Reliability of participant classification in sport and exercise science: application of McKay et al.’s (2022) framework

Accurately classifying samples within sports and exercise science (SES) research has significant implications for how findings are interpreted and applied. Key to this is clear and sufficiently detailed “Participants” sections of manuscripts and frameworks that provide structure for the classification process. The primary aim of this study was to evaluate the inter- and intra-rater reliability of sample classifications made by four experienced academics who applied McKay et al’.s (2022) Participant Classification Framework (PCF) to 130 SES manuscripts. Weighted Cohen’s kappa analyses found inter-rater reliabilities ranging from 0.34 (fair agreement) to 0.74 (substantial), and intra-rater reliabilities ranging from 0.54 (moderate) to 0.90 (almost perfect), evidencing strong internal reliability and reproducible PCF classifications. Tier “0” papers had the highest inter-rater agreement, whilst “Tier 5” and papers with multiple classifications had the lowest. Studies that failed to report sample size and sport type were more frequently classified as “unclear”, whilst ambiguous sex distribution also proved problematic. The findings suggest that current participant reporting standards in the field are insufficient to support consistent application of the PCF. To facilitate the future utility of the PCF and improve the clarity and comparability of SES research, we propose nine “Key Criteria for Classifying SES Research Samples”.