Inter-rater concordance in the classification of COVID-19 in chest X-ray images using the RANZCR template for COVID-19 infection

Introduction: The Royal Australian and New Zealand College of Radiologists (RANZCR) has developed a reporting template to assist in the categorization of COVID-19 in chest X-ray (CXR) images and the levels of COVID-19 infection. Whilst CXRs are reported by radiologists, radiographers are often the first to assess the CXRs, and have the potential to support immediate triaging of patients with COVID-19. However, inter-reader concordance in the use of this reporting template remains underexplored. Methods: 70 CXR examinations comprising of the four categories in the RANZCR chest X-ray (CXR) COVID-19 reporting template were used for the study. These included: ‘typical’ (for COVID-19) (n = 30); ‘indeterminate’ (for COVID-19) (n = 20); ‘other diagnoses favoured’ (n = 10) and ‘normal’ (n = 10). These images were independently categorised using the RANZCR reporting template by three cohorts of readers: 12 radiologists, 13 registered radiographers, and 12 final-year radiography students. A Weighted Kappa (κ_w) was used to evaluate inter-reader agreement within and between the cohort of readers. Results: Radiologists demonstrated fair (κ_w = 0.32) to substantial (κ_w = 0.77) inter-reader agreement, and their overall inter-reader was moderate (κ_w = 0.56). Registered radiographers demonstrated no (κ_w = -0.01) to moderate agreement (κ_w = 0.59), and their overall agreement was fair (κ_w = 0.31). Fourth year student radiographers demonstrated slight (κ_w = 0.004) to substantial (κ_{w =} 0.8) agreement, with a moderate (κ_w = 0.47) overall agreement among final year student radiographers. Conclusion: There are wide variations in the classification of the CXRs using the RANZCR reporting template. Overall, radiologists exhibit superior concordance in CXR categorization using the COVID-19 reporting template. Radiographers demonstrate wide variability, highlighting the need for enhanced education and training to standardise the triaging of these patients undergoing CXR imaging for COVID-19 symptoms.