Use of reporting templates for chest radiographs in a coronavirus disease 2019 context: measuring concordance of radiologists with three international templates

Purpose: Reporting templates for chest radiographs (CXRs) for patients presenting or being clinically managed for severe acute respiratory syndrome coronavirus 2 [coronavirus disease 2019 (COVID-19)] has attracted advocacy from international radiology societies. We aim to explore the effectiveness and useability of three international templates through the concordance of, and between, radiologists reporting on the presence and severity of COVID-19 on CXRs. Approach: Seventy CXRs were obtained from a referral hospital, 50 from patients with COVID-19 (30 rated "classic"COVID-19 appearance and 20 "indeterminate") and 10 "normal"and 10 "alternative pathology"CXRs. The recruited radiologists were assigned to three test sets with the same CXRs but with different template orders. Each radiologist read their test set three times and assigned a classification to the CXR using the Royal Australian New Zealand College of Radiology (RANZCR), British Society of Thoracic Imaging (BSTI), and Modified COVID-19 Reporting and Data System (Dutch; mCO-RADS) templates. Inter-reader variability and intra-reader variability were measured using Fleiss' kappa coefficient. Results: Twelve Australian radiologists participated. The BSTI template had the highest inter-reader agreement (0.46; "moderate"agreement), followed by RANZCR (0.45) and mCO-RADS (0.32). Concordance was driven by strong agreement in "normal"and "alternative"classifications and was lowest for "indeterminate."General consistency was observed across classifications and templates, with intrareader variability ranging from "good"to "very good"for COVID-19 CXRs (0.61), "normal"CXRs (0.76), and "alternative"(0.68). Conclusions: Reporting templates may be useful in reducing variation among radiology reports, with intra-reader variability showing promise. Feasibility and implementation require a wider approach including referring and treating doctors plus the development of training packages for radiologists specific to the template being used.