Comparing radiologists’ and artificial intelligence performance detecting suspicious microcalcifications on screening mammograms: a pilot cross-sectional study

Introduction: Early breast cancer detection through periodic screening is crucial for reducing mortality due to later stage detection. Microcalcifications are common mammographic findings, present in both malignant lesions, benign pathologies, and normal tissues. This study aimed to assess radiologists’ observer performance in determining suspicious calcifications requiring recall, and to compare reader performance to an in-house AI model trained and tested on Australian screening mammograms. Methods: In this pilot proof-of-concept cross-sectional study, radiologists (n=27), breast physicians (n=2), and final year radiology trainees (n=6), completed the same mammographic test set consisting of 30 mammographic cases displaying different types of calcifications (10 breast cancer, 20 normal/benign). An in-house trained artificial intelligence model (Sydney-GMIC) was also applied to the same test set. Performance between readers was compared to AI via Spearman Rank-Order Correlation test. Work experience and caseload trends were compared using independent T tests and Mann-Whitney-U. Results: Sensitivity was significantly higher in radiologists with ≤ 10 years of experience compared to radiology trainees (72.2% vs 53.3%, p=0.042). Furthermore, readers with higher cases read per week (CPW) (i.e. 101-200 CPW compared to 0-20 CPW) had a decreased specificity (58.8% vs 74.6%, p=0.041), but higher sensitivity (68.8% vs 59.2%, p=0.12). In this pilot dataset, the Sydney-GMIC AI model demonstrated higher sensitivity and specificity than the radiologist means. The cases perceived as difficult by AI differed substantially from those challenging for human readers. Conclusions: This study highlights the challenging nature of recalling calcifications from screening mammograms only, with variable performance among readers. The findings of this small pilot study are exploratory in nature, but the AI performance signals the potential utility of AI models in mammographic analysis of screening cases to progress to recall.