Visual search behaviour of readers interpreting mammograms with known different difficulty for humans and AI

This paper investigates visual search behaviour and diagnostic performance of observers with varying levels of experience and specialisation when interpreting screening mammograms of differing difficulty classified by a Correct Classification Rate (CCR) and malignant probability score from an Artificial Intelligence (AI) model. The CCR of each mammographic case was calculated from interpretations of at least 73 radiologists. We used the Sydney-GMIC AI model to generate malignant probability on the mammograms. The case collection included 60 mammograms that were classified as easy, intermediate or difficult for either the radiologists, the AI, or both. Twenty human observers (14 radiologists, 4 radiology trainees and 2 breast physicians) read the same set of cases on a dedicated mammogram workstation and marked abnormal lesions using the BREAST software. Eye movements of the observers were recorded using a Gazepoint GP3 eye tracker. The radiologists achieved the highest average sensitivity of 72.9% and lesion sensitivity of 66.7%, followed by the radiology trainees and breast physicians. While radiology trainees and breast physicians obtained the same AUC, both were lower than that of the radiologists. Average fixation duration was shortest for cases rated “easy” for both AI and radiologists, followed by cases rated “difficult” for AI but “easy” by radiologists, “intermediate” for both, “easy” for AI but “difficult” by radiologists, and longest for cases rated “difficult” for both. Statistically significant difference of fixation duration was found among groups of radiologists, radiology trainees, and breast physicians with different levels of case difficulty.