Investigating the link between radiologists' gaze, diagnostic decision, and image content

Georgia Tourassi, Sophie Voisin, Vincent Paquit, Elizabeth Krupinski

Research output: Contribution to journalArticlepeer-review

44 Scopus citations


Objective: To investigate machine learning for linking image content, human perception, cognition, and error in the diagnostic interpretation of mammograms. Methods: Gaze data and diagnostic decisions were collected from three breast imaging radiologists and three radiology residents who reviewed 20 screening mammograms while wearing a head-mounted eyetracker. Image analysis was performed in mammographic regions that attracted radiologists' attention and in all abnormal regions. Machine learning algorithms were investigated to develop predictive models that link: (i) image content with gaze, (ii) image content and gaze with cognition, and (iii) image content, gaze, and cognition with diagnostic error. Both group-based and individualized models were explored. Results: By pooling the data from all readers, machine learning produced highly accurate predictive models linking image content, gaze, and cognition. Potential linking of those with diagnostic error was also supported to some extent. Merging readers' gaze metrics and cognitive opinions with computer-extracted image features identified 59% of the readers' diagnostic errors while confirming 97.3% of their correct diagnoses. The readers' individual perceptual and cognitive behaviors could be adequately predicted by modeling the behavior of others. However, personalized tuning was in many cases beneficial for capturing more accurately individual behavior. Conclusions: There is clearly an interaction between radiologists' gaze, diagnostic decision, and image content which can be modeled with machine learning algorithms.

Original languageEnglish (US)
Pages (from-to)1067-1075
Number of pages9
JournalJournal of the American Medical Informatics Association
Issue number6
StatePublished - 2013

ASJC Scopus subject areas

  • Health Informatics


Dive into the research topics of 'Investigating the link between radiologists' gaze, diagnostic decision, and image content'. Together they form a unique fingerprint.

Cite this