Limitations of CNNs for approximating the ideal observer despite quantity of training data or depth of network

Research output: Contribution to journalArticlepeer-review


The performance of a convolutional neural network (CNN) on an image texture detection task as a function of linear image processing and the number of training images is investigated. Performance is quantified by the area under (AUC) the receiver operating characteristic (ROC) curve. The Ideal Observer (IO) maximizes AUC but depends on high-dimensional image likelihoods. In many cases, the CNN performance can approximate the IO performance. This work demonstrates counterexamples where a full-rank linear transform degrades the CNN performance below the IO in the limit of large quantities of training data and network layers. A subsequent linear transform changes the images’ correlation structure, improves the AUC, and again demonstrates the CNN dependence on linear processing. Compression strictly decreases or maintains the IO detection performance while compression can increase the CNN performance especially for small quantities of training data. Results indicate an optimal compression ratio for the CNN based on task difficulty, compression method, and number of training images.

Original languageEnglish (US)
Article number060408
JournalJournal of Imaging Science and Technology
Issue number6
StatePublished - 2020
Externally publishedYes

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Chemistry(all)
  • Atomic and Molecular Physics, and Optics
  • Computer Science Applications


Dive into the research topics of 'Limitations of CNNs for approximating the ideal observer despite quantity of training data or depth of network'. Together they form a unique fingerprint.

Cite this