Modeling the statistics of image features and associated text

Kobus Barnard, Pinar Duygulu, David Forsyth

Research output: Contribution to journalConference articlepeer-review


We present a methodology for modeling the statistics of image features and associated text in large datasets. The models used also serve to cluster the images, as images are modeled as being produced by sampling from a limited number of combinations of mixing components. Furthermore, because our approach models the joint occurrence image features and associated text, it can be used to predict the occurrence of either, based on observations or queries. This supports an attractive approach to image search as well as novel applications such a suggesting illustrations for blocks of text (auto-illustrate) and generating words for images outside the training set (auto-annotate). In this paper we illustrate the approach on 10,000 images of work from the Fine Arts Museum of San Francisco. The images include line drawings, paintings, and pictures of sculpture and ceramics. Many of the images have associated free text whose nature varies greatly, from physical description to interpretation and mood. We incorporate statistical natural language processing in order to deal with free text. We use WordNet to provide semantic grouping information and to help disambiguate word senses, as well as emphasize the hierarchical nature of semantic relationships.

Original languageEnglish (US)
Pages (from-to)1-11
Number of pages11
JournalProceedings of SPIE - The International Society for Optical Engineering
StatePublished - 2002
EventDocumentation Recognition and Retrieval IX - San Jose, CA, United States
Duration: Jan 21 2002Jan 22 2002


  • Aspect model
  • Hierarchical clustering
  • Image retrieval
  • Learning image semantics
  • Recognition

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering


Dive into the research topics of 'Modeling the statistics of image features and associated text'. Together they form a unique fingerprint.

Cite this