Evaluating image retrieval

Nikhil V. Shirahatti; Kobus Barnard

doi:10.1109/CVPR.2005.147

Evaluating image retrieval

Nikhil V. Shirahatti, Kobus Barnard

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

36 Scopus citations

Abstract

We present a comprehensive strategy for evaluating image retrieval algorithms. Because automated image retrieval is only meaningful in its service to people, performance characterization must be grounded in human evaluation. Thus we have collected a large data set of human evaluations of retrieval results, both for query by image example and query by text. The data is independent of any particular image retrieval algorithm and can be used to evaluate and compare many such algorithms without further data collection. The data and calibration software are available on-line (http://kobus.ca/research/data). We develop and validate methods for generating sensible evaluation data, calibrating for disparate evaluators, mapping image retrieval system scores to the human evaluation results, and comparing retrieval systems. We demonstrate the process by providing grounded comparison results for several algorithms.

Original language	English (US)
Title of host publication	Proceedings - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005
Publisher	IEEE Computer Society
Pages	955-961
Number of pages	7
ISBN (Print)	0769523722, 9780769523729
DOIs	https://doi.org/10.1109/CVPR.2005.147
State	Published - 2005
Event	2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005 - San Diego, CA, United States Duration: Jun 20 2005 → Jun 25 2005

Publication series

Name	Proceedings - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005
Volume	I

Other

Other	2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005
Country/Territory	United States
City	San Diego, CA
Period	6/20/05 → 6/25/05

ASJC Scopus subject areas

General Engineering

Access to Document

10.1109/CVPR.2005.147

Cite this

Shirahatti, N. V., & Barnard, K. (2005). Evaluating image retrieval. In Proceedings - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005 (pp. 955-961). Article 1467369 (Proceedings - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005; Vol. I). IEEE Computer Society. https://doi.org/10.1109/CVPR.2005.147

Evaluating image retrieval. / Shirahatti, Nikhil V.; Barnard, Kobus.
Proceedings - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005. IEEE Computer Society, 2005. p. 955-961 1467369 (Proceedings - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005; Vol. I).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Shirahatti, NV & Barnard, K 2005, Evaluating image retrieval. in Proceedings - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005., 1467369, Proceedings - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. I, IEEE Computer Society, pp. 955-961, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, San Diego, CA, United States, 6/20/05. https://doi.org/10.1109/CVPR.2005.147

@inproceedings{82dd32b84cd74de8be1ad55c3cd8cdfb,

title = "Evaluating image retrieval",

abstract = "We present a comprehensive strategy for evaluating image retrieval algorithms. Because automated image retrieval is only meaningful in its service to people, performance characterization must be grounded in human evaluation. Thus we have collected a large data set of human evaluations of retrieval results, both for query by image example and query by text. The data is independent of any particular image retrieval algorithm and can be used to evaluate and compare many such algorithms without further data collection. The data and calibration software are available on-line (http://kobus.ca/research/data). We develop and validate methods for generating sensible evaluation data, calibrating for disparate evaluators, mapping image retrieval system scores to the human evaluation results, and comparing retrieval systems. We demonstrate the process by providing grounded comparison results for several algorithms.",

author = "Shirahatti, {Nikhil V.} and Kobus Barnard",

year = "2005",

doi = "10.1109/CVPR.2005.147",

language = "English (US)",

isbn = "0769523722",

series = "Proceedings - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005",

publisher = "IEEE Computer Society",

pages = "955--961",

booktitle = "Proceedings - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005",

note = "2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005 ; Conference date: 20-06-2005 Through 25-06-2005",

}

TY - GEN

T1 - Evaluating image retrieval

AU - Shirahatti, Nikhil V.

AU - Barnard, Kobus

PY - 2005

Y1 - 2005

N2 - We present a comprehensive strategy for evaluating image retrieval algorithms. Because automated image retrieval is only meaningful in its service to people, performance characterization must be grounded in human evaluation. Thus we have collected a large data set of human evaluations of retrieval results, both for query by image example and query by text. The data is independent of any particular image retrieval algorithm and can be used to evaluate and compare many such algorithms without further data collection. The data and calibration software are available on-line (http://kobus.ca/research/data). We develop and validate methods for generating sensible evaluation data, calibrating for disparate evaluators, mapping image retrieval system scores to the human evaluation results, and comparing retrieval systems. We demonstrate the process by providing grounded comparison results for several algorithms.

AB - We present a comprehensive strategy for evaluating image retrieval algorithms. Because automated image retrieval is only meaningful in its service to people, performance characterization must be grounded in human evaluation. Thus we have collected a large data set of human evaluations of retrieval results, both for query by image example and query by text. The data is independent of any particular image retrieval algorithm and can be used to evaluate and compare many such algorithms without further data collection. The data and calibration software are available on-line (http://kobus.ca/research/data). We develop and validate methods for generating sensible evaluation data, calibrating for disparate evaluators, mapping image retrieval system scores to the human evaluation results, and comparing retrieval systems. We demonstrate the process by providing grounded comparison results for several algorithms.

UR - http://www.scopus.com/inward/record.url?scp=33745181962&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33745181962&partnerID=8YFLogxK

U2 - 10.1109/CVPR.2005.147

DO - 10.1109/CVPR.2005.147

M3 - Conference contribution

AN - SCOPUS:33745181962

SN - 0769523722

SN - 9780769523729

T3 - Proceedings - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005

SP - 955

EP - 961

BT - Proceedings - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005

PB - IEEE Computer Society

T2 - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005

Y2 - 20 June 2005 through 25 June 2005

ER -

Evaluating image retrieval

Abstract

Publication series

Other

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this