Share Email Print

Proceedings Paper

Benchmarking without ground truth
Author(s): Simone Santini
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

Many evaluation techniques for content based image retrieval are based on the availability of a ground truth, that is on a "correct" categorization of images so that, say, if the query image is of category A, only the returned images in category A will be considered as "hits." Based on such a ground truth, standard information retrieval measures such as precision and recall and given and used to evaluate and compare retrieval algorithms. Coherently, the assemblers of benchmarking data bases go to a certain length to have their images categorized. The assumption of the existence of a ground truth is, in many respect, naive. It is well known that the categorization of the images depends on the a priori (from the point of view of such categorization) subdivision of the semantic field in which the images are placed (a trivial observation: a plant subdivision for a botanist is very different from that for a layperson). Even within a given semantic field, however, categorization by human subjects is subject to uncertainty, and it makes little statistical sense to consider the categorization given by one person as the unassailable ground truth. In this paper I propose two evaluation techniques that apply to the case in which the ground truth is subject to uncertainty. In this case, obviously, measures such as precision and recall as well will be subject to uncertainty. The paper will explore the relation between the uncertainty in the ground truth and that in the most commonly used evaluation measures, so that the measurements done on a given system can preserve statistical significance.

Paper Details

Date Published: 16 January 2006
PDF: 10 pages
Proc. SPIE 6061, Internet Imaging VII, 60610I (16 January 2006); doi: 10.1117/12.655400
Show Author Affiliations
Simone Santini, Univ. Autónoma de Madrid (Spain)

Published in SPIE Proceedings Vol. 6061:
Internet Imaging VII
Simone Santini; Raimondo Schettini; Theo Gevers, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?