Share Email Print
cover

Proceedings Paper

Toward a fair benchmark for image browsers
Format Member Price Non-Member Price
PDF $14.40 $18.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

The recent literature has shown that the principal difficulty in multimedia retrieval is the bridging of the semantic gap between the user's wishes and his ability to formulate queries. This insight has spawned two main directions of research Query By Example (QBE) with relevance feedback (i.e. learning to improve the result of a previously formulated query) and the research in query formulation techniques, like browsing or query by sketch. Browsing techniques try to help the user in finding his target image, or an image which is sufficiently close to the desired result that it can be used in a subsequent QBE query. From the feature space viewpoint, each browsing system tries to permit the user to move consciously in feature space and eventually reach the target image. How to provide this functionality to the user is presently an open question. In fact even obtaining objective performance evaluation and comparison of these browsing paradigms is difficult. We distinguish between deterministic browsers, which try to optimize the possibility for the user to learn how the system behaves, and stochastic browsers based on more sophisticated Monte-Carlo algorithms thus sacrificing reproducibility to a better performance. Presently, these two browsing paradigms are practically incomparable, except by large scale user studies. This makes it infeasible for research groups to evaluate incremental improvement of browsing schemes. Moreover, automated benchmarks in the current literature simulate a user by a model derived directly from the distance measures used within the tested systems. Such a circular reference cannot provide a serious alternative to real user tests. In this paper, we present an automatic benchmark which uses user- annotated collections for simulating the semantic gap, thus providing a means for automatic evaluation and comparison of the different browsing paradigms. We use a very precise annotation of few words together with a thesaurus to provide sufficiently smooth behavior of the annotation-based user model. We discuss the design and evaluation of this annotation as well as the implementation of the benchmark in n MRML-compliant script with pluggable modules which allow testing of new interaction schemes (Multimedia Retrieval Markup Language).

Paper Details

Date Published: 11 October 2000
PDF: 10 pages
Proc. SPIE 4210, Internet Multimedia Management Systems, (11 October 2000); doi: 10.1117/12.403810
Show Author Affiliations
Wolfgang Mueller, Univ. of Geneva (Germany)
Stephane Marchand-Maillet, Univ. of Geneva (Switzerland)
Henning Mueller, Univ. of Geneva (Switzerland)
Thierry Pun, Univ. of Geneva (Switzerland)


Published in SPIE Proceedings Vol. 4210:
Internet Multimedia Management Systems
John R. Smith; Chinh Le; Sethuraman Panchanathan; C.-C. Jay Kuo, Editor(s)

© SPIE. Terms of Use
Back to Top