Share Email Print
cover

Proceedings Paper

Evaluation methodology for query-based scene understanding systems
Author(s): Todd P. Huster; Timothy D. Ross; Jared L. Culbertson
Format Member Price Non-Member Price
PDF $14.40 $18.00

Paper Abstract

In this paper, we are proposing a method for the principled evaluation of scene understanding systems in a query-based framework. We can think of a query-based scene understanding system as a generalization of typical sensor exploitation systems where instead of performing a narrowly defined task (e.g., detect, track, classify, etc.), the system can perform general user-defined tasks specified in a query language. Examples of this type of system have been developed as part of DARPA’s Mathematics of Sensing, Exploitation, and Execution (MSEE) program. There is a body of literature on the evaluation of typical sensor exploitation systems, but the open-ended nature of the query interface introduces new aspects to the evaluation problem that have not been widely considered before. In this paper, we state the evaluation problem and propose an approach to efficiently learn about the quality of the system under test. We consider the objective of the evaluation to be to build a performance model of the system under test, and we rely on the principles of Bayesian experiment design to help construct and select optimal queries for learning about the parameters of that model.

Paper Details

Date Published: 22 May 2015
PDF: 11 pages
Proc. SPIE 9476, Automatic Target Recognition XXV, 947605 (22 May 2015); doi: 10.1117/12.2177007
Show Author Affiliations
Todd P. Huster, Air Force Research Lab. (United States)
Timothy D. Ross, Air Force Research Lab. (United States)
Jared L. Culbertson, Air Force Research Lab. (United States)


Published in SPIE Proceedings Vol. 9476:
Automatic Target Recognition XXV
Firooz A. Sadjadi; Abhijit Mahalanobis, Editor(s)

© SPIE. Terms of Use
Back to Top