Share Email Print

Proceedings Paper

Ground truth and benchmarks for performance evaluation
Author(s): Ayako Takeuchi; Michael Shneier; Tsai Hong Hong; Tommy Chang; Christopher Scrapper; Geraldine S. Cheok
Format Member Price Non-Member Price
PDF $14.40 $18.00

Paper Abstract

Progress in algorithm development and transfer of results to practical applications such as military robotics requires the setup of standard tasks, of standard qualitative and quantitative measurements for performance evaluation and validation. Although the evaluation and validation of algorithms have been discussed for over a decade, the research community still faces a lack of well-defined and standardized methodology. The range of fundamental problems include a lack of quantifiable measures of performance, a lack of data from state-of-the-art sensors in calibrated real-world environments, and a lack of facilities for conducting realistic experiments. In this research, we propose three methods for creating ground truth databases and benchmarks using multiple sensors. The databases and benchmarks will provide researchers with high quality data from suites of sensors operating in complex environments representing real problems of great relevance to the development of autonomous driving systems. At NIST, we have prototyped a High Mobility Multi-purpose Wheeled Vehicle (HMMWV) system with a suite of sensors including a Riegl ladar, GDRS ladar, stereo CCD, several color cameras, Global Position System (GPS), Inertial Navigation System (INS), pan/tilt encoders, and odometry . All sensors are calibrated with respect to each other in space and time. This allows a database of features and terrain elevation to be built. Ground truth for each sensor can then be extracted from the database. The main goal of this research is to provide ground truth databases for researchers and engineers to evaluate algorithms for effectiveness, efficiency, reliability, and robustness, thus advancing the development of algorithms.

Paper Details

Date Published: 30 September 2003
PDF: 6 pages
Proc. SPIE 5083, Unmanned Ground Vehicle Technology V, (30 September 2003); doi: 10.1117/12.485914
Show Author Affiliations
Ayako Takeuchi, National Institute of Standards and Technology (United States)
Michael Shneier, National Institute of Standards and Technology (United States)
Tsai Hong Hong, National Institute of Standards and Technology (United States)
Tommy Chang, National Institute of Standards and Technology (United States)
Christopher Scrapper, National Institute of Standards and Technology (United States)
Geraldine S. Cheok, National Institute of Standards and Technology (United States)

Published in SPIE Proceedings Vol. 5083:
Unmanned Ground Vehicle Technology V
Grant R. Gerhart; Charles M. Shoemaker; Douglas W. Gage, Editor(s)

© SPIE. Terms of Use
Back to Top