Share Email Print

Proceedings Paper

A scoring, truthing, and registration toolkit for evaluation of target detection and tracking
Author(s): John Irvine; Scott K. Ralph; Mark R. Stevens; Sam Kenyon; David M. Anderson; Magnus S. Snorrason; David Gwilt
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

A major challenge for ATR evaluation is developing an accurate image truth that can be compared to an ATR algorithm's decisions to assess performance. While many standard truthing methods and scoring metrics exist for stationary targets in still imagery, techniques for dealing with motion imagery and moving targets are not as prevalent. This is partially because the moving imagery / moving targets scenario introduces the data association problem of assigning targets to tracks. This problem complicates the truthing and scoring task in two ways. First, video datasets typically contain far more imagery that must be truthed than static collections. Specifying the types and locations of the targets present for a large number of images is tedious, time consuming and error prone. Second, scoring ATR performance is ambiguous when assessing performance over a collection of video sequences. For example, if a target is tracked and successfully identified for 90% of a single video sequence, is the identification rate 90%, or is the single sequence evaluated in its entirety and the vehicle identification simply recorded as correct? In the former case, a bias will be introduced for easily identified targets that show up frequently in a sequence. In the latter case, the bias is avoided but system accuracy could be overstated. In this paper, we present a complete truthing system we call the Scoring, Truthing, And Registration Toolkit (START). The first component is registration, which involves aligning the images of the same scene to a common reference frame. Once that reference frame has been determined, the second component, truthing, is used to specify target identity, posi-tion, orientation, and other scene characteristics. The final component, scoring, is used to assess the performance of a given algorithm as compared to the specified truth. In motion imagery, both stationary and moving targets can be de-tected and tracked over portions of a motion imagery clip. We present an approach to scoring performance in the context that provides a natural generalization of the standard methods for dealing with still imagery.

Paper Details

Date Published: 21 September 2004
PDF: 10 pages
Proc. SPIE 5426, Automatic Target Recognition XIV, (21 September 2004); doi: 10.1117/12.542681
Show Author Affiliations
John Irvine, Science Applications International Corp. (United States)
Scott K. Ralph, Charles River Analytics Inc. (United States)
Mark R. Stevens, Charles River Analytics Inc. (United States)
Sam Kenyon, Charles River Analytics Inc. (United States)
David M. Anderson, Charles River Analytics Inc. (United States)
Magnus S. Snorrason, Charles River Analytics Inc. (United States)
David Gwilt, Air Force Research Lab. (United States)

Published in SPIE Proceedings Vol. 5426:
Automatic Target Recognition XIV
Firooz A. Sadjadi, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?