Share Email Print
cover

Proceedings Paper

Metrics for the comparative evaluation of chemical plume identification algorithms
Author(s): E. Truslow; S. Golowich; D. Manolakis; V. K. Ingle
Format Member Price Non-Member Price
PDF $14.40 $18.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

The detection of chemical agents with hyperspectral longwave infrared sensors is a difficult problem with many civilian and military applications. System performance can be evaluated by comparing the detected gases in each pixel with the ground truth for each pixel using a confusion matrix. In the presence of chemical mixtures the confusion matrix becomes extremely large and difficult to interpret due to its size. We propose summarizing the confusion matrix using simple scalar metrics tailored for specific applications. Ideally, an identifier should determine exactly which chemicals are in each pixel, but in many applications it is acceptable for the output to contain additional chemicals or lack some constituent chemicals. A performance metric for identification problems should give partially correct results a lower weight than completely correct results. The metric we propose using, the Dice metric, weighs each output by its similarity with the truth for each pixel, thereby giving less importance to partially correct outputs, while still giving full scores only to exactly correct results. Using the Dice metric we evaluated the performance of two identification algorithms: an adaptive cosine estimator (ACE) detector bank approach, and Bayesian model averaging (BMA). Both algorithms were tested individually on real background data with synthetically embedded plumes; performance was evaluated using standard detection performance metrics, and then using the proposed identification metric. We show that ACE performed well as a detector but poorly as an identifier; however, BMA performed poorly as a detector but well as an identifier. Cascading the two algorithms should lead to a system with a substantially lower false alarm rate than using BMA alone, and much better identification performance than the ACE detector bank alone.

Paper Details

Date Published: 21 May 2015
PDF: 10 pages
Proc. SPIE 9472, Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXI, 947203 (21 May 2015); doi: 10.1117/12.2177163
Show Author Affiliations
E. Truslow, Northeastern Univ. (United States)
S. Golowich, MIT Lincoln Lab. (United States)
D. Manolakis, MIT Lincoln Lab. (United States)
V. K. Ingle, Northeastern Univ. (United States)


Published in SPIE Proceedings Vol. 9472:
Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXI
Miguel Velez-Reyes; Fred A. Kruse, Editor(s)

© SPIE. Terms of Use
Back to Top