Share Email Print

Proceedings Paper

Information theoretic approach for performance evaluation of multi-class assignment systems
Author(s): Ryan S. Holt; Peter A. Mastromarino; Edward K. Kao; Michael B. Hurley
Format Member Price Non-Member Price
PDF $14.40 $18.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

Multi-class assignment is often used to aid in the exploitation of data in the Intelligence, Surveillance, and Reconnaissance (ISR) community. For example, tracking systems collect detections into tracks and recognition systems classify objects into various categories. The reliability of these systems is highly contingent upon the correctness of the assignments. Conventional methods and metrics for evaluating assignment correctness only convey partial information about the system performance and are usually tied to the specific type of system being evaluated. Recently, information theory has been successfully applied to the tracking problem in order to develop an overall performance evaluation metric. In this paper, the information-theoretic framework is extended to measure the overall performance of any multiclass assignment system, specifically, any system that can be described using a confusion matrix. The performance is evaluated based upon the amount of truth information captured and the amount of false information reported by the system. The information content is quantified through conditional entropy and mutual information computations using numerical estimates of the association probabilities. The end result is analogous to the Receiver Operating Characteristic (ROC) curve used in signal detection theory. This paper compares these information quality metrics to existing metrics and demonstrates how to apply these metrics to evaluate the performance of a recognition system.

Paper Details

Date Published: 27 April 2010
PDF: 12 pages
Proc. SPIE 7697, Signal Processing, Sensor Fusion, and Target Recognition XIX, 76970R (27 April 2010); doi: 10.1117/12.851019
Show Author Affiliations
Ryan S. Holt, MIT Lincoln Lab. (United States)
Peter A. Mastromarino, MIT Lincoln Lab. (United States)
Edward K. Kao, MIT Lincoln Lab. (United States)
Michael B. Hurley, MIT Lincoln Lab. (United States)

Published in SPIE Proceedings Vol. 7697:
Signal Processing, Sensor Fusion, and Target Recognition XIX
Ivan Kadar, Editor(s)

© SPIE. Terms of Use
Back to Top