Share Email Print

Proceedings Paper

Discrimination and confidence error in detector-reported scores
Author(s): Timothy D. Ross; Mark E Minardi
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

A decision system may compute a score that reflects its confidence in one of its decisions. This paper considers methods for evaluating such scores. There is a class of measures-of-performance (MOPs) for each of the score's two roles: discrimination (how well it separates targets and clutter) and confidence (how well it predicts its own accuracy). Area-under-the-ROC and probability of error are considered as discrimination MOPs. Error in the posterior (EP) and normalized cross entropy (NCE) are considered as confidence MOPs. MOPs for the scores are assessed using Monte Carlo simulations where known score distributions are sampled, allowing comparison of true and estimated MOPs. Classical data-direct ROC estimates are found to be equivalent to those based on explicit distribution estimation using probability mass functions (pmfs). An alternative distribution estimation based on histograms is recommended for empirical ROCs, being accurate and avoiding the unnatural stair-step character of data-direct ROCs. Confidence MOPs are more difficult to estimate than discrimination MOPs and NCE estimates are especially poor. EP may be meaningfully estimated through histogram density estimates and it is recommended as a replacement for the AdaptSAPS binned EP confidence MOP.

Paper Details

Date Published: 2 September 2004
PDF: 12 pages
Proc. SPIE 5427, Algorithms for Synthetic Aperture Radar Imagery XI, (2 September 2004); doi: 10.1117/12.542161
Show Author Affiliations
Timothy D. Ross, Air Force Research Lab. (United States)
Mark E Minardi, Air Force Research Lab. (United States)

Published in SPIE Proceedings Vol. 5427:
Algorithms for Synthetic Aperture Radar Imagery XI
Edmund G. Zelnio; Frederick D. Garber, Editor(s)

© SPIE. Terms of Use
Back to Top