Share Email Print

Proceedings Paper

A confidence paradigm for classification systems
Author(s): Nathan J. Leap; Kenneth W. Bauer Jr.
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

There is no universally accepted methodology to determine how much confidence one should have in a classifier output. This research proposes a framework to determine the level of confidence in an indication from a classifier system where the output is a measurement value. There are two types of confidence developed in this paper. The first is confidence in a classification system or classifier and is denoted classifier confidence. The second is the confidence in the output of a classification system or classifier. In this paradigm, we posit that the confidence in the output of a classifier should be, on average, equal to the confidence in the classifier as a whole (i.e., classifier confidence). The amount of confidence in a given classifier is estimated using multiattribute preference theory and forms the foundation for a quadratic confidence function that is applied to posterior probability estimates. Classifier confidence is currently determined based upon individual measurable value functions for classification accuracy, average entropy, and sample size, and the form of the overall measurable value function is multilinear based upon the assumption of weak difference independence. Using classifier confidence, a quadratic function is trained to be the confidence function which inputs a posterior probability and outputs the confidence in a given indication. In this paradigm, confidence is not equal to the posterior probability estimate but is related to it. This confidence measure is a direct link between traditional decision analysis techniques and traditional pattern recognition techniques. This methodology is applied to two real world data sets, and results show the sort of behavior that would be expected from a rational confidence measure.

Paper Details

Date Published: 17 April 2008
PDF: 12 pages
Proc. SPIE 6968, Signal Processing, Sensor Fusion, and Target Recognition XVII, 69680U (17 April 2008); doi: 10.1117/12.776755
Show Author Affiliations
Nathan J. Leap, Air Force Institute of Technology (United States)
Kenneth W. Bauer Jr., Air Force Institute of Technology (United States)

Published in SPIE Proceedings Vol. 6968:
Signal Processing, Sensor Fusion, and Target Recognition XVII
Ivan Kadar, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?