Share Email Print
cover

Proceedings Paper

Training variability in the evaluation of automated classifiers
Format Member Price Non-Member Price
PDF $14.40 $18.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

The evaluation of automated classifiers in computer-aided diagnosis of medical images often involves a training dataset for classifier design and a test dataset for performance estimation in terms of, e.g., area under the receiver operating characteristic (ROC) curve, or AUC. The traditional approach to assess the uncertainty of the estimated AUC only considers the finite testing set as the source of variability. However, a finite training set is also a random sample and the AUC varies with varying training sets. We categorize the assessment of classifiers into three levels and provide analytical expressions for the variance of the estimated AUC at each level: (1) training treated as a fixed effect, the estimated performance generalizable only to the population of testing sets; (2) training treated as a random effect, the estimated performance generalizable to both the population of training sets and the population of testing sets; (3) training treated as a random effect, performance averaged over training sets generalizable to both the population of training sets and the population of testing sets. The two sources of variability - training and testing - in automated classifiers are analogous to readers and cases in the multi-reader multi-case (MRMC) ROC paradigm in reader studies. We show the one-to-one analogy between the automated classifiers and human readers at these three levels as well as the practical difference in estimating their performance, especially regarding variance.

Paper Details

Date Published: 9 March 2010
PDF: 9 pages
Proc. SPIE 7624, Medical Imaging 2010: Computer-Aided Diagnosis, 762404 (9 March 2010); doi: 10.1117/12.844605
Show Author Affiliations
Weijie Chen, U.S. Food and Drug Administration (United States)
Brandon D. Gallas, U.S. Food and Drug Administration (United States)


Published in SPIE Proceedings Vol. 7624:
Medical Imaging 2010: Computer-Aided Diagnosis
Nico Karssemeijer; Ronald M. Summers, Editor(s)

© SPIE. Terms of Use
Back to Top