Share Email Print
cover

Proceedings Paper

Comparison of classifier performance estimators: a simulation study
Author(s): Weijie Chen; Robert F. Wagner; Waleed A. Yousef; Brandon D. Gallas
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

We aim to compare resampling-based estimators of the area under the ROC curve (AUC) of a classifier with a Monte Carlo simulation study. The comparison is in terms of bias, variance, and mean square error. We also examine the corresponding variance estimators of these AUC estimators. We compared three AUC estimators: the hold-out (HO) estimator, the leave-one-out cross validation (LOOCV) estimator, and the leave-pair-out bootstrap (LPOB) estimator. Each performance estimator has its own variability estimator. In our simulations, in terms of the mean square error, HO is always the worst and the ranking of the other two estimators depends on the interplay of sample size, dimensionality, and the population separability. In terms of estimator variability, the LPOB is the least variable estimator and the HO is the most variable estimator. The results also show that the estimation of the variance of LPOB using the influence function approach with a finite data set is unbiased or conservatively biased whereas the estimation of the variance of the LOOCV or the HO is downwardly (i.e., anti-conservatively) biased.

Paper Details

Date Published: 12 March 2009
PDF: 11 pages
Proc. SPIE 7263, Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment, 72630X (12 March 2009); doi: 10.1117/12.811584
Show Author Affiliations
Weijie Chen, CDRH, U.S. Food and Drug Administration (United States)
Robert F. Wagner, CDRH, U.S. Food and Drug Administration (United States)
Waleed A. Yousef, CDRH, U.S. Food and Drug Administration (United States)
Brandon D. Gallas, CDRH, U.S. Food and Drug Administration (United States)


Published in SPIE Proceedings Vol. 7263:
Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment
Berkman Sahiner; David J. Manning, Editor(s)

© SPIE. Terms of Use
Back to Top