Share Email Print
cover

Proceedings Paper

Semi-parametric estimation of the area under the precision-recall curve
Author(s): Berkman Sahiner; Weijie Chen; Aria Pezeshk; Nicholas Petrick
Format Member Price Non-Member Price
PDF $14.40 $18.00

Paper Abstract

Precision and recall are two common metrics used in the evaluation of information retrieval systems. By changing the number of retrieved documents, one can obtain a precision-recall curve. The area under the precision-recall curve (AUCPR) has been suggested as a performance measure for information retrieval systems, in a manner similar to the use of the area under the receiver operating characteristic curve in binary classification. Limited work has been performed in the literature to investigate the bias and variance of AUCPR estimators. The goal of our study was to investigate the bias and variability of a semi-parametric binormal method for estimating the AUCPR, and to compare it to other techniques, such as average precision (AP) and lower trapezoid (LT) approximation. We show how AUCPR can be obtained given the binormal model parameters, and how its variance can be estimated using the delta method. We performed simulation experiments with normal and non-normal data, and investigated the effect of sample size and prevalence. Our results indicated that the semi-parametric binormal approach provided AUCPR estimates with small bias and confidence intervals with acceptable coverage when the sample size was large, and the performance of the binormal model was comparable to or better than alternative methods evaluated in this study when the sample size was small. We conclude that the semi-parametric binormal model can be used to accurately estimate the AUCPR, and that the confidence intervals derived from the model can be at least as accurate as from other alternatives, even for non-normal decision variable distributions.

Paper Details

Date Published: 24 March 2016
PDF: 7 pages
Proc. SPIE 9787, Medical Imaging 2016: Image Perception, Observer Performance, and Technology Assessment, 97870D (24 March 2016); doi: 10.1117/12.2216434
Show Author Affiliations
Berkman Sahiner, U.S. Food and Drug Administration (United States)
Weijie Chen, U.S. Food and Drug Administration (United States)
Aria Pezeshk, U.S. Food and Drug Administration (United States)
Nicholas Petrick, U.S. Food and Drug Administration (United States)


Published in SPIE Proceedings Vol. 9787:
Medical Imaging 2016: Image Perception, Observer Performance, and Technology Assessment
Craig K. Abbey; Matthew A. Kupinski, Editor(s)

© SPIE. Terms of Use
Back to Top