Share Email Print

Proceedings Paper

Statistical properties of a utility measure of observer performance compared to area under the ROC curve
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

The receiver operating characteristic (ROC) curve has become a common tool for evaluating diagnostic imaging technologies, and the primary endpoint of such evaluations is the area under the curve (AUC), which integrates sensitivity over the entire false positive range. An alternative figure of merit for ROC studies is expected utility (EU), which focuses on the relevant region of the ROC curve as defined by disease prevalence and the relative utility of the task. However if this measure is to be used, it must also have desirable statistical properties keep the burden of observer performance studies as low as possible. Here, we evaluate effect size and variability for EU and AUC. We use two observer performance studies recently submitted to the FDA to compare the EU and AUC endpoints. The studies were conducted using the multi-reader multi-case methodology in which all readers score all cases in all modalities. ROC curves from the study were used to generate both the AUC and EU values for each reader and modality. The EU measure was computed assuming an iso-utility slope of 1.03. We find mean effect sizes, the reader averaged difference between modalities, to be roughly 2.0 times as big for EU as AUC. The standard deviation across readers is roughly 1.4 times as large, suggesting better statistical properties for the EU endpoint. In a simple power analysis of paired comparison across readers, the utility measure required 36% fewer readers on average to achieve 80% statistical power compared to AUC.

Paper Details

Date Published: 28 March 2013
PDF: 9 pages
Proc. SPIE 8673, Medical Imaging 2013: Image Perception, Observer Performance, and Technology Assessment, 86730D (28 March 2013); doi: 10.1117/12.2007708
Show Author Affiliations
Craig K. Abbey, Univ. of California, Santa Barbara (United States)
UC Davis Medical Ctr. (United States)
Frank W. Samuelson, U.S. Food and Drug Administration (United States)
Brandon D. Gallas, U.S. Food and Drug Administration (United States)
John M. Boone, UC Davis Medical Ctr. (United States)
Loren T. Niklason, Hologic, Inc. (United States)

Published in SPIE Proceedings Vol. 8673:
Medical Imaging 2013: Image Perception, Observer Performance, and Technology Assessment
Craig K. Abbey; Claudia R. Mello-Thoms, Editor(s)

© SPIE. Terms of Use
Back to Top