Share Email Print
cover

Proceedings Paper

Impact of different study populations on reader behavior and performance metrics: initial results
Author(s): Brandon D. Gallas; Etta Pisano; Elodia Cole; Kyle Myers
Format Member Price Non-Member Price
PDF $14.40 $18.00

Paper Abstract

The FDA recently completed a study on design methodologies surrounding the Validation of Imaging Premarket Evaluation and Regulation called VIPER. VIPER consisted of five large reader sub-studies to compare the impact of different study populations on reader behavior as seen by sensitivity, specificity, and AUC, the area under the ROC curve (receiver operating characteristic curve). The study investigated different prevalence levels and two kinds of sampling of non-cancer patients: a screening population and a challenge population. The VIPER study compared full-field digital mammography (FFDM) to screenfilm mammography (SFM) for women with heterogeneously dense or extremely dense breasts. All cases and corresponding images were sampled from Digital Mammographic Imaging Screening Trial (DMIST) archives. There were 20 readers (American Board Certified radiologists) for each sub-study, and instead of every reader reading every case (fully-crossed study), readers and cases were split into groups to reduce reader workload and the total number of observations (split-plot study). For data collection, readers first decided whether or not they would recall a patient. Following that decision, they provided an ROC score for how close or far that patient was from the recall decision threshold. Performance results for FFDM show that as prevalence increases to 50%, there is a moderate increase in sensitivity and decrease in specificity, whereas AUC is mainly flat. Regarding precision, the statistical efficiency (ratio of variances) of sensitivity and specificity relative to AUC are 0.66 at best and decrease with prevalence. Analyses comparing modalities and the study populations (screening vs. challenge) are still ongoing.

Paper Details

Date Published: 10 March 2017
PDF: 8 pages
Proc. SPIE 10136, Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment, 101360A (10 March 2017); doi: 10.1117/12.2255977
Show Author Affiliations
Brandon D. Gallas, U.S. Food and Drug Administration (United States)
Etta Pisano, Beth Israel Deaconess Medical Ctr. (United States)
Harvard Medical School, Harvard Univ. (United States)
Elodia Cole, Beth Israel Deaconess Medical Ctr. (United States)
Kyle Myers, U.S. Food and Drug Administration (United States)


Published in SPIE Proceedings Vol. 10136:
Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment
Matthew A. Kupinski; Robert M. Nishikawa, Editor(s)

© SPIE. Terms of Use
Back to Top