Share Email Print
cover

Proceedings Paper

Influence of panel size and expert skill on truth panel performance when combining expert ratings
Format Member Price Non-Member Price
PDF $14.40 $18.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

The focus of this manuscript is to investigate the statistical properties of expert panels used as a substitute for clinical truth through a simplistic Monte Carlo simulation model. We use Gaussian models to simulate both normal and abnormal distributions of ideal-observer test statistics. These distributions are designed to produce an ideal observer area under the ROC curve (AUC) of 0.85. Expert observers are modeled as an ideal observer test statistic degraded by a zero-mean Gaussian random variable. Different expert skill levels are achieved by changing the added variance. The experts' skill ranges between 0.6 and 0.8 in AUC. We combine decisions from 2-10 experts into a panel score by taking the median of all expert ratings as the panel test statistic. In experiment 1, truth panels made up of 2, 4, 8 and 10 experts who had the same skill level (AUC=0.8) achieved mean AUCs of 0.82, 0.83, 0.84, and 0.84, respectively. For experiment 2, the experts' skill level was varied uniformly between 0.6 and 0.8 in AUC. Panel performance decreased in experiment 2 compared to the fixed skill level panels in experiment 1. However, panels composed of 8 and 10 experts still achieved an AUC greater than 0.80, the maximum of any individual expert. These simulation experiments, while idealized and simplistic, are a starting point for understanding the implications of using a panel of experts as surrogate truth in ROC studies when a gold standard is not available.

Paper Details

Date Published: 6 April 2005
PDF: 9 pages
Proc. SPIE 5749, Medical Imaging 2005: Image Perception, Observer Performance, and Technology Assessment, (6 April 2005); doi: 10.1117/12.596286
Show Author Affiliations
Nicholas Petrick, NIBIB/CDRH Lab. for the Assessment of Medical Imaging Systems, US FDA (United States)
Brandon D. Gallas, NIBIB/CDRH Lab. for the Assessment of Medical Imaging Systems, US FDA (United States)
Frank W. Samuelson, NIBIB/CDRH Lab. for the Assessment of Medical Imaging Systems, US FDA (United States)
Robert F. Wagner, NIBIB/CDRH Lab. for the Assessment of Medical Imaging Systems, US FDA (United States)
Kyle J. Myers, NIBIB/CDRH Lab. for the Assessment of Medical Imaging Systems, US FDA (United States)


Published in SPIE Proceedings Vol. 5749:
Medical Imaging 2005: Image Perception, Observer Performance, and Technology Assessment
Miguel P. Eckstein; Yulei Jiang, Editor(s)

© SPIE. Terms of Use
Back to Top