Share Email Print
cover

Proceedings Paper

Investigation of methods for analyzing location specific observer performance data
Format Member Price Non-Member Price
PDF $14.40 $18.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

We examined the statistical powers of three methods for analyzing FROC mark-rating data, namely ROC, JAFROC and IDCA. Two classes of observers were simulated: a designer-level CAD algorithm and a human observer. A search-model based simulator was used with the average numbers of false positives per image ranging from 0.21 for the human observer to 10 for CAD. Model parameters were chosen to yield 80% and 85% areas under the predicted ROC curves for both classes of observers and inter-image and inter-modality correlations of 0.1, 0.5 and 0.9 were investigated. The area under the FROC curve up to abscissa α (ranging from 0.18 to 6.7) was used as the IDCA figure-of-merit; the other methods used their well-known figures of merit. For IDCA power increased with α so it should be chosen as large as possible consistent with the need for overlap of the two FROC curves in the x-direction. For CAD the IDCA method yielded the highest statistical power. Surprisingly, JAFROC yielded the highest statistical power for human observers, even greater than IDCA which, unlike JAFROC, uses all the marks. The largest difference occurred for conservative reporting styles and high data correlation: e.g., 0.3453 for JAFROC vs. 0.2672 for IDCA. One reason is that unlike IDCA, the JAFROC figure of merit is sensitive to unmarked normal images and unmarked lesions. In all cases the ROC method yielded the least statistical power and entailed a substantial statistical power penalty (e.g., 24% for ROC vs. 41% for JAFROC). For human observers JAFROC should be used and for designer-level CAD data IDCA should be used and use of the ROC method for localization studies is discouraged.

Paper Details

Date Published: 6 March 2008
PDF: 12 pages
Proc. SPIE 6917, Medical Imaging 2008: Image Perception, Observer Performance, and Technology Assessment, 69170C (6 March 2008); doi: 10.1117/12.770531
Show Author Affiliations
D. P. Chakraborty, Univ. of Pittsburgh (United States)
Hong-Jun Yoon, Univ. of Pittsburgh (United States)


Published in SPIE Proceedings Vol. 6917:
Medical Imaging 2008: Image Perception, Observer Performance, and Technology Assessment
Berkman Sahiner; David J. Manning, Editor(s)

© SPIE. Terms of Use
Back to Top