Share Email Print
cover

Proceedings Paper

JAFROC analysis revisited: figure-of-merit considerations for human observer studies
Format Member Price Non-Member Price
PDF $14.40 $18.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

Jackknife alternative free-response receiver operating characteristic (JAFROC) is a method for measuring human observer performance in localization tasks. JAFROC is being increasingly used to evaluate imaging modalities because it has been shown to have greater statistical power than conventional receiver operating characteristic (ROC) analysis, which neglects location information. JAFROC neglects the non-lesion localization marks ("false positives") on abnormal images. JAFROC1 is an alternative method that includes these marks. Both methods are lesion-centric in the sense that they assign equal importance to all lesions; an image with many lesions would tend to dominate the performance metric, and clinically less significant lesions are treated identically as more significant ones. In this paper weighted JAFROC and JAFROC1 analyses are described that treat each abnormal image (not each lesion) as a unit of measurement and account for different lesion clinical significances (weights). Lesion-centric and weighted methods were tested using a simulator that includes multiple-reader multiple-case multiple-modality location level correlations. For comparison, ROC analysis was also tested where the rating of the highest rated mark on an image was assumed to be its "ROC" rating. The testing involved random numbers of lesions per image, random weights, case-mixes (ratio of normal to abnormal images) and different correlation structures. We found that for either JAFROC or JAFROC1, both lesion-centric and weighted analyses had correct NH behavior and comparable statistical powers. For either lesion-centric or weighted analyses JAFROC1 yielded the highest power, followed by JAFROC and ROC yielded the least power, confirming a recent study using a less flexible single-reader dual-modality simulator. Provided the number of normal cases is not too small, JAFROC1 is the preferred method for analyzing human observer free-response data. For either JAFROC or JAFROC1 weighted analysis is preferable.

Paper Details

Date Published: 12 March 2009
PDF: 12 pages
Proc. SPIE 7263, Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment, 72630T (12 March 2009); doi: 10.1117/12.810859
Show Author Affiliations
D. P. Chakraborty, Univ. of Pittsburgh (United States)
Hong-Jun Yoon, Univ. of Pittsburgh (United States)


Published in SPIE Proceedings Vol. 7263:
Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment
Berkman Sahiner; David J. Manning, Editor(s)

© SPIE. Terms of Use
Back to Top