Share Email Print
cover

Proceedings Paper

Assessment of updated CAD without a new reader study: effect of calibration of computer output on the computer-aided reader performance in CADx
Format Member Price Non-Member Price
PDF $14.40 $18.00

Paper Abstract

It is very resource-demanding to assess each new version of a CAD system through a new reader study. We conjecture that the aided reader performance on a new version can be predicted by using certain characteristics of the computer output and the reader study conducted when the CAD system was initially introduced. This would likely reduce the need for additional reader studies. However, investigations are needed to develop a sound scientific foundation to test this conjecture. In this work, we consider a CADx system that outputs a disease score to aid the physician in making a diagnostic decision on a located lesion. Our major contribution is to show that calibration, reflected as a change in scale, is a characteristic of the computer output that needs to be considered in order to predict the aided reader performance in a new CADx version without a reader study. We used a bivariate bi-beta distribution to model the joint distribution of the decision variable underlying the reader without aid and the decision variable underlying the version 1 computer output in the initial version. We then applied a monotonic transformation to the computer output to simulate the computer output in a new version, i.e., the scores in the two versions differ only in calibration (specifically a change in scale). By further modeling certain mechanisms that the human reader may use for combining the computer output and the reader-alone scores, we computed the aided reader performance in terms of AUC for the new version of the CADx system. Our results show that the aided reader performance could depend on the degree of calibration difference between the two CAD system outputs. We conclude that for the purpose of predicting the aided reader performance of a new version of the CADx system, ROC performance (or any other rank-based metric) of the stand-alone CADx system may not be sufficient by itself.

Paper Details

Date Published: 3 March 2011
PDF: 8 pages
Proc. SPIE 7966, Medical Imaging 2011: Image Perception, Observer Performance, and Technology Assessment, 79661P (3 March 2011); doi: 10.1117/12.878278
Show Author Affiliations
Weijie Chen, U.S. Food and Drug Administration (United States)
Nicholas Petrick, U.S. Food and Drug Administration (United States)
Berkman Sahiner, U.S. Food and Drug Administration (United States)


Published in SPIE Proceedings Vol. 7966:
Medical Imaging 2011: Image Perception, Observer Performance, and Technology Assessment
David J. Manning; Craig K. Abbey, Editor(s)

© SPIE. Terms of Use
Back to Top