Share Email Print
cover

Proceedings Paper

A proposal of the evaluation method of the effectiveness of CAD use by means of the bias with variance characteristic (BVC) analysis and the internal structure analysis of the ROC-curve
Author(s): Toru Matsumoto; Akira Furukawa; Kanae Nisizawa; Shinichi Wada; Shinji Yamamoto; Kohei Murao; Mitsuomi Matsumoto; Shusuke Sone; Kenjiro Fukuhisa; Takeshi Iinuma; Yukio Tateno
Format Member Price Non-Member Price
PDF $14.40 $18.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

In this paper, we propose a methodology for evaluating whether the use of CAD is effective for any given reader or case, first analyzing the results of readers' judgments (0 or 1) by the technique known as analysis of bias-variance characteristics (BVC)1,2, then by combining this with ROC analysis, elucidating the internal structure of the ROC curve. The mean and variance are first calculated for the situation when multiple readers examine a medical image for a single case without CAD and with CAD, and assign the values 0 and 1 to their judgment of whether abnormal findings are absent or present or whether the case is normal or abnormal. The mean of these values represents the degree of bias from the true diagnosis for the particular case, and the variance represents the spread of judgments between readers. When the relationship between the two parameters is examined for several cases with differing degrees of diagnostic difficulty, the mean (horizontal axis) and variance (vertical axis) show a bell-shaped relation. We have named this typical phenomenon arising when images are read, the bias-variance characteristic (BVC) of diagnosis. The mean of the 0 and 1 judgments of multiple readers is regarded as a measure of the confidence level determined for the particular case. ROC curves were drawn by usual methods for diagnoses made without CAD and with CAD. From the difference between the TPF obtained without CAD and with CAD for the same FPF on the ROC curve, we were able to quantify the number of cases, the total number of readers, and the total number of cases for which CAD support was beneficial. To demonstrate its usefulness, we applied this method to data obtained in a reading experiment that aimed to evaluate detection performance for abnormal findings and data obtained in a reading experiment that aimed to evaluate diagnostic discrimination performance for normal and abnormal cases. We analyzed the internal structure of the ROC curve produced when all cases were included, and showed that there is a relationship between the degree of diagnostic difficulty of the case and the benefit of CAD support and demonstrated that there are patients and readers for whom CAD is of benefit and those for whom it is not.

Paper Details

Date Published: 8 March 2007
PDF: 11 pages
Proc. SPIE 6515, Medical Imaging 2007: Image Perception, Observer Performance, and Technology Assessment, 651507 (8 March 2007); doi: 10.1117/12.710450
Show Author Affiliations
Toru Matsumoto, National Institute of Radiological Sciences (Japan)
Akira Furukawa, National Institute of Radiological Sciences (Japan)
Kanae Nisizawa, National Institute of Radiological Sciences (Japan)
Shinichi Wada, Niigata Univ. (Japan)
Shinji Yamamoto, Chukyo Univ. (Japan)
Kohei Murao, Fujitsu Ltd. (Japan)
Mitsuomi Matsumoto, Diichi Hospital (Japan)
Shusuke Sone, JA Azumi General Hospital (Japan)
Kenjiro Fukuhisa, National Institute of Radiological Sciences (Japan)
Takeshi Iinuma, National Institute of Radiological Sciences (Japan)
Yukio Tateno, National Institute of Radiological Sciences (Japan)


Published in SPIE Proceedings Vol. 6515:
Medical Imaging 2007: Image Perception, Observer Performance, and Technology Assessment
Yulei Jiang; Berkman Sahiner, Editor(s)

© SPIE. Terms of Use
Back to Top