Share Email Print

Proceedings Paper

Analysis of the number of distinct findings obtained by multiple readers in an MRMC study: When do findings obtained from the addition of new readers become redundant, or otherwise negligible?
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

The ultimate goal of this project is to investigate whether the effect of a computer-aided detection (CAD) system on readers' performance (especially, in situation of an upgrade of the CAD system, or between two different CAD systems with similar design) can be accurately predicted without having to perform a multi-reader multi-case (MRMC) observer study and, if such prediction is possible, to establish the underlying methodology. Our current study is intended to provide evidence that would substantiate efforts toward such investigation. The objectives of this study were 1) to investigate the relationship between the number of radiologists reading a dataset of thoracic computed tomography (CT) images to identify lung nodules and the number of distinct findings and 2) to determine the number of readers needed to identify almost all clinically distinct findings in a dataset. We used data from a multi-reader multi-case (MRMC) observer study that consisted of six radiologists interpreting 85 thoracic CT examinations. To further illustrate our approach, we also utilized simulated data consisting of twelve readers interpreting 198 samples equally distributed between three levels of detection difficulty. For each possible reader grouping, the number of distinct findings identified by the readers in the group was calculated. Five types of regression models used to describe the relationship between the average number of distinct findings per case and the number of readers needed were compared. The result showed that the logistic model best fitted both the thoracic CT data and the simulated data. Our assumption is that adding more readers after a certain reader set size would mostly add redundant findings and, therefore, the benefit would be negligible. Using this model, the predicted number of readers was found to depend on the type of findings considered. Our study showed that the number of clinically distinct findings that can be identified by radiologists on CT lung examinations without the use of a CAD system may be limited and that identifying almost all of these findings may only require a limited number of readers.

Paper Details

Date Published: 3 March 2011
PDF: 10 pages
Proc. SPIE 7966, Medical Imaging 2011: Image Perception, Observer Performance, and Technology Assessment, 79661M (3 March 2011); doi: 10.1117/12.878103
Show Author Affiliations
Sophie Paquerault, Consultant (United States)
Berkman Sahiner, U.S. Food and Drug Administration (United States)
Anna Kettermann, U.S. Food and Drug Administration (United States)
Laura M. Yarusso, Consultant (United States)
Lubomir M. Hadjiiski, Univ. of Michigan (United States)
Heang-Ping Chan, Univ. of Michigan (United States)

Published in SPIE Proceedings Vol. 7966:
Medical Imaging 2011: Image Perception, Observer Performance, and Technology Assessment
David J. Manning; Craig K. Abbey, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?