Share Email Print
cover

Proceedings Paper

Forming a reference standard from LIDC data: impact of reader agreement on reported CAD performance
Author(s): Robert Ochs; Hyun J. Kim; Erin Angel; Christoph Panknin; Michael McNitt-Gray; Matthew Brown
Format Member Price Non-Member Price
PDF $14.40 $18.00

Paper Abstract

The Lung Image Database Consortium (LIDC) has provided a publicly available collection of CT images with nodule markings from four radiologists. The LIDC protocol does not require radiologists to reach a consensus during the reading process, and as a result, there are varying levels of reader agreement for each potential nodule with no explicit reference standard for nodules. The purpose of this work was to investigate the effects of the level of reader agreement on the development of a reference standard and the subsequent impact on CAD performance. Ninety series were downloaded from the LIDC database. Four different reference standards were created based on the markings of the LIDC radiologists, reflecting four different levels of reader agreement. All series were analyzed with a research CAD system and its performance was measured against each of the four standards. Between the standards with the lowest (any 1 of 4 readers) and highest (all 4 readers) required level of reader agreement, the number of nodules ⩾ 3 mm decreased 48% (from 174 to 90) and CAD sensitivity for nodules ⩾ 3 mm increased from 0.70 ± 0.34 to 0.79 ± 0.35. Between the same reference standards, the number of nodules < 3 mm decreased 84% (from 483 to 75) and CAD sensitivity for nodules < 3 mm increased from 0.30 ± 0.29 to 0.51 ± 0.45. This research illustrates the importance of indicating the method used to form the reference standard, since the method influences both the number of nodules and reported CAD performance.

Paper Details

Date Published: 30 March 2007
PDF: 6 pages
Proc. SPIE 6514, Medical Imaging 2007: Computer-Aided Diagnosis, 65142A (30 March 2007); doi: 10.1117/12.707916
Show Author Affiliations
Robert Ochs, Univ. of California, Los Angeles (United States)
Hyun J. Kim, Univ. of California, Los Angeles (United States)
Erin Angel, Univ. of California, Los Angeles (United States)
Christoph Panknin, Univ. of California, Los Angeles (United States)
Siemens Medical Solutions (Germany)
Michael McNitt-Gray, Univ. of California, Los Angeles (United States)
Matthew Brown, Univ. of California, Los Angeles (United States)


Published in SPIE Proceedings Vol. 6514:
Medical Imaging 2007: Computer-Aided Diagnosis
Maryellen L. Giger; Nico Karssemeijer, Editor(s)

© SPIE. Terms of Use
Back to Top