Share Email Print
cover

Proceedings Paper

MRMC analysis of agreement studies
Author(s): Brandon D. Gallas; Amrita Anam; Weijie Chen; Adam Wunderlich; Zhiwei Zhang
Format Member Price Non-Member Price
PDF $14.40 $18.00

Paper Abstract

The purpose of this work is to present and evaluate methods based on U-statistics to compare intra- or inter-reader agreement across different imaging modalities. We apply these methods to multi-reader multi-case (MRMC) studies. We measure reader-averaged agreement and estimate its variance accounting for the variability from readers and cases (an MRMC analysis). In our application, pathologists (readers) evaluate patient tissue mounted on glass slides (cases) in two ways. They evaluate the slides on a microscope (reference modality) and they evaluate digital scans of the slides on a computer display (new modality). In the current work, we consider concordance as the agreement measure, but many of the concepts outlined here apply to other agreement measures. Concordance is the probability that two readers rank two cases in the same order. Concordance can be estimated with a U-statistic and thus it has some nice properties: it is unbiased, asymptotically normal, and its variance is given by an explicit formula. Another property of a U-statistic is that it is symmetric in its inputs; it doesn't matter which reader is listed first or which case is listed first, the result is the same. Using this property and a few tricks while building the U-statistic kernel for concordance, we get a mathematically tractable problem and efficient software. Simulations show that our variance and covariance estimates are unbiased.

Paper Details

Date Published: 24 March 2016
PDF: 12 pages
Proc. SPIE 9787, Medical Imaging 2016: Image Perception, Observer Performance, and Technology Assessment, 97870F (24 March 2016); doi: 10.1117/12.2217074
Show Author Affiliations
Brandon D. Gallas, U.S. Food and Drug Administration (United States)
Amrita Anam, U.S. Food and Drug Administration (United States)
Univ. of Maryland, Baltimore County (United States)
Weijie Chen, U.S. Food and Drug Administration (United States)
Adam Wunderlich, U.S. Food and Drug Administration (United States)
Zhiwei Zhang, U.S. Food and Drug Administration (United States)


Published in SPIE Proceedings Vol. 9787:
Medical Imaging 2016: Image Perception, Observer Performance, and Technology Assessment
Craig K. Abbey; Matthew A. Kupinski, Editor(s)

© SPIE. Terms of Use
Back to Top