Share Email Print

Proceedings Paper

Using “Reader Disagreement Index” as a predictive reviewer performance monitoring tool for timely intervention
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

Purpose: Blinded Independent Central Review (BICR) is a well-accepted method employed in many oncology registration trials. Ongoing monitoring radiologist “reader” performance is both good clinical trial practice and a requirement by regulatory authorities. We continue to use Reader Disagreement Index (RDI) as an important measure in BICR. In this work we studied RDI as an early indicator to identify an outlier reader during the monitoring of reader performance in BICR. Early indication would enable early intervention and thus possibly improve trial outcomes.
Methods: We performed a retrospective analysis of readers’ RDIs in nineteen different clinical trials. Ninety-two reader performances were examined at five intervals in each trial. These individual trial reviews were conducted by forty-three board-certified radiologist readers using several established imaging assessment trial criteria. The objective was to see how well RDI performance above a threshold at progressive monitoring intervals would “flag” a potential overall end-point performance “issue” for that specific reader.
Results: We present results for the prediction of exceeding threshold (one standard deviation above a study mean RDI). Sensitivity, Specificity, Positive Predictive Value (PPV) and Negative Predictive Value (NPV) were determined for the predicted performance outcomes. We explored interpreting multiple “flags” for each trial to improve the aforementioned metrics.
Conclusions: One would expect that a “flag” of RDI exceeding threshold at an early-stage would likely give a useful prediction of end-point reader performance. We refined our methods to use multiple flags which enable statistically improved Specificity and PPV. Improved predictive capability at early stage intervals coupled with persistent monitoring across subsequent intervals will enable trial managers to focus on specific readers. An earlier indication of possible reader performance issues can permit proactive intervention and enhance good trial practices.

Paper Details

Date Published: 16 March 2020
PDF: 6 pages
Proc. SPIE 11316, Medical Imaging 2020: Image Perception, Observer Performance, and Technology Assessment, 113160A (16 March 2020); doi: 10.1117/12.2549980
Show Author Affiliations
Manish Sharma, PAREXEL International (India)
J. Michael O'Connor, PAREXEL International (United States)
Anitha Singareddy, PAREXEL International (India)
Madhuri Madasu, PAREXEL International (India)
Nicholas Enus, PAREXEL International (United States)
Kassel Fotinos-Hoyer, Parexel International GmbH (Germany)
Yibin Shao, PAREXEL China Co., Ltd. (China)

Published in SPIE Proceedings Vol. 11316:
Medical Imaging 2020: Image Perception, Observer Performance, and Technology Assessment
Frank W. Samuelson; Sian Taylor-Phillips, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?