2.1

Models for patients

Early studies of task-based image quality in medical imaging made use of highly stylized object models, typically in which signals were exactly known and backgrounds were uniform. These studies included theoretical investigations² and those making use of simplistic digital³ or physical⁴ imaging phantoms with ideal, human observers, or both.⁵ It soon became clear that more complex models for the object were needed to avoid conclusions regarding system quality that did not generalize well to more complex imaging scenarios. As a result, investigators turned to statistical models for backgrounds and signals that were defined statistically.^6-8 As highlighted next, we’ve come a very long way since those early efforts to develop models for signals and backgrounds with greater complexity.

Procedural, analytical, or parameterized models are one major area of significant technical advances in recent years. For example, the digital breast model of Graff et al.⁹ makes use of a set of procedures to generate the major anatomical structures of the female breast, including fat and glandular tissues, the ductal tree, vasculature, and ligaments. The model allows for patient characteristics such as breast shape, volume, density, and compressed thickness to be controlled. Source code for the model is freely available.¹⁰ Models for masses¹¹ and microcalcifications¹² based on procedure models allow for the insertion of pathologies in such digital breast objects, resulting in a family of digital breasts with simulated lesions of varying size and shape, and with even the possibility of modeling tumor growth over time.¹³

Hybrid models combine clinical backgrounds with edited clinical lesions to create larger datasets. These models typically leverage the much larger numbers of normal images at hand, and create a new, larger set of abnormal images by inserting lesions that were digitally harvested from abnormal images. These models have been demonstrated in the domain of the raw/detected data^14-16 as well as in the domain of reconstructed images.¹⁷ Because such procedures make use of real clinical backgrounds and pathologies, they have the advantage of clinical realism, so long as the lesion insertion process does not result in noticeable artifacts. However, as these methods start with image domain data, they are less amenable to evaluations of imaging system hardware. They are more likely to be useful in the evaluation of competing reconstruction methods, image processing tools, display devices, and data augmentation for training of artificial intelligence and deep learning (AI/DL) algorithms.

Atlas models derive families of patients through the segmentation of clinical image sets. The XCAT Phantom Program developed by investigators of the Duke University’s Center for Virtual Imaging Trials is a prominent example of this approach.¹⁸ Because these phantoms are derived from clinical images, they are necessarily limited by the imaging hardware used to acquire the underlying images. Research investigators are working to develop approaches for inserting sub-resolution structures and textures from other imaging modalities or via modeling methods.

A very recent entry into the toolbox for creating models of patients is the use of learning models based on Generative Adversarial Networks (GANs). Anastasio’s team at UIUC have contributed several important papers demonstrating this approach.^19-20 The Grand Challenge on “Deep Generative Modeling for Learning Medical Imaging Statistics” hosted by UIUC and the US FDA, and hosted by the American Association of Physicists in Medicine (AAPM), will provide the community with a state-of-the-art understanding of these models’ ability to reproduce visually similar images that also preserve task-based relevant statistical properties.²¹ The beauty of this challenge is that, because the models will be trained on a set of digital breast models with known statistical properties,⁹ the GAN model entries will be able to be evaluated against a gold standard in object space. If such GANs can be shown to preserve diagnostically relevant information in a task-based sense, they will open new avenues for task-based assessments.

2.2

Models for imaging systems

Solid and steady progress has been made on the expansion of realistic physics models that incorporate recent innovations in imaging hardware, including geometries with multiple and/or moving sources, new detector materials and mechanisms, and so on. As computing power and storage has grown, so has our ability to model the many physical interactions that comprise the imaging process with greater realism and precision. This area of endeavor has long been the bedrock of the Physics Conference at the SPIE Medical Imaging Symposium.

Research into the potential for GANs to synthesize images is a more recent area of interest. GANs might be able to learn the conversion from one kind of acquisition system to another.²² GANs are also being investigated for their potential to learn the forward imaging model so as to produce realistic images from digital body phantoms.²³ Much work remains to be done to understand the validity of these approaches in terms of their ability to support the task-based evaluation of imaging systems.

2.3

Models for readers/observers

Broadly speaking, observer models follow two tracks. The first is the study of human perception in order to better understand the ability of humans to extract information from images. From such studies, models for human readers are being developed for use in the evaluation of imaging tools that are intended to create or display data for human interpretation, particularly image reconstruction algorithms, image processing methods, and novel display or visualization systems. The state-of-the-science with respect to modeling human perception includes the elucidation of the human observer’s template for more types of tasks,²⁴ the creation and validation of models for human interpretation that incorporate foveation for visual search in 3D medical images,²⁵ and the prediction of human performance in classification tasks via deep learning models.^26,27

The second track in the modeling of observers is pursued by those who seek to model the Ideal or Bayesian Observer (IO). By definition the IO captures all information in the detected data, without adding noise or uncertainty. The difference in performance between the IO and the human observer for a particular task offers a window into the opportunity for improved inferences from those images via computer-aided diagnosis, or perhaps in the future computer diagnosis. For the purpose of this paper, however, the importance of the IO is in its ability to facilitate the evaluation of image acquisition systems on an absolute scale. A search of the SPIE Digital Library will demonstrate that over many decades, presentations at the SPIE Medical Imaging Symposium have grown our ability to compute and understand IO performance and that of its close cousin, the optimal linear or Hotelling observer, for tasks of increasing range and complexity. Moreover, it has long been known that neural networks can approximate a Bayes optimal discriminant.²⁸ While this was true in theory, in practice the computing requirements did not match the computing power at hand, leaving us still limited with respect to the task-based evaluations we could execute for the IO. That reality is rapidly changing. The SPIE 2020 Medical Imaging Symposium was a breakout year for papers that presented approaches to approximating the ideal observer.^29-31

2.4

Statistical methods

Rigorous science requires error bars. Task-based assessments require the incorporation of all sources of uncertainty in the evaluation process that contribute to uncertainty in the figure of merit. These sources include the variability in the underlying patients being imaged, measurement noise from the imaging system, and uncertainty coming from the readers. In the past 10+ years our community has come to appreciate that Multi-Reader, Multi-Case (MRMC) analysis is essential for testing the significance of differences in competing imaging modalities for detection and discrimination tasks. MRMC methods give the total uncertainty in estimates of ROC-based metrics stemming from the range of case difficulty, reader skill and mindset, and their interactions. That’s not particularly new. What is new is the development of “split-plot” methodologies that allow for the analysis of more general study designs than the fully-crossed design in which every reader reads every case for each modality. Split-plot designs can be used to reduce the number of reads per reader and the number of total reads. Furthermore, for the same number of total reads, the split-plot design has been found to be more statistically efficient.^32-35

There has been considerable progress in recent years in methods for collecting better data from human readers. Data from human observers may be needed to serve as the reference standard for circumstances in which other forms of truth are not available. A prominent example of this is in pathology applications. In order to assess a new digital pathology imaging device, including the many efforts to develop AI/DL methods to assist in pathology tasks, it is common to make use of experts for the “truthing” of pathology data. New tools are emerging for estimating truth from panels of expert readers, including improved approaches to reader training to reduce reader variability, improved collection interfaces, and statistical analysis tools.^36,37 Similarly, better approaches are emerging for collecting data from humans who are serving as “study” readers in a task-based assessment. These also include improved training methods and data collection interfaces that facilitate the collection of more precise reader data on a finer measurement scale.³³ US FDA/CDRH shares examples of reader training materials and data collection interfaces that facilitate the collection of improved reader data, along with powerful statistical software tools for reader or algorithm study design and data analysis.^38,39 The inclusion of iMRMC statistical tools in the FDA/CDRH Regulatory science tools catalog is further indication of the FDA’s interest and support for the development and use of such tools in medical imaging device development and review.

The role of clinicians in task-based assessments of image quality was highlighted in a recent consensus paper from the Society of Nuclear Medicine and Molecular Imaging.⁴⁰ That group noted that clinicians are important partners in these investigations, playing key roles as study readers and expert truthers. They also assist in the selection of appropriate clinical tasks and patient populations.

There has also been significant recent progress related to the assessment of imaging systems for tasks related to the estimation of quantitative imaging biomarkers (QIBs). RSNA’s Quantitative Imaging Biomarker Alliance (QIBA) published a series of papers in 2015 that laid out a comprehensive framework for QIBs, intending to standardize the terminology,⁴¹ methods for evaluating imaging biomarkers,⁴² and methods for comparing imaging biomarkers.⁴³ Now comes the publication of a set of papers in Academic Radiology that expands QIBA’s earlier work to multiparametric quantitative imaging biomarkers.^44-49