Seven challenges for image quality research
Digital images undergo numerous forms of processing during capture, storage, transmission, and ultimately display to the consumer. Since these processing steps can alter the appearance of an image, there exists a need to assess the impacts of processing on final visual quality. In addressing this need, many algorithms for image quality assessment (IQA) have been investigated and developed over the last several decades, with particularly explosive growth during the last 10 years. Today, IQA research is an active subdiscipline of image processing, and many of the techniques and algorithms have proven useful for a variety of electronic imaging applications.1–12
We can trace research on IQA back to early quality evaluations of optical systems and analog television broadcast/display systems.13–24 This early work not only laid the foundation for our current understanding of the factors that affect quality—for example, luminance and spatial resolution, contrast and color range, gradation, brilliance, flicker, and noise—but it also stressed the need to take into account properties of the human visual system during the quality-assessment process.
Modern IQA algorithms estimate quality using a variety of image-analysis techniques. When a reference image is available, local differences between the reference and distorted images are measured in various domains, and such differences are mapped to an estimate of quality. In the absence of a reference image, or when local comparisons are infeasible (for example, when comparing two samples of textures), quality is estimated by distortion-specific detectors and/or statistical features that are measured or applied to the distorted image. Due in large part to fundamental research efforts by members of the Human Vision and Electronic Imaging (HVEI) community, IQA research has particularly benefited from improved models of visual perception. Such vision models have been instrumental for providing IQA algorithms with the ability to take into account key properties of human visual perception, as well as the ability to mimic the biological processing strategies that underlie the visual assessment process.
Our aim is to shed light on the limitations in current knowledge of image quality in the hopes of opening doors for further studies.25 Without question, today's IQA algorithms predict quality for a variety of images and distortion types remarkably well. We have noticed, however, a shift in the focus of IQA research over the last decade from the previously broad objective of gaining a better understanding of how humans judge quality, to the current limited objective of achieving a better fit to the available, ground-truth subjective data. We appear to have gained a substantial amount in the form of new algorithms, but significantly less in the form of new fundamental knowledge on image quality. Despite the explosive growth in IQA algorithms, today's research community is only marginally closer to understanding how humans perceive artifacts in images compared to over 30 years ago. Behind the numerous successes in IQA research lies a long list of unanswered questions and unsolved challenges.
We therefore highlight seven open research challenges. The first six challenges stem from the lack of complete perceptual models for natural images, compound and suprathreshold distortions, multiple distortions, the interactions between the distortions and the images, images containing nontraditional distortions, and enhanced images. The seventh challenge deals with computational efficiency. We have made an effort to address some of these challenges in three recent studies: our computational modeling effort on the database of local detection thresholds in natural images presented at HVEI 2013, our new IQA database designed to measure local image quality, and our investigation into common hardware bottlenecks in IQA algorithms.
As an example, one open challenge in IQA research is to determine how humans judge local quality in an image, and how these local judgments relate to the overall quality. Our preliminary database of local quality ratings has an example of original and distorted images, and ground-truth and predicted local quality ratings from three popular IQA algorithms: multi-scale structural similarity (MS-SSIM),26 visual information fidelity (VIF),27 and most apparent distortion (MAD)28 (see Figure 1). By doing a block-by-block comparison between the predictions and the ground truth, it is apparent that the algorithms fall short in terms of local quality assessment, despite the fact that they perform extremely well at overall (global) quality assessment.
Local quality assessment is but one of many open challenges in IQA research. By identifying these challenges, we hope to raise awareness of not only the current limitations, but also the need for additional psychophysical and computational studies beyond those commonly cited, as well as the need for alternative theories and techniques beyond those commonly employed.
The authors are grateful to the HVEI community for its continued fundamental research on image quality. This work was supported in part by National Science Foundation awards 0917014 and 1054612, and by the US Army Research Laboratory and the US Army Research Office under grant W911NF-10-1-0015.
Oklahoma State University