Seven challenges for image quality research

Examining the limitations of current research into image quality assessment opens doors for further studies.
15 January 2014
Damon Chandler, Thien Phan and Md Mushfiqul Alam

Digital images undergo numerous forms of processing during capture, storage, transmission, and ultimately display to the consumer. Since these processing steps can alter the appearance of an image, there exists a need to assess the impacts of processing on final visual quality. In addressing this need, many algorithms for image quality assessment (IQA) have been investigated and developed over the last several decades, with particularly explosive growth during the last 10 years. Today, IQA research is an active subdiscipline of image processing, and many of the techniques and algorithms have proven useful for a variety of electronic imaging applications.1–12

Purchase SPIE Field Guide to Image ProcessingWe can trace research on IQA back to early quality evaluations of optical systems and analog television broadcast/display systems.13–24 This early work not only laid the foundation for our current understanding of the factors that affect quality—for example, luminance and spatial resolution, contrast and color range, gradation, brilliance, flicker, and noise—but it also stressed the need to take into account properties of the human visual system during the quality-assessment process.

Modern IQA algorithms estimate quality using a variety of image-analysis techniques. When a reference image is available, local differences between the reference and distorted images are measured in various domains, and such differences are mapped to an estimate of quality. In the absence of a reference image, or when local comparisons are infeasible (for example, when comparing two samples of textures), quality is estimated by distortion-specific detectors and/or statistical features that are measured or applied to the distorted image. Due in large part to fundamental research efforts by members of the Human Vision and Electronic Imaging (HVEI) community, IQA research has particularly benefited from improved models of visual perception. Such vision models have been instrumental for providing IQA algorithms with the ability to take into account key properties of human visual perception, as well as the ability to mimic the biological processing strategies that underlie the visual assessment process.

Our aim is to shed light on the limitations in current knowledge of image quality in the hopes of opening doors for further studies.25 Without question, today's IQA algorithms predict quality for a variety of images and distortion types remarkably well. We have noticed, however, a shift in the focus of IQA research over the last decade from the previously broad objective of gaining a better understanding of how humans judge quality, to the current limited objective of achieving a better fit to the available, ground-truth subjective data. We appear to have gained a substantial amount in the form of new algorithms, but significantly less in the form of new fundamental knowledge on image quality. Despite the explosive growth in IQA algorithms, today's research community is only marginally closer to understanding how humans perceive artifacts in images compared to over 30 years ago. Behind the numerous successes in IQA research lies a long list of unanswered questions and unsolved challenges.

We therefore highlight seven open research challenges. The first six challenges stem from the lack of complete perceptual models for natural images, compound and suprathreshold distortions, multiple distortions, the interactions between the distortions and the images, images containing nontraditional distortions, and enhanced images. The seventh challenge deals with computational efficiency. We have made an effort to address some of these challenges in three recent studies: our computational modeling effort on the database of local detection thresholds in natural images presented at HVEI 2013, our new IQA database designed to measure local image quality, and our investigation into common hardware bottlenecks in IQA algorithms.

As an example, one open challenge in IQA research is to determine how humans judge local quality in an image, and how these local judgments relate to the overall quality. Our preliminary database of local quality ratings has an example of original and distorted images, and ground-truth and predicted local quality ratings from three popular IQA algorithms: multi-scale structural similarity (MS-SSIM),26 visual information fidelity (VIF),27 and most apparent distortion (MAD)28 (see Figure 1). By doing a block-by-block comparison between the predictions and the ground truth, it is apparent that the algorithms fall short in terms of local quality assessment, despite the fact that they perform extremely well at overall (global) quality assessment.

Figure 1. Existing image quality assessment (IQA) algorithms cannot accurately perform local quality assessment. (a) The original image. (b) A version of that image distorted via JPEG2000 compression. (c) A map indicating the local quality of each corresponding 85×85 region in which brighter areas denote greater quality. The second row shows maps of the local quality predictions from three popular IQA algorithms: (d) multi-scale structural similarity (MS-SSIM), (e) visual information fidelity (VIF), and (f) most apparent distortion (MAD).

Local quality assessment is but one of many open challenges in IQA research. By identifying these challenges, we hope to raise awareness of not only the current limitations, but also the need for additional psychophysical and computational studies beyond those commonly cited, as well as the need for alternative theories and techniques beyond those commonly employed.

The authors are grateful to the HVEI community for its continued fundamental research on image quality. This work was supported in part by National Science Foundation awards 0917014 and 1054612, and by the US Army Research Laboratory and the US Army Research Office under grant W911NF-10-1-0015.

Damon Chandler, Thien Phan, Md Mushfiqul Alam
School of Electrical and Computer Engineering
Oklahoma State University
Stillwater, OK

1. R. J. Beaton, Quantitative models of image quality, Proc. Hum. Fact. Ergon. Soc. Annu. Mtg. 27(1), p. 41-45, 1983. doi:10.1177/154193128302700111
2. A. J. Ahumada Jr., Computational image quality metrics: a review, Soc. Inf. Display Int'l Symp. Digest Techn. Papers 24, p. 305-308, 1993.
3. M. P. Eckert, A. P. Bradley, Perceptual quality metrics applied to still image compression, Signal Process. 70, p. 177-200, 1998.
4. B. Keelan, Handbook of Image Quality: Characterization and Prediction, Taylor and Francis, 2002.
5. K. Seshadrinathan, T. Pappas, R. Safranek, J. Chen, Z. Wang, H. Sheikh, A. Bovik, Image quality assessment, The Essential Guide to Image Processing, Academic Press, 2009.
6. S. Winkler, Video quality measurement standards: current status and trends, Proc. 7th Int'l Conf. Inf. Commun. Signal Process., p. 848-852, 2009.
7. S. S. Hemami, A. R. Reibman, No-reference image and video quality estimation: applications and human-motivated design, Signal Process. Image Commun. 25(7), p. 469-481, 2010. doi:10.1016/j.image.2010.05.009
8. S. Chikkerur, V. Sundaram, M. Reisslein, L. J. Karam, Objective video quality assessment methods: a classification, review, and performance comparison, IEEE Trans. Broadcast. 57(2), p. 165-182, 2011.
9. K. Seshadrinathan, A. C. Bovik, Automatic prediction of perceptual quality of multimedia signals—a survey, Multimedia Tools Appl. 51(1), p. 163-186, 2011. doi:10.1007/s11042-010-0625-9
10. W. Lin, C. C. Jay Kuo, Perceptual visual quality metrics: a survey, J. Vis. Comun. Image Represent. 22(4), p. 297-312, 2011. doi:10.1016/j.jvcir.2011.01.005
11. D. M. Chandler, Seven challenges in image quality assessment: past, present, and future research, ISRN Signal Process. 2013, p. 905685, 2013. doi:10.1155/2013/905685
12. A. C. Bovik, Automatic prediction of perceptual image and video quality, Proc. IEEE 101(9), 2013.
13. J. Horton, The electrical transmission of pictures and images, Proc. Inst. Radio Eng. 17(9), p. 1540-1563, 1929. doi:10.1109/JRPROC.1929.221858
14. L. Jesty, G. Winch, Television images: an analysis of their essential qualities, Trans. Illum. Eng. 2, p. 316-334, 1937.
15. P. Goldmark, J. Dyer, Quality in television pictures, Proc. IRE 28(8), p. 343-350, 1940. doi:10.1109/JRPROC.1940.229361
16. G. Winch, Colour television: some subjective and objective aspects of color rendering, Proc. IEE, Part IIIA: Television 99(20), p. 854-860, 1952. doi:10.1049/pi-3a.1952.0103
17. L. Jesty, Television as a communication problem, J. Inst. Electr. Eng. 1953(4), p. 181-183, 1953. doi:10.1049/jiee-2.1953.0122
18. P. B. Fellgett, E. H. Linfoot, On the assessment of optical images, Philos. Trans. R. Soc. London, Ser. A 247(931), p. 369-407, 1955.
19. C. Dean, Measurements of the subjective effects of interference in television reception, Proc. IRE 48(6), p. 1035-1049, 1960. doi:10.1109/JRPROC.1960.287505
20. H. Schmid, Measurement of television picture impairments caused by linear distortions, J. SMPTE 77(3), p. 215-220, 1968.
21. D. Sakrison, V. Algazi, Comparison of line-by-line and two-dimensional encoding of random images, IEEE Trans. Inf. Theor. 17(4), p. 386-398, 1971. doi:10.1109/TIT.1971.1054667
22. Z. Budrikis, Visual fidelity criterion and modeling, Proc. IEEE 60(7), p. 771-779, 1972.
23. T. G. Stockham Jr., Image processing in the context of a visual model, Proc. IEEE 60(7), p. 828-842, 1972. doi:10.1109/PROC.1972.8782
24. O. Schade, Image Quality: A Comparison of Photographic and Television Systems , RCA Laboratories, 1975.
25. D. Chandler, Seven challenges for image quality research, SPIE 9014, 2014. (Keynote presentation.)
26. Z. Wang, E. Simoncelli, A. Bovik, Multiscale structural similarity for image quality assessment, Conf. Rec. Thirty-Seventh Asilomar Conf. Signals Syst. Comput., p. 1398-1402, 2003.
27. H. R. Sheikh, A. C. Bovik, Image information and visual quality, IEEE Trans. Image Process. 15(2), p. 430-444, 2006.
28. E. C. Larson, D. M. Chandler, Most apparent distortion: full-reference image quality assessment and the role of strategy, J. Electron. Imag. 19(1), p. 011006, 2010.
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?