Human spatial processing accounts for dynamic range and color

Silver halide photography has a fixed response to scenes. Films count photons, and that results in a count-dependent optical density. The range of the film's sensitivity to light, as well as its optical density, is determined in the factory where the film is made. However, digital imaging makes it possible to render wider ranges of light intensity, thereby more closely reproducing the variations in illumination that are found within scenes. With better imaging hardware, it is now possible to improve the range of light sensors and displays, and increase the number of digital quantization levels. It should also be possible to make more accurate reproductions over a greater range of light intensities. Raw formats in today's commercial cameras enable more bits per pixel.¹ A sensor's dynamic range is the ratio of the amount of light that causes its maximum response to the light that causes its minimum response. Active pixel processing has led to techniques that can increase the sensor's dynamic range to a ratio of 10¹⁰ to 1.^2–4 For displays, Seetzen et al. used an ‘unsharp mask’ technique to illuminate a high-resolution liquid-crystal display with an out-of-focus, modulated array of LED illuminators,⁵ and make digital high-dynamic-range (HDR) displays with the ratio of maximum light emission to minimum emission of 10⁵ to 1 and a 1500cd/m² maximum screen luminance.⁶ As we start to incorporate these techniques in imaging, the question becomes, how much of the 10¹⁰ range of the sensor response and the 10⁵ range of the displays can we actually use?

In 1941, Jones and Condit measured the range of light in 128 outdoor scenes and the camera responses to them.⁷ They also measured the range on each camera's film plane, and showed that the optical veiling glare (scattered light), not the sensor signal-to-noise, sets the usable dynamic range of cameras. They designed negatives that captured all the information in any scene above the glare limit, roughly 4 log units). However, not all this information was necessarily accessible in a print. High-contrast print paper caused loss of scene detail. Recent measurements of the veiling glare limits in cameras and in human vision also show strong scene dependency. Camera optics add an unwanted fraction of the scene's light to the image of the scene on the sensor.⁸ This unwanted glare is present in all multiple exposures, and makes calculations of scene radiance unreliable.⁹

Further, intraocular glare limits the range of light falling on the retina.^10–12 As with cameras, the range of usable light for humans varies with the scene content: with a maximum white background it is about 1.5 log units; with a 50% white background around 2 log units; and with an unlit background slightly over 4 log units. Optical veiling glare limits the range of scene radiances cameras can capture accurately, whereas intraocular glare limits the range of light from the scene that falls on the retina.

Nevertheless, HDR imaging dramatically improves scene rendering. There must be reasons, other than accurate scene rendition, that account for the success of HDR imaging. Two important, but paradoxical observations can help to explain this. First, artists since the Renaissance have rendered HDR scenes in the low-dynamic range of paintings. Outdoor scenes with sun and shade have 1000 to 1 ranges of light, while oil paints have a range of only 30 to 1. Second, humans see over 4 log units of detail in shadows using optic nerves that can transmit only 2 log units of range.¹³

We have found that the paradoxes can be resolved by considering how human spatial image processing counteracts the effects of glare. Humans have two independent and opposing spatial mechanisms. Intraocular veiling glare reduces the luminance range on the retina, while neural ‘simultaneous contrast’ effects increase the apparent differences. Figure 1 shows the classic simultaneous contrast with two identical scene luminance gray patches. The one surrounded by white has more glare than the gray-in-black. If retinal luminance predicted appearance, then the gray-in-white would appear lighter than the gray-in-black. However, through human spatial image processing, simultaneous contrast makes the lower contrast gray-in-white look darker. In this case, contrast overcompensates for glare.

Figure 1. Two opposing spatial mechanisms control the appearance of gray patches.

Glare distorts the luminances of the scene in the image on the retina by lowering retinal contrast. However, simultaneous neural spatial processing works to counteract glare. The image with lower actual contrast appears to have a higher apparent contrast. Both glare and simultaneous contrast are spatial processes and depend on the content of the entire scene.

Color constancy is usually considered to be unrelated to HDR scene rendition. Unlike films that have fixed spectral sensitivities, humans see objects with constant color appearances from variable spectral stimuli. This has traditionally been attributed to pixel-based normalization, similar to white balancing a camera. Many assume that retinal cones adapt to changes in illumination to discount it, so that appearance correlates with reflectance. Such hypothetical mechanisms can account for flat 2D color test targets, but fail to predict the variable appearance of constant paints in complex illumination on 3D colored blocks.¹⁴ Figure 2 shows photographs of two identical arrangements of 3D colored blocks called Mondrians: one in nearly uniform illumination, and the other in directional (HDR) illumination. Adaptation, or any other single-pixel process, cannot predict colors in 3D Mondrians.¹⁵ The small departures from perfect color constancy observed are predicted by spatial comparisons with colors in the rest of the view.¹⁶ Color constancy depends on spatial comparisons that synthesize how colors appear in relation to all the segments of the scene.¹⁷

Figure 2. Two identical arrangements of 3D colored blocks called Mondrians, made with 11 paints, viewed at the same time in the same room. The left is in nearly uniform illumination, while the right is in directional high-dynamic-range (HDR) illumination. Appearances vary with illumination, despite constant reflectance.

At first, HDR imaging may have seemed best suited for improved recordings of scene radiances. However, glare limits the range of light that can be detected by cameras or the retina. All scene regions below middle gray are influenced, more or less, by the glare from the bright scene segments. Instead of accurate radiance reproduction, HDR imaging works well because it preserves the details in the scene's shadows. Spatial image processing preserves this information, but distorts accurate reproduction. Similarly, color constancy is the result of color comparisons of the entire scene.

My future research will focus on studying the correlations between HDR image-processing algorithms and models of human vision.

The author is grateful for the collaboration of Alessandro Rizzi, Carinna Parraman, Vassilios Vonikakis, Bob Sobel, and Mary McCann.