Integral imaging: autostereoscopic images of 3D scenes

A technique that works with incoherent light provides realistic, undistorted images without the need for special glasses.
18 November 2006
Manuel Martínez-Corral, Raúl Martínez-Cuenca, Genaro Saavedra and Bahram Javidi

One of the challenges of the information society is the development, and subsequent broad implemention, of technologies for acquiring and displaying 3D pictures and movies. Research efforts have long been devoted to finding an optimal 3D imaging and display technique. Among existing methods, integral imaging (InI) is especially useful because it provides autostereoscopic intensity images with full parallax, free of any viewing device.

InI is based on so-called integral photography (IP), which was proposed by Lipmann in 1908.1 Recently the technology has approached the level required for 3D TV and displays.2 In the capture stage of an InI system, an array of microlenses generates a collection of 2D elemental images onto a matrix image sensor, such as a CCD: see Figure 1 (top). In the reconstruction stage, the set of elemental images is displayed in front of a second microlens array, providing the observer with a reconstructed 3D image: see Figure 1 (bottom). Note that although all the elemental images are imaged by the corresponding microlenses onto the same plane—the reference image plane—the 3D scene is reconstructed in the image space by the intersection of the ray bundles emanating from each of the microlenses.

Figure 1. (Top) The scheme shows the pickup stage of an integral imaging system. Each elemental image records information from a different perspective. (Bottom) At the display stage, the 3D image is reconstructed by the intersection of ray bundles proceeding from the microlenses.

Despite major progress, InI systems still pose problems in their standard configuration. For example, the reconstructed images InI provides are real (in front of the film) but pseudoscopic, that is, reversed in depth. Also, owing to the spread of ray bundles for out-of-focus parts of the scene, the reconstructed images have very poor depth of field (DOF).

To attenuate the DOF problem, we have experimented with binary amplitude modulation during the capture stage. Specifically, we have shown that inserting a collection of opaque circular masks centered on the entrance pupil of each microlens—see Figure 2 (top)—greatly improves the DOF of the capture setup.3 Moreover, binary amplitude modulation allows the point spread function (PSF) of the microlenses to degrade very slowly for out-of-focus positions. That makes it is possible to define, to a good approximation, an effective PSF over a wide range of axial positions. Consequently, deconvolution tools can be used to improve the quality of the elemental images.4 Since InI requires real-time processing for a high number of elemental images, a low-time-consuming, noniterative method is convenient for deconvolution tasks. Our choice is Wiener filtering, the optimal linear method for signal recovery. To illustrate, we have calculated the elemental images for the pickup architecture shown in Figure 2 (top). In Figure 2 (middle) and (bottom) we show the elemental images obtained with the conventional setup and the proposed hybrid method, respectively.

Figure 2. The scheme (top) shows a pickup architecture in which binary amplitude modulation has been implemented. The elemental images (middle and bottom) were obtained using nonmodulated microlenses, and binary amplitude modulation and deconvolution processing, respectively.

A clever attempt to overcome the problem of pseudoscopic reconstruction was made by Herbert E. Ives, who proposed to record a second set of elemental images using the reconstructed image as the object. When the second set of elemental images is used in a second display stage, a real, undistorted, orthoscopic 3D image emerges. Yet the approach does not completely solve the pseudoscopic-to-orthoscopic (PO) conversion problem. This is because the two-step recording process entails substantial image degradation owing to diffraction effects and also to the pixilated structure of the matrix image sensor and display setups.

What we suggest here is to employ Ives's PO concept but to perform the intermediate stage (Display 1 + Pickup 2) using pixel mapping.5Figure 3 presents this hybrid procedure schematically. In Pickup 1, a set of elemental images is optically recorded in the CCD. Each elemental image records a different perspective of the object, and all the pixels of the elemental cell are charged. When these elemental images are viewed through the central pinhole in Display 1, a faceting effect occurs: a different portion of the reconstructed image is seen through each microlens.6

Figure 3. Shown is a scheme of the pseudoscopic-to-orthoscopic hybrid conversion method.

The result is that, through the upper microlens, the central pinhole can see only the ‘blue’ part of the reconstructed image, which indeed was recorded in the ‘blue’ pixel of the upper microcell. A similar process occurs when the reconstructed image is seen through the other pinholes. This pixel mapping provides elemental image Set 2, which, via optical Display 2, allows the reconstruction of a realistic image.

Manuel Martínez-Corral, Raúl Martínez-Cuenca, Genaro Saavedra
Department of Optics, University of Valencia
Burjassot, Spain

Manuel Martínez-Corral is currently associate professor of optics at the University of Valencia. Since 1999 he has led the 3D Diffraction and Imaging Group. His research interests include focusing properties of light, point-spread-function engineering in 3D microscopy, and 3D-imaging acquisition and display. He has published more than 50 technical articles on these topics in major journals. He has also published some 60 conference proceedings, including more than 10 invited presentations.

Bahram Javidi
Electrical and Computer Engineering Dept., University of Connecticut
Storrs, CT

Recent News