Integral imaging: autostereoscopic images of 3D scenes

One of the challenges of the information society is the development, and subsequent broad implemention, of technologies for acquiring and displaying 3D pictures and movies. Research efforts have long been devoted to finding an optimal 3D imaging and display technique. Among existing methods, integral imaging (InI) is especially useful because it provides autostereoscopic intensity images with full parallax, free of any viewing device.

InI is based on so-called integral photography (IP), which was proposed by Lipmann in 1908.¹ Recently the technology has approached the level required for 3D TV and displays.² In the capture stage of an InI system, an array of microlenses generates a collection of 2D elemental images onto a matrix image sensor, such as a CCD: see Figure 1 (top). In the reconstruction stage, the set of elemental images is displayed in front of a second microlens array, providing the observer with a reconstructed 3D image: see Figure 1 (bottom). Note that although all the elemental images are imaged by the corresponding microlenses onto the same plane—the reference image plane—the 3D scene is reconstructed in the image space by the intersection of the ray bundles emanating from each of the microlenses.

Despite major progress, InI systems still pose problems in their standard configuration. For example, the reconstructed images InI provides are real (in front of the film) but pseudoscopic, that is, reversed in depth. Also, owing to the spread of ray bundles for out-of-focus parts of the scene, the reconstructed images have very poor depth of field (DOF).

To attenuate the DOF problem, we have experimented with binary amplitude modulation during the capture stage. Specifically, we have shown that inserting a collection of opaque circular masks centered on the entrance pupil of each microlens—see Figure 2 (top)—greatly improves the DOF of the capture setup.³ Moreover, binary amplitude modulation allows the point spread function (PSF) of the microlenses to degrade very slowly for out-of-focus positions. That makes it is possible to define, to a good approximation, an effective PSF over a wide range of axial positions. Consequently, deconvolution tools can be used to improve the quality of the elemental images.⁴ Since InI requires real-time processing for a high number of elemental images, a low-time-consuming, noniterative method is convenient for deconvolution tasks. Our choice is Wiener filtering, the optimal linear method for signal recovery. To illustrate, we have calculated the elemental images for the pickup architecture shown in Figure 2 (top). In Figure 2 (middle) and (bottom) we show the elemental images obtained with the conventional setup and the proposed hybrid method, respectively.

A clever attempt to overcome the problem of pseudoscopic reconstruction was made by Herbert E. Ives, who proposed to record a second set of elemental images using the reconstructed image as the object. When the second set of elemental images is used in a second display stage, a real, undistorted, orthoscopic 3D image emerges. Yet the approach does not completely solve the pseudoscopic-to-orthoscopic (PO) conversion problem. This is because the two-step recording process entails substantial image degradation owing to diffraction effects and also to the pixilated structure of the matrix image sensor and display setups.

What we suggest here is to employ Ives's PO concept but to perform the intermediate stage (Display 1 + Pickup 2) using pixel mapping.⁵Figure 3 presents this hybrid procedure schematically. In Pickup 1, a set of elemental images is optically recorded in the CCD. Each elemental image records a different perspective of the object, and all the pixels of the elemental cell are charged. When these elemental images are viewed through the central pinhole in Display 1, a faceting effect occurs: a different portion of the reconstructed image is seen through each microlens.⁶

The result is that, through the upper microlens, the central pinhole can see only the ‘blue’ part of the reconstructed image, which indeed was recorded in the ‘blue’ pixel of the upper microcell. A similar process occurs when the reconstructed image is seen through the other pinholes. This pixel mapping provides elemental image Set 2, which, via optical Display 2, allows the reconstruction of a realistic image.