Integral imaging has received significant attention in the past decade as an alternative to holography and stereoscopy for displaying 3D images. It is on its way to becoming integrated into a range of applications, from medical imaging to home entertainment. The advantage of integral imaging is that it is a passive sensing technique that uses existing environmental light in the scene instead of requiring active, controlled illumination. Unlike stereoscopy, it does not require the use of special glasses. Similar to holography, integral imaging provides full parallax so that viewers have more flexibility in observing 3D objects and scenes from different viewpoints.1
Synthetic aperture integral imaging (SAII) provides reconstructed 3D image slices with high pixel resolution. Although this method offers more details for feature extraction and recognition purposes, the covariance matrix is very large. This has been a major issue for processing high-dimensional reconstructed images using SAII. To deal with this computational issue, I have performed feature extraction using principal component analysis (PCA) to reduce the number of data dimensions. Specifically, I applied Murakami's method to compute eigenvectors and eigenvalues. This novel approach yields accurate object recognition while reducing the size of the data set.
To develop a simple, high-performance integral imaging method for the collection and reconstruction of 3D objects, I acquired 2D images of three small toy cars using a ‘pick-up process’ and then reconstructed 3D images of these objects. The camera, which was attached to a rotating stage, moved horizontally and vertically to capture images of the objects at each position (see Figure 1). Each of these images revealed partial information about the objects and is therefore called an elemental image. The camera positions were spaced equally both horizontally and vertically to facilitate the reconstruction process. The resulting elemental images were superimposed to reconstruct 3D images of the objects plane by plane, using the horizontal and vertical offset values calculated based on the camera parameters and the experimental setup (see Figure 2). The offset values were smaller for farther reconstruction planes than for closer ones. In the reconstructed 3D image, the value of a pixel in each slice was the average of all corresponding pixel values of the superimposed elemental images.
Figure 1. Illustration of the pick-up process.
Figure 2. Illustration of the reconstruction process.
Figure 3. Exemplified slices of 3D reconstructed images of three toy cars used in the experiment.
Figure 4. Training and classification using neural networks. PCA: Principal component analysis.
During the experiment, I placed the three toy cars on a rotatable stage. For each of these objects, I carried out the pick-up process using six different angles and reconstructed six 3D images. Figure 3 shows cropped slices of the 3D reconstructed images of the three objects. I extracted and analyzed the image slices using PCA, keeping a low number of principal components with the highest eigenvalues to obtain a new space that retains most of the variance in the data. To overcome the high memory demands required for computing a large covariance matrix with a large number of dimensions, I applied Murakami's method of PCA to obtain a much smaller-sized covariance matrix.1, 2 To prevent the extraction of redundant data, I extracted only image slices with changes in offset pixel values. In most cases, I computed 95% of the total variance by placing the computed eigenvalues in a descending order and summing them to 95% of the total eigenvalues. By computing a very high percentage of the total variance, I could take many more dimensions into account to obtain enough information to characterize the objects.
I then used 3D reconstructed images with a few dimensions to train weights between the layers of neural networks.3, 4 I projected the 3D reconstructed images onto the reduced-dimensional PCA space prior to object classification by neural networks (see Figure 4). I obtained a highly accurate classification rate with a low number of principal components, suggesting that a few features are sufficient to characterize the objects.
In conclusion, I have introduced a novel integral imaging method for collecting images of 3D objects at multiple perspectives and reconstructing 3D images of these objects. By applying Murakami's method of PCA, I have reduced the number of dimensions in the 3D reconstructed image slices while retaining most of the variance in the data. The results indicate that the combination of PCA and neural networks reduces memory requirements and computational time, and yields highly accurate object classification with a limited amount of extracted information. The next step will be to classify 3D objects in a scattering medium to evaluate the performance of the system.
Cuong M. Do
University of Connecticut (UConn)
Cuong Do received his PhD in electrical engineering from UConn (2009) and has worked as a research specialist in the Department of Electrical and Computer Engineering. He is a member of SPIE, the Optical Society of America, and IEEE.
1. C. Do, 3D object recognition with integral imaging using neural networks, Proc. SPIE
8135, pp. 81350D, 2011. doi:10.1117/12.893046
2. H. Murakami, B. Kumar, Efficient calculation of primary images from a set of images, IEEE Trans. Pattern Anal. Machine Intell. 4, no. 51982, 1982.
3. R. O. Duda, P. E. Hart, D. G. Stork, Pattern Classification, 2nd ed., Wiley Interscience, 2001.
4. C. M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995.