 Astronomy
 Biomedical Optics & Medical Imaging
 Defense & Security
 Electronic Imaging & Signal Processing
 Illumination & Displays
 Lasers & Sources
 Micro/Nano Lithography
 Nanotechnology
 Optical Design & Engineering
 Optoelectronics & Communications
 Remote Sensing
 Sensing & Measurement
 Solar & Alternative Energy
 Sign up for Newsroom EAlerts
 Information for:
Advertisers




Electronic Imaging & Signal Processing
Compressive lightfield imaging
Amit Ashok and Mark A. Neifeld
Compressive lightfield imagers employ fewer photonefficient measurements, enabling higherresolution reconstruction than is possible using their traditional counterparts.
19 August 2010, SPIE Newsroom. DOI: 10.1117/2.1201008.003113
‘Light field’ refers to the spatioangular distribution of light rays in free space emanating from a 3D object volume (see Figure 1).^{1,2} The rapid growth of computing power, following Moore's law, has largely addressed the computational challenge of processing lightfield data to achieve capabilities such as digital refocusing and depth of field control. However, traditional lightfield imagers, for example, the plenoptic camera^{2,3} and the integral imager,^{4,5} suffer from an inherent spatioangular resolution tradeoff^{6} that typically results in ‘lowresolution’ measurements. This tradeoff is one of the main hurdles in extending lightfield imaging to a wider class of applications such as 3D photography and 3D microscopy. Figure 1. The light field, ℓ(s, t, u, v)—parameterized by the spatial location (s,t) and angle/slope (u,v) of each ray—is a 4D scalar quantity. z=0 , z =Δz, z=∞: Observation planes. Recent studies have reported success in mitigating the problem by making a series of measurements—scanning in either the angular or spatial dimension—to synthesize a higherresolution light field.^{7} However, these ‘sampling’ approaches require a large number of measurements over a longer exposure time, which is undesirable in many applications. More important, sampling does not exploit the inherent spatioangular redundancies present in the light field of a natural scene and consequently are photoninefficient. We describe two architectures for compressive lightfield imaging that exploit correlations along these dimensions. Such compressive imagers acquire fewer photonefficient measurements over a shorter exposure time relative to conventional imagers employing noncompressive techniques. The angular compressive lightfield (ACLF) imager employs architecture described elsewhere.^{7} Here a particular configuration of the amplitude mask (K×K elements) modulates the angular dimension of the light field: see Figure 2(a). The resulting measurement is a 2D projection of the light field along that dimension. Alternatively, the spatial compressive lightfield (SCLF) imager employs a modified plenoptic camera architecture, where an amplitude mask (K×K elements) is inserted immediately before each lenslet: see Figure 2(b). Here the amplitude mask modulates the spatial dimension of the light field, and the corresponding measurement represents a 2D projection of the light field along that dimension. Both ACLF and SCLF imagers employ a scheme where the number of measurements M is less than the angular or spatial dimensionality K^{2} of the light field. M measurements are acquired within a total exposure time of T^{exp}=L×T^{exp}_{0}, where T^{exp}_{0} corresponds to the exposure time of a conventional measurement without an amplitude mask. Thus L indicates the number of such exposure times that comprise the total exposure time T^{exp}. Note that the amplitude mask employed in each compressive lightfield imager can be implemented by a programmable liquidcrystal spatial light modulator (LCSLM) or a digitalmirrorarray device (DMD). Figure 2. Compressive lightfieldimager architectures: (a) ACLF and (b) SCLF. P ^{ang}_{k }or P ^{spt}_{k}: k ^{th }projection vector from projection matrix P ^{ang} or P ^{spt}. (m, n): Light field at spatial location (m, n). g ^{ang }_{k}(m, n): k ^{th }measurement at spatial location (m, n) corresponding to the projection vector P ^{ang}_{k}. N: Number of detectors in the local plane array. K: Number of elements along each side of a K×K spatial mask. s _{o}: Object distance. s _{i}: Image distance. f, f _{1}, f _{2}: Focal length of lens. : Local light field centered at spatial location (i, j). : k ^{th} measurement at spatial location (i, j) corresponding to the projection vector P ^{spt}_{k}. In a compressive lightfield imager, the set of amplitudemask configurations comprises the compressive measurement basis. Here we consider two: the principal component (PC), or KarhunenLoève basis, and the binary Hadamard basis. We used a training dataset composed of five highresolution light fields taken from Stanford's lightfield archive^{8} to construct the projection matrices for the PC and the Hadamard bases (K=8). Because these matrices contain negative elements that cannot be physically implemented using an amplitude mask, we used a ‘dualrail’ measurement scheme.^{9} The light field is reconstructed from the compressive measurements using the linear minimum mean square error operator. We evaluate ACLF and SCLF imager performance using two lightfield samples that are distinct from the training dataset. We use the normalized root mean square error (RMSE) metric (expressed as a percentage of the dynamic range) to quantify the fidelity of the lightfield estimate. Here we consider a sensor with 10bit—i.e., 0 to 1023—dynamic range and noise standard deviation of 1. Figure 3(a) shows a plot of the reconstruction RMSE vs. M for the ACLFPC and ACLFH imagers using the PC and Hadamard bases, respectively. Note that for both imagers, the RMSE decreases initially, reaching minimum at M_{opt}, and then starts to increase with increasing M. Two underlying mechanisms determine this behavior: truncation error and measurement signaltonoise ratio.^{10} Comparing an ACLFPC imager with a conventional lightfield (CONV) imager (for CONV, M=K^{2}=64) shows nearly one to two orders of magnitude performance improvement for small values of L (see RMSE data in Table 1). For instance, at L=16, the ACLFPC imager RMSE=3.7%, while the CONV imager RMSE=25%. Observe that for nearly all values of L, the ACLFPC imager outperforms the ACLFH imager in terms of M_{opt} because of the superior compressibility of the PC basis despite its slightly inferior photonthroughput efficiency. Comparing the relative performance of the ACLFPC and ACLFH imagers operating in noncompressive mode, i.e., where M=K^{2}=64, shows that the Hadamard basis always achieves the best performance among all three bases (PC, Hadamard, and identity for CONV) because of its superior lightthroughput efficiency. Figure 4(a) shows reconstructed lightfield images at four different angular positions for the ACLFPC, ACLFH, and CONV imagers. The ACLFPC imager with M=22 and L=16 offers comparable visual image quality as the CONV imager, which requires a four times longer exposure time and three times as many measurements (M=64 and L=64). Figure 3. RMSE performance of (a) ACLFPC and ACLFH imagers and (b) SCLFPC and SCLFH imagers as a function of M for four exposure times specified by L. Table 1. Root mean square error (RMSE) performance of angular compressive lightfield (ACLF) and spatial compressive lightfield (SCLF) imagers, operating in compressive and noncompressive modes, and the conventional lightfield (CONV) imager. L: Increasing exposure time. PC: Principal component basis. H: Binary Hadamard basis. Mopt: Minimum RMSE. RMSE ↓ Exp. Time→  L=16  L=22  L=32  L=64  ACLFPC (M_{opt})  3.7%(16)  3.4%(17)  3.15%(22)  2.6% (30)  ACLFH (M_{opt})  4.0%(26)  3.5%(35)  3.0%(35)  2.1% (60)  SCLFPC (M_{aitopt})  2.35%(11)  2.2%(14)  1.9%(22)  1.4% (27)  SCLFH (M_{opt})  2.4%(23)  2.2%(23)  1.9%(44)  1.2% (44)  ACLFPC (M=64)  8.4%  7.8%  6.85%  4.6%  ACLFH (M=64)  6.8%  5.5%  4.1%  2.2%  SCLFPC (M=64)  4.4%  3.9%  3.3%  2.3%  SCLFH (M=64)  3.6%  3.1%  2.5%  1.55%  CONV (M=64)  25%  18%  12.5%  6.25% 
A plot of the reconstruction RMSE vs. M for the SCLFPC and SCLFH imagers—see Figure 3(b)—shows performance trends that are qualitatively similar to those observed for ACLF imagers, and indicates that the SCLFPC system outperforms the SCLFH. Further, we observe that with the PC basis, the ACLF imager performs better than its SCLF counterpart by a factor of nearly two for small L values. This suggests higher spatial compressibility compared with angular compressibility of light fields using the PC basis. A visual inspection of the light field reconstructions—see Figure 4(b)—confirms this observation. In general, we note that SCLF imagers require fewer compressive measurements to achieve the same RMSE than do ACLF imagers. Figure 4. Lightfield image reconstructions. (a) ACLF imager. (top row) Compressive: L=16 . (left) ACLFPC, M_{opt}=22 and (right) ACLFH, M_{opt}=26. (bottom row) Noncompressive: L=64. (left) ACLFH and (right) CONV. (b) SCLF architecture. (top row) Compressive: L=16. (left) SCLFPC, M_{opt}=11and (right) SCLFH, M_{opt}=22. (bottom row) Noncompressive: L=64. (left) SCLFH and (right) CONV. The class of compressive lightfield imagers discussed here achieves compression in either the spatial or angular dimension of a light field. We believe that it is possible to further improve compressive performance by exploiting the joint spatioangular correlations present in the field. Moreover, employing a hybrid measurement basis^{11} will help to extend application of compressive lightfield imagers to a wider class of natural scenes. We intend to pursue further work along these two directions.
Amit Ashok, Mark A. Neifeld Department of Electrical and Computer Engineering University of Arizona Tucson, AZ Amit Ashok is a senior research scientist. His research includes computational and compressive optical imaging, Bayesian inference, statistical signal processing, and information theory. Mark A. Neifeld is VonBehren Professor of Electrical and Computer Engineering. His research interests include information and communication theoretical methods in image processing, nontraditional imaging that exploits the joint optimization of optical and postprocessing degrees of freedom, coding for nonlinear fiber channels, and applications of slow and fast light for pulse shaping and storage.


