Compressive light-field imaging

‘Light field’ refers to the spatio-angular distribution of light rays in free space emanating from a 3D object volume (see Figure 1).^1,2 The rapid growth of computing power, following Moore's law, has largely addressed the computational challenge of processing light-field data to achieve capabilities such as digital refocusing and depth of field control. However, traditional light-field imagers, for example, the plenoptic camera^2,3 and the integral imager,^4,5 suffer from an inherent spatio-angular resolution trade-off⁶ that typically results in ‘low-resolution’ measurements. This trade-off is one of the main hurdles in extending light-field imaging to a wider class of applications such as 3D photography and 3D microscopy.

Figure 1. The light field, ℓ(s, t, u, v)—parameterized by the spatial location (s,t) and angle/slope (u,v) of each ray—is a 4D scalar quantity. z=0 , z =Δz, z=∞: Observation planes.

Recent studies have reported success in mitigating the problem by making a series of measurements—scanning in either the angular or spatial dimension—to synthesize a higher-resolution light field.⁷ However, these ‘sampling’ approaches require a large number of measurements over a longer exposure time, which is undesirable in many applications. More important, sampling does not exploit the inherent spatio-angular redundancies present in the light field of a natural scene and consequently are photon-inefficient. We describe two architectures for compressive light-field imaging that exploit correlations along these dimensions. Such compressive imagers acquire fewer photon-efficient measurements over a shorter exposure time relative to conventional imagers employing noncompressive techniques.

The angular compressive light-field (ACLF) imager employs architecture described elsewhere.⁷ Here a particular configuration of the amplitude mask (K×K elements) modulates the angular dimension of the light field: see Figure 2(a). The resulting measurement is a 2D projection of the light field along that dimension. Alternatively, the spatial compressive light-field (SCLF) imager employs a modified plenoptic camera architecture, where an amplitude mask (K×K elements) is inserted immediately before each lenslet: see Figure 2(b). Here the amplitude mask modulates the spatial dimension of the light field, and the corresponding measurement represents a 2D projection of the light field along that dimension. Both ACLF and SCLF imagers employ a scheme where the number of measurements M is less than the angular or spatial dimensionality K² of the light field. M measurements are acquired within a total exposure time of T^exp=L×T^exp₀, where T^exp₀ corresponds to the exposure time of a conventional measurement without an amplitude mask. Thus L indicates the number of such exposure times that comprise the total exposure time T^exp. Note that the amplitude mask employed in each compressive light-field imager can be implemented by a programmable liquid-crystal spatial light modulator (LC-SLM) or a digital-mirror-array device (DMD).

Figure 2. Compressive light-field-imager architectures: (a) ACLF and (b) SCLF. P^ang_k or P^spt_k: k^th projection vector from projection matrix P^ang or P^spt. (m, n): Light field at spatial location (m, n). g^ang _k(m, n): k^th measurement at spatial location (m, n) corresponding to the projection vector P^ang_k. N: Number of detectors in the local plane array. K: Number of elements along each side of a K×K spatial mask. s_o: Object distance. s_i: Image distance. f, f₁, f₂: Focal length of lens. : Local light field centered at spatial location (i, j). : k^th measurement at spatial location (i, j) corresponding to the projection vector P^spt_k.

In a compressive light-field imager, the set of amplitude-mask configurations comprises the compressive measurement basis. Here we consider two: the principal component (PC), or Karhunen-Loève basis, and the binary Hadamard basis. We used a training dataset composed of five high-resolution light fields taken from Stanford's light-field archive⁸ to construct the projection matrices for the PC and the Hadamard bases (K=8). Because these matrices contain negative elements that cannot be physically implemented using an amplitude mask, we used a ‘dual-rail’ measurement scheme.⁹ The light field is reconstructed from the compressive measurements using the linear minimum mean square error operator. We evaluate ACLF and SCLF imager performance using two light-field samples that are distinct from the training dataset. We use the normalized root mean square error (RMSE) metric (expressed as a percentage of the dynamic range) to quantify the fidelity of the light-field estimate. Here we consider a sensor with 10-bit—i.e., 0 to 1023—dynamic range and noise standard deviation of 1.

Figure 3(a) shows a plot of the reconstruction RMSE vs. M for the ACLF-PC and ACLF-H imagers using the PC and Hadamard bases, respectively. Note that for both imagers, the RMSE decreases initially, reaching minimum at M_opt, and then starts to increase with increasing M. Two underlying mechanisms determine this behavior: truncation error and measurement signal-to-noise ratio.¹⁰ Comparing an ACLF-PC imager with a conventional light-field (CONV) imager (for CONV, M=K²=64) shows nearly one to two orders of magnitude performance improvement for small values of L (see RMSE data in Table 1). For instance, at L=16, the ACLF-PC imager RMSE=3.7%, while the CONV imager RMSE=25%. Observe that for nearly all values of L, the ACLF-PC imager outperforms the ACLF-H imager in terms of M_opt because of the superior compressibility of the PC basis despite its slightly inferior photon-throughput efficiency. Comparing the relative performance of the ACLF-PC and ACLF-H imagers operating in noncompressive mode, i.e., where M=K²=64, shows that the Hadamard basis always achieves the best performance among all three bases (PC, Hadamard, and identity for CONV) because of its superior light-throughput efficiency. Figure 4(a) shows reconstructed light-field images at four different angular positions for the ACLF-PC, ACLF-H, and CONV imagers. The ACLF-PC imager with M=22 and L=16 offers comparable visual image quality as the CONV imager, which requires a four times longer exposure time and three times as many measurements (M=64 and L=64).

Figure 3. RMSE performance of (a) ACLF-PC and ACLF-H imagers and (b) SCLF-PC and SCLF-H imagers as a function of M for four exposure times specified by L.

A plot of the reconstruction RMSE vs. M for the SCLF-PC and SCLF-H imagers—see Figure 3(b)—shows performance trends that are qualitatively similar to those observed for ACLF imagers, and indicates that the SCLF-PC system outperforms the SCLF-H. Further, we observe that with the PC basis, the ACLF imager performs better than its SCLF counterpart by a factor of nearly two for small L values. This suggests higher spatial compressibility compared with angular compressibility of light fields using the PC basis. A visual inspection of the light field reconstructions—see Figure 4(b)—confirms this observation. In general, we note that SCLF imagers require fewer compressive measurements to achieve the same RMSE than do ACLF imagers.

Figure 4. Light-field image reconstructions. (a) ACLF imager. (top row) Compressive: L=16 . (left) ACLF-PC, M_opt=22 and (right) ACLF-H, M_opt=26. (bottom row) Noncompressive: L=64. (left) ACLF-H and (right) CONV. (b) SCLF architecture. (top row) Compressive: L=16. (left) SCLF-PC, M_opt=11and (right) SCLF-H, M_opt=22. (bottom row) Noncompressive: L=64. (left) SCLF-H and (right) CONV.

The class of compressive light-field imagers discussed here achieves compression in either the spatial or angular dimension of a light field. We believe that it is possible to further improve compressive performance by exploiting the joint spatio-angular correlations present in the field. Moreover, employing a hybrid measurement basis¹¹ will help to extend application of compressive light-field imagers to a wider class of natural scenes. We intend to pursue further work along these two directions.