# Compressive light-field imaging

‘Light field’ refers to the spatio-angular distribution of light rays in free space emanating from a 3D object volume (see Figure 1).^{1,2} The rapid growth of computing power, following Moore's law, has largely addressed the computational challenge of processing light-field data to achieve capabilities such as digital refocusing and depth of field control. However, traditional light-field imagers, for example, the plenoptic camera^{2,3} and the integral imager,^{4,5} suffer from an inherent spatio-angular resolution trade-off^{6} that typically results in ‘low-resolution’ measurements. This trade-off is one of the main hurdles in extending light-field imaging to a wider class of applications such as 3D photography and 3D microscopy.

Recent studies have reported success in mitigating the problem by making a series of measurements—scanning in either the angular or spatial dimension—to synthesize a higher-resolution light field.^{7} However, these ‘sampling’ approaches require a large number of measurements over a longer exposure time, which is undesirable in many applications. More important, sampling does not exploit the inherent spatio-angular redundancies present in the light field of a natural scene and consequently are photon-inefficient. We describe two architectures for compressive light-field imaging that exploit correlations along these dimensions. Such compressive imagers acquire fewer photon-efficient measurements over a shorter exposure time relative to conventional imagers employing noncompressive techniques.

The angular compressive light-field (ACLF) imager employs architecture described elsewhere.^{7} Here a particular configuration of the amplitude mask (*K*×*K* elements) modulates the angular dimension of the light field: see Figure 2(a). The resulting measurement is a 2D projection of the light field along that dimension. Alternatively, the spatial compressive light-field (SCLF) imager employs a modified plenoptic camera architecture, where an amplitude mask (*K*×*K* elements) is inserted immediately before each lenslet: see Figure 2(b). Here the amplitude mask modulates the spatial dimension of the light field, and the corresponding measurement represents a 2D projection of the light field along that dimension. Both ACLF and SCLF imagers employ a scheme where the number of measurements *M* is less than the angular or spatial dimensionality *K*^{2} of the light field. *M* measurements are acquired within a total exposure time of T^{exp}=L×T^{exp}_{0}, where T^{exp}_{0} corresponds to the exposure time of a conventional measurement without an amplitude mask. Thus *L* indicates the number of such exposure times that comprise the total exposure time T^{exp}. Note that the amplitude mask employed in each compressive light-field imager can be implemented by a programmable liquid-crystal spatial light modulator (LC-SLM) or a digital-mirror-array device (DMD).

In a compressive light-field imager, the set of amplitude-mask configurations comprises the compressive measurement basis. Here we consider two: the principal component (PC), or Karhunen-Loève basis, and the binary Hadamard basis. We used a training dataset composed of five high-resolution light fields taken from Stanford's light-field archive^{8} to construct the projection matrices for the PC and the Hadamard bases (K=8). Because these matrices contain negative elements that cannot be physically implemented using an amplitude mask, we used a ‘dual-rail’ measurement scheme.^{9} The light field is reconstructed from the compressive measurements using the linear minimum mean square error operator. We evaluate ACLF and SCLF imager performance using two light-field samples that are distinct from the training dataset. We use the normalized root mean square error (RMSE) metric (expressed as a percentage of the dynamic range) to quantify the fidelity of the light-field estimate. Here we consider a sensor with 10-bit—i.e., 0 to 1023—dynamic range and noise standard deviation of 1.

Figure 3(a) shows a plot of the reconstruction RMSE vs. * M* for the ACLF-PC and ACLF-H imagers using the PC and Hadamard bases, respectively. Note that for both imagers, the RMSE decreases initially, reaching minimum at

*M*

_{opt}, and then starts to increase with increasing

*M*. Two underlying mechanisms determine this behavior: truncation error and measurement signal-to-noise ratio.

^{10}Comparing an ACLF-PC imager with a conventional light-field (CONV) imager (for CONV, M=K

^{2}=64) shows nearly one to two orders of magnitude performance improvement for small values of

*L*(see RMSE data in Table 1). For instance, at L=16, the ACLF-PC imager RMSE=3.7%, while the CONV imager RMSE=25%. Observe that for nearly all values of

*L*, the ACLF-PC imager outperforms the ACLF-H imager in terms of M

_{opt}because of the superior compressibility of the PC basis despite its slightly inferior photon-throughput efficiency. Comparing the relative performance of the ACLF-PC and ACLF-H imagers operating in noncompressive mode, i.e., where M=K

^{2}=64, shows that the Hadamard basis always achieves the best performance among all three bases (PC, Hadamard, and identity for CONV) because of its superior light-throughput efficiency. Figure 4(a) shows reconstructed light-field images at four different angular positions for the ACLF-PC, ACLF-H, and CONV imagers. The ACLF-PC imager with M=22 and L=16 offers comparable visual image quality as the CONV imager, which requires a four times longer exposure time and three times as many measurements (M=64 and L=64).

RMSE ↓ Exp. Time→ | L=16 | L=22 | L=32 | L=64 |

ACLF-PC (M_{opt}) | 3.7%(16) | 3.4%(17) | 3.15%(22) | 2.6% (30) |

ACLF-H (M_{opt}) | 4.0%(26) | 3.5%(35) | 3.0%(35) | 2.1% (60) |

SCLF-PC (M_{aitopt}) | 2.35%(11) | 2.2%(14) | 1.9%(22) | 1.4% (27) |

SCLF-H (M_{opt}) | 2.4%(23) | 2.2%(23) | 1.9%(44) | 1.2% (44) |

ACLF-PC (M=64) | 8.4% | 7.8% | 6.85% | 4.6% |

ACLF-H (M=64) | 6.8% | 5.5% | 4.1% | 2.2% |

SCLF-PC (M=64) | 4.4% | 3.9% | 3.3% | 2.3% |

SCLF-H (M=64) | 3.6% | 3.1% | 2.5% | 1.55% |

CONV (M=64) | 25% | 18% | 12.5% | 6.25% |

A plot of the reconstruction RMSE vs. *M* for the SCLF-PC and SCLF-H imagers—see Figure 3(b)—shows performance trends that are qualitatively similar to those observed for ACLF imagers, and indicates that the SCLF-PC system outperforms the SCLF-H. Further, we observe that with the PC basis, the ACLF imager performs better than its SCLF counterpart by a factor of nearly two for small *L* values. This suggests higher spatial compressibility compared with angular compressibility of light fields using the PC basis. A visual inspection of the light field reconstructions—see Figure 4(b)—confirms this observation. In general, we note that SCLF imagers require fewer compressive measurements to achieve the same RMSE than do ACLF imagers.

The class of compressive light-field imagers discussed here achieves compression in either the spatial or angular dimension of a light field. We believe that it is possible to further improve compressive performance by exploiting the joint spatio-angular correlations present in the field. Moreover, employing a hybrid measurement basis^{11} will help to extend application of compressive light-field imagers to a wider class of natural scenes. We intend to pursue further work along these two directions.

Amit Ashok is a senior research scientist. His research includes computational and compressive optical imaging, Bayesian inference, statistical signal processing, and information theory.

Mark A. Neifeld is VonBehren Professor of Electrical and Computer Engineering. His research interests include information and communication theoretical methods in image processing, nontraditional imaging that exploits the joint optimization of optical and postprocessing degrees of freedom, coding for nonlinear fiber channels, and applications of slow and fast light for pulse shaping and storage.

*J. Math. Phys.*18, pp. 51-151, 1939. Trans. P. Moon and G. Timoshenko