The first video standards were formalized in 1940, enabling standard definition television. Since then, single-channel communications bandwidth has increased by six to seven orders of magnitude. Information processing systems have grown from discrete relays to integrated circuits, with more than a billion transistors. Over the same period, video image quality has improved from standard definition to high definition, a net fivefold improvement in pixel count. However, even the most recent ‘4K by 4K’ ultra-high definition displays improve 1940 standards by less than two orders of magnitude. This anemic growth is primarily because video bandwidth cannot exceed broadcast capacity, and because of the assumption that images matched to human visual acuity are sufficient. However, optical and electronic limitations have also played a role.
Cable television offered possibilities for reducing fundamental bandwidth restrictions, but the greatest developments in video quality have been on Internet and cellular networks. Even when display capacity is limited to just a few megapixels, networked users seek interactive media that is digitally zoomable in space, color, focus, and time (between slow- and accelerated motion). Broadcast and user interface restrictions are removed, so that image quality is limited by camera capacity and, ultimately, by atmospheric conditions and photon dynamics (quantum optics).
Pixel count is the most common measure of image quality. A generalized pixel count includes the number of colors, focal- and temporal ranges a camera resolves. In most applications, atmospheric distortion limits the number of 2D pixels and range values a camera sees, and photon flux limits the number of temporal and spectral values. Under typical conditions, the fundamental information limit exceeds 1015 pixels per second.1 Current cameras operate far below this limit.
A multiscale camera,2,3 which consists of a large-aperture, spherical objective lens surrounded by an array of smaller aperture microcameras, enables near diffraction-limited imaging over a wide resolution range.4 Multiscale cameras easily map onto parallel electronic interfaces for broadband digital data management.5 Figure 1 shows an image integrated from 10 microcameras in the 382 microcamera-capacity ‘AWARE 10’ system. AWARE 10 supports 25 microradian resolution, which is approximately 10 times better than human acuity over a 100 degree field of view (see Figures 2 and 3). To achieve this level of resolution, AWARE 10 uses fast, local focus in each microcamera. The results are seen in the photograph in Figure 2, which shows focal disparity between the foreground tree and the background building, just a few meters behind.
Figure 1. An image taken using the multiscale ‘AWARE 10’ camera, demonstrating high resolution over a wide field of view.
A detail of Figure 1
, showing fast focal disparity between the foreground tree and the building background at 100m.
Detail of Figure 1
, showing window at 200m.
While AWARE 10 demonstrates the feasibility of imaging at the optical limits of transverse resolution, capturing the full information capacity of the field requires camera operation at >10,000 frames per second with >100 colors and >10 focal ranges. We can reach these limits using physical layer compression, where we analog-code pixel data to reduce the bandwidth necessary to read out the image. Under development are strategies using specific lenses, sensor design, and coded apertures—where we use a transmission mask to modulate image data for physical layer compression—to achieve multi-aperture6 and coded-aperture7–9 coding. This would enable us to achieve compressive temporal and focal10 sampling, to make the needed bandwidth reductions. Compression by factors of 100–1000, combined with parallel electronic read-out and compressive analog-digital electronic conversion, may allow for the development of cameras approaching the physical limit of one million gigapixels per second.1
The first quarter century of digital photography has been characterized by ‘megapixel wars,’ as designers have struggled to make systems that match the capabilities of human vision and film-based cameras. Recent progress shows that continuing developments will transform concept imaging, as digital photography achieves spatial, temporal, spectral and focal resolution well in excess of human capabilities. We expect to make 10–20 AWARE cameras available for testing and event capture in 2013, and anticipate that these systems will evolve to accommodate very high-speed resolution video recording and broadcast over the next decade.
This work was conducted under the Defense Advanced Research Projects Agency (DARPA) ‘AWARE’ Wide Field of View program, under the direction of Nibir Dhar.
David Brady, Daniel Marks
Electrical and Computer Engineering
David J. Brady is Fitzpatrick Professor of Photonics and leads the Duke Imaging and Spectroscopy Program. He is the author of Optical Imaging and Spectroscopy and is a Fellow of SPIE, the Institute of Electrical and Electronics Engineers (IEEE), and the Optical Society of America (OSA).
Daniel L. Marks is an associate research professor, and leads the AWARE program optics design group. His numerous contributions to computational optical imaging systems include development of optical projection tomography, interferometric synthetic aperture microscopy and compressive coherence imaging.
1. D. J. Brady, D. L. Marks, S. Feller, M. E. Gehm, D. R. Golish, E. M. Vera, D. S. Kittle, Petapixel photography and the limits of camera information capacity, Proc. SPIE
8657, p. 8657-35, 2013. doi:10.1117/12.2014274
2. D. J. Brady, N. Hagen, Multiscale lens design, Opt. Express 17(13), p. 10659-10674, 2009.
3. D. J. Brady, M. E. Gehm, R. A. Stack, D. L. Marks, D. S. Kittle, D. R. Golish, E. M. Vera, S. D. Feller, Multiscale gigapixel photography, Nature 486(7403), p. 386-389, 2012.
4. D. L. Marks, E. J. Tremblay, J. E. Ford, D. J. Brady, Microcamera aperture scale in monocentric gigapixel cameras, Appl. Opt. 50(30), p. 5824-5833, 2011.
5. D. R. Golish, E. M. Vera, K. J. Kelly, Q. Gong, P. A. Jansen, J. M. Hughes, D. S. Kittle, D. J. Brady, M. E. Gehm, Development of a scalable image formation pipeline for multiscale gigapixel photography, Opt. Express 20(20), p. 22048-22062, 2012.
6. M. Shankar, N. P. Pitsianis, D. J. Brady, Compressive video sensors using multichannel imagers, Appl. Opt. 49(10), p. B9-B17, 2010.
7. N. P. Pitsianis, D. J. Brady, X. B. Sun, Sensor-layer image compression based on the quantized cosine transform, Proc. SPIE
5817, p. 250-257, 2005. doi:10.1117/12.604921
8. D. J. Brady, Optical Imaging and Spectroscopy, Wiley/OSA, 2009.
9. Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, S. K. Nayar, Video from a single coded exposure photograph using a learned over-complete dictionary, 2011 IEEE Int'l Conf. Comp. Vision (ICCV)
, p. 287-294, 2011. doi:10.1109/ICCV.2011.6126254
10. D. J. Brady, D. L. Marks, Coding for compressive focal tomography, Appl. Opt. 50(22), p. 4436-4449, 2011.