Theory and Methods of Lightfield Photography (SC980)Course Level: Intermediate
Lightfield photography is based on capturing discrete representations of all light rays in a volume of 3D space. Since light rays are characterized with 2D position and 2D direction (relative to a plane of intersection), lightfield photography captures 4D data. In comparison, conventional photography captures 2D images. Multiplexing this 4D radiance data onto conventional 2D sensors demands sophisticated optics and imaging technology. Rending an image from the 4D lightfield is accomplished computationally based on creating 2D integral projections of the 4D radiance. Optical transformations can also be applied computationally, enabling effects such as computational focusing anywhere in space. This course presents a comprehensive development of lightfield photography, beginning with theoretical ray optics fundamentals and progressing through real-time GPU-based computational techniques. Although the material is mathematically rigorous, our goal is simplicity. Emphasizing fundamental underlying ideas leads to the development of surprisingly elegant analytical techniques. These techniques are in turn used to develop and characterize computational techniques, model lightfield cameras, and analyze resolution. The course also demonstrates practical approaches and engineering solutions. The course includes a hands-on demonstration of several working plenoptic cameras that implement different methods for radiance capture, including the micro-lens approach of Lippmann, the mask-enhanced "heterodyning" camera, the lens-prism camera, multispectral and polarization capture, and the plenoptic 2.0 camera. One section of the course is devoted specifically to the commercially available Lytro camera. Various computational techniques for processing captured data are demonstrated, including basic rendering, Ng's Fourier slice algorithm, the heterodyned light-field approach for computational refocusing, glare reduction, super-resolution, artifact reduction, and others.
This course will enable you to:
- formulate arbitrary lens systems in terms of matrix optics, i.e., to use matrix operations to express ray propagation
- formulate typical lightfield photography problems in terms of the radiance in 4D ray space using ray propagation computations, enabling you to design and construct different plenoptic cameras both theoretically and as an engineering task
- classify plenoptic cameras into version 1.0 and 2.0 and analyze the reasons for the higher resolution of 2.0 cameras
- construct your own Plenoptic, 3D, HDR, multispectral or Superresolution cameras
- write GPU-based applications to perform lightfield rendering of the captured image in real time
- develop approaches to artifact reduction
This course is intended for anyone interested in learning about lightfield photography. Prerequisites are basic familiarity with ray optics, image processing, linear algebra, and programming. Deeper involvement in one or several of those areas is a plus, but not required to understand the course.
Todor Georgiev is a principal engineer at Qualcomm. With background in theoretical physics, he concentrates on applications of mathematical methods taken from physics to image processing. Todor was previously with Adobe Systems, where he authored the Photoshop Healing Brush (a tool on which Poisson image editing was based). He works on theoretical and practical ideas in optics and computational photography, including plenoptic cameras and radiance capture. He has a number of papers and patents in these and related areas.
Andrew Lumsdaine received his PhD degree in electrical engineering and computer science from the Massachusetts Institute of Technology in 1992. He is presently a professor of computer science at Indiana University, where he is also the director of the Center for Research in Extreme Scale Technologies. His research interests include computational science and engineering, parallel and distributed computing, programming languages, numerical analysis, and computational photography. He is a member of the IEEE, the IEEE Computer Society, the ACM, and SIAM.