We will describe how learning techniques can be used to estimate the 3D shape of objects using optical 2D measurements.

Deep learning algorithms offer a powerful means to automatically analyze the content of medical images. However, many biological samples of interest are difficult to resolve with a standard optical microscope. Here, we use a convolutional neural network (CNN) not only to classify images, but also to optimize the physical layout of the microscope itself. We merge an optical model of image formation into the pipeline of a CNN to simultaneously determine an ideal illumination arrangement to highlight important sample features during image acquisition, along with a set of convolutional weights to classify the detected images post-capture.

The development of PlumeNET, a thermal imagery based classifier for aerosolized chemical and biological warfare agents, is detailed. PlumeNET is a convolutional neural network designed for the real-time classification of threat-like plumes from background clutter. The model weights were trained from scratch within the tensorflow API using thermal imagery of simulant plumes recorded at various test events. The performance between different convolutional neural network architectures are compared and benchmarked against models developed using transfer learning from existing models presented in the literature. An analysis of the final PlumeNET model layers through activation mapping is performed.

We propose a computer-aided detection method for breast cancer screening using convolutional neural network (CNN). First, all mammographic images are divided into small rectangular. Each regional image is fed to a CNN for training or testing. A bounding box can be predicted from the CNN outputs, which indicates a cancerous area. Both current and prior mammographic scans are used for breast cancer detection. Two CNNs are created by using the current mammograms and the difference images between current and prior scans, respectively. The prediction results from two CNNs are fused using a supervised classifier (e.g., k-nearest neighbors). The performance of the proposed CAD method is very promising.

Three-dimensional (3D) imaging has recently been applied to human gesture recognition using depth maps from RGB-D sensors. An alternative which has been scarcely explored is 3D Integral Imaging, which has shown to give very competitive results in object reconstruction and recognition tasks, even under challenging conditions (e.g. low illumination, occlusions). In this paper we present results corresponding to the application of the integral imaging 3D acquisition technique for the recognition of human gestures, when there are occlusions that may hinder the recognition capability. We also present results comparing its capability against that given by an RGB-D sensor (Kinect) and that obtained when only one of the cameras in the camera array is used. Our results show that Integral Imaging compares more or less similarly to Kinect and the monocular case when there are not occlusions, but much more favorably when there are.

Computational optical systems allow novel architectures for capturing real world scenes in multiple dimensions and showing them on displays which enable 3D perception for human vision. In this talk, I will explore one such 3D display technique which is based on integral imaging, known as Integral displays or light field displays. I will discuss issues of human 3D perception related to these displays, highlighting their challenges and advantages. Similar to displays, capturing content for these also provides interesting computational imaging challenges. I will talk about two specific capture methods to show content from the real world on these displays.

Recently, 3D imaging sensors such as time-of-flight (ToF) sensors are rapidly gaining widespread adoption in many applications due to their cost effectiveness, simplicity, and compact size. However, current generation ToF cameras suffer from low spatial resolution due to physical fabrication limitations. In this paper, we propose a compressive sensing based ToF sensor capable of achieving high spatial resolution imaging. Based on the proposed model, we developed a prototype 1-megapixel compressive ToF camera that achieves 4 - 6 × improvement in spatial resolution. We will also show the spatial resolution improvement for ToF sensors using the multiple-frame super-resolution technique. By extending the multiple-frame super-resolution technique to the complex-value modeled ToF imaging, the spatial resolution has been greatly improved for the depth image with nine low-resolution ToF measurements.

A novel Fresnel zone light field spectral imager was recently developed that provides snapshot spectral imaging with no scanning or moving parts by using a Fresnel zone plate as the primary optic within a plenoptic camera to encode spectral information on the detector array rather than distance. This talk provides the first performance comparison of standard digital refocusing, 3D deconvolution by Richardson-Lucy, and super resolution plenoptic post-processing algorithms for collecting spectral information about a scene which is very important for the performance of computational photography systems like this. This talk presents the results of this study.

For high ranging resolution and to record full waveform signals, full waveform LiDAR requires short pulse width lasers, high bandwidth sensors and A/Ds, and large on-board memories. To relax these requirements, temporal super-resolution is studied. In temporal super-resolution, the reflected beam is split into several, and then transpose through different distances. The minimum distance difference determines the resolution. To reduce the data amount, each beam is modulated by an EOM, and then detected, digitized, and sub-sampled. An experiment using a 40ns pulse width laser with a 25MHz detector and a 25 MHz A/D converter demonstrates a 1.5m ranging resolution.

Non-spherical aerosols, particularly aggregates and those comprised of rough surfaces, produce complex light scattering patterns that deviate considerably from those of their spherical counterparts. Consequently, discerning particle morphology from the complex scattering pattern, i.e., the inverse problem, is difficult at best. Additional information is required to associate uniquely the interference pattern resulting from scattered light and the particle’s morphology (size, shape, etc.). This uniqueness challenge of the inverse problem may be overcome by incorporating digital holographic imaging into the light scattering apparatus. Using a color CCD camera, we demonstrate that two-dimensional light scattering patterns and digital holograms from individual flowing aerosol particles may be recorded simultaneously at different wavelengths revealing the complex scattering pattern along with the size, shape, and orientation of the particle at the instant the scattering occurs. Knowing the exact scattering pattern associated with an exact particle morphology will improve the understanding the radiative characteristics of non-spherical atmospheric aerosols, and reduce uncertainties of important physical parameters such as radiative forcing of aerosols.

In this talk, we will present the applications of inverse scattering principles with digital holography. First, I will present the recently developed 3-D holotomography setup using a dynamic mirror device, which is an optical analogous to X-ray computed tomography. In particular, I will discuss the visualization of 3D refractive index distributions of biological cells and tissues measured with the 3-D holotomography using the transfer function method. In addition, we also discuss the application of inverse scattering principle for highly scattering layers. With wavefront shaping techniques using digital holography, we demonstrate ultra-high-definition dynamic holographic display exploiting large space-bandwidth in volume speckle.

The temporal resolution of quantitative phase imaging (QPI) methods such as Fourier Ptychographic Microscopy and Differential Phase Contrast is limited due to their need for multiple illumination-encoded measurements. This inhibits their use for imaging fast moving live samples. We present a novel approach to correct motion during QPI that relies on a color-multiplexed navigator signal to register measurements prior to phase retrieval. The motion navigator enables accurate estimation of complex sample motion and standard techniques are used to registration measurements between time points. To validate our method, we demonstrate experimentally the reduction in blur for live sample motion.

Following our previous work on a structured illumination microscope (SIM) based on a two-dimensional (2D) pattern generated by the incoherent illumination of a Fresnel biprism using one single linear source (e. g. slit), in this contribution, we report on a new system in which several equidistant slits are used to generate a 3D structured pattern. The pattern in our proposed incoherent SIM system is characterized by axially-localized tunable-frequency fringes. 3D experimental images of test objects are compared with simulated results to validate our implementation.

In the visible range, imaging and sensing techniques with coherent light encode information as intensity modulations which are accessible to the available square-law detectors. Systems typically modally-limited the fields so that the resulting variations are resolvable. When they are not resolvable, the field is generally referred to as 'lacking coherence', or 'pseudo-incoherent'. We argue that such a situation can be viewed as merely the multiplexing of numerous, independently-coherent channels, and therefore can be separated by a variety of computational- and compressive-sensing approaches for demultiplexing. As such, the modal-limitation of such systems can be relaxed, improving system performance.

Computational imaging refers to image formation techniques that use digital computation instead of optically capturing every scene pixel. In a computational imager, each detector measures coded information that is digitally post-processed to extract relevant scene information. Computational imaging acquires an image using smaller, cheaper, and/or lower bandwidth components, but not without compromise. NATO SET-232 has undertaken the task of investigating computational imaging and compressive sensing for EO/IR systems. This paper presents an overview of the NATO SET-232 computational imaging and compressive sensing technologies, limitations of the design trade space, algorithm and conceptual design considerations, and field performance assessment and modeling.

There are many applications that have corrupted or missing pixels in images. Here, we present sparsity based image completion algorithms that can achieve high performance in image reconstruction. Through extensive experiments using various types of images, it was demonstrated that our algorithms can deal with extremely high missing rates (up to 99.9%) and relatively large missing blocks.

Long image acquisition time is a critical problem in single-pixel-imaging. Here, we propose a new high-speed single-pixel compressive imaging method. We developed an ADMM based optimization algorithm to handle images with multiple features. The proposed method solves an optimization problem with the objectives of Total-Variation and ell-1-norm with a data-fidelity constraint. The algorithm is highly parallel and is also implemented in GPUs. The proposed method significantly reduces reconstruction times. The system produces high resolution images and can also be used for super-resolution by changing the detector. We verified the system experimentally and compared performance of our algorithm with similar methods.

The single pixel camera is an imaging system particularly well suited for scene acquisition where a matrix detector is unavailable. Because of the long acquisition times, the system is usually aided with common compressive sensing techniques. However, if the scene has moving elements, the recovery will show poor results. In the article we purpose a novel technique for recovery of a moving scene, where there are both a global motion (e.g. because of the camera motion) and motion of the objects in the scene with help of a 'Russian-Dolls' multi-scale sensing matrix.

In this talk, I will provide an overview of computational imaging technologies under development by the NU Comp Photo Lab, covering a large span of physical scales: from the nanoscopic to the astronomic. First, I will introduce a Synthetic Aperture Visual Imaging (SAVI) technique to image at high resolution over remote sensing scales (e.g. 100m-10km). SAVI uses camera arrays coupled with coherent illumination as an effective method of improving spatial resolution in long distance images by a factor of ten and beyond, essentially providing telescopic resolution using commodity hardware. Next, I will introduce a novel structured light technique designed to operate at macroscopic scales (e.g. 1-10m) called Motion Contrast 3D scanning (MC3D). MC3D maximizes bandwidth and light source power to avoid performance trade-offs in structured light 3D acquisition. Lastly, I will present research on coherent diffraction imaging and tomographic reconstruction at the nanoscale.

We present the adaptation of Fourier ptychography to fixed source, reflection-mode EUV microscope.

We present a theoretical formulation and an experimental demonstration of a novel technique for fast compression-less terahertz imaging based on broadband Fourier optics. The technique exploits k-vector/frequency duality in Fourier optics which allows to use a single-pixel detector to perform angular scan along a circular path, while the broadband spectrum is used to scan along the radial dimension in Fourier domain. The proposed compression-less image reconstruction technique (hybrid inverse transform) requires only a small number of measurements that scales linearly with the image linear size, thus promising real-time acquisition of high-resolution THz images.

We present a high-temporal resolution 4D-XCT with feature-based iterative reconstruction method(FBIR) by involving motion modeling as well as imposing feature priors in the reconstruction process. The 4D reconstruction is acquired through an iterative minimization of the cost function which is obtained by combining the forward motion model and the structural feature-based priors. The scheme is applied to the study of the mechanical response of a porous structure (cuttlefish bone) which achieves high temporal resolution and demonstrates robustness against noise, limited views and motion induced blurring.

X-ray phase imaging requires both sufficient spatial coherence of the beam and computational processing of the resulting images. In propagation-based phase imaging with tabletop sources, coherence is typically achieved through the use of low-power microfocus sources. We have previously demonstrated the use of focusing polycapillary optics to generate the required coherence by focusing the X-rays from a large spot source to a focal spot of the required size. However, the resulting phase images suffered from artifacts produced by the structure of the polycapillary optic. Here, we demonstrate the correction of these artifacts and the resulting improvements in phase reconstructions.

X ray phase imaging can offer significantly improved image contrast between materials of similar atomic number compared to traditional imaging, although it typically requires small, low power sources to generate the required spatial coherence. We have demonstrated the use of a simple wire mesh and Fourier transform techniques to overcome this limitation, essentially by observing shifts in the mesh. However, the resolution of that technique is limited by the mesh period. Here we demonstrate greatly improved spatial resolution by the application of wider windowing and appropriate combinations of Fourier components from multiple images acquired while spatially shifting the grid.

We propose and demonstrate that a multimode optical fiber can be used to measure the spectral phase of ultrafast optical pulses. The speckle pattern formed at the end of a multimode fiber provides a fingerprint which can be used to identify the spectral amplitude and phase of an unknown pulse. This technique allows for single-shot pulse characterization in a simple experimental setup. In addition to demonstrating a novel pulse characterization scheme, this work further illustrates the potential of complex photonic structures such as multimode fibers as versatile optical sensing platforms.

Near-field imaging can be accomplished by multiplexing scene information onto a collection of measurements using dynamic metasurface antennas (DMA). This approach can result in cumbersome post-processing, especially when DMAs are moving in a SAR configuration. We address this problem by modifying the range migration algorithm (RMA). The measurements are transformed into those from an effective multistatic configuration, for which specific forms of the algorithm have been derived. We demonstrate the adaptation of the RMA to near-field imaging using a DMA as central hardware of a SAR system, and discuss the effects of various common approximations on the image quality.

In this article we discuss a technique for providing an approximate solution for problems which are formally singular. We develop a method for solving problems with a high degree of singularity (those for which the number of equations is far less than the number of variables). The technique is applied to digital zoom and the results compared with standard interpolation techniques. The development of multi spectral data cubes for tomographic type multi spectral imaging systems is shown with several simulated results based on real data.

A phase only filter is implemented in the pupil plane of an imaging system to engineer a new point spread function with a low peak intensity. Blurred detected images are then reconstructed in post processing through Wiener Deconvolution. A Differential Evolution algorithm is implemented to optimize these filters for high SNR across the MTF. These filters are tested experimentally using a reflective Spatial Light Modulator (SLM) in the pupil of a system and successfully show the peak Irradiance reduced 100 times the diffraction limit. These results are compared to expected performance.

In the modern tactical imaging environment, new computational imaging (CI) systems and algorithms are being used to improve speed and accuracy for detection tasks. Therefore, a measurement technique is needed to predict the performance of complex non-shift invariant EO/IR imaging systems, including CI systems. Detection performance of traditional imaging systems can be modeled using current system metrics and measurements such as Modulation Transfer Function (MTF), Signal to Noise (SNR), and instantaneous Field of View (iFOV). In this correspondence, we propose a technique to experimentally measure a detection sensitivity metric for non-traditional CI systems. The experimental results are compared with theoretical expected values though the Night Vision Integrated Performance Model (NV-IPM). Additionally, we demonstrate the experimental results for a variety of imaging systems (IR, visible, and color), target sizes and orientations, as well as SNR values.

Accurate methods for breast cancer diagnosis are of capital importance for patient's care and treatment. Since the introduction of DCE-MRI, a set of computer aided diagnosis (CAD) systems have been developed with satisfactory performance results. However, in the case of challenging lesions as non-mass like tumors, the low specificity of such methods remains unsatisfactory. Here we propose a new approach to the problem by taking advantage of ten robust classification methods to describe the dynamic behavior of the different tissues by multiple dynamic curves. This CAD system is able to detect, segment and classify malign tissues by using the whole information encoded in the dynamic curves in unseen data with an estimated performance at least at 95% accuracy, 95% sensitivity and 95% specificity.

In the last decade, research for large-scale image retrieval systems has shifted from using local features to applying global aggregation methods such as Fisher Vectors, VLAD or CNN-based methods to encode images with a short fixed-sized vector. We argue that this is reasonable for established public retrieval datasets since they typically show one large object in image center but fails in real-world scenarios dealing with small objects in cluttered backgrounds. Combining public datasets, we therefore generate novel evaluation setups and our experiments reveal that systems based on local features and proper discriminative extensions outperform the global vector methods.