In this paper, we present a unifying framework for retinex that is able to reproduce many of the existing retinex implementations within a single model. The fundamental assumption, as shared with many retinex models, is that the observed image is a multiplication between the illumination and the true underlying reflectance of the object. Starting from Morel’s 2010 PDE model for retinex, where illumination is supposed to vary smoothly and where the reflectance is thus recovered from a hard-thresholded Laplacian of the observed image in a Poisson equation, we define our retinex model in similar but more general two steps. First, look for a filtered gradient that is the solution of an optimization problem consisting of two terms: The first term is a sparsity prior of the reflectance, such as the TV or H1 norm, while the second term is a quadratic fidelity prior of the reflectance gradient with respect to the observed image gradients. In a second step, since this filtered gradient almost certainly is not a consistent image gradient, we then look for a reflectance whose actual gradient comes close. Beyond unifying existing models, we are able to derive entirely novel retinex formulations by using more interesting non-local versions for the sparsity and fidelity prior. Hence we define within a single framework new retinex instances particularly suited for texture-preserving shadow removal, cartoon-texture decomposition, color and hyperspectral image enhancement.

We propose a vector space approach for relighting a Lambertian convex object with distant light source, whose crucial task is the decomposition of the reflectance function into albedos (or reflection coefficients) and lightings based on a set of images of the same object and its 3-D model. Making use of the fact that reflectance functions are well approximated by a low-dimensional linear subspace spanned by the first few spherical harmonics, this inverse problem can be formulated as a matrix factorization, in which the basis of the subspace is encoded in the spherical harmonic matrix S. A necessary and sufficient condition on S for unique factorization is derived with an introduction to a new notion of matrix rank called nonseparable full rank. An SVD-based algorithm for exact factorization in the noiseless case is introduced. In the presence of noise, the algorithm is slightly modified by incorporating the positivity of albedos into a convex optimization problem. Implementations of the proposed algorithms are done on a set of synthetic data.

In digital cameras and mobile phones, there is an ongoing trend to increase the image resolution, decrease the sensor size and to use lower exposure times. Because smaller sensors inherently lead to more noise and a worse spatial resolution, digital post-processing techniques are required to resolve many of the artifacts. Color filter arrays (CFAs), which use alternating patterns of color filters, are very popular because of price and power consumption reasons. However, color filter arrays require the use of a post-processing technique such as demosaicing to recover full resolution RGB images. Recently, there has been some interest in techniques that jointly perform the demosaicing and denoising. This has the advantage that the demosaicing and denoising can be performed optimally (e.g. in the MSE sense) for the considered noise model, while avoiding artifacts introduced when using demosaicing and denoising sequentially. In this paper, we will continue the research line of the wavelet-based demosaicing techniques. These approaches are computationally simple and very suited for combination with denoising. Therefore, we will derive Bayesian Minimum Squared Error (MMSE) joint demosaicing and denoising rules in the complex wavelet packet domain, taking local adaptivity into account. As an image model, we will use Gaussian Scale Mixtures, thereby taking advantage of the directionality of the complex wavelets. Our results show that this technique is well capable of reconstructing fine details in the image, while removing all of the noise, at a relatively low computational cost. In particular, the complete reconstruction (including color correction, white balancing etc) of a 12 megapixel RAW image takes 3.5 sec on a recent mid-range GPU.

In this paper, we proposed a new technique for demosaicing a unique RGBZ color-depth imaging sensor, which captures color and depth images simultaneously, with a specially designed color-filter-array (CFA) where two out of six RGB color rows are replaced by “Z” pixels that capture depth information but no color information. Therefore, in an RGBZ image, the red, green and blue colors are more sparsely sampled than in a standard Bayer image. Due to the missing rows in the data image, commonly used demosaicing algorithms for the standard Bayer CFA cannot be applied directly. To this end, our method first fills-in the missing rows to reconstruct a full Bayer CFA, followed by a color-selective adaptive demosaicing algorithm that interpolates missing color components. In the first step, unlike common bilinear interpolation approaches that tend to blur edges, our edge-based directional interpolation approach, derived from de-interlacing techniques, emphasizes reconstructing more straight and sharp edges with fewer artifacts and thereby preserves the vertical resolution in the reconstructed the image. In the second step, to avoid using the newly estimated pixels for demosaicing, the bilateral-filter-based approach interpolates the missing color samples based on weighted average of adaptively selected known pixels from the local neighborhoods. Tests show that the proposed method reconstructs full color images while preserving edges details, avoiding artifacts, and removing noise with high efficiency.

The automatic recomposition of a digital photograph to a more pleasing composition or alternate aspect ratio is a very powerful concept. The human face is arguably one of the most frequently photographed and important subjects. Although evidence suggests only a minority of photos contain faces, the vast majority of images used in consumer photobooks contain faces. Face detection and facial understanding algorithms are becoming ubiquitous to the computational photography community and facial features have a dominating influence on both aesthetic and compositional properties of the displayed image. We introduce a fully automatic recomposition algorithm, capable of zooming in to a more pleasing composition, re-trimming to alternate aspect ratios, or a combination thereof. We use facial bounding boxes, input and output aspect ratios, along with derived composition rules to introduce a facecrop algorithm with superior performance to more complex saliency or region of interest detection algorithms. We further introduce sophisticated facial understanding rules to improve user satisfaction further. We demonstrate through psychophysical studies the improved subjective quality of our method compared to state-of-the-art techniques.

Recent advances allow for the construction of filters with precisely defined frequency response for use in Raman chemical spectroscopy. In this paper we give a probabilistic interpretation of the output of such filters and use this to give an algorithm to design optimal filters to minimize the mean squared error in the estimated photon emission rates for multiple spectra. Experiments using these filters demonstrate that detecting as few as ~10 Raman scattered photons in as little time as ~30μs can be sufficient to positively distinguish chemical species. This speed should allow "chemical imaging" of samples.

The limitations in neutron flux and resolution (1/D) of current neutron imaging systems can be addressed with a Coded Source Imaging system with magnification (xCSI). More precisely, the multiple sources in an xCSI system can exceed the flux of a single pinhole system for several orders of magnitude, while maintaining a higher 1 / D with the small sources. Moreover, designing for an xCSI system reduces noise from neutron scattering, because the object is placed away from the detector to achieve magnification. However, xCSI systems are adversely affected by correlated noise such as non-uniform illumination of the neutron source, incorrect sampling of the coded radiograph, misalignment of the coded masks, mask transparency, and the imperfection of the system Point Spread Function (PSF). We argue that a model-based reconstruction algorithm can overcome these problems and describe the implementation of a Simultaneous Iterative Reconstruction Technique algorithm for coded sources. Design pitfalls that preclude a satisfactory reconstruction are documented.

Bright Field (BF) electron tomography (ET) has been widely used in the life sciences to characterize biological specimens in 3D. While BF-ET is the dominant modality in the life sciences it has been generally avoided in the physical sciences due to anomalous measurements in the data due to a phenomenon called “Bragg scatter” - visible when crystalline samples are imaged. These measurements cause undesirable artifacts in the reconstruction when the typical algorithms such as Filtered Back Projection (FBP) and Simultaneous Iterative Reconstruction Technique (SIRT) are applied to the data. Model based iterative reconstruction (MBIR) provides a powerful framework for tomographic reconstruction that incorporates a model for data acquisition, noise in the measurement and a model for the object to obtain reconstructions that are qualitatively superior and quantitatively accurate. In this paper we present a novel MBIR algorithm for BF-ET which accounts for the presence of anomalous measurements from Bragg scatter in the data during the iterative reconstruction. Our method accounts for the anomalies by formulating the reconstruction as minimizing a cost function which rejects measurements that deviate significantly from the typical Beer’s law model widely assumed for BF-ET. Results on simulated as well as real data show that our method can dramatically improve the reconstructions compared to FBP and MBIR without anomaly rejection, suppressing the artifacts due to the Bragg anomalies.

The monochromatic single frame pixel count of a camera is limited by diffraction to the space-bandwidth product, roughly the aperture area divided by the square of the wavelength. We have recently shown that it is possible to approach this limit using multiscale lenses for cameras with space bandwidth product between 1 and 100 gigapixels. When color, polarization, coherence and time are included in the image data cube, camera information capacity may exceed 1 petapixel/second. This talk reviews progress in the construction of DARPA AWARE gigapixel cameras and describes compressive measurement strategies that may be used in combination with multiscale systems to push camera capacity to near physical limits.

Scanning electron microscopes (SEMs) are used in neuroscience and materials science to image centimeters of sample area at nanometer scales. Since imaging rates are in large part SNR-limited, large collections can lead to weeks of around-the-clock imaging time. To increase data collection speed, we propose and demonstrate on an operational SEM a fast method to sparsely sample and reconstruct smooth images. To accurately localize the electron probe position at fast scan rates, we model the dynamics of the scan coils, and use the model to rapidly and accurately visit a randomly selected subset of pixel locations. Images are reconstructed from the undersampled data by compressed sensing inversion using image smoothness as a prior. We report image fidelity as a function of acquisition speed by comparing traditional raster to sparse imaging modes. Our approach is equally applicable to other domains of nanometer microscopy in which the time to position a probe is a limiting factor (e.g., atomic force microscopy), or in which excessive electron doses might otherwise alter the sample being observed (e.g., scanning transmission electron microscopy).

Computer simulation of metal alloys is an emerging trend in materials development. Simulated replicas of fabricated alloys are based on the segmentations of alloy micrographs. Therefore, accurate segmentation of visible precipitates is paramount to simulation accuracy. Since the shape and size of precipitates are key indicators of physical alloy properties, automated segmentation algorithms must account for abundant prior information of precipitate shape. We present a new method for constructing a prior enforcing rectangular shape which can be applied within a min-cut framework for maximum a-posteriori segmentation.

Despite the ubiquitous use of multi-photon and confocal microscopy measurements in biology, the core techniques typically suffer from fundamental compromises between signal to noise (S/N) and linear dynamic range (LDR). In this study, direct synchronous digitization of voltage transients coupled with statistical analysis is shown to allow S/N approaching the theoretical maximum throughout an LDR spanning more than 8 decades, limited only by the dark counts of the detector on the low end and by the intrinsic nonlinearities of the photomultiplier tube (PMT) detector on the high end. Synchronous digitization of each voltage transient represents a fundamental departure from established methods in confocal/multi-photon imaging, which are currently based on either photon counting or signal averaging. High information-density data acquisition (up to 3.2 GB/s of raw data) enables the smooth transition between the two modalities on a pixel-by-pixel basis and the ultimate writing of much smaller files (few kB/s). Modeling of the PMT response allows extraction of key sensor parameters from the histogram of voltage peak-heights. Applications in second harmonic generation (SHG) microscopy are described demonstrating S/N approaching the shot-noise limit of the detector over large dynamic ranges.

In this paper, we propose the scanning electron microscope (SEM) image blurring model and apply this model to the joint deconvolution and segmentation method which performs deconvolution and segmentation simultaneously. In the field of materials science and engineering, automated image segmentation techniques are critical and getting exact boundary shape is especially important. However, there are still some difficulty in getting good segmentation results when the images have blurring degradation. SEM images have blurring due in part to complex electron interactions during acquisition. To improve segmentation results at object boundaries, we incorporate prior knowledge of this blurring degradation into the existing EM/MPM segmentation algorithm. Experimental results are presented to demonstrate that the proposed method can be used to improve the segmentation of microscope images of materials.

Segmenting materials images is a laborious and time-consuming process and automatic image segmentation algorithms usually contain imperfections and errors. Interactive segmentation is a growing topic in the areas of image processing and computer vision, which seeks to and a balance between fully automatic methods and fully manual segmentation processes. By allowing minimal and simplistic interaction from the user in an otherwise automatic algorithm, interactive segmentation is able to simultaneously reduce the time taken to segment an image while achieving better segmentation results. Given the specialized structure of materials images and level of segmentation quality required, we show an interactive segmentation framework for materials images that has two key contributions: 1) a multi-labeling framework that can handle a large number of structures while still quickly and conveniently allowing manual interaction in real-time, and 2) a parameter estimation approach that prevents the user from having to manually specify parameters, increasing the simplicity of the interaction. We show a full formulation of each of these contributions and example results from their application.

A grid-based Bayesian array (GBA) for robust visual tracking has recently been developed, which proposes a novel method of deterministic sample generation and sample weighting for position estimation. In particular, a target motion model is constructed, predicting target position in the next frame based on estimations in previous frames. Samples are generated by gridding within an ellipsoid centered at the prediction. For localization, radial edge detection is applied for each sample to determine if it is inside the target boundary. Sample weights are then assigned according to the number of the edge points detected around the sample and its distance from the predicted position. The position estimation is computed as the weighted sum of the sample set. In this paper, we enhance the capacity of the GBA tracker in accommodating the tracking of targets in video with erratic motion, by introducing adaptation in the motion model and iterative position estimation. The improved tracking performance over the original GBA tracker are demonstrated in tracking a single leukocyte in vivo and ground vehicle target observed from UAV videos, both undergoing abrupt changes in motion. The experimental results show that the enhanced GBA tracker outperforms the original by tracking more than 10% of the total number of frames, and increases the number of video sequences with all frames tracked by greater than 20%.

Pose estimation and tracking of articulated objects like humans is particularly difficult due to the complex occlusions among the articulated parts. Without the benefit of multiple views, resolution of occlusions becomes both increasingly valuable and challenging. We propose a method for articulated 3D pose estimation from monocular video which uses nonparametric belief propagation and employs a novel and efficient approach to occlusion reasoning. We present a human tracking application, and evaluate results using the HumanEva II data set.

In this paper we discuss the main characteristics (that we consider to be essential) for the design of an efficient optimizer in the context of highly non-convex functions. We consider a specific model known as Marked Point Process (MPP). Given that the probability density is multimodal, and given the size of the configuration space, an exploration phase is essential at the beginning of the algorithm. Next, the fine details of the density function should be discovered. We propose efficient kernels to efficiently explore the different modes of the density, and other kernels to discover the details of each mode. We study the algorithm theoretically to express convergence speeds and to select its best parameters. We also present a simple and generic method to parallelize the optimization of a specific class of MPP models. We validate our ideas first on synthetic data of configurations of different sizes to prove the efficiency of the proposed kernels. Finally we present results on three different applications.

Touch sensitive screens are used in many applications ranging in size from smartphones and tablets to display walls and collaborative surfaces. In this study, we consider optical touch sensing, a technology best suited for large-scale touch surfaces. Optical touch sensing utilizes cameras and light sources placed along the edge of the display. Within this framework, we first find a sufficient number of cameras necessary for identifying a convex polygon touching the screen, using a continuous light source on the boundary of a circular domain. We then find the number of cameras necessary to distinguish between two circular objects in a circular or rectangular domain. Finally, we use Matlab to simulate the polygonal mesh formed from distributing cameras and light sources on a circular domain. Using this, we compute the number of polygons in the mesh and the maximum polygon area to give us information about the accuracy of the configuration. We close with summary and conclusions, and pointers to possible future research directions.

Imaging in low light is problematic as sensor noise can dominate imagery, and increasing illumination or aperture size is not always effective or practical. Computational photography offers a promising solution in the form of the light field camera, which by capturing redundant information offers an opportunity for elegant noise rejection. We show that the light field of a Lambertian scene has a 4D hyperfan-shaped frequency-domain region of support at the intersection of a dual-fan and a hypercone. By designing and implementing a filter with appropriately shaped passband we accomplish denoising with a single all-in-focus linear filter. Drawing examples from the Stanford Light Field Archive and images captured using a commercially available lenselet- based plenoptic camera, we demonstrate that the hyperfan outperforms competing methods including synthetic focus, fan-shaped antialiasing filters, and a range of modern nonlinear image and video denoising techniques. We show the hyperfan preserves depth of field, making it a single-step all-in-focus denoising filter suitable for general-purpose light field rendering. We include results for different noise types and levels, over a variety of metrics, and in real-world scenarios. Finally, we show that the hyperfan’s performance scales with aperture count.

A major issue in electron tomography is the misalignment of the projections contributing to the reconstruction. The current alignment techniques currently use fiducial markers such as gold particles. When the use of markers is not possible, the accurate alignment of the projections is a challenge. We describe a new method for the alignment of transmission electron microscopy (TEM) images series without the need of fiducial markers. The proposed approach is composed of two steps. The first step consists of an initial alignment process, which relies on the minimization of a cost function based on robust statistics measuring the similarity of a projection to its previous projections in the series. It reduces strong shifts resulting from the acquisition between successive projections. The second step aligns the projections finely. The issue is formalized as an inverse problem. The pre­ registered projections are used to initialize an iterative alignment-refinement process which alternates between (i) volume reconstructions and (ii) registrations of measured projections onto simulated projections computed from the volume reconstructed in (i). The accuracy of our method is very satisfying; we illustrate it on simulated data and real projections of different zeolite supports catalyst.

Model- based iterative reconstruction (MBIR) is increasingly widely applied as an improvement over conventional, deterministic methods of image reconstruction in X-ray CT. A primary advantage of MBIR is potentially dras­ tically reduced dosage without diagnostic quality loss. Early success of the method has naturally led to growing numbers of scans at very low dose, presenting data which does not match well the simple statistical models heretofore considered adequate. This paper addresses several issues arising in limiting cases which call for refine­ ment of standard data models. The emergence of electronic noise as a significant contributor to uncertainty, and bias of sinogram values in photon-starved measurements are demonstrated to be important modeling problems in this new environment. We present also possible ameliorations to several of these low-dosage estimation issues.

Scanning transmission electron tomography is often used to reveal the internal structure of material samples. In this technique, a tilt series of tomographic projections is reconstructed to obtain volumetric information. The reconstructed scene is then segmented into regions with homogeneous properties to localize and quantify various material elements. Unfortunately, physical constraints limit the extent of the projection tilt series, leading to artifacts in the reconstructed volume, which can make subsequent segmentation difficult. In this work we use a different, discrete tomography, approach wherein we directly reconstruct only a limited and discrete set of pixel amplitudes, effectively performing the reconstruction and segmentation in a joint fashion. Unlike existing methods, the approach is based on direct formulation of the problem in the discrete domain. Solution of the subsequent challenging optimization problem is achieved through the iterative use of graph-cut methods applied to a physically motivated surrogate cost function. We show reconstruction results using synthetic phantom images for limited angle scenarios and compare them to conventional reconstruction techniques.

Size and concentration of airborne particulate matter (PM) are important indicators of air pollution events and public health risks. It is therefore important to monitor size resolved PM concentrations in the ambient air. This task, however, is hindered by the highly dynamic spatiotemporal variations of the PM concentrations. Satellite remote sensing is a common approach for gathering spatiotemporal data regarding aerosol events but its current spatial resolution is limited to a large grid that does not fit high varying urban areas. Moreover, satellite-borne remote sensing has limited revisit periods and it measures along vertical atmospheric columns. Thus, linking satellite-borne aerosol products to ground PM measurements is extremely challenging. In the last two decades visibility analysis is used by the US Environmental Protection Agency (US-EPA) to obtain quantitative representation of air quality in rural areas by horizontal imaging. However, significantly fewer efforts have been given to utilize the acquired scene characteristics (color, contrast, etc.) for quantitative parametric modeling of PM concentrations. We suggest utilizing the image effective bandwidth, a quantitative measure of image characteristics, for predicting PM concentrations. For validating the suggested method, we have assembled a large dataset that consists of time series imaging as well as measurements from air quality monitoring stations located in the study area that report PM concentrations and meteorological data (wind direction and velocity, relative humidity, etc.). Quantitative and qualitative statistical evaluation of the suggested method shows that dynamic changes of PM concentrations can be inferred from the acquired images.

In this paper, we propose an efficient synthetic refocusing method from multiple coded aperture images for 3D user interaction. The proposed method is applied to a flat panel display with a sensor panel which forms lens-less multi-view cameras. To capture the scene in front of the display, the modified uniformly redundant arrays (MURA) patterns are displayed on the LCD screen without the backlight. Through the imaging patterns on the LCD screen, MURA coded images are captured in the sensor panel. Instead of decoding all coded images to synthetically generate a refocused image, the proposed method only decodes one coded image corresponding to the refocusing image at a certain distance after circularly shifting and averaging all coded images. Further, based on the proposed refocusing method, the depth of an object in front of the display is estimated by finding the most focused image for each pixel through a stack of the refocused images at different depth levels. Experimental results show that the proposed method captures an object in front of the display, generates refocused images at different depth levels, and accurately determines the depth of an object including real human hands near the display

A contrast enhancement algorithm is developed for enhancing the contrast of x-ray images. The algorithm is based on Laplacian pyramid image processing technique. The image is decomposed into three frequency sub-bands- low, medium, and high. Each sub-band contains different frequency information of the image. The detail structure of the image lies on the high frequency sub-band and the overall structure lies on the low frequency sub-band. Apparently it is difficult to extract detail structure from the high frequency sub-bands. Enhancement of the detail structures is necessary in order to find out the calcifications on the mammograms, cracks on any object such as fuel plate, etc. In our proposed method contrast enhancement is achieved from high and medium frequency sub-band images by decomposing the image based on multi-scale Laplacian pyramid and enhancing contrast by suitable image processing. Standard Deviation-based Modified Adaptive contrast enhancement (SDMACE) technique is applied to enhance the low-contrast information on the sub-bands without overshooting noise. An alpha-trimmed mean filter is used in SDMACE for sharpness enhancement. After modifying all sub-band images, the final image is derived from reconstruction of the sub-band images from lower resolution level to upper resolution level including the residual image. To demonstrate the effectiveness of the algorithm an x-ray of a fuel plate and two mammograms are analyzed. Subjective evaluation is performed to evaluate the effectiveness of the algorithm. The proposed algorithm is compared with the well-known contrast limited adaptive histogram equalization (CLAHE) algorithm. Experimental results prove that the proposed algorithm offers improved contrast of the x-ray images.