Computer vision relies heavily on images and movies collected from camera recordings. Because these are essentially measurements taken from the real world, they are intrinsically noisy. Often, one of the first processing steps for advanced applications is to reduce the level of noise in the images. Standard image denoising techniques usually operate on single (streams of) images and assume a uniform noise level across them. If multiple streams of mutually dependent images (i.e., they share edges and so forth) are available and local noise levels can be estimated, such information can be exploited to improve denoising quality. We extend three standard denoising methods to exploit the additional information available in timeofflight (TOF) camera data. One of these methods—extended anisotropic diffusion—can be implemented to operate in real time. It also improves TOF camera image quality, especially in highnoise regions, while still preserving edges and details in lownoise regions.
Timeofflight range imaging records intensity and distance (range) information for each pixel by measuring the reflectance level and travel time of light issued by a light source close to the camera and reflected by objects in the recorded scene. Measurements of intensity and range made using this acquisition scheme are affected by certain kinds of noise.^{1} In particular, very high levels of noise occur in rangeimage regions with low levels of reflected light^{2} (e.g., dark image patches) because less data is available for estimating range. By the same token, regions of high reflection exhibit noise levels orders of magnitude lower. This inhomogeneity makes several standard denoising approaches unsuitable.
Figure 1. Real laboratory scene recording. Top: Averaged range. Bottom: Extended anisotropic diffusion (EAD) denoised range. Despite good denoising performance of EAD, the edges are well preserved. Gray levels in the bottom image reflect range in meters. Full details are available elsewhere.
^{1}
We have described three alternatives that extend standard denoising approaches to make use of the additional information available in TOF imaging by virtue of two linked (intensity and range) images.^{1,3} The intensity image gives information about noise levels in rangeimage regions that can be used for local adaptation of the denoising parameters. The alternative methods are these: We adapt the threshold in wavelet thresholding to local noise levels. We perform image segmentation to determine image regions of similar noise characteristics and the appropriate denoising parameters. Or we modify the diffusion (i.e., smoothing) process of anisotropic (actually, PeronaMalik) diffusion filtering to take different noise levels into account, which we call extended anisotropic diffusion, or EAD.
Table 1.Selected performance results—rms error and its standard deviation with respect to true ranges for 50 trials—of denoising using standard averaging and the evaluated EAD, segmentation (clustering), and wavelet methods applied to synthetic data for different average (but locally intensitydependent) noise levels and scene characteristics. Significantly best performers for each scene type are highlighted in bold. SNR: Signaltonoise ratio.
Scene characteristics  Averaging  EAD  Clustering  Wavelet  SNR (dB) 
Very low noise 
0.15±0.020 
0.03±0.005 
0.09±0.039 
0.07±0.011 
11.69±2.336 
Low noise 
0.32±0.036 
0.05±0.009 
0.10±0.053 
0.14±0.022 
6.85±3.336 
Standard noise 
0.52±0.061 
0.12±0.033 
0.08±0.065 
0.22±0.037 
0.11±4.070 
Strong noise 
0.78±0.097 
0.18±0.037 
0.19±0.116 
0.33±0.047 
−1.28±3.508 
Very strong noise 
1.17±0.137 
0.28±0.041 
0.26±0.164 
0.46±0.098 
−4.66±3.219 
Strong bumps 
0.54±0.074 
0.12±0.037 
0.21±0.069 
0.24±0.045 
2.02±2.781 
Strong tilts 
0.54±0.070 
0.08±0.016 
0.39±0.159 
0.26±0.037 
4.67±2.455 

Each of these methods improves the denoising technique traditionally used on TOF image data—temporal averaging—given the same number of images for each method. Still, the most robust and effective variant is EAD, which allows realtime processing of images (at least with GPU—graphics processing unit —computing support) and qualitatively achieves the best and fairly robust results (see Table 1 and Figure 1).
Standard isotropic Gaussian smoothing can be formulated as differential equations.^{4} Here, ϕ is the range image, the smoothing strength is influenced by the length of time tof diffusion propagation, and ∇ is the operator that computes the vector of partial derivatives, ∂). Thus,
similar to the common formulation of anisotropic diffusion,
which introduces (an example for) the term D to inhibit diffusion between neighboring image pixels if they are located on an edge in the image (c sets the strength of diffusion inhibition).^{5} This formulation takes into account prior knowledge that strong edges are less likely to be caused by noise than are isolated peaks or weaker edges.
We extend anisotropic diffusion along these lines by incorporating further prior knowledge taken from the intensity image. Specifically, this is knowledge about the local noise level, derived from the reciprocal relationship between intensity and range noise level, and knowledge about edges in the intensity image that makes edges in the range image more probable. Color and material changes are often caused by their belonging to different objects in different ranges.
As a result, in EAD we extend D to be
The local (i.e., locationdependent) noise variance term in the numerator scales diffusion according to the local noise level, while the terms in the denominator inhibit diffusion at edges in the range image ϕ, edges in the intensity image I, and edges in the noise level image σ^{2}_{ϕ} derived from I or subsequent range images. To increase robustness, these images, and specifically σ^{2}_{ϕ}, can be smoothed before application of this term.
In summary, we have taken a number of standard image denoising approaches, extended them to take into account the special noise structure in TOF camera images, and evaluated the modified versions on synthetic and real image data. A comparison with common temporal denoising shows that, given the same amount of image data, all these methods are superior to averaging or to their nonextended denoising counterparts. The EAD method in particular proved to be very useful in terms of computation, as well as in denoising. We are continuing our research on TOF camera data with respect to scene analysis as well as further improving denoising for streamed image data.
The author is grateful for the support of the Austrian Funding Program COMET.
Holger Schöner
Software Competence Center Hagenberg (SCCH) GmbH
Hagenberg, Austria
Holger Schöner received an MS from University of Colorado at Boulder and from the Technical University of Berlin, where, in 2005, he also received his PhD in computer science. At SCCH he is working on and researching applied problems in data mining, machine learning, and image processing.
References:
1. H. Schöner, F. Bauer, A. Dorrington, B. Heise, V. Wieser, A. Payne, M. J. Cree, B. Moser, Image processing for threedimensional scans generated by timeofflight range cameras,
J. Electron. Imag. 21(2), p. 023012, 2012.
doi:10.1117/1.JEI.21.2.023012
2. M. Frank, M. Plaue, H. Rapp, U. Köthe, B. Jähne, F. A. Hamprecht, Theoretical and experimental error analysis of continuouswave timeofflight range cameras,
Opt. Eng. 48(1), p. 013602, 2009. doi:
10.1117/1.3070634
3. B. Moser, F. Bauer, P. Elbau, B. Heise, H. Schöner, Denoising techniques for raw 3D data of TOF cameras based on clustering and wavelets,
Proc. SPIE 6805, p. 68050E, 2008.
doi:10.1117/12.765541
4. G. Aubert, P. Kornprobst, Mathematical Problems in Image Processing 147, Springer, New York, 2006.
5. P. Perona, J. Malik, Scalespace and edge detection using anisotropic diffusion, IEEE Trans. Pattern Anal. Machine Intell. 12, p. 629639, 1990.