In military scenarios it is of crucial importance to observe over long distances, or in other words, to have a large recognition range. Visible light and IR cameras are those mainly used for this task. However, the recognition range—especially for ‘land-to-land’ observation (i.e., along horizontal paths at ground/sea level)—is hampered by the optical effects of atmospheric turbulence, which are caused by local variations in the refractive index of air. These in turn are caused by temperature and humidity variations along the observation path. In images, atmospheric turbulence is observed as locally varying blur and beam wander. If we could reduce these effects the recognition range would be extended significantly.
Many hardware and software solutions have been developed in the past. Some hardware solutions use adaptive optics1 and are used for astronomical applications. However, these are expensive and have only one mode, that of performing the same processing on the whole image. Software solutions are much cheaper and can be used in combination with existing cameras. However, a lot of these software methods2, 3 also use a global approach (i.e., applying a uniform process across the whole image), whereas for land-to-land observation the effects of atmospheric turbulence are local. We propose a software solution that performs local processing to reduce these effects.
Figure 1 shows a flowchart of our method, which can be divided into three parts: first, frame selection to select the sharpest frames; second, local motion compensation to stabilize the observed scene; and third, multi-frame deblurring to increase the resolution and remove the blur in each frame. This processing results in a more stable image, which is more comfortable and less tiresome for the operator to watch. In addition, more details will be visible, because the resolution is enhanced and the image is deblurred. Our solution is flexible and can be implemented in real time.
Figure 1. Flowchart of our turbulence compensation method.
Figure 2. Left: Region-of-interest of deteriorated input frame. Right: Result after processing with our method.
The frame-selection part is based on the Gaussian gradient magnitude (GGM), which is used as a sharpness measure. Frames with a large GGM are selected for further processing, while other, more blurry frames are neglected. To stabilize the scene, first a reference frame is calculated by averaging a fixed number of frames over time. Optical flow—a well-known image registration method—is used to estimate the local motion in each frame relative to the reference frame. Next, we simultaneously apply resolution enhancement and deblurring, which are deduced with super-resolution (SR) reconstruction methods.4 More details about our method can be found elsewhere.5
To test the performance of our method we processed real-life images captured in South Africa during the False Bay Atmospheric Experiment (FATMOSE). These images were captured with a visible-light monochromatic camera with a resolution of 640 × 480 pixels over a distance of 15.7km looking at a beach station. We processed a region-of-interest (ROI) of 150 frames in which the front of the beach station is visible. In Figure 2, a ROI of one of the original frames is shown together with a processed frame. After processing the frames with our method, the Sea Rescue sign of the beach station is much easier to read. Further results, using images captured in Dayton, OH, are shown in a short video that is available online.6
In summary, software methods seem a promising way of increasing the recognition range for military observers. We have developed a software method that can perform this task in an efficient way. It is flexible, robust, fast (can be implemented in real-time), and produces state-of-the-art results. We are now adapting our method for imagery containing moving objects.
The authors would like to thank FATMOSE trial colleagues, the NATO SET-165 group for capturing the Dayton dataset, and the Netherlands Ministry of Defense for funding this research.
Adam van Eekeren, Klamer Schutte, Piet Schwering
Netherlands Organization for Applied Scientific Research (TNO)
The Hague, The Netherlands
Adam van Eekeren received his PhD from Delft University, the Netherlands. Currently he is a research scientist at the Intelligent Imaging Department at TNO. His research interests include image restoration, image quality assessment, and object detection.
Klamer Schutte is a senior research scientist in the Intelligent Imaging Department, working at TNO since 1996. His current research includes image enhancement, scene analysis, and image-based behavior analysis.
Piet Schwering received his PhD from Leiden University, the Netherlands. He is currently a senior research scientist in the Electronic Defense Department at TNO and has worked in infrared surveillance, long-range imaging, infrared search and track, system concepts and data fusion, and detection and classification for 25 years.
1. Adaptive Optics in Astronomy, Cambridge University Press, 1999.
2. P. B. W. Schwering, S. P. van den Broek, M. van Iersel, EO system concepts in the littoral, Proc. SPIE
6542, p. 654230, 2007. doi:10.1117/12.719217
3. M. van Iersel, A. M. J. van Eijk, Estimating turbulence in images, Proc. SPIE
7814, p. 78140Q, 2010. doi:10.1117/12.861162
4. R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, E. A. Watson, High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system, Opt. Eng.
37(1), p. 247-260, 1998. doi:10.1117/1.601623
5. A. W. M. van Eekeren, M. C. Kruithof, K. Schutte, J. Dijk, M. van Iersel, P. B. W. Schwering, Patch-based local turbulence compensation in anisoplanatic conditions, Proc. SPIE 8355, p. 83550T, 2012. (In press.)