Super-resolution

Viewing low-resolution images limits many tasks. For example, one might have difficulty identifying, resolving, or reading street signs, notices in front of a building, or features, actions, or objects carried by someone in an image. Image quality is limited by a sensor's native resolution or other factors caused by undersampling images spatially or electronically using low-cost sensors. Consequently, subtle, detailed information (i.e., high-frequency components) is lost.

Many approaches have been developed to increase image resolution, one of which uses ‘microdither’^1,2: a special sensor or scanner captures multiple images, where each image is captured by displacing the sensor a known fraction of a pixel (subpixel). The individual frames are then combined into a high-resolution image. An alternative approach employs ‘super-resolution image reconstruction’ (SRIR).^1,2 This software technology produces high-resolution images using low-cost, low-resolution video equipment. On the basis of a very small number of digital input frames, resolution can be doubled or even quadrupled. Image quality can thus be improved using existing sensors without the need for hardware upgrades.

SRIR explores the subpixel movement of an image sequence acquired using an imager affected by natural jitter (such as a handheld camera), for which the subpixel motion characteristics are unknown. When low-resolution images have subpixel shifts between successive frames, they convey different information about the same scene. Therefore, the information contained in a low-resolution image sequence can be combined to obtain a high-resolution (alias-free) picture. SRIR on the basis of multiple snapshots provides far more detailed information than any interpolated image from a single snapshot. Dealing with natural jitter is more cost effective and practical than using microdither scanners. For example, an imager may often be mounted on a moving platform. In a rescue mission, the camera may be carried by a helicopter or a moving vehicle. In a military-reconnaissance situation, the camera may be carried by a person, or unmanned ground or aerial vehicles. In environments such as law enforcement and security cameras might be stationary. However, when the scene changes between frames (due to, e.g., a passing person or vehicle), subpixel shifts could be generated by the motion of either the entire scene or part of the scene.

The US Army Research Laboratory has conducted extensive research into SRIR technology³ for several years. This has resulted in the development of a practical and accurate algorithm that has been tested for several useful types of imaging products, including those produced by color video, forward-look IR, and laser detection and ranging (LADAR)⁴ cameras with various sensor configurations. The algorithm effectively increases the bandwidth (resolution) of the reconstructed output image using the existing low-cost imaging device based on a sequence of low-resolution (undersampled) imagery. It recovers the lost high-frequency information by estimating the subpixel shifts between the successive images and subsequently combining information from the individual images. The subpixel shifts are unknown because of uncontrolled natural camera jitter.

Figure 1. Thermal images before and after super-resolution processing. If anything were hiding behind the trees, the super-resolved image (right) would reveal it clearly.

One of the difficulties faced in developing super-resolution algorithms is the number of undersampled frames required to improve the final image. Many existing algorithms use more than 100 frames. However, our technique uses a very small number of input images, based on information theory, and exploits both the frequency and spatial domains to obtain an alias-free, high-resolution result. The algorithm produces images with 2–4 times higher resolution using 4–16 frames.

Using a relatively small number of frames acquired at subpixel shifts through natural jitter of an imaging device mounted on a moving platform (instead of by continuous camera motion, as often also required by the existing algorithms) offers a flexibility that is crucial in the operation of innovative practical systems, e.g., the imaging system available for situational awareness, requiring low memory and fast processing. Using a small number of frames also provides a realistic solution to producing high-resolution images of a scene containing one or more fast-moving targets.

Figure 2. (Top) Grayscale and (bottom) color-coded flash LADAR imagery for (left) low-resolution and (right) super-resolved images of a triangular target at 5m. The orientation of the triangular hole is difficult to discern in the low-resolution image as the hole resembles a blurred circle. By contrast, the orientation is clear in the super-resolved image. The color scale at the bottom right is a distance scale (expressed in meters).

In conclusion, we have developed a practical and accurate SRIR algorithm and tested various sensor configurations. While initially developed for military applications, the possible software applications range from home video to law enforcement, security, fingerprint recognition, medical imaging, and industrial applications, where high-resolution images are desirable from low-resolution cameras. SRIR can be applied to virtually any undersampled digital-imaging device (most low-resolution cameras are undersampled), such as handheld video or dashboard-mounted cameras in patrol cars, or security cameras that swivel. The technique could be used with in-camera firmware or as desktop video- or photo-editing software. Examples of thermal and LADAR images before and after processing are shown in Figures 1 and 2, respectively. We will extend our research to applications of SRIR to 3D lightweight flash LADAR sensors that will benefit both autonomous and semi-autonomous robot navigation.