Standard-definition content often looks blurred when viewed on a high-definition television set. The reason for this loss in quality is often the resizing engine, which increases the content's spatial dimensions to fit onto the screen. Typically, these engines treat video as a sequence of independent images or frames and employ basic resizing methods such as nearest-neighbor, linear, or cubic-spline interpolation for each frame. Because these approaches essentially apply the same method at every pixel, regardless of the image structures around it, the resulting resized video can have blurred features and detail that appears wiped out. Recently, image and video resizing have benefited from advances in our understanding of visual perception and an increase in computing power.
It is well known that strong edges and gross features play an important role in perception of image quality.1 Based on this, several single image-resizing methods such as new-edge-directed2 and iterative curve-based interpolation3 emphasize faithful reproduction of strong edges. However, retaining edge sharpness comes at the expense of overblurring fine details. Similarly, algorithms that preserve detail produce a halo effect around the edges.3
Our research has focused on designing image- and video-resizing methods for practical systems that preserve the integrity of edges and produce perceptually pleasing results. We decompose an input image into multiple layers,4 such as a base and residue layer. The former contains the gross features such as strong edges and the latter captures finer details. Each one can be resized using appropriate methods and then blended to give the final, larger image. For instance, we can design algorithms for the base layer that retain the integrity of the strong edges to an arbitrary degree. Resizing the detail layer and combining it with the base layer can produce an image that is perceptually superior to ones using several existing schemes. Figure 1 compares the results of our proposed algorithm to cubic interpolation for a magnification factor of four.
Figure 1. (a) Original picture. Results of resizing by 4×using (b) cubic-spline interpolation and (c) our proposed algorithm.
We have also extended the idea of layer-based resizing to video. Our current research is focused on scaling video using periodic high-resolution (HR) stills. This option exists in the emerging hybrid-camera market where devices can capture high- or standard-definition video at medium frame rates of 15–30 frames per second (fps), while simultaneously capturing very-high-resolution still pictures (at 12 megapixels and higher) at lower frequencies (1–5fps).
(a) Original video. After resizing by a magnification factor of two with (b) the example-based method and (c) our approach using HR stills. (Frame 18 of the source video5
(a) Original video. After enlarging 2× using (b) the example-based method and (c) our wavelet-based algorithm for video resizing using HR stills. (Frame 77 of the source video5
The technology used in hybrid cameras may soon be found in mobile-phone cameras, where constraints in form factor, battery life, and processing power only allow video capture at much lower spatial resolution and HR stills. Video-resizing methods that use HR stills as guides would therefore be very useful for producing HR video or enlarging particular frames.
Current approaches use superresolution methods6 such as optical flow-based framework,7 example-based,8 and robust correspondence-based methods.9 Most of these algorithms have high computational complexity and may be unsuitable for real-time and embedded systems such as camera phones.
We are researching approaches for resizing video using HR stills. We implicitly use a layered approach to fill in details using HR stills. Figure 2 compares our method with the current state-of-the-art algorithms. We are also developing a wavelet-based, computationally inexpensive technique geared towards embedded scenarios, with a marginal penalty in image quality (see Figure 3).
We have focused on developing technologies that resize images and video. We envision using this resizing approach in applications like mobile phones, where images captured at low resolution (e.g., 1MB) would appear as if they were taken at much higher resolution (e.g., 5MB). These methods could also be used in the digital-zoom features employed in low-end camera phones. Resizing and quality enhancement of compressed images and video (with and without HR guides) are important directions for further research.
I thank my team, S. V. Basavaraj, Amit Prabhudesai, Sudha Velusamy, and Radhika Raghavendran, as well as Asha Vijaykumar for providing me with the figures for this article and for their comments. Comments from Chandra Sekhar Reddy, Karthik Balasubramaniam, and Anshul Sharma have been very helpful.
Ajit S. Bopardikar
Samsung Advanced Institute of Technology India Laboratories
Samsung India Software Operations
Ajit Bopardikar earned his PhD from the Center for Imaging Science, Rochester Institute of Technology, in 2000. He has worked extensively on signal and image processing and has published widely in these areas. He is the co-author (with Raghuveer Rao) of a textbook on wavelet transforms. He is currently a chief engineer focusing on superresolution and image resizing.