SPIE Membership Get updates from SPIE Newsroom
  • Newsroom Home
  • Astronomy
  • Biomedical Optics & Medical Imaging
  • Defense & Security
  • Electronic Imaging & Signal Processing
  • Illumination & Displays
  • Lasers & Sources
  • Micro/Nano Lithography
  • Nanotechnology
  • Optical Design & Engineering
  • Optoelectronics & Communications
  • Remote Sensing
  • Sensing & Measurement
  • Solar & Alternative Energy
  • Sign up for Newsroom E-Alerts
  • Information for:


Print PageEmail PageView PDF

Electronic Imaging & Signal Processing

Mobile receiver design increases the resolution of compressed video

A new concept in image enhancement has made possible a video receiver that allows significant reduction in transmitted bandwidth while keeping the quality of received images high.
1 August 2007, SPIE Newsroom. DOI: 10.1117/2.1200707.0821

When compressing a video sequence, modern techniques achieve high compression ratios by removing image details (high-frequency information), producing a degradation of the decoded image. This degradation is negligible for high bit rates but can result in unbearable quality where bit rates are very low and compression ratios must therefore be high. Sometimes the lost information can be recovered from frames that occur shortly before or after the one that is degraded. This is known as super-resolution (SR) but its high computational load usually prevents real-time implementation of the process. Moreover, such algorithms are designed in such a way that a feasible implementation would require a huge re-design effort.

Although several algorithms for SR have been proposed (a comprehensive classification is provided by Borman1), all lack a practicable real-time implementation: for those who know the field, this even includes the Segall2 and Gunturk3 compression environments. Here, we jointly consider the algorithm specification and the system design as a whole process, where the restrictions imposed by the platform are taken into account by the algorithm from the first design steps. Our aim is to contribute towards research in low-cost and high-performance multimedia devices. To this end, we propose a mobile system for real-time video reception over the Digital Audio Broadcast4 (DAB) network, using programmable logic devices (PLDs) with embedded microprocessors as the target technology. As the data rate of DAB is limited to 1.8432Mbs−1, the bandwidth of the encoded sequence must be drastically reduced, decreasing the quality due to compression losses.

Figure 1. Overview of a system to improve the quality of highly-compressed video images.

To increase the image quality for these very-low bit rates, we have developed a novel technique that we call extended super-resolution (XSR). Prior to MPEG-4 (Moving Picture Experts Group standard 4) coding, pre-processing is performed, composed by a segmentation phase according to motion and texture, and by a guided down-sampling phase. As a result, a set of texture and motion maps are obtained that will also be compressed and included into the bit-stream as user data side information. A system-level design and simulation tool called CASSE,5 has been used for the application and architectural modeling, as well as for the exploration of the best hardware/software partition of the MPEG-4 decoder.6

System overview

The system works with a typical ITU-R 601 (750×576 pixels) video signal, as can be seen in Figure 1. The sequence is pre-processed to get QCIF (Quarter Common Interface Format, designed for teleconferencing) frame size (176×144 pixels) and to extract some image characteristics that will keep important information for the XSR process. The down-sampled sequence is encoded using the MPEG-4 standard, delivering a reduced bandwidth bitstream. Next, a MPEG-2 transport stream (TS) is generated to be transmitted over IP (internet protocol) tunnelling through the DAB network. Alternately, and for testing purposes, the IP datagrams can be coded in the receiver data interface (RDI) and transmitted through an optical link (IEC-958). On reception, the user selects the channel identifier, which enables the data containing the MPEG-4 bitstream to be decoded. After decoding, an XSR process increases the resolution to CIF (352×288 pixels) size, where a LCD-display is used for visualization.

Figure 2. The Excalibur XA10 architecture and the system-task mapping.
Design space exploration

The receiver (Figure 2) is developed using the Integrator CM922T-XA10 development board, which comprises an Excalibur XA10 device and off-chip memory. The Excalibur device is structured as two parts, the stripe and the programmable logic device. The former is composed of an ARM9 embedded processor and some embedded programmable on-chip peripherals. The starting point was a functional model of the application, for which it was decomposed into concurrent tasks that communicate with each other using a Kahn process network (KPN). The mapping of this model comprises an allocation of all the tasks into the elements of the model for a specific architectural platform. The final architecture consists of two hardware co-processors defined for the front- and back-end tasks, (Vin and Vout), a hardware accelerator that executes the inverse discrete cosine transform (IDCT) task and several remaining tasks that are executed on the ARM9 processor.

Figure 3. A schematic of the extended super-resolution process.
Extended super resolution

The XSR algorithm can be divided into a pre-processing stage on the transmitter side and a post-processing stage on the receiver side (see Figure 3). The pre-processing consists of a block-based high-resolution (HR) image segmentation into three types of regions: motion and textured (MT), flat (F), and no motion but textured (T), followed by a smart down-sampling procedure. In order to combine information from multiple frames, we use principal component analysis (PCA). While the computed motion vectors are very accurate in areas with texture, they are usually more prone to error in flat areas: a situtation that can be substantially recovered using simple interpolation. In T regions, the down-sampling process is modified to attenuate the loss of high frequencies and the SR is applied to MT regions. Finally, the low-resolution (LR) image sequence is lossy compressed using an MPEG-4 encoder while the segmentation maps are losslessly compressed and included into the bit-stream as user data information. The SR algorithm used is based on non-uniform interpolation technique.7

We present the results for a CIF sequence call Students. Our model is compared with an alternative technique where the CIF sequence is interpolated instead of being super-resolved. Decoded video sequences are compared for bit rates ranging from 32Kbps to 4Mbps. Figure 4 shows the rate-distortion curves for the two choices. The graphic shows that the XSR algorithm outperforms the interpolated system for all bit rates. Figure 5 shows Frame 8 for both choices and Figure 6 shows some details of the same frame.

Figure 4. Rate-distortion characteristics of the Students sequence: compares the average quality and bit-rates between the interpolated and the super-resolved sequence after decompression.

Figure 5.(a) Frame 8 of the Students sequence interpolated and, (b) the same frame after the XSR process.

Figure 6. Detail of the tumbler in order to compare the perceptual quality between (a) the interpolation and (b) the proposed XSR method.

The proposed XSR technique, mapped onto a mobile video receptor using DAB technology for transmission, provides a final video sequence with a quality higher than that provided by simple image interpolation. In order to prove the feasibility of the proposed method, the decoder system has been implemented using a configurable system-on-chip platform, following a tailored hardware/software design methodology. Experimental results validate the utility of the proposed system, which achieves a throughput of 15 QCIF frames per second. However, we expect that enhancing the methodology and optimizing the hardware coprocessor will allow a significant speed up of the system.

This work has been supported by the Spanish Ministry of Education and Sciences under the ARTEMI+ project TEC2006-13599-C02- 02/MIC, and also with pre-doctoral grants AP2003-4310 and BES- 2004-4186.

Gustavo M. Callicó
Electronic Engineering and Control
University of Las Palmas de Gran Canaria (ULPGC)
Las Palmas de Gran Canaria, Spain
Integrated Circuits Division
Research Institute for Applied Microelectronics
Las Palmas de Gran Canaria, Spain 

Gustavo M. Callicó received his degree in telecommunications engineering in 1995 and his PhD in 2003: both from the ULPGC. He has been an assistant professor there since 1997. He joined the Research Institute for Applied Microelectronics (IUMA) in 1994.

Antonio Nunez
Electronic Engineering and Control
University of Las Palmas de Gran Canaria
Las Palmas de Gran Canaria, Spain
Microelectronics Technology
Research Intitute for Applied Microelectronics
Las Palmas de Gran Canaria, Spain

Prof. Antonio Nunez received his higher engineering degree in 1974 from the School of Telecommunication Engineering, and his the PhD degree in 1981, both from the Technical University of Madrid. In 1981 he was a research scientist with the Electrical Engineering Department of EPFL Lausanne, Switzerland, and both a visiting scientist (1986-1987) and visiting professor (1987-1988) at the School of Electrical Engineering at Purdue University, IA.