Imaging systems are traditionally developed using structured analysis and design techniques at best. Such approaches tend to be rigid with respect to changing needs, technologies, devices and algorithms-for example, when additional compression algorithms are needed or attached devices are changed large parts of software applications employing those techniques and interfacing with those devices must be modified to accommodate the change. In a larger perspective, these systems are difficult or impossible to reuse; each new problem requires a new solution. This is generally undesirable and often not necessary, but only if best practices in software engineering are employed. These best practices have been explored and documented in detail with regard to object-oriented systems, which suggests that it is an appropriate paradigm to employ in the development of future imaging systems. This work examines these best practices, in the form of patterns and design principles, with reference to imaging systems.

This paper discusses the real-time implementation of a fast and accurate auto-focus method on the Texas Instruments DM270, a programmable processor designed specifically for digital still cameras. The DM270's programmable auto-focus hardware filter is utilized to obtain a sharpness function from a captured image. This function is then used to drive a rule-based search algorithm, which varies the focusing step size depending on the slope of the sharpness function. This leads to faster focusing speeds as compared to the standard global search algorithm. A wide variety of filters are tested by examining their performances in terms of focusing accuracy. The results show that the filters approximating the first derivative operator generate the best focusing accuracy under various focusing conditions.

Antialiasing is still a challenge in real-time computer graphics. In this paper we present a real-time selective antialiasing solution that builds upon our experience. We investigated existing approaches to real-time antialiasing and finally found a new simpler solution. Our new idea is to use the z-buffer directly for extracting visible edges information. The solution presented here can be summarized as follows: 1) select objects edges by applying on the z-buffer spatial convolution with Laplacian; 2) filter out aliasing artifacts by applying lowpass spatial convolution filtering to selected pixels. In this approach the same circuit architecture can be used for selection and the antialiasing of selected pixels. The major advantage of using spatial convolution in the context of antialiasing is hat general purpose hardware real-time convolution filters are well known and available. Method presented here can be used to improve image quality in graphics accelerators but also applications such as real-time ray tracing.

Since the shape of a 3D object moving in 3D space changes a lot in 2D image due to translation and rotation, it is very difficult to track the object using the SSD algorithm which finds the matching object in the input image using the template of the moving object. To solve the problem, this paper presents an enhanced SSD algorithm which updates the template based on an extended snake algorithm adaptive to the shape variation. The proposed snake algorithm uses the derivative of the area as the constraint energy to determine the boundary of an interested area considering the progressive variation of the shape. The performance of the proposed algorithm has been proved by the experiments where a mobile robot with one camera tracks a 3D target object translating and rotating arbitrarily in the 3D workspace.

Color image and video sequence restoration and improvement are complicated due to presence of various kinds of random noise. Impulsive noise is introduced by acquisition or broadcasting errors into communication channels. Non linear filters can provide good performance in terms of the signal-to-noise ratio in different levels of corruption as soon as minimum error chromaticity and minimum perceptual error. This paper presents the capability and real-time processing features of several processing techniques such as “directional processing”, “non parametric approaches” and “order statistics” filters. Some of such the filters were: Median Filter (MF), Vector Median Filter (VMF), -Trimmed Mean Filter (ATMF), Generalized Vector Directional Filter (GVDF), Adaptive Multichannel Non Parametric Filter (AMNF), Median M-type K-Nearest Neighbour (MM-KNN) filter, Wilcoxon M-type K-Nearest Neighbour (WM-KNN) filter, Ansari-Bradley-Siegel-Tukey M-Type K-Nearest Neighbor (ABSTM-KNN) filter, etc. Extensive simulations in reference color RGB images “Lena”, “Mandrill”, “Peppers” and QCIF format video sequences (Miss America, Flowers and Foreman) have demonstrated that the proposed filters consistently can outperform the known nonlinear filters. The used performance criteria in color imaging were the traditional ones: PSNR, MAE and other specific for color imaging, NCD and MCRE. The real-time implementation of image filtering was realized on the DSP TMS320C6701. The processing time of proposed filters includes the duration of data acquisition, processing and store data. We simulated impulse corrupted color image QCIF sequences to demonstrate that some of the proposed and analyzing filters potentially could provide on line processing to quality video transmission of the images.

Recently, the novel approach to Hough Transform (HT) calculation, regarded to as the Hough-Green Transform (HGT), was proposed. This approach is based on tracing object contours on bitonal imagery; therefore it is usually much more computationally effective than the Standard HT (SHT) on objects with a large number of interior pixels, as the acceleration is proportional to area by circumference ratio. Although the purely software sequential HGT implementation is very effective compared to the SHT and the Radon transform based alternatives, the algorithm allows designing a parallel and scalable real-time architecture.

Video tracking systems are inherently provided with non-rigid objects with various shapes and sizes, which often result in poor match of an initial model with the actual input shape. The hierarchical approach to the active shape model(ASM) is essential for video tracking systems to deal with such unstable inputs. The contribution of this paper is to improve the performance of active shape model-based real-time object tracking. We also propose a hierarchical processing framework to reduce the computational overhead. In this paper, we present a Kalman filter-based hierarchical ASM in order to estimate a dynamic shape in video object tracking. The experimental results show that the proposed hierarchical active shape model using Kalman filter is efficient. The experimental results also show good results when objects are partially occluded by other objects.

This paper proposes a real-time digital auto-focusing algorithm using a priori estimated set of point spread functions (PSFs). A priori set of PSFs are estimated by establishing the relation between two-dimensional PSF and onedimensional step response whose elements are samples of profile of degraded step edge. From the priori estimated set, the proposed auto-focusing algorithm can select the optimal PSF by the focusing criterion based on the frequency domain analysis. We then use the constrained least square (CLS) filter to obtain the in-focused image with the estimated optimal PSF. The proposed algorithm can be implemented in real-time because the set of PSFs are already estimated and the filtering is performed in the frequency domain.

This paper proposes a new approach of fusing skin color and motion information to detect face areas in moving pictures. For reducing the computation time and also the effect of lighting conditions, the block based analysis technique is used in the fusion process. Motion is detected from a difference image segmented into blocks having 4x4 pixels, and a block is classified as a MB (motion block) if the sum of pixel values in the blocks is larger than a specific threshold. Similarly, each block is marked as a FB (face block) if it contains more than one skin color pixels. Then the candidates for face regions (CF) are selected by fusing MB and FB images using the logical AND operation. Among CFs, the face regions are selected by using the heuristic rules on human faces such as the ratio of width and height, the high frequency portion in a face region, etc. In the final step, the selected CFs are tested by the facial feature scoring scheme which selects the CF having a score larger than a threshold as a face. The experimental results have shown that the proposed algorithm determines face regions in 0.08 sec with the correct decision rate of 91.7% and false detection rate 4.26%.

This paper presents a real-time image acquisition and segmentation system. The system involves three main processes: acquisition, filtering and segmentation. Image acquisition is performed in the steel industry, where thermographic linear images are captured from strips (10 Km long and 1 meter wide) at a temperature between 100°C and 200°C while they are moving forward along a track. During the acquisition process a relationship between each pixel in the linear image and real-world units is established using a theoretical model whose parameters have been adjusted after a calibration process. After the acquisition, linear images are spatially filtered to reduce noise, and online with these processes (acquisition and filtering), segmentation is applied to the linear images to divide them into homogeneous temperature zones. Two different segmentation methods are evaluated: region merging and edge detection. To compare the segmentation algorithms an empiric segmentation evaluation method is defined. The segmentation evaluation method lies in comparing the results obtained from the algorithm with the theoretical segmentation defined by a group of experts. The evaluation method will determine the best segmentation algorithm, the optimal parameter and the effectiveness obtained using a test set

This paper describes a novel approach to the real-time visualization of 3D imagery obtained from a 3D millimeter wave radar. The radar system uses two scanning beams to provide all weather 3D distance measurements of objects appearing on the ground. This information is displayed using our high-end 3D visualization engine capable of delivering models of up to 100,000 polygons with 30 frames per second. The resulting 3D models can then be viewed from any angle and subsequently processed to integrate match them against 3D model data stored in a synthetic database. The resulting Synthetic Radar Vision System will provide a truly novel way to obtained all weather 3D images. The paper will focus on the real-time imaging and display aspects of our solution, and will discuss technical details of the radar design itself. Engineering challenges will be outlined in the context of a practical application.

Numerous researchers have proposed the use of robotic aerial explorers to perform scientific investigation of planetary bodies in our solar system. One of the essential tasks for any aerial explorer is to be able to perform scientifically valuable imaging surveys. The focus of this paper is to discuss the challenges implicit in, and recent observations related to, acquiring mission-representative imaging data from a small fixed-wing UAV, acting as a surrogate planetary aerial explorer. This question of successfully performing aerial explorer surveys is also tied to other topics of technical investigation, including the development of unique bio-inspired technologies. Imaging results from two seasons of flights at Haughton Crater, Devon Island, Canada, a well documented Mars Analog site, are presented.

Minimally-invasive interventions are an important domain of medical real-time imaging modalities. Image processing algorithms that enhance interventional images run within hard real-time and latency constraints due to the required hand-eye coordination of physicians which perform the intervention. To support research activities, we present a flexible software architecture that allows to transfer image enhancement algorithms from research to clinical validation. The software architecture especially pays regard to multimodality interventional scenarios where an intervention runs in close succession to the acquisition of diagnostic data. Including the additional information of such diagnostic acquisitions enables content-based image enhancement. The proposed software architecture administers threads for a graphical user interface, data acquisition, offline preparation of diagnostic data, and the context-based real-time enhancement itself. Using this architecture, it is possible to run arbitrary complex content-based image analysis in real-time with only 9% computational overhead during the latency introducing algorithm run time. The proposed architecture is exemplified with an application for navigation support in cardiac CathLab interventions where diagnostic exposure acquisitions and interventional fluoroscopy can alternate in close succession.

In this article we present a generic, flexible and robust approach for an intelligent real-time video-surveillance system. A previous version of the system was presented in [1]. The goal of these advanced tools is to provide help to operators by detecting events of interest in visual scenes and highlighting alarms and compute statistics. The proposed system is a multi-camera platform able to handle different standards of video inputs (composite, IP, IEEE1394 ) and which can basically compress (MPEG4), store and display them. This platform also integrates advanced video analysis tools, such as motion detection, segmentation, tracking and interpretation. The design of the architecture is optimised to playback, display, and process video flows in an efficient way for video-surveillance application. The implementation is distributed on a scalable computer cluster based on Linux and IP network. It relies on POSIX threads for multitasking scheduling. Data flows are transmitted between the different modules using multicast technology and under control of a TCP-based command network (e.g. for bandwidth occupation control). We report here some results and we show the potential use of such a flexible system in third generation video surveillance system. We illustrate the interest of the system in a real case study, which is the indoor surveillance.

To clearly identify the face given in a surveillance image, this paper proposes a new method that magnifies the face image large enough and brings the magnified face image in focus. For this purpose, the compound lens system consisted of the zooming and focusing lenses is analysed to derive the relationship between the positions of lenses and the image size. Once the face size in the surveillance image and the target face size to achieve are given, the positions of the lenses are determined by the derived relationship. To adjust the positions of the lenses to obtain the focused image, the four point measurement algorithm is proposed. It calculates the focus measures of at most 4 positions and estimates the position having the maximum focus measure. The algorithm has been implemented on the camera system whose lenses are controlled by fast motors. The experimental results have shown that the magnified and focused image can be obtained in 0.77 seconds on average.

In this paper, a new real-time and intelligent face tracking system basing on the pan/tilt controlled stereo camera is proposed. In the proposed system moving human face is firstly detected from a sequence of the stereo image captured by the stereo camera system using a threshold value of YCbCr color model and its distance and area coordinates are measured and then, finally the stereo camera is controlled by using the pan/tilt system for tracking the moving face in real-time. Especially, color information of the face and geometric coordinate information of the stereo camera in which the pan/tilt system is embedded are used for extraction of 3D information of the human's face. From some experiments using a sequence of 1,280 frames of stereo input image, it is analyzed that the horizontal and vertical displacement on the center position of the face after tracking are kept to be very low values of 0.6% for 1,280 frames on average. Also, the error ratio between the measured and computed values is kept to be very low value of 0.5% on average in cartesian coordinates. This good experimental result finally suggests a possibility of implementing a new face tracking system having a high degree of accuracy and a very fast response time on the target by using the proposed algorithm.

The MPEG experts are currently developing the MPEG-21 set of standards and this includes a framework and specifications for digital rights management (DRM), delivery of quality of services (QoS) over heterogeneous networks and terminals, packaging of multimedia content and other things essential for the infrastructural aspects of multimedia content distribution. Considerable research effort is being applied to these new developments and the capabilities of MPEG-21 technologies to address specific application areas are being investigated. One such application area is broadcasting, in particular the development of digital TV and its services. In more practical terms, digital TV addresses networking, events, channels, services, programs, signaling, encoding, bandwidth, conditional access, subscription, advertisements and interactivity. MPEG-21 provides an excellent framework of standards to be applied in digital TV applications. Within the scope of this research work we describe a new model based on MPEG-21 and its relevance to digital TV: the digital broadcast item model (DBIM). The goal of the DBIM is to elaborate the potential of MPEG-21 for digital TV applications. Within this paper we focus on a general description of the DBIM, quality of service (QoS) management and metadata filtering, digital rights management and also present use-cases and scenarios where the DBIM’s role is explored in detail.

Block-based motion estimation is widely used in the field of video compression due to its feature of high processing speed and competitive compression efficiency. In the chain of compression operations, however, motion estimation still remains to be the most time-consuming process. As a result, any improvement in fast motion estimation will enable practical applications of MPEG techniques more efficient and more sustainable in terms of both processing speed and computing cost. To meet the requirements of real-time compression of videos and image sequences, such as video conferencing, remote video surveillance and video phones etc., we propose a new search algorithm and achieve fast motion estimation for MPEG compression standards based on existing algorithm developments. To evaluate the proposed algorithm, we adopted MPEG-4 and the prediction line search algorithm as the benchmarks to design the experiments. Their performances are measured by: (i) reconstructed video quality; (ii) processing time. The results reveal that the proposed algorithm provides a competitive alternative to the existing prediction line search algorithm. In comparison with MPEG-4, the proposed algorithm illustrates significant advantages in terms of processing speed and video quality.

Motion estimation and compensation is the key to high quality video coding. Block matching motion estimation is used in most video codecs, including MPEG-2, MPEG-4, H.263 and H.26L. Motion estimation is also a key component in the digital restoration of archived video and for post-production and special effects in the movie industry. Sub-pixel accurate motion vectors can improve the quality of the vector field and lead to more efficient video coding. However sub-pixel accuracy requires interpolation of the image data. Image interpolation is a key requirement of many image processing algorithms. Often interpolation can be a bottleneck in these applications, especially in motion estimation due to the large number pixels involved. In this paper we propose using commodity computer graphics hardware for fast image interpolation. We use the full search block matching algorithm to illustrate the problems and limitations of using graphics hardware in this way.

We propose a real-time object segmentation method for MPEG encoded video. Computational superiority is the main advantage of compressed domain processing. We exploit the macro-block structure of the encoded video to decrease the spatial resolution of the processed data, which exponentially reduces the computational load. Further reduction is achieved by temporal grouping of the intra-coded and estimated frames into a single feature layer. In addition to computational advantage, compressed-domain video possesses important features attractive for object analysis. Texture characteristics are provided by the DCT coefficients. Motion information is readily available without incurring cost of estimating a motion field. To achieve segmentation, the DCT coefficients for I-frames and block motion vectors for P-frames are combined and a frequency-temporal data structure is constructed. Starting from the blocks where the AC-coefficient energy and local inter-block DC-coefficient variance is small, the homogeneous volumes are enlarged by evaluating the distance of candidate vectors to the volume characteristics. Affine motion models are fit to volumes. Finally, a hierarchical clustering stage iteratively merges the most similar parts to generate an object partition tree as an output.

Nonlinear spatial transforms and fuzzy pattern classification with unimodal potential functions are already established in signal processing. They have proved to be excellent tools in feature extraction and classification. We propose an inspection method for pattern recognition and classification of two dimensional translation variant security elements such as stripes, kinegrams and others, which are widely used as applications in bank note printing. The system is based on discrete non linear translation invariant circular transforms and fuzzy pattern classification. Nonlinear discrete circular transforms are adaptable transforms, which can be optimized for different application tasks, such as translation variant object analysis and position location. They are mainly used as generators for feature vectors. Even though, the feature vector is theoretically translation invariant, the object movement creates a translation tolerant feature vector, because in real systems and applications many problems can occur, such as signal and optical distortions. Therefore, the features should be further analysed by a fuzzy pattern classifier. Implementation of the transforms and fuzzy pattern classifier in radix-2-structures is possible, allowing fast calculations with a computational complexity of O(N) up to O(Nld(N)). Furthermore, the algorithms can be implemented in one Field Programmable Gate Array (FPGA), which operates with 40 MHz clock rate.

Real-time image registration is potentially an enabling technology for the effective and efficient use of many image-guided diagnostic and treatment procedures relying on multimodality image fusion or serial image comparison. Mutual information is currently the best known image similarity measure for multimodality image registration. Mutual information calculation is a memory-intensive task that does not benefit from cache-based memory architecture in standard software implementations (i.e., the calculation incurs a large number of cache misses). Previous attempts to perform image registration in real time focused on parallel supercomputer implementations, which achieved real-time performance using large, expensive supercomputers that are simply impractical for deployment in a hospital. We have developed a custom hardware architecture that, in a single-module PC-based implementation, achieves registration speeds comparable to those of a 64-processor parallel supercomputer. The single-module speedup results from using parallel memory access and parallel calculation pipelines. The total speedup can be increased by using several modules in parallel. The architecture is designed for linear, mutual information-based registration and can be extended to elastic (nonlinear) registration.

In this paper, we consider a natural paradigm for lifting of crisp-set binary filters to fuzzy filters for hardware implementation and process the gray-scale realizations of binary images as [0,1]-valued fuzzy binary images. We present the implementation of the filtering algorithms for smoothing, peak detection and edge detection of such fuzzy images using the Xilinx Virtex series of FPGA for real-time processing of image sequences. The erosion filter forms the core for all of the filtering algorithms and the dilation filter itself is implemented as a function of the erosion filter. Smoothing is achieved using fuzzy opening of the input image using the user defined fuzzy structuring element. A fuzzy top-hat transform is used for peak detection. As opposed to gray-scale top-hat, which detects only the narrow peaks, the fuzzy top-hat is shown to detect both the narrow as well as wide peaks within the same image. Edge detection algorithm uses the fuzzy morphological gradient wherein the set minus operation has been performed between the dilated and the eroded images. Pipelined architectures are used for the erosion filter design and the use of flops has been maximized to achieve a high clock rate. The throughput measurements and the results generated by the implemented filters are also presented.

Image compression is a computationally intensive task, which can be undertaken most efficiently by dedicated hardware. If a portable device is to carry out real-time compression on a variety of image types, then it may be useful to reconfigure the circuitry dynamically. Using commercial off-the shelf (COTS) chips, reconfiguration is usually implemented by a complete re-load from memory, but it is also possible to perform a partial reconfiguration. This work studies the use of programmable hardware devices to implement the lossless JPEG compression algorithm in real-time on a stream of independent image frames. The data rate is faster than can be compressed serially in hardware by a single processor, so the operation is split amongst several processors. These are implemented as programmable circuits, together with necessary buffering of input and output data. The timing of input and output, bearing in mind the different, and context-dependent amounts of data due to Huffman coding, is analyzed using storage-timing graphs. Because there may be differing parameters from one frame to the next, several different configurations are prepared and stored, ready to load as required. The scheduling of these reconfigurations, and the distribution/recombination of data streams is studied, giving an analysis of the real-time performance.

In this article we present the segmentation chain based on contour detection including contour detection followed by thinning and crest restoration of contours. The thinning and crest restoration algorithms are based on topological notions. These algorithms allow us to enhance the segmentation process. This article presents the study to implement edge detection, thinning and crest restoration algorithms on a XCV 300 Virtex FPGA based architecture. The proposed architecture for the thinning and crest restoration operators is based on an optimized architecture for the topological classifier of the characteristics of the point.

This paper presents an efficient VLSI architecture and a low complexity implementation of BinDCT coprocessor for wireless video application. The coprocessor architecture was implemented in VHDL and was synthesized with 0.18 mm CMOS technology. The footprint of the 2-D BinDCT coprocessor, which includes memory buffer, is 0.1173 mm². The BinDCT coprocessor can calculate video in CIF format at 30 frames per second at 5 MHz clock rate with 1.55-volt power supply. The BinDCT coprocessor dissipates 12.05 mW. With its fast transform, compact size and low power consumption, the BinDCT coprocessor is an excellent candidate for DCT-based wireless multimedia coding systems.

Recently, backgrounds modeling methods that employ Time-Adaptive, Per Pixel, and Mixture of Gaussians (TAPPMOG) model have become more and more popular owing to their intrinsic appealing properties in video surveillance. Nevertheless, they are not able parse to monitor global changes in the scene, because they model the background as a set of independent pixel processes. In this paper, Gibbs Distributions-Markov Random Field (GDMRF) model is applied to the background modeling, and then the Simulated Annealing algorithm is developed to extract the background from video sequences. Experimental comparison between our methods and a classic pixel-based approach reveals that our proposed method is really effective in recovering from situations of sudden global illumination changes of the background, and can perfectly adapt the object moving in the background.

Volume rendering has been a key technology in the visualization of data sets from various disciplines. However, real-time volume rendering of large scale data sets is still a challenging field due to the vast memory, bandwidth and computational requirements. In this paper, to visualize small to medium scale data set in real-time, we first proposed a new kind of volume rendering graphic processor based on object-order splatting algorithm in which flexible transfer function configuration and software optimization such as early opacity termination and transparent voxel occlusion can be achieved. At the same time, the processor also integrates an eight-way interleaved memory system and an efficient address calculation module to accelerate the voxel traversal process and maintain high cache hit rate. Multiple parallel rendering pipelines embedded also can achieve local parallelism on board. Second, in order to render large scale data sets, a real-time and general-purpose volume rendering architecture is also presented in this paper. By utilizing graphic processors on PC clusters, large scale data sets can be visualized resulted from the high parallel speedup among graphic processors.

In this paper we present a new bus protocol satisfying extreme real time demands. It has been applied to a high performance quality inspection system which can involve up to eight sensors of various types. Thanks to the modular configuration this multi-sensor inspection system acts on the outside as a single sensor image processing system. In general, image processing systems comprise three basic functions (i) image acquisition, (ii) image processing and (iii) output of processed data. The data transfers for these three fundamental functions can be accomplished either by individual bus systems or by a single bus. In case of using a single bus the system complexity of the implementation, i.e. Development of protocols, hardware employment and EMC technical considerations, is far smaller. An important goal of the new protocol design is to support extremely fast communication between individual processing modules. For example, the input data (image acquisition) is transferred in real time to individual processing modules. Concurrent to this communication the processed data are being transferred to the output module. Therefore, the key function of this protocol is to realize concurrent data paths (data rates over 1.2 Gbit/s) by using principles of pipeline architectures and methods of time division multiplex. Moreover, the new bus protocol enables concurrent data transfers via a single bus system. In this paper the function of the new bus protocol including hardware layout and innovative bus arbiter are described in details.

This paper proposed an approach of online content filtering system, which can filter unexpected content from Internet, support searching, detecting, recognizing images, video and multimedia data. The approach consists of three parts: first is texture feature extraction with quasi-Gabor filters. These filters are constructed in different directions and sizes in frequency domain of images. This avoids convolution and multiplication with images spatially. Second, the extracted features are sent to Kohonon neural networks to perform decreasing dimension. The outputs of Kohonon network are then fed to a neural network classifier to get the final classification result. The proposed approach has been applied in our content monitoring system, which can filter unexpected images and alarm by pre-defined requirement.

In this paper we are going to present the first real-time S-parameter positron imager. This is a useful tool in solid state technology for mapping the lateral defect types and concentrations on a material sample. This technology has been developed for two major categories of researchers, the first being those that have a focused low energy positron beam and second those that do not. Here we describe the design and implementation of a real-time automated scanning system that rasters a sample surface with a 0.5mm diameter positron source (or beam focus) so as to give an S-parameter image of a sample. The source (or beam) rasters across a region of a semiconductor sample in rectilinear motion while gamma ray energies E_γ are processed using a standard HP Ge spectroscopy system and a 14 bit nuclear ADC. Two other ADCs are used to obtain the x, y coordinate data corresponding to each event by storing voltage pulses from the x & y stepper motor drives (or saddle coil currents) gated with the event pulses. Using these event data triplets (x, y, Eγ) the S-parameter is computed in real time for each pixel region and is used to refresh a color image display on the screen coordinates. Optimal use is made of processing time and the system resources. This user-friendly system is efficient for producing high resolution S-parameter images of the sample. (patent pending 2003)