- Biomedical Optics & Medical Imaging
- Defense & Security
- Electronic Imaging & Signal Processing
- Illumination & Displays
- Lasers & Sources
- Micro/Nano Lithography
- Optical Design & Engineering
- Optoelectronics & Communications
- Remote Sensing
- Sensing & Measurement
- Solar & Alternative Energy
- Sign up for Newsroom E-Alerts
- Information for:
Electronic Imaging & Signal Processing
Image-based motion analysis
New software will analyze moving targets in sequences of images: abilities include target tracking, multi-sensor triangulation, and model-based pose estimation.
24 May 2006, SPIE Newsroom. DOI: 10.1117/2.1200604.0192
The quantitative motion analysis of sequences of images poses a challenge in disciplines ranging from automotive crash-test analysis to biological motion studies and military test-range instrumentation. The images can be created using high-speed or standard video cameras in a wide variety of formats. In many cases, analysis requires not only the image data, but various supporting (ancillary or meta) data that can vary over time: these include camera location and pointing, camera frame rate, or radar range data.
Both the processing and analysis requirements can vary widely, from simple, single-camera 2D (x-y) tracking to geometric-lens-distortion correction, model-based pose estimation, multi-camera triangulation, target-shape analysis, and radiometric analysis. Unique to image-based motion analysis is the need to deal with sequences of images—possibly exploiting ancillary data—to develop time histories of the desired features of objects, such as position, shape, and intensity.
In traditional motion analysis, each target in each frame is manually designated. While this is acceptable for short sequences with only a few targets, it becomes impractical otherwise. To address the needs and challenges of image-based motion analysis, I developed the Visual Fusion motion analysis software package (see Figure 1): a software analysis tool suitable for a wide spectrum of users with expertise ranging from personnel without degrees to Ph.D.-level investigators in a broad range of application fields. It also required the development and integration of many kinds of image-analysis technology with much wider applications: these include multi-target tracking, triangulation-error analysis, automatic outlining of complex-shaped targets, and the correction of complex optical distortions.
Figure 1. Motion-analysis software user interface showing target detection and tracking, plotting of position versus time, Fourier transform to determine frequency, and curve fitting to provide more exact estimation.
Motion analysis can be seen as consisting of the following steps: image pre-processing; target detection and feature extraction (shape, moments, centroid, leading edge, intensity); correction for optical distortion; calibration from focal-plane pixel space to units such as azimuth and elevation or x-y distance units (e.g. meters); calibration from pixel grey scale to radiometric intensity units; and detect-to-track association to develop time histories or tracks (see Figure 2). This can be followed by multi-sensor triangulation, model-based pose estimation, determination of the motion of one object relative to another, numerical derivatives or curve fitting to estimate velocity and acceleration, or frequency analysis. Each of these involve a variety of adjustable parameters or even algorithms, necessitating the development of a user interface that provides simple operations for many applications, but allows the experienced user to adjust parameters for more challenging problems.
Figure 2. Multi-target tracking and surface relief, alternative image display.
A basic requirement for most applications is to be able to manipulate and visualize both the image and ancillary data. This means the software must be able to read a wide range of different image and data formats and allow the user to easily view the imagery or data. Some of the basic tools available at this level include the ability to adjust gray scale to enhance features of interest and perform operations such as edge enhancement or convolution filtering and electronic zoom. Additionally, it is sometimes informative to display imagery in a 3D relief format where bright regions appear as higher elevations.
Basic target detection involves pre-processing, followed by threshold detection (dark targets on light or light targets on dark), and then either correlation detection (in which the software finds the various user-selected targets) or detection of pre-defined targets, such as standard quadrant patterns. Simple radial polynomial optical system correction can be used to correct for wide angle lens distortion. However, I elected to implement a more robust bi-linear interpolation on a grid of points, thus allowing correction of off-axis imaging systems or external windows, such as curved windscreens. Robust multi-target detect-to-track assignment (basically a modified global-nearest-neighbor algorithm) is used to develop time histories of targets.1–4
Passive imaging sensors are 2D-angle-only sensors. To generate 3D target tracks, one must either bring in range data from radar and perform multi-sensor triangulation, or use model-based pose estimation. While developing the former I was surprised to find that, although triangulation itself is well known, little work had been published on error analysis to determine the accuracy of triangulation results. As a result, I developed the theory for triangulation error analysis5 and some methodology for practical application of the theory.6
Though model-based pose estimation is a well known technique,7 classical approaches suffer from the need for an initial pose guess. This is impractical in many situations and becomes unstable in important scenarios involving approximately planar objects. Recent work8,9 has greatly improved performance in this area.
Image-based motion analysis has recently moved from manual frame-by-frame point and click analysis to much more automated and sophisticated computer analysis. This brings together a rich set of image-processing and numerical analysis techniques. While commercial software is available, an understanding of the underlying techniques and principles can be valuable in accurately obtaining the most information from these tools.
Jack Sanders-Reed is a Technical Fellow with the Boeing Company, specializing in image-based motion analysis, image exploitation, and electro-optic systems. Developer of the Visual Fusion motion analysis software package, he has more than 20 peer reviewed and conference publications to his name, has published numerous papers in Optical Engineering and the Proceedings of the SPIE, and teaches the image-based motion analysis course at the SPIE Defense & Security Symposium.