Capturing 3D hyperspectral image cubes for dynamic events

Hyperspectral imaging systems capture spectral and spatial information and produce a hyperspectral data cube, or ‘hypercube,’ from the data. These hypercubes consist of 2D spatial images with spectra at each pixel. Hyperspectral imaging is used in many applications, including agricultural health monitoring from aircraft or satellites, mineral detection and identification, food processing, and medical diagnostics. Typical hyperspectral systems use some form of scanning—tunable filters¹ or laser illumination,² for example—to build up a hypercube over multiple camera frames, and are therefore unable to temporally resolve dynamic events. Some systems image only a single spatial slice of the cube at one time and must scan over the spatial dimension using platform motion or a rotating mirror.³ For applications that image dynamic events (those that change within the scanning time), these methods are not sufficient to accurately capture the 4D data content (2D spatial + spectral + temporal).

We have developed a new device, the 4D imaging spectrometer⁴ (4DIS), capable of the high spectral and temporal resolution required for quickly evolving events. The sensor obtains an image cube with each camera frame, enabling capture and analysis of ‘hypervideo’ sequences. The fundamental concept of the 4DIS is shown in Figure 1. The lens forms an image onto a 2D array of fiber optics. At the opposite end, the fibers are rearranged to form a linear column. In other words, the 2D spatial content is collapsed to 1D with the 2D information preserved. The column of fibers defines the input slit of an imaging spectrograph (e.g., Offner), from which data is captured using a scientific 2D imaging camera. The output from this camera contains the spatial and spectral information that can be remapped to a 3D image cube in post-processing. We have developed 4DIS sensors in several spectral regions: ultraviolet, visible to near-infrared (VNIR), short-wave infrared, mid-wave infrared, and long-wave infrared. 4DIS sensors have been built with very high frame rates (up to 100,000 cubes/s). Our VNIR 4DIS is shown in Figure 2.

Figure 1. Fundamental concept of the 4D imaging spectrometer (4DIS) sensor: a lens focuses the image onto a 2D fiber array feeding to an imaging spectrometer. The fiber reformats the 2D array of pixels (x, y) into a 1D line (x*y), which is remapped at the end to produce the 3D image cube. λ: Wavelength.

Figure 2. Photograph of our visible-to-near-infrared 4DIS sensor.

The computational requirements for processing the acquired data are relatively low, enabling true, real-time display and analysis of the spectral information using standard computers. Real-time display supports efficient focusing and boresighting as well as scene recognition and target tracking. This feature presents a distinct advantage over other snapshot sensors, such as the computed tomographic imaging spectrometer,⁵ which require intensive computation to deconvolve the raw data.

The hypervideo 4DIS sensors have been employed for a wide variety of dynamic events including muzzle flash and gun identification, rocket plumes, atmospheric reentry observations, improvised explosive device detonations, lightning characterization, and real-time target detection.⁶ To illustrate the 4DIS data product, Figure 3 shows a single-frame hypercube collected during a missile launch event. The inset shows the 2D image at one wavelength band (2.1μm) and the plotted spectra correspond to individual pixels in the image. The spectral data reveals distinct features that arise due to the emissive elements in the exhaust plume—water (H₂O), carbon dioxide (CO₂), and hydrochloric acid—and also the attenuation due to specific atmospheric gases (H₂O and CO₂).

Figure 3. Spatially resolved spectra of a rocket exhaust plume. Blue text indicates emissive elements and black text shows atmospheric absorption. The inset displays the 2D image at one wavelength band (2.1μm). The plotted spectra correspond to individual pixels in the image. H₂O: Water. HCl: Hydrochloric acid. CO₂: Carbon dioxide. CO: Carbon monoxide.

The spectral and temporal resolution of our device is relatively high and its low spatial resolution can be mitigated by fusing the 4DIS data with imagery from a boresighted context camera in real time. An example of fused hypervideo data is shown in Figure 4. In this demonstration, the blood flow to two fingers of a human hand was constricted by a rubber band, which was then removed. The 4DIS data was processed using a modified band-ratio technique in which certain specific multispectral bands are selected from the hyperspectral image cube, with the resulting value in this case corresponding to the saturated oxygenation value in hemoglobin (SO₂). This SO₂ spectral value is displayed on the color scale shown in the image, which lacks a high spatial resolution, and is combined with the context camera imagery, which provides high spatial resolution with no spectral content. This fused image thus provides high spatial resolution and maintains spectral content. With the rubber band on, the blood oxygenation level can be seen to decrease in the affected fingers. When the rubber band is removed, the oxygenation values quickly return to a normal level. Although this is a contrived example, it demonstrates the general ability of the system to provide spatially resolved spectra of dynamic events. This capability could be particularly important in biomedical applications that require the temporal evolution of specific spectral signatures to be monitored.

Figure 4. Fusion of spectral analysis from hypervideo data with context camera imagery. In this example, a rubber band was placed on the index and middle fingers of a human hand and then taken off about 2 minutes later. The spectral analysis was designed to display the oxygen saturation (SO₂) value of hemoglobin.

In summary, our hyperspectral imaging system provides snapshot hypervideo with high-resolution 4D information (2D spatial + spectral + temporal) and is therefore suitable for the scanning of transient events. Currently, we are working on designs for the next-generation 4DIS sensor, which will enable improved signal quality and higher hypercube rates by providing increased sensitivity, increased spatial resolution by using innovative design concepts, and reduced size and weight by using compact spectrometers and cameras.