Nature has equipped the colorful mantis shrimp with a set of hyperspectral eyes, allowing it to see visible, infrared, and ultraviolet light. With these eyes, the shrimp can distinguish between different types of coral that all look the same to human eyes.
Humans are also interested in seeing objects through hyperspectral vision for applications in food sorting, medicine, agriculture, and many other domains.
Research on bringing this vision-discrimination power to more and more practical applications has resulted in the development of compact, low-cost, and easy-to-use hyperspectral cameras.
So, forget Google glasses; get ready put on “mantis-shrimp glasses.”
What is hyperspectral vision?
As most optical scientists know, standard camera sensors capture light in three broadband spectral bands (red, blue, and green) that are visible to the human eye. Spectral image sensors, on the other hand, collect more detailed information. They sense the light in many small-wavelength bands. The result is a spectral cube with one spectral and two spatial dimensions.
The spectral dimension visualizes light intensity in function of wavelength. The spectral cube may be interpreted as a stack of images, one for each wavelength.
A typical spectral cube.
In contrast to multispectral sensors, which sense a discrete number of bands, hyperspectral sensors detect a high number of narrow bands, continuously positioned across the spectrum.
The spectral information gained from spectral cameras can be used in all sorts of applications where objects or materials can be identified based on their different characteristics. For example, they could be used for food sorting, such as picking out the ideally roasted coffee beans or the brightest green peas. They can also detect skin cancer even before you can see anything with the naked eye. Or they could be used in farming to detect weeds around the crops and to spray herbicide only where needed.
Challenges for hyperspectral cameras outside the lab
Although there are many applications for hyperspectral cameras, there are also many problems. Many of today’s hyperspectral cameras are expensive, slow, big, and difficult to use. For this reason, they are mostly used as a scientific tool in dedicated laboratory settings.
Major innovations are needed before hyperspectral imaging could be used in more practical applications.
And this is just what researchers at the Belgian research institute imec are doing. With its expertise in complementary metal–oxide–semiconductor (CMOS) technology, imec researchers are moving hyperspectral imaging out of the exclusive domain it is used in today.
They started with a commercially available image sensor and integrated a group of 100 spectral filters on top of that imager. These filters were arranged in the shape of a wedge.
Conceptual drawing of imec’s hyperspectral line-scan imager with 100 static spectral filter structures.
To enable low-cost processing of such a microscopic wedge filter, a design was introduced that is able to compensate for process variability. The result is a compact and fast hyperspectral camera made with mass-producible and fully CMOS-compatible process technology.
The integrated spectral filters are narrow-banded Fabry–Pérot interference filters. The Fabry–Pérot filter is typically made of a transparent layer (called a cavity) with a mirror at each side of that layer. The height of the cavity defines the central wavelength of the optical filter, and the reflectivity of the mirrors defines the full width half maximum (FWHM) of the filter.
Different hyperspectral imager designs can be realized by using these filters. The hyperspectral filters can be processed, in principle, on any image sensor to match different application specifications.
Similarly, the spectral range can be tuned for the desired application. Both a line-scan and a snapshot camera prototype were developed based on this principle.
Compact high-speed camera with imec hyperspectral sensor enables real-world spectral imaging.
Line-scan hyperspectral camera
The line-scan hyperspectral prototype can identify and classify objects that have controlled translational movement such as on a conveyor belt for food inspection. For the line-scan camera, optical filters were integrated on top of a CMOS sensor, as a stepwise wedge. The line filters for the different wavelengths are built by varying the cavity height in one direction over the sensor while keeping the cavity height fixed over the second dimension.
Two conceptual drawings of the hyperspectral snapshot imager with 32 tiled spectrum bands.
The prototype camera is faster than conventional line-scan hyperspectral cameras that typically scan the image line per line. The new camera is able to cover multiple spatial lines in one spectral band.
This development is particularly interesting for applications in need of a low-cost, high-speed, and compact hyperspectral camera in combination with high spatial/spectral resolution. It can be flexibly tuned to different applications just by adjusting the software. For example, it can be used to inspect carrots one day and beans the next day.
A 200mm wafer image with both types of spectral filter layouts, snapshot and line-scan, monolithically integrated with CMOS image sensors.
Snapshot hyperspectral camera
The snapshot hyperspectral camera prototype can capture an entire hyperspectral image (cube) at one discrete point in time without scanning.
To achieve operation at video rates, the optical filters on top of the image sensor are organized in a tiled configuration. Each of these filters senses only one narrow band of wavelengths.
Second, an optical component is added to the system which duplicates the scene onto each filter tile. Lastly, an objective lens forms an image of the scene on the optical duplicator. In this way, the three dimensions of the spectral cube (two spatial and one spectral) are mapped on the two dimensions of the sensor.
Security applications for the snapshot-camera concept include analyzing dynamic scenes where objects move randomly, such as in detecting license plates and tracking intruders.
In agriculture, the camera could identify individual animals to allow precision livestock farming. For food control, the camera could be used to monitor processes such as coffee-bean roasting.
–Els Parton is a scientific editor and editor-in-chief of imec’s InterConnect magazine. She holds a PhD in bioscience engineering from the University of Leuven (KU Leuven) (Belgium).
–Murali Jayapala is a senior researcher in imec’s Integrated Imaging group where he focuses on system designs for next-generation imaging systems. He has an MsEE in systems science and automation from the Indian Institute of Science and a PhD in applied sciences from KU Leuven.
–Andy Lambrechts leads imec’s Integrated Imaging team and works on hyperspectral imaging, lens-free microscopy, and other activities that combine imec’s process technology with systems and software knowledge to enable new applications in the vision domain. He has an MSc and PhD in electrical engineering from KU Leuven.
Imec is a non-profit research organization in nano-electronics that serves as a bridge between the fundamental research at universities and the technology development in industry.
With expertise in chip processing and system design, imec’s industry partners include those in healthcare, energy, and in information and communications technologies (ICT).
Headquartered in Leuven, Belgium, imec has R&D teams and/or offices in The Netherlands, China, Taiwan, India, Japan, and the United States.