Autonomous vehicle industry remains cool to thermal imaging

Just before 10 pm on 18 March 2018, an Uber employee named Rafaela Vasquez was behind the wheel of a Volvo, cruising just under the speed limit down a four-lane road in Tempe, Arizona. She was not watching the road. Instead, she gazed at her dashboard-mounted cell phone, which was streaming a reality TV singing competition. Vasquez wasn't being completely negligent, however. After all, the car had eyes of its own: a lidar imager, seven optical cameras, and ten radars. It also had a computer brain, algorithmically programmed to prevent the car from hitting other vehicles, crashing into fixed objects, and running over pedestrians.

On that last point, Uber's Volvo failed spectacularly. It struck a pedestrian named Elaine Herzberg. The National Transportation Safety Board's investigation of the crash reported that while the car's sensors had indeed picked up on something up ahead, they couldn't provide the computer with sufficient information for it to conclusively figure out what that thing was. At least, not before Herzberg was dead on the road. Since the Tempe tragedy, many experts in the remote sensing community have pointed out that if Uber's sensor array had included a thermal infrared camera the car likely would have identified Hertzberg in time to hit the brakes. However, before the autonomous vehicle industry writ large warms up completely to thermal infrared, the technology's proponents will have to make more than just a case for superior safety.

Every type of self-driving car sees the world through a different mix of sensors. Each sensor type has strengths and weaknesses, and autonomous vehicle makers assemble their arrays so one sensor's super sight peers into another one's blind spot. For instance, lidar can map the car's surroundings in incredible detail. However, it goes blind in rain or snow. Radar, on the other hand, does not. But, radar doesn't have the spatial resolution to really pick out pedestrians with confidence. And neither radar nor lidar does a great job of sensing things like traffic or brake lights, things optical cameras see very clearly. Visual learning algorithms behind optical cameras also do a pretty good job of picking out pedestrians ... as long as there's enough light. The circle of redundancies goes round and round.

Thermal infrared imaging could step in to solve the low light problem. The infrared portion of the radiofrequency spectrum is huge, encompassing wavelengths too long to be visible light, and too short to be radio—everything from 700 nanometers to 1 millimeter. The thermal imaging portion of that bandwidth is a relatively small subset. Far infrared detects wavelengths between 8 and 15 microns. "Objects seen in this bandwidth emit radiation completely passively," says John Lester Miller, the CEO of Cascade Electro Optics and chair of the SPIE conference on infrared technology and applications. This means thermal imagers can see in complete darkness.

A typical human pedestrian casts a thermal silhouette at around 9.25 microns. Added to an autonomous vehicle's remote sensing array, this kind of information could dramatically improve the car's reaction time. Take the Tempe tragedy, for example. According to the National Transportation Safety Board's report, Uber's Volvo detected an anomaly about six seconds before it impacted her. However, the car's computer couldn't quite figure out what it was detecting. It first classified Herzberg as an unknown object. Then, another vehicle. Then, it thought she was a bike. With each classification, the computer recalculated Herzberg's assumed trajectory. It wasn't until about 1.3 seconds before hitting Herzberg that the car decided it needed to engage emergency braking.

Thermal imagers in cars today

It's highly likely that a thermal imager would have picked up Herzberg's signature. After all, thermal imagers have been helping human drivers identify pedestrians, animals, and cyclists for nearly two decades. In 2000, Cadillac debuted its Night Vision system on the Deville product line. It wasn't true night vision—which amplifies available light—but thermal infrared. Using a grill-mounted, 3-inch passive sensor, Cadillac's system (developed by Raytheon) picked up thermal imagery from the oncoming road, then transmitted it to a grayscale display in the car's center dashboard. Cooler objects were darker, while warm stuff, like bodies, showed bright white.

About five years after Cadillac, BMW introduced thermal infrared, too. And, in 2008, they added an upgrade: the onscreen display would automatically detect, and emphasize, humans or animals in the line of traffic. Now this tool wasn't just enhancing situational awareness, it was a step towards the type of situational awareness AI needs to develop for automated driving.

FLIR Systems designed BMW's thermal infrared cameras. And while the system was initially meant to help drivers see better at night, FLIR's head of product Mike Walters says the company is actively working on applications for autonomous vehicles. "Our sensors can see the whole environment, and since they're looking at emitted energy, not reflected light, they don't get blinded by the sun," he says. The company has also been tuning its data using publicly available neural networks—a type of AI that mimics a biological brain's nodal reference structure—and Walters says he's pretty happy with the results. FLIR has also released its thermal imaging library as open access, so anyone can train their self-navigating AI to recognize heat signatures.

The three-legged stool

So why hasn't thermal imaging already taken the autonomous vehicle industry by storm? "Auto manufacturers are tough customers," says SPIE Fellow John Lester Miller. "They want everything cheap, reliable, and durable." Thermal imagers haven't yet met all those criteria to the self-driving industry's standards. And those standards—price, reliability, and durability—often run in opposition to one another. For instance, Miller says, some auto industry specs can exceed what's considered military grade, but there's no way even the smartest, safest, most self-drivingest car in the world would survive in the consumer market if built on a military-grade budget.

And these standards aren't just applied to thermal imaging. Uber's Tempe trial was a deliberate attempt to prove their self-driving technology could process the environment using fewer sensors. In 2016, when Uber debuted its autonomous Volvos, it simultaneously retired its fleet of Ford Focuses, each of which was equipped with seven lidar units. Lidar units are notoriously pricey—the 360˚ spinning Velodyne models Uber used on its rooftops cost around $85,000 apiece. Lidar is, at this point, essential for self-driving cars. But it still needs backup. As mentioned before, it conks out during rain or snow, plus it doesn't pick up every environmental queue. "There's no single sensor that does this correctly," says Walters. So, self-driving companies are stuck purchasing multi-sensor assemblies. Radar, optics, thermal. This stuff adds up.

The challenge for automakers is figuring which mix of sensors meets their price point, while also keeping their car from riding off the road, into other vehicles, or perpetrating another pedestrian tragedy. If thermal infrared makes it into a self-driving company's mix of sensors, it's going to have to become worth it, either through increased functionality or lower prices. How low? "It's probably too soon to commoditize," says Michael Dudzik, president of IQM Research Institute. "The car companies will pick a number based on what their market studies tell them."

And cost per unit isn't the only challenge facing thermal imagers. "Nobody has made these sensors on the scale that auto manufacturers are going to need," says Miller. "They want millions!" That means setting up what could effectively be a multi-billion—possibly trillion—dollar network to mine the rare minerals required to create the laser diodes necessary for thermal imagers. Of course, FLIR points out that it has produced over 1 million of its Lepton imagers. But, that was over the course of five years, so they haven't necessarily solved all the issues inherent in scaling up.

Zero tolerance for error

Car companies go through great pains to duplicate the complexity of decision making in the human brain, but it's more than duplication that these companies are after. Autonomous vehicles need to be darn near perfect drivers. Herzberg's death was a tragedy, no doubt. However, in 2016—the most recent year with available records—37,461 people died in car-related accidents. And not only is that statistically low for the last 50 years, it's more than 6,000 deaths lower than the annual average (43,588) of auto fatalities since 1960.

Given how poorly people internalize relative risk—consider the nationwide panic that happens after a single shark attack—autonomous cars probably won't build momentum in the consumer market if they continue killing humans. That means their brains need to upgrade, and fast. "Each driving system has millions of lines of code supporting it," says Dudzik. "Each sensor you add means the system gets more complex." This means more chances for mistakes, and more opportunities for the car to come up against an anomaly it can't resolve.

The Society of Automobile Engineers International has a numerical scale for judging vehicle autonomy. It runs from zero to five. A zero is no automation—a rating that encompasses everything from a ‘68 Mustang that's missing a side mirror, all the way to a BMW equipped with thermal sensor to help its human driver swerve around a deer in the road. Level five full automation: napping in the backseat while your robot chauffeur ferries you though rush hour. The very best autonomous technology is currently peaking at around a 2.5.

Thermal imaging might be the technology autonomous vehicles need to reach level five. "Overall I'm pretty positive that thermal photonics are going to play a big role in turning self-driving vehicles into consumer products," says Miller. "There's always issues and challenges, that's why we don't make minimum wage in this industry." There's no doubt this industry will continue taking on the challenges.

In the end, it comes down to one question: How much are consumers willing to pay in order to take their eyes off the road?

Nick Stockton is a freelance reporter based in Pittsburgh who writes about transportation, remote sensing, and the environment. His work has appeared in WIRED, Scientific American, Popular Science, and several other publications.