A vision-based collision warning system for unmanned aerial vehicles

Unmanned aerial vehicle (UAV) technology has reached a level of maturity that enables autonomous flight missions. However, due to certain safety shortcomings, UAVs are generally not able to fly autonomously outside the line of sight in shared airspace. This is a major barrier to the industry's growth and development¹.

One of the safety issues is collision avoidance.² UAVs fly blindly in autonomous mode using GPS-based waypoint navigation. This makes them dangerous to other static and dynamic objects, such as small manned aircrafts, other UAVs, tall buildings, or items such as cranes, which may not have been accounted for when calculating the route. There are several methods for detecting large intruder aircraft, including radar, radio transponders, and vision systems. However, for smaller platforms, vision systems are the only promising solution because of the constraints of size, weight, power demand, and cost. The ultimate device would monitor a 220^°×70^° field of view, detect a Cessna-sized aircraft from 2000 meters, would weigh no more than 80 grams, and would consume less than 1W.³

We are developing a prototype of a vision-based intruder detection system that meets these specifications. Its required optical resolution is 0.08-0.1^°/pixel. Therefore, we capture and process the video flow—which comprises frames of a minimum 3megapixels—in real time. To avoid spatial uncertainties, we used synchronized global shutter sensors. In addition, we selected multiple-panel cameras with S-mount objectives in order to cover the extremely large viewing angle as well as to limit the cost, size, and weight.

In our first prototype, we used five wide video cameras (see Figure 1). For the computation, we used a multiple-core foveal processor in a field programmable gate array (FGPA). The processor comprises three major blocks. The first is a pre-processor, which segments the image into sky and terrain background areas, and it identifies potential locations of the intruder aircraft. The second block is a foveal unit, which further processes the images of these locations. While the pre-processor is dealing with the entire image, the foveal unit processes small windows (fovea) around the potential location. The third unit is the FPGA processor, which manages the two processor blocks, the data flow, and the communication. It is also responsible for decision-making. We used a solid state drive to save the video flow during the test and verification flights.⁴ However, this would not be included in the final device.

Figure 1. Prototype of the multi-camera vision system to detect intruder aircraft.

Algorithmically, the challenge is to identify and track very small spots on a complex image. We applied different algorithms in the various image regions according to the edge density of the background.⁵ For low edge density (clear sky or low contrast gray clouds) we used an intra-frame method to detect high-contrast small spots that are not located on cloud boundaries. For high edge density regions (mostly terrain background), we used inter-frame techniques (see Figure 2). Here, we cut the image into segments, and calculated the displacement of these. Then, we applied subtractive detection of the moving objects.

Figure 2. Intruder aircraft in front of low-contrast cloud and high-contrast terrain backgrounds.

We collected flight video data with our system. The intruder aircraft were either distant large descending airliners, Cessna class small planes, or other UAVs. The detection rates on these scenarios depended on the size of the intruder aircraft and the cloud contrast. In the best case, when the intruder size was 4x5–7x4 pixels, and it was flown against a mixed cloudy and blue sky background, the intruder was found in 94.3% of the frames. The worst case, when the spot size was between 3x2 - 6x3 in front of strongly structured clouds, the detection rate was 86.4%. In all cases, there were some false tracks detected as well. However, it was easy to discard them because they were moving, which means that they were not on a collision course.

In summary, we are developing a multi-camera vision system prototype for UAVs that can identify intruder aircraft on a collision course. The system is capable of detecting and tracking multiple targets, and identifying those on a hazardous track.

In future work, we aim to team up with an industrial partner to implement our prototype as a low-cost, ultra-light, miniature flyable system.