Embedded architecture enables intelligent vehicles

New vehicles from the automotive industry include a growing number of progressively more sophisticated embedded systems, including a variety of advanced driver assistance systems (ADASs), such as lane departure warning¹ and adaptive cruise control. For reasons of cost and technology, these innovations draw upon limited resources, leading to highly specific, optimized architectures for each application and platform. The goal is to develop dependable products capable of real-time environment recognition and decision-making for passenger safety.

The level of intelligence demanded by such systems can only be achieved by adapting high-level cognitive architectures and developing new methods for designing integrated circuits. Algorithm parallelism, for which hardware is especially well-suited, and the computational power of field-programmable gate arrays (FPGAs) must be exploited.

With these parameters in mind, we are exploring the possibility of scaling down artificial intelligence techniques (neural networks) and BB1-inspired blackboard architecture^2,3 to effectively use limited resources. Our initial focus is to detect traffic signals.

Figure 1. Phases (1-4) and agents (A-C) of an execution cycle in blackboard architecture. Adaptation of the BB1 blackboard architecture ^2,3. HMI: human machine interface.

FPGAs allow implementation of intrinsically parallel techniques, such as artificial neural networks.^4,5 However, low-cost FPGAs, such as might be used in industrial applications, provide extremely limited resources. We therefore propose a new approach that stresses conceptual and hardware architectural design. We use a low-cost Altera Cyclone FPGA.

Our system is based on images from a charge-coupled device (CCD) camera installed on the dashboard facing the road. Images are analyzed for features that may indicate the presence of a traffic signal, such as arcs and corners. Three neural networks perform this analysis, each specialized in identifying one type of feature. They carry out phase 1 (see Figure 1) with execution coordinated by agent A, the ‘manager.’ The position and number of features is analyzed in phase 2 and assigned a priority level by agent B, the ‘rater.’ Results are conveyed to the driver through the `executor’ agent, a human-machine interface, labelled C.

The order in which the agents operate and the data and resources used in the process correspond from a conceptual viewpoint to blackboard architecture,^2,3 a well-known type of cognitive architecture developed to facilitate data-sharing in artificial intelligence.

Insufficient resources prevent strict implementation of blackboard architecture with our hardware. A single neural network agent requires virtually all the FPGA's resources. We use at least three agents, and so must either employ several FPGAs or share a single one. Taking the latter alternative, we have built a single hardware neural network (double-layer perceptron) into the FPGA. We call it the Generic Neural Network (GNN), and it is shared between all agents. This represents a form of hardware reconfiguration.⁶

The GNN is configured for each agent in runtime. It is loaded with network weights and agent biases that are stored in memory after being previously calculated off-line with Matlab. Weights of unused neurons are set to zero. Reconfiguration and execution times are sufficiently short to allow new frame processing from the camera every 40ms, the phase alternating line (PAL) rate.

Each layer in the GNN includes its own control logic. A layer of n neurons can perform the corresponding n operations in parallel due to independent access to the memories of each neuron. To prevent precision losses, over- and underflows, the data path has been dimensioned.

Designing intelligent embedded systems for the automotive industry requires adapting high-level techniques to limited resources. By sharing the same hardware for the entire system, we reduced components to create a generic, runtime configurable hardware neural network. System parts and operation employ a blackboard architecture, which provides a high degree of modularity, necessary for consistent scalability. System functionality can be enhanced by adding new data (weights and biases) to memory. In the near term, the system will include pedestrian detection. The same architecture could also be easily deployed on several FPGAs, or over hybrid FPGA/digital-signal-processing/analog boards.

The functional and platform scalability we describe will enable future applications with increased intelligence. Many applications require advanced perception algorithms and sensory fusion as well as distributed sensing and action, for which blackboard architecture is particularly well suited.