Context-aware tracking with wide-area motion imagery

Wide-area motion imagery (WAMI) generates high-resolution images to enable tracking and recording of vehicle and pedestrian movements over city-sized areas. Sensors are mounted on aircraft to provide a wide field of view, but the resulting image may include a very large number of individuals and objects. This presents difficulties in identifying a target by physical characteristics alone. To overcome this, we can use WAMI alongside a ‘ context-aware’ tracker: that is, one which is sensitive to the target's interactions with its surroundings (or ‘ network’). We can then record an individual's ‘ pattern of life,’ by following their movements over several days (their trace), and during a single day (their tracelets).

Using WAMI video, we focus on targets that display abnormal behaviors: for example, a suspected drug dealer who drives to the same place every other day. We identify the location of interest (a drug house, in our example) from an established database. The tracker records cars and pedestrians that visit the house, enabling us to construct a network that includes the location (the ‘ node’ for our model). We also record the cars' and pedestrians' traces and tracelets (the ‘ links’) to obtain targets' information (see Figure 1). Finally, we analyze individuals' behavior using a reasoning model that employs trace analysis to obtain the pattern of life. Furthermore, when new locations are identified, we update the knowledge database for future movement analysis.

Figure 1. A diagram of the system for obtaining pattern of life recognition using wide-area motion imagery (WAMI) and a context-aware multi-target tracker (CAMT).

We pinpoint our selected location using the WAMI sensor's viewing angle and a terrain map. Our context-aware multi-target tracker (CAMT) uses a spatio-temporal model, maximum consistency context, that seeks the most consistent associations in its neighborhood.¹ This determines if there is a link between two detected targets across two consecutive frames, and selects reliable context information, filtering out noisy distraction.

To determine the features of a moving target, we use a particle filter: a sampling technique for estimating the probability density function of the variables in the system. The target's features are observations of its underlying states. Our algorithm automatically updates its estimation of the probability distribution of changes in these states over time. Figure 2 shows a tracking result using the CAMT over a very large number of targets.

Using our technique over a specific period, we construct information networks and examine the semantic attributes of each network element to determine patterns. We detect nodes and links using DJ-Cluster, an algorithm based on density and join-based clustering.² Using the network structure—the detected nodes, links, and associated attributes— we can generate features based on target activities and events. From these we obtain the pattern of life. As we accumulate temporal-spatial analysis of targets' journeys over an extended period, we can recognize specific actions, from which we infer behavior during a particular time frame. Recognizing activities in the region of interest, we can also determine the pattern of life. ^3–8

Figure 2. An example of CAMT results to track pattern of life in multiple targets, using WAMI data. Each line segment indicates the short-term trajectory of a target, or a connection between its positions in the last several frames. The colors indicate movement in different directions.

In future work, we will extend our methods for information extraction to include full motion videos that provide greater detail of the target. These would have higher resolution and would be taken from a side view, as well as from above. We would also include text to complement the video input, providing target information.