A new approach for real-time target tracking in videos

Video segmentation is fundamentally challenging, but it has many computer vision applications. The goal of video segmentation is to isolate a target object from its background across a sequence of frames. It is a 3D problem that incorporates the dimension of time. That makes the input an order of magnitude larger than 2D problems, so computational efficiency is a major challenge. Time-efficient tracking of algorithms allows for tailing targets of interest in real-time and tracing these objects in large visual data sets. These capabilities are useful in law enforcement, analyzing shoppers' behavior, sports broadcasting (having a camera zoom on a ball at all times), and many more applications.

Tracking algorithms in the literature are categorized into three main classes, the active-contour approach,^{1, 2} statistical and stochastic methods,^3,4 and graph-theory-based tracking.^{5, 6} The active-contour approach uses continuous models coupled with consistency constraints to delineate a target object's boundary. Since digital images or videos are innately discrete, these methods introduce errors when converting the discrete input to a certain continuous function and when converting the continuous output back to a discrete solution. Statistical and stochastic schemes, by comparison, rely heavily on iterative steps that are computationally intense and neither guarantee optimal solution nor the same output over different runs with the same input data. The third approach, which we employ, represents the issue as a graph problem.

Until recently, graph formulations incorporated a set of motion-consistency-constraints, meaning the object's location in the current frame is constrained to appear close to its location in the previous frame, shifted by the estimated motion. This approach is vulnerable to occlusions and often results in computationally complex problems. In contrast to existing graph-theoretic methods, our approach represents motion as a feature (like color or position), which applies neither heuristics nor approximations. A further distinction of our algorithm is that we estimate motion as part of the tracking process, while the other approaches estimate it by optical flow algorithms. While considered to be most accurate, the convergence of optical-flow methods is slow, and often thousands of iterations are necessary for solving the problem. Our approach incorporates the significantly less computationally intensive MPEG-4 motion estimation schemes⁷while achieving high-quality tracking.

Figure 1. Representing frames from surveillance sequences taken from the Context Aware Vision using Image-based Active Recognition (CAVIAR) data set. CAVIAR is a project of the European Commission's Information Society Technology program.

Figure 2. The resulting trajectories for the PETS09 (tasks S2.L1).

We devised a generic, graph-theory-based tracking scheme in videos. Our method casts the tracking problem as a variant of the normalized cut (NC') problem⁸ and solves it with Hochbaum's Pseudo Flow (HPF) algorithm,⁹ which was shown to be most efficient for the current problem.¹⁰ The robustness of our approach allows for incorporating computationally less intensive block-matching, motion-estimation schemes, such as the ones used within the MPEG-4 video coding framework.¹¹ Although block-matching techniques generate noisy and coarse motion fields, we used them because they have two advantages. First, they have faster computation times, and there is a broad variety of off-the-shelf software and hardware components for performing this task that can easily be incorporated into the segmentation scheme. Second, if the videos are already compressed, then the motion information is inherent in their compressed form and is available from the video encoder.¹² In that case, there is no need to apply any motion estimation algorithm. These two advantages enable real-time execution.

Our evaluation of the method on standard (see Figures 1 and 2) and non-standard benchmark videos showed that the integration of motion into the segmentation process provided a superior segmentation output. For quantitative comparison of our method's performance to the state-of-the-art tracking algorithms,^13–15 we employed the multi object tracking (MOT) measures ¹⁶ for precision (MOTP) and accuracy (MOTA). As our algorithm does not tag or identify the different objects, this quantity is omitted in the MOTP calculations. As a base for the quantitative comparison, the PETS2009 S2.L1 and S2.L2 benchmarks (see Figure 2) were arbitrarily selected. Our method presents comparable results in significantly shorter running times. The fact that we can use an MPEG-4 motion field, which is coarse and noisy, indicates that our algorithm is robust, even though that is not a measurable quantity. Along with the time efficiency of the algorithm, this robustness makes the suggested scheme a suitable choice for many video segmentation applications.

Going forward, we plan to exploit the robustness and efficiency of the HPF algorithm in other vision problems. Another line of research is to implement our tracking scheme on a computer chip so the processing will be done on the camera itself. The latter will allow broadcasting of only the target of interest from the camera.