New wide-area surveillance techniques for protection of pipeline infrastructure

An automated mechanism can detect and issue warnings of machinery threats to pipeline rights-of-way.
30 January 2015
Vijayan Asari, Paheding Sidike, Chen Cui and Varun Santhaseelan

There are millions of miles of pipes buried along the length and breadth of the United States. The areas through which these pipelines run cannot be used for other activities. Furthermore, machinery—such as construction equipment and heavy vehicles—are major threats to pipeline infrastructure. Monitoring is therefore required to know whether a pipeline's right-of-way is threatened at any time. Rapid advances in sensor technologies have enabled the use of high-end video acquisition systems to monitor the right-of-way of pipelines and have generated a huge amount of data. It is, however, very costly to employ analysts to scan through this data and to identify right-of-way threats. An automated mechanism that can detect these threats and issue warnings is therefore warranted.

Purchase SPIE Field Guide to IR Systems, Detectors and FPAsSeveral object detection and recognition algorithms have been proposed previously.1–5 Car body edges, edges of a front windshield, and shadows can be used as features for car detection.6 An alternative simple vehicle detection algorithm involves exploring four elongated edge operators.7 A top-down matching method has also been developed for vehicle detection from high-resolution aerial imagery.8 On-line boosting, with an interactive training framework, is another option for automatic car detection.9 A feature-based approach to car detection uses scale-invariant transform features and an affinity propagation algorithm.10 We previously designed a three-stage pattern recognition framework to detect construction equipment in various lighting conditions and different object orientations using monogenic signal representation.11, 12 The majority of these techniques, however, are either computationally expensive or unable to deal with the complex environments associated with aerial imagery. None of these approaches consider the potential security issues of the detected objects for wide-area surveillance.

We have therefore developed a multistage framework for aerial imagery analysis.13 This can be used for the automatic detection and identification of machinery threats to pipeline right-of-way. The three phases involved in our proposed framework can be categorized as background elimination, part-based object detection, and risk assessment. With our approach we can take into account the constraints associated with aerial imagery, such as low resolution, lower frame rate, large variations in illumination, and motion blurs.

We developed the background elimination model with the aim of providing information regarding the contents of an image, which can be used in the process of threat detection. Our proposed segmentation technique has two parts, which are known as local textural features-based segmentation (LTFS) and adaptive perception-based segmentation (APS). LTFS is based on the properties of the area around every pixel within an image. As such, the output of the LTFS only contains prominent information from the input image (e.g., abnormal regions or a fully connected inhomogeneous area). Through LTFS, we are able to eliminate the majority of the background so that only some protruding regions remain (see Figure 1). These regions indicate the possible location of the target. Our LTFS method provides global background elimination, but it cannot be trained to eliminate specific undesired objects (e.g., trees, houses, or roads) that are not considered as threats. We therefore also developed the APS model (see Figure 2). In our APS algorithm, a test image is divided into various segments and passed through the trained model to detect the presence of an object. A sample result of our APS model, in which buildings are being segmented, is shown in Figure 3.

Figure 1. A sample result of the local textural features-based segmentation (LTFS) algorithm. (a) The original red, green, blue (RGB) image. (b) The LTFS output. The yellow circle indicates the threat object that has been identified.

Figure 2. Complete architecture for the proposed adaptive perception-based segmentation (APS) algorithm.

Figure 3. Illustration of an APS algorithm result, in which buildings are segmented.

With aerial imagery, a major challenge for detection is when the object of interest is partially occluded, e.g., by shrubs, trees, or buildings. A part-based model has been demonstrated as an attractive option for object recognition because of its ability to cope with partial occlusions and large appearance variations.14, 15 In our proposed part-based model an object is first partitioned into a certain number of parts, depending on the size of the object. We then use local phase information to extract informative attributes for describing the individual parts. Our next step is to convert the object parts into a larger number of clusters. In this process, we group similar parts into the same cluster and a histogram of oriented phase is used to describe the specific pattern of the parts. We subsequently organize each of the detected parts and their attributes as an integrated entity. Since a target can be represented by a certain number of patterns, we can train a classifier to detect the local patterns of the target. An occluded object can thus be detected by parts in the input scene. Multiple parts of an object can be detected without any false alarm with our technique (see Figure 4).

Figure 4. A sample result from the proposed part-based algorithm. The original image is shown in (a) and the part-based detection, with background elimination, is shown in (b). The yellow circle in (a) indicates the location of the object of interest. The red and green regions in (b) are the bigger and smaller parts of the detected object, respectively.

The output of our part-based object detection technique is the pixel location of the threat object in the input image. In real-life applications, however, a pipeline operator must also know the geolocation of an object to prevent any damage to the pipeline. A registration process that links the images to a geographical map is therefore required. In addition, some detected machinery threats may be far from a pipeline, or have a low threat probability (i.e., when they are not being operated). Considering these issues, we have designed an additional framework (see Figure 5) that can automatically analyze the geolocation and temperature information of a detected object, and can assign a risk level to any given threat (high, medium, or low). We have simulated and tested this proposed technique under various sets of challenging conditions to investigate its reliability and feasibility for monitoring the right-of-way of pipelines. In these tests, we have achieved over 85% accuracy for machinery threat detection. We have therefore demonstrated that our method can be used as a practical approach for wide-area surveillance and to protect pipeline infrastructures.

Figure 5. The proposed framework for automatic geolocation, temperature monitoring, and risk assessment of detected objects.

We have designed and tested a novel wide-area surveillance technique that can be used to protect underground pipeline infrastructures. Our multistage framework uses different segmentation and classification approaches to automatically detect machinery threats to pipeline rights-of-way, and to issue warnings regarding the severity of the detected objects. We are currently focusing on hardware-software integration and performance acceleration aspects, so that we can enable real-time processing in an onboard flight environment. We are now planning to modify the detection framework using a cascade of classifiers—with various feature set levels—to eliminate the possibility of false alarms caused by occlusions and cluttered backgrounds, without affecting the detection accuracy.

Vijayan Asari, Paheding Sidike, Chen Cui
Electrical and Computer Engineering
University of Dayton
Dayton, OH

Vijayan Asari is a professor in electrical and computer engineering and is the Ohio Research Scholars Endowed Chair in Wide Area Surveillance. He received his PhD from the Indian Institute of Technology, Madras. He also holds three patents and has published more than 450 research papers. His research areas include image processing, computer vision, machine learning, and pattern recognition. He is a senior member of IEEE and SPIE.

Paheding Sidike received his BS in electrical information engineering from Northwest University for Nationalities, Lanzhou, China, and his MS in electrical engineering from the University of South Alabama. He is currently working toward a PhD in electrical and computer engineering and serves as a research assistant. He has published more than 20 research papers in the areas of image processing, computer vision, machine learning, and pattern recognition. He is a member of SPIE, IEEE, and Tau Beta Pi.

Chen Cui is a graduate research assistant. She received her BS and MS in electrical engineering from Dalian Jiaotong University and the University of Dayton in 2011 and 2013, respectively. She is currently pursuing her PhD in electrical and computer engineering. Her research interests include image processing, pattern recognition, machine learning, computer vision, and computer architecture.

Varun Santhaseelan
Auviz Systems
Campbell, CA

Varun Santhaseelan received his PhD in electrical engineering from the University of Dayton, where he specialized in computer vision and image processing. His areas of research include removal of dynamic noise from videos, analysis of wide-area motion imagery, scene understanding in aerial imagery, and change detection methodologies. He obtained his MS in embedded software from Gannon University, Erie, PA, and his BTech from the College of Engineering, Trivandrum, India.

1. C. Cui, A. D. Braun, V. K. Asari, Robust textural features for real time face recognition, Proc. SPIE 9408, 2015. (In press.)
2. P. Sidike, V. K. Asari, M. S. Alam, Illumination-invariant pattern recognition using fringe-adjusted joint transform correlator and monogenic signal, Proc. SPIE 9024, p. 90240C, 2014. doi:10.1117/12.2040752
3. P. Sidike, T. Aspiras, V. K. Asari, M. S. Alam, A rotation-invariant pattern recognition using spectral fringe-adjusted joint transform correlator and histogram representation, Proc. SPIE 9094, p. 90940F, 2014. doi:10.1117/12.2050467
4. V. Santhaseelan, V. K. Asari, Tracking in wide area motion imagery using phase vector fields, IEEE Conf. Comput. Vis. Pattern Recognit., p. 823-830, 2013. doi:10.1109/CVPRW.2013.123
5. A. Mathew, V. K. Asari, Rotation-invariant histogram features for threat object detection on pipeline right-of-way, Proc. SPIE 9026, p. 902604, 2014. doi:10.1117/12.2039663
6. T. Zhao, R. Nevatia, Car detection in low resolution aerial image, Proc. 8th IEEE Int'l Conf. Comput. Vis. 1, p. 710-717, 2001. doi:10.1109/ICCV.2001.937593
7. H. Moon, R. Chellappa, A. Rosenfeld, Performance analysis of a simple vehicle detection algorithm, Image Vis. Comput. 20, p. 1-13, 2002.
8. S. Hinz, Detection of vehicles and vehicle queues in high resolution aerial images, Photogrammetrie Fernerkunding Geoinf. 3, p. 201-213, 2004.
9. H. Grabner, T. T. Nguyen, B. Gruber, H. Bischof, On-line boosting-based car detection from aerial images, ISPRS J. Photogrammetry Remote Sens. 63, p. 382-396, 2008.
10. S. Sahli, Y. Ouyang, Y. Sheng, D. A. Lavigne, Robust vehicle detection in low-resolution aerial imagery, Proc. SPIE 7668, p. 76680G, 2010. doi:10.1117/12.850387
11. B. M. Nair, V. Santhaseelan, C. Cui, V. K. Asari, Intrusion detection on oil pipeline right of way using monogenic signal representation, Proc. SPIE 8745, p. 87451U, 2013. doi:10.1117/12.2015640
12. M. Felsberg, G. Sommer, The monogenic signal, IEEE Trans. Signal Process. 49, p. 3136-3144, 2001.
13. V. K. Asari, Recent progress in wide-area surveillance: protecting our pipeline infrastructure, Proc. SPIE 9408, 2015. (In press.)
14. P. F. Felzenszwalb, R. B. Girshick, D. McAllester, D. Ramanan, Object detection with discriminatively trained part-based models, IEEE Trans. Pattern Anal. Machine Intell. 32, p. 1627-1645, 2010.
15. P. F. Felzenszwalb, R. B. Girshick, D. McAllester, Cascade object detection with deformable part models, IEEE Conf. Comput. Vis. Pattern Recognit., p. 2241-2248, 2010. doi:10.1109/CVPR.2010.5539906
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?