Share Email Print

Proceedings Paper

On covariance structure in noisy, big data
Author(s): Randy C. Paffenroth; Ryan Nong; Philip C. Du Toit
Format Member Price Non-Member Price
PDF $14.40 $18.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

Herein we describe theory and algorithms for detecting covariance structures in large, noisy data sets. Our work uses ideas from matrix completion and robust principal component analysis to detect the presence of low-rank covariance matrices, even when the data is noisy, distorted by large corruptions, and only partially observed. In fact, the ability to handle partial observations combined with ideas from randomized algorithms for matrix decomposition enables us to produce asymptotically fast algorithms. Herein we will provide numerical demonstrations of the methods and their convergence properties. While such methods have applicability to many problems, including mathematical finance, crime analysis, and other large-scale sensor fusion problems, our inspiration arises from applying these methods in the context of cyber network intrusion detection.

Paper Details

Date Published: 30 September 2013
PDF: 12 pages
Proc. SPIE 8857, Signal and Data Processing of Small Targets 2013, 88570E (30 September 2013); doi: 10.1117/12.2037882
Show Author Affiliations
Randy C. Paffenroth, Numerica Corp. (United States)
Ryan Nong, Numerica Corp. (United States)
Philip C. Du Toit, Numerica Corp. (United States)

Published in SPIE Proceedings Vol. 8857:
Signal and Data Processing of Small Targets 2013
Oliver E. Drummond; Richard D. Teichgraeber, Editor(s)

© SPIE. Terms of Use
Back to Top