Traditionally, information or data fusion has sought to combine information from multiple physical sensors, such as radar and acoustic sensors, to improve our knowledge of the physical environment. Recent changes in information fusion involve; (1) an interest in characterizing the human landscape as well as the physical landscape, (2) use of humans as observers or "soft" sensors, (3) explicit use of human pattern recognition via advanced visualization and sonficiation, and (4) use of multiple humans for dynamic, ad hoc, collaboration for analysis. This paper describes these new trends and describes on-going research at the Pennsylvania State University in all of these areas.

In this paper, we propose a distributed predator aware situation assessment system (PORTENT) to model and detect potential events occurring within an uncertain environment. PORTENT draws inspiration from how the mammalian brain detects and makes rational decisions through assessing fragmented signals of threat, within uncertainty, at different speeds. PORTENT represents the faster system using standard signal detection theory and the slower more accurate system as the integration of sensory data over time, until a certain level of confidence is reached. We also consider strategies to how both these systems could be combined optimally, to enhance PORTENT situation assessment performance. Our experimental simulations to verify the PORTENT concept demonstrates the effectiveness of our approach.

Situation Awareness (SA) is the perception of environmental elements within a volume of time and space, the comprehension of their meaning, and the projection of their future status. In a military environment the most critical elements to be tracked are followed elements are either friendly or hostile forces. Poor knowledge of locations of friendly forces easily leads into the situation in which the troops could be under firing by own troops or in which decisions in a command and control system are based on incorrect tracking. Thus the Friendly Force Tracking (FFT) is a vital part of building situation awareness. FFT is basically quite simple in theory; collected tracks are shared through the networks to all troops. In real world, the situation is not so clear. Poor communication capabilities, lack of continuous connectivity n and large number of user on different level provide high requirements for FFT systems. In this paper a simple architecture for Friendly Force Tracking is presented. The architecture is based on NFFI (NATO Friendly Force Information) hubs which have two key features; an ability to forward tracking information and an ability to convert information into the desired format. The hub based approach provides a lightweight and scalable solution, which is able to use several types of communication media (GSM, tactical radios, TETRA etc.). The system is also simple to configure and maintain. One main benefit of the proposed architecture is that it is independent on a message format. It communicates using NFFI messages, but national formats are also allowed.

From the advances in computer vision methods for the detection, tracking and recognition of objects in video streams, new opportunities for video surveillance arise: In the future, automated video surveillance systems will be able to detect critical situations early enough to enable an operator to take preventive actions, instead of using video material merely for forensic investigations. However, problems such as limited computational resources, privacy regulations and a constant change in potential threads have to be addressed by a practical automated video surveillance system. In this paper, we show how these problems can be addressed using a task-oriented approach. The system architecture of the task-oriented video surveillance system NEST and an algorithm for the detection of abnormal behavior as part of the system are presented and illustrated for the surveillance of guests inside a video-monitored building.

Cyberspace is an indispensable part of the economy and society, but has been "polluted" with many compromised computers that can be abused to launch further attacks against the others. Since it is likely that there always are compromised computers, it is important to be aware of the (dynamic) cyber security-related situation, which is however challenging because cyberspace is an extremely large-scale complex system. Our project aims to investigate a theoretical framework for trustworthy cyber sensing. With the perspective of treating cyberspace as a large-scale complex system, the core question we aim to address is: What would be a competent theoretical (mathematical and algorithmic) framework for designing, analyzing, deploying, managing, and adapting cyber sensor systems so as to provide trustworthy information or input to the higher layer of cyber situation-awareness management, even in the presence of sophisticated malicious attacks against the cyber sensor systems?

Because sensor networks are often deployed in hostile environments where their security and integrity may be compromised, it is essential to maximize the reliability and trustworthiness of existing and envisioned sensor networks. During operations, the sensor network must be robust to deception, node compromise, and various other attacks, while maintaining the operator's situational awareness regarding the health and integrity of the system. To address these needs, we have designed a Framework to Ensure and Assess Trustworthiness in Sensor systems (FEATS) to identify attacks on sensor system integrity and inform the operator of sensor data trustworthiness. We have developed and validated unsupervised anomaly detection algorithms for sensor data captured from an experimental acoustic sensor platform under a number of attack scenarios. The platform, which contains four audio microphones, was exposed to two physical attacks (audio filtering and audio playback) as well as a live replay attack (replaying live audio data that is captured at a remote location), which is analogous to a wormhole attack in the routing layer. With our unsupervised learning algorithms, we were able to successfully identify the presence of various attacks.

Traditional Intrusion Detection and Prevention (IDP) systems scan packets quickly by applying simple byte-wise pattern signatures to network flows. Such a protocol-agnostic approach can be compromised with polymorphic attacks: slight modifications of exploits that bypass pattern signatures but still reach corresponding vulnerabilities. To protect against these attacks, a solution is to provision the IDP system with protocol awareness, at the risk of degrading performance. To balance vulnerability coverage against network performance, we introduce a hardware-aware, compiler-based platform that leverages hardware engines to accelerate the core functions of protocol parsing and protocol-aware signature evaluation.

In recent years the internet has facilitated an explosion of growth in social networks, allowing individuals to interact with one another in a variety of different contexts. Interactions between individuals in networks such as twitter and NASDAQ produce events which co-occur in time. If we make the assumption that events in networks are anonymized such that there is no mapping from the event back to the individual who produced it, we are left with a data stream consisting of spatially and temporally interleaved events with no attribution. We model this property of event co-occurrence in order to recreate this mapping by assuming a strong coupling between temporal co-occurrence and spatial variance an arbitrary individual's behavior. We present a few algorithms based on this model, which produce partitions of tracks, where each track is indicative of the behaviors from a single individual in the network. Results using the algorithms indicate that the models are valid showing a high degree of spatio-temporal consistency among behaviors in networks. This suggests the need for further exploration of new behavior models and algorithms centered around this property.

The future smart grid will enable cost savings and lower energy use by means of smart appliances and smart meters which support dynamic load management and real-time monitoring of energy use and distribution. The introduction of two-way communications and control into power grid introduces security and privacy concerns. This talk will survey the security and privacy issues in smart grids using the NIST reference model, and relate these issues to cyber security in the Internet.

Two primary elements of the US energy policy are demand management and efficiency and renewable sources. Major objectives are clean energy transmission and integration, reliable energy transmission, and grid cyber security. Development of the Smart Grid seeks to achieve these goals by lowering energy costs for consumers, achieving energy independence and reducing greenhouse gas emissions. The Smart Grid is expected to enable real time wide-area situation awareness (SA) for operators. Requirements for wide-area SA have been identified among interoperability standards proposed by the Federal Energy Regulatory Commission and the National Institute of Standards and Technology to ensure smart-grid functionality. Wide-area SA and enhanced decision support and visualization tools are key elements in the transformation to the Smart Grid. This paper discusses human factors research to promote SA in the electric power grid and the Smart Grid. Topics that will be discussed include the role of human factors in meeting US energy policy goals, the impact and challenges for Smart Grid development, and cyber security challenges.

Problem of crack detection has attracted the attention of several investigators in the areas like defense, aeronautics, and marine industries. In this paper we suggest fuzzy logic approach for detection of cracks and also deciding about the severity of the crack. The data obtained from data acquisition system is processed and results presented by using various software. Fuzzy rules are developed to determine the severity of the crack and a light controller used to indicate the severity of the crack. The simplicity of the approach makes it very useful in many fields.

Spatio-temporal reasoning plays a significant role in situation management that is performed by intelligent agents (human or machine) by affecting how the situations are recognized, interpreted, acted upon or predicted. Many definitions and formalisms for the notion of spatio-temporal reasoning have emerged in various research fields including psychology, economics and computer science (computational linguistics, data management, control theory, artificial intelligence and others). In this paper we examine the role of spatio-temporal reasoning in situation management, particularly how to resolve situations that are described by using spatio-temporal relations among events and situations. We discuss a model for describing context sensitive temporal relations and show have the model can be extended for spatial relations.

In this paper we discuss issues in testing various cognitive fusion algorithms for situation management. We provide a proof-of-principle discussion and demo showing how gaming technologies and platforms could be used to devise and test various fusion algorithms, including input, processing, and output, and we look at how the proof-of-principle could lead to more advanced test beds and methods for high-level fusion in support of situation management. We develop four simple fusion scenarios and one more complex scenario in which a simple rule-based system is scripted to govern the behavior of battlespace entities.

Detection of intrusions is a continuing problem in network security. Due to the large volumes of data recorded in Web server logs, analysis is typically forensic, taking place only after a problem has occurred. This paper describes a novel method of representing Web log information through multi-channel sound, while simultaneously visualizing network activity using a 3-D immersive environment. We are exploring the detection of intrusion signatures and patterns, utilizing human aural and visual pattern recognition ability to detect intrusions as they occur. IP addresses and return codes are mapped to an informative and unobtrusive listening environment to act as a situational sound track of Web traffic. Web log data is parsed and formatted using Python, then read as a data array by the synthesis language SuperCollider [1], which renders it as a sonification. This can be done either for the study of pre-existing data sets or in monitoring Web traffic in real time. Components rendered aurally include IP address, geographical information, and server Return Codes. Users can interact with the data, speeding or slowing the speed of representation (for pre-existing data sets) or "mixing" sound components to optimize intelligibility for tracking suspicious activity.

The current trend in high-accuracy aircraft navigation systems is towards using data from one or more inertial navigation subsystem and one or more navigational reference subsystems. The enhancement in fault diagnosis and detection is achieved via computing the minimum mean square estimate of the aircraft states using, for instance, Kalman filter method. However, this enhancement might degrade if the cause of a subsystem fault has some effect on other subsystems that are calculating the same measurement. One instance of such case is the tragic incident of Air France Flight 447 in June, 2009 where message transmissions in the last moment before the crash indicated inconsistencies in measured airspeed as reported by Airbus. In this research, we propose the use of mathematical aircraft model to work out the current states of the airplane and in turn, using these states to validate the readings of the navigation equipment throughout smart diagnostic decision tree network. Various simulated equipment failures have been introduced in a controlled environment to proof the concept of operation. The results have showed successful detection of the failing equipment in all cases.

Persistent surveillance is an intricate process requiring monitoring, gathering, processing, tracking, and characterization of many spatiotemporal events occurring concurrently. Data associated with events can be readily attained by networking of hard (physical) sensors. Sensors may have homogeneous or heterogeneous (hybrid) sensing modalities with different communication bandwidth requirements. Complimentary to hard sensors are human observers or "soft sensors" that can report occurrences of evolving events via different communication devices (e.g., texting, cell phones, emails, instant messaging, etc.) to the command control center. However, networking of human observers in ad-hoc way is rather a difficult task. In this paper, we present a Twitter web-service for soft agent reporting in persistent surveillance systems (called Web-STARS). The objective of this web-service is to aggregate multi-source human observations in hybrid sensor networks rapidly. With availability of Twitter social network, such a human networking concept can not only be realized for large scale persistent surveillance systems (PSS), but also, it can be employed with proper interfaces to expedite rapid events reporting by human observers. The proposed technique is particularly suitable for large-scale persistent surveillance systems with distributed soft and hard sensor networks. The efficiency and effectiveness of the proposed technique is measured experimentally by conducting several simulated persistent surveillance scenarios. It is demonstrated that by fusion of information from hard and soft agents improves understanding of common operating picture and enhances situational awareness.

In most organizations, IT (information technology) infrastructure exists to support the organization's mission. The threat of cyber attacks poses risks to this mission. Current network security research focuses on the threat of cyber attacks to the organization's IT infrastructure; however, the risks to the overall mission are rarely analyzed or formalized. This connection of IT infrastructure to the organization's mission is often neglected or carried out ad-hoc. Our work bridges this gap and introduces analyses and formalisms to help organizations understand the mission risks they face from cyber attacks. Modeling an organization's mission vulnerability to cyber attacks requires a description of the IT infrastructure (network model), the organization mission (business model), and how the mission relies on IT resources (correlation model). With this information, proper analysis can show which cyber resources are of tactical importance in a cyber attack, i.e., controlling them enables a large range of cyber attacks. Such analysis also reveals which IT resources contribute most to the organization's mission, i.e., lack of control over them gravely affects the mission. These results can then be used to formulate IT security strategies and explore their trade-offs, which leads to better incident response. This paper presents our methodology for encoding IT infrastructure, organization mission and correlations, our analysis framework, as well as initial experimental results and conclusions.

Broadband wireless access standards, together with advances in the development of commercial sensing and actuator devices, enable the feasibility of a consumer service for a multi-sensor system that monitors the conditions within a residence or office: the environment/infrastructure, patient-occupant health, and physical security. The proposed service is a broadband reimplementation and combination of existing services to allow on-demand reports on and management of the conditions by remote subscribers. The flow of on-demand reports to subscribers and to specialists contracted to mitigate out-of-tolerance conditions is the foreground process. Service subscribers for an over-the-horizon connected home/office (OCHO) monitoring system are the occupant of the premises and agencies, contracted by the service provider, to mitigate or resolve any observed out-of-tolerance condition(s) at the premises. Collectively, these parties are the foreground users of the OCHO system; the implemented wireless standards allow the foreground users to be mobile as they request situation reports on demand from the subsystems on remote conditions that comprise OCHO via wireless devices. An OCHO subscriber, i.e., a foreground user, may select the level of detail found in on-demand reports, i.e., the amount of information displayed in the report of monitored conditions at the premises. This is one context of system operations. While foreground reports are sent only periodically to subscribers, the information generated by the monitored conditions at the premises is continuous and is transferred to a background configuration of servers on which databases reside. These databases are each used, generally, in non-real time, for the assessment and management of situations defined by attributes like those being monitored in the foreground by OCHO. This is the second context of system operations. Context awareness and management of conditions at the premises by a second group of analysts and decision makers who extract information from the OCHO data in the databases form the foundation of the situation management problem.

Providing current, time-sensitive imagery and geospatial information to deployed tactical military forces or first responders continues to be a challenge. This challenge is compounded through rapid increases in sensor collection volumes, both with larger arrays and higher temporal capture rates. Focusing on the needs of these military forces and first responders, ITT developed a system called AGILE (Advanced Geospatial Imagery Library Enterprise) Access as an innovative approach based on standard off-the-shelf techniques to solving this problem. The AGILE Access system is based on commercial software called Image Access Solutions (IAS) and incorporates standard JPEG 2000 processing. Our solution system is implemented in an accredited, deployable form, incorporating a suite of components, including an image database, a web-based search and discovery tool, and several software tools that act in concert to process, store, and disseminate imagery from airborne systems and commercial satellites. Currently, this solution is operational within the U.S. Government tactical infrastructure and supports disadvantaged imagery users in the field. This paper presents the features and benefits of this system to disadvantaged users as demonstrated in real-world operational environments.

With particular focus on distributed situational awareness in defense and security applications, we propose a value-ofinformation based middleware framework as a paradigm shift from crawl-index based centralized search. The proposed approach seeks to improve efficiency in search and retrieval by transforming sensors into programmable information points and enabling ubiquitous data and software flows through the infrastructure of the distributed network. We explain the different components of such a middleware framework to organize, tag and notify emerging spatial, temporal and causal patterns from the sensor measurements. We conclude the paper with a brief discussion on the top-down programming model that can realize the framework as a reconfigurable sensor query system.

Deploying Video Analytics in operational environments is extremely challenging. This paper presents a methodological approach developed by the Video Surveillance and Biometrics Section (VSB) of the Science and Engineering Directorate (S&E) of the Canada Border Services Agency (CBSA) to resolve these problems. A three-phase approach to enable VA deployment within an operational agency is presented and the Video Analytics Platform and Testbed (VAP/VAT) developed by the VSB section is introduced. In addition to allowing the integration of third party and in-house built VA codes into an existing video surveillance infrastructure, VAP/VAT also allows the agency to conduct an unbiased performance evaluation of the cameras and VA software available on the market. VAP/VAT consists of two components: EventCapture, which serves to Automatically detect a "Visual Event", and EventBrowser, which serves to Display & Peruse of "Visual Details" captured at the "Visual Event". To deal with Open architecture as well as with Closed architecture cameras, two video-feed capture mechanisms have been developed within the EventCapture component: IPCamCapture and ScreenCapture.

Unveiling unusual or hostile events by observing manifold moving persons in a crowd is a challenging task for human operators, especially when sitting in front of monitor walls for hours. Typically, hostile events are rare. Thus, due to tiredness and negligence the operator may miss important events. In such situations, an automatic alarming system is able to support the human operator. The system incorporates a processing chain consisting of (1) people tracking, (2) event detection, (3) data retrieval, and (4) display of relevant video sequence overlaid by highlighted regions of interest. In this paper we focus on the event detection stage of the processing chain mentioned above. In our case, the selected event of interest is the encounter of people. Although being based on a rather simple trajectory analysis, this kind of event embodies great practical importance because it paves the way to answer the question "who meets whom, when and where". This, in turn, forms the basis to detect potential situations where e.g. money, weapons, drugs etc. are handed over from one person to another in crowded environments like railway stations, airports or busy streets and places etc.. The input to the trajectory analysis comes from a multi-object video-based tracking system developed at IOSB which is able to track multiple individuals within a crowd in real-time [1]. From this we calculate the inter-distances between all persons on a frame-to-frame basis. We use a sequence of simple rules based on the individuals' kinematics to detect the event mentioned above to output the frame number, the persons' IDs from the tracker and the pixel coordinates of the meeting position. Using this information, a data retrieval system may extract the corresponding part of the recorded video image sequence and finally allows for replaying the selected video clip with a highlighted region of interest to attract the operator's attention for further visual inspection.

To move from data to information in almost all science and defense applications requires a human-in-the-loop to validate information products, resolve inconsistencies, and account for incomplete and potentially deceptive sources of information. This is a key motivation for visual analytics which aims to develop techniques that complement and empower human users. By contrast, the vast majority of algorithms developed in machine learning aim to replace human users in data exploitation. In this paper we describe a recently introduced machine learning problem, called rare category detection, which may be a better match to visual analytic environments. We describe a new design criteria for this problem, and present comparisons to existing techniques with both synthetic and real-world datasets. We conclude by describing an application in broad-area search of remote sensing imagery.

In time-sensitive environments, such as DHS emergency operations centers (EOCs), it is imperative for decision makers to rapidly understand and address key logical relationships that exist between tasks, entities, and events, even as conditions fluctuate. These relationships often have important temporal characteristics, such as tasks that must be completed before others can be started (e.g., buses must be transported to an area before an evacuation process can begin). Unfortunately, traditional temporal display methods, such as mission timelines, typically reveal only rudimentary event details and fail to support user understanding of and reasoning about critical temporal constraints and interrelationships across multiple mission components. To address these shortcomings, we developed a visual language to enhance temporal data displays by explicitly and intuitively conveying these constraints and relationships to decision makers. In this paper, we detail these design strategies and describe ongoing evaluation efforts to assess their usability and effectiveness to support decision-making tasks in complex, time-sensitive environments. We present a case study in which we applied our visual enhancements to a timeline display, improving the perception of logical relationships among events in a Master Scenario Event List (MSEL). These methods reduce the cognitive workload of decision makers and improve the efficacy of identification.

A typical approach to exploring Light Detection and Ranging (LIDAR) datasets is to extract features using pre-defined segmentation algorithms. However, this approach only provides a limited set of features that users can investigate. To expand and represent the rich information inside the LIDAR data, we introduce a linked feature space concept that allows users to make regular, conjunctive, and disjunctive discoveries in non-uniform LIDAR data by interacting with multidimensional transfer functions. We achieve this by providing interactions for creating multiple scatter-plots of varying axes, establishing chains of plots based on selection domains, linking plots using logical operators, and viewing selected brushing results in both a 3D view and selected scatter-plots. Our highly interactive approach to visualizing LIDAR feature spaces facilitates the users' ability to explore, identify, and understand data features in a novel way. Our approach for exploring LIDAR data can directly lead to better understanding of historical LIDAR datasets, and increase the turnaround time and quality of results from time-critical LIDAR collections after urban disasters or on the battlefield.

In this paper we describe multidisciplinary experimental research concentrated on stereoscopic presentation of geospatial imagery data obtained from various sensors. Source data were different in scale, texture, geometry and content. None of image processing techniques allows processing such a data simultaneously. However, augmented reality system allows subjects to fuse multi-sensor, multi-temporal data and terrain reality into single model. Augmented reality experimental set, based on head-mounted display was designed to efficiently superimpose LIDAR point-clouds for comfortable stereoscopic perception. Practical research experiment performed indicates feasibility of the stereoscopic perception data obtained on-the-fly. One of the most interesting findings is that source LIDAR point-clouds do not have to be preprocessed or enhanced for being in the experiments described.

In this work, we propose new automation tools to process 2D building geometry data for effective communication and timely response to critical events in commercial buildings. Given the scale and complexity of commercial buildings, robust and visually rich tools are needed during an emergency. Our data processing pipeline consists of three major components, (1) adjacency graph construction, representing spatial relationships within a building (between hallways, offices, stairways, elevators), (2) identification of elements involved in evacuation routes (hallways, stairways), (3) 3D building network construction, by connecting the oor elements via stairways and elevators. We have used these tools to process a cluster of five academic buildings. Our automation tools (despite some needed manual processing) show a significant advantage over manual processing (a few minutes vs. 2-4 hours). Designed as a client-server model, our system supports analytical capabilities to determine dynamic routing within a building under constraints(parts of the building blocked during emergencies, for instance). Visualization capabilities are provided for easy interaction with the system, on both desktop (command post) stations as well as mobile hand-held devices, simulating a command post-responder scenario.

Visualization tools for cyber security often overlook related research from the information visualization domain. Cyber security data sets are notoriously large, yet many of the popular analysis tools use 3D techniques and parallel coordinates which have been shown to suffer issues of occlusion when applied to large data sets^1,2. While techniques exist to ameliorate these issues they are typically not used. In this paper we evaluate several cyber security visualization tools based on established design principles and human-computer interaction research. We conclude by enumerating challenges, requirements, and recommendations for future work.

To address the unique requirements of cyber Command and Control (C2), new visualization methods are needed to provide situation awareness and decision support within the cyber domain. A key challenge is the complexity of relevant data: it is immense and multidimensional, includes streaming and log data, and comes from multiple, disparate applications and devices. Decision makers must be afforded a view of a) the current state of the cyber battlespace, b) enemy and friendly capabilities and vulnerabilities, c) correlations between cyber events, and d) potential effects of alternative courses of action within cyberspace. In this paper we present requirements and designs for Visualization for Integrated Cyber Command and Control (VIC3).

Maintaining Situational Awareness (SA) is crucial to the success of high tempo operations, such as war fighting and mass casualty events (bioterrorism, natural disasters). Modern computer and software applications attempt to provide command and control manager's situational awareness via the collection, integration, interrogation and display of vast amounts of analytic data in real-time from a multitude of data sources and formats [1]. At what point does the data volume and displays begin to erode the hierarchical distributive intelligence, command and control structure of the operation taking place? In many cases, people tasked with making decisions, have insufficient experience in SA of high tempo operations and become overwhelmed easily as vast amounts of data begin to be displayed in real-time as an operation unfolds. In these situations, where data is plentiful and the relevance of the data changes rapidly, there is a chance for individuals to target fixate on those data sources they are most familiar. If these individuals fall into this type of pitfall, they will exclude other data that might be just as important to the success of the operation. To counter these issues, it is important that the computer and software applications provide a means for prompting its users to take notice of adverse conditions or trends that are critical to the operation. This paper will discuss a new method of displaying data called a Crisis View^TM, that monitors critical variables that are dynamically changing and allows preset thresholds to be created to prompt the user when decisions need to be made and when adverse or positive trends are detected. The new method will be explained in basic terms, with examples of its attributes and how it can be implemented.