Share Email Print
cover

Proceedings Paper

Evaluation of statistical methods for the evaluation of observer trials for the assessment of the effectiveness of signature measures
Author(s): Patrick Dunau; Daniel Fitz; Karin U. Stein
Format Member Price Non-Member Price
PDF $14.40 $18.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

The statistical methods discussed in this paper are drawn from the area of machine learning or data mining as well as from descriptive statistics. These techniques are discussed with focus on their applicability to the results of observer trials in order to evaluate the effectiveness of signature measures. Signature measures aim at the change of the apparent signature of an object, e.g. a vehicle. So signature measures can be camouflage against infrared sensory, or they can be used for deception reasons. In order to evaluate the effectiveness of signature measures, observer trials provide an efficient method. The department of Signatorics of Fraunhofer IOSB developed a software tool named CARPET (Computer Aided inteRactive Performance Evaluation Tool) for the realization of observer trials. The benefit of this system is the reproducibility and uniformity of trials for every observer. The results from this system consist of marks, that were placed at particular times, as well as computer mouse positions recorded for each human observer. Based on the information gathered from these marks together with the known target object positions the statistical treatment can be done. For the statistics it has to be known to which target object the marks belong. The first problem considered in this paper concentrates on the correct labeling of the marks according to the target objects. The labeling is done using an expectation maximization scheme with the k-means clustering algorithm. The next step involves a second labeling. In this step a linear discriminant is used to decide whether a mark should be considered a hit or miss for every particular target object. After these decisions, a receiver-operating characteristics (ROC) analysis is performed in order to evaluate the detectability of each target object. Furthermore the sample mean and sample covariance formulas are used on the so called hit sets in order to approximate Gaussian distributions for every hit set. These Gaussians facilitate the evaluation of the accuracy and the precision of the hit sets. Accuracy and precision offer information about the quality of the marks set by the observers.

Paper Details

Date Published: 21 October 2015
PDF: 9 pages
Proc. SPIE 9653, Target and Background Signatures, 96530T (21 October 2015); doi: 10.1117/12.2194088
Show Author Affiliations
Patrick Dunau, Fraunhofer-Institut für Optronik, Systemtechnik und Bildauswertung (Germany)
Daniel Fitz, Fraunhofer-Institut für Optronik, Systemtechnik und Bildauswertung (Germany)
Karin U. Stein, Fraunhofer-Institut für Optronik, Systemtechnik und Bildauswertung (Germany)


Published in SPIE Proceedings Vol. 9653:
Target and Background Signatures
Karin U. Stein; Ric H. M. A. Schleijpen, Editor(s)

© SPIE. Terms of Use
Back to Top