What Sherlock Holmes would do in the digital age

Information forensics is a newly emerging investigative field concerned about all aspects of information processing. It involves the development of frameworks, algorithms, and methodologies for tracing traitors, protecting contents, detecting tampering, and analyzing components to protect intellectual rights. In addition, information forensics models and analyzes the behavior of users of multimedia social networks. As a result of recent advances in information technology, information assurance and forensics are now recognized as critical issues.

Information forensics is primarily focused on reconstructing what may have compromised a given information content and on investigating who may have done what, when, and how. Typical investigations are accordingly multidisciplinary, involving the interplay of signal processing, cryptology, communication and information theory, and game theory, as well as the psychology of human visual or auditory perception.

For example, performing multimedia forensics requires collecting evidence. Analogous to human fingerprints in crime forensics, digital domains also have fingerprints, either extrinsic or intrinsic. Extrinsic fingerprints are signals proactively and artificially inserted by system designers into multimedia data to facilitate the tracing of a recipient, identify the origin of the data, or detect changes and tampering. For instance, a unique identifier can be inserted in each copy of a given information to identify all receiving users. When copies are leaked or misused, an authority can use the embedded extrinsic fingerprint to trace the culprits. In this context, traitor tracing technology can now be used to address the threat of several users (colluders) who combine their copies of the same content embedded with different fingerprints to undermine the fingerprints (see Figure 1).¹ These multi-user attacks, known as collusion attacks, provide methods for removing an identifying fingerprint, and effectively threaten digital multimedia rights. To mitigate the serious threat posed by collusion, we have developed theories and algorithms for constructing forensic fingerprints that can resist collusion, identify colluders, and corroborate their guilt.^2–7

When no proactive protection is available, a forensics investigation can use the invisible traces left on the information contents as it cycles through operations and devices. These intrinsic fingerprints can provide powerful evidence concerning the history and provenance of a given digital content. We have also designed techniques that can identify components inside a visual device (such as a camera) solely from its output by inferring what algorithms or processing are employed and estimating their parameter settings. For example, we introduced a new methodology for forensic analysis of digital camera images based on the observation that color interpolation leaves distinct intrinsic traces on images. These intrinsic fingerprints can then be identified and used to verify the authenticity of the digital data. Using a detailed imaging model and applying component analysis techniques, we can determine which interpolation algorithm is being used, estimate the parameter settings, and thus determine the brand and model of the camera that took the picture. The classification accuracy rate is in the high 90% range.⁸

Figure 1. Fingerprint embedding and fingerprint extraction and detection ca overcome collusion attacks.

Our methodology can also detect tampering. This is because any change or inconsistencies among the estimated in-camera fingerprints—including the presence of new postcamera fingerprints—suggests that the image has undergone some sort of processing after the initial capture, such as tampering or steganographic embedding (see Figure 2).

Figure 2. Two photos (left) taken with different cameras are combined to form a new tampered image (upper right). Our tampering detection system can determine that the tampered image consists of two portions from different cameras (lower right).

Building upon component forensics knowledge, we can extend our non-intrusive methodology to address a number of larger forensic issues, such as discovering technology infringement and protecting intellectual property rights (infringement forensics), identifying the type and model of acquisition device (acquisition forensics), detecting a variety of contents tampering and verifying integrity (tampering forensics), and building universal detectors capable of detecting unseen and challenging steganography schemes (steganography forensics).⁹

We are also developing methodologies to study user dynamics in media security and provide a framework to model and analyze user dynamics.^10,11 The users of multimedia security systems often form complex social networks with different and often conflicting objectives that influence member decisions and performance. Behavior forensics seeks to define the complex dynamics among users, investigate how they interact with each other, and analyze how user dynamics affect multimedia security as well as forensics (see Figure 3).

Figure 3. Attackers and security investigators form special networks characterized by different types of behavior dynamics.

These type of investigations should significantly improve our understanding of multimedia security and forensics, while providing important guidelines for the systematic design of efficient media security systems.