Firearms identification through partonomy

Identifying partially occluded (hidden) objects is necessary in security or intelligence systems when visually monitoring an area for weapons and assessing the level of threat they pose. In real-life situations, such as when a weapon is held in the hand, over the shoulder, or in a suitcase or bag, some of the weapon will necessarily be occluded. Existing automated schemes for weapon detection have the disadvantage of relying on active scanning of co-operative subjects in a controlled environment. For instance, millimeter-wave scanners require the subject to assume and hold a fixed position. At a cost of approximately $170,000 each and a false positive rate approaching 25%, such systems address a different threat scenario from our methodology, which consists of ubiquitous high-definition cameras costing less than $50. However, automating recognition of the whole by a part is a very challenging problem. For example, recognizing an AK-47 assault rifle from the unique shape of its individual components (e.g., muzzle, gas-piston, safety lever, or curved magazine) is a non-trivial exercise even for a human: see Figure 1. We have introduced a new approach in which individual parts of a weapon are considered as functional equivalents of the entire weapon.

Figure 1. (a) An image of an AK47 along with part of its boundary, and the core of the weapon. (b) The boundary part without the magazine and the core of this part (the inscribed polygon).

Our approach links conceptual and visual hierarchies of the ontology (partial model of the world) to conduct semantic analysis using the results of a visual search.^1–3 The conceptual ontology is augmented with a meronymic component: see Figure 2. Meronymy, also called partonymy, is a central organizing principle of most ontologies defining of which parts an object is composed. (A meronym is a constituent part.) Meronymic relations (part-to-whole, whole-to-part) satisfy transitive but not reflective or symmetric properties.

Figure 2. Partial meronomy (i.e., the hierarchial structure of concepts that have meronymic relationships) showing the AK47 and how it is related to its parts via the has-object-as-part (‘hoap’) relation and to the weapon part types and weapon type ontologies via the ‘isa’ relation, the two essential relations for the current project.

Meronymic (partonymic) principles let us distinguish functionally defined parts of a gun, for example, as well as parts that are essential to a gun (trigger, barrel) from parts that are not essential (stock, scope): see Figure 2. These distinctions help us to assess the threat posed by a disassembled or assembled gun, as well as to identify a partially occluded gun in an image. Adding classes of meronymic information provides additional metadata (useful for threat assessment) and allows for fast and accurate search of a weapon by using visible parts as a query.

We updated the visual ontology to include firearms' geometrical parts and their descriptors (used as partonomy node labels). For this purpose, we performed automatic image segmentation and semi-automatic weapon decomposition to extract the boundary and the convex hull (CH) of the weapons' parts to augment the existing visual and conceptual hierarchies to partonyms.

To segment an image to background and objects, we applied a shrinking active contour (S-ACES).^{2, 3} A drawback of this approach is that overcrowded images degrade its performance. To overcome the problem and extract multiple objects, we employed a mathematical morphology approach, but this has the drawback that too many, and sometimes unnecessary, details are extracted, which slows the process.

Automatic partitioning of objects is a hot topic with applications in computer graphics, vision, and recognition, yet despite intensive research efforts, no solid breakthrough has been made. Some methods determine the largest convex object (‘pilled potato’) inscribed in another object.⁴ Others detect the ‘longest stick’ in an object.⁴ To split an object into central and peripheral geometric parts, we applied an expanding active contour, whose initial point is selected by the user.⁵

We created visual and conceptual hierarchy trees from the firearm partonymy and labeled them using boundary- and CH-defined numerical sequences. We generated one sequence for the part's boundary and composed another by that part's CH.

We generated a number of visual hierarchies and determined the one closest to the conceptual by calculating tree distances.^{1, 2} We extended every leaf of the visual hierarchy by attaching a node for each geometric part of a weapon, and did the same for the conceptual hierarchy by attaching a node for every functional weapon part seen. The two kinds of nodes are linked offline by an expert. The links facilitate weapon identification by visual parts, and metadata extraction, which is used to assess the threat with a formula introduced elsewhere.^{1, 2}

Essentially, we treat searching for the parts of an object as the same problem of searching for the object itself,^{1, 2} and we make a probabilistic decision at the end of the parallel searches. Thus, if a given search identifies the presence of the butt, the stock, and the barrel of a rifle, we infer the presence of the trigger and trigger guard, and presume they are occluded by the hand of the shooter. Needless to say, special attention needs to be paid to the spatial arrangement of the parts (disassembled weapon vs. assembled weapon), especially when assessing threats.

Integrating the conceptual and visual partonomies also helps to prevent proliferation of potential parts. Any fraction of a partially occluded object could potentially be defined as a part of that object. In our partonomy, however, we added only whole parts of object, i.e., those that have been identified in the conceptual ontology as such. For example, the butt of a rifle is added to the ontology, but not a barrel with the first two inches occluded.

We currently have an ontology containing 153 weapons and 30 parts.^5,6 We performed experiments querying the visual partonomy with every weapon and with every part. The rate of success in the two cases was 100%. Also, for every query our method detected parts that are contained in the firearm. For each part, our method correctly detected firearms that contain the part.⁵ We have also queried the ontology with unknown weapons. Our method correctly detected the second ancestor in 70.5% of the cases.⁶(The first-level ancestor of a concept in an ontology is the immediate parent, while the second-level ancestor we call the grandparent. For example, in Figure 2 the first ancestor of ‘AK_47’ is ‘assault rifle.’ The second ancestor is not shown.) The runtime for a search is 0.2–0.7s on a 1.6GHz PC, while the image segmentation is about 0.095s on a 2.67GHz PC on images of 320×103 pixels. Our work continues with extending the existing partonomy and elaborating the methods for geometric weapons partitioning; matching and identification in the visual partonomy; and linking visual with conceptual partonomies.