Putting Statistics Into Forensic Firearms Identification

How good a match is it?

Published: Monday, February 26, 2018 - 13:01

On February 14, 1929, gunmen working for Al Capone disguised themselves as police officers, entered the warehouse of a competing gang, and shot seven of their rivals dead. The St. Valentine’s Day Massacre is famous not only in the annals of gangland history, but also in the history of forensic science. Capone denied involvement, but an early forensic scientist named Calvin Goddard linked bullets from the crime scene to Tommy guns found at the home of one of Capone’s men. Although the case never made it to trial—and Capone’s involvement was never proved in a court of law—media coverage introduced millions of readers to Goddard and his strange-looking microscope.

That microscope had a split screen that allowed Goddard to compare bullets or cartridge cases, the metal cases a gun ejects after firing a bullet, side by side. If markings on the bullets or cases matched, that indicated that they were fired from the same gun. Firearms examiners still use that same method today, but it has an important limitation: After visually comparing two bullets or cartridge cases, the examiner can offer an expert opinion as to whether they match. But they cannot express the strength of the evidence numerically, the way a DNA expert can when testifying about genetic evidence.

This is not Calvin Goddard. Wilmer Souder, a physicist and early forensic scientist at the National Bureau of Standards, now NIST, compares two bullets using a comparison microscope. Souder learned forensic techniques from Calvin Goddard, another early scientist in the field. Credit: Photo by NBS/NIST; source: NARA

Now, a team of researchers at the National Institute of Standards and Technology (NIST) has developed a statistical approach for ballistic comparisons that may enable numerical testimony. While other research groups are also working on this problem, the advantages of the NIST approach include a low error rate in initial tests and that it is relatively easy to explain to a jury. The researchers described their approach in Forensic Science International.

When comparing two cartridge cases, the NIST method produces a numerical score that describes how similar they are. It also estimates the probability that random effects might cause a false positive match—a concept similar to match probabilities for DNA evidence.

“No scientific method has a zero error rate,” says John Song, a NIST mechanical engineer and the lead author of the study. “Our goal is to give the examiner a way to estimate the probability of this type of error so the jury can take that into account when deciding guilt or innocence.”

The new approach also seeks to transform firearm identification from a subjective method that depends on an examiner’s experience and judgement to one that is based on objective measurements. A landmark 2009 report from the National Academy of Sciences and a 2016 report from the President’s Council of Advisors on Science and Technology both called for research that would bring about this transformation.

The theory behind forensic ballistics

When a gun is fired, and the bullet blasts down the barrel, it encounters ridges and grooves that cause it to spin, increasing the accuracy of the shot. Those ridges dig into the soft metal of the bullet, leaving striations. At the same time that the bullet explodes forward, the cartridge case explodes backward with equal force against the mechanism that absorbs the recoil, called the breech face. This stamps an impression of the breech face into the soft metal at the base of the cartridge case, which is then ejected from the gun.

The theory behind firearm identification is that microscopic striations and impressions left on bullets and cartridge cases are unique, reproducible, and therefore, like “ballistic fingerprints” that can be used to identify a gun. If investigators recover bullets or cartridge cases from a crime scene, forensic examiners can test-fire a suspect's gun to see if it produces ballistic fingerprints that match the evidence.

But bullets and cartridge cases that are fired from different guns might have similar markings, especially if the guns were consecutively manufactured. This raises the possibility of a false positive match, which can have serious consequences for the accused.

A fired bullet with rifling impressions from the barrel of a gun (left). A fired cartridge case and fired bullet (right). Experts can often identify the weapon used based on rifling impressions on the bullet or impressions on the primer (the silver-colored metal) at the base of the cartridge case. Credit: Robert Thompson/NIST

A statistical approach

In 2013, Song and his NIST colleagues developed an algorithm that compares three-dimensional surface scans of the breech face impressions on cartridge cases. Their method, called Congruent Matching Cells, or CMC, divides one of the scanned surfaces into a grid of cells, then searches the other surface for matching cells. The greater the number of matching cells, the more similar the two surfaces, and the more likely they are to have come from the same gun.

In their recent study, the researchers scanned 135 cartridge cases that were fired from 21 different 9-millimeter pistols. This produced 433 matching image pairs and 4,812 non-matching pairs. To make the test even more difficult, most of the pistols were consecutively manufactured.

The CMC algorithm classified all the pairs correctly. Furthermore, almost all the non-matching pairs had zero matching cells, with a handful having one or two due to random effects. All the matching pairs, on the other hand, had at least 18 matching cells. In other words, the matching and non-matching pairs fell into highly separated distributions based on the number of matching cells.

“That separation indicates that the probability of random effects causing a false positive match using the CMC method is very low,” says coauthor and physicist Ted Vorburger.

Typical results for a comparison of breech face impressions on cartridge case primers, using the NIST technique known as Congruent Matching Cells, or CMC. In pair A, almost all the cells from the first image match cells from the second image, indicating that the two cartridge cases were likely fired by the same gun. In pair B, some cells find similar cells, but they are randomly distributed, and therefore, not considered matching. Only the area of interest for each primer is shown. Portions of the primer surface that were not compared appear in white. The color scale indicates relative surface height in micrometers. Click for larger image. Credit: Johannes Soons/NIST

A better way to testify

Using well-established statistical methods, the authors built a model for estimating the likelihood that random effects would cause a false positive match. Using this method, a firearms expert would be able to testify about how closely the two cartridges match based on the number of matching cells, and also the probability of a random match, similar to the way forensic experts testify about DNA.

Although this study did not include enough test-fires to calculate realistic error rates for actual casework, the study has demonstrated the concept. “The next step is to scale up with much larger and more diverse datasets,” say Johannes Soons, a NIST mechanical engineer and coauthor of the study.

With more diverse datasets, researchers will be able to create separate models for different types of guns and ammunition. That would make it possible to estimate random match rates for the various combinations that might be used in a crime.

Other groups of researchers are working on ways to express the strength of evidence numerically, not only for firearms but also fingerprints and other types of pattern evidence. Many of those efforts use machine learning and artificial intelligence-based algorithms to compare patterns in the evidence. But it can be difficult to explain how machine-learning algorithms work.

“The CMC method can be easily explained to a jury,” says Song. “It also appears to produce very low false-positive error rates.”

About The Author

NIST

Founded in 1901, the National Institute of Standards and Technology (NIST) is a nonregulatory federal agency within the U.S. Department of Commerce. Headquartered in Gaithersburg, Maryland, NIST’s mission is to promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve our quality of life.