Text: Forensic science is going through some problems. Illustration: Door of CSI Lab covered in yellow tape with words “field in crisis, do not trust.”
Text: On TV it looks fun and easy. Illustration: Two investigators look at photographic evidence, which appears in background showing: mug shot of woman named Scarface Diane, size 11 sandal and Birkenstock footprint. The male investigator listens as the female investigator says, “We picked her up at the farmer’s market.”
Text: But in real life the certainty behind the forensic scientific methods isn’t so certain
Text: Since 1992, at least 362 wrongfully convicted US prisoners have been cleared by the Innocent Project (www.innocenceproject.org). Nearly half involved bad forensic evidence. Illustration: Orange and brown silhouettes of prisoners in handcuffs.
Text: 96% involved scientifically invalid testimony … including 33 cases where the suspect had been sentenced to death. Illustration: Tall stack of papers in an inbox labeled “reviewed.”
Text: One man served nearly 30 years in prison for murder after prosecutors claimed there was only 1 chance in 10 million the crime scene hairs were not from him. Decades later, a DNA re-analysis exonerated him. Illustration: Prisoner in orange jumpsuit sitting on bed in cell with barred window and tally marks on wall.
Text: One of the hairs turned out to be from a dog. Illustration: Yellow dog, wearing black mask over eyes and striped black-and-white shirt, holds revolver with front paw.
Text: Forensic science experts sometimes get pretty confident in their work. Maybe too confident. Illustration: Green landscape with large billboard, showing male investigator in eyeglasses giving thumbs-up signal, and the words “Error rate: 0%” and “Yeeeeaaaaaah”; plane flies in sky pulling banner that reads “100% confidence”; and trash can filled with garbage including report card that reads “fingerprints, 5% errors; hair analysis, 11% errors; bite marks, 25% errors” and note that reads “estimates vary.”
Text: There have been lots of committees and reports. They all point to the need for better statistical foundations in forensic science. For example, when evidence isn’t 100% certain, how can we squeeze information out of it anyway? Illustration: Signs with names of groups and years: National Research Council, 2003, 2004, 2008 and 2009; National Commission on Forensic Science, 2013; Organization of Scientific Area Committees for Forensic Science, 2014; and President’s Council of Advisors on Science and Technology, 2016.
Text: Luckily, Probably and Statistics have powerful tools to help. Both are good at figuring out what’s known — and what’s not — and making conclusions with confidence. Illustration: Female superhero with red hair, blue superhero suit and red cape punches through wall while male superhero with green-black hair and similar outfit shows a screen with mathematical equation to two startled people.
Illustration: Two superheroes with word bubble. They are saying: “And we do it with math!”
Text: Statistician Hal Stern at the University of California, Irvine, and his colleagues have studied how one tool in particular can help crime experts. Illustration: Statistician in blue shirt and eyeglasses with word bubble. He is saying: “It’s called the Likelihood Ratio.” Footnote in side panel says: Stern, H.S. (2017). Statistical Issues in Forensic Science, Annual Review of Statistics and Its Application, 4, 225-244.
Text: It’s a very handy tool — mathematically sophisticated but also broadly applicable. Illustration: Animated golden balance scale with weight pan on each side. Left pan is labeled “P(Evidence| Defense Claim)” and right pan is labeled “P(Evidence|Prosecution Claim)”
Text: But first, let’s look hypothetically at how all this might go the old-fashioned way — without the Likelihood Ratio tool.
Text: We start by assuming, for the sake of argument, that the crime scene evidence is linked to the suspect. Illustration: Large green arrow points to sandals worn by woman with long hair, wearing red shirt and teal pants; large gloved green hand gives thumbs-up signal to indicate sandal matches inset photo of Birkenstock footprint.
Text: Then we look for enough apparent differences to negate that. Illustration: Blue computer screen with words “statistically different” and arrows pointing to patterned sole of sandal and to photo labeled “Birkenstock” that shows footprint with different pattern.
Text: In other words, when it comes to forensic evidence the onus can often be on the Defense to prove it’s not a match. Illustration: Defense lawyer in blue suit, red tie and red boxing gloves marked with letter “D” faces off clown bop bag labeled “prosecution.”
Text: Notice how the system was rigged? We set up a hypothesis that favors the Prosecution, then try to knock it down. Illustration: Defense lawyer in blue suit, red tie and red boxes gloves marked with letter “D” punches clown bop bag labeled “prosecution.”
Text: The problem: Real life is messy. Sometimes crime scene evidence is not from the suspect, in fact, but there just aren’t enough clues left to say much of anything. Illustration: Two investigators look at a photo of bird, footprint, cigarette butt and leaf. One has a word bubble, saying: “There were pigeons.”
Text: And in that case, it would be the Prosecution’s argument that’s left standing by default. Illustration: Defense lawyer in blue suit, red tie and red boxing gloves is hunched over, sweaty and tired, while smiling clown bop bag labeled “prosecution” stands upright.
Text: But “presumed guilty until proven innocent” is not the idea our legal system was built on. Illustration: A colonial American man writes with a feather quill onto a scroll of paper.
Text: A better method? Set up BOTH the Prosecutors’ and Defenses hypothesis — and then see how strong the evidence is for or against EACH of them. Illustration: Two lawyers in suits and boxing gloves spar with clown-shaped bop bags that keep bouncing back; one is defense lawyer boxing “prosecution” bop bag, and other is prosecutor boxing “defense” bop bag.
Text: Here’s how the Likelihood Ratio can do this. (With a little help from Probability and Statistics.) It considers each side and weighs them independently. Illustration: Two superheroes in blue and orange suits and red capes, with the letters S and P on their chests that stand for Statistics and Probability, stand on either side of a golden balance scale with weight pans labeled “prosecution” and “defense.”
Text: Because they’re two separate questions.
Illustration: Prosecutor in blue suit and eyeglasses with word bubble. She is saying: “Assume the Defense is right — that the evidence was not from the suspect. Now … how surprising is it to see evidence like this?”
Text: That is, what are the chances of a match this good from a random person in the relevant population? Illustration: Forensic scientist in white lab coat, shrugging with palms up, surrounded by question marks and various footprints. In word bubble, she is saying: “Hmm, size 11 Birkenstocks in this town? Pretty common. Not too surprising.”
Text: The more surprising it would be to find evidence like this if the Defense were right … the greater the weight in favor of the Prosecution. Illustration: Worker in blue cap and overalls stands to left of large balance plate, stacking round weights.
Text: And that’s why the FBI and other agencies are building huge databases with information from manufacturers about the most common tire treads, shoe soles, auto body paints, windshield glass, textile fibers… Illustration: Row of green filing cabinets with two drawers open, one overflowing with various shoes and the other with tires.
Text: And on the other side… Illustration: Prosecutor in blue suit and eyeglasses with word bubble. She is saying: “Now assume the Prosecution is right — the evidence was from the suspect. How surprising would evidence like this be?”
Text: That is, what are the chances that a real match would show so many differences and seem to be such a poor fit with the evidence? Illustration: Forensic scientist in white lab coat compares sole of a shoe with photo of a shoeprint. In word bubble, she is saying: “Well, Scarface Diane’s sandals are brand-new, but the print at the scene looked worn down. That’s weird. Kind of surprising.”
Text: The more surprising it would be to find evidence like this if the Prosecution were right … the greater the weight in favor of the Defense. Illustration: Worker in blue hat and overalls stands to right of balance plate, stacking large round weights.
Text: Laboratory testing helps. Scientists can measure the real-world variability in evidence. Illustration: Criminal wearing eye mask, striped shirt, black cap and pants tiptoes past forensic scientist seated in director chair speaking on megaphone. She is saying: “OK, let’s try that again, but this time pretend you’re tiptoeing out of the house with a safe in your hands.”
Illustration: Character Scarface Diane, a woman with light long hair and a scar on right side of face, holds round blue weight in right hand and several red weights in left. She is saying: “OK, we have the weight in favor of each side — now what?”
Illustration: Two hands side by side, one higher up holding several red weights while lower hand holds one blue weight. Label says, “Likelihood of the evidence under the Prosecution’s claim / Likelihood of the evidence under the Defense’s claim.” Superheroes named Probability and Statistics exclaim, “Voilà! A Likelihood Ratio!”
Illustration: Character named Scarface Diane looks at illustration showing stack of four red weights (labeled prosecution) over one blue weight (labeled defense) and says: “Aha! So the bigger the Likelihood Ratio, the more likely it would be to see this evidence under the Prosecution’s arguments than under the Defense’s.”
Illustration: Two blue birds use beaks to carry either side of banner that reads, “Likelihood Ratio = 4” as character Scarface Diane is saying: “So that shoe print was evidence more in favor of the Prosecution than the Defense. But how much more?”
Illustration: Superheroes Statistics and Probability watch two large coins flip in the air. Statistics says, “That evidence is about as strong as pointing to two heads in two coin flips and claiming the coin was two-headed.” Probability says, “Not super strong.”
Illustration: Superheroes Statistics and Probability look distressed and say together, “In most of science this would considered ‘weak’ evidence.” Character Scarface Diane grins widely with both thumbs up.
Text: Different types of evidence — and different crime scenes — give different Likelihood Ratios. For example, Likelihood Ratios for DNA evidence can reach into the millions. Illustration: Two pigs flying with wings hold aloft in snouts banner that reads: “Likelihood Ratio = 1,000,000”
Text: That would be like pointing to 20 heads in 20 coin flips as evidence of a two-headed coin. Way stronger than that shoe print. Illustration: Superhero Probability holds red umbrella as coins drop and bounce like raindrops. He is saying: “Extremely strong!”
Text: The Likelihood Ratio and its devil’s-advocate-for-both-sides approach might seem like common sense. But it’s not in widespread use yet, says Hal Stern. Illustration: Two identical lawyers in blue three-piece suits, red ties and red devil’s masks, with red hands and pointy red tail, stand behind side-by-side tables — one labeled “Defense,” the other “Prosecution.”
Text: That’s partly because statisticians and forensic scientists still need to do more work together. Illustration: Two scientists in white lab coats face each other, speaking. Their word bubbles show scientific formulas and symbols.
Text: Though that’s getting harder and harder in today’s political climate. Illustration: Three newspapers with headlines that read: “Attorney General Scuttles Forensics Partnership with Scientists, Statisticians”; “Trump Tries to Axe Forensic Science Funding”; and “New Department of Justice Forensics Science Adviser: Not a Scientist.”
Text: But some people have another worry: Likelihood Ratios would be too scary and hard for juries to understand. (And maybe for lawyers and judges.) Illustration: In courtroom, jurors, judge and lawyers watch in horror as smiling woman holds up paper labeled “Likelihood Ratio” that produces purple monster blob with four eyes, sharp teeth and tentacles.
Text: But Hal Stern is hopeful. He’s co-director of the Center for Statistics and Applications in Forensic Evidence, where researchers across the world come together to study the issues. Illustration: Group of four international scientists speak by a table with placards that read, “Engineers,” “Lawyers,” “Forensic Statisticians” and “Forensic Scientists.”
Text: They also help train and educate people about forensic science and statistics — including the Likelihood Ratio. Illustration: Statistician Hal Stern in blue shirt and eyeglasses points to chalkboard with the words “Likelihood Ratio” in classroom of students.
Text: Meanwhile, US advocates continue working to exonerate wrongfully convicted prisoners like Johnny Tallbear, sentenced to life without parole for a murder he didn’t commit. A forensic analyst and key expert in his trial was later found by the FBI to have made outright errors in several of her cases, going “beyond the acceptable limits of science.” DNA evidence exonerated him earlier this year. He had served 26 years — more than a third of his life, innocent, behind bars. Illustration: Johnny Tallbear, wearing orange jumpsuit, sits in prison cell.
Text: About 1.5 million people are sitting in US prisons today. Experts estimate that at least 15,000 of them are not guilty — including 1 out of every 25 sentenced to death. And that also means there are thousands of criminals who have gone free. Illustration: Group of prisoners standing together, wearing white T-shirts under orange jumpsuits. Faces of three men and one woman are visible.
Text: So anything that brings more science to forensic science can only be a good things. Illustration: Light background sketch of domed government building.
Illustration: Diverse group of characters, including prosecution and defense lawyers, Superheroes Probability and Statististics, and criminal suspect march together, smiling and carrying signs that read: “We like LIKE-lihood ratios,” “Probability beautiful,” “Better statistically read than dead,” “Make forensic science great again!” and “Yes, stats can.”