Lecture 15: Admissibility of Scientific Evidence, Statistical Mistakes in the Courtroom

(version 7 Jan 2008)

This material is copyrighted and MAY NOT be used for commercial purposes

 You are visitor number   since 7 Jan 2008 

Expert witnesses and the Admissibility of scientific evidence

What makes one qualified to be an expert witness? Likewise, even if one is an expert witness, under what conditions will the court allow scientific evidence to be admitted into trial?

In deciding if someone is an expert in a particular field, the court looks at education, training, practical background and perhaps the outside opinion of other experts.

The conditions for allowing a witness to qualified as an expert are separate from admitting the testimony of that expert witness.

For example, someone could be qualified by the courts as an expert witness on (say) human auras, but this, by itself, does not allow any testimony of human auras to be admitted. Rather, a further test (a series of conditions that the court agrees are satisfied) is required to admit the evidence.

Thus expert testimony is not qualified "just because somebody with a diploma says it is so" (United States v. Ingham, 42 M.J. 218, 226 [A.C.M.R. 1995]). In addition to appropriate qualifications of the expert, the proposed testimony must meet certain criteria for reliability. In the United States, two models for evaluating the proposed testimony are used, Daubert and Frye.

Before the trial, a Daubert (or Frye) hearing takes place in front of the judge, but not the jury, and the judge considers the evidence presented at such a hearing to determine whether an expert's "testimony rests on a reliable foundation and is relevant to the task at hand." (Daubert, 509 U.S. at 597).

The Frye Test

Frye was the standard test for admissibility of expert witness testimony used by both federal and state courts from 1923 until 1993, and was based on the 1923 ruling in Frye v. United States. The Frye test requires that the scientific principles upon which the work is based on

The Frye test is a conservative standard intended to be an obstacle for introduction of evidence based on new scientific principles or dubious scientific grounds. Using Frye, a judge must test the viability of expert testimony before allowing it in court using three tests:

  1. The scientific tests reliability must be established by its general acceptance in the relevant scientific community

  2. The testifying witness must be properly qualified

  3. The proponents of the evidence must demonstrate that the correct scientific procedures were used

The Daubert Test

While the Frye standard is still used in some state courts, starting in 1993, all federal courts and most state courts use the test formulated from the US Supreme Court case of Daubert v. Merrell Dow Pharmaceuticals . A Daubert hearing considers four questions (prongs):

  1. Whether a "theory or technique . . . can be (and has been) tested"

  2. Whether it "has been subjected to peer review and publication".

  3. Whether, in respect to a particular technique, there is a high "known or potential rate of error"

  4. the general acceptance of the theory in the scientific community

  5. whether the proffered testimony is based upon the expert's special skill (this last test was added in the 7th Circuit court)

Kumho Tire Company v. Carmichael case: Daubert is Extended.

The Daubert case was about scientific evidence and did not address whether it was to be used outside the "hard" sciences. The federal courts were sharply divided over the applicability of Daubert to experts having technical or specialized knowledge (engineers) or opinions based on experience and training. The U.S. Supreme Court finally settled the issue in 1999 in Kumho Tire v. Carmichael, when the court extended Daubert to testimony provided by all experts (including engineers) in federal court.

Presenting DNA evidence in trial

As new DNA procedures were developed, e.g.,

All of these went through at least one (and often several in each state) Frye/Daubert hearings. Once the high court in a state (for state cases) or a federal rule (likely appealed) set a precedent, the requirement of a hearing on the admissibility of the type of DNA testimony in question was often (but not always) foregone.

Hence, much expert testimony initially was by experts (Ph.D.s in forensics, genetics, and statistics).

Once the precedents were established, the expert testimony in trials was given by criminalists (i.e. , CSI types).

Probability of failing to exclude (''match probability'') calculations are presented using the NRC II recommendations. Typically NRC 4.1 in the US, but the more conservation (giving a higher match probability and hence more favorable to a defendant) NRC 4.2 is becoming more common.

DNA evidence is unique in that with other forms of evidence (fingerprints, handwriting, ballistics), the expert says "in my expert opinion, this (print, writing, gun). came from the suspect"

Rather, DNA evidence ties to place a specific probability on how likely a random person would have the same DNA profile.

While extremely precise (if done carefully), one can also be fast and loose with probability, and hence some care is needed.

Misuse of Probability in Court

People v. Collins

A classic misuse of statistics was in the State of California vs. Collins, a 1968 case.

Witnesses said that a robbery in Los Angeles was done by a black male with a beard and mustache and a white female with a blond ponytail, who escaped in a yellow car.

Mr. Collins was indeed a black man with beard/mustache who had a white girlfriend with a yellow car and a blond ponytail.

The prosecutor had an `` instructor in mathematics'' introduce the jury to the product rule, and the prosecutor then offered the following probabilities (for LA)

The prosecutor argued that the odds were thus one in 12 million. The jury found Collins guilty.

The California Supreme Court set aside the conviction, noting that mathematics

The Sally Clark Case

A much more striking miscarriage of justice is the Sally Clark Case in England.

British housewife Sally Clark was accused in 1998 of killing her two sons, as a child of hers died at 11 weeks of age, and a child conceived later died at 8 weeks of age.

The defense claimed that these were two cases of sudden infant death syndrome; neither side offered any other explanation for the two deaths.

Prosecution expert witness Sir Roy Meadow (most famous for a classic 1977 paper on Munchausen Syndrome by Proxy) testified that the probability of two children in the same family dying from sudden infant death syndrome is about 1 in 73 million

Clark was convicted in 1999. Upon appeal her conviction was upheld in October of 2000, but was finally freed on a second appeal in Jan 2003. Tragically, she died 15 March 2007, at the age of 42.

A more formal statistical analysis of the Clark case is as follows:

The Prosecutor's fallacy

The Clark case is a classic example of the so-called The Prosecutor's fallacy, a term coined in 1987 by Thompson & Schumann. (Jsotr link to original article)

Suppose you just won the lottery. The police stop by and arrest you, because you must have bribed someone involved with the drawing. After all, winning the lottery is an extremely rare event.

What is the flaw in the prosecution's' case against you? While the chances of any one particular person winning the lottery are very small, someone will because there are a large number of plays. Hence, we expect lottery winners. We need more to argue cheating.

We can frame this using conditional probablity, where P(A|B) is the probablity of event A occuring giving we know B.

Let E = the particular evidence, I = you are innocent. What a court really wants is P(I | E) -- the probability you are innocent given the evidence. However, what we often have is P( E | I) -- the probability of the evidence given you are innocent.

The random match probability for DNA is P(E | I) -- what is the chance that a random person (i.e., not involved in the crime) has such a DNA profile.

The connection between the two conditional probabilities, P(I | E) and P(E | I) occurs through Bayes theorem,

For our purpose, we can express this as Odds(I | E) = Odds(I) * P(E | I) / P(E | guilty)

For DNA evidence, P(E | guilty) = 1, as you did contribute the DNA sample.

Hence for DNA, Odds(I | E) = Odds(I) * random person matching.

Hence, if the prior odds of being guilty are 1:1 (or higher), we can say this is it highly unlikely you did not leave the crime sample

Suppose the match probability is one in a million.

Let's consider the Sally Clark case and suppose that the chance of two (or more) SIDS deaths is 1/1,500,000 (it is likely much higher).

The Defense Attorney's Fallacy.

Defense Attorneys can also (and often do) make incorrect probability statements.

Consider the 1 in a million match.

A Defense attorney might say this means there are roughly 300 person matching this in the US, and hence the odds are 1/300 that their client is guilty.

This ignored the P(I) part. Presumably, there is other evidence against the client. For example, in a rape case, there are roughly (accounting for sex and age) 100,000,000 possible candidates, and so 100 potential matches.

However, the Defense Attorney's Fallacy is to assume that all 100 of these potential matches have equal chance of committing the crime. Other factors, such as opportunity (lack of an alibi), motive, and/or means greatly reduces the potential set of individuals from the millions to tens or thousands.