The NAS Report and Foundations of Good Science

Transcription

1 The NAS Report and Foundations of Good Science March 29, 2012 NACDL/Innocence Network Kansas City, MO Andrea Roth, UC Berkeley School of Law

2 Goals for the Next Hour Learn what the National Academy of Sciences 2009 Report says about what makes good science Learn what terms like scientific method, error rate, and validation really mean Discuss how to identify and challenge nonscientific aspects of forensic evidence

3 The NAS Report Who National Research Council of the National Academy of Sciences, under D.C. Circuit Judge Harry Edwards Why Funding from Congress DNA exonerations

4 NAS Report: Overarching Conclusions Descriptive: Other than DNA, nearly all forensic individualization sciences fail to possess the most basic attributes of science Prescriptive: We need an independent agency (NIFS), a research culture in forensic science, and more robust judicial review of forensic disciplines

5 Is this stuff even science? The Law s greatest dilemma in its heavy reliance on forensic evidence, however, concerns the question of whether and to what extent there is any science in any given forensic science discipline. (87)

6 How can I tell if an expert s method is based on good science? [See if it follows the scientific method.] Great! What s the scientific method?

7 What is the Scientific Method? Isaac Newton: Scientific method refers to the body of techniques for investigating phenomena, acquiring new knowledge, or correcting and integrating previous knowledge. It is based on gathering observable, empirical, and measurable evidence subject to specific principles of reasoning. Andrea Roth, paraphrasing the Internet: An objective means of testing guesses we have about why things are the way they are

8 Steps of the Scientific Method

9 Steps of the Scientific Method Clearly define your question Which popcorn pops the best? State the hypothesis to be tested Will the most expensive one Orville Redenbacher pop the best? Create a repeatable procedure to collect measurable data Pop each brand for 3 minutes in same microwave, repeat 5 times, see what % of kernels pop View results Act II Premium Theater pops the best! Draw conclusion Highest cost doesn t necessarily mean highest quality! After only 5 trials, we can t be 100% sure about this conclusion

10 Is this method scientifically valid? Does it accurately and reliably measure/do what it claims to measure/do?

11 Reliability v. Validity A method is reliable if it produces consistent, repeatable results Reliable but not Valid Neither Valid nor Reliable Valid and Reliable!!!!

12 Accuracy v. Precision

13 Important means of ensuring reliability and validity Data is measurable fluffy versus # of kernels popped Results are repeatable Using a different microwave each time? Procedure is documented? Level of uncertainty is known Do we have a sense of how confident we should be in the conclusion based on the results? Lack of Bias Does experimenter work for Orville Redenbacher? Is experimenter s best friend double-checking her results?

14 A Closer Look at: Uncertainty and Error

15 What do scientists mean by measuring uncertainty? There is no zero error rate in science Every test leaves uncertainty about what the real answer is Key is to measure that uncertainty!

16 What do scientists mean by measuring uncertainty? Common terms: Sampling error: Pepsi challenge Measurement error: 3 breathalyzer results; 5 kernel pop sessions False positive ( type I error ) / false negative ( type II ): Diagnostic test for breast cancer

17 If no valid means of measuring uncertainty, evidence should be inadmissible Probative value of method depends on how well it detects/predicts the proposition it s being admitted for If test can t say with enough confidence what the real value/answer may be, it s not reliable evidence for that proposition e.g. Massachusetts OUI breathalyzer rule

18 From the horse s mouth Few forensic science methods have developed adequate measures of the accuracy of inferences made by forensic scientists. All results for every forensic science method should indicate the uncertainty in the measurements that are made, and studies must be conducted that enable the estimation of those values. NAS Report, p 184

19 Uncertainty with respect to what? Hair analysis method might have low error rate for declaring brunettes, but high error rate for declaring identity of particular brunette (e.g. Jennifer Friedman)

20 So, what s an error rate? % of cases in which the analysis led (or is expected to lead) to a false conclusion Can have different types of error rates false positive rate (% of times test says YES when answer is NO) false negative rate (% of time test says NO when answer is YES)

21 A Special Kind of Uncertainty/Error: Pattern Evidence and Lack of Data on Variability of Patterns p.s. If you remember nothing else from today, let it be the next 4 slides

22 Uncertainty and Match Evidence Match between suspect and evidence (latent prints, hair/fiber, bite mark, toolmark, DNA, facial mapping) Offered to show defendant is the source Scientific method: How uncertain are we, after result (match), of our conclusion (he s the source)?

23 Need Variability Data! Intraindividual variability (e.g. how much one s finger impressions change from impression to impression) Interindividual variability (e.g. how much impressions of many fingerprints vary across a population, and in what ways) Variability data Match statistics!

24 Without the probability assessment, the jury does not know whether the patterns are as common as pictures with two eyes, or as unique as the Mona Lisa. US v. Yee (N.D. Ohio 1991)

25 Big take-away point: the lack of data about pattern variability is the NAS s primary criticism of nearly every forensic discipline discussed in Ch. 5 of the report.

26 Even DNA AZ, MD databases FBI refusal to open up CODIS Mueller, L.D Can simple population genetic models reconcile partial match frequencies observed in large forensic databases? Journal of Genetics, 87 :

27 Example: Facial Comparison The SWGANTH guidelines state clearly that given the difficulty in scientifically assessing the uniqueness of human faces, statements regarding the value of craniofacial superimpositions should be carefully presented. However, accurately determining the suitable weighting of techniques which do not currently have associated known error rates is problematic. Evison et al. (2012)

28 Confidence Intervals Ex: 3 breathalyzer values are.08,.09,.10. Average is.09. There s a 95% chance that true value is between.07 and.11 (plus or minus 2 standard deviations from the mean measurement). Bottom # is lower bound ; top # is upper bound

29 A Closer Look at: Bias

30 Sources of bias that make a method / result unscientific Contextual bias e.g. post-9/11 Mayfield warm fuzzies Reviewing your friends/colleagues (ACE-V) Confirmation bias Anchoring / Sunken cost Other cognitive biases Too quick to assume lack of coincidence (e.g. birthday problem)

31 Important steps to minimize bias Independence of laboratories from law enforcement Sequential unmasking Blind proficiency testing Protocols for ID procedures, testing Independent peer review

32 Putting it all together: Validation Studies

33 What are validation studies? John Butler: The process of demonstrating that a laboratory procedure is robust, reliable, and reproducible in the hands of the personnel performing the test in that laboratory.

34 Two stages of validation studies Developmental validation Testing of new technologies, software, etc. by the developing company, agency, lab Internal validation Verifying through testing that procedures established through development validation reach valid results in one s own laboratory.

35 Why is a validation study important? Only way to confirm through reason that a method produces accurate results! Cautionary tale: Dr. Michael West, bite mark analyst

36 Validated for what? If lab has to change its method, lab should do new validation studies... E.g., ballistics lab s validation studies only test methods for comparing intact bullets The bullet they are viewing is deformed so they can t look at half the sections of the bullet to see if they match. Method might be valid, but we don t know, because it hasn t been validated for this purpose

37 Critical steps in a validation study Competence in testing and calibration Assessment of uncertainty of results Minimization of bias Peer review of the validation study (so that experts in the field can review, question, and check the repeatability of the results) Renewed validation when conditions change (e.g. new knowledge)

38 Validating Pattern Analysis Methods 1. Precisely define attributes (for DNA, alleles; for FPs, minutiae; for bullets, striations) 2. Measure variability of these attributes in pop s that are representative and sufficiently large 3. Use these results to set objective limits on number and type of attributes needed for identification 4. ID sources of error and assess level of uncertainty associated with ID based on #/type of attributes 5. Use all of the above to develop protocols that minimize error and bias 6. Validate said protocols for anticipated use 7. When new info or situations arise, add to protocols and revalidate!

39 Example - Toolmarks Looking method using a comparison microscope No validation studies showing that this method works to accurately call matches between bullets

40 Example Facial Comparison Looking method video superimposition Subjective assessment of observer Facial mapping several points sliding scale of confidence powerful, limited, inconclusive No validation studies: Ability to discern a match Ability to discern identity from a match

41 Use of these terms in NAS Report [C]laims that [friction ridge] analyses have zero error rates are not scientifically plausible. ACE-V provides a broadly stated framework for conducting friction ridge analyses. However, this framework is not specific enough to qualify as a validated method for this type of analysis. Because not enough is known about the variabilities among individual tools and guns, we are not able to specify how many points of similarity are necessary for a given level of confidence in the result. [A]s casework capacity increases, pressure to... extend opinions beyond the scientific method... also increases significantly.

42 How to incorporate these concepts into your practice

43 Discovery Use NAS Report as blueprint Ask for SOPs (testing & quality control), developmental/internal validation studies for equipment, software, lab, technique Ask for any scientific literature the expert bases her opinion on Ask for any communications between analysts and police Any information on proficiency testing Ask specific questions about procedure or ask expert during meeting about procedures

44 Admissibility Daubert: Explicitly based on scientific validity, from testability, peer review/publication, known/potential error rate, SOPs, general acceptance Frye: General acceptance for what purpose? Post-NAS? (note: AAFS endorsed the report)

45 NAS on Admissibility There are two very important questions that should underlie the law s admission of and reliance upon forensic evidence in criminal trials: (1)the extent to which a particular forensic discipline is founded on a reliable scientific methodology that gives it the capacity to accurately analyze evidence and report findings and (2)The extent to which practitioners in a particular forensic discipline rely on human interpretation that could be tainted by error, the threat of bias, or the absence of sound operational procedures and robust performance standards. (87)

46 Motions in Limine CX Limit language of expert s opinion Establish scientific method go down list? You didn t do X, you didn t do Y Defense case Call expert on scientific method? Establish NAS Report as a learned treatise? Jury Instructions In assessing scientific testimony, Closing Use science fair analogies?

47 Questions?