BIOM5010: Statistics #2F. Hypothesis Testing. onlinestatbook.com 11(A,B,C)

Size: px

Start display at page:

Download "BIOM5010: Statistics #2F. Hypothesis Testing. onlinestatbook.com 11(A,B,C)"

Basil McCarthy
7 years ago
Views:

1 Slides 2F.1 BIOM5010: #2F Hypothesis Testing onlinestatbook.com 11(A,B,C)

2 Slides 2F.2 What is H 0? xkcd.com/892/

3 Logic of hypothesis testing Slides 2F.3 Example from text: 1. give martinis to Bond 2. ask whether it was shaken or stirred. Assume: Mr. Bond correct on 13 of the 16 taste tests. Question: Can Mr. Bond to detect shaken vs. stirred martinis? If yes, then we predict he can do it for a future martini If no, then that test was a one off

4 p-values Slides 2F.4 p-value: probability of getting the result (or a more extreme one) just by chance Example from text: 1. give martinis to Bond 2. ask whether it was shaken or stirred. Assume: Mr. Bond correct on 13 of the 16 taste tests. Question: Can... detect shaken vs. stirred martinis? Maybe he was just lucky? probability just guessing would be correct 13/16 times or more = (~1%). Does this mean there is a 1% chance he was just lucky? No: if he were just lucky (and we don't know how likely this is) this outcome would have a 1% chance

5 Interpret p values Slides 2F.5 An animal trainer claims a bird can determine whether or not numbers are divisible by 7. experiment assessing claim: bird correct 9/16 choices. Using the binomial calculator, probability of being correct 9/16 times is Thus, bird who only guessing would do this well 40% of the time, these data don't provide evidence the bird can tell the difference. QUESTION: Would you conclude that there is a 0.40 probability that the bird can tell the difference? As a scientist, you would be very skeptical that the bird had this ability. You would think the probability very low.

6 Slides 2F.6 Null Hypothesis Null Hypothesis: H 0 The hypothesis that an apparent effect is due to chance Typically that a test parameter is zero Difference between (Bond's ability) and (chance) = 0 Statistical Test 1. Try to explain your data using H 0 2. Calculate p-value = probability H 0 explains data 3. Interpret: Use p to decide whether result is significant Fisher interpretation: p is the strength of evidence against H 0

7 Slides 2F.7 What we want Data Inference How likely is my theory (ie. How unlikely H 0 ) What we have How likely is my data? H 0

8 Slides 2F.8 Hypothetical story H:A > B I don't belive you I'll do tests and get data

9 Slides 2F.9 Hypothetical story H:A > B I don't belive you I'll do tests and get data H 0 rejected p=0.05. See! OK. That eliminates H 0. But what about all the other Alterternatives? Eliminated H: A > B with p=0.05 H 0 : effect is from chance H C : effect is 3 rd variable H D : effect is sampling bias

10 Significance Slides 2F.10 A p value below a threshold α is significant Typical α values are 0.05 or 0.01 (significance level) This is used in a paper as follows: Based on data A, when B is present, outcome C is significantly more likely (p=0.024)

11 Slides 2F.11 Be careful to interpret significance correctly What does significance mean? Statistical Significance does not mean A significant finding of the study was that A is larger than B Statistical significance does not mean the result is important or noteworthy (e.g. It only means it signifies something (ie. Unlikely to be explained by chance) Example: look at F=(Brain Mass)/(Body Mass) Even if there is a 0.1% difference between two groups, if we have a large enough study we can detect the difference. Can we say F A was significantly larger than F B? Can we say A significant finding was F A >F B?

12 Questions Slides 2F.12 Why is it wrong to say the p-value is the chance the Null Hypothesis is true? What does the p-value mean? What is the chance the H 0 is true? What is the significance level in a study? What does significance mean? Does it mean that the result is important? Is there a relationship between statistical significance and the normal sense of significance? What are good synonyms for the normal sense of significance?

13 Slides 2F.13 Errors Eg. Test for cancer. H 0 = Subject is normal (ie. Not cancer) Type I error Reject H 0 even though it is true eg. Say test results indicate cancer, but patient is OK Other names: False Positive, False Match Type II error Accept H 0 even though it is false eg. Say patient is OK, even though patient has cancer Other names False negative, False Non-Match AcceptHH 0 RejectHH 0 HH 0 true TTTT FFFF HH 0 false FFFF TTTT

14 Slides 2F.14 FP = False Positive (Type I) TP = True Positive FN = False Negative (Type II) TN = True Negative Error Rates AcceptHH 0 RejectHH 0 HH 0 true TTTT FFFF HH 0 false FFFF TTTT True Negative Rate = Specificity True Negatives / All H 0 = TN / (TN + FP) eg. Likelihood of saying OK if patient is OK True Positive Rate = Sensitivity True Positives / All (Not H 0 ) = TP / (FN + TP) eg. Likelihood of saying cancer if patient has cancer

15 Positive Predictive Value Slides 2F.15 Example: In test of 1000 patients, 100 with cancer, FP = 20, FN = 20. Calculate: Normals = = 900 TP = Patients FN = = 80 TN = Normals FP = = 880 Sensitivity =True Positive Rate = Test Cancer / All Cancer = TP/Patients = (100-20) / 100 = 0.8 Specificity = True Negative Rate = Test not Cancer / All Not Cancer = TN/Normals = (900-20)/900 = 0.98

16 Slides 2F.16 Example: Sensitivity = 80/ 100 = 0.8 Specificity = 880/900 = 0.98 Positive Predictive Value If cancer occurs in 1% of subjects, what can we tell a subject if their test is positive? Can we say you have 80% chance of cancer? Why not? Similar to misunderstanding of what rejecting H 0 means Specificity Know: Patient has cancer => Q: chance test is cancer? Want: Test is cancer => Q: chance patient has cancer?

17 Positive Predictive Value Slides 2F.17 Example: In test of 1000 patients, 100 with cancer, FP = 20, FN = 20. PPV is the predictive value of a positive result PPPPPP = sensitivity prevalence sensitivity prevalence + 1 specificity 1 prevalence sens = 0.8. spec = prev =.01 PPV =.8x.01 / (.8x.01 + (1.98)(1.01)) = 0.29 So a positive test only indicates a 29% chance of a positive result. Problems with PPV: Depends on accurate knowledge of prevalence Also there is NPV = Negative predictive value

18 Questions Slides 2F.18 What is sensitivity? What does a high sensitivity mean? Can we have a low specificity and high sensitivity? What does this tell us? How can we have high sensitivity but low PPV? What does that mean?

Independent samples t-test. Dr. Tom Pierce Radford University

Independent samples t-test. Dr. Tom Pierce Radford University Independent samples t-test Dr. Tom Pierce Radford University The logic behind drawing causal conclusions from experiments The sampling distribution of the difference between means The standard error of