HEL-8020 Analyse av registerdata i forskning: Multiple testing

Transcription

1 Multiple testing Page 1 of 34 HEL-8020 Analyse av registerdata i forskning: Multiple testing Førsteamanuensis Georg Elvebakk Institutt for matematikk og statistikk, UiT 27. april :15-09:00

2 Multiple testing Page 2 of 34 Multiple testing Assume we have five significant results. HURRAY, SIGNIFICANT RESULTS But we have also 95 non-significant results?????????? How can we adjust for multiple tests?

3 Multiple testing Page 3 of 34 Simulated data, H 0 is true - Testing a single hypothesis Example: Comparing patients (BMI) at two hospitals. H 0 : Equal mean BMI - Reject H 0 if P-value 0.05, (the significance level) - Calculations: N Mean BMI P-value Conclusion Hospital A Not rejected Hospital B

4 Multiple testing Page 4 of 34 Simulated data, H 0 is true - Testing several hypotheses Example: Comparing patients (BMI) at15 hospitals. N Mean BMI Hospital A Hospital B Hospital C Hospital D etc Hospital O The ANOVA analysis (not shown here) tests: H 0 : All hospitals have equal BMI

5 Multiple testing Page 5 of 34 Simulated data, H 0 is true - Testing several hypotheses - Next step is to try to locate significantly different hospitals. - Pairwise comparisons of all 15 hospitals. Sign. level 0.05: Hospital Simulated data pairs P-value Decision A vs B Not rejected A vs C Not rejected etc B vs C Not rejected B vs D Not rejected etc M vs N Not rejected M vs O Rejected N vs O Not rejected 105 tests 2 rejected

6 Multiple testing Page 6 of 34 Simulated data, H 0 is true - Testing several hypotheses Figure:. A scatter plot of the 105 ordered P-values from the pairwise comparisons of hospitals: 105 pairwise comparisons of hospitals 105 pairwise comparisons of hospitals Rank order Rank order Ordered P-values(BMI) Ordered P-values(BMI) - Two of the P-values are less than 0.05, i.e. significant.

7 Multiple testing Page 7 of 34 About decisions from testing hypotheses Decision based on data Reality H 0 accepted H 0 rejected True H 0 : Equal BMI Correct Type I error False H 0 Different BMI Type II error Correct P (Type I error) = 0.05 = significance level Interpretation of significance level 0.05: * P(Type I error)= * 5% Type I errors. * Expected five Type I errors in 100 tests. * Expected 50 Type I errors in 1000 tests.

8 Multiple testing Page 8 of 34 Back to example with simulated BMI data: 105 comparsons of hospitals. Two of 105 hypotheses were rejected Two Type I errors We know these are Type I errors because we simulated equal-mean BMI data. And with a significance level of 5% we expect approximately type I errors (wrongly rejected null hypotheses).

9 Multiple testing Page 9 of 34 Real data example: BMI. - Comparing patients (BMI) at 41 hospital wards. Data from the norwegian Ryggregister, patients. - One-way ANOVA in SPSS: H 0 : All hospitals have equal BMI

10 Multiple testing Page 10 of 34 ANOVA (in SPSS):

11 Multiple testing Page 11 of 34 SPSS results: Tests of Between-Subjects Effects Dependent Variable: BMI Source df Mean Square F Sig. Corrected Model 2598,541 a 40 64,964 3,422,000 Intercept AvdNavn Error Total Corrected Total a , , ,180, , ,964 3,422, , , , F = 3.422, P-value = and H 0 is rejected. - Conclusion: There are at least two different hospital wards. - Typically use multiple comparison post hoc test to investigate.

12 Multiple testing Page 12 of 34 Observed means and group sizes Number Mean

13 Multiple testing Page 13 of

14 Multiple testing Page 14 of 34 Testing for differences - Next: Locate the wards with significant different mean BMI s. - There are 41 (41 1) 2 = 820 pairwise comparisons. - Use two-sample T-test for each pair? - For instance ward i and j: T = X i X j S p 1ni + 1 n j H 0 T ni +n j 2 - Compute the p-values and conclude...

15 Multiple testing Page 15 of 34 Results presented as sactter plot of 820 ordered P-values: 820 pairwise comparisons of hospital wards 820 pairwise comparisons of hospital wards Rank order Rank order Ordered P-values(BMI) Ordered P-values(BMI) - Seems definately to be some significante differences here. - But 146 P-values are less than 0.05, should we reject all these hypotheses?

16 Multiple testing Page 16 of 34 New measure of error - If you only do one test (compare two groups) it is common to use a level of 5%. (probability of wrongly rejecting a true H 0 ). - If you do several (m) tests this will be repated for each test, and the probability of incorrectly claiming at least one significant difference can then be much larger than 5%. - Possible solution: In multiple testing it is common to controll: The probability of at least one Type I error = The family-wise error rate (FWER) It can be shown that if individual tests all have the level of α m, then the total FWER m α m

17 Multiple testing Page 17 of 34 Consequences of this - Want FWER (prob. of at least one Type I error) to be small (α = 5%)? You must then start with an ever smaller significance level α m for each test! -The BMI example: FWER 820 α m Significance Max FWER level per test all tests

18 Multiple testing Page 18 of 34 Leads to Bonferroni correction - So a common method for achieving FWER α: Bonferroni: Example: FWER α = 0.05: Testing m hypothesis. Reject H 0 if P-value α m. This guarantees FWER α. m Bonferroni level in each test 5 α m = = α m = = α m = =

19 Multiple testing Page 19 of 34 Significant differences by Bonferroni - Tests and P-values as before, rejection level stricter. - Significant differences for 27 P-values less than in the real data BMI example: S 26 S 27 S S S S S 35 S S 36 S S S 37 S S 38 S S S S S S S S S S S S 41

20 Multiple testing Page 20 of 34 Other methods: Tukey, Dunnett - Bonferroni easy, but may be very conservative. FWER << α. - Low power tests, hard to detect significant differences. - Several other methods (Post Hoc) in for achieving FWER α (in SPSS). - Methods for different situations: All/some pairwise comparisons, comparing contrasts, balanced/unbalanced, unequal variance. - In addition to Bonferroni, we will look at Tukey and Dunnett.

21 Multiple testing Page 21 of 34 SPSS

22 Multiple testing Page 22 of 34 Tukey s method -Controls the FWER for all pairwise comparisons. -Balanced data (equal numbers in grpups) FWER = α. -Unbalanced data (unequal numbers, Tukey-Kramer ) FWER α. But still often less conservative than Bonferroni. - Based on computing critical region/p-value from the correct distribution of the maximum difference between mean T-test statistic.

23 Multiple testing Page 23 of 34 Significant differences by Tukey BMI example: 30 significant differences with P-values less than α = 0.05: S 26 S 27 S S S S 35 S S S 36 S S S 37 S S 38 S S S S S S S S S S S S S S S S 41

24 Multiple testing Page 24 of 34 Dunnett s method - Related to Tukeys method, but looks at pairwise multiple comparison of k 1 treatments against a single control treatment. FWER = α. - Can again find the corrrect distribution of the T-test statistic.

25 Multiple testing Page 25 of 34 BMI example, 820 tests, summarized P-values 0.05, this is 18% of all P-values P-values , the Bonferroni correction P-values are significant by the Tukey method. Question: Are the FWER methods too conservative? Answer: May depend upon the number of tests m?

26 Multiple testing Page 26 of 34 Alternative to FWER methods? - In FWER controlling methods we have that P (at least 1 type I errror) α - A quite strict criterion in cases with hundreds or thousands of tests. - Nearly all significant differences should be true, but many true differences may go undetected. - These methods are not always recommended for a very large number of tests. (Bioinformatics?). - Another approach (equally strict no matter the number), is to control not the number but the rate of false discoveries.

27 Multiple testing Page 27 of 34 False discovery rate (FDR) - Another type of methods are those controlling the false discovery rate. (Not in SPSS.) FDR V R The proportion Type I errors among all rejections R = Number of rejected hypotheses among all m tests. V = Number of true H 0 rejected, Type I error.

28 Multiple testing Page 28 of 34 Remember: Comparing mean BMI at 41 hospital wards. 820 pairwise comparisons of hospital wards 820 Rank order Ordered P-values(BMI) 146 of original P-values are less than 0.05 (18% of all P-values).

29 Multiple testing Page 29 of 34 Benjamini-Hochberg (BH) Threshold (one of several FDRmethods) 1) Decide an upper limit α for FDR, for example ) Calculate all m P-values (T-tests). 3) Order the P-values from smallest til largest: P (1), P (2),... P (m). 4) Plot the ordered P-values against their ordering number. 5) Estimate the number of true H 0 (here m 0 = 740). 6) Find the largest rank order number y such that P (y) yα m 0 and reject all hypotheses with P-value P (y)

30 Multiple testing Page 30 of 34 Graphical picture 820 pairwise comparisons of hospital wards 200 Rank order Ordered P-values(BMI) - The line crossing at 80 indicates m 0 = = 740 true H P-values are less than the FDR limit.

31 Multiple testing Page 31 of 34 BMI example, 820 tests, summarized P-values 0.05, this is 18% of all P-values P-values , the Bonferroni correction P-values are significant by the Tukey method P-values are less than the FDR limit.

32 Multiple testing Page 32 of 34 Summary: m (multiple) tests. - Type I error: Rejecting a true H 0. - Significance level = 0.05 means we expect 5% Type I errors. - Family-wise error rate: FWER = P (at least one Type I error) m α m. - Reducing significance level will controll FWER (Bonferroni). - Many other methods for controlling FWER (several in SPSS). - FDR (False discovery rate) is controlling the proportion of Type I errors.

33 Multiple testing Page 33 of 34 General advice - Act carefully when performing multiple tests. - The least one can do, without any help of a statistcal package, is to reduce the significance level for each test (Bonferroni).

34 Multiple testing Page 34 of 34