Multiple Comparisons

Transcription

1 Multiple Comparisons October 16th & 18th, 2007 Reading: Chapter 7 HH Multiple Comparisons p. 1/2

2 Simultaneous Inferences Individual tests of hypotheses or confidence intervals Suppose you try to control the Type I error of K hypothesis tests at level α. Pr(at least one Type I error) = 1 - (1 α) K. This leads to an unacceptable error threshold. Multiple comparison procedure: control the family-wise error rate (FWE): FWE = Pr(reject at least one true hypothesis under any configuration of true and false hypotheses) FDR = false discovery rate (the expected proportion of falsely rejected hypotheses). Multiple Comparisons p. 2/2

3 Common Multiple Comparisons Procedure Bonferroni method Tukey procedure Dunnett procedure Scheffe and Extended Tukey: simultaneously comparing all possible contrasts. Multiple Comparisons p. 3/2

4 Bonferroni Method Bonferroni inequality: P( R i ) P(R i ). Perform m related tests and conduct each test at level : FWE α. α m Conservative multiple comparison procedure Useful in situations when the statistics associated with the m inferences have nonidentical probability distributions. Multiple Comparisons p. 4/2

5 Tukey Procedure Examines all pairwise comparisons by making use of the information about the joint distribution of the statistics used in the inferences. Less conservative than Bonferroni. If there are a groups, there will be ( a 2) pairwise tests. Confidence intervals are constructed using critical values from the Studentized range distribution. Intervals based on the Studentized range statistic, Tukey Honest Significant Differences method. See ptukey and qtukey functions in R. Multiple Comparisons p. 5/2

6 More on Tukey s method Confidence level exact when sample sizes are equal across the a groups. If the sample sizes are unequal, confidence intervals are conservative. R adjusts for slightly unbalanced design see comments in help(tukeyhsd). In R, use either TukeyHSD or simint with type= Tukey option. Multiple Comparisons p. 6/2

7 Tukey Graphical Output Female mice diet example 95% family wise confidence level N.N85 lopro N.R40 lopro N.R50 lopro NP lopro R.R50 lopro N.R40 N.N85 N.R50 N.N85 NP N.N85 R.R50 N.N85 N.R50 N.R40 NP N.R40 R.R50 N.R40 NP N.R50 R.R50 N.R50 R.R50 NP Differences in mean levels of DIET Multiple Comparisons p. 7/2

8 Dunnett s Procedure Compare the mean of one population with each of the means of the remaining populations (e.g., compare a control to different treatments). Uses the percentiles of a marginal distribution of a multivariate t distribution. Multiple Comparisons p. 8/2

9 Example of Dunnett s Procedure Random sample of 50 men matched for pounds overweight was randomly separated into 5 equal groups. Each group was given exactly one of the weight loss agents: A, B, C, D, or E. After a fixed period of time, each man s weight loss was recorded. Multiple Comparisons p. 9/2

10 Weight Loss Boxplots Weight Loss A B C D E group Multiple Comparisons p. 10/2

11 Dunnett Output Dunnett contrasts groupa groupd ( groupb groupd ( groupc groupd ( groupe groupd ( % one sided confidence intervals Multiple Comparisons p. 11/2

12 Scheffe s Method Simultaneously compare all possible contrasts. Uses a percentile of an F distribution to construct simultaneous confidence intervals a a c c j ȳ j ± (a 1)F 0.05,a 1,N a s 2 j s = ˆσ j=1 N = a j=1 n j j=1 Results specify the constrast s significant difference from zero. n j Multiple Comparisons p. 12/2

13 Turkey Data Six turkeys were randomly assigned to each of 5 diet groups and fed for the same length of time. Control diet A1: control + amount 1 of additive A A2: control + amount 2 of additive A B1: control + amount 1 of additive B B2: control + amount 2 of additive B Multiple Comparisons p. 13/2

14 Turkey Boxplots 10 8 Weight Gain 6 4 control A1 A2 B1 B2 diet Multiple Comparisons p. 14/2

15 Scheffe simultaneous 95% CI control vs treatment: (-4.28, -2.58) A vs B: (-2.71, -1.19) amount: (-2.69, -1.17) A vs B by amount: (-0.312, 1.21) Multiple Comparisons p. 15/2

16 Contrast Analysis treatment vs control: averaged over the 4 treatments, turkeys receiving a dietary additive gain significantly more weight than ones not receiving an additive. additive: turkeys receiving additive B gain significantly more weight than turkeys receiving additive A. amount: turkeys receiving amount 2 gain significantly more weight than turkeys receiving amount 1. interaction between additive and amount: the extent of increased weight gain as a result of receiving amount 2 rather than amount 1 is not significantly different for the two additives Multiple Comparisons p. 16/2

17 Extended Tukey Procedure Can modify Tukey procedure to cover the family of all possible contrasts when the sample sizes are equal across groups. a j=1 c j ȳ j ± q α 2 s a n j=1 c j Wider CI compared to Scheffe s method. Appropriate for situations with a small number of more complicated contrasts. Multiple Comparisons p. 17/2

18 Graphical Displays The standard tabular and graphical outputs do not convey some aspects of a multiple comparison analysis. For example, in the Tukey pairwise comparison, the standard output just shows the CI for the difference. The mean of each group being compared is obscured. The standard displays do not show the relative distances between adjacent sorted sample means. Multiple Comparisons p. 18/2

19 MMC Plot Mean-Mean Multiple Comparisons displays Horizontal axis shows the contrast value (e.g., for a comparison between two groups, it would show the difference between the two sample means). Vertical axis shows the sample mean of each subgroup. This allows visualization of the relative distances between the sample means of the different subgroups. Multiple Comparisons p. 19/2

20 Example with Turkey data Pairwise confidence intervals from Tukey procedure: R code > turkeyci <- simint(wt.gain diet, data=turk type= Tukey ) > plot(turkeyci) Multiple Comparisons p. 20/2

21 Pairwise confidence intervals for turkey da Tukey contrasts dieta1 dietcontrol ( ) dieta2 dietcontrol ( ) dietb1 dietcontrol ( ) dietb2 dietcontrol ( ) dieta2 dieta1 ( ) dietb1 dieta1 ( ) dietb2 dieta1 ( ) dietb1 dieta2 ( ) dietb2 dieta2 ( ) dietb2 dietb1 ( ) % two sided confidence intervals Multiple Comparisons p. 21/2

22 MMC plot with Turkey data R code > tmp0 <- t(simint(wt.gain diet, data=turkey type= Tukey )$cmatrix > tmp1 <- simint.mmc(wt.gain diet, data=turk method= Tukey, whichf= diet, lmat=tm lmat.rows=2:6) > plot(tmp1) Multiple Comparisons p. 22/2

23 Pairwise confidence intervals for turkey da B2 multiple comparisons of means of wt.gain simultaneous 95% confidence limits, Tukey method dietb2 dieta2 dietb2 dietb1 dietb2 dieta A2 B1 dietb1 dieta2 B2 B2 dietb2 dietcontrol dieta2 dieta1 dietb1 dieta1 5.5 A1 B1 A2 A2 B1 dieta2 dietcontrol dietb1 dietcontrol A1 A1 dieta1 dietcontrol 33333control ean.gain level control contrast value contrast Multiple Comparisons p. 23/2