Multiple Comparisons. Cohen Chpt 13

Transcription

1 Multiple Comparisons Cohen Chpt 13

2 How many t-tests? We do an experiment, 1 factor, 3 levels (= 3 groups). The ANOVA gives us a significant F-value. What now? 4 levels, 1 factor: how many independent comparisons? What is the probability of making at least one false alarm?

3 Familywise alpha Probability that we will have one or more false alarms = familywise alpha = experimentwise alpha Increases as number of possible comparisons increases For 7 levels, we have a.66 chance of making at least one F. A.

4 Other possible comparisons As well as asking about each possible comparison between 2 groups, complex comparisons are also possible: Do groups a,b,c collectively differ from groups d,e,f,g? Are treatments more effective on weekends than weekdays? Any arbitrary combination of group means can be compared to any other such combination!!! Honesty is the best policy: What do you set out to test? What do you decide to test afterwards?

5 Basic test types: 1. Planned comparisons a priori few in number theoretically motivated 2. Post hoc comparisons based on looking at the data exploratory risky business

6 Post hoc tests

7 Strategy 1: Fisher s Protected t Basic strategy: only perform pairwise t-tests if the ANOVA provides a significant F. Insight: requiring significant F reduces the number of opportunities for false alarms. Problem: opportunities remain, especially with many levels. Solution: Don t use it! (Use it for 3-group situations only if you like)

8 Strategy 2: Tukey s HSD Test Honestly Significant Difference Insight: The greatest chance of making a type 1 error (F.A.) arises in comparing the largest mean with the smallest. If we can protect against an F.A. in this case, all other comparisons are also protected. If this comparison is not significant, nor is any other one! Advantage: provides protection of familywise alpha

9 Doing Tukey HSD tests (equal n) Suppose we do an experiment with one factor, 5 levels. There is the ANOVA table: Source SS df MS F p Between <.05 Within (error) Here are the group means: I II III IV V

10 Here are the differences between the means. Which are significant? I V IV III II I V IV III II 82 0 Need to find a critical value for the difference between group means

11 The studentized range statistic, q This is a statistic (just like F or t), for which an expected distribution is known under the null hypothesis of no differences. However, we don t compute this for our data. Rather, we use this formula to figure out a critical value for the numerator thus: set alpha to desired level (.05) use tables to obtain critical value of q for appropriate degrees of freedom work backward to get critical difference between means

12 1. Get critical q using correct degrees of freedom 1. Numerator = no of groups 2. Denominator = d.f. for MS W (use closest available) Here, d.f. = 5, 55, q crit = 3.98 (Table A11 in Cohen) = in this case Which pairs of means exceed this? 1 & 2, 1 & 3

13 Tukey HSD in R If you have done an ANOVA, it is only a short step to doing a HSD in R (this is new in R, hurrah!) >myaov = aov(score ~ rate, dat) >summary(myaov) Df Sum Sq Mean Sq F value Pr(>F) rate ** Residuals

14 > TukeyHSD(myaov, ordered=t) Tukey multiple comparisons of means 95% family-wise confidence level factor levels have been ordered Fit: aov(formula = score ~ rate, data = dat) $rate diff lwr upr slow-fast normal-fast normal-slow

15 Practical use of Tukey s HSD Fine if all groups have equal n Used only for post hoc testing: stronger tests are available if only a small number of comparisons are to be made (planned comparison) Computation is a little trickier if group sizes are a little unequal (use R) Do not use if group sizes are very unequal. Several other post hoc methods exist (Newman-Keuls, Scheffe, Bonferroni). Have fun!

16 Planned comparisons Post hoc comparisons are fine for exploratory studies. Ideally, however, we know in advance which differences we expect to find in our data. and these may or may not be simple differences between two means. Unlike post hoc tests, these can be done whether or not the ANOVA is significant.

17 How many comparisons? The biggest problem with post hoc comparisons was the proliferation of possible comparisons. Planned comparisons put a lid on that. For a single factor with 6 levels, there are at most 5 (n-1) independent comparisons among means possible. Independent comparisons are also called orthogonal contrasts.

18 Planned Comparison: an Example A study by the UCD Institute of Blatant Propaganda looked at the influence of the medium used (lecture, movie) on tendency of subjects to change their attitude towards Fianna Fail. The media used were: A movie, favourable to FF A lecture on FF, also favourable A combination of lecture and movie. Subjects were assigned at random to groups, each having been given a preliminary attitude test. After the treatment, each was tested again, and the change in attitude was the dependent variable. Based on an example in Hays, 1988

19 Problem: mere repetition of the test may affect a subject s score, so a control group without any exposure to lecture or movie was included. Problem: Perhaps seeing any movie or hearing any lecture would cause a change in score. so 2 more control groups were introduced: Experimental Groups Control Groups I II III IV V VI Movie Lecture Mov + Lec Nothing Neutral Neutral Movie Lecture

20 The investigators now had the following specific questions: [1] Do the experimental groups (as a whole) differ from the control groups? [2] Among Experimental groups, is the Movie+Lecture different from average effect of either Movie alone or Lecture alone? [3] Is the Experimental Lecture different from the Experimental Movie? [4] Among control groups, does Nothing differ from either Movie or Lecture? Each specific question can be expressed as a comparison among sample means..

21 [1] Do the experimental groups (as a whole) differ from the control groups? Experimental Groups Control Groups I II III IV V VI Movie Lecture Mov + Lec Nothing Neutral Neutral Movie Lecture Comparison weights

22 [2] Among Experimental groups, is the Movie+Lecture different from average effect of either Movie alone or Lecture alone? Experimental Groups Control Groups I II III IV V VI Movie Lecture Mov + Lec Nothing Neutral Neutral Movie Lecture

23 [3] Is the Experimental Lecture different from the Experimental Movie? Experimental Groups Control Groups I II III IV V VI Movie Lecture Mov + Lec Nothing Neutral Neutral Movie Lecture

24 [4] Among control groups, does Nothing differ from either Movie or Lecture? Experimental Groups Control Groups I II III IV V VI Movie Lecture Mov + Lec Nothing Neutral Neutral Movie Lecture

25 Planned Comparisons.. And there we stop with planned comparisons. Cohen does not provide sufficient detail to do these with confidence, so if you want to do a planned comparison, you will need a more detailed reference work (e.g. Hays, (1988), Statistics, Harcourt Brace Jovanovich. Remember though: if you have strong and precise theoretical questions before you run the experiment, planned comparisons are considerably more powerful than post hoc tests.