Effect Sizes Based on Means

Transcription

1 CHAPTER 4 Effect Sizes Based on Means Introduction Raw (unstardized) mean difference D Stardized mean difference, d g Resonse ratios INTRODUCTION When the studies reort means stard deviations, the referred effect size is usually the raw mean difference, the stardized mean difference, or the resonse ratio. These effect sizes are discussed in this chater. RAW (UNSTANDARDIZED) MEAN DIFFERENCE D When the outcome is reorted on a meaningful scale all studies in the analysis use the same scale, the meta-analysis can be erformed directly on the raw difference in means (henceforth, we will use the more common term, raw mean difference). The rimary advantage of the raw mean difference is that it is intuitively meaningful, either inherently (for examle, blood ressure, which is measured on a known scale) or because of widesread use (for examle, a national achievement test for students, where all relevant arties are familiar with the scale). Consider a study that reorts means for two grous (Treated Control) suose we wish to comare the means of these two grous. Let 1 2 be the true (oulation) means of the two grous. The oulation mean difference is defined as D ¼ 1 2 : ð4:1þ In the two sections that follow we show how to comute an estimate D of this arameter its variance from studies that used two indeendent grous from studies that used aired grous or matched designs. Introduction to Meta-Analysis. Michael Borenstein, L. V. Hedges, J. P. T. Higgins H. R. Rothstein 2009 John Wiley & Sons, Ltd. ISBN:

2 22 Effect Size Precision Comuting D from studies that use indeendent grous We can estimate the mean difference D from a study that used two indeendent grous as follows. Let X 1 X 2 be the samle means of the two indeendent grous. The samle estimate of D is just the difference in samle means, namely D ¼ X 1 X 2 : ð4:2þ Note that uercase D is used for the raw mean difference, whereas lowercase d will be used for the stardized mean difference (below). Let S 1 S 2 be the samle stard deviations of the two grous, n 1 n 2 be the samle sizes in the two grous. If we assume that the two oulation stard deviations are the same (as is assumed to be the case in most arametric data analysis techniques), so that , then the variance of D is V D 5 n 1 þ n 2 n 1 n 2 S 2 ooled ; ð4:3þ where sffiffiffiffiffiffiffiffiffiffi S ooled 5 ðn 1 1ÞS 2 1 þðn 2 1ÞS 2 2 : n 1 þ n 2 2 ð4:4þ If we don t assume that the two oulation stard deviations are the same, then the variance of D is V D 5 S2 1 n 1 þ S2 2 n 2 : ð4:5þ In either case, the stard error of D is then the square root of V, ffiffiffiffiffiffi V D: ð4:6þ For examle, suose that a study has samle means X , X , samle stard deviations S , S , samle sizes n 1 5 n The raw mean difference D is D 5 103:00 100:00 5 3:00: If we assume that then the ooled stard deviation within grous is sffiffiffiffi ð50 1Þ5:5 2 þ ð50 1Þ4:5 2 S ooled 5 5 5:0249: 50 þ 50 2 The variance stard error of D are given by 50 þ 50 V D : :0100; 1: :0050:

3 Chater 4: Effect Sizes Based on Means If we do not assume that then the variance stard error of D are given by V D 5 5:52 50 þ 4: :0100 1: :0050: In this examle formulas (4.3) (4.5) yield the same result, but this will be true only if the samle size /or the estimate of the variances is the same in the two grous. Comuting D from studies that use matched grous or re-ost scores The revious formulas are aroriate for studies that use two indeendent grous. Another study design is the use of matched grous, where airs of articiants are matched in some way (for examle, siblings, or atients at the same stage of disease), with the two members of each air then being assigned to different grous. The unit of analysis is the air, the advantage of this design is that each air serves as its own control, reducing the error term increasing the statistical ower. The magnitude of the imact deends on the correlation between (for examle) siblings, with a higher correlation yielding a lower variance ( increased recision). The samle estimate of D is just the samle mean difference, D. If we have the difference score for each air, which gives us the mean difference X diff the stard deviation of these differences (S diff ), then D 5 X diff ; ð4:7þ V D 5 S2 diff n ; ð4:8þ where n is the number of airs, ffiffiffiffiffiffi V D: ð4:9þ For examle, if the mean difference is 5.00 with stard deviation of the difference of n of 50 airs, then D 5 5:0000; V D 5 10: :0000; ð4:10þ ffiffiffiffiffiffiffiffiffi 2:00 5 1:4142: ð4:11þ

4 24 Effect Size Precision Alternatively, if we have the mean stard deviation for each set of scores (for examle, siblings A B), the difference is D ¼ X 1 X 2 : ð4:12þ The variance is again given by V D 5 S2 diff n ; ð4:13þ where n is the number of airs, the stard error is given by ffiffiffiffiffiffi V D: ð4:14þ However, in this case we need to comute the stard deviation of the difference scores from the stard deviation of each sibling s scores. This is given by qffiffiffiffiffiffiffiffiffiffiffiffi S diff 5 S 2 1 þ S2 2 2 r S 1 S 2 ð4:15þ where r is the correlation between siblings in matched airs. If S 1 5 S 2, then (4.15) simlifies to qffiffiffiffiffiffiffiffi S diff 5 2 S 2 ooledð1 rþ: ð4:16þ In either case, as r moves toward 1.0 the stard error of the aired difference will decrease, when r 5 0 the stard error of the difference is the same as it would be for a study with two indeendent grous, each of size n. For examle, suose the means for siblings A B are , with stard deviations 10 10, the correlation between the two sets of scores is 0.50, the number of airs is 50. Then D 5 105:00 100:00 5 5:0000; V D 5 10: :0000; ffiffiffiffiffiffiffiffiffi 2:00 5 1:4142: In the calculation of V D, the S diff is comuted using ffiffiffiffiffiffiffiffiffiffiffi S diff þ : :0000 or qffiffiffiffiffiffiffiffiffi S diff ð1 0:50Þ 5 10:0000: The formulas for matched designs aly to re-ost designs as well. The re ost means corresond to the means in the matched grous, n is the number of subjects, r is the correlation between re-scores ost-scores.

5 Chater 4: Effect Sizes Based on Means Calculation of effect size estimates from information that is reorted When a researcher has access to a full set of summary data such as the mean, stard deviation, samle size for each grou, the comutation of the effect size its variance is relatively straightforward. In ractice, however, the researcher will often be working with only artial data. For examle, a aer may ublish only the -value, means samle sizes from a test of significance, leaving it to the meta-analyst to back-comute the effect size variance. For information on comuting effect sizes from artial information, see Borenstein et al. (2009). Including different study designs in the same analysis Sometimes a systematic review will include studies that used indeendent grous also studies that used matched grous. From a statistical ersective the effect size (D) has the same meaning regardless of the study design. Therefore, we can comute the effect size variance from each study using the aroriate formula, then include all studies in the same analysis. While there is no technical barrier to using different study designs in the same analysis, there may be a concern that studies which used different designs might differ in substantive ways as well (see Chater 40). For all study designs (whether using indeendent or aired grous) the direction of the effect (X 1 X 2 or X 2 X 1 ) is arbitrary, excet that the researcher must decide on a convention then aly this consistently. For examle, if a ositive difference will indicate that the treated grou did better than the control grou, then this convention must aly for studies that used indeendent designs for studies that used re-ost designs. In some cases it might be necessary to reverse the comuted sign of the effect size to ensure that the convention is followed. STANDARDIZED MEAN DIFFERENCE, d AND g As noted, the raw mean difference is a useful index when the measure is meaningful, either inherently or because of widesread use. By contrast, when the measure is less well known (for examle, a rorietary scale with limited distribution), the use of a raw mean difference has less to recommend it. In any event, the raw mean difference is an otion only if all the studies in the meta-analysis use the same scale. If different studies use different instruments (such as different sychological or educational tests) to assess the outcome, then the scale of measurement will differ from study to study it would not be meaningful to combine raw mean differences. In such cases we can divide the mean difference in each study by that study s stard deviation to create an index (the stardized mean difference) that would be comarable across studies. This is the same aroach suggested by Cohen (1969, 1987) in connection with describing the magnitude of effects in statistical ower analysis.

6 Effect Size Precision The stardized mean difference can be considered as being comarable across studies based on either of two arguments (Hedges Olkin, 1985). If the outcome measures in all studies are linear transformations of each other, the stardized mean difference can be seen as the mean difference that would have been obtained if all data were transformed to a scale where the stard deviation within-grous was equal to 1.0. The other argument for comarability of stardized mean differences is the fact that the stardized mean difference is a measure of overla between distributions. In this telling, the stardized mean difference reflects the difference between the distributions in the two grous ( how each reresents a distinct cluster of scores) even if they do not measure exactly the same outcome (see Cohen, 1987, Grissom Kim, 2005). Consider a study that uses two indeendent grous, suose we wish to comare the means of these two grous. Let 1 1 be the true (oulation) mean stard deviation of the first grou let 2 2 be the true (oulation) mean stard deviation of the other grou. If the two oulation stard deviations are the same (as is assumed in most arametric data analysis techniques), so that , then the stardized mean difference arameter or oulation stardized mean difference is defined as : ð4:17þ In the sections that follow, we show how to estimate from studies that used indeendent grous, from studies that used re-ost or matched grou designs. It is also ossible to estimate from studies that used other designs (including clustered designs) but these are not addressed here (see resources at the end of this Part). We make the common assumtion that , which allows us to ool the estimates of the stard deviation, do not address the case where these are assumed to differ from each other. Comuting d g from studies that use indeendent grous We can estimate the stardized mean difference () from studies that used two indeendent grous as d 5 X 1 X 2 S within : ð4:18þ In the numerator, X 1 X 2 are the samle means in the two grous. In the denominator S within is the within-grous stard deviation, ooled across grous, sffiffiffiffiffiffiffiffiffiffi ðn 1 1ÞS 2 1 S within 5 þðn 2 1ÞS 2 2 n 1 þ n 2 2 ð4:19þ where n 1 n 2 are the samle sizes in the two grous, S 1 S 2 are the stard deviations in the two grous. The reason that we ool the two samle

7 Chater 4: Effect Sizes Based on Means estimates of the stard deviation is that even if we assume that the underlying oulation stard deviations are the same (that is ), it is unlikely that the samle estimates S 1 S 2 will be identical. By ooling the two estimates of the stard deviation, we obtain a more accurate estimate of their common value. The samle estimate of the stardized mean difference is often called Cohen s d in research synthesis. Some confusion about the terminology has resulted from the fact that the index, originally roosed by Cohen as a oulation arameter for describing the size of effects for statistical ower analysis is also sometimes called d. In this volume we use the symbol to denote the effect size arameter d for the samle estimate of that arameter. The variance of d is given (to a very good aroximation) by V d 5 n 1 þ n 2 n 1 n 2 d 2 þ 2ðn 1 þ n 2 Þ : ð4:20þ In this equation the first term on the right of the equals sign reflects uncertainty in the estimate of the mean difference (the numerator in (4.18)), the second reflects uncertainty in the estimate of S within (the denominator in (4.18)). The stard error of d is the square root of V d, SE d 5 ffiffiffiffiffi V d: ð4:21þ It turns out that d has a slight bias, tending to overestimate the absolute value of in small samles. This bias can be removed by a simle correction that yields an unbiased estimate of, with the unbiased estimate sometimes called Hedges g (Hedges, 1981). To convert from d to Hedges g we use a correction factor, which is called J. Hedges (1981) gives the exact formula for J, but in common ractice researchers use an aroximation, 3 J 5 1 4df 1 : ð4:22þ In this exression, df is the degrees of freedom used to estimate S within, which for two indeendent grous is n 1 þ n 2 2. This aroximation always has error of less than less than ercent when df 10 (Hedges, 1981). Then, g 5 J d; V g 5 J 2 V d ; ð4:23þ ð4:24þ SE g 5 ffiffiffiffiffi V g: ð4:25þ For examle, suose a study has samle means X , X , samle stard deviations S , S , samle sizes n 1 5 n We would estimate the ooled-within-grous stard deviation as

8 28 Effect Size Precision sffiffiffiffi ð50 1Þ5:5 2 þ ð50 1Þ4:5 2 S within 5 5 5:0249: 50 þ 50 2 Then, d 5 5 0:5970; 5: þ 50 V d þ 0: þ ð 50Þ 5 0:0418; SE d 5 0: :2044: The correction factor (J), Hedges g, its variance stard error are given by 3 J :9923; g 5 0:9923 0: :5924; v g 5 0: : :0411; SE g 5 0: :2028: The correction factor (J) is always less than 1.0, so g will always be less than d in absolute value, the variance of g will always be less than the variance of d. However, J will be very close to 1.0 unless df is very small (say, less than 10) so (as in this examle) the difference is usually trivial (Hedges, 1981). Some slightly different exressions for the variance of d ( g) have been given by different authors even the same authors at different times. For examle, the denominator of the second term of the variance of d is given here as 2(n 1 þ n 2 ). This exression is obtained by one method (assuming the n s become ffiffiffi large with fixed). An alternate derivation (assuming n s become large with n fixed) leads to a denominator in the second term that is slightly different, namely 2(n 1 þ n 2 2). Unless n 1 n 2 are very small, these exressions will be almost identical. Similarly, the exression given here for the variance of g is J 2 times the variance of d, but many authors ignore the J 2 term because it is so close to unity in most cases. Again, while it is referable to include this correction factor, the inclusion of this factor is likely to make little ractical difference. Comuting d g from studies that use re-ost scores or matched grous We can estimate the stardized mean difference () from studies that used matched grous or re-ost scores in one grou. The formula for the samle estimate of d is

9 Chater 4: Effect Sizes Based on Means d 5 Y diff S within 5 Y 1 Y 2 S within : ð4:26þ This is the same formula as for indeendent grous (4.18). However, when we are working with indeendent grous the natural unit of deviation is the stard deviation within grous so this value is tyically reorted (or easily imuted). By contrast, when we are working with matched grous, the natural unit of deviation is the stard deviation of the difference scores, so this is the value that is likely to be reorted. To comute d from the stard deviation of the differences we need to imute the stard deviation within grous, which would then serve as the denominator in (4.26). Concretely, when working with a matched study, the stard deviation within grous can be imuted from the stard deviation of the difference, using S diff S within 5 ffiffiffi ; ð4:27þ 2ð1 rþ where r is the correlation between airs of observations (e.g., the retest-osttest correlation). Then we can aly (4.26) to comute d. The variance of d is given by V d 5 1 n þ d2 21 ð rþ; ð4:28þ 2n where n is the number of airs. The stard error of d is just the square root of V d, SE d 5 ffiffiffiffiffi V d : ð4:29þ Since the correlation between re- ost-scores is required to imute the stard deviation within grous from the stard deviation of the difference, we must assume that this correlation is known or can be estimated with high recision. Otherwise we may estimate the correlation from related studies, ossibly erform a sensitivity analysis using a range of lausible correlations. To comute Hedges g associated statistics we would use formulas (4.22) through (4.25). The degrees of freedom for comuting J is n 1, where n is the number of airs. For examle, suose that a study has re-test ost-test samle means X , X , samle stard deviation of the difference S diff 5 5.5, samle size n 5 50, a correlation between re-test ost-test of r The stard deviation within grous is imuted from the stard deviation of the difference by 5:5 S within 5 ffiffiffiffiffiffiffiffi 5 7:1005: 21 ð 0:7Þ Then d, its variance stard error are comuted as d 5 5 0:4225; 7:1000

10 30 Effect Size Precision v d þ 0:42252 ð21 ð 0:7ÞÞ5 0:0131; 2 50 SE d 5 0: :1143: The correction factor J, Hedges g, its variance stard error are given by 3 J :9846; g 5 0:9846 0: :4160; V g 5 0: : :0127; SE g 5 0: :1126: Including different study designs in the same analysis As we noted earlier, a single systematic review can include studies that used indeendent grous also studies that used matched grous. From a statistical ersective the effect size (d or g) has the same meaning regardless of the study design. Therefore, we can comute the effect size variance from each study using the aroriate formula, then include all studies in the same analysis. While there are no technical barriers to using studies with different designs in the same analysis, there may be a concern that these studies could differ in substantive ways as well (see Chater 40). For all study designs the direction of the effect (X 1 X 2 or X 2 X 1 ) is arbitrary, excet that the researcher must decide on a convention then aly this consistently. For examle, if a ositive difference indicates that the treated grou did better than the control grou, then this convention must aly for studies that used indeendent designs for studies that used re-ost designs. It must also aly for all outcome measures. In some cases (for examle, if some studies defined outcome as the number of correct answers while others defined outcome as the number of mistakes) it will be necessary to reverse the comuted sign of the effect size to ensure that the convention is alied consistently. RESPONSE RATIOS In research domains where the outcome is measured on a hysical scale (such as length, area, or mass) is unlikely to be zero, the ratio of the means in the two grous might serve as the effect size index. In exerimental ecology this effect size index is called the resonse ratio (Hedges, Gurevitch, & Curtis, 1999). It is imortant to recognize that the resonse ratio is only meaningful when the outcome

11 Chater 4: Effect Sizes Based on Means Study A Resonse ratio Log resonse ratio Study B Resonse ratio Log resonse ratio Study C Resonse ratio Log resonse ratio Summary Resonse ratio Summary Log resonse ratio Figure 4.1 Resonse ratios are analyzed in log units. is measured on a true ratio scale. The resonse ratio is not meaningful for studies (such as most social science studies) that measure outcomes such as test scores, attitude measures, or judgments, since these have no natural scale units no natural zero oints. For resonse ratios, comutations are carried out on a log scale (see the discussion under risk ratios, below, for an exlanation). We comute the log resonse ratio the stard error of the log resonse ratio, use these numbers to erform all stes in the meta-analysis. Only then do we convert the results back into the original metric. This is shown schematically in Figure 4.1. The resonse ratio is comuted as R 5 X 1 X 2 ð4:30þ where X 1 is the mean of grou 1 X 2 is the mean of grou 2. The log resonse ratio is comuted as lnr 5 lnðrþ 5 ln X 1 5 ln X 1 ln X2 : ð4:31þ X 2 The variance of the log resonse ratio is aroximately V lnr 5 S 2 ooled þ 2!; ð4:32þ n 1 X 1 n 2 X 2 where S ooled is the ooled stard deviation. The aroximate stard error is SE ln R 5 ffiffiffiffiffiffiffiffi : ð4:33þ Note that we do not comute a variance for the resonse ratio in its original metric. Rather, we use the log resonse ratio its variance in the analysis to yield V lnr

12 32 Effect Size Precision a summary effect, confidence limits, so on, in log units. We then convert each of these values back to resonse ratios using R 5 exðlnrþ; LL R 5 exðll lnr Þ; UL R 5 exðul lnr Þ; ð4:34þ ð4:35þ ð4:36þ where LL UL reresent the lower uer limits, resectively. For examle, suose that a study has two indeendent grous with means X , X , ooled within-grou stard deviation , samle size n 1 5 n Then R, its variance stard error are comuted as R 5 61:515 51: :2058; lnr 5 lnð1:2058þ5 0:1871; V lnr 5 19: ð61:515þ 2 þ 1 10 ð51:015þ 2 SE lnr 5 0: :1581:! 5 0:0246: SUMMARY POINTS The raw mean difference (D) may be used as the effect size when the outcome scale is either inherently meaningful or well known due to widesread use. This effect size can only be used when all studies in the analysis used recisely the same scale. The stardized mean difference (d or g) transforms all effect sizes to a common metric, thus enables us to include different outcome measures in the same synthesis. This effect size is often used in rimary research as well as meta-analysis, therefore will be intuitive to many researchers. The resonse ratio (R) is often used in ecology. This effect size is only meaningful when the outcome has a natural zero oint, but when this condition holds, it rovides a unique ersective on the effect size. It is ossible to comute an effect size variance from studies that used two indeendent grous, from studies that used matched grous (or re-ost designs) from studies that used clustered grous. These effect sizes may then be included in the same meta-analysis.