On Knowing What We Don't Know: An Empirical Comparison of Methods to Detect Publication Bias in Meta-Analysis

Transcription

1 Publication Bias 1 On Knowing What We Don't Know: An Empirical Comparison of Methods to Detect Publication Bias in Meta-Analysis Gianna Rendina-Gobioff Jeffrey D. Kromrey University of South Florida Paper presented at the annual meeting of the American Educational Research Association, San Diego, April 12 16, 2004

2 Publication Bias 2 On Knowing What We Don't Know: An Empirical Comparison of Methods to Detect Publication Bias in Meta-Analysis Abstract The performance of methods for detecting publication bias in meta-analysis was evaluated using Monte Carlo methods. Four methods of bias detection were investigated: Begg s Tau, Egger s Regression, Funnel Plot Regression and Trim and Fill. Five factors were included in the simulation design: number of primary studies in each meta-analysis, sample sizes of primary studies, population variances in primary studies, magnitude of population effect size and magnitude of selection bias. Results were evaluated in terms of Type I error control and statistical power. Results suggest poor Type I error control in many conditions for all of the methods examined except Begg s Tau using sample size rather than the estimated variance. Statistical power was typically very low for conditions in which Type I error rates were adequately controlled, although power increased with larger sample sizes in the primary studies and larger numbers of studies in the meta-analysis.

3 Publication Bias 3 On Knowing What We Don't Know: An Empirical Comparison of Methods to Detect Publication Bias in Meta-Analysis As researchers we rely heavily on the literature available through professional publications to inspire and guide future studies. Meta-analytic syntheses of research provide important summaries of collections of empirical studies and indications of directions that future inquiry should take. However, due to publication bias we cannot assume that the resources available are complete and accurate. One type of publication bias is selection bias that occurs when the decision to publish, either by the researcher or the editorial staff of a journal, is influenced by research results (often the attainment of statistical significance). Essentially, when researchers do not submit studies or publishers reject submissions with small treatment effects or non-significant results, the resource of literature on the topic of interest becomes biased (Thornton & Lee, 2000). When the decision to publish is based on statistical significance there is a direct relationship between publication probability and sample size. Essentially, the greater the sample size the greater the chances of a statistically significant result and thus a greater chance of being published. Research with small samples and large treatment effects are more likely to be published than small sample studies with smaller treatment effects. This results in a relationship between the effect sizes and sample sizes in the published literature. Other study design factors that may influence publication bias are the failure to use randomization, the absence of a control group and the heterogeneity of treatment effects related to moderating variables (Begg, 2001; Macaskill, Walter, & Irwig, 2001). The implications of publication bias for meta-analysis research methods are very important. Due to practical issues the selection of articles for meta-analysis is usually restricted to published articles. Although non-published research may be included, it is impossible to collect all the potential research endeavors for a topic when one does not know that they exist. Essentially, the file drawers of all researchers are not made available to the meta-analyst (Rosenthal, 1979). The investigation and accurate

4 Publication Bias 4 identification of publication bias is extremely important because the results from metaanalytic studies provide a summary of effect sizes for a pool of research on a topic. In other words the conclusions drawn from meta-analysis research have a greater impact than one research study on a topic. Although there are statistical methods that have been suggested for correcting for publication bias (Hedges, 1992; Hedges & Vevea, 1996), before one corrects for bias one must detect its presence. Typically researchers have examined a funnel plot to detect publication bias, yet recent literature provides formal significance tests. However, because these methods often provide conflicting results, research that provides a comparison across the various methods is needed (Sutton, Abrams, Jones, Sheldon, & Song, 2000). Therefore, the focus of this research was to examine methods for detecting publication bias. Specifically, this research empirically compared the performance of four methods for identifying publication bias: Begg's Tau, Egger's regression, funnel plot regression and the trim and fill method. Analysis Options for Detecting Publication Bias As an aid for the presentation of methods for detecting publication bias, a hypothetical sample of 10 studies was simulated with an average sample size of five observations in each of two groups. For each study, the sample effect size was calculated. These data (presented in Table 1) will be used to illustrate each method. Begg s Tau. Begg's Tau method (Begg & Mazumdar, 1994) examines the relationship between the standardized treatment effect and the variance of the treatment effect using Kendall's Tau. The standardized treatment effects are estimated as: d d = d B i i i * ν i Where d i is the i th observed study effect size d i is the weighted average effect size = wd i w i i

5 Publication Bias 5 Where w i is the weight or inverse of the effect size variance (vi) * 1 vi is the standardized variance of the treatment effect = v i ( v i ) The relationship between the standardized treatment effect and the variance is illustrated in Figure 1. Ranks are assigned for the observed standardized treatment effects and the variances of those treatment effects (alternatively, the sample sizes may be ranked rather than the estimated variances). The correlation between these ranked values (Kendall s tau) leads to a statistical test for the presence of publication bias. The standardized treatment effects ( d ) are provided in Table 1. In this sample of effect sizes the weighted mean effect size is and the sum of the reciprocals of the sampling variances is The standardized treatment effect for the first study is B i 1 d B 1 d d ( ) 1 i = = = * ν Cliff and Charlin (1991) derived the estimated sampling variance of Kendall s tau as 2 2 ( ) ( ) 2 tih 12 n n 1 t12 2 tih12 n n 1 t12 i h i h var( t12) = n n n n ( 1)( 2)( 3) where tih12 = a concordance indicator (equal to 1 if observations i and h are ranked in the same order on variables 1 and 2, equal to -1 if they have opposite ranks, and equal to zero if they are tied), and t12 = the sample estimate of tau. t12 The ratio z = is compared to a critical value from a standard normal var( t ) 12 distribution to provide a test of the null hypothesis that tau = 0. For these data,

6 Publication Bias z = = , p >.05, and the null hypothesis is not rejected. Thus, these data do not provide sufficient evidence that publication bias exists in this sample of studies. Egger's Regression Method. The Egger regression method (Egger, Davey Smith, Schneider & Minder, 1997) treats the standardized treatment effect as the criterion and the precision of effect size estimation (the inverse of its standard error) as the predictor in a regression model (estimated by either OLS or WLS, with observations weighted by the inverse of their variances). The standardized treatment effects are estimated as: d E i Where di is the observed study effect size for study i vi is the variance of the effect size for study i E For example, for study 1 in Table 1, d 1 = = For Egger s Regression Method, the precision of the effect size is estimated as: = Examining the example data, an OLS regression with the standardized treatment effect as the criterion and the precision as the predictor resulted in a slope of and intercept of (presented in Figure 2). Although not conducted for these sample data, this regression model can also be estimated with WLS regression. The slope of this regression equation provides an estimate of the true effect and the intercept is expected to have the value of zero when no publication bias exists. Thus, a test of the null hypothesis that the regression intercept equals zero provides a test of publication bias. In the example, the null hypothesis would not be rejected (t = , p >.05, df=8), thus these data do not suggest a relationship between standardized treatment effects and precision of estimation. Therefore, these data do not provide sufficient evidence that publication bias exists in this sample of studies. Funnel Plot Regression Method. The funnel plot regression method, suggested by Macaskill, Walter, and Irwig (2001), uses a regression model with the criterion variable d i ν i 1 v. 1

7 Publication Bias 7 being the treatment effect and study size being the predictor variable. Estimation by WLS is recommended for such a model, using as weights the inverse of the estimation 1 variance (e.g., ν ). For example, for study 1 in Table 1, the weight is: i v = = 1 A WLS regression with the treatment effect as the criterion and the study size as the predictor resulted in a slope of and intercept of for the example data (presented in Figure 3). In contrast to the Egger method, in this regression equation the slope will indicate no publication bias when it has the value of zero and the intercept in this regression equation will indicate the true effect. Thus, a test of the null hypothesis that the regression slope equals zero provides a test of publication bias. In the example, the null hypothesis would not be rejected (t = , p >.05, df = 8). Therefore, these data do not suggest a relationship between the observed effect sizes and sample size, and these data do not provide sufficient evidence that publication bias exists in this sample of studies. Trim and Fill Method. The Trim and Fill method, introduced by Duval and Tweedie (2000a, 2000b), is a nonparametric approach which is based on the funnel plot. Using symmetry assumptions the observed studies are ranked using the absolute values of the effect sizes; positive ranks for studies with effect sizes greater than the mean effect size, negative ranks for studies with effect sizes less than the mean effect size. The ranks are estimated as: r = rank d d * i i i ( ) ( ) ( ) * For example, for study 1 in Table 1, ( ) r1 = rank = rank = 8. A negative algebraic sign is assigned to ranks where the d i is less than the d i. For example, the rank for study 4 would be assigned a negative algebraic sign: ( ( )) ( ) * r rank rank 4 = = = 10

8 Publication Bias 8 Using these ranks the number of research studies missing from the funnel plot due to publication bias is estimated by: * R0 = γ 1= 0 1= 1 where R0 is the estimated number of studies concealed due to publication bias, * * γ = = = 0, k r h where k is the number of studies included in the meta-analysis, and * r h is the largest negative rank Publication bias is evidenced when R0 > 3, with power greater than 0.80 and =.05 (Duval & Tweedie, 2000a). In this example, the R0 is not greater than 3 and thus would suggest that these data do not provide sufficient evidence that publication bias exists in this sample of studies. Purpose of the Study Although the investigation of publication bias has received an abundance of attention in the physical sciences, the problem has been somewhat neglected in the social sciences. The need for research that examines publication bias under conditions that are applicable to the social sciences has driven this research. While Macaskill, Walter, and Irwig (2001) provided the framework for this study, the conditions chosen were selected to represent those commonly reported in the fields of psychology and education. The researchers hope that this research will bring attention in the social sciences to the issues and concerns surrounding publication bias in meta-analysis. Method This research was a Monte Carlo study designed to simulate meta-analyses. The use of simulation methods allows the control and manipulation of research design factors and the incorporation of sampling error into the analyses. Observations in primary studies were generated under known population conditions, and primary studies were combined to simulate meta-analyses.

9 Publication Bias 9 This simulation was modeled after that reported by Macaskill, Walter and Irwig (2001), but extends the conditions examined to more adequately represent the data structures evident in meta-analyses conducted in psychology and education. Each primary study consisted of two groups of observations (e.g., a treatment and control group). For each primary study, the Hedges g effect size (Hedges & Olkin, 1985) was calculated based on the simulated data. The Monte Carlo study included five factors in the design. These factors were (a) the number of primary studies in each meta-analysis (10, 20, 50, and 100), (b) the sample sizes of the two groups in each primary study (with mean sample sizes ranging from 5 in each group to 100 in each group, as well as unbalanced conditions), (c) group variances in the primary studies (variance ratios of 1:2, 1:4, and 1:8, as well as a homogeneous variance condition), (d) the magnitude of the population effect size ( = 0.00, 0.20, 0.50, 0.80), and (e) the magnitude of the selection bias (no bias, moderate bias, and strong bias). The selection bias in the meta-analyses was manipulated by generating a weight function to establish the probability of a simulated study being included in the metaanalysis, with the probability of inclusion being inversely related to the p-value obtained in the simulated study (Begg & Mazumdar, 1994). That is, samples that yield statistically significant results are more likely to be published than those that do not yield statistically significant results. The weight functions are illustrated in Figure 4. Table 2 provides the proportion of simulated studies that were available for meta-analysis based on the selection bias described above. As would be expected the proportion available/selected for meta-analysis was greater for the smaller publication bias than the larger publication bias. The overall range for the proportion selected across all conditions was from (few studies being selected) to (almost all studies being selected). In each simulated meta-analysis, publication bias was assessed using Begg s Tau method (using both the estimated variance and the sample size), Egger s regression method, the funnel plot regression method suggested by Macaskill, Walter and Irwig (2001) and the trim and fill method suggested by Duval and Tweedie (2000a, 2000b). For

10 Publication Bias 10 the Egger Regression, OLS estimation was employed because preliminary analyses suggested that WLS estimates were substantially biased. The proportion of simulated meta-analyses for which publication bias was indicated by each detection method was calculated. For simulation conditions with no publication bias, these proportions represent estimates of Type I error rates. Conversely, for conditions that included publication bias, these proportions estimate the statistical power. This research was conducted using SAS/IML version 8.2. Conditions for the study were run under Windows and Unix platforms. Normally distributed random variables were generated using the RANNOR random number generator in SAS. A different seed value for the random number generator was used in each execution of the program. The program code was verified by hand-checking results from benchmark datasets. For each condition examined in the Monte Carlo study, meta-analyses were simulated. The use of samples provides a maximum 95% confidence interval width around an observed proportion that is ± (Robey & Barcikowski, 1992). The relative effectiveness of the publication bias detection methods was evaluated in terms of Type I error control and statistical power. Results Type I Error Control The distributions of estimated Type I error rates across the conditions examined in this study are presented as box and whisker plots in Figure 5. In general, the use of Egger s Regression or Kendall s Tau based on sample estimated effect size variance, Tau (V), resulted in liberal Type I error control (with the estimated Type I error rates exceeding.90 in some conditions). Conversely, the funnel plot and the trim and fill methods evidenced conservative Type I error control in most conditions. Only Kendall s Tau based on sample sizes, Tau (N), resulted in Type I error control that was very close to the nominal alpha level across the majority of conditions examined.

11 Publication Bias 11 To determine the influence of the Monte Carlo study design factors on the Type I error control of these five tests, the proportions of simulation conditions that resulted in adequate Type I error control according to Bradley s (1975) liberal criterion were calculated. These proportions, disaggregated by the study design factors, are presented in Table 3 and the corresponding average estimated Type I error rates are presented in Table 4. For Egger s Regression, as the number of studies in each meta-analysis (k) increased, the proportion of conditions with adequate Type I error control decreased from 0.55 (with k = 10) to 0.17 (with k = 100). Table 4 indicates that this test became increasingly liberal with larger number of studies, yielding an average Type I error rate of 0.29 with k = 100. Similar results were evident with the use of Kendall s Tau (V), with the average estimated Type I error rate reaching as high as 0.22 with k = 100. Both the funnel plot and trim and fill methods evidenced the opposite relationship with k: with larger numbers of studies in the meta-analysis, the proportions of conditions with adequate Type I error control increased. The use of Kendall s Tau (N) maintained the average Type I error rate near the nominal alpha across values of K, although the proportion of conditions with adequate Type I error control decreased slightly as k increased, reaching a low of 0.94 with k = 100. The proportions of conditions with adequate Type I error control and the average estimated Type I error rates are graphed in Figures 6 and 7, respectively. With small sample sizes in the primary studies, Egger s Regression, Kendall s Tau (V) and the trim and fill approach evidenced adequate Type I error control in fewer than one quarter of the conditions (Figures 8-11). These proportions increased with larger samples, with Kendall s Tau (V) performing adequately in a larger proportion of conditions than either Egger s Regression or trim and fill (reaching as high as 0.89 with sample sizes of 40 and 60 in the two groups). With small samples, the funnel plot regression showed better Type I error control than did Egger s Regression, Kendall s Tau (V) or the trim and fill (except for conditions with positive pairing of sample size with variance), but the best Type I error control was evidenced by Kendall s Tau (N).

12 Publication Bias 12 For the latter test, the worst performance in Type I error control was for conditions with sample sizes of 40 and 60, in which only 91% of the conditions evidenced adequate control. All methods except the trim and fill approach evidenced decreasing performance as the population effect size increased (Figures 12 and 13). For example, Kendall s Tau (V) showed adequate control in 94% of the conditions with a population effect size of zero, but in only 17% of the conditions with a population effect size of 0.8. With larger population effect sizes, the Egger Regression, Kendall s Tau (V), and Trim and Fill became excessively liberal in Type I error control. The trim and fill approach evidenced its best performance with small population effects (41% of the conditions with an effect size of 0.2), but showed decreasing Type I error control in conditions with larger population effect sizes and in conditions with a population effect size of zero. The Kendall s Tau (N) test also became liberal with larger population effect sizes, but it still maintained adequate Type I error control in 93% of the conditions with a population effect size of 0.8. As expected, all of the methods showed decreasing performance in Type I error control under conditions of variance heterogeneity in the primary studies, with the average Type I error rates increasing with greater degrees of heterogeneity (Figures 14 and 15). With homogeneous variance conditions or conditions with a 1:2 variance ratio, all of the methods except the trim and fill maintained adequate Type I error control in more than half of the conditions examined. With the largest degree of heterogeneity examined (variance ratio of 1:8), the funnel plot method still maintained adequate control in 56% of the conditions, while the Kendall s Tau (N) test maintained control in 95% of the conditions. Statistical Power The statistical power of each test was estimated only for the conditions in which Type I error was adequately controlled (using Bradley s, 1975, liberal robustness criterion). An initial overview of these power results is provided in Table 5, which

13 Publication Bias 13 presents the maximum power estimates that were observed, disaggregated by the design factors of the Monte Carlo study. These results underscore the limited statistical power of these tests. For example, the highest power observed for Kendall s Tau (V) was only (a slightly higher maximum power, 0.671, was observed for Kendall s Tau (N)). The power estimates for Egger s Regression rarely exceeded 0.5 (a value reached only for conditions with k = 100, and n1 + n2 = 100). Even lower estimates were observed for the funnel plot and the trim and fill methods (neither of which evidenced power exceeding 0.40). As expected, the maximum power increased with larger numbers of studies in the meta-analyses, with larger sample sizes in the primary studies, and with larger magnitudes of publication bias. The funnel plot method and Kendall s Tau (N) appeared to be more influenced by unbalanced designs in the primary studies than were the other procedures. The funnel plot, Kendall s Tau (N), and the trim and fill methods evidenced increasing power as the population effect size increased. In contrast, the Egger Regression and Kendall s Tau (V) showed a decrease in maximum power with the largest population effect size examined. Similarly, higher levels of power were evident with heterogeneous population variances for Kendall s Tau (N) and the funnel plot methods, but lower power was evidenced for the other procedures. More detailed power estimates are provided for a subset of conditions in Table 6 (for balanced designs in primary studies, with nj = 10 and nj = 50). This table illustrates the abysmal power for all five procedures with small numbers of studies in the metaanalysis (regardless of the sample sizes in these studies) and with larger meta-analyses conducted from small sample primary studies. For conditions in which Type I error is adequately controlled, the Kendall Tau (V) procedure evidences distinct power advantages relative to Kendall s Tau (N). For example, with homogeneous variances, k = 100, n1 = n2 = 50, a large magnitude of publication bias, and population effect size of 0.50, the power of Kendall s Tau (V) was estimated to be 0.623, in contrast to a power estimate of for Kendall s Tau (N). Similar power advantages are occasionally

14 Publication Bias 14 evident for Egger s Regression (see Figure 16), but the number of conditions in which this procedure provides adequate Type I error control is very small. Discussion and Conclusions The results for conditions in which no publication bias was present suggests that only the Kendall s Tau (N) procedure maintained adequate Type I error control across the majority of conditions examined. The Egger Regression and Kendall s Tau (V) methods suggested liberal Type I error control in most conditions, while the Funnel Plot Regression and the trim and fill methods evidenced conservative control with small numbers of studies in the meta-analysis. The contrast in Type I error control between the two approaches to Kendall s Tau is likely related to the fact that the estimation variance of Cohen s d is a function of as well as sample size: σˆ ( d ) 2 n + n = + nn 2( n + n ) That is, even in the absence of publication bias, a non-zero correlation is expected between the sample value of Cohen s d and the estimated variance of d. In contrast, no relationship is expected between the sample value of Cohen s d and the sample size if publication bias is not present. Despite the overall satisfactory Type I error control exhibited by Kendall s Tau (N), statistical power was wanting in all conditions examined. The increase in power with larger numbers of studies in a meta-analysis and with larger sample sizes in the primary studies was anticipated, but power remained substantially below frequently used target power of 0.80 even under conditions with reasonably large meta-analyses simulated (i.e., with k = 100 and n1 + n2 = 100). Clearly, more research is needed to develop methods for adequately screening meta-analytic data sets for publication bias. Meta-analysis has become increasingly important for the synthesis of research results in a variety of fields, including education, the behavioral sciences and medicine. The accuracy of inferences derived from meta-analysis depends upon the appropriate application of statistical tools. As the use of meta-analytic methods becomes more

15 Publication Bias 15 commonplace, researchers must remain mindful of the limitations of certain estimates. The results of this study underscore the need to exercise caution in the interpretation of the results obtained from meta-analyses. This research furnishes valuable cautionary information about the poor Type I error control of several methods advocated for detecting publication bias, and the generally impoverished levels of statistical power.

16 Publication Bias 16 References Begg, C. B. (2001). Publication bias. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp ). New York: Russell Sage Foundation. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50(4), Bradley, J. V. (1978). Robustness? British Journal of Mathematical and Statistical Psychology, 31, Cliff, N. & Charlin, V. (1991). Variances and covariances of Kendall s tau and their estimation. Multivariate Behavioral Research, 26, Duval, S. & Tweedie, R. (2000a). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, Duval, S. & Tweedie, R. (2000b). A nonparametric Trim and Fill method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, Egger, M., Davey Smith, G., Schnedier, M. & Minder, C. Bias in meta-analysis detected by a simple graphical test. British Medical Journal, 315, Hedges, L. (1992). Modeling publication selection effects in meta-analysis. Statistical Science, 7(2), Hedges & Olkin (1985). Statistical Methods for Meta-Analysis. San Diego, CA: Academic Press. Hedges, L. & Vevea, J. (1996). Estimating effect size under publication bias: Small sample properties and robustness of a random effects selection model. Journal of Educational and Behavioral Statistics, 21(4),

17 Publication Bias 17 Macaskill, P., Walter, S., & Irwig, L. (2001). A comparison of methods to detect publication bias in meta-analysis. Statistics in Medicine, 20, Robey, R. R. & Barcikowski, R. S. (1992). Type I error and the number of iterations in Monte Carlo studies of robustness. British Journal of Mathematical and Statistical Psychology, 45, Rosenthal, R. (1979). The "file-drawer problem" and tolerance for null results. Psychological Bulletin, 86, Sutton, S.J., Abrams, K.R., Jones, D.R., Sheldon, T.A., & Song, F. (2000). Methods for Meta-Analysis in Medical Research. Chichester, NY: John Wiley & Sons, LTD. Thornton, A. & Lee, P. (2000). Publication bias in meta-analysis: Its causes and consequences. Journal of Clinical Epidemiology, 53,

18 Publication Bias 18 Table 1: Hypothetical sample of 10 effect sizes. Study n1 n2 1 di B * E d i ν i ν i ν i d i ν i d i Sum Mean Effect Size =

19 Publication Bias 19 Table 2 Proportion of Simulated Studies Available for Meta-Analysis. Number of Studies (K) Mean Min Max Primary Study Sample Size (n1 n2) 4, , , , , , , , , Population Effect Size Population Variance Ratio 1: : : : Magnitude of Publication Bias Small Large

20 Publication Bias 20 Table 3 Estimated Proportion of Conditions with Adequate Type I Error Control by Monte Carlo Design Factors. Trim and Egger Kendall (V) Kendall (N) Funnel Fill Number of Studies (K) Primary Study Sample Size (n 1 n 2) 4/ / / / / / / / / Population Effect Size Population Variance Ratio 1: : : : Note. Type I error estimates were based on 10,000 samples of each condition.

21 Publication Bias 21 Table 4 Mean Estimated Type I Error Rates by Monte Carlo Design Factors. Egger Kendall (V) Kendall (N) Funnel Number of Studies (K) Trim and Fill Primary Study Sample Size (n 1 n 2) 4/ / / / / / / / / Population Effect Size Population Variance Ratio 1: : : : Note. Type I error estimates were based on 10,000 samples of each condition.

22 Publication Bias 22 Table 5 Maximum Power Estimates by Monte Carlo Design Factors. Number of Studies (K) Egger Ken (V) Ken (N) Funnel Trim and Fill Primary Study Sample Size (n 1 n 2) 4, , , , , , , , , Population Effect Size Population Variance Ratio 1: : : : Magnitude of Publication Bias Small Large Note. Statistical power estimates were computed only for conditions in which Type I error was adequately controlled. Power estimates were based on 10,000 samples of each condition.

23 Publication Bias 23 Table 6 Estimated Power for Selected Conditions. Homogeneous Variances Population Effect Size = 0.0 Population Effect Size = 0.2 Population Effect Size = 0.5 Population Effect Size = 0.8 k n j Bias Egger Ken V Ken N Funnl T & F Egger Ken V Ken N Funnl T & F Egger Ken V Ken N Funnl T & F Egger Ken V Ken N Funnl T & F Sml Lrg Sml Lrg Sml Lrg Sml Lrg Sml Lrg Sml Lrg Heterogeneous Variances (1:4) Population Effect Size = 0.0 Population Effect Size = 0.2 Population Effect Size = 0.5 Population Effect Size = 0.8 k n j Bias Egger Ken V Ken N Funnl T & F Egger Ken V Ken N Funnl T & F Egger Ken V Ken N Funnl T & F Egger Ken V Ken N Funnl T & F Sml Lrg Sml Lrg Sml Lrg Sml Lrg Sml Lrg Sml Lrg Note. Statistical power estimates were computed only for conditions in which Type I error was adequately controlled. Power estimates were based on 10,000 samples of each condition.

24 Publication Bias 24 3 Begg's Standardized Effect Size Standardized Variance Figure 1: Begg's Tau plot with standardized effect size by standardized variance

25 Publication Bias 25 3 Egger's Standardized Effect Size y = X Precision Figure 2: Egger's regression plot with standardized effect size by precision

26 Publication Bias Effect Size y = X Sample Size Figure 3: Funnel plot regression method with effect size by sample size

27 Publication Bias Moderate Bias: exp(-2p**1.5) Strong Bias: exp(-4p**1.5) Weight p-value Figure 4. Weight functions to establish probability of inclusion in a meta-analysis as a function of the p-value.

28 Publication Bias Rate Egger Kendal (V) Kendal (N) Funnel Trim and Fill Method Figure 5: Distribution of Type I Error Rate Estimates Across Conditions Simulated

29 Publication Bias Proportion of Samples Egger Kendall (V) Kendall (N) Funnel Trim and Fill Analysis Method Figure 6: Proportions of Conditions with Adequate Type I Error Control by Number of Studies in the Meta-analysis

30 Publication Bias Average Type I Error Rate Egger Kendall (V) Kendall (N) Funnel Trim and Fill Analysis Method Figure 7: Mean Type I Error Rate by Number of Studies in the Meta-analysis

31 Publication Bias Proportion of Samples Egger Kendall (V) Kendall (N) Funnel Trim and Fill Analysis Method 5/5 10/10 50/50 Figure 8: Proportions of Conditions with Adequate Type I Error Control by Sample Size in Primary Studies (Equal N)

32 Publication Bias Proportion of Samples Egger Kendall (V) Kendall (N) Funnel Trim and Fill Analysis Method 4/6 6/4 8/12 12/8 40/60 60/40 Figure 9: Proportions of Conditions with Adequate Type I Error Control by Sample Size in Primary Studies (Unequal N)

33 Publication Bias Average Type I Error Rate Egger Kendall (V) Kendall (N) Funnel Trim and Fill Analysis Method 5/5 10/10 50/50 Figure 10: Mean Type I Error Rate by Sample Size in Primary Studies (Equal N)

34 Publication Bias Average Type I Error Rate Egger Kendall (V) Kendall (N) Funnel Trim and Fill Analysis M ethod 4/6 6/4 8/12 12/8 40/60 60/40 Figure 11: Mean Type I Error Rate by Sample Size in Primary Studies (Unequal N)

35 Publication Bias Proportion of Samples Egger Kendall (V) Kendall (N) Funnel Trim and Fill Analysis Method Figure 12: Proportions of Conditions with Adequate Type I Error Control by Population Effect Size

36 Publication Bias Average Type I Error Rate Egger Kendall (V) Kendall (N) Funnel Trim and Fill Analysis Method Figure 13: Mean Type I Error Rate by Population Effect Size

37 Publication Bias Proportion of Samples Egger Kendall (V) Kendall (N) Funnel Trim and Fill Analysis Method 1:1 1:2 1:4 1:8 Figure 14: Proportions of Conditions with Adequate Type I Error Control by Population Variance Ratio

38 Publication Bias Average Type I Error Rate Egger Kendall (V) Kendall (N) Funnel Trim and Fill Analysis Method 1:1 1:2 1:4 1:8 Figure 15: Mean Type I Error Rate by Population Variance Ratio

39 Publication Bias Estimated Power Egger Kendall (V) Kendall (N) Funnel Trim and Fill Analysis Method k=20, nj=10 k=20, nj=50 k=50, nj=10 k=50, nj=50 k=100, nj=10 k=100, nj=50 Figure 16: Statistical Power Estimates for Large Publication Bias, Homogeneous Variances, and Small Population Effect Size.