Improving the error rates of the Begg and Mazumdar test for publication bias in fixed effects meta-analysis

Transcription

1 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 RESEARCH ARTICLE Improving the error rates of the Begg and Mazumdar test for pubication bias in fixed effects meta-anaysis Miriam Gjerdevik 1,2* and Ivar Heuch 1 Open Access Abstract Background: The rank correation test introduced by Begg and Mazumdar is extensivey used in meta-anaysis to test for pubication bias in cinica and epidemioogica studies. It is based on correating the standardized treatment effect with the variance of the treatment effect using Kenda s tau as the measure of association. To our knowedge, the operationa characteristics regarding the significance eve of the test have not, however, been fuy assessed. Methods: We propose an aternative rank correation test to improve the error rates of the origina Begg and Mazumdar test. This test is based on the simuated distribution of the estimated measure of association, conditiona on samping variances. Furthermore, Spearman s rho is suggested as an aternative rank correation coefficient. The attained eve and power of the tests are studied by simuations of meta-anayses assuming the fixed effects mode. Resuts: The significance eves of the origina Begg and Mazumdar test often deviate consideraby from the nomina eve, the nu hypothesis being rejected too infrequenty. It is proven mathematicay that the assumptions for using the rank correation test are not stricty satisfied. The pairs of variabes fai to be independent, and there is a correation between the standardized effect sizes and samping variances under the nu hypothesis of no pubication bias. In the meta-anaysis setting, the adverse consequences of a fase negative test are more profound than the disadvantages of a fase positive test. Our aternative test improves the error rates in fixed effects meta-anaysis. Its significance eve equas the nomina vaue, and the Type II error rate is reduced. In sma data sets Spearman s rho shoud be preferred to Kenda s tau as the measure of association. Concusions: As the attained significance eves of the test introduced by Begg and Mazumdar often deviate greaty from the nomina eve, modified rank correation tests, improving the error rates, shoud be preferred when testing for pubication bias assuming fixed effects meta-anaysis. Keywords: Meta-anaysis, Pubication bias, Rank correation, Kenda s tau, Spearman s rho, Error rates Background Meta-anaysis is a systematic procedure for assessing and combining statistica information based on resuts of avaiabe independent studies regarding the same topic. In recent years, meta-anaytic methods have become increasingy popuar in various fieds of medicine. Resuts from meta-anaysis are subject to criticism for many reasons, an important concern being possibe sma study *Correspondence: miriam.gjerdevik@igs.uib.no 1 Department of Mathematics, University of Bergen, P. O. Box 7800, N-5020 Bergen, Norway 2 Department of Goba Pubic Heath and Primary Care, University of Bergen, P. O. Box 7804, N-5018 Bergen, Norway effects such as pubication bias. Pubication bias arises when the pubished studies reevant for incusion in a meta-anaysis do not represent a studies of the probem of interest [1]. In particuar, studies that are ess ikey to get pubished appear to be the ess concusive ones [2,3]. The chance that studies with sma sampe size and ow statistica precision are pubished is increased if they show stronger treatment effects [4,5]. Pubication bias may affect the concusions of meta-anayses and systematic reviews and resut in a biased overa estimate of the treatment effect. Hence data shoud be evauated for pubication bias before a meta-anaysis is conducted [6] Gjerdevik and Heuch; icensee BioMed Centra Ltd. This is an Open Access artice distributed under the terms of the Creative Commons Attribution License ( which permits unrestricted use, distribution, and reproduction in any medium, provided the origina work is propery credited. The Creative Commons Pubic Domain Dedication waiver ( appies to the data made avaiabe in this artice, uness otherwise stated.

2 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 Page 2 of 16 Traditionay, funne pots [7], a simpe graph of the measure of precision (e.g. sampe sizes or inverse variances of the component studies versus the summary outcome measures, have been used as a visua too to detect sma study effects. The funne graph is based on the fact that precision in estimating the underying effect wi increase as the sampe size of the component studies increases [8]. The estimated effects shoud in principe be symmetricay distributed about the true unknown effect. Resuts from studies with sma sampe size shoud scatter widey at the bottom of the graph, with the spread narrowing as the sampe sizes increase. Assuming that a studies in the anaysis estimate the same effect, the pot shoud resembe a symmetrica inverted funne if pubication bias is not present. If pubication bias is present, however, the pot wi tend to be skewed, due to the fact that sma and non-significant studies are ess ikey to appear in the pubished iterature. This induces a correation in the graph. Severa authors have provided forma and objective tests for pubication bias. Egger et a. [8] based their test on a simpe inear regression of the effect estimate against its standard error, weighted by the inverse of the variance of the effect estimate. A modified regression approach was introduced by Macaski et a. [4]. Begg and Mazumdar [6] expoited the fact that pubication bias wi tend to induce a correation between the treatment effects and their variances. They constructed a test by examining the correation between the two factors and standardized the effect sizes prior to performing a rank correation test based on Kenda s tau. Asymmetry in funne pots may aso, however, occur due to heterogeneity. Statistica heterogeneity is present when the true effects being evauated vary between studies, and this underying heterogeneity may be detectabe if the variation between the studies is above that expected by chance [9]. The proposed tests for pubication bias actuay test for sma study effects. They seek to assess whether funne pot asymmetry is ikey to have occurred by chance and are thus not abe to distinguish between sma study effects such as pubication bias and heterogeneity. However, as we wi conduct simuations studies of meta-anayses using the fixed effects mode, which assumes no heterogeneity, we wi often use the ess precise phrase pubication bias. The meaning shoud be cear from the context. The regression based approach proposed by Egger et a. [8]andtherankcorreationtestprovidedbyBeggand Mazumdar [6] are widey used in meta-anaysis to test for sma study effects in cinica and epidemioogica studies. As of December 2013, the origina artice by Egger et a. [8] had been cited more than 7,000 times in the Web of Science database [10]. Sti, the artice by Begg and Mazumdar [6] had been cited more than 2,500 times, a number which demonstrates great infuence in the iterature. During 2013, the number of new citations exceeded 600. Figure 1 iustrates the increasing impact and reevance of the rank correation test amost two decades after it was pubished, athough it is not quite so important as the Egger test [8]. Concerns have been expressed, however, about the possibe ack of power of both tests [4,6]. It is aso we-known that when the outcome is binary and the intervention effect is expressed as an odds ratio or a reative risk, the variance of the estimator is mathematicay reated to the estimate itsef [11,12]. This resuts in the nu hypothesis being rejected too frequenty, and severa authors have proposed aternative tests to correct for this bias [5,13,14]. Nonetheess, concentrating on the rank correation test, it is sti an open question whether the significance eve is satisfactory in the case of a continuous outcome variabe. Improving the error rates of the Begg and Mazumdar test for pubication bias in meta-anaysis, assuming a normay distributed outcome, is thus a primary objective of our artice. The significance eve of the Begg and Mazumdar test is attained when there is no seection bias present in the meta-anaysis. Begg and Mazumdar carried out simuations corresponding to such situations. They did not, however, incude the resuts in their paper but merey stated that In a cases the nomina significance eve was ess than 5% [6]. This remark shoud be interpreted with caution, but probaby indicates that the significance eve of the test is ess than the nomina 5% eve. Nonetheess, the issue of the significance eve warrants further investigation. Description of the Begg and Mazumdar test Suppose that a meta-anaysis consists of k studies. Let t 1, t 2,..., t k and v 1, v 2,..., v k denote the estimated effect sizes and samping variances from these studies. As the effect sizes are not identicay distributed under the nu hypothesis of no pubication bias, Begg and Mazumdar [6] standardize the effect sizes prior to performing a rank correation test. They correate ti and v i, i = 1, 2,..., k, where ti = t i t ( v 1/2. i Here, v j t = v j t j is the standard weighted average of the effect sizes and ( v i = v i v j is the variance of t i t.

3 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 Page 3 of 16 Figure 1 Number of yeary cites for the Begg and Mazumdar artice. Begg and Mazumdar [6] use a rank correation test based on Kenda s tau, defined as 2S 2(C D τ = = k(k 1 k(k 1. The test invoves evauating C, the number of pairs of studies that are ranked in the same order with respect to the two factors t and v, andd, the number of pairs of studies that are ranked in the reverse order. Here t and v denote the vectors consisting of t1, t 2,..., t k and v 1, v 2,..., v k, respectivey. The standardized test statistic is defined as z = C D k(k 1(2k + 5/18, a variabe which is asymptoticay N(0, 1, a direct resut foowing from the properties of Kenda s tau [15]. It is aso possibe to empoy an exact distribution of Kenda s tau for sma vaues of k, athough this is not done by Begg and Mazumdar. The denominator shoud be modified if there are tied observations [15]. This artice is organized as foows. In the Methods section we suggest an agorithm intended to improve the error rates of the Begg and Mazumdar test for pubication bias. Additionay, this section outines the simuation procedure used to study and compare the new agorithm to the origina test in fixed effects meta-anaysis. In the Resuts section we expain why the Begg and Mazumdar method has a poor significance eve. The performance of the adjusted test is assessed and compared to the resuts of the origina test. Exampes are given. The Discussion section incudes an overa evauation of the rank correation tests for pubication bias presented in our paper and is foowed by the Concusion section. Methods Improvement of the Begg and Mazumdar test: Method and agorithm We woud ike to deveop a test based on rank correation making it easy to adjust the actua significance eve in the case of a normay distributed outcome variabe. Thiscanbedoneempoyingthesimuateddistributionof the estimated measure of association, conditiona on the samping variances. The foowing agorithm summarizes the procedure: Given k estimated effects, t 1, t 2,..., t k, and their variances, v 1, v 2,..., v k. 1. For each repication, indexed j = 1, 2,..., n: (a Generate k estimated effects t (j 1, t (j 2,..., t (j k, where t (j i N(δ, v i and δ is the common effect size in the fixed effects mode. Standardize these effects as described in the

4 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 Page 4 of 16 Background section, obtaining t (j 1, t (j 2,..., t (j k. (b Correate t (j 1, t (j 2,..., t (j k and v 1, v 2,..., v k by computing Kenda s tau, τ j. 2. Determine the intervas of rejection, e.g., by finding the percenties based on the empirica distribution of τ 1, τ 2,..., τ n. These intervas depend on the prescribed significance eve α. 3. Correate the standardized effects, t 1, t 2,..., t k, based on the actua data, and the variances, v 1, v 2,..., v k, and compute Kenda s tau, τ. 4. Reject the nu hypothesis of no pubication bias if τ is within the rejection intervas or compute the p-vaue. We denote this the adjusted Begg and Mazumdar test. The R code for the adjusted procedure is provided in Additiona fie 1. The intervas of rejection are estimated with errors. These errors may infuence the resuts when testing for pubication bias, and in order to minimize the errors, n shoud be arge. A drawback with this procedure is that we condition on the samping variances, and we may possiby deveop better methods if this is not done. In addition we assume that the estimated effect sizes are normay distributed. This may not aways be the case in studies with a sma sampe size, and in particuar, effect sizes are not normay distributed if the outcome is binary. Kenda s tau is scae invariant. Hence the adjusted Begg and Mazumdar method wi sti work we if the variances are systematicay underestimated. It shoud be noted that the simuation procedure itsef in steps 1 and 2 does not depend on the observed vaues t 1, t 2,..., t k of the random variabes invoved, but ony on the fixed variances. Thus the procedure is not a bootstrap in the ordinary sense. Spearman s rho versus Kenda s tau as the measure of association The Begg and Mazumdar test uses Kenda s tau as the measure of association. It requires evauation of the test statistic S = C D, where C and D are the numbers of concordant and discordant pairs, respectivey. Thisstatisticcanonytakeafinitenumberofdistinct vaues, and the exact distribution of Kenda s tau is discrete. The significance eve may thus be incorrect, i.e., the actua significance eve of the test may not equa the nomina vaue, especiay for sma vaues of k. This wi be of ess concern, however, as the vaue of k increases. In order to improve the eve, one may aternativey appy the mid-p-vaue [16], which deviates from the p- vaue in that ony haf the probabiity of the observed vaue of the test statistic is incuded in the tai. Additionay, one coud empoy the adjusted Begg and Mazumdar test based on Spearman s rho. This statistic is defined as r s = 1 6 d2 i k 3 k, where d i = x i y i is the difference between the ranks of observation i for the two variabes. The distribution of Kenda s tau converges faster towards the norma distribution than that of Spearman s rho [17]. This may be one of the reasons why Begg and Mazumdar chose Kenda s tau as a basis for their test procedure. However, having deveoped efficient computer programs for cacuating the exact distribution of the rank correation coefficient, this is no onger a vaid argument in our setting. The exact distribution of Spearman s rho is aso discrete, but the test statistic can take more vaues compared to Kenda s tau. Consequenty, the actua significance eve of Spearman s rho converges faster towards the nomina eve than Kenda s tau, i.e., for smaer vaues of k. Simuation procedure Simuations are needed in order to study the significance eve of the Begg and Mazumdar test and to examine the operationa characteristics of our new agorithm and compare to the origina test. We appy the simuation procedure introduced by Begg and Mazumdar, and a detaied description is given in the foowing subsections. Study seection Begg and Mazumdar [6] assume that the samping distribution of t is norma, i.e. t i N (δ, v i and t i is independent of t j for i = j. The component studies are designed to estimate a common effect size δ (we assume the fixed effects mode, with variances depending on the sampe sizes in the individua studies. A asymmetry in the simuated funne pot is thus due to pubication bias or chance; it cannot be expained by heterogeneity. The effect size, t, is in each case a summary estimate based on a data set of a certain size. Due to the Centra Limit Theorem, t wi possess an asymptotic norma distribution in most situations. According to Begg and Mazumdar [6], the assumption of normaity is therefore reasonabe. When a particuar vaue of t i has been generated, the study is pubished (incuded in the meta-anaysis with probabiity given by an appropriate weight function. Begg and Mazumdar use different weight functions. The weight function dependingon the p-vaue for the hypothesis that the true underying effect is zero is defined as w i (t i (p i = exp ( bp a i,

5 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 Page 5 of 16 with suitaby defined constants ( a and b.thisweightfunction is evauated at p i = t i /v 1/2 i for one-sided ( seection, and p i = 2 t i /v 1/2 i for two-sided seection. The absoute vaue was inadvertenty eft out in the artice of Begg and Mazumdar. An aternative weight function based ony on the assumption that the observed effect estimate determines the chance of pubication is defined as w(t i = exp ( b ( t i a. The main objective in conducting a meta-anaysis is to estimate the true underying effect, δ. This can be done using t = v j t j / v j. The bias induced by the seection mode, β, isdefinedasβ = E( t δ. The expected vaue is here defined with respect to the distribution of t after the seection of studies. Scenarios Foowing the scenarios of Begg and Mazumdar, we considered two vaues for k, the number of component studies in a meta-anaysis, k = 25 and k = 75. For each simuated meta-anaysis, the studies were generated in such a manner that after seection for incusion in the metaanaysis, there were three equa-sized groups of studies with different variances. In each simuation, the midde group had a standardized variance of 1. When k = 25, nine studies had variance 1. Two ranges of standardized variances were empoyed: arge (v = 0.1, 1.0, 10.0 and sma (v = 0.5, 1.0, 2.0. The range from the study with the smaest variance to the study with the argest variance was characterized by the ogarithm to the base 10 of the ratio of these variances. The order of magnitude was 2 and 0.6 using Begg and Mazumdar s choices of variances. The parameter δ refecting the effect under study varied from zero, the nu vaue, through 3.0 standard deviations from the nu vaue. The scae was given in standard deviation units for the effect estimator for a study in the midde group having variance equa to 1.0. Pubication bias in the meta-anaysis was simuated using the seection functions presented in the previous subsection. Strong pubication bias was obtained choosing a = 1.5 and b = 4.0, whereas a = 3.0 and b = 4.0 corresponded to moderate pubication bias, see Figure 2. The choice of simuation parameters was justified by Begg and Mazumdar [6], and the same parameters were ater used by Macaski et a. [4] in their simuation studies. The seection mode based on the p-vaue has aso been described by Copas [18] and used by Preston et a. [19]. We generated each simuated meta-anaysis in the foowing way. An effect size was randomy generated from a norma distribution having one of the variances under study. Its mean was the true, underying effect. The probabiity of seection for incusion in the meta-anaysis was cacuated by the reevant seection mode. We chose the mode based on the p-vaue. The decision to incude or Figure 2 Seection mechanism. The weight function for seecting studies for incusion in the meta-anaysis as a function of the p-vaue. The vaue of a determines the seection strength (a = 1.0: substantia seection strength, a = 1.5: strong seection strength, a = 3.0: moderate seection strength.

6 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 Page 6 of 16 excude this study in the meta-anaysis was made based on a biased-coined randomization using the computed probabiity of pubication. This procedure was repeated unti a study with this first variance was incuded in the meta-anaysis. We repeated the process, this time using a second variance under study. To avoid ties, we added asmatermɛ = to each variance which had previousy occurred in the simuation process of this particuar meta-anaysis. The process was continued unti k studies had been seected with the required mixture of variances. The rank correation test based on Kenda s tau (or aternativey Spearman s rho was then cacuated. The summary estimate of the true underying effect was computed, and the number of studies required to generate k pubished studies was recorded. We repeated the entire process 5,000 times, a number originay chosen by Begg and Mazumdar. The two-sided empirica power and significance eve of the test were cacuated at the nomina 5% eve. Since the error rates are expressed as proportions in a simuation study, the power estimates have a maximum standard error of 0.707%. Resuts Attained significance eve and power of the origina Begg and Mazumdar test based on Kenda s tau We first performed simuations in the situation without pubication bias, in order to contro the resuts of the Begg and Mazumdar method. The estimated significance eve of their test for pubication bias found by us is shown in Tabes 1 and 2, in the same format as used by Begg and Mazumdar to report rejection rates. The first tabe gives the significance eves for sma meta-anayses consisting of k = 25 component studies. The second gives resuts for arge meta-anayses that consist of k = 75 component studies each. Our resuts confirm the impression that the attained significance eve does not exceed the nomina one, but overa the deviations between the two vaues can be considerabe. The range of variances is obviousy an important factor infuencing the significance eve of the test. When the spread of variances is arge (v = 0.1, 1.0, 10.0, the significance eve is roughy 0.02 or ower and deviates consideraby from the nomina vaue of When the range of variances is sma (v = 0.5, 1.0, 2.0, the significance eve is approximatey The vaue of k does not seem to affect the eve. We assessed the significance eve aso choosing k = 300 and a arge range of variances. The eve is roughy the same as for k = 25 and k = 75, i.e The nu hypothesis is rejected too infrequenty. Due to these, perhaps unexpected, resuts, we beieve a more thorough investigation of the operationa characteristics is justified. A simuations were undertaken using R [20]. The rank correation test based on Kenda s tau was cacuated Tabe 1 Significance eve for the Begg and Mazumdar test for pubication bias: Sma meta-anayses* Leve [% seected for incusion, bias] Range of variances Large Sma Treatment effect (δ % 3.96% [100%,.00] [100%,.00] % 4.36% [100%,.00] [100%,.00] % 4.30% [100%,.00] [100%,.00] % 3.68% [100%,.00] [100%,.00] % 3.58% [100%,.00] [100%, -.00] % 4.48% [100%,.00] [100%,.00] % 4.24% [100%,.00] [100%, -.00] *k = 25 studies; nomina significance eve v = 0.1, 1.0, 10.0, v = 0.5, 1.0, 2.0. Vaues deviating significanty from the nomina eve 5.00% over the 5000 simuations (using a 5% eve in the binomia test are typed in bodface. using the R function cor.test(, with the ogica indicator exact set to NULL. An exact p-vaue for the hypothesis of no association is cacuated in this function if there are ess than 50 paired sampes containing finite vaues and there are no ties. Otherwise, the asymptoticay norma test statistic z, defined in the Methods section, is used. Simiar simuations using the statistic z for sma meta-anaysis (k = 25 were aso carried out. The resuts confirm the observation that the overa significance eve does not equa the nomina vaue if the asymptotic distribution of Kenda s tau is used and is sti beow the nomina vaue. Thus, use of the exact or asymptotic distribution of Kenda s tau does not expain the poor significance eve when k = 25. In the simuation procedure presented by Begg and Mazumdar, the variances are treated as fixed constants. Additiona simuations were carried out empoying random variances drawn from a suitabe distribution. As an exampe, for k = 25 and δ = 0 the eve was ony when drawing variances from an inverse gamma distribution with shape parameter 3.3 and rate parameter 300. The use of random variances does not remove the probem regarding the poor significance eve. This is as expected because the fixed variance case has a strong resembance to the Mann-Kenda trend test [21].

7 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 Page 7 of 16 Tabe 2 Significance eve for the Begg and Mazumdar test for pubication bias: Large meta-anayses* Leve [% seected for incusion, bias] Range of variances Large Sma Treatment effect (δ % 4.12% [100%, -.00] [100%,.00] % 4.74% [100%,.00] [100%, -.00] % 4.54% [100%,.00] [100%,.00] % 4.30% [100%,.00] [100%, -.00] % 4.24% [100%,.00] [100%, -.00] % 4.10% [100%,.00] [100%,.00] % 4.22% [100%,.00] [100%,.00] *k = 75 studies; nomina significance eve v = 0.1, 1.0, 10.0, v = 0.5, 1.0, 2.0. Vaues deviating significanty from the nomina eve 5.00% over the 5000 simuations (using a 5% eve in the binomia test are typed in bodface. Power estimates were found empoying the simuation procedure described in Methods. The simuations were restricted to one-sided seection depending on the p-vaue for the hypothesis that the true underying effect is zero. The rank correation test is not adequate for two-sided seection. At δ = 0, the generated studies are symmetricay distributed around zero. Consequenty, any seection function that is dependent on the two-sided p-vaue wi resut in a pattern in which the effect estimates and variances are uncorreated [6]. The power of the Begg and Mazumdar test is shown in Tabes 3 and 4. Best power is achieved for arge metaanayses where the spread of variance is arge, the seection strength is strong and the treatment effect is sma. As an exampe, the power is 0.99 when k = 75, v = 0.1, 1.0, 10.0, a = 1.5 and δ = 0. When the seection strength is smaer (a = 3, using the same vaues of the other parameters reduces the power to When k = 25, the power is moderate even for quite strong seection effects (a = 1.5. It is estimated at 0.57 when δ = 0 and v = 0.1, 1.0, The range of variances, which is inversey reated to the spread of sampe sizes, aso has a substantia impact on the power. Reducing the range of variances in the atter situation (v = 0.5, 1.0, 2.0, the power is ony The test is generay not powerfu when the underying treatment effect is far from the nu vaue. However, this is not a major concern; in these situations Tabe 3 Power for the Begg and Mazumdar test for pubication bias: Sma meta-anayses* Power [% seected for incusion, bias] Seection strength Strong** Moderate*** Range of variances Large Sma Large Sma Treatment effect (δ.0 57% 22% 33% 13% [36%,.34] [37%,.74] [57%,.25] [57%,.54].5 51% 21% 23% 11% [54%,.16] [52%,.54] [74%,.09] [73%,.34] % 16% 13% 8% [65%,.07] [67%,.36] [82%,.04] [85%,.20] % 13% 9% 6% [72%,.05] [80%,.23] [87%,.02] [92%,.10] % 8% 5% 5% [78%,.03] [88%,.14] [90%,.02] [96%,.05] % 6% 3% 4% [82%,.02] [93%,.07] [93%,.01] [98%,.03] 3.0 9% 5% 3% 4% [86%,.02] [96%,.04] [94%,.01] [99%,.01] *k = 25 studies; nomina significance eve **a = 1.5, *** a = 3.0. v = 0.1, 1.0, 10.0, v = 0.5, 1.0, 2.0.

8 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 Page 8 of 16 Tabe 4 Power for the Begg and Mazumdar test for pubication bias: Large meta-anayses* Power [% seected for incusion, bias] Seection strength Strong** Moderate*** Range of variances Large Sma Large Sma Treatment effect (δ.0 99% 61% 88% 38% [36%,.34] [36%,.74] [56%,.24] [56%,.54].5 99% 59% 77% 31% [53%,.16] [52%,.54] [74%,.09] [72%,.34] % 50% 54% 21% [64%,.07] [67%,.36] [82%,.04] [84%,.19] % 35% 35% 12% [71%,.04] [79%,.23] [86%,.02] [92%,.10] % 22% 21% 7% [77%,.03] [88%,.13] [90%,.02] [96%,.05] % 12% 13% 5% [81%,.02] [93%,.07] [92%,.01] [98%,.03] % 7% 8% 5% [85%,.02] [96%,.04] [94%,.01] [99%,.01] *k = 75 studies; nomina significance eve **a = 1.5, ***a = 3.0. v = 0.1, 1.0, 10.0, v = 0.5, 1.0, 2.0. there is ony a sma bias in the estimate of the treatment effect because the number of unpubished studies is ower. Additiona simuations were carried out using a nomina eve of When the range of variances is arge, the attained significance eves are roughy haf the nomina, i.e., about The significance eve is estimated at when k = 25 and δ = 0. Empoying a sma spread of variances, the significance eves are about When k = 25 and δ = 0, the empirica eve is As a consequence of increasing the significance eve, the power of the test is aso increased. Giving an exampe, the power equas 0.72 and 0.32 when k = 25, a = 1.5, δ = 0andthe spread of variances is arge and sma, respectivey. BeggandMazumdarjustifiedthetworangesofvariances used in their simuations by considering reevant vaues in pubished studies. In the context of metaanaysis, however, the variances take more than three different vaues, and their choices of variances do not represent a reaistic distribution. Separate simuations confirmed that the distribution of the variances infuences the resuts concerning the significance eve and power, but these resuts are nevertheess not in confict with our genera assessments and concusions. As an exampe, the simuation resuts when generating meta-anayses consisting of 25 component studies, of which 13 studies have the argest variance (v = 10.0 or v = 2.0, 9 have variance 1.0 and 3 have the owest variance (v = 0.1 or v = 0.5, are consistent with the resuts in Tabe 3, see Additiona fie 2. The significance eve was roughy 0.02 and 0.04 using the arge and sma range of variances, respectivey. An expanation of the poor significance eve We compute the conditiona covariance between two standardized effects given the variances. Straightforward cacuations give (t i, tj v 1, v 2,..., v k Cov ( Cov t i v = v t v1, v 2,..., v k = ( ( v 1/2 i [ Cov ( Cov ( Cov + Cov ( v i 1/2 ( v j 1 1/2 ( v v t, t j v 1 1/2 ( 2 v j v ( t i v, t j v v1, v 2,..., v k t i v, v t v1, v 2,..., v k, v t v1, v 2,..., v k t j v ( v t, v ] t v1, v 2,..., v k, where i, j = 1, 2,..., k and i = j. We assume that the k pairs of random variabes considered in a meta-anaysis, (t 1, v 1, (t 2, v 2,..., (t k, v k,are 2

9 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 Page 9 of 16 independent and have the same bivariate distribution. As a consequence, t i is independent of t j given v 1, v 2,..., v k. It foows that the first term inside the square brackets equas zero. For the same reason, ( Cov t i v ( = Cov = v Cov + v = v. The ast term equas Cov, v t v 1, v 2,..., v k t j v, v t v 1, v 2,..., v k ( t j, v j t j v 1, v 2,..., v k Cov t j, v t v 1, v 2,..., v k =j ( v t, v t v 1, v 2,..., v k ( = Var v t v 1, v 2,..., v k = v. It readiy foows that Cov (ti, tj v 1 1, v 2,..., v k = (v i 1/2 (v j 1/2 v. Because the variabes ti are standardized, we have Corr (ti, tj v 1, v 2,..., v k = Cov (ti, tj v 1, v 2,..., v k. It has therefore been mathematicay proven that the standardized treatment effects given the variances fai to be independent. As a resut, the standardized treatment effects are not independent even if the variances are regarded as random variabes. Furthermore, there is a correation between the standardized effect sizes and samping variances under the nu hypothesis of no pubication bias. It foows that the vectors t and v are not independent under the nu hypothesis. The assumptions for using the rank correation test are not stricty satisfied. We note that these resuts are independent of k and the distribution of t, meaning that the probems sti occur if the vaue of k isargeandifthestandardizedeffectsare normay distributed. Our cacuations are consistent with a remark given by Begg [2], who caimed that a sma correation is produced because the empirica standardization is based on the estimated mean effect, which wi be positivey biased. However, when k = 25 and v = 0.1, 1.0, 10.0, then 0.13 Corr (ti, tj v 1, v 2,..., v k < 0. The arger correations are not negigibe. Attained significance eve and power for the adjusted Begg and Mazumdar test The effect sizes are standardized by Begg and Mazumdar to obtain a set of estimates that can be assumed to be independent and identicay distributed under the nu hypothesis of no pubication bias [4]. However, even after the standardization the assumptions for using the traditiona rank correation test are not met. In the context of potentia pubication bias in metaanaysis, the adverse consequences of a fase negative test are much more profound than those of a fase positive test [2]. One shoud not concude that pubication bias is absent when it actuay is present. It is essentia to contro or restrict the Type II error rate, and one shoud thus not be satisfied with a test that imits the significance eve beneath the nomina vaue. Adjusting the significance eve is a primary concern. Additionay, there are good reasons to choose a higher nomina significance eve (e.g., α = 0.10 than the conventiona vaue For a fixed sampe size, increasing the significance eve wi increase the power of a given test method. Choosing a higher significance eve is consequenty a conservative measure in a meta-anaysis setting. We assessed the properties of the adjusted rank correation test, first using Kenda s tau as our test statistic. The adjusted interva of rejection for the test statistic was found in any particuar situation with fixed variances using ony the first two steps in the agorithm, choosing n = 100, 000. The area of acceptance was within the 2.5 and 97.5 percenties from the empirica distribution of the simuated vaues of the measure of association. The nomina eve of 0.05 was chosen to make resuts comparabe to those of Begg and Mazumdar [6]. We used the same rejection intervas for a vaues of δ, the true underying effect, that is, we ony found new intervas for each vaue of k and each range of variances. The vaue of δ is unknown, but the significance eve does not seem to depend on δ when the effect sizes are normay distributed (see Tabes 1 and 2. Without oss of generaity, one may thus choose δ = 0. We then carried out the simuation procedure given by Begg and Mazumdar, empoying the adjusted rejection intervas. In order to reduce the standard error, the entire process was now repeated 10,000 times. The maximum standard error from the atter simuations is then reduced from 0.707% to 0.500%. When k = 25, the attained significance eve is and for v = 0.1, 1.0, 10.0 and v = 0.5, 1.0, 2.0, respectivey. The agorithm obviousy improves the Type I error rates. Tabe 5 presents the power of the adjusted test for k = 25. As is to be expected, the Type II error rate is reduced compared to the origina test introduced by Begg and Mazumdar. When δ = 0, v = 0.1, 1.0, 10.0 and a = 1.5, the adjusted method increases the power to 0.73 compared to the origina When the seection

10 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 Page 10 of 16 Tabe 5 Power for the adjusted Begg and Mazumdar test based on Kenda s tau: Sma meta-anayses* Power [% seected for incusion, bias] Seection strength Strong** Moderate*** Range of variances Large Sma Large Sma Treatment effect (δ.0 73% 24% 48% 16% [36%,.34] [37%,.74] [57%,.25] [57%,.54].5 69% 23% 37% 14% [54%,.16] [52%,.54] [74%,.09] [73%,.35] % 20% 25% 10% [65%,.07] [67%,.37] [82%,.04] [85%,.20] % 15% 17% 7% [72%,.05] [80%,.23] [87%,.02] [92%,.10] % 10% 12% 6% [78%,.03] [88%,.13] [90%,.02] [96%,.05] % 7% 9% 5% [82%,.02] [93%,.07] [93%,.01] [98%,.03] % 6% 7% 5% [86%,.02] [97%,.04] [94%,.01] [99%,.01] *k = 25 studies; nomina significance eve **a = 1.5, *** a = 3.0. v = 0.1, 1.0, 10.0, v = 0.5, 1.0, 2.0. strength is moderate (a = 3, the corresponding numbers are 0.48 and 0.33 for the adjusted and origina Begg and Mazumdar test, respectivey. In cases where the range of variances is sma, the attained significance eve is cose to the nomina vaue for the Begg and Mazumdar test. Consequenty, there is ony a minor improvement in the Type II error rate when correcting the significance eve. Nevertheess, the adjusted method is ceary preferabe to the origina test. Simiar concusions are drawn when the meta-anayses comprise 75 studies, and the resuts of the power simuations are shown in Additiona fie 3. The attained significance eve is and for v = 0.1, 1.0, 10.0 and v = 0.5, 1.0, 2.0, respectivey. Figure 3 shows histograms of the simuated distribution of τ under the nu hypothesis for k = 25, together with kerne density estimates. These are compared to the density of the asymptotic distribution of Kenda s tau. The histograms are based on 100,000 simuated vaues of τ, and the kerne density estimates were computed appying the R function density [20]. When there is a arge spread of variances (Figure 3A, the simuated distribution of τ ceary deviates from the theoretica. This difference decreases when the range of variances is sma (Figure 3B. This resut supports our findings in Tabes 1 and 3; when the attained eve of the Begg and Mazumdar test is cose to the nomina, it performs equay we as the adjusted test. Additiona simuations were performed choosing a = 1.0 when k = 25 for both the origina and the adjusted Begg and Mazumdar test. This vaue of a generates even stronger seection bias than a = 1.5, see Figure 2. In ine with earier findings, the power was increased for both methods with the substantia pubication bias, and the adjusted test sti gave the best Type II error rates. The resuts are shown in Additiona fie 4. As aready expained, it may be reasonabe to choose a higher significance eve than 0.05, and simuations were carried out for the adjusted method at a 0.10 significance eve. These additiona simuations were aso performed when the meta-anayses comprised k = 25 studies, this vaue of k being more reevant than k = 75 in cinica and epidemioogica settings. The trade-off effect reduces the Type II error rate at the expense of the Type I error rate. The resuts are shown in Tabe 6. The power is sti poor when the spread of variances is sma, and the power estimates are beow However, the test gives respectabe power in the remaining situations, at east when the treatment effect is cose to the nu vaue. At a = 1.5, the power ranges between 0.83 and 0.30 for a treatment effects. The attained significance eve was cose to the nomina

11 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 Page 11 of 16 Figure 3 Histograms of the simuated distribution of τ under the nu hypothesis aong with kerna density estimates. These are compared to the density of the asymptotic distribution of Kenda s tau. A: sma meta-anayses (k = 25 and a arge range of variances (v = 0.1, 1.0, 10.0; B: sma meta-anayses (k = 25 and a sma range of variances (v = 0.5, 1.0, 2.0. vaue and was estimated at and when v = 0.1, 1.0, 10.0 and v = 0.5, 1.0, 2.0, respectivey. Many meta-anayses incude much ess than 25 studies. For that reason, we performed additiona simuations in the Begg and Mazumdar setting to assess the actua significance eve of the adjusted test as k decreases, sti using Kenda s tau as our test statistic. The resuts depend on the range of variances. When the spread of variances is arge (v = 0.1, 1.0, 10.0, a reasonaby correct significance eve is attained for k = 16. For a sma spread of variances (v = 0.5, 1.0, 2.0, the correct significance eve is achieved for k 18. The power of the different tests for pubication bias is generay of great concern. This probem wi not decrease for sma vaues of k. The best power is, as usua, attained when strong seection bias is present and δ = 0. When, in addition, the range of variance is arge and k = 16, the simuated power of the test is With a sma range of variances and k = 18, the test achieves a power of The estimated significance eves are and , respectivey. What if we use Spearman s rho instead of Kenda s tau as the measure of association? Additiona simuations then demonstrate that k 10 is sufficient to achieve an acceptabe significance eve when the range of variances is arge (v = 0.1, 1.0, For sma spread of variances (v = 0.5, 1.0, 2.0, a reasonaby correct significance eve cannot be guaranteed for k < 11. The power attained given strong seection bias and δ = 0isineachcase0.33 and 0.13, respectivey. The attained significance eves are and The resuts of the adjusted test based on Spearman s rho in the Begg and Mazumdar simuation setting are presented in Tabes 7 and 8 at a nomina significance eve of 0.05 and 0.10, respectivey. Both tabes dispay resuts for sma meta-anayses (k = 25. We see that the power estimates are simiar to the ones attained using Kenda s tau. The attained significance eves are approximatey equa to the nomina vaue in a cases. At the nomina 0.05 eve, it is estimated at when v = 0.1, 1.0, 10.0 and at when v = 0.5, 1.0, 2.0. The corresponding vaues at the nomina 10% eve are and Exampes We compare the new, adjusted test to that of Begg and Mazumdar, appying two exampes from the iterature [22-24]. These exampes have previousy been used by Begg and Mazumdar [6] and Begg [2], respectivey, in iustration of rank correation tests for pubication bias; the potentia probem of pubication bias was recognized

12 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 Page 12 of 16 Tabe 6 Power for the adjusted Begg and Mazumdar test based on Kenda s tau: Sma meta-anayses* Power [% seected for incusion, bias] Seection strength Strong** Moderate*** Range of variances Large Sma Large Sma Treatment effect (δ.0 83% 36% 63% 23% [36%,.34] [37%,.74] [57%,.25] [57%,.54].5 80% 34% 53% 22% [54%,.16] [52%,.54] [74%,.09] [73%,.34] % 30% 39% 17% [65%,.07] [67%,.36] [82%,.04] [85%,.20] % 23% 27% 13% [72%,.05] [80%,.23] [87%,.03] [92%,.11] % 18% 21% 10% [78%,.03] [88%,.13] [90%,.02] [96%,.05] % 13% 17% 10% [82%,.02] [93%,.08] [93%,.01] [98%,.03] % 10% 14% 10% [86%,.02] [96%,.04] [94%,.01] [99%,.01] *k = 25 studies; nomina significance eve **a = 1.5, ***a = 3.0. v = 0.1, 1.0, 10.0, v = 0.5, 1.0, 2.0. in both exampes and was supported by the funne pots in their papers. The raw data can be found in the origina artices [22,23] and are aso presented by Begg and Mazumdar [6] and Begg [2]. Exampe 1 In the first meta-anaysis, Cottingham and Hunter [22] studied the association between Chamydia trachomatis and ora contraceptive use. The anaysis was based on 29 case-contro studies and two prospective studies. Begg and Mazumdar excuded the prospective studies, and they are thus aso excuded from our exampe. In order to correct for tied vaues in the dataset, we appied the R function Kenda [25] to compute the rank correation based on Kenda s tau and its p-vaue when using the traditiona rank correation test. For the modified tests, we used the R function cor [20]. Appying the Begg and Mazumdar test, Kenda s tau equas 0.21, and we obtain a two-sided p-vaue of Hence we do not reject the hypothesis of no pubication bias, using a nomina significance eve of This vaue deviates from the resut reported by Begg and Mazumdar (p-vaue of 0.08, and ony some of this discrepancy can be expained by Begg and Mazumdar not accounting for ties. The adjusted tests based on Kenda s tau and Spearman s rho give midp-vaues of and The vaue of Spearman s rho is Both resuts indicate pubication bias, which is in agreement with the visua assessment of the funne pot [6]. Exampe 2 The second exampe [23,24] is a review of randomized experiments on the effects of teacher expectancy on pupi IQ. A tota number of 19 studies were incuded in the anaysis. The Begg and Mazumdar test, again using the R function Kenda to account for ties, gives a two-sided p-vaue of The rank correation coefficient equas Appying the adjusted method based on Kenda s tau, we obtain a mid-p-vaue of 0.067, and the adjusted method based on Spearman s rho gives a mid-p-vaue of Spearman s rho equas The resuts are consistent with the findings of the BeggandMazumdartestandthevisuainterpretationof the funne pot [2], athough a bit more conservative in the meta-anaysis setting. Discussion Athough the test introduced by Begg and Mazumdar is we known and often cited in pubished work (Figure 1, it has serious drawbacks. To our knowedge, these disadvantages have not been adequatey discussed in the iterature. The significance eve does not equa the nomina eve, the main reason being the non-zero conditiona covariance between the standardized treatment effects

13 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 Page 13 of 16 Tabe 7 Power for the adjusted Begg and Mazumdar test based on Spearman s rho: Sma meta-anayses* Power [% seected for incusion, bias] Seection strength Strong** Moderate*** Range of variances Large Sma Large Sma Treatment effect (δ.0 74% 24% 52% 16% [36%,.34] [37%,.74] [57%,.25] [57%,.54].5 69% 23% 39% 14% [54%,.16] [52%,.54] [74%,.09] [73%,.34] % 20% 26% 10% [65%,.07] [67%,.37] [82%,.04] [85%,.20] % 15% 17% 7% [72%,.05] [80%,.23] [87%,.03] [92%,.10] % 10% 12% 5% [78%,.03] [88%,.13] [90%,.02] [96%,.05] % 7% 9% 5% [82%,.02] [93%,.07] [93%,.01] [98%,.03] % 6% 8% 5% [86%,.02] [97%,.04] [94%,.01] [99%,.01] *k = 25 studies; nomina significance eve **a = 1.5, ***a = 3.0. v = 0.1, 1.0, 10.0, v = 0.5, 1.0, 2.0. given the variances. The adverse consequences are difficut to ignore. We propose improvements of the Begg and Mazumdar test when the outcome is assumed to be normay distributed. Athough it is more computer intensive, we recommend using the adjusted Begg and Mazumdar method when assuming fixed effects meta-anaysis, which attains significance eves that equa the nomina vaue. This reduces the Type II error. When outcomes are binary, however, there are tests particuary designed for handing the additiona difficuties that arise in this situation [5,13,14]. Sterne et a. [26] give an overview of these tests. Nevertheess, resuts of a recenty pubished simuation study [27] indicate that even these tests might not perform we in many situations. Overa, we advocate the use of the adjusted method based on Spearman s rho; it makes it easier to contro the Type I error rate for sma vaues of k compared to the adjusted test based on Kenda s tau. Sterne at a. [26] do not recommend tests for funne pot asymmetry when there are fewer than ten studies in the meta-anaysis, due to the fact that test power is usuay too ow to distinguish chance from rea asymmetry. Our findings support this reasoning. An additiona argument is that we are not abe to contro the significance eve for vaues of k smaer than ten. However, if Kenda s tau is used as a basis for the test instead of Spearman s rho, the minimum number of studies shoud be at east 16 in order to contro the significance eve. Our study has not evauated situations with heterogeneity in effects between studies; in these situations the minimum number of studies required may be substantiay higher than ten. Athough the improved tests are more powerfu than the test by Begg and Mazumdar, their genera power is sti imited, particuary for moderate amounts of bias and when the tota number of studies incuded in the meta-anaysis is typica of standard practice in medica appications. Tests for sma-study effects shoud routiney be performed prior to conducting a meta-anaysis. Nevertheess, it is important not to rue out the possibiity of sma-study effects when the tests do not produce significant resuts. Even when evidence of sma-study effects is found in the meta-anaysis, carefu consideration shoud be given to possibe expanations, e.g., pubication bias and heterogeneity. There are some imitations to our study that need to be addressed. We have ony regarded the fixed effects mode, i.e. no heterogeneity is assumed between the component studies in the meta-anaysis. This is consistent with the origina Begg and Mazumdar formuation [6]. The mode is therefore sufficient for demonstrating the

14 Gjerdevik and Heuch BMC Medica Research Methodoogy 2014, 14:109 Page 14 of 16 Tabe 8 Power for the adjusted Begg and Mazumdar test based on Spearman s rho: Sma meta-anayses* Power [% seected for incusion, bias] Seection strength Strong** Moderate*** Range of variances Large Sma Large Sma Treatment effect (δ.0 84% 36% 64% 25% [36%,.34] [36%,.74] [57%,.25] [57%,.54].5 80% 35% 53% 22% [54%,.16] [52%,.54] [74%,.09] [73%,.34] % 31% 38% 18% [65%,.07] [67%,.37] [82%,.04] [85%,.20] % 25% 27% 13% [72%,.05] [80%,.23] [87%,.03] [92%,.10] % 18% 22% 11% [78%,.03] [88%,.13] [90%,.02] [96%,.05] % 13% 17% 10% [82%,.02] [93%,.08] [93%,.01] [98%,.03] % 11% 14% 10% [86%,.02] [97%,.04] [94%,.01] [99%,.01] *k = 25 studies; nomina significance eve **a = 1.5, ***a = 3.0. v = 0.1, 1.0, 10.0, v = 0.5, 1.0, 2.0. adverse consequences of their test. To fuy understand the properties of our proposed method it shoud aso be assessed in a random effects mode. Such modes assign arger weights to sma studies, which are thus ess ikey to be triviaized, and this may affect the behaviour of our test. Nevertheess, such an assessment requires a whoe new simuation study and is beyond the scope of this paper. In addition, a more reaistic distribution of the component studies in the meta-anayses shoud be used in new simuations. Furthermore, the seection function considered in our paper is very simpistic and depends ony on the p-vaue. We beieve it is sufficient for iustrating the imitations of the Begg and Mazumdar test and for introducing our simuation procedure. Nevertheess, in a new simuation study more sophisticated seection modes, using more of the avaiabe information, shoud be empoyed. Baker and Jackson [28] and Bowden et a. [29] propose methods invoving information on journa impact factor and pubishing author. The adjusted test corrects the significance eve. It therefore forms a better basis for comparing the different test statistics introduced in the iterature, e.g., those presented by Egger et a. [8] and Macaski et a. [4]. Severa papers have compared different methods (e.g. [4,30]. However, as severa tests fai to give the correct Type I error, it is difficut to evauate and compare the performance of the test statistics. The genera idea behind our proposed simuation method, invoving the empirica distribution, is of course not nove. Nevertheess, we have not seen this simpe method used in the iterature of pubication bias. The procedure may not ony be appied to the rank correation tests but aso to other test statistics introduced in this fied. New simuation studies shoud be conducted comparing the performance of different statistics when we are abe to contro their significance eve. Unfortunatey, such studies are aso beyond the scope of our artice. The methods considered in this paper test for pubication bias in meta-anaysis. Severa authors address the issue of how to proceed if a test for pubication bias is significant. Duva and Tweedie [31,32] introduce an iterative non-parametric method to provide an adjusted estimate of the treatment effect, known as the trimand-fi method. The method is based on the symmetry property of the funne graph and is a usefu suppement to the methods discussed in this paper. Bürkner and Doeber [27] aso suggest using the trim-and-fi method as an option for handing pubication bias in meta-anaysis when outcomes are binary. The Copas seection mode [33] is a parametric statistica mode which combines the random effects mode with a seection mode. However,