Meta-analysis in Psychological Research.

Transcription

1 Internatonal Journal of Psychologcal Research, 010. Vol. 3. No. 1. ISSN mpresa (prnted ISSN electrónca (electronc Sánchez-Meca, J., Marín-Martínez, F., (010. Meta-analyss n Psychologcal Research. Internatonal Journal of Psychologcal Research, 3 (1, Meta-analyss n Psychologcal Research. l meta-análss en la nvestgacón pscológca. Julo Sánchez-Meca and Fulgenco Marín-Martínez Unversty of Murca, Span ABSTRAT Meta-analyss s a research methodology that ams to quanttatvely ntegrate the results of a set of emprcal studes about a gven topc. Wth ths purpose, effect-sze ndces are obtaned from the ndvdual studes and the characterstcs of the studes are coded n order to examne ther relatonshps wth the effect szes. Statstcal analyss n meta-analyss requres the weghtng of each effect estmate as a functon of ts precson, by assumng a fxed- or a randomeffects model. Ths paper outlnes the steps requred for carryng out the statstcal analyses n a meta-analyss, the dfferent statstcal models that can be assumed, and the consequences of the assumptons n nterpretng ther results. The statstcal analyses are llustrated wth a real example. Key words: Meta-analyss, effect sze, fxed-effects models, random-effects models, mxed-effects models. RSUM l meta-análss es una metodología de nvestgacón que pretende ntegrar cuanttatvamente los resultados de un conjunto de estudos empírcos sobre un determnado problema. on este propósto, se calculan índces del tamaño del efecto y se codfcan las característcas de los estudos con objeto de examnar su relacón con los tamaños del efecto. l análss estadístco en meta-análss requere ponderar cada estmacón del efecto en funcón de su precsón asumendo un modelo de efectos fjos o de efectos aleatoros. n este trabajo se presentan las etapas necesaras para realzar un metaanálss, los dferentes modelos estadístcos que pueden asumrse y las consecuencas de asumr dchos modelos en la nterpretacón de sus resultados. Fnalmente, los análss estadístcos se lustran con datos de un ejemplo real. Palabras clave: Meta-análss, tamaño del efecto, modelos de efectos fjos, modelos de efectos aleatoros, modelos de efectos mxtos. Artcle receved/artículo recbdo: December 15, 009/Dcembre 15, 009, Artcle accepted/artículo aceptado: March 15, 010/Marzo 15/010 Dreccón correspondenca/mal Address: Julo Sánchez-Meca, Dpto. Pscología Básca y Metodología, Facultad de Pscología, ampus de spnardo, Unversdad de Murca, Murca, Span, -mal: jsmeca@um.es Fulgenco Marín-Martínez, Unversty of Murca, Span INTRNATIONAL JOURNAL OF PSYHOLOGIAL RSARH esta ncluda en PSRINFO, NTRO D INFORMAION PSIOLOGIA D OLOMBIA, OPN JOURNAL SYSTM, BIBLIOTA VIRTUAL D PSIOLOGIA (ULAPSY-BIRM, DIALNT y GOOGL SHOLARS. Algunos de sus artculos aparecen en SOIAL SIN RSARH NTWORK y está en proceso de ncluson en dversas fuentes y bases de datos nternaconales. INTRNATIONAL JOURNAL OF PSYHOLOGIAL RSARH s ncluded n PSRINFO, NTRO D INFORMAIÓN PSIOLÓGIA D OLOMBIA, OPN JOURNAL SYSTM, BIBLIOTA VIRTUAL D PSIOLOGIA (ULAPSY-BIRM, DIALNT and GOOGL SHOLARS. Some of ts artcles are n SOIAL SIN RSARH NTWORK, and t s n the process of ncluson n a varety of sources and nternatonal databases. Internatonal Journal of Psychologcal Research 151

2 Internatonal Journal of Psychologcal Research, 010. Vol. 3. No. 1. ISSN mpresa (prnted ISSN electrónca (electronc Meta-analyss n Psychologcal Research 1. Introducton In the last 30 years meta-analyss has become a very useful methodologcal tool for accumulatng research on a gven topc. The huge growth of research n psychology has made t very dffcult to synthesze the results n any feld wthout the help of statstcal methods to summarze the evdence. Unlke tradtonal revews on a gven topc, whch are essentally subjectve n nature, meta-analyss ams to mbue the research revew wth the same scentfc rgor that s demanded of emprcal studes: objectvty, systematzaton and replcablty. Thus, metaanalyss s a method used to quanttatvely ntegrate the results of a set of emprcal studes on a gven research queston. Wth ths purpose, the results of each ndvdual study ncluded n a meta-analyss have to be quantfed n the same metrc, usually by calculatng an effect-sze ndex, and then the effect estmates are statstcally analyzed n order to: (a obtan an average estmate of the effect magntude, (b assess heterogenety among the effect estmates, and (c search for characterstcs of the studes that can explan the heterogenety (ooper, 010; ooper, Hedges, & Valentne, 009; Hunter & Schmdt, 004; Lpsey & Wlson, 001; Pettcrew & Roberts, 006. Sánchez-Meca, J., Marín-Martínez, F., (010. Meta-analyss n Psychologcal Research. Internatonal Journal of Psychologcal Research, 3 (1, Phases n a Meta-analyss A meta-analyss s a scentfc nvestgaton and, consequently, t nvolves carryng out the same phases as n an emprcal study. However, some of the phases have a few specfctes that t s necessary to menton. Bascally, we can conduct a meta-analyss n sx phases: (1 Defnng the research queston; ( lterature search; (3 codng of studes; (4 calculatng an effect-sze ndex; (5 statstcal analyss and nterpretaton, and (6 publcaton (ooper, 010; gger, Davey Smth, & Altman, 001; Lpsey & Wlson, 001; Lttell, orcoran, & Plla, 008; Sánchez- Meca & Marín-Martínez, 010. (1 Defnng the research queston. As n any emprcal study, the frst step n a meta-analyss s to defne the research queston as clearly and objectvely as possble. Ths mples proposng conceptual and operatonal defntons of the dfferent concepts and constructs related to the research queston. For example, n a meta-analyss about the effcacy of psychologcal treatments of obsessvecompulsve dsorder (OD, constructs such as psychologcal treatment, obsessve-compulsve dsorder, and the measurement tools to assess effcacy were defned n ths phase (Rosa-Alcázar, Sánchez-Meca, Gómez- onesa, & Marín-Martínez, 008. As meta-analyss ams to ntegrate sngle studes, the analyss unt s not the partcpant, but the sngle study. Therefore, the sample sze n a meta-analyss s the number of studes that t has been possble to recover regardng the research queston. Meta-analyss s beng appled n many dfferent felds n psychology, but especally n evaluatng the effectveness of treatments, nterventons, and preventon programs n such settngs as mental health, educaton, socal servces, or human resources. Other psychologcal felds where meta-analyss s also beng appled nclude areas such as gender dfferences n chldhood, adolescence or wth adults of many apttudes and atttudes; psychometrc valdty of employment tests, and relablty generalzaton of psychologcal tests n general (ook, ooper, ordray et al., 199. Nowadays, t s very common to fnd meta-analytc studes on very dfferent topcs n any scentfc psychology journal. Therefore, clncans and researchers should have a suffcent knowledge base for correctly nterpretng and/or carryng out meta-analyses. ( Lterature search. Once the research queston s formulated, the next step conssts of defnng the elgblty crtera of the sngle studes, that s, the characterstcs a study must fulfll n order to be ncluded n the meta-analyss. The selecton crtera wll depend on the purpose of the meta-analyss, but t s always necessary to specfy the types of study desgns that wll be accepted (e.g., only expermental desgns, or also quas-expermental ones, etc.. For example, n the meta-analyss on OD (Rosa-Alcázar et al., 008 n order to be ncluded n the meta-analyss the studes had to fulfll several crtera: (a to apply a psychologcal treatment to adult patents wth OD; (b to nclude a control group wth OD patents; (c to report statstcal data for calculatng the effect szes; (d to have at least 5 partcpants n each group, and (e to be publshed between 1980 and 006. In ths phase the dfferent strateges used to locate the sngle studes are also specfed. No meta-analyss s complete wthout a search of electronc databases specfyng the keywords used (e.g., PsycInfo, MedLne, RI. Ths search strategy s usually complemented by carryng out searches by hand of relevant journals and books for the topc of nterest, and by checkng the references of the papers ncluded n the meta-analyss. Addtonally, t s very advsable to try to locate unpublshed papers that mght fulfll the selecton crtera, n order to counteract publcaton bas. Ths can be done by Ths artcle s dvded nto four sectons. Frstly, the phases n whch a meta-analyss s carred out are presented. Then we outlne the man statstcal methods n meta-analyss. In the next secton statstcal methods for meta-analyss are llustrated usng a real example. Fnally, we present some concludng remarks. 15 Internatonal Journal of Psychologcal Research

3 Internatonal Journal of Psychologcal Research, 010. Vol. 3. No. 1. ISSN mpresa (prnted ISSN electrónca (electronc Sánchez-Meca, J., Marín-Martínez, F., (010. Meta-analyss n Psychologcal Research. Internatonal Journal of Psychologcal Research, 3 (1, sendng letters to well-known researchers n the feld requestng unpublshed papers about the topc. (3 odng of studes. Once we have the sngle studes ncluded n the meta-analyss, the next step s to record the man characterstcs of the studes n order to later explan the heterogenety exhbted by the effect szes. The characterstcs of the studes, or moderator varables, are classfed as substantve, methodologcal, and extrnsc varables. Substantve characterstcs are those related to the research queston of the meta-analyss, whereas methodologcal varables are characterstcs related to the study desgn. Fnally, extrnsc varables refer to those characterstcs that, despte are not related wth the subjects nor the study desgn, could also have an nfluence n the results. In the OD example (Rosa-Alcázar et al., 008, substantve characterstcs coded n the studes ncluded the type of psychologcal treatment (e.g., cogntve therapy, exposure technques, the mean age of the partcpants and the llness hstory (n years. Some of the methodologcal characterstcs coded ncluded the type of desgn (expermental versus quas-expermental, attrton n the posttest, and the sample sze. Moreover, extrnsc varables such as the country where the study was carred out and the educaton profle of the man author were also coded. The codng norms of the moderator varables are wrtten n a codebook. Some study characterstcs are dffcult to code due to ncomplete or ambguous reportng n the sngle studes. Therefore, the relablty of the codng process should be analyzed. To ths end, two (or more researchers should ndependently apply the codebook to all or a sample of the sngle studes. Then, usng the codng records made by the researchers, agreement ndces are appled (e.g., kappa coeffcents, ntraclass correlatons n order to assess the relablty of the codng process. (4 alculatng an effect-sze ndex. In the codng process of sngle studes, an effect-sze ndex also has to be calculated n order to quantfy the results of each study n a common metrc. Dependng on the study desgn and the type of dependent varables (contnuous, dchotomous, dfferent effect-sze ndces can be appled. Thus, when the studes have a two-group desgn and the outcome measure s contnuous, the most approprate effect-sze ndex s the standardzed mean dfference or d. Ths s defned as the dfference between the two means dvded by a pooled wthn-study standard devaton. Furthermore, when the dependent varable s dchotomous, several rsk ndces can be appled: (a the rsk dfference, rd, defned as the dfference between the falure (or success proportons for the two groups; (b the rsk rato, rr, defned as the rato between the two proportons, and (c the odds rato, or, defned as the rato between the odds of the two groups. Fnally, when the study appled a correlatonal desgn, a correlaton coeffcent can be used as the effect-sze ndex (e.g., the Pearson correlaton coeffcent, ts Fsher s Z transformaton, the pont-bseral correlaton coeffcent, the ph coeffcent, etc.. Table 1 presents some of the usual effect-sze ndces appled n meta-analyss together wth ther estmated samplng varances, σˆ, as they are used n the statstcal analyses of a meta-analyss (cf. Borensten, Hedges, Hggns, & Rothsten, 009; ooper et al., 009. Once the effect-sze ndex most approprate to the characterstcs of the studes has been selected, t s appled to each sngle study and ts samplng varance s also calculated wth the correspondng formulas (cf., e.g., Borensten et al., 009. When a meta-analyss ncludes studes wth dfferent desgns (e.g., correlatonal and twogroup desgns, there are formulas to transform dfferent effect-sze ndces nto each other. For example, t s possble to transform correlaton coeffcents nto d ndces, and vce versa; or odds ratos nto d ndces (Sánchez-Meca, Marín-Martínez, & hacón-moscoso, 003. (5 Statstcal analyss and nterpretaton. The dataset n a meta-analyss s composed of a matrx where the rows are the studes and the columns are the moderator varables, the effect-sze ndex calculated n each study, and ts samplng varance. Wth these data t s possble to carry out statstcal analyses, whch have the followng three man objectves: (1 to calculate an average effect sze and ts confdence nterval; (b to assess the heterogenety of the effect szes around the average, and (c to search for moderator varables that can explan the heterogenety (Sutton & Hggns, 008. The man characterstc of metaanalyss s that statstcal methods are used for ntegratng the study results. More detals about how to statstcally analyze a meta-analytc database are presented n the next pont of ths artcle. (6 Publcaton. Fnally, the results of a metaanalyss have to be publshed followng the same structure as any other scentfc paper: Introducton, method, results, and dscusson and conclusons (Botella & Gambara, 006; Rosenthal, A lterature revew on the topc s outlned n the ntroducton, together wth defntons of the constructs and varables mpled n the research queston, and the objectves and hypotheses of the meta-analyss. In the method secton the followng should be ncluded: the selecton crtera of the studes, the search strategy of the studes, the codng process of the study characterstcs, the effect-sze ndex calculated n the sngle studes, and the statstcal analyses that were carred out n the metaanalytc ntegraton. In the results secton the characterstcs of the studes are presented, together wth the effect-sze dstrbuton, the mean effect sze, the heterogenety assessment, and the results of the statstcal analyses for searchng for moderator varables related to the effect szes. Fnally, n the dscusson and concluson secton the results Internatonal Journal of Psychologcal Research 153

4 Internatonal Journal of Psychologcal Research, 010. Vol. 3. No. 1. ISSN mpresa (prnted ISSN electrónca (electronc of the meta-analyss are compared wth prevous ones, the mplcatons for future research are mentoned, and the Sánchez-Meca, J., Marín-Martínez, F., (010. Meta-analyss n Psychologcal Research. Internatonal Journal of Psychologcal Research, 3 (1, lmtatons and the man conclusons of the meta-analyss are also outlned. Table 1. ffect-sze ndces and ther respectve estmated wthn-study samplng varances ffect-sze ndex T stmated samplng varance, Mean dfference Standardzed mean dfference Rsk dfference Natural logarthm of the rsk rato Natural logarthm of the odds rato D y y 3 y d rd p p y S Lrr Loge ( p / p Lor Log e p p (1 p (1 p n + n V ( d n n V ( rd p S S V( D + n n (1 p n σˆ d + ( n + n 1 p V ( Lrr n p + p 1 p + n p (1 p n V ( Lor a b c d Pearson correlaton (1 rxy V ( r coeffcent r xy xy y and Fsher s Z Z r Log e 1+ r 1 r y : means for expermental and control groups. xy xy S and 1 V ( Z r 3 S : varances for expermental and control groups. n and n : sample szes for expermental and control groups. S: pooled standard devaton of the two groups. n + n. p and p : success (or falure proportons for expermental and control groups. a, b, c, and d: cell frequences of success and falure for expermental and control groups. 3. Statstcal Methods n Meta-analyss The man characterstc of meta-analyss s the use of statstcal methods to ntegrate the study results. In order to do ths, an effect sze estmate s calculated from each sngle study as well as a set of moderator varables (substantve and methodologcal characterstcs that can explan the varablty n the effect sze dstrbuton. The statstcal analyss n a meta-analyss proceeds n three steps (Lpsey & Wlson, 001: (1 the obtanng of an average effect sze and a confdence nterval around t; ( the assessment of the heterogenety of the effect szes, and (3 f there s a large heterogenety, the search for moderator varables that may be related to the effect szes. The effect szes obtaned from the sngle studes dffer among themselves n terms of ther precson, as they are calculated from dfferent sample szes. ffect szes obtaned from large samples are more accurate than those obtaned from small ones. As a consequence, statstcal methods n meta-analyss take nto account the accuracy of each effect sze by weghtng them as a functon of ts precson (Marín-Martínez & Sánchez-Meca, n press; Sánchez-Meca & Marín-Martínez, In partcular, statstcal theory shows that the most approprate method (n terms of the mnmum varance estmate for weghtng effect szes n a meta-analyss nvolves usng the nverse varance of each effect sze estmate as the weghtng factor (ooper et al., 009; Hedges & Olkn, (1 Averagng effect szes. The frst step n the statstcal analyses conssts n calculatng an average effect sze that summarzes the overall effect magntude of the meta-analyzed studes. The statstcal model for carryng out these calculatons assumes a random-effects model, whch consders that the effect sze, T, n each sngle study s estmatng a dfferent populaton effect sze, θ, that s, T θ + u, where u represents the samplng error n T due to 154 Internatonal Journal of Psychologcal Research

5 Internatonal Journal of Psychologcal Research, 010. Vol. 3. No. 1. ISSN mpresa (prnted ISSN electrónca (electronc the fact that the sngle study s based on a random sample selected from the populaton of potental partcpants (Feld, 003; Hedges & Vevea, 1998; Schmdt, Oh, & Hayes, 009. The samplng error s quantfed through the wthnstudy samplng varance, σ. Thus, t s assumed that n a gven meta-analyss the ncluded studes consttute a random sample of the studes whch could have been carred out about the same topc. Moreover, for the ncluded studes t s almost sure that the research condtons dffer someway (e.g., n the therapst s experence, the treatment s desgn and length, etc., so t s reasonable to suspect that the effect szes could vary owng to these dfferences. Thus, a dstrbuton of populaton effect szes, θ, wth a mean populaton effect sze, µ θ, s assumed, that s, θ µ θ + ε, wth ε beng the devatons of the populaton effect szes from ts mean. The varablty of the populaton effect szes s called the between-studes varance, τ, or heterogenety varance. Hence, n a random-effects model t s assumed that each effect sze estmate ncludes two varablty sources: the wthn-study varance, σ, and the betweenstudes varance, τ. The statstcal model can be formulated as: T µ θ +ε +u. (1 When ε 0, then the random-effects model becomes a fxed-effects model, where there s only one varablty source, the wthn-study varance σ, and all of the studes are estmatng the same populaton effect sze. Thus, the statstcal model s smplfed to T µ θ + u, and µ θ θ. In practce the meta-analyst wll have to decde whch statstcal model to apply, the fxed- or the randomeffects model. The consequences of assumng a randomeffects model or a fxed-effects one concern the nterpretaton of the results and also the results obtaned themselves. On the one hand, a meta-analyst that apples a fxed-effects model s assumng that hs/her results can only be generalzed to an dentcal populaton of studes to that of the ndvdual studes ncluded n the meta-analyss, whereas n a random-effects model the results can be generalzed to a wder populaton of studes. On the other hand, the error attrbuted to the effect sze estmates n a fxed-effects model s smaller than n a random-effects model, whch s why n the frst model the confdence ntervals are narrower and the statstcal tests more lberal than n the second one. The man consequence of assumng a fxed-effects model when the meta-analytc data come from a random-effects model s that we may attrbute more precson to the effect sze estmates than s really approprate and that we may fnd statstcally sgnfcant relatonshps between varables that are actually spurous. Sánchez-Meca, J., Marín-Martínez, F., (010. Meta-analyss n Psychologcal Research. Internatonal Journal of Psychologcal Research, 3 (1, onsequently, t s more realstc to assume random-effects models n meta-analyss. In order to apply statstcal nference, t s usually assumed that the effect sze dstrbuton, T, n a randomeffects model follows a normal dstrbuton wth populaton mean µ θ and varance equal to the sum of the two varablty sources, σ + τ, that s, T (µ θ ; σ + τ. Thus, the unformly mnmum varance unbased estmator of µ θ, T U, s gven by (Vechtbauer, 005: wt TU w, ( where w are the optmal weghts, defned as w 1 σ + τ. The varance of T U s gven by: ( 1 V ( TU. (3 w However, n practce the optmal weghts cannot be calculated, because the parametrc wthn-study varances, σ, and the parametrc between-studes varance, τ, are unknown. Therefore, the two knds of varance have to be estmated from the data. In general, good estmators of the wthn-study varance for the dfferent effect-sze ndces have been proposed n the meta-analytc lterature (cf. e.g., Borensten et al., 009. About a dozen dfferent estmators have been proposed for estmatng the between-studes varance (Sánchez-Meca & Marín-Martínez, 008; Vechtbauer, 005. Of these, the most usually appled are those based on the moments method,, proposed by DerSmonan and Lard (1986, and the one based on restrcted maxmum lkelhood, (Thompson & Sharp, The moments method estmator s gven by: Q ( k 1, (4 c where k s the number of studes n the metaanalyss; Q s the heterogenety statstc defned as: Q ~ ( T T w ~, (5 Internatonal Journal of Psychologcal Research 155

6 Internatonal Journal of Psychologcal Research, 010. Vol. 3. No. 1. ISSN mpresa (prnted ISSN electrónca (electronc wth w ~ 1 beng the estmated weghts by σˆ assumng a fxed-effects model, and T ~ beng the average effect sze also by assumng a fxed-effects model, that s: ~ T wt ~ w~. Fnally, n equaton (4, c s obtaned by: ( w ~ c w~ w~. (6 In quaton (4, when Q < (k 1, then s truncated to 0 to avod negatve values. The between-studes varance estmator based on restrcted maxmum lkelhood,, s obtaned by teratng untl convergence the equaton (Thompson & Sharp, 1999: [( T T σˆ ] 1 + (7 wth w ˆ 1 (σˆ +, where s ntally 0 or t s estmated by any of the nonteratve estmators of the between-studes varance (e.g., and T s gven by: T T. (8 In each teraton of quatons (7 and (8, each estmate of τ must be checked n order to avod negatve values. Once we have an estmate of the between-studes varance ( or ther estmated wthn-study varances, calculate an average effect sze by: T T and the effect estmates, T, and σˆ, t s possble to, (9 wth w ˆ 1 (σˆ +. Then a confdence nterval for T s usually calculated by assumng a normal dstrbuton: Sánchez-Meca, J., Marín-Martínez, F., (010. Meta-analyss n Psychologcal Research. Internatonal Journal of Psychologcal Research, 3 (1, T ± z V(, (10 α / T where z α/ s the 100(α/ percentle of the standard normal dstrbuton, α beng the sgnfcance level; and V (T s the samplng varance of the average effect sze, whch s obtaned by: 1 V ( T. (11 ˆ w Although quaton (10 s the usual procedure for calculatng a confdence nterval around the overall effect sze, ths method does not take nto account the uncertanty produced by the fact that the wthn-study and the betweenstudes varances have to be estmated. As a consequence, the confdence nterval wll underestmate the nomnal confdence level. A confdence nterval that better fts the nomnal confdence level s that proposed by Hartung (1999; see also Sánchez-Meca & Marín-Martínez, 008; Sdk & Jonkman, 003, 006, whch assumes a Student t- dstrbuton wth k 1 degrees of freedom and estmates the samplng varance of the overall effect sze by an mproved formula: T ± t V, (1 ( k 1, α / W T where V W( T s the mproved samplng varance estmate and s gven by: ( T T VW ( T ( k 1. (13 Fnally, together wth the average effect sze and ts confdence nterval, t s very nformatve to present a graph that was specally developed for meta-analyss named forest plot. A forest plot s a graphcal presentaton of each effect sze estmate wth ts confdence nterval and the overall effect sze also wth ts confdence nterval. Thus, a forest plot s somethng lke a photograph of the effect estmates obtaned n the meta-analyss (Borensten et al., 009; Hggns & Green, 008. ( Assessng heterogenety. Whlst t s mportant n meta-analyss to obtan an overall effect sze, t s even more mportant to assess the heterogenety of the effect estmates around ts mean. We need to know whether the varablty n the effect szes s due only to samplng error or f there s more varablty than can be explaned by samplng error. Ths queston s usually answered by 156 Internatonal Journal of Psychologcal Research

7 Internatonal Journal of Psychologcal Research, 010. Vol. 3. No. 1. ISSN mpresa (prnted ISSN electrónca (electronc applyng the heterogenety Q statstc, whch was defned n quaton (5. Under the null hypothess of heterogenety due only to samplng error, the Q statstc follows a hsquare dstrbuton wth k 1 degrees of freedom. Thus, by comparng Q wth the 100(1 α percentle of χ k 1 dstrbuton, t s possble to make a statstcal decson about ths queston. When a meta-analyss has a small number of studes, the Q statstc has low statstcal power (Harwell, 1997; Sánchez-Meca & Marín-Martínez, Thus, t s usual to assess heterogenety by complementng the Q statstc wth the I ndex, a percentage that nforms us about the extent of varablty n the effect sze dstrbuton due to true heterogenety (that s, heterogenety not due to samplng error, but to the nfluence of many dfferent moderator varables. The I ndex s calculated by (Hggns & Thompson, 00: Q ( k 1 I 100. (14 Q When Q < (k 1 then I s truncated to 0. Hggns and Thompson (00 proposed a tentatve classfcaton of I by statng that I values around 5%, 50%, and 75% can be consdered as reflectng small, medum, and large heterogenety, respectvely. (3 Searchng for moderator varables. When the Q statstc acheves a statstcally sgnfcant result and the I ndex s of medum to large magntude, then the overall effect sze calculated n the frst step of the statstcal analyses does not adequately represent all of the study results. As a consequence, the next step n the analyses conssts n searchng for moderator varables that can explan the heterogenety. In ths phase of the analyss, the effect estmates, T, act as the dependent varable, whereas the moderator varables are potental predctors that may be related to the effect estmates. Dependng on the categorcal or contnuous nature of the moderator varables, analyses of varance (ANOVAs or regresson analyses are appled n order to examne the nfluence of these predctors on the effect magntude. In all cases, however, weghtng methods are appled that take nto account the precson of the effect estmates. In partcular, the most approprate statstcal model for testng the nfluence of moderator varables n meta-analyss s to assume a mxed-effects model, where the moderator varable s the fxed-effects component and the studes are the random-effects component n the model (Konstantopoulos & Hedges, 009; Raudenbush, 009. For categorcal moderator varables, ANOVAs are appled by weghted least squares estmaton. An ANOVA for testng the sgnfcance of a categorcal moderator Sánchez-Meca, J., Marín-Martínez, F., (010. Meta-analyss n Psychologcal Research. Internatonal Journal of Psychologcal Research, 3 (1, varable wth m categores conssts of calculatng a weghted average effect sze for each category, T j, and obtanng the Q B statstc by: Q ( T j T m w ˆ B j, (15 j wth w ˆj 1 V ( T j, and V ( T j 1 j. The Q B 0 θ 1 µ θ m statstc s the weghted between-categores sum of squares of the ANOVA. Under the null hypothess of no dfference between the mean effect szes for the m categores ( H : µ..., the Q B statstc follows a hsquare dstrbuton wth m 1 degrees of freedom. Thus, from comparng the Q B statstc wth the 100(1 α percentle of χ dstrbuton, t s possble to decde m 1 whether the moderator varable s statstcally related to the effect sze. The result of Q B s complemented wth a msspecfcaton test that can be appled separately for each category of the moderator varable. Thus, the Q statstc for the jth category s obtaned by: Q m j ( T T j w ˆ Wj j j m j. (16 A dfferent Q W j statstc s calculated for each category of the moderator varable n order to examne the heterogenety of the effect szes wthn a gven category. Thus, under the null hypothess of homogeneous effect szes n the jth category, the Q statstc follows a hsquare dstrbuton wth m j 1 degrees of freedom, where m j s the number of effect szes n the jth category. Therefore, by comparng Q wth the 100(1 α percentle of m j 1 W j W j χ dstrbuton, t s possble to decde whether the effect szes n the jth category are homogeneous. In addton, a global msspecfcaton test for all ANOVA model conssts of calculatng the sum of the m Q statstcs as follows: W j Q W Q W Q. (17 1 W m The Q W statstc s the weghted wthn-categores sum of squares of the ANOVA. Thus, under the null hypothess of global homogenety for all categores, the Q W statstc follows a h-square dstrbuton wth k m W j Internatonal Journal of Psychologcal Research 157

8 Internatonal Journal of Psychologcal Research, 010. Vol. 3. No. 1. ISSN mpresa (prnted ISSN electrónca (electronc degrees of freedom. By comparng Q W wth the 100(1 α percentle of χ dstrbuton, t s possble to decde k m whether the ANOVA model s globally msspecfed. When the moderator varable s contnuous or we are nterested n examnng the nfluence of a set of moderator varables (contnuous and/or categorcal, weghted smple or multple lnear regresson models can be appled. By assumng a mxed-effects model, where the moderator varables are the fxed-effects component and the studes the random-effects component, the lnear model s gven by: T Xβ + u + ε, (18 wth T beng a k by 1 vector of effect sze estmates wth elements {T }, X s a k by P matrx of predctors, wth P p + 1 columns (p beng the number of predctors, β s a P by 1 vector of parametrc regresson coeffcents wth elements {β j }, u s a k by 1 vector of wthn-study estmaton errors wth elements {u }, and ε s a scalar wth the between-studes varance {ε}. T has varance V(u + ε τ I + V, wth I beng a k by k dentty matrx and V beng a k by k dagonal matrx wth elements { v σ + τ }. The vector of regresson coeffcents, β, s estmated by: βˆ -1 ( X'WX X'WT, (19 ˆ 1 wth W V, W beng a k by k dagonal matrx wth the weghts for each effect sze, { ŵ }, whch are estmated by the nverse of the sum of the wthn-study and the betweenstudes varances: w ˆ 1 (σˆ +. In ths case, the between-studes varance s estmated by an extenson of quatons (4 or (7 to the case of a regresson model wth p predctors. For example, an extenson of the moments method estmator s gven by: Q ( 1 k p, (0 tr( W tr 1 [ WX( X' WX X' W] where Q s the weghted resdual sum of squares of the model and s obtaned by: Q T' WT -. (1 Q R The between-studes varance estmator based on restrcted maxmum lkelhood for a weghted regresson model can be consulted n Thompson and Sharp (1999. Sánchez-Meca, J., Marín-Martínez, F., (010. Meta-analyss n Psychologcal Research. Internatonal Journal of Psychologcal Research, 3 (1, A test for the statstcal sgnfcance of the full model s gven by the Q R statstc, whch s the weghted regresson sum of squares and s gven by: Q βˆ 'S -1ˆ β, ( R ˆ β where S s the matrx of varances and covarances for βˆ the regresson coeffcents. Under the null hypothess of no relatonshp between the composte of predctors and the effect szes (H 0 : β 0, Q R follows a h-square dstrbuton wth P degrees of freedom. By comparng Q R wth the 100(1 α percentle of χ P dstrbuton, t s possble to decde f the full model shows a statstcally sgnfcant relatonshp wth the effect sze. At the same tme, statstcal tests for ndvdual predctors can also be appled n order to examne the nfluence of each predctor once that of the other predctors n the model has been partalzed. For a gven regresson coeffcent, ˆ β, the null hypothess of no effect s tested by: βˆ j Z, (3 V ( βˆ j wth V ˆ beng the jth dagonal element of the β ( j P by P matrx for the varances and covarances of the regresson coeffcents. Thus, comparng Z wth the 100(1 α/ percentle of the standard normal dstrbuton, t s possble to determne the statstcal sgnfcance of a gven predctor n the multple regresson model. Fnally, a specfcaton test of the regresson model s appled by means of the Q statstc defned n quaton (1. Under the null hypothess that the model s well specfed (H 0 : τ WLS 0, Q follows a h-square dstrbuton wth k p 1 degrees of freedom. Thus, by comparng Q wth the 100(1 α percentle of j k p 1 dstrbuton, t s possble to examne the model msspecfcaton. 4. An Illustratve xample In order to llustrate the calculatons n a typcal meta-analyss, Table presents some of the data obtaned n a meta-analyss on the effcacy of psychologcal treatments for obsessve-compulsve dsorder (OD; Rosa- Alcázar et al., 008. Ths meta-analyss s composed of 4 studes that compared two groups of patents wth OD, one recevng a psychologcal treatment (expermental χ 158 Internatonal Journal of Psychologcal Research

9 Internatonal Journal of Psychologcal Research, 010. Vol. 3. No. 1. ISSN mpresa (prnted ISSN electrónca (electronc Sánchez-Meca, J., Marín-Martínez, F., (010. Meta-analyss n Psychologcal Research. Internatonal Journal of Psychologcal Research, 3 (1, group and the other one not recevng treatment (control group. The effect-sze ndex calculated n each study was the standardzed mean dfference, d, defned as the dfference between the means for the treatment and control groups dvded by a pooled estmate of the standard devatons for the two groups. Postve values for d ndcated a lower level of obsessons and compulsons after treatment n the treated group n comparson wth the control group, whereas negatve values for d ndcated a hgher level. Table also ncludes the sample szes for the two groups (n and n, as well as the estmated wthnstudy samplng varance for each effect sze ( σˆ. Table. Dataset of the meta-analyss about the effcacy of psychologcal treatments for OD. Study Year Desgn n n d σˆ Desgn type: 1, quas-expermental;, expermental. n and n are the sample szes for the expermental and control groups, respectvely. d s the standardzed mean dfference between the means for the expermental and control groups. σˆ s the estmated wthn-study samplng varance. In the example, d values correspond to the term T used n the prevous secton to represent the effect estmates. The statstcal analyses should begn wth a forest plot to graphcally represent the ndvdual effect estmates and ther confdence ntervals, together wth an average effect sze. Fgure 1 presents a forest plot for the example data. Fgure 1. Forest plot for the example data. The effect-sze ndex s the standardzed mean dfference, d. The mean effect sze was calculated by assumng a randomeffects model wth. The confdence nterval for the mean effect sze was calculated from the classcal method. Once we have a general mpresson of the effect sze dstrbuton, the statstcal analyses begn by calculatng an average effect estmate. By assumng a random-effects model, ths mples estmatng the betweenstudes varance, τ. In ths paper we have presented two alternatve estmators of τ : that based on the moments method,, and that based on restrcted maxmum lkelhood, those obtaned from a random-effects model: d Internatonal Journal of Psychologcal Research 159 Study Mean d (and 95% I Heterogenety: ; Q 53.45, df 3 (p ; I² 56.97% Std. Mean Dfference and 95% I 1.43 [0.39,.46] 1.07 [0.44, 1.69] 0.9 [0.40, 1.45] 0.91 [0.38, 1.44] 0.8 [-0.44, 1.00] 1.65 [0.94,.35] 1.01 [0.3, 1.78] 1.00 [0.6, 1.38] 0.73 [0.36, 1.10] 1.88 [0.85,.91] 1.08 [0.30, 1.87].33 [1.13, 3.5] 0.3 [-0.91, 1.36] 0.19 [-0.69, 1.07] 0.98 [0.37, 1.59] 1.6 [0.94,.30] 3.00 [1.65, 4.35] 0.86 [-0.3,.04] 1.49 [0.50,.49] 0.60 [-0.0, 1.39] 0.67 [-0.01, 1.36] 0.49 [-0.18, 1.16] 3.78 [.08, 5.48] 1.59 [0.56,.6] [0.84, 1.31] Std. Mean Dfference and 95% I Favours control Favours treatment. By applyng quatons (4 and (7 to the example data, we obtan and For comparson purposes, Table 3 presents dfferent average effect szes and confdence ntervals dependng on the statstcal model assumed (fxed- versus random-effects model, the between-studes varance estmator ( versus, and the confdence nterval method (classcal versus mproved by Hartung, The weghted mean effect sze that we obtaned by applyng quaton (9 was d when we used n the weghtng factor, and d for. Thus, changng the between-studes varance estmator does not seem to affect the mean effect sze estmate. Assumng a fxed-effects model the mean effect sze s also smlar to

10 Internatonal Journal of Psychologcal Research, 010. Vol. 3. No. 1. ISSN mpresa (prnted ISSN electrónca (electronc Sánchez-Meca, J., Marín-Martínez, F., (010. Meta-analyss n Psychologcal Research. Internatonal Journal of Psychologcal Research, 3 (1, Table 3. Summary statstcs for the average effect sze and ts confdence nterval calculated from dfferent methods Statstcal model τ estmator I method d 95%. I. d l d u Wdth of the I R model lasscal R model Improved R model lasscal R model Improved F model -- lasscal R: random-effects model. F: fxed-effects model. I: confdence nterval. d: average effect sze. d l and d u : lower and upper confdence lmts for the average effect sze. Followng ohen s (1988 benchmarks for nterpretng the practcal sgnfcance of an effect sze, we can consder that d values around 0.0, 0.50, and 0.80 can be nterpreted as reflectng an effect of small, medum, and large magntude, respectvely. Therefore, a mean effect sze n our example of d can be nterpreted as ndcatng a hgh effect of psychologcal treatments n reducng obsessons and compulsons of patents wth OD. Table 3 also shows confdence ntervals for the average effect sze dependng on the method selected (classcal versus mproved by Hartung, 1999 and on the between-studes varance estmator (moments method versus restrcted maxmum lkelhood. Wth the classcal method for calculatng a confdence nterval around the mean effect sze the wdth of the confdence nterval (0.463 and for and, respectvely was smaller than that of the mproved method (0.577 and 0.57 for and. The classcal method s, therefore, slghtly more lberal n comparson wth the mproved method. The most lberal method, however, s the confdence nterval whch assumes a fxed-effects model as t does not take nto account the between-studes varablty among the effect szes. Once we have an estmate of the overall effect magntude n the meta-analyss, the next step n the analyses conssts of assessng the heterogenety of the effect szes. By applyng quaton (5 to our example data, we obtaned Q( , p.0003, whch enabled us to reject the null hypothess of homogenous effect szes. The statstcally sgnfcant result for the Q statstc s complemented wth the calculaton of the I ndex by quaton (14, reachng a moderate heterogenety, I 56.97%. Therefore, we can conclude that the effect szes were clearly heterogeneous and, as a consequence, the next step n the analyses s to search for moderator varables whch are able to explan the effect sze varablty. In order to llustrate how to test dfferent moderator varables on the effect szes, here we have selected two of them: one categorcal varable and the other contnuous. As an example of a categorcal moderator varable, we have selected the desgn type, dstngushng between expermental (random assgnment to the groups versus quas-expermental desgns (nonrandom assgnment. For comparson purposes, Table 4 shows the weghted ANOVA results for the desgn type by assumng a mxed-effects model wth two dfferent estmators of the between-studes varance ( and and a fxedeffects model. In the three cases we obtaned, usng quaton (15, a nonstatstcally sgnfcant result for the Q B statstc, leadng to the concluson that the type of desgn does not seem to affect the effect szes, although quasexpermental desgns presented a mean effect sze thatwas slghtly lower than that of the expermental ones. Wecan also observe how the Q B statstc for the fxed-effects model was the most lberal of the three models appled. Table 4. Results of the weghted A OVA appled on the desgn type by assumng a random-effects model wth and, and for a fxed-effects model. Mxed-ffects Model wth Desgn type 95%. I. k d j d l d u Quas-exptal xptal ANOVA Q B ( , p.18 results Q W ( , p.098 Mxed-ffects Model wth %. I. k Desgn type d j d l d u Quas-exptal xptal Q Wj Q Wj D F 3 19 D F 3 19 p p Internatonal Journal of Psychologcal Research

11 Internatonal Journal of Psychologcal Research, 010. Vol. 3. No. 1. ISSN mpresa (prnted ISSN electrónca (electronc ANOVA results Q B (1 1.75, p.189 Q W ( , p.039 Fxed-ffects Model Desgn type k 95%. I. d j Quas-exptal xptal ANOVA results d l d u Q B (1.37, p.14 Q W ( , p.0004 Q Wj D F 3 19 p The specfcaton test dd not reach the same results n the three models. Thus, assumng a mxed-effects model, the Q W statstc calculated by quaton (17 reached statstcal sgnfcance dependng on the between-studes varance estmator used: p.098 for and p.039 for. For the fxed-effects model the Q W statstc reached statstcal sgnfcance. Therefore, dependng on the statstcal model assumed, we can conclude that the categorcal model was msspecfed or not. A more detaled analyss of the model specfcaton conssts n examnng the separate Q Wj statstcs calculated by quaton (16 n relaton to each category of the moderator varable, n order to determne whch categores were homogeneous around ts mean effect sze. Table 4 shows that the four effect szes n the category quas-expermental desgn seemed to be homogeneous around ts mean, whereas the 0 effect szes n the category expermental desgn dd not seem to be homogeneous. Table 5.. Results of the weghted regresson analyss appled on the publcaton year by assumng a random-effects model wth and, and for a fxed-effects model. To llustrate how to analyze the nfluence of a contnuous moderator varable on the effect szes, we have selected the year of publcaton of the study and appled weghted regresson analyses by assumng a fxed-effects or a mxed-effects model wth two between-studes varance estmators ( and. Table 5 presents the man results for the three models appled. By applyng quatons ( and (3, t s possble to test f there s a statstcally sgnfcant relatonshp between the moderator varable and effect sze. As Table 5 shows, n all three cases we found a statstcally sgnfcant result for the Q R and the Z statstcs. For example, by assumng a mxed-effects model wth 0.156, calculated by quaton (0, we obtaned Q R ( , p.035, or Z.107, p.035. Note that n a smple regresson model the statstcal sgnfcance for testng the full model, Q R, concdes wth that of the Z test for the moderator varable. The postve sgn of the Sánchez-Meca, J., Marín-Martínez, F., (010. Meta-analyss n Psychologcal Research. Internatonal Journal of Psychologcal Research, 3 (1, regresson coeffcent, ˆ β 1, calculated by quaton (19, for the year of publcaton means that the most recent studes showed larger effect szes than those of the older studes. Table 5 also presents the results of testng the model specfcaton wth Q statstc by applyng quaton (1. In ths case, dependng on the statstcal model appled the result for Q ether reached or dd not reach statstcal sgnfcance. Thus, assumng a mxed-effects model the msspecfcaton test dd not reach statstcal sgnfcance when usng [Q ( 8.830, p.150], whereas t was margnally statstcally sgnfcant when usng (p.065. Assumng a fxed-effects model, the msspecfcaton test was hghly statstcally sgnfcant (p Regress. oeff. onstant Year Full model results Mxed-ffects Model wth oncludng Remarks βˆ j S( βˆ j Z p Q R ( , p.035 Q ( 8.830, p.150 Mxed-ffects Model wth Regress. oeff. onstant Year Full model results Regress. oeff. onstant Year Full model results βˆ j S( βˆ j Z p Q R ( , p.09 Q ( 3.747, p.065 Fxed-ffects Model βˆ j S( βˆ j Z p Q R ( , p.07 Q ( , p Usng meta-analyss to summarze the evdence about a gven research problem has mportant advantages n comparson wth narratve revews. Frstly, meta-analyses can be replcated, as all decsons and steps carred out n ther process are made explct. Secondly, by applyng statstcal methods ther conclusons are more relable and precse. Thrdly, ther emphass on the effect sze wll contrbute to ensurng that researchers pay more attenton to the effect magntude, resultng n a lesser nterest n statstcal sgnfcance tests. Fnally, meta-analyss also contrbutes towards promotng vdence-based Practce n Internatonal Journal of Psychologcal Research 161

12 Internatonal Journal of Psychologcal Research, 010. Vol. 3. No. 1. ISSN mpresa (prnted ISSN electrónca (electronc Psychology, a new methodologcal approach that ams to encourage professonals to base ther practce to the greatest extent possble on scentfc evdence obtaned from research. Sánchez-Meca, J., Marín-Martínez, F., (010. Meta-analyss n Psychologcal Research. Internatonal Journal of Psychologcal Research, 3 (1, Hedges, Hggns, & Rothsten, 005; RFR S Nevertheless meta-analyss has problems and lmtatons. On the one hand, the valdty and accuracy of the results n a meta-analyss depend on the qualty of the emprcal studes ntegrated. If the sngle studes offer based estmatons of the effects, then the meta-analytc results wll also be based. An assessment of the methodologcal qualty of the sngle studes s therefore one of the man requstes n any meta-analyss (cf. e.g., Valentne & ooper, 008. In addton, meta-analyss can suffer publcaton bas f t s only based on publshed studes. As a consequence, an analyss of publcaton bas s essental n any meta-analyss (Rothsten, Sutton, & Borensten, 005. Moreover, meta-analyss can suffer selecton bas, when the selecton crtera for ncludng sngle studes n the meta-analyss are affected by theoretcal or substantve preferences of the meta-analyst. A relablty analyss of the selecton process of the studes should be therefore accomplshed n order to avod bas n ths step of the meta-analyss. Fnally, meta-analyses can be affected by reportng bas when the sngle studes only reported statstcal data on the outcomes wth postve results for the hypothess tested. A detaled analyss of the desgn and the dependent varables ncluded n the sngle studes should be carred out to assess whether studes are selectvely reportng ther statstcal results. As meta-analyses can suffer defcences and bases n ther development and n ther reportng practces, they should be read crtcally. To ths end, several protocols and statements have been publshed that enable consumers of meta-analyses to assess ther methodologcal qualty. It s worth notng the recent publcaton of the PRISMA checklst ( Preferred Reported Items for Systematc revews and Meta-Analyses; Moher, Lberat, Tetzlaff et al., 009, a set of gudelnes to assess the methodologcal qualty n reportng practces of meta-analyses. Another endeavor along the same lnes s the publcaton of the AMSTAR protocol for crtcal apprasal of meta-analyses (Shea, Grmshaw, Wells et al., 007. Fnally, several software programs have been developed for carryng out statstcal analyses n metaanalyss. Davd B. Wlson has developed macros for dong meta-analyss n SPSS, SAS, and STATA. The macros can be freely obtaned from the web ste The ochrane ollaboraton has developed RevMan 5.0., another free program for carryng out meta-analyss that can be obtaned from the web ste of ths ollaboraton ( Fnally, there s a commercal program omprehensve Meta-analyss.0 (Borensten, Borensten, M. J., Hedges, L. V., Hggns, J. P. T., & Rothsten, H. (005. omprehensve Metaanalyss (Vers.. nglewood lffs, NJ: Bostat, Inc. Borensten, M. J., Hedges, L. V., Hggns, J. P. T., & Rothsten, H. R. (009. Introducton to metaanalyss. hchester, UK: Wley. Botella, J., & Gambara, H. (006. Dong and reportng a meta-analyss. Internatonal Journal of lncal and Health Psychology, 6, ohen, J. (1988. Statstcal power analyss for the behavoral scences ( nd ed.. Hllsdale, NJ: rlbaum. ook, T. D., ooper, H., ordray, D. S., Hartmann, H., Hedges, L. V., Lght, R. J., Lous, T. A., & Mosteller, F. (199. Meta-analyss for explanaton: A casebook. New York: Russell Sage Foundaton. ooper, H. (010. Research synthess and meta-analyss: A step-by-step approach (3 rd ed.. Thousand Oaks, A: Sage. ooper, H., Hedges, L. V., & Valentne, J.. (ds.(009. The handbook of research synthess and metaanalyss ( nd ed.. New York: Russell Sage Foundaton. DerSmonan, R., & Lard, N. (1986. Meta-analyss of clncal trals. ontrolled lncal Trals, 7, gger, M., Davey Smth, G., & Altman, D. G. (ds. (001. Systematc revews n health care: Metaanalyss n context ( nd ed.. London: BMJ Pub. Group. Feld, A. P. (003. The problems of usng fxed-effects models of meta-analyss on real-world data. Understandng Statstcs,, Hartung, J. (1999. An alternatve method for metaanalyss. Bometrcal Journal, 41, Harwell, M. (1997. An emprcal study of Hedges's homogenety test. Psychologcal Methods,, Hedges, L. V., & Olkn, I. (1985. Statstcal methods for meta-analyss. New York: Academc Press. Hedges, L. V., & Vevea, J. L. (1998. Fxed- and randomeffects models n meta-analyss. Psychologcal Methods, 3, Hggns, J. P. T., & Green, S. (ds.(008. ochrane handbook for systematc revews of nterventons. hchester, UK: Wley-Blackwell. 16 Internatonal Journal of Psychologcal Research

13 Internatonal Journal of Psychologcal Research, 010. Vol. 3. No. 1. ISSN mpresa (prnted ISSN electrónca (electronc Sánchez-Meca, J., Marín-Martínez, F., (010. Meta-analyss n Psychologcal Research. Internatonal Journal of Psychologcal Research, 3 (1, Hggns, J. P. T., & Thompson, S. G. (00. Quantfyng heterogenety n a meta-analyss. Statstcs n Medcne, 1, Hunter, J.., & Schmdt, F. L. (004. Methods of metaanalyss: orrectng error and bas n research synthess ( nd ed.. Sage. Konstantopoulos, S., & Hedges, L. V. (009. Fxed effects models. In H. ooper, L. V. Hedges, & J.. Valentne (ds., The handbook of research synthess and meta-analyss ( nd ed. (pp New York: Russell Sage Foundaton. Lpsey, M. W., & Wlson, D. B. (001. Practcal Metaanalyss. Thousand Oaks, A: Sage Lttell, J. H., orcoran, J., & Plla, V. (008. Systematc revews and meta-analyss. Oxford, UK: Oxford Unversty Press. Marín-Martínez, F., & Sánchez-Meca, J. (010. Weghtng by nverse varance or by sample sze n randomeffects meta-analyss. ducatonal and Psychologcal Measurement, 70, Moher, D., Lberatt, A., Tetzlaff, J., Altman D. G., and the PRISMA Group (009. Preferred Reportng Items for Systematc revews and Meta-Analyses: The PRISMA statement. PLOS Medcne, 6(7: e do: journal.pmed Pettcrew, M., & Roberts, H. (006. Systematc revews n the socal scences: A practcal gude. Malden, MA: Blackwell. Raudenbush, S. W. (009. Random effects models. In H. ooper, L. V. Hedges, & J.. Valentne (ds., The handbook of research synthess and metaanalyss ( nd ed. (pp New York: Russell Sage Foundaton. Rosa-Alcázar, A. I., Sánchez-Meca, J., Gómez-onesa, A., & Marín-Martínez, F. (008. Psychologcal treatment of obsessve-compulsve dsorder: A meta-analyss. lncal Psychology Revew, 8, Rosenthal, R. (1995. Wrtng meta-analytc revews. Psychologcal Bulletn, 118, Rothsten, H. R., Sutton, A. J., & Borensten, M. (ds. (005. Publcaton bas n meta-analyss: Preventon, assessment, and adjustments. hchester, UK: Wley. Sánchez-Meca, J., & Marín-Martínez, F. (1997. Homogenety tests n meta-analyss: A Monte arlo comparson of statstcal power and Type I error. Qualty and Quantty, 31, Sánchez-Meca, J., & Marín-Martínez, F. (1998. Weghtng by nverse-varance or by sample sze n metaanalyss: A smulaton study. ducatonal and Psychologcal Measurement, 58, Sánchez-Meca, J., & Marín-Martínez, F. (008. onfdence ntervals for the overall effect sze n random-effects meta-analyss. Psychologcal Methods, 13, Sánchez-Meca, J., & Marín-Martínez, F. (010. Metaanalyss. In P. Peterson,. Baker, & B. McGaw (ds., Internatonal encyclopeda of educaton, Vol. 7 (3 rd ed. (pp Oxford: lsever Sánchez-Meca, J., Marín-Martínez, F., & hacón-moscoso, S. (003. ffect-sze ndces for dchotomzed outcomes n meta-analyss. Psychologcal Methods, 8, Schmdt, F. L., Oh, I.-S., & Hayes, T. L. (009. Fxed versus random effects models n meta-analyss: Model propertes and an emprcal comparson of dfference n results. Brtsh Journal of Mathematcal and Statstcal Psychology, 6, Shea, B. J., Grmshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel,., Porter, A.., Tugwell, P., Moher, D., & Bouter, L. M. (007. Development of AMSTAR: A measurement tool to assess the methodologcal qualty of systematc revews. BM Medcal Research Methodology, 7. do: / Sdk, K., & Jonkman, J. N. (003. On constructng confdence ntervals for a standardzed mean dfference n meta-analyss. ommuncatons n Statstcs: Smulaton & omputaton, 3, Sdk, K, & Jonkman, J. N. (006. Robust varance estmaton for random effects meta-analyss. omputatonal Statstcs and Data Analyss, 50, Sutton, A. J., & Hggns, J. P. T. (008. Recent developments n meta-analyss. Statstcs n Medcne, 7, Thompson, S. G., & Sharp, S. J. (1999. xplanng heterogenety n meta-analyss: A comparson of methods. Statstcs n Medcne, 18, Valentne, J.., & ooper, H. (008. A systematc and transparent approach for assessng the methodologcal qualty of nterventon effectve research: The Study Desgn and Implementaton Assessment Devce (Study DIAD. Psychologcal Methods, 13, Vechtbauer, W. (005. Bas and effcency of metaanalytc varance estmators n the random-effects model. Journal of ducatonal and Behavoral Statstcs, 30, Internatonal Journal of Psychologcal Research 163