How to calculate effect sizes from published research: A simplified methodology


 Karin Doyle
 1 years ago
 Views:
Transcription
1 WORKLEARNING RESEARCH How o alulae effe sizes from published researh: A simplified mehodology Will Thalheimer Samanha Cook A Publiaion Copyrigh 2002 by Will Thalheimer All righs are reserved wih one exepion. Individuals are permied o make opies of his doumen in is enirey for personal use. Published Augus 2002
2 How o alulae effe sizes from published researh ariles: A simplified mehodology Will Thalheimer Samanha Cook Harvard Universiy Overview This arile provides a simplified mehodology for alulaing Cohen s d effe sizes from published experimens ha use ess and Fess. Aompanying his arile is a Mirosof Exel Spreadshee o speed your alulaions. Boh he spreadshee and his arile are available as free downloads a /effe_sizes.hm. Why we use effe sizes Whereas saisial ess of signifiane ell us he likelihood ha experimenal resuls differ from hane expeaions, effesize measuremens ell us he relaive magniude of he experimenal reamen. They ell us he size of he experimenal effe. Effe sizes are espeially imporan beause hey allow us o ompare he magniude of experimenal reamens from one experimen o anoher. Alhough peren improvemens an be used o ompare experimenal reamens o onrol reamens, suh alulaions are ofen diffiul o inerpre and are almos always impossible o use in fair omparisons aross experimenal paradigms. A simple mehodology Alhough exensive ariles have been wrien deailing mehods for alulaing effe sizes from published researh ariles (e.g., Rosnow & Rosenhal, 1996; Rosnow, Rosenhal, & Rubin, 2000), a leas some of us he firs auhor inluded require a simpler approah. This arile provides a mehod o alulae Cohen s d from boh ess and some Fess of signifiane. Aompanying his arile is a Mirosof Exel Spreadshee ha an be used o ompue Cohen s d from published daa. Cohen s d has wo advanages over oher effesize measuremens. Firs, is burgeoning populariy is making i he sandard. Thus, is alulaion enables immediae omparison 2
3 o inreasingly larger numbers of published sudies. Seond, Cohen s (1992) suggesion ha effe sizes of.20 are small,.50 are medium, and.80 are large enables us o ompare an experimen s effesize resuls o known benhmarks. The simple mehodology offered below is no new bu is drawn from previously published ariles, mos noably Rosnow and Rosenhal (1996) and Rosnow, Rosenhal, and Rubin (2000). We have simplified he mehodology no by hanging he formulas and alulaions bu by disarding as muh as possible he jargon and ompuaional raionales ypially inluded in ariles wrien for researh audienes. This arile is an aemp o provide a praial mehodology o enable he alulaion of effe sizes. Wha is an effe size? In essene, an effe size is he differene beween wo means (e.g., reamen minus onrol) divided by he sandard deviaion of he wo ondiions. I is he division by he sandard deviaion ha enables us o ompare effe sizes aross experimens. Beause  ess and Fess uilize differen measures of sandard deviaion, wo separae alulaions are required. You will find i useful o keep his disinion in mind as you read his doumen and uilize he aompanying spreadshee. Table of Conens Calulaing Cohen s d from ess Page 4 Calulaing Cohen s d from ess: When you don have sandard deviaions or sandard errors Page 5 Calulaing Cohen s d from ess: When you have sandard errors insead of sandard deviaions Page 6 Calulaing Cohen s d from Fess: Page 7 Calulaing Cohen s d from Fess: When you don have MSE s Page 8 Referenes Page 9 How o ie his arile Page 9 Aknowledgemens Page 9 3
4 Calulaing Cohen s d from ess (1) d = x s x pooled d = Cohen s d effe size x = mean (average of reamen or omparison ondiions) s = sandard deviaion The arile should lis he means ( x ) of he reamen ondiion and he omparison ondiion. Use hose numbers in he formula and alulae he pooled sandard deviaion by using Formula 1a below. Afer you use Formula 1a, simply finish alulaing Formula 1 o ge Cohen s d. (1a) s pooled = ( n 2 1) s + ( n n + n 1) s 2 s = sandard deviaion The arile should lis he number of subjes (n) and he sandard deviaions (s) of he reamen ondiion and he omparison ondiion. Use hose numbers o make your alulaions. If he arile does no lis he sandard deviaions, use eiher Formula 2 or Formula 3 below if possible. 4
5 Calulaing Cohen s d from ess: When you don have sandard deviaions or sandard errors. When an experimen ha uses a es does no lis sandard deviaions, you an alulae Cohen s d as follows using he saisi: n (2) n d = n n n 2 d = Cohen s d effe size = saisi The arile should lis he saisi, whih i will usually do, for example, wih he following noaion: (29) = 3.12, where 29 is he degrees of freedom and 3.12 is he saisi. The arile should also lis he number of subjes (n) wihin eah ondiion. Use hose numbers o make your alulaions. If he arile does no lis he number of subjes in eah ondiion bu does lis he oal number of subjes and if you an assume ha boh ondiions have roughly equal numbers of subjes you an esimae Cohen s d by using Formula 2a below. (2a) Warning: Some sudies using repeaedmeasure designs (where eah subje is measured several imes wihin he same ondiion) inorrely use experimenal rials, insead of subjes, as he unis of analysis. The formulas on his page anno be used for hese sudies beause he saisi is no relevan o he number of subjes (n) in he sudy. These sudies are ofen easy o spo beause hey have ourageously high degrees of freedom. d 2 n 2 5
6 Calulaing Cohen s d from ess: When you have sandard errors insead of sandard deviaions. When an experimen ha uses a es does no lis sandard deviaions bu does lis sandard errors (SE), you an alulae he sandard deviaions as follows and hen use he resuling numbers in Formula 1a: (3) s = SE n s = sandard deviaion SE = sandard error This formula assumes ha he arile liss he sandard error (SE) and number of subjes (n) wihin eah ondiion. Use hose numbers o make your alulaions. 6
7 Calulaing Cohen s d from Fess (4) d = x x n + n 2 MSE n + n d = Cohen s d effe size x = mean (average of reamen or omparison ondiion) MSE = mean squared error If sandard deviaions are available, use Formulas 1 and 1a above beause MSE s will no produe a preise Cohen s d when he F es is a omparison among more han wo ondiions. Oherwise, oninue. The arile should lis he means ( x ) of he reamen ondiion and he omparison ondiion, and he mean squared error (MSE). Use hose numbers in he formula o ge Cohen s d. Be areful o sele he orre MSE if many are lised. Noe ha only when he Fes numeraor degrees of freedom are equal o 1 when he F es ompares one ondiion o one oher ondiion will he MSE produe an exa Cohen s d effe size. In his ase, he Fes is equivalen o a es. Seleing oher MSE s may no produe valid resuls. 7
8 Calulaing Cohen s d from Fess: When you don have MSE s. When an experimen ha uses an Fes does no lis he MSE, you an alulae Cohen s d as follows using he F saisi. This alulaion should only be used when he Fes ompares one ondiion o one oher ondiion. n (5) n d = F n n n 2 d = Cohen s d effe size F = F saisi This formula an ONLY be used when he Fes ompares wo ondiions (when he firs degrees of freedom is equal o one). The arile should lis he F saisi, whih i will usually do, for example, wih he following noaion: F (1,39) = 3.12, where 1 is he degrees of freedom based on he number of ondiions, and 39 is he degrees of freedom based on he number of subjes. The arile should also lis he number of subjes (n) wihin eah ondiion. 8
9 Referenes Cohen, J. (1992). A power primer. Psyhologial Bullein, 112, Rosnow, R. L., & Rosenhal, R. (1996). Compuing onrass, effe sizes, and ounernulls on oher people s published daa: General proedures for researh onsumers. Psyhologial Mehods, 1, Rosnow, R. L., Rosenhal, R., & Rubin, D. B. (2000). Conrass and orrelaions in effesize esimaion. Psyhologial Siene, 11, How o ie his arile Thalheimer, W., & Cook, S. (2002, Augus). How o alulae effe sizes from published researh ariles: A simplified mehodology. Rerieved November 31, 2002 from hp://worklearning.om/effe_sizes.hm. (NOTE: You should replae he fiional November 31 dae wih he dae on whih he arile was downloaded.) Aknowledgemens We would like o hank Allison Sieber for opyediing his doumen and Don Rubin for supporing he seond auhor s involvemen in his effor. 9
CO2 Cost Pass Through and Windfall Profits in the Power Sector
CO Cos Pass Through and Windfall Profis in he Power Seor Jos Sijm, Karsen Neuhoff and Yihsu Chen May 006 CWPE 0639 and EPRG 067 These working papers presen preliminary researh findings, and you are advised
More informationMining the Most Interesting Rules
Appears in Pro. of he Fifh ACM SIGKDD In l Conf. on Knowledge Disovery and Daa Mining, 145154, 1999. Mining he Mos Ineresing Rules Robero J. Bayardo Jr. IBM Almaden Researh Cener hp://www.almaden.ibm.om/s/people/bayardo/
More informationOUTOFBAG ESTIMATION. Leo Breiman* Statistics Department University of California Berkeley, CA. 94708 leo@stat.berkeley.edu
1 OUTOFBAG ESTIMATION Leo Breiman* Saisics Deparmen Universiy of California Berkeley, CA. 94708 leo@sa.berkeley.edu Absrac In bagging, predicors are consruced using boosrap samples from he raining se
More informationCostSensitive Learning by CostProportionate Example Weighting
CosSensiive Learning by CosProporionae Example Weighing Bianca Zadrozny, John Langford, Naoki Abe Mahemaical Sciences Deparmen IBM T. J. Wason Research Cener Yorkown Heighs, NY 0598 Absrac We propose
More informationDoes Britain or the United States Have the Right Gasoline Tax?
Does Briain or he Unied Saes Have he Righ Gasoline Tax? Ian W.H. Parry and Kenneh A. Small March 2002 (rev. Sep. 2004) Discussion Paper 02 12 rev. Resources for he uure 1616 P Sree, NW Washingon, D.C.
More informationFollow the Leader If You Can, Hedge If You Must
Journal of Machine Learning Research 15 (2014) 12811316 Submied 1/13; Revised 1/14; Published 4/14 Follow he Leader If You Can, Hedge If You Mus Seven de Rooij seven.de.rooij@gmail.com VU Universiy and
More informationBIS Working Papers. Globalisation, passthrough. policy response to exchange rates. No 450. Monetary and Economic Department
BIS Working Papers No 450 Globalisaion, passhrough and he opimal policy response o exchange raes by Michael B Devereux and James Yeman Moneary and Economic Deparmen June 014 JEL classificaion: E58, F6
More informationExchange Rate PassThrough into Import Prices: A Macro or Micro Phenomenon? Abstract
Exchange Rae PassThrough ino Impor Prices: A Macro or Micro Phenomenon? Absrac Exchange rae regime opimaliy, as well as moneary policy effeciveness, depends on he ighness of he link beween exchange rae
More informationAre Under and Overreaction the Same Matter? A Price Inertia based Account
Are Under and Overreacion he Same Maer? A Price Ineria based Accoun Shengle Lin and Sephen Rasseni Economic Science Insiue, Chapman Universiy, Orange, CA 92866, USA Laes Version: Nov, 2008 Absrac. Theories
More informationThe concept of potential output plays a
Wha Do We Know (And No Know) Abou Poenial Oupu? Susano Basu and John G. Fernald Poenial oupu is an imporan concep in economics. Policymakers ofen use a onesecor neoclassical model o hink abou longrun
More informationKONSTANTĪNS BEŅKOVSKIS IS THERE A BANK LENDING CHANNEL OF MONETARY POLICY IN LATVIA? EVIDENCE FROM BANK LEVEL DATA
ISBN 9984 676 20 X KONSTANTĪNS BEŅKOVSKIS IS THERE A BANK LENDING CHANNEL OF MONETARY POLICY IN LATVIA? EVIDENCE FROM BANK LEVEL DATA 2008 WORKING PAPER Lavias Banka, 2008 This source is o be indicaed
More informationAsymmetry of the exchange rate passthrough: An exercise on the Polish data 1
Asymmery of he exchange rae passhrough: An exercise on he Polish daa Jan Przysupa Ewa Wróbel 3 Absrac We propose a complex invesigaion of he exchange rae passhrough in a small open economy in ransiion.
More informationEDUCATION POLICIES AND STRATEGIES
EDUCATION POLICIES AND STRATEGIES Naional Educaion Secor Developmen Plan: A resulbased planning handbook 13 Educaion Policies and Sraegies 13 Educaion Policies and Sraegies 13 Naional Educaion Secor Developmen
More informationBoard of Governors of the Federal Reserve System. International Finance Discussion Papers. Number 1003. July 2010
Board of Governors of he Federal Reserve Sysem Inernaional Finance Discussion Papers Number 3 July 2 Is There a Fiscal Free Lunch in a Liquidiy Trap? Chrisopher J. Erceg and Jesper Lindé NOTE: Inernaional
More informationA Simple Introduction to Dynamic Programming in Macroeconomic Models
Economics Deparmen Economics orking Papers The Universiy of Auckland Year A Simple Inroducion o Dynamic Programming in Macroeconomic Models Ian King Universiy of Auckland, ip.king@auckland.ac.nz This paper
More informationBIS Working Papers No 365. Was This Time Different?: Fiscal Policy in Commodity Republics. Monetary and Economic Department
BIS Working Papers No 365 Was This Time Differen?: Fiscal Policy in Commodiy Republics by Luis Felipe Céspedes and Andrés Velasco, Discussion Commens by Choongsoo Kim and Guillermo Calvo Moneary and Economic
More informationI M F S T A F F D I S C U S S I O N N O T E
I M F S T A F F D I S C U S S I O N N O T E February 29, 2012 SDN/12/01 Two Targes, Two Insrumens: Moneary and Exchange Rae Policies in Emerging Marke Economies Jonahan D. Osry, Aish R. Ghosh, and Marcos
More informationImproved Techniques for Grid Mapping with RaoBlackwellized Particle Filters
1 Improved Techniques for Grid Mapping wih RaoBlackwellized Paricle Filers Giorgio Grisei Cyrill Sachniss Wolfram Burgard Universiy of Freiburg, Dep. of Compuer Science, GeorgesKöhlerAllee 79, D79110
More informationDynamic Contracting: An Irrelevance Result
Dynamic Conracing: An Irrelevance Resul Péer Eső and Balázs Szenes Sepember 5, 2013 Absrac his paper considers a general, dynamic conracing problem wih adverse selecion and moral hazard, in which he agen
More informationResearch Division Federal Reserve Bank of St. Louis Working Paper Series
Research Division Federal Reserve Bank of S. Louis Working Paper Series Wihsanding Grea Recession like China Yi Wen and Jing Wu Working Paper 204007A hp://research.slouisfed.org/wp/204/204007.pdf March
More informationWhich Archimedean Copula is the right one?
Which Archimedean is he righ one? CPA Mario R. Melchiori Universidad Nacional del Lioral Sana Fe  Argenina Third Version Sepember 2003 Published in he YieldCurve.com ejournal (www.yieldcurve.com), Ocober
More informationThe Macroeconomics of MediumTerm Aid ScalingUp Scenarios
WP//6 The Macroeconomics of MediumTerm Aid ScalingUp Scenarios Andrew Berg, Jan Goschalk, Rafael Porillo, and LuisFelipe Zanna 2 Inernaional Moneary Fund WP//6 IMF Working Paper Research Deparmen The
More informationWhy Have Economic Reforms in Mexico Not Generated Growth?*
Federal Reserve Bank of Minneapolis Research Deparmen Saff Repor 453 November 2010 Why Have Economic Reforms in Mexico No Generaed Growh?* Timohy J. Kehoe Universiy of Minnesoa, Federal Reserve Bank of
More informationWhen Should Public Debt Be Reduced?
I M F S T A F F D I S C U S S I ON N O T E When Should Public Deb Be Reduced? Jonahan D. Osry, Aish R. Ghosh, and Raphael Espinoza June 2015 SDN/15/10 When Should Public Deb Be Reduced? Prepared by Jonahan
More informationThE Papers 07/02. Do sunk exporting costs differ among markets? Evidence from Spanish manufacturing firms.
ThE Papers 07/02 Deparameno de Teoría e Hisoria Económica Universidad de Granada Do sunk exporing coss differ among markes? Evidence from Spanish manufacuring firms. Blanes Crisóbal, José Vicene. Universidad
More informationOneWay Analysis of Variance (ANOVA) Example Problem
OneWay Analysis of Variance (ANOVA) Example Problem Introduction Analysis of Variance (ANOVA) is a hypothesistesting technique used to test the equality of two or more population (or treatment) means
More informationISSN 15183548. Working Paper Series
ISSN 583548 Working Paper Series Nonlinear Mechanisms of he Exchange Rae PassThrough: A Phillips curve model wih hreshold for Brazil Arnildo da Silva Correa and André Minella November, 006 ISSN 583548
More informationFIRST PASSAGE TIMES OF A JUMP DIFFUSION PROCESS
Adv. Appl. Prob. 35, 54 531 23 Prined in Norhern Ireland Applied Probabiliy Trus 23 FIRST PASSAGE TIMES OF A JUMP DIFFUSION PROCESS S. G. KOU, Columbia Universiy HUI WANG, Brown Universiy Absrac This paper
More informationThe Simple Analytics of Helicopter Money: Why It Works Always
Vol. 8, 201428 Augus 21, 2014 hp://dx.doi.org/10.5018/economicsejournal.ja.201428 The Simple Analyics of Helicoper Money: Why I Works Always Willem H. Buier Absrac The auhor proides a rigorous analysis
More information