How to calculate effect sizes from published research: A simplified methodology

Size: px

Start display at page:

Download "How to calculate effect sizes from published research: A simplified methodology"

Karin Doyle
8 years ago
Views:

1 WORK-LEARNING RESEARCH How o alulae effe sizes from published researh: A simplified mehodology Will Thalheimer Samanha Cook A Publiaion Copyrigh 2002 by Will Thalheimer All righs are reserved wih one exepion. Individuals are permied o make opies of his doumen in is enirey for personal use. Published Augus 2002

by Will Thalheimer All righs are reserved wih one exepion.

2 How o alulae effe sizes from published researh ariles: A simplified mehodology Will Thalheimer Samanha Cook Harvard Universiy Overview This arile provides a simplified mehodology for alulaing Cohen s d effe sizes from published experimens ha use -ess and F-ess. Aompanying his arile is a Mirosof Exel Spreadshee o speed your alulaions. Boh he spreadshee and his arile are available as free downloads a /effe_sizes.hm. Why we use effe sizes Whereas saisial ess of signifiane ell us he likelihood ha experimenal resuls differ from hane expeaions, effe-size measuremens ell us he relaive magniude of he experimenal reamen. They ell us he size of he experimenal effe. Effe sizes are espeially imporan beause hey allow us o ompare he magniude of experimenal reamens from one experimen o anoher. Alhough peren improvemens an be used o ompare experimenal reamens o onrol reamens, suh alulaions are ofen diffiul o inerpre and are almos always impossible o use in fair omparisons aross experimenal paradigms. A simple mehodology Alhough exensive ariles have been wrien deailing mehods for alulaing effe sizes from published researh ariles (e.g., Rosnow & Rosenhal, 1996; Rosnow, Rosenhal, & Rubin, 2000), a leas some of us he firs auhor inluded require a simpler approah. This arile provides a mehod o alulae Cohen s d from boh -ess and some F-ess of signifiane. Aompanying his arile is a Mirosof Exel Spreadshee ha an be used o ompue Cohen s d from published daa. Cohen s d has wo advanages over oher effe-size measuremens. Firs, is burgeoning populariy is making i he sandard. Thus, is alulaion enables immediae omparison 2

Boh he spreadshee and his arile are available as free downloads a /effe_sizes.hm.

3 o inreasingly larger numbers of published sudies. Seond, Cohen s (1992) suggesion ha effe sizes of.20 are small,.50 are medium, and.80 are large enables us o ompare an experimen s effe-size resuls o known benhmarks. The simple mehodology offered below is no new bu is drawn from previously published ariles, mos noably Rosnow and Rosenhal (1996) and Rosnow, Rosenhal, and Rubin (2000). We have simplified he mehodology no by hanging he formulas and alulaions bu by disarding as muh as possible he jargon and ompuaional raionales ypially inluded in ariles wrien for researh audienes. This arile is an aemp o provide a praial mehodology o enable he alulaion of effe sizes. Wha is an effe size? In essene, an effe size is he differene beween wo means (e.g., reamen minus onrol) divided by he sandard deviaion of he wo ondiions. I is he division by he sandard deviaion ha enables us o ompare effe sizes aross experimens. Beause - ess and F-ess uilize differen measures of sandard deviaion, wo separae alulaions are required. You will find i useful o keep his disinion in mind as you read his doumen and uilize he aompanying spreadshee. Table of Conens Calulaing Cohen s d from -ess Page 4 Calulaing Cohen s d from -ess: When you don have sandard deviaions or sandard errors Page 5 Calulaing Cohen s d from -ess: When you have sandard errors insead of sandard deviaions Page 6 Calulaing Cohen s d from F-ess: Page 7 Calulaing Cohen s d from F-ess: When you don have MSE s Page 8 Referenes Page 9 How o ie his arile Page 9 Aknowledgemens Page 9 3

The simple mehodology offered below is no new bu is drawn from previously published ariles, mos noably Rosnow and Rosenhal (1996) and Rosnow, Rosenhal, and Rubin (2000).

4 Calulaing Cohen s d from -ess (1) d = x s x pooled d = Cohen s d effe size x = mean (average of reamen or omparison ondiions) s = sandard deviaion The arile should lis he means ( x ) of he reamen ondiion and he omparison ondiion. Use hose numbers in he formula and alulae he pooled sandard deviaion by using Formula 1a below. Afer you use Formula 1a, simply finish alulaing Formula 1 o ge Cohen s d. (1a) s pooled = ( n 2 1) s + ( n n + n 1) s 2 s = sandard deviaion The arile should lis he number of subjes (n) and he sandard deviaions (s) of he reamen ondiion and he omparison ondiion. Use hose numbers o make your alulaions. If he arile does no lis he sandard deviaions, use eiher Formula 2 or Formula 3 below if possible. 4

Afer you use Formula 1a, simply finish alulaing Formula 1 o ge Cohen s d.

5 Calulaing Cohen s d from -ess: When you don have sandard deviaions or sandard errors. When an experimen ha uses a -es does no lis sandard deviaions, you an alulae Cohen s d as follows using he saisi: n (2) n d = n n n 2 d = Cohen s d effe size = saisi The arile should lis he saisi, whih i will usually do, for example, wih he following noaion: (29) = 3.12, where 29 is he degrees of freedom and 3.12 is he saisi. The arile should also lis he number of subjes (n) wihin eah ondiion. Use hose numbers o make your alulaions. If he arile does no lis he number of subjes in eah ondiion bu does lis he oal number of subjes and if you an assume ha boh ondiions have roughly equal numbers of subjes you an esimae Cohen s d by using Formula 2a below. (2a) Warning: Some sudies using repeaed-measure designs (where eah subje is measured several imes wihin he same ondiion) inorrely use experimenal rials, insead of subjes, as he unis of analysis. The formulas on his page anno be used for hese sudies beause he -saisi is no relevan o he number of subjes (n) in he sudy. These sudies are ofen easy o spo beause hey have ourageously high degrees of freedom. d 2 n 2 5

whih i will usually do, for example, wih he following noaion: (29) = 3.12, where 29 is he degrees of freedom and 3.12 is he saisi. The arile should also lis he number of subjes (n) wihin eah ondiion.

6 Calulaing Cohen s d from -ess: When you have sandard errors insead of sandard deviaions. When an experimen ha uses a -es does no lis sandard deviaions bu does lis sandard errors (SE), you an alulae he sandard deviaions as follows and hen use he resuling numbers in Formula 1a: (3) s = SE n s = sandard deviaion SE = sandard error This formula assumes ha he arile liss he sandard error (SE) and number of subjes (n) wihin eah ondiion. Use hose numbers o make your alulaions. 6

sandard deviaions as follows and hen use he resuling numbers in Formula 1a: (3) s = SE n s = sandard deviaion SE =

7 Calulaing Cohen s d from F-ess (4) d = x x n + n 2 MSE n + n d = Cohen s d effe size x = mean (average of reamen or omparison ondiion) MSE = mean squared error If sandard deviaions are available, use Formulas 1 and 1a above beause MSE s will no produe a preise Cohen s d when he F- es is a omparison among more han wo ondiions. Oherwise, oninue. The arile should lis he means ( x ) of he reamen ondiion and he omparison ondiion, and he mean squared error (MSE). Use hose numbers in he formula o ge Cohen s d. Be areful o sele he orre MSE if many are lised. Noe ha only when he F-es numeraor degrees of freedom are equal o 1 when he F- es ompares one ondiion o one oher ondiion will he MSE produe an exa Cohen s d effe size. In his ase, he F-es is equivalen o a -es. Seleing oher MSE s may no produe valid resuls. 7

The arile should lis he means ( x ) of he reamen ondiion and he omparison ondiion, and he mean squared error (MSE). Use hose numbers in he formula o ge Cohen s d.

8 Calulaing Cohen s d from F-ess: When you don have MSE s. When an experimen ha uses an F-es does no lis he MSE, you an alulae Cohen s d as follows using he F saisi. This alulaion should only be used when he F-es ompares one ondiion o one oher ondiion. n (5) n d = F n n n 2 d = Cohen s d effe size F = F saisi This formula an ONLY be used when he F-es ompares wo ondiions (when he firs degrees of freedom is equal o one). The arile should lis he F saisi, whih i will usually do, for example, wih he following noaion: F (1,39) = 3.12, where 1 is he degrees of freedom based on he number of ondiions, and 39 is he degrees of freedom based on he number of subjes. The arile should also lis he number of subjes (n) wihin eah ondiion. 8

n (5) n d = F n n n 2 d = Cohen s d effe size F = F saisi This formula an ONLY be used when he F-es ompares wo ondiions (when he firs degrees of freedom is equal o one).

9 Referenes Cohen, J. (1992). A power primer. Psyhologial Bullein, 112, Rosnow, R. L., & Rosenhal, R. (1996). Compuing onrass, effe sizes, and ounernulls on oher people s published daa: General proedures for researh onsumers. Psyhologial Mehods, 1, Rosnow, R. L., Rosenhal, R., & Rubin, D. B. (2000). Conrass and orrelaions in effe-size esimaion. Psyhologial Siene, 11, How o ie his arile Thalheimer, W., & Cook, S. (2002, Augus). How o alulae effe sizes from published researh ariles: A simplified mehodology. Rerieved November 31, 2002 from hp://work-learning.om/effe_sizes.hm. (NOTE: You should replae he fiional November 31 dae wih he dae on whih he arile was downloaded.) Aknowledgemens We would like o hank Allison Sieber for opyediing his doumen and Don Rubin for supporing he seond auhor s involvemen in his effor. 9

Conrass and orrelaions in effe-size esimaion. Psyhologial Siene, 11, 446-453. How o ie his arile Thalheimer, W., & Cook, S. (2002, Augus).

Duration and Convexity ( ) 20 = Bond B has a maturity of 5 years and also has a required rate of return of 10%. Its price is $613.

Duration and Convexity ( ) 20 = Bond B has a maturity of 5 years and also has a required rate of return of 10%. Its price is $613. Graduae School of Business Adminisraion Universiy of Virginia UVA-F-38 Duraion and Convexiy he price of a bond is a funcion of he promised paymens and he marke required rae of reurn. Since he promised