Calculating EffectSizes


 Anastasia Foster
 1 years ago
 Views:
Transcription
1 Calculating EffectSizes David B. Wilson, PhD George Mason University August 2011 The Heart and Soul of Metaanalysis: The Effect Size Metaanalysis shifts focus from statistical significance to the direction and magnitude of effect Key to this is the effect size It is the dependent variable of metaanalysis Encodes research findings on a numerical scale Different types of effect sizes for different research situations Each type may have multiple methods of computation 1
2 Overview Main types of Effect Sizes Logic of the Standardized Mean Difference Methods of Computing the Standardized Mean Difference Logic of the Oddsratio and Riskratio Methods of Computing the Oddsratio Adjustments, such as for baseline differences Issues related to the variance estimate Most Common Effect Size Indexes Standardized mean difference (d or g) Group contrast (e.g., treatment versus control) Inherently continuous outcome construct Oddsratio and Riskratio (OR and RR) Group contrast (e.g., treatment versus control) Inherently dichotomous (binary) outcome construct Correlation coefficient (r ) Two inherently continuous constructs 2
3 Necessary Characteristics of Effect Sizes Numerical values produced must be comparable across studies Must be able to compute its standard error Must not be a direct function of sample size The Standardized Mean Difference Compares the means of two groups Standardized this difference 3
4 Visual Example of dtype Effect Size Small Sample Size Bias Adjustment d is slightly upwardly bias when based on small samples. This can be fixed with the adjustment below: It is common practice to apply this adjustment whenever using d. 4
5 Standard Error and Variance of d The standard error (and variance) of d is mostly a function of sample size: Methods of Computing d Lots of methods of computing d Goal is to reproduce what you would get with means, standard deviations, and sample sizes Some methods are straightforward 5
6 Computing d from a ttest Formula for d: Formula for t: Therefore: Computing d from a pvalue from a ttest An exact pvalue from a ttest can be convert into a tvalue, assuming you have the degreesoffreedom Then proceed as above 6
7 Computing d from dichotomous outcomes Some studies may report outcome dichotomously d can be estimated from these data Several methods Logit Cox Logit Probit Arcsine Logit method of computing d Logistic distribution similar to the normal distribution Logged oddsratios conform to the logistic distribution Standard deviation of the logistic distribution is Thus, we can rescale a logged OR into d Monte Carlo simulations suggest that this performs fairly well; some advantages to the Cox logit method (divide by 1.65) 7
8 More complicated estimates of d Can think of d as having two parts Numerator: mean difference Denominator: pooled standard deviation Trick is to get reasonable estimates of each Often have one but not the other Estimates of numerator of d Covariate adjusted means Difference in gain score means Unstandardized regression coefficient for treatment dummy 8
9 Estimates of denominator of d Gain score standard deviation Overall standard deviation for the outcome Standard error Oneway ANOVA Be care: don't just use any standard deviation laying around See Online Effect Size Calculator On CEBCP Website: On Campbell Website: 9
10 Logic of the OddsRatio and RiskRatio Oddsratio is the ratio of odds An odds is the probability of failure over the probability of success Riskratio is the ratio of risks A risk is simply the probability of failure An oddsratio or riskratio of 1 indicates no difference between the groups Oddsratio more difficult to interpret than riskratios Risk ratios are sensitive to base event rate; oddsratios are not Logic of the OddsRatio and RiskRatio Oddsratios and riskratios contrast two groups on dichotomous outcome I.e., a 2 by 2 frequency table where a, b, c, and d, are cell frequencies 10
11 Computing the OddsRatio and RiskRatio Oddsratio is computed as: Riskratio is computed as Computing the OddsRatio and RiskRatio The oddsratio can also be computed from the proportion successful in each group (p). Oddsratio is computed as: 11
12 Computing the Standard Error and Variance of the OddsRatio and Risk Ratio Standard error of the logged Oddsratio: Standard error of the logged Riskratio: Variance: Computing the OddsRatio from Other Data Only so many was you can report binary data Makes computing oddsratios and risk ratios relatively easy Generally have Full twobytwo table Percent of events (successes/failures) in each group Proportion of events (successes/failures) in each group Actual oddsratio or riskratio Convert dtype effect size Chisquare 12
13 Computing the OddsRatio from a χ 2 Oddsratio can be computed from a χ 2 if you have the overall event rate for the sample (total number of successes or failures) and the sample size in each group Computations reproduce the twobytwo table See online calculator or Lipsey/Wilson Practical Metaanalysis Appendix B Adjustments to Effect Size Hunter and Schmidt  Psychometric metaanalysis Measurement unreliability, invalidity Range restriction Artificial dichotomization Hunter and Schmidt adjustments generally not done in a Campbell or Cochrane review Pretest of outcome For quasiexperimental designs, may remove some bias Must still use posttest standard deviation in denominator Adds additional error (variance estimate should be bigger) Sensitivity analysis recommended 13
14 Issues Related to Variance Estimate Method of computing effect size may affect variance estimate For example d computed from dichotomous data numerator of d is adjusted for baseline or other covariates Study data may involve clusters Possible solution: rescale standard error from correct model provided in study (if available) where z is either a z or t test associate with the treatment effect from a complex statistical model. Questions? 14
15 P.O. Box 7004 St. Olavs plass 0130 Oslo, Norway 15
EffectSize Indices for Dichotomized Outcomes in MetaAnalysis
Psychological Methods Copyright 003 by the American Psychological Association, Inc. 003, Vol. 8, No. 4, 448 467 108989X/03/$1.00 DOI: 10.1037/108989X.8.4.448 EffectSize Indices for Dichotomized Outcomes
More information
Lee A. Becker. Effect Size (ES) 2000 Lee A. Becker
Lee A. Becker Effect Size (ES) 2000 Lee A. Becker I. Overview II. Effect Size Measures for Two Independent Groups 1. Standardized difference between two groups.
More informationResults from the 2014 AP Statistics Exam. Jessica Utts, University of California, Irvine Chief Reader, AP Statistics jutts@uci.edu
Results from the 2014 AP Statistics Exam Jessica Utts, University of California, Irvine Chief Reader, AP Statistics jutts@uci.edu The six freeresponse questions Question #1: Extracurricular activities
More informationSome Practical Guidance for the Implementation of Propensity Score Matching
DISCUSSION PAPER SERIES IZA DP No. 1588 Some Practical Guidance for the Implementation of Propensity Score Matching Marco Caliendo Sabine Kopeinig May 2005 Forschungsinstitut zur Zukunft der Arbeit Institute
More informationStandardized or simple effect size: What should be reported?
603 British Journal of Psychology (2009), 100, 603 617 q 2009 The British Psychological Society The British Psychological Society www.bpsjournals.co.uk Standardized or simple effect size: What should be
More informationGuide to Biostatistics
MedPage Tools Guide to Biostatistics Study Designs Here is a compilation of important epidemiologic and common biostatistical terms used in medical research. You can use it as a reference guide when reading
More informationBeyond Baseline and Followup: The Case for More T in Experiments * David McKenzie, World Bank. Abstract
Beyond Baseline and Followup: The Case for More T in Experiments * David McKenzie, World Bank Abstract The vast majority of randomized experiments in economics rely on a single baseline and single followup
More informationNCSS Statistical Software
Chapter 06 Introduction This procedure provides several reports for the comparison of two distributions, including confidence intervals for the difference in means, twosample ttests, the ztest, the
More informationPRINCIPAL COMPONENT ANALYSIS
1 Chapter 1 PRINCIPAL COMPONENT ANALYSIS Introduction: The Basics of Principal Component Analysis........................... 2 A Variable Reduction Procedure.......................................... 2
More informationDesign of Experiments (DOE)
MINITAB ASSISTANT WHITE PAPER This paper explains the research conducted by Minitab statisticians to develop the methods and data checks used in the Assistant in Minitab 17 Statistical Software. Design
More informationFixedEffect Versus RandomEffects Models
CHAPTER 13 FixedEffect Versus RandomEffects Models Introduction Definition of a summary effect Estimating the summary effect Extreme effect size in a large study or a small study Confidence interval
More informationOneWay Analysis of Variance (ANOVA) Example Problem
OneWay Analysis of Variance (ANOVA) Example Problem Introduction Analysis of Variance (ANOVA) is a hypothesistesting technique used to test the equality of two or more population (or treatment) means
More informationUnderstanding and Using ACS SingleYear and Multiyear Estimates
Appendix. Understanding and Using ACS SingleYear and Multiyear Estimates What Are SingleYear and Multiyear Estimates? Understanding Period Estimates The ACS produces period estimates of socioeconomic
More informationHow to Deal with The LanguageasFixedEffect Fallacy : Common Misconceptions and Alternative Solutions
Journal of Memory and Language 41, 416 46 (1999) Article ID jmla.1999.650, available online at http://www.idealibrary.com on How to Deal with The LanguageasFixedEffect Fallacy : Common Misconceptions
More informationREAD MORE READ BETTER? A METAANALYSIS OF THE LITERATURE ON THE RELATIONSHIP BETWEEN EXPOSURE TO READING AND READING ACHIEVEMENT M. Lewis S. J.
READ MORE READ BETTER? A METAANALYSIS OF THE LITERATURE ON THE RELATIONSHIP BETWEEN EXPOSURE TO READING AND READING ACHIEVEMENT M. Lewis S. J. Samuels University of Minnesota 1 Abstract While most educators
More informationIntroduction to. Hypothesis Testing CHAPTER LEARNING OBJECTIVES. 1 Identify the four steps of hypothesis testing.
Introduction to Hypothesis Testing CHAPTER 8 LEARNING OBJECTIVES After reading this chapter, you should be able to: 1 Identify the four steps of hypothesis testing. 2 Define null hypothesis, alternative
More informationThe InStat guide to choosing and interpreting statistical tests
Version 3.0 The InStat guide to choosing and interpreting statistical tests Harvey Motulsky 19902003, GraphPad Software, Inc. All rights reserved. Program design, manual and help screens: Programming:
More informationUnrealistic Optimism About Future Life Events
Journal of Personality and Social Psychology 1980, Vol. 39, No. 5, 806820 Unrealistic Optimism About Future Life Events Neil D. Weinstein Department of Human Ecology and Social Sciences Cook College,
More informationSawtooth Software. How Many Questions Should You Ask in ChoiceBased Conjoint Studies? RESEARCH PAPER SERIES
Sawtooth Software RESEARCH PAPER SERIES How Many Questions Should You Ask in ChoiceBased Conjoint Studies? Richard M. Johnson and Bryan K. Orme, Sawtooth Software, Inc. 1996 Copyright 19962002, Sawtooth
More informationCan political science literatures be believed? A study of publication bias in the APSR and the AJPS
Can political science literatures be believed? A study of publication bias in the APSR and the AJPS Alan Gerber Yale University Neil Malhotra Stanford University Abstract Despite great attention to the
More informationANOVA ANOVA. TwoWay ANOVA. OneWay ANOVA. When to use ANOVA ANOVA. Analysis of Variance. Chapter 16. A procedure for comparing more than two groups
ANOVA ANOVA Analysis of Variance Chapter 6 A procedure for comparing more than two groups independent variable: smoking status nonsmoking one pack a day > two packs a day dependent variable: number of
More informationEffect Sizes. Null Hypothesis Significance Testing (NHST) C8057 (Research Methods 2): Effect Sizes
Effect Sizes Null Hypothesis Significance Testing (NHST) When you read an empirical paper, the first question you should ask is how important is the effect obtained. When carrying out research we collect
More informationCalculating the number needed to be exposed with adjustment for confounding variables in epidemiological studies
Journal of Clinical Epidemiology 55 (2002) 525 530 Calculating the number needed to be exposed with adjustment for confounding variables in epidemiological studies Ralf Bender*, Maria Blettner Department
More informationNational Evaluation of Student Support Services: Examination of Student Outcomes After Six Years
U.S. DEPARTMENT OF EDUCATION National Evaluation of Student Support Services: Examination of Student Outcomes After Six Years Final Report National Evaluation of Student Support Services: Examination of
More information8 INTERPRETATION OF SURVEY RESULTS
8 INTERPRETATION OF SURVEY RESULTS 8.1 Introduction This chapter discusses the interpretation of survey results, primarily those of the final status survey. Interpreting a survey s results is most straightforward
More informationSome benefits of dichotomization in psychiatric and criminological research
100 Criminal Behaviour and Mental Health, 10, 100 122 2000 Whurr Publishers Ltd Some benefits of dichotomization in psychiatric and criminological research DAVID P. FARRINGTON 1 and ROLF LOEBER 2 1 Institute
More informationMisunderstandings between experimentalists and observationalists about causal inference
J. R. Statist. Soc. A (2008) 171, Part 2, pp. 481 502 Misunderstandings between experimentalists and observationalists about causal inference Kosuke Imai, Princeton University, USA Gary King Harvard University,
More informationNSSE MultiYear Data Analysis Guide
NSSE MultiYear Data Analysis Guide About This Guide Questions from NSSE users about the best approach to using results from multiple administrations are increasingly common. More than three quarters of
More informationAn Introduction to Regression Analysis
The Inaugural Coase Lecture An Introduction to Regression Analysis Alan O. Sykes * Regression analysis is a statistical tool for the investigation of relationships between variables. Usually, the investigator
More informationIntroduction to Linear Regression
14. Regression A. Introduction to Simple Linear Regression B. Partitioning Sums of Squares C. Standard Error of the Estimate D. Inferential Statistics for b and r E. Influential Observations F. Regression
More information