Methods for Meta-analysis in Medical Research Alex J. Sutton University of Leicester, UK Keith R. Abrams University of Leicester, UK David R. Jones University of Leicester, UK Trevor A. Sheldon University of York, UK Fujian Song University of York, UK JOHN WILEY & SONS, LTD Chichester New York Weinheim Brisbane Singapore Toronto
Contents Preface Acknowledgements Part A: Meta-Analysis Methodology: The Basics 1. Introduction - Meta-analysis: Its Development and Uses. 1 Evidence-based health care.2 Evidence-based everything!.3 Pulling together the evidence - systematic reviews.4 Why meta-analysis?.5 Aim of this book.6 Concluding remarks References XV xvii 1 3 3 4 5 8 12 13 13 2. Deflning Outcome Measures used for Combining via Meta-analysis.. 2.1 Introduction 2.2 Non-comparative binary outcomes 2.2.1 Odds 2.2.2 Incidence rates 2.3 Comparative binary outcomes 2.3.1 The Odds ratio 2.3.2 Relative risk (or rate ratio/relative rate) 2.3.3 Risk differences between proportions (or the absolute risk reduction) 2.3.4 The number needed to treat 2.3.5 Comparisons of rates 2.3.6 Other scales of measurement used in summarizing binary data 2.3.7 Which scale to use?.. 17 17 18 18 19 20 20 23 25 27 28 28 28
vi CONTENTS 2.4 Continuous data 28 2.4.1 Outcomes defined on their original metric (mean difference) 29 2.4.2 Outcomes defined using standardized mean differences 31 2.4 Ordinal outcomes 33 2.5 Summary/Discussion 33 References 34 3. Assessing Between Study Heterogeneity 37 3.1 Introduction 37 3.2 Hypothesis tests for presence of heterogeneity 38 3.2.1 Standard x 2 test 38 3.2.2 Extensions/alternative tests 39 3.2.3 Example: Testing for heterogeneity in the cholesterol lowering trial dataset 40 3.3 Graphical informal tests/explorations of heterogeneity 41 3.3.1 Plot of normalized (z) scores 41 3.3.2 Forest plot 42 3.3.3 Radial plot (Galbraith diagram) 46 3.3.4 VAbbe plot. 47 3.4 Possible causes of heterogeneity 48 3.4.1 Specific factors that may cause heterogeneity in RCTs 49 3.5 Methods for investigating and dealing with sources of heterogeneity 50 3.5.1 Change scale of outcome variable 51 3.5.2 Include covariates in a regression model (meta-regression) 51 3.5.3 Exclude studies 52 3.5.4 Analyse groups of studies separately 52 3.5.5 Use of random effects models 52 3.5.6 Use of mixed-effect models 53 3.6 The validity of pooling studies with heterogeneous outcomes 53 3.7 Summary/Discussion 53 References 54 4. Fixed Effects Methods for Combining Study Estimates 57 4.1 Introduction 57 4.2 General fixed effect model - the inverse variance-weighted method 58 4.2.1 Example: Combining odds ratios using the inverse variance-weighted method 59 4.2.2 Example: Combining standardized mean differences using a continuous outcome scale 62
CONTENTS vii 4.3 Specific methods for combining odds ratios 63 4.3.1 Mantel-Haenszel method for combining odds ratios 64 4.3.2 Peto's method for combining odds ratios 66 4.3.3 Combining odds ratios via maximum-likelihood techniques 68 4.3.4 Exact methods of interval estimation 69 4.3.5 Discussion of the relative merits of each method 69 4.4 Summary/Discussion 70 References 71 5. Random Effects Models for Combining Study Estimates 73 5.1 Introduction 73 5.2 Algebraic derivation for random effects models by the weighted method 74 5.3 Maximum likelihood and restricted maximum likelihood estimate solutions 75 5.4 Comparison of estimation methods 76 5.5 Example: Combining the cholesterol lowering trials using a random effects model 76 5.6 Extensions to the random effects model 80 5.6.1 Including uncertainty induced by estimating the between study variance 80 5.6.2 Exact approach to random effects meta-analysis of binary data 81 5.6.3 Miscellaneous extensions to the random effects model 82 5.7 Comparison of random with fixed effect models 83 5.8 Summary/Discussion 84 References 84 6. Exploring Between Study Heterogeneity 87 6.1 Introduction 87 6.2 Subgroup analyses 88 6.2.1 Example: Stratification by study characteristics 89 6.2.2 Example: Stratification by patient characteristics 89 6.3 Regression models for meta-analysis 93 6.3.1 Meta-regression models (fixed-effects regression) 93 6.3.2 Meta-regression example: a meta-analysis of Bacillus Calmette-Guerin (BCG) vaccine for the prevention of tuberculosis (TB) 95 6.3.3 Mixed effect models (random-effects regression) 97 6.3.4 Mixed model example: A re-analysis of Bacillus Calmette-Guerin (BCG) vaccine for the prevention of tuberculosis (TB) trials 99
viii CONTENTS 6.3.5 Mixed modelling extensions 99 6.4 Summary/Discussion 104 References 105 7. Publication Bias 109 7.1 Introduction 109 7.2 Evidence of publication and related bias 110 7.2.1 Survey of authors 110 7.2.2 Published versus registered trials in a meta-analysis 110 7.2.3 Follow-up of cohorts of registered studies 111 7.2.4 Non-empirical evidence 111 7.2.5 Evidence of language bias 111 7.3 The seriousness and consequences of publication bias for meta-analysis 112 7.4 Predictors of publication bias (factors effecting the probability a study will get published) 112 7.5 Identifying publication bias in a meta-analysis 112 7.5.1 The funnel plot 113 7.5.2 Rank correlation test 116 7.5.3 Linear regression test 117 7.5.4 Other methods to detect publication bias 119 7.5.5 Practical advice on methods for detecting publication bias' 119 7.6 Taking into account publication bias or adjusting the results of a meta-analysis in the presence of publication bias 119 7.6.1 Analysing only the largest studies 120 7.6.2 Rosenthal's 'file drawer' method 120 7.6.3 Models which estimate the number of unpublished studies, but do not adjust 122 7.6.4 Selection models using weighted distribution theory 123 7.6.5 The'Trim and Fill'method 123 7.6.6 The sensitivity approach of Copas 125 7.7 Broader perspective solutions to publication bias 126 7.7.1 Prospective registration of trials 126 7.7.2 Changes in publication process and journals 126 7.8 Including unpublished information 127 7.9 Summary/Discussion 127 References 128 8. Study Quality 133 8.1 Introduction 133 8.2 Methodological factors that may affect the quality of studies 134 8.2.1 Experimental studies 135
CONTENTS ix 8.2.2 Observational Studies 136 8.3 Incorporating study quality into a meta-analysis 137 8.3.1 Graphical plot 137 8.3.2 Cumulative methods 138 8.3.3 Regression model 138 8.3.4 Weighting 140 8.3.5 Excluding studies 142 8.3.6 Sensitivity analysis 143 8.4 Practical implementation 143 8.5 Summary/Discussion 144 References 144 9. Sensitivity Analysis 147 9.1 Introduction 147 9.2 Sensitivity of results to inclusion criteria 147 9.3 Sensitivity of results to meta-analytic methods 150 9.3.1 Assessing the impact of choice of study weighting 150 9.4 Summary/Discussion 151 References 151 10. Reporting the Results of a Meta-analysis 153 10.1 Introduction 153 10.2 Overview and structure of a report 154 10.3 Graphical displays used for reporting the findings of a meta-analysis 155 10.3.1 Forest plots 155 10.3.2 Radial plots 157 10.3.3 Funnel plots 157 10.3.4 Displaying the distribution of effect size estimates 158 10.3.5 Graphs investigating length of follow-up 158 10.4 Summary/Discussion 158 References 158 Part B: Advanced and Specialized Meta-analysis Topics 161 11. Bayesian Methods in Meta-analysis 163 11.1 Introduction 163 11.2 Bayesian methods in health research 163 11.2.1 General introduction 163 11.2.2 General advantages/disadvantages of Bayesian methods 166 11.2.3 Example: Bayesian analysis of a single trial using a normal conjugate model 167
x CONTENTS 11.3 Bayesian meta-analysis of normally distributed data 169 11.3.1 Example: Combining trials with continuous outcome measures using Bayesian methods 171 11.4 Bayesian meta-analysis of binary data 171 11.4.1 Example: Combining binary outcome measures using Bayesian methods 173 11.5 Empirical Bayes methods in meta-analysis 175 11.6 Advantages/disadvantages of Bayesian methods in meta-analysis 176 11.6.1 Advantages 176 11.6.2 Disadvantages 178 11.7 Extensions and specific areas of application 179 11.7.1 Incorporating study quality 179 11.7.2 Inclusion of covariates 180 11.7.3 Model selection 180 11.7.4 Hierarchical models 181 11.7.5 Sensitivity analysis 181 11.7.6 Comprehensive modelling 182 11.7.7 Other developments 183 11.8 Summary/Discussion 183 References 183 / 12. Meta-analysis of Individual Patient Data 191 12.1 Introduction 191 12.2 Procedural methodology 193 12.2.1 Data collection 193 12.2.2 Checking data 193 12.3 Issues involved in carrying out IPD meta-analyses 193 12.4 Comparing meta-analysis using IPD or summary data? 194 12.5 Combining individual patient and summary data 195 12.6 Summary/Discussion 196 References 196 13. Missing Data 199 13.1 Introduction 199 13.2 Reasons for missing data 200 13.3 Categories of missing data at the study level 200 13.4 Analytic methods for dealing with missing data 201 13.4.1 General missing data methods which can be applied in the meta-analysis context 201 13.4.2 Missing data methods specific to meta-analysis 202 13.4.3 Example: Dealing with missing standard deviations of estimates in a meta-analysis 202
CONTENTS xi 13.5 Bayesian methods for missing data 203 13.6 Summary/Discussion 203 References 204 14. Meta-analysis of Different Types of Data 205 14.1 Introduction 205 14.2 Combining ordinal data 205 14.3 Issues concerning scales of measurement when combining data 206 14.3.1 Transforming scales, maintaining same data type 207 14.3.2 Binary outcome data reported on different scales 207 14.3.3 Combining studies whose outcomes are reported using different data types 208 14.3.4 Combining summaries of binary outcomes with those of continuous outcomes 208 14.3.5 Non-parametric method of combining different data type effect measures 208 14.4 Meta-analysis of diagnostic test accuracy 209 14.4.1 Combining binary test results 209 14.4.2 Combining ordered categorical test results 215 14.4.3 Combining continuous test results. 215 14.5 Meta-analysis using surrogate markers 215 14.6 Combining a number of cross-over trials using the patient preference outcome 216 14.7 Vote-counting methods 217 14.8 Combining/7-values/significance levels 218 14.8.1 Minimum p method 219 14.8.2 Sum of z's method 220 14.8.3 Sum of logs method 220 14.8.4 Logit method 220 14.8.5 Other methods of combining significance levels 220 14.8.6 Appraisal of the methods 221 14.8.7 Example of combining/7-values 221 14.9 Novel applications of meta-analysis using non-standard methods or data 223 14.10 Summary/Discussion 223 References 223 15. Meta-analysis of Multiple and Correlated Outcome Measures 229 15.1 Introduction 229 15.2 Combining multiple/^-values 230 15.3 Method for reducing multiple outcomes to a single measure for each study 231
xii CONTENTS 15.4 Development of a multivariate model 231 15.4.1 Model of Raudenbush et al. 231 15.4.2 Model of Gleser and Olkin 232 15.4.3 Multiple outcome model for clinical trials 232 15.4.4 Random effect multiple outcome regression model 232 15.4.5 DuMouchel's extended model for multiple outcomes 233 15.4.6 Illustration of the use of multiple outcome models 233 15.5 Summary/Discussion 236 References 236 16. Meta-analysis of Epidemiological and Other Observational Studies... 239 16.1 Introduction 239 16.2 Extraction and derivation of study estimates 240 16.2.1 Scales of measurement used to report and combine observational studies 243 16.2.2 Data manipulation for data extraction 243 16.2.3 Methods for transforming and adjusting reported results 244 16.3 Analysis of summary data 246 16.3.1 Heterogeneity of observational studies 246 16.3.2 Fixed or random effects? 247 16.3.3 Weighting of observational studies 247 16.3.4 Methods for combining estimates of observational studies 247 16.3.5 Dealing with heterogeneity and combining the OC and breast cancer studies 248 16.4 Reporting the results of meta-analysis of observational studies 248 16.5 Use of sensitivity and influence analysis 248 16.6 Study quality considerations for observational studies 249 16.7 Other issues concerning meta-analysis of observational studies, 250 16.7.1 Analysing individual patient data from observational studies 250 16.7.2 Combining dose-response data 251 16.7.3 Meta-analysis of single case research 253 16.8 Unresolved issues concerning the meta-analysis of observational studies 254 16.9 Summary/Discussion 255 References 255
CONTENTS xiii 17. Generalized Synthesis of Evidence - Combining Different Sources of Evidence 259 17.1 Introduction 259 17.2 Incorporating single-arm studies: models for incorporating historical controls 259 17.2.1 Example 260 17.3 Combining matched and unmatched data 262 17.4 Approaches for combining studies containing multiple and/or different treatment arms 263 17.4.1 Approach of Gleser and Olkin 264 17.4.2 Models of Berkey et al. 264 17.4.3 Method of Higgins 264 17.4.4 Mixed model of DuMouchel 264 17.5 The confidence profile method 265 17.6 Cross-design synthesis 266 17.6.1 Beginnings 267 17.6.2 Bayesian hierarchical models 267 17.6.3 Grouped random effects models of Larose and Dey 271 17.6.4 Synthesizing studies with disparate designs to assess the exposure effects on the incidence of a rare adverse event 271 17.6.5 Combining the results of cancer studies in humans and other species 272 17.6.6 Combining biochemical and epidemiological evidence 272 17.6.7 Combining information from disparate toxicological studies using stratified ordinal regression 272 17.7 Summary/Discussion 273 References 273 18. Meta-analysis of Survival Data 277 18.1 Introduction 277 18.2 Inferring/estimating and combining (log) hazard ratios 278 18.3 Calculation of the'log-rank'odds ratio 278 18.4 Calculation of pooled survival rates 279 18.5 Method of Hunink and Wong 279 18.6 Iterative generalized least squares for meta-analysis of survival data at multiple times 280 18.6.1 Application of the model 281 18.7 Identifying prognostic factors using a log (relative risk) measure 282
xiv CONTENTS 18.8 Combining quality of life adjusted survival data 282 18.9 Meta-analysis of survival data using individual patient data 283 18.9.1 Pooling independent samples of survival data to form an estimator of the common survival function 283 18.9.2 Is obtaining and using survival data necessary? 283 18.10 Summary/Discussion 284 References 284 19. Cumulative Meta-analysis 287 19.1 Introduction 287 19.2 Example: Ordering by date of publication 288 19.3 Using study characteristics other than date of publication 290 19.3.1 Example: Ordering the cholesterol trials by baseline risk in the control group 290 19.4 Bayesian approaches 291 19.5 Issues regarding uses of cumulative meta-analysis 291 19.6 Summary/Discussion 292 References 292 / 20. Miscellaneous and Developing Areas of Application in Meta-analysis 295 20.1 Introduction 295 20.2 Alternatives to conventional meta-analysis 295 20.2.1 Estimating and extrapolating a response surface 295 20.2.2 Odd man out method 296 20.2.3 Best evidence synthesis 296 20.3 Developing areas 297 20.3.1 Prospective meta-analysis 297 20.3.2 Economic evaluation through meta-analysis ' 298 20.3.3 Combining meta-analysis and decision analysis > 299 20.3.4 Net benefit model synthesizing disparate sources of information 299 References 299 Appendix I: Software Used for the Examples in this Book 301 Subject index 309