Using GRADE to develop recommendations for immunization: recent advances Holger Schünemann, MD, PhD Chair and Professor, Department of Clinical Epidemiology & Biostatistics Professor of Medicine Michael Gent Chair in Healthcare Research McMaster University, Hamilton, Canada Robert Koch Institute, Berlin September 15, 2011
Disclosure Co chair GRADE Working Group Work with various guideline groups using GRADE American College of Physicians (ACP) Clinical Practice Guidelines Committee WHO: Advisory Committee for Health Research, Expert Advisory Panel on Clinical Practice Guidelines and Clinical Research Methods and Ethics & chair of various guideline panels; funding for guideline development No direct/personal for profit payments
Outline for today Introduction to GRADE Advances GRADE and public health Observational studies Criteria for upgrading and overall quality of evidence Transparent judgments for developing recommendations DECIDE project
Hierarchy of evidence based on quality STUDY DESIGN Randomized Controlled Trials Cohort Studies and Case Control Studies Case Reports and Case Series, Non systematic observations Expert Opinion BIAS
GRADE Working Group Grades of Recommendation Assessment, Development and Evaluation Aim: to develop a common, transparent and sensible system for grading the quality of evidence and the strength of recommendations (over 100 systems) International group of guideline developers, methodologists & clinicians from around the world (>200 contributors) since 2000 International group: ACCP, AHRQ, Australian NMRC, BMJ Clinical Evidence, CC, CDC, McMaster Uni., NICE, Oxford CEBM, SIGN, UpToDate, USPSTF, WHO CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005, AJRCCM 2006, Chest 2006, BMJ 2008
GRADE: recommendations & quality of (a body of)evidence Clear separation, but judgments required: 1) Recommendation: 2 grades conditional (aka weak) or strong (for or against an action)? Balance of benefits and downsides, values and preferences, resource use and quality of evidence 2) 4 categories of quality of evidence: (High), (Moderate), (Low), (Very low)? methodological quality of evidence likelihood of bias related to recommendation by outcome and across outcomes *www.gradeworking-group.org
GRADE Quality of Evidence In the context of making recommendations: The quality of evidence reflects the extent of our confidence that the estimates of an effect are adequate to support a particular decision or recommendation.
Likelihood of and confidence in an outcome
Determinants of quality RCTs observational studies 5 factors that can lower quality 1. limitations in detailed design and execution (risk of bias criteria) 2. Inconsistency (or heterogeneity) 3. Indirectness (PICO and applicability) 4. Imprecision (number of events and confidence intervals) 5. Publication bias 3 factors can increase quality 1. large magnitude of effect 2. plausible residual bias or confounding 3. dose-response gradient
GRADE Uptake World Health Organization Allergic Rhinitis in Asthma Guidelines (ARIA) American Thoracic Society American College of Physicians (ACP) Canadian Task Force for the Preventive Services European Respiratory Society European Society of Thoracic Surgeons British Medical Journal Infectious Disease Society of America UpToDate National Institutes of Health and Clinical Excellence (NICE) Scottish Intercollegiate Guideline Network (SIGN) Cochrane Collaboration Clinical Evidence Agency for Health Care Research and Quality (AHRQ) Partner of GIN Over 60 major organizations (over 260 members)
Formulate question P I C O Outcome Outcome Outcome Outcome Systematic review Select outcomes Rate importance Critical Critical Important Not important Outcomes across studies Create evidence profile with GRADEpro Summary of findings & estimate of effect for each outcome Rate quality of evidence for each outcome High Moderate Low Very low Grade down Grade up Randomization increases initial quality 1. Risk of bias 2. Inconsistency 3. Indirectness 4. Imprecision 5. Publication bias 1. Large effect 2. Dose response 3. Confounders Guideline development Formulate recommendations: For or against (direction) Strong or weak/conditional (strength) By considering: Quality of evidence Balance benefits/harms Values and preferences Revise if necessary by considering: Resource use (cost) Grade overall quality of evidence across outcomes based on lowest quality of critical outcomes We recommend using We suggest using We recommend against using We suggest against using
Healthcare problem Healthy people Herd immunity Long term perspective Few RCTs Lots of other things recommendation
Outline for today Introduction to GRADE Advances GRADE and public health Observational studies Criteria for upgrading and overall quality of evidence Transparent judgments for developing recommendations DECIDE project
Is immunization different from other public health or clinical fields? Herd immunity One outcome among others Incidence after introduction of vaccine? Numerator of effect larger than denominator (in terms of effect) = NNT < 1? Population intervention How does it differ from public health messages (e.g. health lifestyle)? Frequently, not always i.e. individualized decision making
Is immunization different from other public health or clinical fields? Healthy people Applies to other public health or prevention topics Possibly only an issue of baseline risk (burden of disease) No (or more difficult to do) RCTs Will not discuss because not appropriate
GRADE and observational studies Users of GRADE have expressed concern that GRADE places greater confidence on the results of randomized studies (RCTs) population or public health interventions and environmental health, health policy making and often surgery, where conducting RCTs is either challenging or unethical Consequently, the best quality of evidence for these questions will come from observational studies Some argue that it would be unreasonable to grade such best quality evidence, typical of most public health questions, as low
GRADE and observational studies Argument is not valid for several reasons: inability to obtain RCT data would not eliminate or minimize the bias associated with observational data quality of evidence from observational studies can lead to moderate and even high quality evidence within the GRADE framework why is this overlooked? need to be able to compare confidence in estimates of effect across healthcare questions GRADE applied to many PH questions (clarifies our lack of confidence in many interventions) applies to many clinical areas as well
GRADE and need for RCTs GRADE provides a framework to decide when RCTs would be desirable or not desirable Weak recommendations (due to low or very low quality evidence) Low or very low quality evidence across outcomes RCTs (often) not needed High quality of a body of evidence (either RCTs or observational studies with special strength) Strong recommendations
Outline for today Introduction to GRADE Advances GRADE and public health Observational studies Criteria for upgrading and overall quality of evidence Transparent judgments for developing recommendations DECIDE project
Discussions at 2011 GRADE meeting in Geneva Philippe Duclos presented two issues: 1. Population level dose response relations 2. Inappropriate claims of association
Factors for upgrading the quality of evidence 1. dose response relation Vaccine efficacy 50% of population immunized 20% lower risk 70% of population immunized 40% lower risk 90% of population immunized 80% lower risk In particular if across different settings and populations 2. all plausible residual confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed
Vaccines & autism MMR vaccine associated with autism? Given early report, likely overreporting Despite overreporting (opposing plausible bias and confounding): no association Increase quality of evidence for no association Confirmed by withdrawal of publication
Example & solution Association between an oral rotavirus vaccine and intussusception confirmed Concerns that other live oral vaccines could also be associated with intussusception Studies from different investigators and different countries Well done observational studies with good controlling for confounding & differential ascertainment biases and good precision for several studies (self controlled case series, population based, and use of large linked data bases) All hypothesis testing consistent to show no risk of association GRADE quality of evidence = low quality as no criteria to increase the quality apply in this setting?
Solution 1 = opposing residual bias and confounding Remove bias and confounding Possibly biased studies (e.g. case control studies) showing no association despite opposing bias Unbiased observational studies (e.g. Self control studies) showing no effect Increase confidence in no effect = increase QoE
Solution 2 = non critical outcome Given that no association exists and the outcome likely be very rare the outcome should not be rated as critical for decision making (perhaps important) Thus overall quality of evidence not affected even if low quality of evidence
Overall quality of a body of evidence The quality of evidence reflects the extent of our confidence that the estimates of an effect are adequate to support a particular decision or recommendation. Guideline developers must specify and determine importance of all relevant outcomes Overall quality of evidence is based on the lowest quality of all critical outcomes
Meta-analyses of several critical and important outcomes (one PICO) High Moderate Due to imprecision Low Due to imprecision and risk of bias High 0.5 0.75 1 1.25 1.5 Better Relative Risk Worse Moderate Low based on critical outcomes
Meta-analyses of several critical outcomes (one PICO) Threshold of acceptable harm for strong recommendation based on sure benefit in mortality and stroke High Moderate Due to imprecision High High 0.5 0.75 1 1.25 1.5 Better Relative Risk Worse High
Meta-analyses of several critical outcomes (one PICO) Threshold of acceptable harm for strong recommendation based on sure benefit in mortality and stroke High High Moderate due to risk of bias High 0.5 0.75 1 1.25 1.5 Better Relative Risk Worse Moderate
Outline for today Introduction to GRADE Advances GRADE and public health Observational studies Criteria for upgrading and overall quality of evidence Transparent judgments for developing recommendations DECIDE project
Outline for today Introduction to GRADE Advances GRADE and public health Observational studies Criteria for upgrading and overall quality of evidence Transparent judgments for developing recommendations DECIDE project
Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence Ten partners in seven countries: University of Dundee, UK Norwegian Knowledge Centre for the Health Services, Norway Research Institute of the Hospital of Santa Creu and Sant Pau, Spain Italian Cochrane Centre, Italy University of Amsterdam, the Netherlands World Health Organisation, International Freiburg University Hospital, Germany National Institute for Health and Clinical Excellence, UK Scottish Intercollegiate Guidelines Network, UK Finnish Medical Society Duodecim, Finland www.decide-collaboration.eu
GRADEpro redesign Handbook revision Guideline Development tool
Summary If adaptation necessary for immunization, GRADE should adapt, not immunization community GRADE applied to many PH interventions, but many other fields face absence of RCTs New examples for upgrading criteria DECIDE will bring many answers