Using GRADE to develop recommendations for immunization: recent advances

Transcription

1 Using GRADE to develop recommendations for immunization: recent advances Holger Schünemann, MD, PhD Chair and Professor, Department of Clinical Epidemiology & Biostatistics Professor of Medicine Michael Gent Chair in Healthcare Research McMaster University, Hamilton, Canada Robert Koch Institute, Berlin September 15, 2011

2

3 Disclosure Co chair GRADE Working Group Work with various guideline groups using GRADE American College of Physicians (ACP) Clinical Practice Guidelines Committee WHO: Advisory Committee for Health Research, Expert Advisory Panel on Clinical Practice Guidelines and Clinical Research Methods and Ethics & chair of various guideline panels; funding for guideline development No direct/personal for profit payments

4 Outline for today Introduction to GRADE Advances GRADE and public health Observational studies Criteria for upgrading and overall quality of evidence Transparent judgments for developing recommendations DECIDE project

5 Hierarchy of evidence based on quality STUDY DESIGN Randomized Controlled Trials Cohort Studies and Case Control Studies Case Reports and Case Series, Non systematic observations Expert Opinion BIAS

6 GRADE Working Group Grades of Recommendation Assessment, Development and Evaluation Aim: to develop a common, transparent and sensible system for grading the quality of evidence and the strength of recommendations (over 100 systems) International group of guideline developers, methodologists & clinicians from around the world (>200 contributors) since 2000 International group: ACCP, AHRQ, Australian NMRC, BMJ Clinical Evidence, CC, CDC, McMaster Uni., NICE, Oxford CEBM, SIGN, UpToDate, USPSTF, WHO CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005, AJRCCM 2006, Chest 2006, BMJ 2008

7 GRADE: recommendations & quality of (a body of)evidence Clear separation, but judgments required: 1) Recommendation: 2 grades conditional (aka weak) or strong (for or against an action)? Balance of benefits and downsides, values and preferences, resource use and quality of evidence 2) 4 categories of quality of evidence: (High), (Moderate), (Low), (Very low)? methodological quality of evidence likelihood of bias related to recommendation by outcome and across outcomes *

8 GRADE Quality of Evidence In the context of making recommendations: The quality of evidence reflects the extent of our confidence that the estimates of an effect are adequate to support a particular decision or recommendation.

9 Likelihood of and confidence in an outcome

10 Determinants of quality RCTs observational studies 5 factors that can lower quality 1. limitations in detailed design and execution (risk of bias criteria) 2. Inconsistency (or heterogeneity) 3. Indirectness (PICO and applicability) 4. Imprecision (number of events and confidence intervals) 5. Publication bias 3 factors can increase quality 1. large magnitude of effect 2. plausible residual bias or confounding 3. dose-response gradient

11 GRADE Uptake World Health Organization Allergic Rhinitis in Asthma Guidelines (ARIA) American Thoracic Society American College of Physicians (ACP) Canadian Task Force for the Preventive Services European Respiratory Society European Society of Thoracic Surgeons British Medical Journal Infectious Disease Society of America UpToDate National Institutes of Health and Clinical Excellence (NICE) Scottish Intercollegiate Guideline Network (SIGN) Cochrane Collaboration Clinical Evidence Agency for Health Care Research and Quality (AHRQ) Partner of GIN Over 60 major organizations (over 260 members)

12 Formulate question P I C O Outcome Outcome Outcome Outcome Systematic review Select outcomes Rate importance Critical Critical Important Not important Outcomes across studies Create evidence profile with GRADEpro Summary of findings & estimate of effect for each outcome Rate quality of evidence for each outcome High Moderate Low Very low Grade down Grade up Randomization increases initial quality 1. Risk of bias 2. Inconsistency 3. Indirectness 4. Imprecision 5. Publication bias 1. Large effect 2. Dose response 3. Confounders Guideline development Formulate recommendations: For or against (direction) Strong or weak/conditional (strength) By considering: Quality of evidence Balance benefits/harms Values and preferences Revise if necessary by considering: Resource use (cost) Grade overall quality of evidence across outcomes based on lowest quality of critical outcomes We recommend using We suggest using We recommend against using We suggest against using

13 Healthcare problem Healthy people Herd immunity Long term perspective Few RCTs Lots of other things recommendation

15 Is immunization different from other public health or clinical fields? Herd immunity One outcome among others Incidence after introduction of vaccine? Numerator of effect larger than denominator (in terms of effect) = NNT < 1? Population intervention How does it differ from public health messages (e.g. health lifestyle)? Frequently, not always i.e. individualized decision making

16 Is immunization different from other public health or clinical fields? Healthy people Applies to other public health or prevention topics Possibly only an issue of baseline risk (burden of disease) No (or more difficult to do) RCTs Will not discuss because not appropriate

17 GRADE and observational studies Users of GRADE have expressed concern that GRADE places greater confidence on the results of randomized studies (RCTs) population or public health interventions and environmental health, health policy making and often surgery, where conducting RCTs is either challenging or unethical Consequently, the best quality of evidence for these questions will come from observational studies Some argue that it would be unreasonable to grade such best quality evidence, typical of most public health questions, as low

18 GRADE and observational studies Argument is not valid for several reasons: inability to obtain RCT data would not eliminate or minimize the bias associated with observational data quality of evidence from observational studies can lead to moderate and even high quality evidence within the GRADE framework why is this overlooked? need to be able to compare confidence in estimates of effect across healthcare questions GRADE applied to many PH questions (clarifies our lack of confidence in many interventions) applies to many clinical areas as well

19 GRADE and need for RCTs GRADE provides a framework to decide when RCTs would be desirable or not desirable Weak recommendations (due to low or very low quality evidence) Low or very low quality evidence across outcomes RCTs (often) not needed High quality of a body of evidence (either RCTs or observational studies with special strength) Strong recommendations

21 Discussions at 2011 GRADE meeting in Geneva Philippe Duclos presented two issues: 1. Population level dose response relations 2. Inappropriate claims of association

22 Factors for upgrading the quality of evidence 1. dose response relation Vaccine efficacy 50% of population immunized 20% lower risk 70% of population immunized 40% lower risk 90% of population immunized 80% lower risk In particular if across different settings and populations 2. all plausible residual confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed

23 Vaccines & autism MMR vaccine associated with autism? Given early report, likely overreporting Despite overreporting (opposing plausible bias and confounding): no association Increase quality of evidence for no association Confirmed by withdrawal of publication

24 Example & solution Association between an oral rotavirus vaccine and intussusception confirmed Concerns that other live oral vaccines could also be associated with intussusception Studies from different investigators and different countries Well done observational studies with good controlling for confounding & differential ascertainment biases and good precision for several studies (self controlled case series, population based, and use of large linked data bases) All hypothesis testing consistent to show no risk of association GRADE quality of evidence = low quality as no criteria to increase the quality apply in this setting?

25 Solution 1 = opposing residual bias and confounding Remove bias and confounding Possibly biased studies (e.g. case control studies) showing no association despite opposing bias Unbiased observational studies (e.g. Self control studies) showing no effect Increase confidence in no effect = increase QoE

26 Solution 2 = non critical outcome Given that no association exists and the outcome likely be very rare the outcome should not be rated as critical for decision making (perhaps important) Thus overall quality of evidence not affected even if low quality of evidence

27 Overall quality of a body of evidence The quality of evidence reflects the extent of our confidence that the estimates of an effect are adequate to support a particular decision or recommendation. Guideline developers must specify and determine importance of all relevant outcomes Overall quality of evidence is based on the lowest quality of all critical outcomes

28 Meta-analyses of several critical and important outcomes (one PICO) High Moderate Due to imprecision Low Due to imprecision and risk of bias High Better Relative Risk Worse Moderate Low based on critical outcomes

29 Meta-analyses of several critical outcomes (one PICO) Threshold of acceptable harm for strong recommendation based on sure benefit in mortality and stroke High Moderate Due to imprecision High High Better Relative Risk Worse High

30 Meta-analyses of several critical outcomes (one PICO) Threshold of acceptable harm for strong recommendation based on sure benefit in mortality and stroke High High Moderate due to risk of bias High Better Relative Risk Worse Moderate

32

34 Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence Ten partners in seven countries: University of Dundee, UK Norwegian Knowledge Centre for the Health Services, Norway Research Institute of the Hospital of Santa Creu and Sant Pau, Spain Italian Cochrane Centre, Italy University of Amsterdam, the Netherlands World Health Organisation, International Freiburg University Hospital, Germany National Institute for Health and Clinical Excellence, UK Scottish Intercollegiate Guidelines Network, UK Finnish Medical Society Duodecim, Finland

35

36 GRADEpro redesign Handbook revision Guideline Development tool

37 Summary If adaptation necessary for immunization, GRADE should adapt, not immunization community GRADE applied to many PH interventions, but many other fields face absence of RCTs New examples for upgrading criteria DECIDE will bring many answers