Using Criteria to Appraise a Meta-analyses Paul Cronin B.A., M.B. B.Ch. B.A.O., M.S., M.R.C.P.I.,.F.R.C.R. Department of Radiology, Division of Cardiothoracic Radiology, University of Michigan, Ann Arbor, Michigan
AUR 59th Annual Meeting Westin Boston Waterfront, Boston, Massachusetts April 12-15, 2011 March 23-26, 2010
Critical Appraisal of Secondary Research Reviews of the scientific literature Selective summaries Highly subjective Objective summaries Quantitative information syntheses of the best research evidence The challenge is where along this spectrum the review lies
Principles of Critical Appraisal Secondary Literature Systematic review and Meta-analysis of diagnostic test accuracy (1) Finding the evidence Framing objectives of the review Identifying the relevant literature (2) Assessment of study quality and applicability to the clinical problem at hand
Principles of Critical Appraisal Secondary Literature Systematic review and Meta-analysis of diagnostic test accuracy (3) Summarizing the evidence Qualitatively and if appropriate, quantitatively (meta-analysis) (4) Clinical Interpretation Application of findings and development of recommendations
Questions Question 1: What is the PICO of the study and is it close enough to your PICO? Question 2: How well was the study done? Question 3: What do the results mean and could they have been due to chance?
Question 1 What is the PICO of the study and is it close enough to your PICO?
Finding the Evidence Defining the research question P Population / problem I Intervention / index test C Control / reference test O Outcome
Identifying the Evidence Literature search strategy 1. The target condition of interest (all subheadings) 2. The name of the test (all subheadings) and text 3. Combine the results of these searches with Boolean operator AND 4. Further search terms to restrict the search should only be introduced if this approach produces unmanageably large and nonspecific results
Question 2 How well was the study done?
Biases Studies that are chosen for inclusion Selection bias Published in the first place Publication bias Level of importance attributed to the study results by the secondary researcher Results are summarized
QFAS Steps EBP Secondary research Critical appraisal issue Q Formulate a question (PICO) Formulate a question (PICO) Does the research ask a clearly focused question (PICO) and use it to direct the search? F Find the best evidence Find the best evidence Did the search find all the best evidence? A Appraise the included studies Appraise the included studies Have the studies been critically appraised? S Synthesis the results (summary tables and plots) Have the results been synthesized with appropriate summary tables and plots?
Question Does the research ask a clearly focused question (PICO) and use it to direct the search? Excluded studies should be recorded with reasons for the exclusions Helps to eliminate selection bias
Find Search strategy Did the search find all the best evidence? Use PubMed for MEDLlNE, OLDMEDLINE, EMBASE, OVID, CANCERLIT, and the Cochrane Library databases, etc
Find Search strategy Did the search find all the best evidence? Searching of relevant journals Conference proceedings Reference lists of articles in the searches Non-English as well as English papers Overcoming selection bias
Find Overcoming publication bias Much primary research missed because of publication bias Reluctant to submit/publish null or negative results Good-quality analysis takes account of unpublished studies
Find Overcoming publication bias Discussed in the methods section Check Clinical trials registries http://www.controlled-trials.com Contacting experts Conference proceedings Internet Other sources of unpublished literature
Appraise Have the studies been critically appraised (using appropriate quality criteria)? Searching has been well done Validity of the results and conclusions depends on the quality of the individual studies included Good-quality analysis should include a critical appraisal for each of the studies
Appraise Have the studies been critically appraised (using appropriate quality criteria)? This ideally should be appraised by two assessors working independently Points of disagreement discussed and a consensus reached Critical assessment can be subjective - avoided a single reviewer
Quality Appraisal of Evidence 1. Methods and criteria used for the selection of the study population 2. Selection of the reference standard 3. Execution of index test strategy, comparator and reference standard 4. Interpretation of index test, comparator and reference standard 5. Analysis of the results
Quality Appraisal of Evidence QUADAS QUality Assessment Diagnostic Accuracy Studies
The QUADAS tool Yes No Unclear 1. Was the spectrum of patients representative of the patients who will receive the test in practice? ( ) ( ) ( ) 2. Were selection criteria clearly described? ( ) ( ) ( ) 3. Is the reference standard likely to correctly classify the target condition? ( ) ( ) ( ) 4. 5. Is the time period between reference standard and index test short enough to be reasonably sure that the target condition did not change between the two tests? Did the whole sample or a random selection of the sample, receive verification using a reference standard of diagnosis? ( ) ( ) ( ) ( ) ( ) ( ) 6. Did patients receive the same reference standard regardless of the index test result? ( ) ( ) ( ) 7. Was the reference standard independent of the index test (i.e. the index test did not form part of the reference standard)? ( ) ( ) ( ) 8. Was the execution of the index test described in sufficient detail to permit replication of the test? ( ) ( ) ( ) 9. Was the execution of the reference standard described in sufficient detail to permit its replication? ( ) ( ) ( ) 10. Were the index test results interpreted without knowledge of the results of the reference standard? ( ) ( ) ( ) 11. Were the reference standard results interpreted without knowledge of the results of the index test? ( ) ( ) ( ) 12. Were the same clinical data available when test results were interpreted as would be available when the test is used in practice? ( ) ( ) ( ) 13. Were uninterpretable/ intermediate test results reported? ( ) ( ) ( ) 14. Were withdrawals from the study explained? ( ) ( ) ( )
Synthesis Have the results been synthesized with appropriate summary tables and plots? Summary of the included studies (table) Show the types of studies Tests Number of subjects Results (95% confidence intervals) Graphic presentations forest plots
Synthesis Have the results been synthesized with appropriate summary tables and plots? Heterogeneity should be assessed and discussed
Question 3 What do the results mean?
Results Weighted values to each individual study according to their size Expressed in a standard way Sensitivity, specificity, PPV, NPV, AUROC curve RR, OR, etc Traditionally displayed in a figure called a forest plot
Forest Plot Individual studies are represented by a black square and a horizontal line Corresponds to point estimate and 95% CI The size (area) of the black square reflects the weight of the study Weight is proportional to the inverse of the study variance
Forest Plot The result is not significant at conventional levels (P > 0.05) The diamond at the bottom represents the combined or pooled result of all studies with its 95% CI
Statistical Methods for Pooling Data SROC Hierarchical summary ROC (HSROC) Bivariate randomeffects model
Summary ROC Curve
Exploring Heterogeneity Heterogeneity can be assessed using the 'eyeball' test Formally with statistical tests, such as I 2 and the Cochran chi-square (Q) test The amount of heterogeneity is calculated as the I 2 value 0 if none near 1 if a lot
Exploring Heterogeneity The statistical significance is assessed using the Cochran Q test If Cochran Q is statistically significant - there is definite heterogeneity If Cochran Q is not statistically significant but the ratio of Cochran Q and the degrees of freedom (Q/df) is greater than 1 - there is possible heterogeneity
Exploring Heterogeneity The statistical significance is assessed using the Cochran Q test If Cochran Q is not statistically significant and Q/df is less then 1 - then heterogeneity is very unlikely
Exploring Publication Bias Funnel plot +/- linear regression
Assessment of Publication Bias Funnel plot Non-parametric linear regression Parametric linear regression formally test for funnel plot asymmetry
Clinical Interpretation Fagan plot (Bayes Nomogram) of the Evidence
Clinical Interpretation Likelihood ratio scattergraph 1. Left Upper Quadrant, Likelihood Ratio Positive > 10, Likelihood Ratio Negative <0.1: Exclusion & Confirmation 2. Right Upper Quadrant, Likelihood Ratio Positive >10, Likelihood Ratio Negative >0.1: Confirmation Only 3. Left Lower Quadrant, Likelihood Ratio Positive <10, Likelihood Ratio Negative <0.1: Exclusion Only 4. Right Lower Quadrant, Likelihood Ratio Positive <10, Likelihood Ratio Negative >0.1: No Exclusion or Confirmation of the Evidence
Clinical Interpretation Predictive Values and Probability Modifying Plot of the Evidence
Probability Modifying Plot
Section/topic # Preferred Reporting Items for Systematic Reviews and Meta-Analyses Checklist item TITLE Title ABSTRACT Structured summary INTRODUCTION Rationale Objectives METHODS Protocol and registration Eligibility criteria Information sources Search Study selection Data collection process Data items Risk of bias in individual studies Summary measures Synthesis of results 1 Identify the report as a systematic review, meta-analysis, or both. 2 Provide a structured summary including, as applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number. 3 Describe the rationale for the review in the context of what is already known. 4 Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS). 5 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number. 6 Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale. 7 Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched. 8 Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated. 9 State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis). 10 Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators. 11 List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made. 12 Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis. 13 State the principal summary measures (e.g., risk ratio, difference in means). 14 Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g., I 2 ) for each meta-analysis. Reported on page #
Section/topic Risk of bias across studies Additional analyses RESULTS Study selection Study characteristics Risk of bias within studies Results of individual studies Synthesis of results Risk of bias across studies Additional analysis DISCUSSION Summary of evidence Limitations Conclusions FUNDING Funding # Checklist item 15 Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies). 16 Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified. 17 Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram. 18 For each study, present characteristics for which data were extracted (e.g., study size, PICOS, followup period) and provide the citations. 19 Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12). 20 For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot. 21 Present results of each meta-analysis done, including confidence intervals and measures of consistency. 22 Present results of any assessment of risk of bias across studies (see Item 15). 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]). 24 Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., healthcare providers, users, and policy makers). 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at review-level (e.g., incomplete retrieval of identified research, reporting bias). 26 Provide a general interpretation of the results in the context of other evidence, and implications for future research. 27 Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review. Reported on page #
AMSTAR 2
Summary Question The review is a systematic review based on a clear clinical question and the PICO is similar to ours
Summary Find The inclusion criteria and search methods are stated in the methods section Inclusion criteria were based on the clinical question A comprehensive search of the literature was conducted, including MEDLINE (PubMed) and EMBASE
Summary Find Contacted the authors of included papers directly and checked the reference lists for further relevant papers They also searched the Cochrane controlled trial register for unpublished clinical trials
Summary Appraise The studies were critically appraised using the QUODAS, DELPHI, Jadad quality scores Scores are shown in the paper
Summary Synthesise The paper includes a clear summary table of the included studies Forest plots Heterogeneity analysis for the outcomes Publication bias Clinical interpretation of the evidence
Using Criteria to Appraise a Meta-analyses Paul Cronin B.A., M.B. B.Ch. B.A.O., M.S., M.R.C.P.I.,.F.R.C.R. Department of Radiology, Division of Cardiothoracic Radiology, University of Michigan, Ann Arbor, Michigan