PCORI Methodology Standards: Academic Curriculum. 2016 Patient-Centered Outcomes Research Institute. All Rights Reserved.

PCORI Methodology Standards: Academic Curriculum 2016 Patient-Centered Outcomes Research Institute. All Rights Reserved.

Module 5: Step 3 Search the Literature Category 11: Systematic Reviews Prepared by Eric Bass, MD, MPH Tianjing Li, MD, PhD Presented by Eric Bass, MD, MPH

Step 3: Search the Literature Select highest-yield electronic databases: General (e.g., PUBMED, EMBASE, Cochrane database of reviews and trials) Subject-specific databases (e.g., CINAHL, PsychINFO, OTSeeker) Consider other sources: Citation databases, such as Web of Science and Scopus Registries, such as ClinicalTrials.gov and WHO portal Regulatory agencies, such as Food and Drug Administration (FDA) Manufacturers Grey literature (dissertations, conferences), such as OpenGrey Use other methods: Citations in reviews and eligible articles Table of contents of relevant journals Query experts 3

Is It Sufficient to Use PUBMED and EMBASE? Topic Database Articles available Bicycle-related brain injuries % of total Unique articles EMBASE 31 61 8 MEDLINE 33 65 3 PsychINFO 16 34 2 Web of Science 27 53 5 Source: Lawrence, D. W. (2008). What is lost when searching only one literature database for articles relevant to injury prevention and safety promotion? Injury Prevention: Journal of the International Society for Child and Adolescent Injury Prevention, 14(6), 401 404. http://doi.org/10.1136/ip.2008.019430 4

Build a Boolean PubMed Search Strategy Start with: (Population OR synonym) AND (intervention OR synonym) AND (comparison OR synonym) Broaden or limit search with Boolean terms (AND, OR, NOT) Consider limiting by study type Cochrane has a highly sensitive search strategy for randomized controlled trials 5

Building a Search Strategy Use controlled vocabulary (medical subject headings) and key words If you use only controlled vocabulary, you miss articles because indexing is imperfect If you use only key words, you miss articles that don t use the same terms Use Clinical Queries to search by study category Etiology, diagnosis, therapy, prognosis, prediction Search for systematic reviews Keep a record of search strings and results using Advanced Search Builder and My NCBI See online tutorial at http://www.nlm.nih.gov/bsd/disted/pubmed.html Adapt search to other databases Document when, where, how, and what! 6

Criteria for Assessing the Search Strategy Accurate translation of question into search concepts Correct choice of Boolean operators and line numbers Adequate translation of strategy for each database Inclusion of relevant subject headings Absence of spelling errors Source: Sampson, M., McGowan, J., Cogo, E., Grimshaw, J., Moher, D., & Lefebvre, C. (2009). An evidence-based practice guideline for the peer review of electronic search strategies. Journal of Clinical Epidemiology, 62(9), 944 952. http://doi.org/10.1016/j.jclinepi.2008.10.012. 7

How Important Is It to Include Unpublished Results? Analysis of 42 systematic reviews of nine new drugs, comparing results with and without unpublished FDA data: For one harm outcome, summary estimate of harm was greater with unpublished data included Efficacy estimate Lower Higher Identical Source: Hart, B., Lundh, A., & Bero, L. (2012). Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ (Clinical Research Ed.), 344, d7202. 8

Review Citations Titles (n=10,475) Abstracts (n=3,612) Full-text articles (n=449) Eligible articles (n=63) 9

Potential Causes of Bias and Error in Searching the Literature Bias Publication bias Redundant reporting Exclusion of important sources Incomplete development of search strategy Error Inadequate monitoring of search process Imperfect reliability of reviewers 10

What Can Be Done to Minimize Bias and Error When Reviewing Citations for Eligible Studies? Establish process for reviewing citations with quality control Use trained reviewers with documented instructions Use independent dual-reviewer process Use key articles to validate search Assess samples early Discuss discrepancies Watch for duplication 11

Relevant IOM Standards for Searching the Literature Standard 2.6: Develop a systematic review protocol Describe context and rationale for review from decision-making and research perspective Describe study screening and selection criteria (inclusion and exclusion) Describe which outcomes, time points, interventions, and comparisons will be addressed Describe search strategy for identifying relevant evidence Describe procedures for study selection and data extraction, and how to identify and resolve disagreement between researchers in study selection and data extraction Describe approach to critically appraising studies Describe method for evaluating body of evidence, including quantitative and qualitative synthesis Describe and justify planned analyses of differential treatment effects according to subgroups, how intervention is delivered, or how outcome measured Describe proposed timetable 12

Relevant IOM Standards for Searching the Literature Standard 3.1: Conduct a comprehensive systematic search for evidence Work with information specialist trained in systematic reviews to plan search strategy PCORI acknowledges that such expertise is not limited to librarians Design the search strategy to address each question Use an independent information specialist to peer-review the search strategy Search bibliographic databases and citation indexes Search literature cited by eligible studies Update search at intervals appropriate to pace of generation of new information Search subject-specific databases and regional bibliographic databases if other databases are unlikely to provide all relevant evidence 13

Relevant IOM Standards for Searching the Literature Standard 3.2: Take action to address potentially biased reporting of results Search grey literature databases, clinical trial registries, and other sources of unpublished information about studies Invite researchers to clarify information about study eligibility, study characteristics, and risk of bias Invite study sponsors and researchers to submit unpublished data, including unreported outcomes Hand-search selected journals and conference abstracts Conduct a web search Search for studies reported in languages other than English, if appropriate 14

Relevant IOM Standards for Searching the Literature Standard 3.3: Screen and select studies Include or exclude studies on the basis of protocol s prespecified criteria Use observational studies as well as randomized trials to assess harms of interventions Use two or more members of the team, working independently, to screen and select studies Train screeners using written documentation, and test and retest screeners to improve accuracy and consistency Use one of two strategies to select studies: Read all full-text articles identified in the search Read full text of articles identified after screening titles and abstracts Taking account of risk of bias, consider using observational studies to address gaps in evidence from randomized trials on the benefits of interventions 15

Relevant IOM Standards for Searching the Literature Standard 3.4: Document the search Provide a line-by-line description of the search strategy, including date of every search for each database, web browser, etc. Document disposition of each report identified, including reasons for exclusion 16