HTA. Quality Assessment Tools Project Report. Canadian Agency for Drugs and Technologies in Health

Transcription

1 Canadian Agency for Drugs and Technologies in Health Agence canadienne des médicaments et des technologies de la santé HTA Quality Assessment Tools Project Report July 2012 Supporting Informed Decisions

2 Until April 2006, the Canadian Agency for Drugs and Technologies in Health (CADTH) was known as the Canadian Coordinating Office for Health Technology Assessment (CCOHTA). Publications can be requested from: CADTH Carling Avenue Ottawa ON Canada K1S 5S8 Tel.: Fax: or downloaded from CADTH s website: Cite as: Bai A, Shukla VK, Bak G, Wells G. Quality Assessment Tools Project Report. Ottawa: Canadian Agency for Drugs and Technologies in Health; Production of this report is made possible by financial contributions from Health Canada and the governments of Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Northwest Territories, Nova Scotia, Nunavut, Ontario, Prince Edward Island, Saskatchewan, and Yukon. The Canadian Agency for Drugs and Technologies in Health takes sole responsibility for the final form and content of this report. The views expressed herein do not necessarily represent the views of Health Canada or any provincial or territorial government. Reproduction of this document for non-commercial purposes is permitted provided appropriate credit is given to CADTH. CADTH is funded by Canadian federal, provincial, and territorial governments. Legal Deposit 2012 National Library of Canada ISBN: (print) ISBN: (online) July 2012 PUBLICATIONS MAIL AGREEMENT NO RETURN UNDELIVERABLE CANADIAN ADDRESSES TO CANADIAN AGENCY FOR DRUGS AND TECHNOLOGIES IN HEALTH CARLING AVENUE OTTAWA ON K1S 5S8

3 Reviewers CADTH staff would like to thank the following people for their time, assistance, and expert input throughout the project, including guidance on the approach and methods, and constructive feedback on drafts of this report. Gordon H. Guyatt, BSc, MD, MSc, FRCPC Distinguished Professor (Clinical Epidemiology & Biostatistics, and Medicine) McMaster University Co-founder and co-chair of the GRADE working group. Holger Schünemann, M.D., Ph.D. (Epi), M.Sc., FRCP(C) Chair and Professor Department of Clinical Epidemiology and Biostatistics McMaster University Hamilton, Ontario, Canada Andy Oxman Research Director Global Health Unit Norwegian Knowledge Centre for the Health Services Norway Kari A.O. Tikkinen Visiting Research Fellow (clinical epidemiologist and urologist) McMaster University and University of Helsinki

4 Canadian Agency for Drugs and Technologies in Health Quality Assessment Tools Project Report Annie Bai, MSc, MD 1 Vijay K. Shukla, RPh, PhD 1 Greg Bak, MLIS, PhD 1 George Wells, PhD 2,3 July Canadian Agency for Drugs and Technologies in Health (CADTH), Ottawa, Ontario, Canada 2 Corresponding Author 3 University of Ottawa Heart Institute, Ottawa, Ontario

5 Acknowledgments The authors acknowledge the continuing effort and support of whole QAT working group. We appreciate the considerable support, commitment, and contributions from CADTH s internal staff at the time of this work: Hayley Fitzsimmons (information specialist) who validated the original search strategy then updated the literature search; Samantha Verbrugghe (research assistant) who helped manage literature selection results and all references involved in the project. In addition, we would like to extend our appreciation to external researchers: Sarah Milne who participated in two key steps of QAT project, that is, identifying and evaluating potential tools; Kathleen Duclos and Renee Lafleur who checked the data of the original QAT project and selected the literature for updating QAT project; and Kasey Parker who provided input in writing the discussion session of this report. We owe our thanks as well to external experts for their valuable input: Dr. David Atkins, Dr. Brian Haynes, Dr. David Moher, Dr. Cynthia Mulrow, Dr. Andy Oxman, Dr. Barnaby Reeves, Dr. Beverley Shea, and Dr. Paul Shekelle as well as Ms. Pam McLean-Veysey. We would also like to thank Denis Bélanger, Barb Shea and the late Barbara Wells for their support to this project.

6 Abbreviations AHRQ CADTH CCOHTA COMPUS EPCs EGSs OBSs QAIs QAT RCTs SRs Agency for Healthcare Research and Quality Canadian Agency for Drugs and Technologies in Health Canadian Coordinating Office for Health Technology Assessment Canadian Optimal Medication Prescribing and Utilization Service evidence-based practice centers evidence grading systems observational studies quality assessment instruments quality assessment tools randomized controlled trials systematic reviews

7 TABLE OF CONTENTS EXECUTIVE SUMMARY... iii 1 INTRODUCTION Rationale for the QAT Project Goals of this QAT Report Quality Assessment in Systematic Reviews of Scientific Evidence Assessing the quality of individual studies Grading the strength of a body of evidence Existing QAIs and EGSs AHRQ Evidence Report Objective of the QAT Project METHODS Overview of QAT Project Methods Details of QAT Project Methods RESULTS Summary of QAT Project Results Collection of existing QAIs and EGSs Collection of existing QAIs and EGSs from review articles Conducted initial expert consultation (First round expert consultation... Step 4) Searched and selected individual QAIs and EGSs (QAT 2: 2000 to August 2005) (Step 5) Identification of potential QAIs and EGSs for evaluation (Second round combined analysis Step 6) Evaluation of the potential QAIs and EGSs identified Evaluation of the potential QAIs for SRs Evaluation of the potential QAIs for RCTs Evaluation of the potential QAIs for OBSs Evaluation of the potential EGSs Consultation on the QAIs and EGSs selected Conducted second expert consultation (Step 8) Chose QAIs and EGSs for CADTH (Step 9) Conducted stakeholder consultation (Step 10) Updating QAIs and EGSs Updating QAT 1 (Step 11) Updating QAT 2 (Step 12) DISCUSSION Tools selected through QAT project AMSTAR for SRs SIGN 50 checklist for RCTs SIGN 50 checklist for OBSs GRADE 2004 for EGS Application of the evaluation tools selected...29 Quality Assessment Tools Project Report i

8 4.3 Methodological issues Literature search Study funding Strengths and limitations of the QAT project CONCLUSION REFERENCES APPENDIX A: Products of QAT Project... A-1 APPENDIX B: Search strategy... A-10 APPENDIX C: Selection criteria... A-22 APPENDIX D: Literature selection... A-26 APPENDIX E: Reference lists of QAIs and EGSs... A-32 APPENDIX F: Evaluation results... A-70 APPENDIX G: QAIs and EGSs selected... A-79 ii Quality Assessment Tools Project Report

9 EXECUTIVE SUMMARY Introduction In March 2004, the Canadian Optimal Medication Prescribing and Utilization Service (COMPUS) was launched by the Canadian Coordinating Office for Health Technology Assessment (CCOHTA) now the Canadian Agency for Drugs and Technologies in Health (CADTH) as a service to federal, provincial, and territorial jurisdictions and other stakeholders. COMPUS was a nationally coordinated program, funded by Health Canada. To meet the goals of the COMPUS program, relevant and rigorously derived evidence-based information was required for making recommendations on optimal drug prescribing and use. However, the quality of scientific evidence varies, depending on the study design, conduct, and analysis; as well existing quality assessment tools also vary. CADTH staff embarked on this Quality Assessment Tools (QAT) project to identify the most appropriate tools for evaluating and grading evidence. The term quality means different things in different contexts. For this review the quality terminology is considered in the context of risk of bias. CADTH used a systematic approach to identify the most appropriate quality assessment instruments (QAIs) for assessing the quality of systematic reviews (SRs), randomized controlled trials (RCTs) and observational studies (OBSs) (mainly cohort and case-control studies), as well as evidence grading systems (EGSs) for rating the strength of a body of evidence. The tools selected by CADTH should be of the highest scientific credibility, be user-friendly, and be supported by most experts and stakeholders. The QAT project was originally conducted from January 2005 to October 2005 and subsequently updated from September 2007 to December The work of this project has been used to support the CADTH optimal use projects at that time and has been referenced in publications and presented at conferences. To date, many requests for details about this project were received from outside organizations. Given the interest and the important role of quality assessment tools in technology assessment, the authors are sharing the approach they took to conduct this project, as well as their findings and conclusions in the current report. Objective The objective of this QAT project was to identify appropriate QAIs by various study designs and EGSs for CADTH optimal use work. The objective of this report is to document the work undertaken between 2005 and 2007 through the QAT project. Methods Accepting the findings from Agency for Healthcare Research and Quality (AHRQ) evidence report No. 47 1, published in 2002, and building upon this report, CADTH applied a systematic Quality Assessment Tools Project Report iii

10 process to select QAIs for assessing the quality of SRs, RCTs and OBSs (cohort and case-control studies), as well as to select an EGS, mainly through: updating the review of QAIs and EGSs; and identifying the most appropriate QAIs and EGSs, that is the ones that were most feasible and efficient for practice use. The whole process was divided into 10 steps for the original project, plus two steps for updating. The QAT working group classified these steps into five main sections as follows: Step 1: Assemble the QAT working group Collection of existing QAIs and EGSs Step 2: Search and select review articles (January 2000 to February 2005) Step 3: Identify existing QAIs and EGSs from review articles Step 4: Consult experts with above collection Step 5: Search and select individual QAIs and EGSs (January 2000 to August 2005) Identification of potential QAIs and EGSs Step 6: Identify potential QAIs and EGSs from existing ones for evaluation Evaluation of potential QAIs and EGSs Step 7: Evaluate potential QAIs and EGSs identified Consult on appropriate QAIs and EGSs selected Step 8: Consult experts with evaluation results Step 9: Choose appropriate QAIs and EGSs for COMPUS Step 10: Collect stakeholders input on CADTH s choices Update the original QAT project (2005 to September 2007) Step 11: Update the search and selection of review articles Step 12: Update the search and selection of individual QAIs and EGSs Building upon the original search strategy used in the AHRQ report for literature dating from 1995 to June 2000, and taking into account the recommendations of the AHRQ team, a very sensitive search for systematic review articles of QAIs and EGSs was designed and carried out. Published literature was identified by searching the following bibliographic databases: PubMed, MEDLINE, Embase, BIOSIS Previews, and The Cochrane Library. CADTH s methodological filter was applied to limit retrieval to systematic reviews. The search was also limited to documents published between January 2000 and February Later, a highly specific supplemental literature search was conducted for the articles applying or reporting individual QAIs or EGSs, published between January 2000 and August This supplemental search was run on PubMed, MEDLINE, Embase, BIOSIS Previews, and The Cochrane Library. While updating the report, the two original search strategies were modified and validated for searching the relevant literature for both reviews and articles containing individual tools between 2005 and iv Quality Assessment Tools Project Report

11 September All searches were without language restriction. See Appendix B for the detailed search strategies. Two reviewers independently conducted the literature selection with agreement on the final articles included and excluded. Review articles were included if they systematically collected and evaluated existing QAIs for SRs, RCTs or OBSs and EGSs or contained a comparison or evaluation of at least two existing QAIs or EGSs. QAIs and EGSs were collected by types of study from the included review articles and reference lists of these existing QAIs and EGSs were created after removing duplicates. Additional QAIs and EGSs were collected from experts input and supplemental literature search. Potential tools were identified among existing generic checklists or scales of QAIs for SRs, RCTs, and OBSs (cohort and case-control), and generic EGSs if they were recommended in the review articles or had not been assessed before. Potential QAIs and EGSs identified were evaluated by using the AHRQ evaluation grids, which consist of domain criteria and elements created within each domain based on generally accepted standard epidemiologic methods. Strengths and weaknesses of different instruments or systems were shown in grids. The highest scoring QAIs and EGSs from each grid represented the proposed selections. Furthermore, the rigour of the development process, inter-rater reliability, and instructions provided as well as the length of tools, ease of use, and time consumed were considered for the instruments or systems with the same highest evaluation scores within each type of study. Any disagreement was resolved by group discussion. The entire process for the project was repeated in carrying out the update in 2007 to identify QAIs and EGSs between 2005 and September Results Overall, out of 4,126 citations retrieved from the original literature searches (3,006 in the search for review articles and 1,120 in the supplemental search), plus 122 references selected from expert consultations, 267 QAIs (57 for SRs, 94 for RCTs, 99 for OBSs, and 17 for multiple designs) and 60 EGSs were identified. Among these existing tools, 75 generic checklists and scales of QAIs (20 for SRs, 32 for RCTs, and 23 for OBSs) and 23 generic EGSs were identified using the pre-specified selection criteria as potential included instruments and were assessed using the AHRQ evaluation grids. Seven QAIs (two for SRs, three for RCTs, and two for OBSs) and 10 EGSs fully address all evaluation domains and obtained the same highest scores within each type of study. Among them, four QAIs (two for SRs, one for RCTs, and one for OBSs) and six EGSs were selected for further expert consultation after removing three less efficient instruments and four non-guideline systems respectively. With the input from experts and stakeholders, plus the consideration of CADTH needs, four checklists of QAIs and one EGS were chosen: AMSTAR 2005 for SRs (unpublished) then AMSTAR SIGN for RCTs 3 SIGN for cohort studies 4 and case-control studies 5 GRADE 2004 EGS 6 Quality Assessment Tools Project Report v

12 Above results are summarized in the following table: Number of tools QAIs EGSs Total No. of QAIs SRs RCTs OBSs Multiple design Existing Review articles tools Expert input collected Supplemental search Check from QAT group Total No. of collection Potential tools evaluated Tools with highest scores Tools for expert consultations Appropriate tools selected * 0 1 EGS = evidence grading system; OBS = observation study; QAI = quality assessment instrument; QAT = quality assessment tool; RCT = randomized controlled trial; SR = systematic review. *There are two methodological checklists provided for cohort and case-control studies respectively. No new QAIs and EGSs were identified through reviewing 1,601 citations retrieved from the updating literature searches (825 in the search for review articles and 776 in the supplemental search). Conclusions In the QAT project, a total of 267 existing QAIs and 60 existing EGSs were collected according to the type of study. Through the analyses and evaluation conducted by the QAT working group, four QAIs (one for each type of study) and one EGS were selected among those existing tools as the most appropriate ones for CADTH to use to make evidence-based recommendations on optimal drug prescribing and use. The project involved four separate literature searches, two rounds of combined analyses, two expert consultations, and one stakeholder consultation. Those selected QAIs and EGS have been applied systematically and consistently in CADTH optimal use projects to make evidence evaluation more transparent, and thus helped reviewers and expert panels more effectively translate evidence into comprehensive, reliable, and practical recommendations. We are confident that the work and selections of the QAT project are an important piece of CADTH s evaluation methodology and will solidify our evaluation foundation. vi Quality Assessment Tools Project Report

13 1 INTRODUCTION This report documents the work undertaken between 2005 and 2007 by the quality assessment tools (QAT) project. The goal of this project is to identify the most appropriate quality assessment tools for evaluating and grading evidence. The term quality means different things in different contexts. For example, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group defines quality as confidence in estimate. For this review we used the Agency for Healthcare Research and Quality (AHRQ) perspective of quality as risk of bias or internal validity. 1.1 Rationale for the QAT Project In March 2004, the Canadian Optimal Medication Prescribing and Utilization Service (COMPUS) was launched by the Canadian Coordinating Office for Health Technology Assessment (CCOHTA) now the Canadian Agency for Drugs and Technologies in Health (CADTH) as a service to federal, provincial, and territorial jurisdictions and other stakeholders. COMPUS was a nationally coordinated program, funded by Health Canada. The goal of COMPUS was to optimize drug-related health outcomes and the cost-effective use of drugs by identifying and promoting optimal drug prescribing and use. To meet this goal, CADTH required relevant and rigorously derived evidence-based information; however, the quality of scientific evidence varies, depending on the study design, and how it was conducted and analyzed. Thus, CADTH staff embarked on this project to identify the most helpful quality assessment tools. Quality assessment instruments (QAIs) are normally applied to transparently evaluate information from various types of studies, while evidence grading systems (EGSs) are applied to systematically rate the strength of a body of evidence. Choosing appropriate tools among the various existing QAIs and EGSs is not only an academic exercise, but the choice must also meet the practical needs, such as the efficiency of tool use. CADTH s mission is to provide timely, relevant, and rigorously derived evidence-based information to decision-makers, and to support the decision-making processes. Building on this foundation, plus considering the study designs most commonly encountered in clinical research on drugs, CADTH used a systematic approach to identify the most appropriate QAIs for assessing the quality of systematic reviews (SRs), randomized controlled trials (RCTs) and observational studies (OBSs) (mainly cohort and case-control studies), as well as EGSs for rating the strength of a body of evidence. The tools selected by CADTH should have the highest scientific credibility, be user-friendly, and be supported by most experts and stakeholders. 1.2 Goals of this QAT Report The QAT project was originally conducted in 2005 and subsequently updated in A summary of the original QAT project was posted on the CADTH website (formerly known as CCOHTA) in December 2005 (Appendix A-1). Since that time, the work has been referenced in publications 7,8, presented at conferences 9-11 (Appendix A-2 and Appendix A-3), and used to Quality Assessment Tools Project Report 1

14 support the work of CADTH. To date, many requests for details about this project were received from outside organizations. Given the interest and the important role of quality assessment tools in technology assessment, the authors are sharing the approach they took to conduct this work, as well as their findings and conclusions. 1.3 Quality Assessment in Systematic Reviews of Scientific Evidence Systematic reviews (SRs) represent a rigorous approach for compiling scientific evidence to answer specific research questions, and are increasingly being used to support evidence-based health care decisions. Compared with narrative reviews, one of the strengths of SRs is that they provide a measure of quality for each study included in the review by assessing how well the study was designed, conducted, and analyzed. Researchers place more emphasis on the results from studies with higher quality. 1 The evidence showed that poor quality studies may bias pooled estimates in SRs and that synthesizing studies of varying quality together may bias the combined effect measures of interest. 12 To take account of differences in study quality and the impact of those differences on interpretations of the scientific evidence, a direct and explicit approach to assessing study quality and rating strength of evidence in reviews is needed Assessing the quality of individual studies Study quality usually refers to the internal validity of a study. It is defined in some research as the extent to which a study s design, conduct, and analysis has minimized selection, measurement, and confounding biases. 1 Different instruments used for assessing the quality of the same study can lead to different quality rankings. 13 The type(s) of study design(s) being considered by researchers play an important role in the conduct of SRs, and the features that are important to assess differ by study types (e.g., SRs, RCTs, and OBSs). Therefore, one size fits all QAIs may be less efficient in measuring the quality of different types of study designs. QAIs were usually designed as components, checklists, or scales. Checklists and scales were more commonly used for rating study quality Grading the strength of a body of evidence Grading the strength of a body of evidence, which incorporates judgments of study quality as one of several factors, is crucial for developing evidence-based clinical recommendations. Strength refers to the size of the estimated risk and its accompanying confidence intervals. 1 Frameworks for grading the strength of a body of evidence are much less uniform than QAIs. Confidence in a recommendation is affected by three well-established attributes quality, quantity, and consistency, which are defined as follows: Quality is concerned with the quality of all relevant studies for a given topic. Quantity encompasses several aspects, such as the number of studies that have evaluated the question, the overall sample size across all of the studies, and the magnitude of the treatment effect, which is along the lines of strength from causality assessment. Consistency is whether 2 Quality Assessment Tools Project Report

15 investigations with both similar and different study designs report similar findings and can be assessed only if numerous studies are done. 1 EGSs should incorporate all three of these attributes. 1.4 Existing QAIs and EGSs Overall, the existing instruments varied widely. 14 Many QAIs are available as checklists and scales for RCTs and observational studies; however, fewer instruments have been developed specifically for assessing systematic reviews. 1 Many instruments were modified from generic counterparts for topic-specific applications; and many were developed based on expert opinion rather than empirical research, and few amongst them used rigorous development techniques. 1 Approaches for grading the strength of a body of evidence are rapidly evolving. The earliest approach to grade the level of evidence based on study design alone was published by the Canadian Task Force on the Periodic Health Examination in Since then, a number of derivative systems have been proposed and mainly used for clinical practice guidelines. 1 Although the EGSs relying on a study design hierarchy are simple, and easy to understand and use, they are increasingly unacceptable because of the lack of consideration of quality, quantity, and consistency of primary studies. The framework of EGSs is much less uniform than QAIs, which complicates the selection of one or more EGSs to be applied. 1 At the time of the QAT project initiation, the number of comprehensive evaluations of QAIs and EGSs was limited. Moher et al identified 25 scales and nine checklists available for RCTs. 16 Deeks et al conducted a health technology assessment and identified a total of 194 QAIs available for non-randomized interventions. 17 An international expert group critically appraised six prominent EGSs and concluded that all of them had important shortcomings, and that a new system should be developed to address the major limitations. 19 A systematic review on evaluation tools, published by the AHRQ in 2002, identified and assessed 121 QAIs by different types of study designs, as well as 40 EGSs for grading the strength of a body of evidence AHRQ Evidence Report 1 AHRQ, a national agency in the United States, focuses its mission to improve the quality, effectiveness, and appropriateness of clinical care by facilitating the translation of evidencebased research findings into clinical practice through its Evidence-Based Practice Centers (EPCs). 1 Building on an earlier report 14 and contributions from collaborating experts from the public and private sectors, AHRQ systematically identified and examined tools to rate the strength of scientific evidence for use in making evidence-based health-care decisions. The evaluation results were presented in Evidence Report / Technology Assessment Number 47. Such tools included: quality scales, quality checklists, and study design characteristics (components) for rating the quality of individual articles Quality Assessment Tools Project Report 3

16 methodologies for grading the strength of a body of scientific evidence; that is, an accumulation of many individual articles that address a common scientific issue. Following is a brief summary of the AHRQ project: A MEDLINE search (1995 to June 2000) was conducted for relevant articles published in English on either rating the quality of individual studies or grading a body of scientific evidence. Information from existing bibliographies, members of a technical expert panel, EPCs, and review groups was also sought to supplement these sources. Four study quality grids were developed with methodological domains, accounting for differences in study designs systematic reviews and meta-analyses, randomized controlled trials, observational studies, and diagnostic studies to compare and characterize existing instruments for assessing the quality of individual studies. One evidence strength grid was developed to assess the systems for rating the strength of bodies of evidence. Overall, 121 existing tools (20 for systematic reviews, 49 for RCTs, 19 for observational studies, 18 for diagnostic test studies, and 40 for grading a body of evidence) were identified, compared, and evaluated in this report. Among them, 19 generic QAIs and seven EGSs that fully address their key quality domains were identified and recommended as starting point tools when conducting systematic reviews. The advice and assistance of international experts were solicited in the preliminary stages of this project, and the entire report was subjected to extensive peer review by experts in the field and AHRQ staff. However, this report did not provide guidance on the specific QAIs and EGSs to use. Potential users were encouraged to consider the feasibility, ease of use, and likely applicability of QAIs and EGSs to their own particular projects. 1.6 Objective of the QAT Project The objective of this QAT project was to identify specific QAIs by various study designs and EGSs for CADTH. To accomplish this we built on the AHRQ evidence report No published in 2002 by: updating the review of QAIs and EGSs; and identifying the most appropriate QAIs and EGSs, that is the ones that were most feasible and efficient for CADTH work. 4 Quality Assessment Tools Project Report

17 2 METHODS 2.1 Overview of QAT Project Methods Accepting the findings from AHRQ evidence report No and building upon this report, CADTH applied a systematic process to select QAIs for assessing the quality of SRs, RCTs, and OBSs (cohort and case-control studies) as well as to select an EGS. This project was originally conducted from January 2005 to October 2005 and subsequently updated from September to December The whole process was divided into 10 steps for the original project (Steps 1 to 10) plus two steps for updating (Steps 11 to 12). The overall road map of the project methods is presented in Figure 1, and the key concepts referred to in the AHRQ report and applied to this project are presented in Table1. Generally, the whole project was broken down into five main sections as follows. Within the first section, two comprehensive literature searches and reviews (QAT 1 and QAT 2) were conducted for collecting existing QAIs and EGSs respectively. Collection of existing QAIs and EGSs evaluated in reviews, applied in study reports, and raised from expert consultation (Steps 2 to 5). o QAT 1 focused on groups of QAIs and EGSs (more than one at least for each group) o evaluated in existing review articles (Step 2). QAT 2 focused on individual QAIs and EGSs used in specific study reports (Step 5). Identification of potential QAIs and EGSs from existing ones for (Step 6). Evaluation of the potential QAIs and EGSs identified using AHRQ evaluation grids (Step 7). Consultation with panel experts and stakeholders on QAIs and EGSs selected (Steps 8 to 10). Formal updating of the project following the same process (Steps 11 to 12). Quality Assessment Tools Project Report 5

18 Figure 1: Road Map of QAT Project Methods Step 1 Assemble QAT working group Step 2 QAT 1: Search and select review articles (2000 to Feb. 2005) Step 3 Step 4 First round combined analysis: Identify QAIs and EGSs from review articles First round expert consultation: input from experts on the collection of QAT 1 Collect existing QAIs and EGSs Step 5 Step 6 QAT 2: Search and select individual QAIs and EGSs (2000 to Aug. 2005) Second round combined analysis: Identify QAIs and EGSs for evaluation Identify potential QAIs and EGSs Step 7 Step 8 Step 9 Step 10 Evaluation: Evaluate potential QAIs and EGSs identified Second round expert consultation: Collect experts comments on evaluation results Choose appropriate QAIs and EGSs for COMPUS Stakeholder consultation: Collect stakeholders comments on the choices Evaluate potential QAIs and EGSs Consult on QAIs and EGSs selected Step 11 Step 12 Updating QAT 1: Select review articles (2005 to Sept. 2007) Updating QAT 2: Select individual QAIs and EGSs (2005 to Sept. 2007) Update QAIs and EGSs 6 Quality Assessment Tools Project Report

19 Table 1: Key Concepts Referred to in the AHRQ Report 1 and Applied to the QAT Project Concepts Generic instrument Specific instrument Type of instrument Domain of study methodology Domains for rating the overall strength of a body of evidence Scale Checklist Component Guidance document Quality Quantity Consistency Definitions Instrument could be used to assess quality of any study of the type considered on that grid (page 33 1 ). Instrument is designed to be used to assess study quality for a particular type of outcome, intervention, exposure, test, etc. (page 33 1 ). Instruments that contain several quality items that are scored numerically to provide a quantitative estimate of overall study quality (page 33 1 ). Instruments that contain a number of quality items, none of which is scored numerically (page 33 1 ). Individual aspect of study methodology (e.g., randomization, blinding, follow-up) that has a potential relation to bias in the estimation of effect (page 33 1 ). Publication in which study quality is defined or described, but does not provide an instrument that could be used for evaluative applications (page 33 1 ). A domain of study methodology or execution reflects factors to be considered in assessing the extent to which the study s results are reliable or valid (i.e., study quality). Each domain has specific elements that one might use in determining whether a particular instrument assessed that domain; in some cases, only one element defines a domain (page 34 1 ). The quality of all relevant studies for a given topic, where quality is defined as the extent to which a study s design, conduct, and analysis has minimized selection, measurement, and confounding biases (page 42 1 ). The construct quantity refers to the extent to which there is a relationship between the technology (or exposure) being evaluated and the outcome, as well as to the amount of information supporting that relationship. Three main factors contribute to quantity: the magnitude of treatment effect the number of studies that have evaluated the given topic the overall sample size across all included studies (page 42 1 ). Consistency is the degree to which a body of scientific evidence is in agreement with itself and with outside information. More specifically, a body of evidence is said to be consistent when numerous studies done in different populations using different study designs to measure the same relationship produce essentially similar or compatible results (page 43 1 ). For any given topic, it refers to the extent to which similar findings are reported from work using similar and different study designs (page 42 1 ). AHRQ = Agency for Healthcare Research and Quality; QAT = Quality Assessment Tool. Quality Assessment Tools Project Report 7

20 2.2 Details of QAT Project Methods Step 1: Assemble QAT working group The working group was comprised of internal researchers (AB, VS, and SV), information specialists (GB and HF), a methodology expert (GW), and external researchers (SM, KD and RL). The group was mandated to direct and supervise the process of the selection and evaluation of QAIs and EGSs. Group members took charge of different roles, mainly literature search, as well as the collection and evaluation of QAIs and EGSs, and consultation. Step 2: Search and select review articles (QAT 1: 2000 to February 2005) To achieve the initial identification of QAIs and EGSs, review articles were targeted if they formally evaluated existing QAIs or EGSs. Formal evaluation was defined as the existing QAIs or EGSs were reviewed with domains, validity, and reliability. Search strategy Building upon the original search strategy used in the AHRQ report for literature dating from 1995 to June of 2000, and taking into account the recommendations of the AHRQ team, a sensitive search for systematic review articles of QAIs and EGSs was designed and carried out by an Information Specialist (GB). Published literature was identified by searching the following bibliographic databases: PubMed; MEDLINE; Embase; BIOSIS Previews; and The Cochrane Library. CADTH s methodological filter was applied to limit retrieval to systematic reviews. The search was also limited to documents published between January 2000 and February 2005 without language restrictions. Targeted hand and grey literature searches were also conducted. The QAT 1 search strategy is presented in Appendix B-1. Selection criteria Specifically, the review articles were included for further analysis if they systematically collected and evaluated existing QAIs for SRs/meta-analyses, RCTs or OBSs, and EGSs; or if they contained a comparison or evaluation of at least two existing QAIs or EGSs. To avoid missing any relevant review articles, first level selection included the articles regarding the principles, methods and tools of methodological quality assessment and critical appraisal, and articles generally introducing evidence-based medicine, methodological issues, levels of evidence, research designs, biases, and development of evidence-based practice guidelines. At second level selection, the articles were excluded if they only provided general methodological knowledge of evidence-based medicine, critical appraisal, SRs, and guidelines; or if they introduced only one individual QAI or EGS. The detailed selection criteria of QAT 1 are presented in Appendix C-1. Selection methods Two reviewers (AB and VS) independently screened the titles and abstracts at the first level selection. The full texts of potentially relevant articles identified by either reviewer were retrieved and reviewed at the second level. Two reviewers discussed to resolve any disagreements of selection at the second level. If consensus could not be reached between them, the conflict was resolved by the third party (GW). 8 Quality Assessment Tools Project Report

21 Potentially relevant non-english full articles were screened by one reviewer using three questions (i.e., Did the article have more than one QAI or EGS included? Did the article compare QA tools? Was the article a systematic review?), which were consistent with the inclusion criteria. The full articles that passed the screening were translated into English and Englishversion full articles were independently reviewed by the two reviewers. The screening questions are presented in Appendix C-2. Step 3: Identify QAIs and EGSs from review articles (first round combined analysis) The key components of all included review articles were extracted and tabulated by the number and type of QAIs evaluated and the number of EGSs, search date and database, evaluation domains, recommendations, etc. Individual QAIs and EGSs were identified and collected by one reviewer (AB) from the included review articles. This extraction was checked twice by other two reviewers (SM and KD) respectively. All extracted QAIs and EGSs were listed in one table by study design; that is, QAIs for SRs, RCTs and OBSs/non-randomized studies, multiple design type studies, and EGSs. Duplicates of QAIs or EGSs were identified within each type of study. Reference lists of QAIs collected by the type of study and EGS were created after removing duplicates. Step 4: Conduct initial expert consultation (first round) Based on the search experience of the AHRQ report and QAT 1, it was determined that any search for individual QAIs and EGSs was unlikely to be comprehensive. Therefore, besides a sensitive search, external experts were consulted to ensure comprehensiveness. The consulted experts included all lead authors of the included review articles of QAT1, the expert panel of AHRQ evidence report No.47, 1 and experts who worked in the evidence evaluation methodology field and who were recommended by the QAT working group and COMPUS advisory committee. A package was sent to Canadian and international experts identified to ensure that important review articles on QAT and/or EGS were not missed. The package included a covering letter; an introduction to the CADTH optimal use program; and details about the QAT project, search strategy of and selection of review articles of interest. Experts were asked to help in identifying relevant review articles, other than those included, as well as individual QAIs and EGSs. The input from experts was tabulated by study designs and then compared with the reference lists of existing QAIs and EGSs derived from the included review articles of QAT 1 (SV). The table was checked (RL) to ensure the all experts input had been fully considered. The additional full articles identified from experts input were reviewed by two reviewers (AB and VS) and added to the existing reference lists if they met the selection criteria of QAT 1 or provided individual generic QAIs or EGSs. Quality Assessment Tools Project Report 9

22 Step 5: Search and select individual QAIs and EGSs (QAT 2: 2000 to August 2005) A second literature search was conducted to capture individual QAIs and EGSs published after 2000, as a supplement of the collection from the review articles and experts input. Search strategy A highly specific supplemental literature search was conducted (GB) for the articles applying or reporting individual QAIs or EGSs. Published literature was identified by searching the following bibliographic databases: PubMed and The Cochrane Library including the methodology register. The search was limited to documents published between 2000 and August 2005, without language restriction. See Appendix B-2 for the detailed QAT 2 search strategy. Selection criteria Generic checklists or scales of QAIs for SRs, RCTs, non-rcts, OBSs (cohort and case-control) and generic EGSs were collected from various types of publications. Considering the further application and completeness of quality assessment, instruments for specific use and quality assessment components were excluded respectively. Guidance documents, such as QUOROM for SRs and CONSORT for RCTs, were also excluded because they have not been developed as tools for assessing the quality of individual studies per se. Selection criteria are presented in Appendix C-3. Selection methods First stage selection: The titles and abstracts were screened by two reviewers (AB and VS) independently at the first level selection, and potentially relevant ones that contained individual QAIs or EGSs were identified by either of the reviewers. The full text of potentially relevant articles was retrieved and independently reviewed by the same two reviewers at the second level screening to identify QAIs and EGSs of interest, presented in references, tables, or appendices. Any disagreement was resolved by consensus between the two reviewers or with the judgment from a third party (GW). Non-English full articles were reviewed by the persons who knew those languages. Second stage selection: An additional selection stage was set up for further checking the QAIs and EGSs identified at the second level selection. The individual QAIs and EGSs of interest directly shown in the included articles were reviewed by two reviewers (AB and VS) to identify generic QAIs and EGSs, then they were compared with the existing reference lists derived from QAT 1 and expert consultation to remove duplicates (SV). If included articles only provided the references of QAIs and EGSs of interest, the references were extracted in one Excel table and compared with the existing reference lists by one person (SV) and checked by another person (KD). The full articles of the references not listed were retrieved and reviewed by the two reviewers (AB and VS). All generic checklists or scales of QAIs and generic EGSs identified were added to the existing reference lists for further analysis. Step 6: Identify potential QAIs and EGSs for evaluation (second round combined analysis) Potential QAIs and EGSs, namely ones recommended in review articles plus generic checklists or scales of QAIs and generic EGSs not assessed before, were identified for by 10 Quality Assessment Tools Project Report

23 types of study. The identification was conducted by one reviewer (AB) and checked by another person (KD) as follows: QAIs and EGSs recommended by any review articles were included in the. If any tools overlapped between reviews, AHRQ evaluation results were referred to first before the recommendations from the other reviews where considered. If there was a big difference in the criteria of the evaluation and recommendation between the AHRQ report and other review articles, a further selection of the tools contained in the other reviews was conducted based on the criteria modified by the working group. If some review articles only evaluated QAIs and EGSs, but did not provide any recommendation information, tools from those reviews were included for after confirming that they were not covered by the reviews with recommendations, regardless of which tools were recommended. Generic checklists or scales of QAIs and generic EGSs identified by expert consultation and QAT 2 were included for as they were not assessed by any review article. Full articles of the potential QAIs and EGSs collected from reviews other than the AHRQ report were retrieved. Two reviewers (AB and VS) independently checked them using the selection criteria of QAT 2 to ensure that generic checklists or scales of QAIs and generic EGSs were included in the. Step 7: Evaluate potential QAIs and EGSs identified Briefly, the potential QAIs and EGSs identified above were evaluated by the types of study using the AHRQ evaluation grids, including domain criteria and elements created within each domain. Strengths and weaknesses of different instruments or systems were shown in grids, with columns denoting the evaluation domains of interest and the rows for the individual tools. Each of the QAIs and EGSs was awarded either a "yes" ( fully addressed a domain), "partial" ( addressed a domain to some extent), or a "no" ( did not deal with a domain) for each domain. 1 The domains applied in the QAT project are presented in Table 2. Type of Study QAI for SR QAI for RCT QAI for OBS Table 2: AHRQ Evaluation Grids 1 Applied in the QAT Project by Types of Study Domain 1 Domain 2 Domain 3 Domain 4 Domain 5 Domain 6 Domain 7 research questions study population search strategy * in/exclusio n criteria data extraction study quality / validity * data funding * synthesis * randomization * blinding * interventions outcomes statistical analysis * funding * comparability exposure / outcome statistical funding * NA NA of subjects * intervention measure analysis EGS quality quantity consistency NA NA NA NA AHRQ = Agency for Healthcare Research and Quality; EGS = evidence grading system; NA = not available; OBS = observational study; QAI = quality assessment instrument; QAT = quality assessment tool; RCT = randomized controlled trial; SR = systematic review. *Domains containing empirical elements. Quality Assessment Tools Project Report 11

24 The details of the evaluation grids (domains and elements) and evaluation rules were presented in the methods section of the AHRQ report, and a brief summary is presented in the following paragraph. The majority of domains and their elements were based on standard good practice epidemiologic methods that were generally accepted. Methodological research has proved that elements with a demonstrable basis in empirical research can affect the conduct and analysis of a study. 1 If domains have empirical elements, they were generally assigned more weight (see domains* in Table 2). For example, the domain of randomization contains three empirical elements (adequate sequence generation, adequate concealment, and baseline similarity) for RCTs. Most of the domains in Table 2 had at least one element identified as essential to be considered for judging whether instruments fully cover that domain or not; for example, the element adequate concealment method for the domain randomization. For domains with multiple elements, a yes rating was required to consider whether that instrument had to address some specified elements or a majority of elements. The remaining domains presented in Table 2 (i.e., those not marked by an asterisk) contain elements that were derived from best practices, and were considered critical for study design, but had not been tested in real life. Specifically to the three domains for EGSs, quality included only one element based on methodological rigour, the extent to which bias was minimized; quantity combined three elements numbers of studies, sample size or power, and the magnitude of effect with a full yes requiring two of the three elements to be covered; consistency had only one element for a summary finding from more than one study reviewed, with a dichotomous yes or no indicating whether a system took consistency into account in its view of the strength of evidence. All of these domains with either empirical or best practice elements were applied in the AHRQ report as the criteria to identify the acceptable QAIs and EGSs with confidence and without major modifications for the current use. 1 The evaluation results of the QAIs and EGSs recommended in the AHRQ report were directly applied in this project. In order to be consistent with the AHRQ evaluation, two reviewers (SM and AB) randomly selected one QAI for each type of study and one EGS from the AHRQ recommendations to repeat the evaluation. After resolving the discrepancy between the AHRQ evaluation and the two reviewers, one potential instrument identified from other than the AHRQ report for each type of study was randomly selected for the evaluation conducted by two reviewers independently. After reaching consensus between them, one reviewer (SM) evaluated the rest of potential QAIs and EGSs identified and the other reviewer (AB) checked the evaluation results. Any disagreement between the two reviewers was solved by group discussion (SM, AB, VS, and GW). The highest scoring QAIs and EGSs from each grid represented the proposed selections. To identify more appropriate tools for use, the descriptive information of tools provided in the AHRQ report was also referred to when further considering the instruments or systems with the same highest evaluation scores within each type of study, including rigour of development process, inter-rater reliability, instructions provided, etc. Moreover, the length of tools, ease of use, and time consumed for assessing one article were considered. After removing relatively unpractical ones, the choices of tools were proposed. 12 Quality Assessment Tools Project Report