PCORI Methodology Standards: Academic Curriculum Patient-Centered Outcomes Research Institute. All Rights Reserved.

Transcription

1 PCORI Methodology Standards: Academic Curriculum 2016 Patient-Centered Outcomes Research Institute. All Rights Reserved.

2 Module 6: Structured Reporting of Diagnostic Comparative Effectiveness Study Results Category 10: Studies of Diagnostic Tests Prepared by Najlla Nassery, MD, MPH Jodi Segal, MD, MPH Presented by Najila Nassery, MD, MPH

3 Structured Reporting of Diagnostic Comparative-Effectiveness Study Results Thorough and unambiguous reporting of information in a diagnostic comparativeeffectiveness study is essential for accurate interpretation of its results and subsequent integration of the results into systematic reviews for decision making An empirical evaluation of 487 published diagnostic accuracy studies in 30 systematic reviews found that only one of seven studies qualified as being high quality (Leeflang, et al., 2007) This was due in part to the poor reporting of study design features and patient characteristics The PCORI methodology standard for studies of diagnostic tests recommends that investigators consult and use broadly accepted checklists for reporting study results and assessing study quality Source: Leeflang, M., et al. (2007). Clinical Chemistry, 53(2),

4 Structured Checklists for Reporting Study Results and Assessing Study Quality PCORI specifically encourages investigators to consult and use the following checklists: CONSORT (Consolidated Standards of Reporting Trials) 2010 STARD (Standards for the Reporting of Diagnostic Accuracy Studies) checklist QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies 2) These recommended checklists contain most of the elements described in the PCORI methodology standards These checklists have generally been described as easy to use in several empirical evaluations 4

5 CONSORT 2010 The CONSORT checklist has been proposed for use by researchers reporting randomized controlled trials since 2001 The checklist was updated in 2010, has been published in 11 medical journals, and has wide support from >600 journals, although compliance is not required by these journals CONSORT has 25 items focusing on five categories: 1. Title and abstract 2. Introduction 3. Methods Description of participants, interventions, objectives, outcomes, sample size, randomization method, allocation concealment, blinding, statistical methods 4. Results 5. Discussion 5

6 CONSORT 2010 Section/Topic Item No Checklist item Title and abstract 1a Identification as a randomised trial in the title 1b Structured summary of trial design, methods, results, and conclusions (for specific guidance see CONSORT for abstracts) Introduction Background and 2a Scientific background and explanation of rationale objectives 2b Specific objectives or hypotheses Methods Trial design 3a Description of trial design (such as parallel, factorial) including allocation ratio 3b Important changes to methods after trial commencement (such as eligibility criteria), with reasons Participants 4a Eligibility criteria for participants 4b Settings and locations where the data were collected Interventions 5 The interventions for each group with sufficient details to allow replication, including how and when they were actually administered Outcomes 6a Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed 6b Any changes to trial outcomes after the trial commenced, with reasons Sample size 7a How sample size was determined 7b When applicable, explanation of any interim analyses and stopping guidelines Randomisation: Sequence generation 8a Method used to generate the random allocation sequence 8b Type of randomisation; details of any restriction (such as blocking and block size) Allocation concealment Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to 9 mechanism conceal the sequence until interventions were assigned Implementation 10 Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions Blinding 11a If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how 11b If relevant, description of the similarity of interventions Reported on page No Source: CONSORT 2010 checklist of information to include when reporting a randomized trial. Available at: Accessed August 10,

7 CONSORT 2010 Section/Topic Item No Checklist item Statistical methods 12a Statistical methods used to compare groups for primary and secondary outcomes 12b Methods for additional analyses, such as subgroup analyses and adjusted analyses Results Participant flow (a For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary 13a diagram is strongly outcome recommended) 13b For each group, losses and exclusions after randomisation, together with reasons Recruitment 14a Dates defining the periods of recruitment and follow-up 14b Why the trial ended or was stopped Baseline data 15 A table showing baseline demographic and clinical characteristics for each group Numbers analysed 16 For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence 17a Outcomes and estimation interval) 17b For binary outcomes, presentation of both absolute and relative effect sizes is recommended Ancillary analyses 18 Results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing pre-specified from exploratory Harms 19 All important harms or unintended effects in each group (for specific guidance see CONSORT for harms) Discussion Limitations 20 Trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses Generalisability 21 Generalisability (external validity, applicability) of the trial findings Interpretation 22 Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence Other information Registration 23 Registration number and name of trial registry Protocol 24 Where the full trial protocol can be accessed, if available Funding 25 Sources of funding and other support (such as supply of drugs), role of funders Reported on page No Source: CONSORT 2010 checklist of information to include when reporting a randomized trial. Available at: Accessed August 10,

8 STARD The STARD checklist is used for reporting on diagnostic accuracy studies STARD was published in seven medical journals in 2003 for use in reporting of accuracy tests More than 200 medical journals have mentioned the STARD statement in their instructions to authors The checklist contains 25 items, in similar categories as in CONSORT, but focuses on diagnostic test accuracy issues Source: STARD Statement: STAndards for the Reporting of Diagnostic accuracy studies. Available at: Accessed August 10,

9 STARD Section and Topic TITLE/ABSTRACT/ KEYWORDS Item # 1 Identify the article as a study of diagnostic accuracy (recommend MeSH heading 'sensitivity and specificity'). INTRODUCTION 2 State the research questions or study aims, such as estimating diagnostic accuracy or comparing accuracy between tests or across participant groups. METHODS Participants 3 The study population: The inclusion and exclusion criteria, setting and locations where data were collected. 4 Participant recruitment: Was recruitment based on presenting symptoms, results from previous tests, or the fact that the participants had received the index tests or the reference standard? 5 Participant sampling: Was the study population a consecutive series of participants defined by the selection criteria in item 3 and 4? If not, specify how participants were further selected. 6 Data collection: Was data collection planned before the index test and reference standard were performed (prospective study) or after (retrospective study)? Test methods 7 The reference standard and its rationale. 8 Technical specifications of material and methods involved including how and when measurements were taken, and/or cite references for index tests and reference standard. 9 Definition of and rationale for the units, cut-offs and/or categories of the results of the index tests and the reference standard. 10 The number, training and expertise of the persons executing and reading the index tests and the reference standard. 11 Whether or not the readers of the index tests and reference standard were blind (masked) to the results of the other test and describe any other clinical information available to the readers. Statistical methods 12 Methods for calculating or comparing measures of diagnostic accuracy, and the statistical methods used to quantify uncertainty (e.g. 95% confidence intervals). 13 Methods for calculating test reproducibility, if done. RESULTS Participants 14 When study was performed, including beginning and end dates of recruitment. 15 Clinical and demographic characteristics of the study population (at least information on age, gender, spectrum of presenting symptoms). 16 The number of participants satisfying the criteria for inclusion who did or did not undergo the index tests and/or the reference standard; describe why participants failed to undergo either test (a flow diagram is strongly recommended). Test results 17 Time-interval between the index tests and the reference standard, and any treatment administered in between. 18 Distribution of severity of disease (define criteria) in those with the target condition; other diagnoses in participants without the target condition. 19 A cross tabulation of the results of the index tests (including indeterminate and missing results) by the results of the reference standard; for continuous results, the distribution of the test results by the results of the reference standard. 20 Any adverse events from performing the index tests or the reference standard. Estimates 21 Estimates of diagnostic accuracy and measures of statistical uncertainty (e.g. 95% confidence intervals). 22 How indeterminate results, missing data and outliers of the index tests were handled. 23 Estimates of variability of diagnostic accuracy between subgroups of participants, readers or centers, if done. 24 Estimates of test reproducibility, if done. DISCUSSION 25 Discuss the clinical applicability of the study findings. On page # Source: STARD Statement: STAndards for the Reporting of Diagnostic accuracy studies. Available at: Accessed August 10,

10 QUADAS-2 QUADAS was published in 2003 for use in assessing the quality of diagnostic accuracy studies used in systematic reviews The checklist was updated in 2011 (QUADAS-2) 10

11 Center for Disease Control s ACCE Framework The ACCE project (Analytical validity, Clinical validity, Clinical utility, and Ethical, legal, and social implications) was sponsored in 2000 ACCE framework involves evaluating a test and its application to a specific clinical scenario by examining: Analytical validity: How well does the test measure what it is supposed to measure? Clinical validity: How well does the test predict the phenotype of concern (e.g., risk of disease, prognosis, or response to therapy)? Clinical utility: Does clinical use of the test provide a health benefit? and by considering the ethical, legal, and social implications of using the test in individual patients 11

12 Center for Disease Control s ACCE Framework Source: CDC. (2010). Genomic Testing: ACCE Model Process for Evaluating Genetic Tests. Retrieved August 10, 2015, from 12

13 References Carlos, R., Gareen, I., Gatsonis, C., Gorelick, J., Kessler, L., Lau, J., Rutter, C., Schmid, C., Tosteson, A. N. A., Trikalinos, T. (2012). Standards in the Design, Conduct and Evaluation of Diagnostic Testing for Use in Patient Centered Outcomes Research. PCORI. Available at: Testing-for-Use-in-Patient-Centered-Outcomes-Research.pdf. Accessed August 7, CDC. (2010). Genomic Testing: ACCE Model Process for Evaluating Genetic Tests. Retrieved August 10, 2015, from CONSORT 2010 checklist of information to include when reporting a randomized trial. Available at: Accessed August 10,

14 References Leeflang, M., Reitsma, J., Scholten, R., Rutjes, A., Di Nisio, M., Deeks, J., & Bossuyt, P. (2007). Impact of adjustment for quality on results of metaanalyses of diagnostic accuracy. Clinical Chemistry, 53(2), STARD Statement: STAndards for the Reporting of Diagnostic accuracy studies. Available at: Accessed August 10, Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., for the QUADAS-2 Group. (2011). QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine, 155(8),