Quality and critical appraisal of clinical practice guidelines a relevant topic for health care?

Transcription

1 Quality and critical appraisal of clinical practice guidelines a relevant topic for health care? Françoise Cluzeau, PhD St George s Hospital Medical School, London on behalf of the AGREE Collaboration

2 WHY APPRAISE GUIDELINES? A guideline is a form of intervention. It can potentially affect a lot of patients. Users need to have confidence in recommendations. Professional and governmental agencies have to ensure guidelines are good before recommending them.

3 BACKGROUND TO AGREE UK work National workshop Appraisal instrument developed and validated in the UK Used in the UK, Europe & Canada EU funding Funding obtained from the 4th Framework of EU Programme for 3 years

4

5 The AGREE Collaboration FINLAND SCOTLAND ENGLAND & WALES DENMARK NETHERLANDS GERMANY SPAIN FRANCE SWITZERLAND ITALY USA Canada New Zealand

6 WHY AN INTERNATIONAL COLLABORATION? Clinical guidelines play an increasingly important role in healthcare in most countries. Ensuring their quality is a shared concern. There is a need for international guidance on guideline development. The whole is greater that the sum of its parts.

7 OBJECTIVES Develop compatible approaches to the development of clinical guidelines. Establish a structure for the appraisal and monitoring of Clinical guidelines. Define quality criteria relevant to guidelines. Promote and encourage the diffusion of these criteria through international exchanges and collaborative links.

8 INTEGRATED RESEARCH PROGRAMME 1 Create an international instrument to appraise the quality of clinical guidelines. 2 Formulate examples of good guideline for guideline developers. 3 Describe and compare guideline development programmes. 4 Undertake content analysis of appraised guidelines. 5 Explore methodologies for assessing individual recommendations.

9 APPRAISAL OF GUIDELINES for RESEARCH & EVALUATION (AGREE) INSTRUMENT The AGREE Collaboration September 2001

10 PURPOSE OF THE AGREE INSTRUMENT To provide a systematic framework for appraising the quality of Clinical guidelines.

11 Quality of clinical practice guidelines is the confidence that: 1. The biases of guideline development have been addressed adequately. 2. The recommendations are both internally and externally valid, and are feasible for practice. AGREE Instrument assesses quality of reporting, and quality of some aspects of recommendations. It provides an assessment of the predicted validity of a guideline (that it will achieve its intended outcome).

12 PURPOSE OF THE AGREE INSTRUMENT To assess local, regional, national or international guidelines. Can be applied to guidelines in any disease area. Can be used by: policy makers, guideline users, guideline developers, educators.

13 STRUCTURE 23 items 4-point Likert Scale Overall assessment User guide Domains 1. Scope & purpose (3) 2. Stakeholder involvement (4) 3. Rigour of development (7) 4. Clarity & presentation (4) 5. Applicability (3) 6. Editorial independence (2)

14 DEVELOPMENT PROCESS Subgroup reviews literature First draft circulated for comments First workshop, instrument modified First validation study Second workshop, Instrument revised Final validation study

15 VALIDATION PROCESS Phase 1 The instrument was applied to 100 guidelines from 11 countries. All countries followed the same protocol. Each guideline was independently assessed by 4 appraisers (194 altogether). Instrument was used in English. Phase 2 Refined instrument tested on 33 guidelines randomly selected from 1st study. Same countries participated. New set of appraisers recruited (70) Each country assessed 3 guidelines

16 CALCULATING GUIDELINES SCORES Mean item scores were calculated for each guideline by averaging scores across the 4 appraisers. Standardised guideline domain scores were calculated by summing up the 4 appraisers scores and dividing by maximum possible score.

17 ANALYSIS 1 Refining the instrument Factor Analysis (principal Components Analysis) was used with mean guidelines item scores from phase1. Appraisers comments reviewed. Reliability Internal consistency (Cronbach s α) Inter-rater agreement (Intra Class Correlation Coefficient)

18 ANALYSIS 2 Content validity (comments from partners and experts) Face validity (appraiser s comments from questionnaire) Construct validity (three hypotheses tested) Criterion validity (Kendall s Tau B rank correlation between appraisers s domain scores & overall assessment)

19 THREE HYPOTHESES 1.Guidelines originating from established programmes would have higher domain scores than those produced outside established programmes. 2. Guidelines supported by well-documented technical information, will have domain scores higher than those without this documentation. 3. Nationally produced guidelines should be higher quality than regional or local ones.

20 RESULTS Refining the instrument PCA analysis suggested 5 quality domains coherent with the conceptual framework. A sixth domain was introduced for editorial independence. A new item was included for the sixth domain. Two items were removed. 10 items reworded.

21 RELIABILITY Domains Cronbach α Intraclass Correlation Individual appraiser (ICC) Mean of 4 appraisers 1. Scope & purpose Stakeholder involvement Rigour of development Clarity & presentation Applicability Editorial independence

22 FACE VALIDITY 1 st field testing n (%) 2 nd field testing n (%) Number of appraisers Found Instrument useful 182 (94) 70 (100) Found user guide helpful 184 (95) 69 (99) Reported difficulties with items 92 (47) 20 (29) Median time to appraise a guideline = 90 minutes

23 CONSTRUCT VALIDITY: hypothesis 1 D 1 D2 D 3 D4 D5 D 6 A * B A: Guidelines developed within a guideline programme (n = 20) B: Guidelines developed outside a guideline programme (n = 13) * p<0.05

24 CONSTRUCT VALIDITY: hypothesis 2 D 1 D2 D 3 D4 D5 D 6 A B C ** A: No background information (n = 7) B: Some information /references (n = 10) C: Detailed documentation (n = 16) **p=0.001

25 CONSTRUCT VALIDITY: hypothesis 3 D 1 D2 D 3 D4 D5 D 6 A B * A: Guidelines developed before national policy (n = 13) B: Guidelines developed after national policy (n = 20) * p<0.05

26 CRITERION VALIDITY Domain 1 Domain 2 Domain 3 Domain 4 Domain 5 Domain 6 Overall Domain Domain Domain Domain Domain Domain overall

27 GUIDELINES DOMAIN SCORES DOMAIN Mean SD Range Scope & purpose Stakeholder Rigour of development Clarity Applicability Editorial independence

28 CONCLUSIONS AGREE is the first appraisal instrument for clinical guidelines to be developed and tested internationally. It can be used consistently by a wide range of professionals from different cultural backgrounds. It has satisfactory reliability. There is some suggestion of validity.

29 The Council of Europe has recommended its use OUTCOMES The AGREE Instrument has been translated into 7 languages. EU projects are encouraged to assess guidelines with the AGREE Instrument WHO has endorsed the AGREE Instrument

30 RESEARCH & POLICY IMPLICATIONS The AGREE Instrument creates a common framework for assessing guidelines that can be applied internationally. Can be used to compare quality of guidelines across countries and disease areas. Provides standards for reporting of clinical guidelines published in medical journals. Governmental agencies may want to use AGREE to judge the quality of national guidelines and to make policy decisions.

31 REMAINING ISSUES Validity. Defining quality by rigour of reporting may not capture the essence of the quality feature. Quality concepts may not be universal. Need to link systematically developed guidelines with implementation strategies.

32 WHAT NEXT? g Publications, monographs, conferences and workshops. g Disseminate AGREE Instrument through workshops, teaching and website. g Involves all original countries + 6 further European countries, and outreach visits to 7 central & East European countries. g Develop training manual and evaluation template. g Produce electronic library of guideline development tools

33 FUTURE PROJECTS AND PROPOSALS gestablish an international guideline network. gundertake further research programme through 6th framework of the EU.

34 Website address Agreecollaboration.org