EVIPNet Capacity-Building Workshop Ethiopian Health and Nutrition Research Institute, Addis Ababa, Ethiopia 18 to 22 February 2008

Transcription

1 EVIPNet Capacity-Building Workshop Ethiopian Health and Nutrition Research Institute, Addis Ababa, Ethiopia 18 to 22 February 2008 Assessment Criteria for Systematic Reviews (Last updated by John Lavis on 13 February 2008) A. Considerations related to the quality of a systematic review Table 2: Excerpted from Shea et al. BMC Medical Research Methodology :10 doi: / AMSTAR is a measurement tool created to assess the methodological quality of systematic reviews. 1. Was an 'a priori' design provided? The research question and inclusion criteria should be established before the conduct of the review. t 2. Was there duplicate study selection and data extraction? There should be at least two independent data extractors and a consensus procedure for disagreements should be in place. t 3. Was a comprehensive literature search performed? At least two electronic sources should be searched. The report must include years and databases used (e.g. Central, EMBASE, and MEDLINE). Key words and/or MESH terms must be stated and where feasible the search strategy should be provided. All searches should be supplemented by consulting current contents, reviews, textbooks, specialized registers, or experts in the particular field of study, and by reviewing the references in the studies found. t 4. Was the status of publication (i.e. grey literature) used as an inclusion criterion? The authors should state that they searched for reports regardless of their publication type. The authors should state whether or not they excluded any reports (from the systematic review), based on their publication status, language etc. 5. Was a list of studies (included and excluded) provided? A list of included and excluded studies should be provided. t t

2 2 6. Were the characteristics of the included studies provided? In an aggregated form such as a table, data from the original studies should be provided on the participants, interventions and outcomes. The ranges of characteristics in all the studies analyzed e.g. age, race, sex, relevant socioeconomic data, disease status, duration, severity, or other diseases should be reported. 7. Was the scientific quality of the included studies assessed and documented? 'A priori' methods of assessment should be provided (e.g., for effectiveness studies if the author(s) chose to include only randomized, double-blind, placebo controlled studies, or allocation concealment as inclusion criteria); for other types of studies alternative items will be relevant. 8. Was the scientific quality of the included studies used appropriately in formulating conclusions? The results of the methodological rigor and scientific quality should be considered in the analysis and the conclusions of the review, and explicitly stated in formulating recommendations. 9. Were the methods used to combine the findings of studies appropriate? For the pooled results, a test should be done to ensure the studies were combinable, to assess their homogeneity (i.e. Chi-squared test for homogeneity, I 2 ). If heterogeneity exists a random effects model should be used and/or the clinical appropriateness of combining should be taken into consideration (i.e. is it sensible to combine?). 10. Was the likelihood of publication bias assessed? An assessment of publication bias should include a combination of graphical aids (e.g., funnel plot, other available tests) and/or statistical tests (e.g., Egger regression test). 11. Was the conflict of interest stated? Potential sources of support should be clearly acknowledged in both the systematic review and the included studies. Shea et al. BMC Medical Research Methodology :10 doi: / t t t t t t

3 3 B. Considerations related to local applicability 1. Are there important differences in the structural elements of health systems (i.e., governance, financial and delivery arrangements) between where the research was done and where it could be applied that might mean a policy option could not work in the same way? e.g., Research on the effectiveness of bulk purchasing arrangements in lowering prices for prescription drugs may have been done in countries with no concentration in the ownership of pharmacies, whereas many policymakers may work in countries with a monopoly in pharmacy ownership 2. Are there important differences in on-the-ground realities and constraints (i.e., governance, financial and delivery arrangements) between where the research was done and where it could be applied that might substantially alter the potential benefits of a policy option? And can these challenges be addressed in the short-term to medium-term? e.g., Research on the effectiveness of a team-based approach to maternity care in reducing both maternal and child morbidity may have been done in countries with midwives and traditional birth attendants, whereas many policymakers may work in countries with neither type of health provider 3. Are there likely to be important differences in the baseline conditions between where the research was done and where it could be applied that might mean a policy option would have different absolute effects, even if the relative effectiveness was the same? e.g., Research on the effectiveness of a strategy for promoting HIV testing among pregnant women may have been done in countries where less than 10% of pregnant women were offered HIV testing, whereas many policymakers may work in countries where 85% or more of pregnant women are offered HIV testing 4. Are there important differences in the perspectives and influence of health system stakeholders (i.e., political challenges) between where the research was done and where it could be applied that might mean a policy option will not be accepted or taken up in the same way? And can these challenges be addressed in the short-term to medium-term? Research on the effectiveness (and safety) of nurse practitioners in providing routine medical care for children may have been done in countries with shortages of physicians and weak medical associations, whereas many policymakers may work in countries with a surplus of physicians and a very strong and vocal medical association

4 4 C. Considerations related to equity A policy option may have an impact on disadvantaged populations and equity in a health system. Disadvantages should be considered in relation to each of the following potentially relevant dimensions: place of residence, race (i.e., ethnic origin), occupation, gender, religion, education, socioeconomic status, and social network and capital (PROGRESS). 1. Are there plausible reasons for anticipating differences in the relative effectiveness of the policy option in disadvantaged settings within the country? 2. Are there likely to be different baseline conditions within the country, so that the problem would be more or less important in disadvantaged settings within the country? 3. Are there likely to be different baseline conditions in disadvantaged settings within the country, so that the absolute effectiveness would be different? 4. Are there important considerations that should be given to implementing the policy option to ensure that inequities are not increased and that they are reduced, if possible (e.g. in terms of ensuring access in disadvantaged settings)? 5. Would the policy option likely reduce or increase health inequities within the country, or would it result in no change?

5 5 D. Considerations related to scaling up 1. Are there important challenges that will need to be addressed when rolling out or scaling up the policy option? 2. How complex is the policy option and does this have implications for scale up? 3. What are the total costs of expanding coverage of the policy option and sustaining it and what are the implications for scale up? 4. What are the requirements that the policy option imposes on government capacity (e.g., effective regulatory capacity) and the implications for scale up? 5. What requirements does the policy option place upon patients, health providers and managers, and what are the implications for scale up? 6. Would widespread implementation of the policy option be likely to have important impacts on the healthcare system or other sectors and, if so, what are the implications for scale up? 7. Is the policy option likely to be difficult to sustain or are its effects likely to change over time? For example, is it likely that the policy option will have deteriorating benefits without the ongoing training and support necessary to ensure that it is properly implemented?