Article Four Different Types of Evidence / Literature Reviews

Transcription

1 Article Four Different Types of Evidence / Literature Reviews The rapid growth in the number of reviews undertaken can partly be explained by the current emphasis on evidence-based practice. Healthcare practitioners such as pharmacists need accurate, up-to-date information about what works and they often look to evidence reviews for this. In addition, changes in information technology means that a vast amount of evidence is increasingly accessible to a wider audience. This often needs to be collated and represented in order to be used by practitioners, policy-makers and commissioners. This article outlines the key features of different types of review and discusses their similarities and differences. However, it is important to remember that the terms used to describe review approaches are not always well defined or used consistently. This article will help you to identify just what type of review you are reading and to understand its potential uses and limitations. Systematic reviews are the most scientifically robust reviews and, as such, are regarded as the gold standard. However, it is not always appropriate to undertake a systematic review. For example, there may be insufficient numbers of primary studies (original research) available or the methodological nature of the studies may make them unsuitable for systematic review. In this sense, there is no single ideal type of review. In the same way that the research method you employ is dependent upon your research question 1 ; the type of review undertaken will also be determined by the question or topic it seeks to address. Systematic Reviews Systematic reviews are reviews in which the evidence from a number of studies is gathered together in one report. The review pools and analyses all available data to assess the overall strength of the evidence 2. Systematic methods are used to identify original research which is relevant to the review question. These primary studies are then evaluated to ensure they are methodologically and analytically robust. Studies that do not meet stringent quality criteria are excluded from the review. Reviewing only high quality studies that are relevant to your review question ensures that bias is minimised and reliable results are produced. One example of a systematic review is that conducted by Watson et al Oral versus intra-vaginal imidazole and triazole anti-fungal agents for the treatment of uncomplicated vulvovaginal candidiasis (thrush): a systematic review. 1 Pharmacy Research & Evaluation Resources, series one, article one 2 Sense about Science, 2009, Systematic Reviews Royal Pharmaceutical Society 2011 Page 1 of 10

2 By definition, this was a review of previous studies which compared oral versus intra-vaginal anti-fungal treatments. The review team devised clear and explicit inclusion, exclusion, methodological and quality criteria which ensured that the studies included were appropriate. During the search stage of the review, 833 potential publications were identified. Of these, 811 were excluded as not relevant to the purpose of the review. Of the remaining 22 comparative trials, a further 5 were excluded on methodological or clinical grounds (i.e. random allocation was not specified or the participants had recurrent rather than acute candidiasis). Although it may seem counterintuitive to eliminate apparently relevant studies, it is essential to evaluate a study s methodological quality before inclusion. If studies with poor methodology are included then their results might distort the review s conclusion. Ensuring that included studies are robust is more important than simply getting more data 3. The key components of a systematic review are: A clearly stated set of objectives with pre-defined eligibility criteria for studies An explicit, reproducible methodology A systematic search that attempts to identify all studies that would meet the eligibility criteria An assessment of the validity of the findings of the included studies, for example through the assessment of risk of bias A systematic presentation, and synthesis, of the characteristics and findings of the included studies Because of their importance in delivering evidence-based practice, systematic reviews should be an integral part of a pharmacist s professional life. To coincide with the release of this series, Dr Margaret Watson, a pharmacist with an impressive track record of published systematic reviews, presented a webinar which provides additional information and guidance on understanding and undertaking systematic reviews. The webinar discussed: The rationale for conducting systematic reviews The difference between systematic and non-systematic reviews The key components of systematic reviews The quality markers of systematic reviews The relevance of systematic reviews to pharmacy practice Access to the webinar can be found here =0D74481C3C1D12D5 Meta-analysis 3 Sense about Science, 2009, Systematic Reviews Royal Pharmaceutical Society 2011 Page 2 of 10

3 Meta-analysis refers to the use of statistical techniques to integrate the results of primary studies. Combining the results of several studies can give a more reliable and precise estimate of an intervention s effectiveness than one study alone. A meta-analysis provides increased numbers of participants, reduces random error, narrows confidence intervals, and provides a greater chance of detecting a real effect as statistically significant (i.e. increases statistical power) 4. Many systematic reviews limit their inclusion criteria to randomised controlled trials (RCTs) with comparable designs, particularly if they are reviewing the evidence on the efficacy of a particular medicine. Because of this, metaanalysis is possible and desirable in some (thought not all) systematic reviews. However, not all systematic reviews limit their inclusion criteria to RCTs, either because insufficient numbers of RCTs have been conducted, or because the review question would not be appropriately addresses through the conduct of RCTs, or the analysis of existing RCT data. This means that meta-analysis is not always possible or sensible in order to conduct a meta-analysis you need to be able to compare like with like. Meta-analysis of methodologically diverse or poor quality studies could be seriously misleading as errors or biases in individual studies would be compounded and the very act of synthesis may give credence to poor quality studies. When used appropriately meta-analysis has the advantage of being explicit in the way that data from individual studies are combined. It is a powerful tool in helping avoid misinterpretation and allowing meaningful conclusions to be drawn across studies 5. Narrative synthesis Narrative synthesis refers to the adoption of a textual (as opposed to statistical) approach that provides an analysis of the relationships within and between studies and an overall assessment of the robustness of the evidence. A narrative synthesis of studies may be undertaken where studies are too diverse (either clinically or methodologically) to combine in a meta-analysis. Even where a meta-analysis is possible, aspects of narrative synthesis will usually be required in order to fully interpret the collected evidence. Because narrative synthesis is inherently a more subjective process than meta-analysis, it is crucial that the approach used to analyse and assess the evidence is rigorous and transparent. This will reduce the potential for bias and ensure the review is replicable. 4 Centre for Reviews and Dissemination, 2009, Systematic Reviews: CRD s guidance for undertaking reviews in health care, 5 As above Royal Pharmaceutical Society 2011 Page 3 of 10

4 Narrative review normally consists of the following four elements, each of which needs to be adequately explained with the review. Developing a theory of how the intervention works, why and for whom Developing a preliminary synthesis of findings of included studies Exploring relationships within and between studies Assessing the robustness of the synthesis The idea of narrative synthesis within a systematic review should not be confused with broader terms like narrative review, which are sometimes used to describe reviews that are not systematic 6. Realist Reviews Some healthcare interventions, such as administering a specific drug for the treatment of an acute condition, are conceptually simple and their effectiveness will generally have been evaluated in RCTs. Evaluating and comparing interventions such as these are most appropriately addressed through a systematic review. Many healthcare interventions though, are more complex and their effectiveness is dependent upon the context in which they are applied. Economic, environmental, social and individual factors can all play a part in determining whether or not an intervention will succeed. Alternative forms of review need to be undertaken in order to examine the relationship between the intervention and context which is so often central to an intervention s efficacy. Indeed, the systematic review of oral versus intra-vaginal anti-fungal treatment for thrush alludes to this social context by attempting to establish patient preferred route of administration. Because there was limited data which addressed this secondary outcome of the review, the authors were unable to definitively state patient preferred route. However, as pharmacists you will be acutely aware of the need to provide treatments with which patients will adhere. A drug that has been proved to be more effective than another through RCTs will not necessarily be more effective if it is not taken as prescribed in the real world of the patient. Realist reviews are concerned with the identification of underlying causal mechanisms, how they work, and under what conditions. Fundamentally, they focus attention on finding out what works, for whom, how and in what circumstances 7. To achieve this, realist reviews typically draw on a wide range of studies that use diverse research methods (both quantitative and qualitative). 6 Centre for Reviews and Dissemination, 2009, Systematic Reviews: CRD s guidance for undertaking reviews in health care, 7 Rycroft-Malone, J. et al 2010, A realistic evaluation: the case of protocol-based care, Implementation Science 2010,58 Royal Pharmaceutical Society 2011 Page 4 of 10

5 In contrast to systematic reviews, realist reviews do not begin with a clearly defined protocol setting out the criteria under which studies will be excluded. This does not mean however, that the review will begin without a search strategy. Reviewers should have given much thought to the key words they will search for and the databases they will use. Unlike a systematic review though, the search process will be more iterative and will be refined and developed as the review progresses. The search is likely to begin with a background search to get a feel for the literature what is there, what form it takes, where it seems to be located and how much there is. This will then be followed by a search for empirical evidence from a range of primary studies using a variety of research strategies. At this stage, the reviewer should provide a formal audit trail. In realist reviews, studies are not quality assessed in the same way as when undertaking systematic reviews. During synthesis, the relative contribution of each source is assessed and the reasoning behind decisions reached is made explicit. For example, the analysis spells out the grounds for being cautious about A because of what we have learned from B and what was indicated in C. The worth of studies is established in synthesis and not as a preliminary prequalification exercise. Unlike systematic reviews, realist reviews are not standardized or replicable 8. The key components of a realist review are: A focus on context and process They draw on a range of methodologically diverse material The protocol is iterative rather than pre-defined The value of studies is established in synthesis They are not standardized or replicable Reviews which combine quantitative and qualitative studies In addition to realist reviews, other review methods enable the inclusion of both quantitative and qualitative empirical studies. This is usually done to enhance the relevance of the review in the decision making process by including some aspect of social context. One example of this is Raynor et al s (2007) A systematic review of quantitative and qualitative research on the role and effectiveness of written information available to patients about individual medicines. 8 Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005), Realist Review a new method of systematic review designed for complex policy interventions, Royal Pharmaceutical Society 2011 Page 5 of 10

6 The aim of this review was: To establish the role and value of written information available to patients about individual medicines from the perspective of patients, carers and professionals and, To determine how effective this information is in improving patients knowledge and understanding of treatment and health outcomes. In order to undertake this review, databases was searched for research on (a) the role and value and (b) the effectiveness of written patient information for individual medicines. The role and value studies were defined as those examining the use and usefulness of written medicines information. Effectiveness trials were defined as RCTs which examined how well written medicines information works. From over 50,000 hits, 413 studies were considered. Of these, 64 papers reporting 70 studies were included: 36 papers reporting 43 RCTs in the effectiveness strand and 28 in the role and value strand. Qualitative Reviews There are many methods available for synthesising qualitative data in order to produce a review on a particular research question or topic. In order to understand or undertake qualitative reviews it is essential that you have a good understanding of qualitative research methods and methods of analysis (both of which will be covered in future series of these Pharmacy Practice Research & Evaluation Resources). In essence, qualitative reviews or qualitative synthesis is sometimes defined as a process resulting in a 'whole' which is more than the sum of its parts. However, the methods vary in the extent to which they attempt to 'go beyond' the primary studies and transform the data. Some methods textual narrative synthesis, ecological triangulation and framework synthesis focus on describing and summarising their primary data (often in a highly structured and detailed way) and translating the studies into one another. Others metaethnography, grounded theory, thematic synthesis, meta-study, metanarrative and critical interpretive synthesis seek to push beyond the original data to a fresh interpretation of the phenomena under review 9. Scoping reviews Scoping reviews aim to map the key concepts underpinning a research area and the main sources and types of evidence available. Scoping reviews provide wide coverage of the available literature but can vary enormously in the degree to which they extract, analyse and re-present the available evidence. The extent to which the review provides an analysis of evidence 9 Barnett-Page, E & Thomas, J. (2009), Methods for the synthesis of qualitative research: a critical review, Royal Pharmaceutical Society 2011 Page 6 of 10

7 depends on the purpose of the review and Arksey & O Malley have identified four common reasons why a scoping review might be undertaken 10 : 1. To examine the extent, range and nature of research activity This type of rapid review might not describe research findings in any detail but is a useful way of mapping fields of study where it is difficult to visualise the range of material in existence. 2. To determine the value of undertaking a full systematic review In these cases a preliminary mapping of the literature might be undertaken to identify whether or not a full systematic review is feasible (does any literature exist?) or relevant (have systematic reviews already been conducted?) and the potential costs of conducting a full systematic review. 3. To summarise and disseminate research findings This kind of scoping study might describe in more detail the findings and range of research in particular areas of study, thereby providing a mechanism for summarising and disseminating research findings to practitioners, policy makers and consumers who might otherwise lack the time or resources to undertake such work themselves (Antman et al. 1992). 4. To identify research gaps in the existing literature This type of scoping study takes the process of dissemination one step further by drawing conclusions from existing literature regarding the overall state of research activity. Specifically designed to identify gaps in the evidence base where no research has been conducted, the study may also summarise and disseminate research findings as well as identify the relevance of a full systematic review in specific areas of interest. As with any review, the first stage is to devise a well thought-out search strategy which includes key terms to be used and databases to be searched. In contrast to systematic reviews, the final parameters of the search need not be finalised prior to beginning the review. Like realist reviews it is often more relevant to maintain a wide approach in order to generate breadth of coverage. Decisions about how to set parameters can be made once some sense of the volume and general scope of the field has been gained. 11 As has been mentioned throughout these resources, using existing knowledge and networks can prove extremely fruitful and this is equally true in generating information about primary research. Likewise, contacting relevant national or local organisations working in the field is useful in identifying unpublished work. In their example of conducting a scoping review whose research question was: What is known from the existing literature about the effectiveness and cost-effectiveness of services to support carers of people 10 Arksey, H. & O Malley, L. (2005), Scoping studies: towards a methodological framework, 11 Arksey, H. & O Malley, L. (2005), Scoping studies: towards a methodological framework, Royal Pharmaceutical Society 2011 Page 7 of 10

8 with mental health problems?, the reviewers contacted a number of relevant organisations including Carers UK, the Sainsbury Centre for Mental Health, the Mental Health Foundation, the King s Fund, and the National Schizophrenia Fellowship 12. Once a review is underway, familiarity with the available literature will enable the development of inclusion and exclusion criteria to determine which potential articles are actually relevant to addressing the review question. As with other types of review, data from primary sources then needs to be extracted and organised in some way in order to undertake analysis and / or present the review s findings. How data is organised and / or analysed will be dependant on the purpose of the scoping review and the types of evidence retrieved. For example, you might simply be providing a map of the existing literature in order to determine what is out there and whether there are gaps in the evidence base. If your aim is to summarise findings, techniques more commonly used to interpret qualitative data might be used. This usually involves sifting, charting and sorting material according to key issues and themes 13. Identified themes can include information about the processes involved in the intervention or the context in which the intervention was used. Although useful in many respects, scoping reviews do have their limitations and these need to be borne in mind. The quantity of data generated during a scoping review can be considerable. This can lead to difficult decisions regarding the compromise between breadth (covering all available material) and depth (providing a detailed analysis and appraisal of a smaller number of studies). In addition, scoping reviews do not quality assess primary studies in any formal sense and consequently, they provide a narrative or descriptive account of available research 14. Unlike systematic reviews, scoping studies are not appropriate to answer a clinical question. The key components of a scoping review are: to map key concepts to map main sources and types of evidence to identify gaps in the evidence base to determine the value of undertaking a systematic review focus on providing wide coverage need to compromise between breadth and depth Rapid Evidence Reviews 12 As above 13 Arksey, H. & O Malley, L. (2005), Scoping studies: towards a methodological framework, 14 As above Royal Pharmaceutical Society 2011 Page 8 of 10

9 Rapid evidence reviews are used to summarise the available research evidence within the constraints of a given timetable, typically three months or less. Rapid evidence assessments differ from full systematic reviews in terms of time constraints. Consequently there are limitations on the extent of the searches and other review activities. Whilst attempting to be as comprehensive as possible, rapid evidence assessments usually make compromises to meet their tight deadlines; therefore they may fail to identify potentially relevant studies. They are useful to policy-makers who need to make decisions quickly, but should be viewed as provisional appraisals, rather than full systematic reviews 15. Conclusion Different reviews employ different methods and these are dependent upon the question which the review seeks to address. In order to assess the strengths and weaknesses of any review, it is essential to have a good understanding of both review methods and research methods. This will enable you determine whether or not the ones used were appropriate and what the limitations of the review are. It is important never to accept the findings of either primary or secondary research at face value. For example, not everything that is called a systematic review has used the necessary methods that would enable it to be so titled. The findings of the evidence are therefore unlikely to be as robust as those which would result form a properly conducted systematic review. Review authors should clearly state how their review was undertaken so that you can judge its value and determine whether or not it is applicable to the context in which you want to apply its findings. In addition, all reviews should provide sufficient information about the primary sources included to enable you to determine what type of evidence they were reviewing and what kinds of inferences can be made form these sources. Reviews are a valuable source of information for busy healthcare professionals but it is essential that they are subjected to the same critical appraisal as primary research. Useful Links Notting Systematic Review Course National Collaborating Centre for Methods and Tools Introduction to evidence-informed decision making online training 15 Centre for Reviews and Dissemination, 2009, Systematic Reviews: CRD s guidance for undertaking reviews in health care, Royal Pharmaceutical Society 2011 Page 9 of 10

10 Further Reading Trisha Greenhalgh, How to Read a paper 1997 BMJ Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005), Realist Review a new method of systematic review designed for complex policy interventions, Essay.pdf Royal Pharmaceutical Society 2011 Page 10 of 10