The Cochrane Public Health Group

Transcription

1 The Cochrane Public Health Group Guide for developing a Cochrane protocol This Guide has been produced by the Cochrane Public Health Group (CPHG) to make the process of preparing a protocol for a public health review in Cochrane format as clear as possible. The editorial base has developed guidelines that detail each section of the protocol and what should be included. These guidelines are a distillation of the Cochrane Handbook for Systematic Reviews of Interventions, review group policies, recommendations from the CPHG Editorial Team and also incorporate the Methodological Standards for the Conduct of Cochrane Intervention Reviews, released in November Please refer to The Cochrane Collaboration Training website at to access online training modules for Cochrane review authors.

2 Table of contents TABLE OF CONTENTS GETTING STARTED FILLING OUT THE PROTOCOL TEMPLATE BACKGROUND Description of the condition Description of the intervention How the intervention might work Why it is important to do this review OBJECTIVES Resources METHODS Criteria for considering studies for this review Types of studies Types of participants Types of interventions Types of outcome measures SEARCH METHODS DATA COLLECTION AND ANALYSIS Selection of studies Data extraction and management Assessment of risk of bias in included studies Measures of treatment effect Unit of analysis issues Dealing with missing data Assessment of heterogeneity Assessment of reporting biases Data synthesis Subgroup analysis and investigation of heterogeneity Sensitivity analysis ACKNOWLEDGEMENTS CONTRIBUTIONS OF AUTHORS POTENTIAL CONFLICT OF INTEREST REFERENCES - OTHER REFERENCES Additional references Other published versions of the review REFERENCES APPENDIX 1 GUIDANCE FOR ESTABLISHING A REVIEW ADVISORY GROUP APPENDIX 2 - PUBLIC HEALTH GROUP EDITORIAL PROCESS APPENDIX 3 RISK OF BIAS ASSESSMENT APPENDIX 4 DATA EXTRACTION AND ASSESSMENT FORM APPENDIX 5 CHECKLIST FOR SUBMISSION Last updated: 24 November

3 Getting started Now that your title has been registered there are several resources you will need to help you to complete your protocol. These are listed below and we have provided relevant links throughout these documents. All are available at and If you have problems downloading these files please contact the Review Group Coordinator. You should also have received an notifying you that a user account has now been set up for you in Archie. Once you have activated this account you will be able to check out your review. 1) Cochrane Handbook for Systematic Reviews of Interventions The Cochrane Handbook explains the Review process (parts have been summarised in this document but you will need to refer to The Cochrane Handbook for more complete guidance) (Higgins and Green 2008). The handbook is incorporated in the RevMan 5.1 software and also available at This document includes a chapter specific to public health reviews (Chapter 21). 2) RevMan 5.1 This is the program which you must use to write your protocol and later your review. It is free software that you can download easily to your computer. A tutorial is accessible from the help menu in RevMan. ALL AUTHORS ARE ASKED TO WORK THROUGH THE REVMAN TUTORIAL BEFORE WORKING ON YOUR PROTOCOL. 3) RevMan user guide This guide can be accessed as a PDF from the help menu in RevMan and provides a detailed description about how to use RevMan. You should continue to refer to this user guide throughout the review production process. 4) Cochrane style guide This guide explains the style conventions for a Cochrane review. 5) Other resources There are a number of other resources available to support you. Many are listed on the Cochrane Public Health Group website ( 6) Advisory group The CPHG recommends that all review teams establish a Review Advisory Group (RAG) to advise them on the scope of their review and its relevance to end-users. One author, usually the lead author, should take responsibility for establishing the RAG prior to commencing work on the protocol. The CPHG can provide you with assistance in identifying relevant RAG members but ultimately it is the author team s responsibility. Appendix 1 outlines guidance for establishing and working with a RAG Last updated: 24 November

4 Filling out the protocol template This document provides advice on what to include in each of the sections presented to review authors in a RevMan protocol template. The headings described below will align with the set headings of the protocol template. The advice reflects CPHG editorial policy and following these guiding principles should ensure fewer modifications are requested at editorial revision stage. See Appendix 2 for an overview of the stages each review goes through prior to publication. You should also refer to Chapter 4 of the Cochrane Handbook as you complete the protocol. This document does not repeat information in the Cochrane Handbook but rather provides supplementary material specific to CPHG requirements. Background Description of the condition This section should describe the condition or issue that the intervention/s under review are aiming to address, including information on the historical, (and perhaps political), social, economic, geographical and biological perspectives of the problem or issue so as to set a context for the review. This will help to establish the rationale for the review and explain the importance of the questions being asked. It may be appropriate to change this heading to Description of the Issue. Description of the intervention Define all terms and interventions clearly and try to set a tone that does not pre-judge the value of the intervention (i.e. the likely effectiveness of the intervention/s). Provision of examples of interventions and their components here will help the reader gain a better understanding of the interventions the authors refer to thereafter in the Types of Interventions of the Methods section. The complexity of the intervention in seeking to address the problem or issue (especially if it is delivered in many different contexts, using different methods and tools or includes many interventions to meet the desired outcome/s) needs to be acknowledged, even if the review is only looking at a component of the problem or issue. Areas of uncertainty about the intervention and issues that may be controversial or the subject of public concern should be highlighted. While this section may require you to cover technicalities of the intervention, it is important to write this clearly and in plain language to aid reader comprehension. We recommend including definitions of key concepts in this section. How the intervention might work In this section identify the theoretical underpinnings and refer to literature that identifies a potential pathway of effect between intervention and outcomes. You might like to include a logic model here which shows the connections between determinants of health, interventions and outcomes. Logic models can also help to identify the interventions of interest and important outcomes, or provide a logical rationale for why only a component of an intervention is being reviewed (and point to where other reviews may need to be carried out to complete the evidence picture). The following diagram from the published protocol for Community-based Last updated: 24 November

5 interventions for enhancing access to and/or consumption of fruits and vegetable access among 5-18 year olds (Ganann et al 2010) is an example of how a logic model might be constructed. Why it is important to do this review Clearly describe the justification for doing this review. Cochrane reviews should include a focus on exploring uncertainty about effects and/or uncertainty about effects within particular subgroups. You should include supporting references to existing and ongoing primary research and reviews (including non-cochrane reviews) of the research topic, to highlight what has been learned from past efforts as well as to point out any inconsistencies, methodological strengths and weaknesses, and evidence gaps that still remain. The contribution of your planned review should be emphasised by clearly stating the unresolved questions and controversies that will be addressed. To instruct the end-user on the potential application of review findings, include a brief statement on how this could inform practice or policy decisions. This section should be comprehensive but concise (1-2 paragraphs per subsection). Resources Cochrane Handbook - Chapters 4 and 5 The Cochrane Public Health Group Training Handbook ( see How to Ask an Answerable Question Last updated: 24 November

6 Objectives The objective/s of the review should be clear and specific with a precise statement that covers both intervention/s or exposure/s to be reviewed, the population of interest (e.g. whole population or particular sub-group/s) and the types of outcome measures of interest. One sentence may be enough; however, for some review questions it may be more appropriate to break it down into primary and secondary objectives. The intention to identify and report on adverse consequences or effects of the intervention/s in the review also should be included in this section. The objectives should be consistent with the review title. Resources Methods Cochrane Handbook Chapter 4 The purpose of the methods sections of a protocol is to describe how the review will be conducted and to theoretically allow for the review to be replicated by others. Please refer to the Cochrane-Campbell Methods Group Equity Checklist ( throughout the development of your protocol. This highlights methodological aspects to consider at both the protocol and the review stage to help you identify the impact of interventions on equity in your review. Criteria for considering studies for this review This section should make it clear how studies will be selected for inclusion in the analysis. All inclusion/exclusion criteria must be explicitly stated within this section, even if mentioned again elsewhere throughout your protocol. The protocol needs to reflect your intention to include information that will allow the review author, and hence the reader, to judge what factors may increase or decrease effectiveness, and under what circumstances/contexts. You do not need to provide any text under this heading. Rather complete each of the sections below. Review authors should note that if non-randomised studies are to be included detailed descriptions of your approach will be needed throughout your protocol. Suggestions are provided throughout this guidance document. Note that any changes to eligibility criteria or outcomes studies will need to be justified in the review. Types of studies In this section you need to list the study designs you will include in your review. Whilst you do not need to document your study selection in detail, study design/s chosen should be determined by, and appropriate for, the intervention and the research question. You should consider the risk of bias of each study design and value added to the review. Clear justification for choice of eligible study designs must be included. Randomised controlled trials should be included if they are feasible for the interventions and outcomes of interest. If an RCT is not able to be used as a study Last updated: 24 November

7 design due to ethical or other reasons the review needs to be explicit about which study designs are appropriate to include and why. It may be appropriate to include more than one study design, even if it is likely that RCTs are available, to increase external validity of the review findings. Studies should be included irrespective of their publication status, unless explicitly justified in your protocol. The CPHG accepts the following study designs as appropriate to the review question: RCTs, cluster RCTs, non-randomised controlled studies (including controlled before and after studies and interrupted time series studies (with three time points before and after the intervention). Whilst there are challenges associated with using study labels, authors are encouraged to refer to Box 13.1.a of the Cochrane Handbook to ensure they are clear about the types of study design descriptors they might find in the literature. You could consider including a list of definitions of your chosen study designs. The focus for eligibility must be on the features of a study s design, rather than the design labels used. You should state in this section if you will be collecting and extracting information from qualitative studies emanating from included intervention studies. Qualitative research studies can be used in the review to help contextualize the major findings, rather than provide causative understandings. The extent to which you search for and include qualitative studies will depend on the questions you wish to answer in your review. Types of participants Clearly describe the population for which the intervention under review is targeted at or likely to have an effect on, and clearly specify inclusion and exclusion criteria for your review. These may be based on individuals or communities. Examples of defining characteristics may be geographic (e.g. where they live or work), demographic (e.g. age, sex) and/or social factors (e.g. education level, at-risk groups (as specified by the authors). Any restrictions with respect to specific population characteristics, settings or factors needs to be justified and reflect information presented in the Background section. Many reviews published within the remit of the CPHG will have community as their participants. Reviews should provide a definition for community that is appropriate for the intervention. It is important to distinguish between community-wide and community-based interventions, if these terms are used in your review. Types of interventions List the intervention/s that will be included in the review, and specify clear inclusion and exclusion criteria (e.g. any essential components, minimum duration of the intervention, etc.). This section should not describe or define interventions - this detail should be included in the Background section (Description of the intervention). This section should therefore be brief and outline the types or groups of included interventions, as referred to in the Background section. Authors should also include a list of the interventions that will be excluded from the review. Include specific examples of relevant interventions/programs of both included and excluded studies when possible. If relevant to the inclusion criteria, identify any specific intervention/s your intervention/s of interest should be compared against (i.e. what the control group will be). Types of outcome measures The outcomes you plan to report for your review should be pre-specified in your review to avoid bias in reporting your findings. If possible, you should also define in advance the details of what will be considered acceptable outcome measures for your review and also how outcome Last updated: 24 November

8 measures will be selected when there are several possible measures (e.g. multiple definitions, assessors or scales). You should include all important outcomes in the protocol, even if you believe there will be insufficient data present in the existing research to include in the synthesis. As well as potential benefits, you must consider any potential adverse consequences of the intervention and ensure that these are addressed in your review. Public health questions often have a large number of outcomes of interest we encourage authors to think carefully about what the main outcomes of interest are likely to be for end-users. Once you have a complete list of the outcomes of interest, you should identify a short list of main outcomes. The Cochrane Handbook suggests no more than seven main outcomes should be chosen, however this should be examined on an individual review basis. These outcomes will be used to summarise the key findings of your completed review, for example in the abstract and Summary of Findings tables (if you include this in your review). For some review questions, it may be appropriate to summarise findings for groups of similar outcomes. From among the main outcomes, select a small number of primary outcomes (usually 3 or less). A primary outcome is of core interest to your review i.e. the outcome that best answers your effectiveness question. You should include at least one undesirable outcome (adverse effect), known or hypothesised, with a rationale for why you think these may arise and why they are important to include in the review. All other main outcomes become secondary outcomes, along with any other additional outcomes the review intends to address. These may include unintended outcomes that occur as a result of the intervention. These are considered helpful in explaining intervention effects or the integrity of the intervention. If you have a large number of secondary outcomes you should consider the most appropriate ways of categorising these. If you are planning to include a Summary of Findings table in your review (not compulsory but strongly recommended in The Cochrane Handbook), you should state the outcomes you will be prioritising for this here. For public health reviews, it may often be appropriate to include measures of changes in social and environmental determinants of population health, as opposed to or in addition to direct health outcomes. For example, changes in children s purchasing habits (as a result of new canteen policies in a school), and reduction in violent behaviour (as a result of adoption of responsible drinking policies at a sports club). You must clarify in advance whether any outcomes will be used as criteria for including studies (rather than as a list of the outcomes of interest within included studies). Outcome measure often do not necessarily form part of the criteria for including studies in a review, however some reviews do legitimately restrict eligibility to specific outcomes. Furthermore, if relevant to your topic, you may wish to specify requirements for the timing of outcome measurement, such as a minimum period after the intervention before which an effect would not be detectable (e.g. the effect of an intervention in pregnant women on the rate of babies born with low birth weight could not be observable for several months after the intervention). Resources Cochrane Handbook Chapters 4, 5, 13 and 21 Last updated: 24 November

9 Chapter 21 in Cochrane Handbook on Reviews in Health Promotion and Public Health ( The Cochrane Non-Randomised Studies Methods Group has provided information on how non-randomised studies should be dealt with in Cochrane reviews (see Chapter 13 in Cochrane Handbook ( Cochrane Consumers and Communication Group have useful information on study designs at : ) Refer to Cochrane Health Equity website for information on assessing issues relevant to equity, along with a checklist for authors that can be used and submitted along with the draft protocol ( Search methods Include a description of the search strategy to be used to retrieve studies. You should list the sources to be searched (e.g. reference databases, personal contacts, hand searches of journals). At a minimum you should search Medline, Embase and CENTRAL. Include a rationale for the choice of literature sources (e.g. ERIC for reviews of educational interventions) if possible and the years to be covered. It may be useful to refer to your logic model to justify your approach. Consider including databases relevant for health equity. Relevant trial/study registers and repositories of results must also be included in your search to reduce the risk of publication bias and identify ongoing studies. Reference lists of included studies and any relevant systematic reviews identified must also be searched and this should be noted in your protocol. You should also describe the mechanisms you intend to use or resources that you have available to enable you to retrieve potentially relevant documents, especially ones that are unpublished and/or written in languages other than English. For example, you may plan to contact relevant individuals and organisations for information about unpublished or ongoing studies. You should not apply inclusion restrictions based on publication status or language. The CPHG can assist authors in identifying potential options for translations if they are required. You should consider mechanisms for searching grey literature, for example using databases such as OpenSIGLE. If searching restrictions are necessary, a rationale must be provided. You should at least make a first attempt at a search strategy (ideally utilising an information specialist available to the review author team). The CPHG Trials Search Coordinator, Ruth Turley (TurleyRL@cardiff.ac.uk) will provide feedback once you have submitted your protocol for editorial review. Your protocol should include (as an appendix) a complete electronic search strategy for one database (usually Medline). Consider including terms or concepts in your search strategy that are relevant for health equity. Note that generally CPHG review authors will need to conduct their own searches. We therefore recommend that an information specialist be recruited to the author team. You should outline in your protocol who will conduct the searches. Resources Cochrane Handbook Chapters 6 and 21 Chapter 21 in Cochrane Handbook on Reviews in Health Promotion and Public Health ( Last updated: 24 November

10 Data collection and analysis Selection of studies This section should outline how results of the various searches will be assessed for inclusion, by whom and what you will do if more information is required to determine eligibility (e.g. contacting the authors for additional information or finding a process paper associated with the study which outlines further information). Specify how many review authors will independently review all abstracts and titles and how disputes regarding inclusion will be resolved (there must be at least two, and we recommend a third being available to resolve disputes). Specify that all irrelevant titles should be excluded and that full-text papers will be obtained where titles are deemed to be relevant or where eligibility is unclear. Again, state how many review authors will assess these full text papers and how disagreement will be resolved. You should note that you will keep a record of reasons for excluding studies. Documentation regarding inclusion decisions must be sufficient to complete a PRISMA flow chart and a table of Characteristics of excluded studies. If any statistics will be used to determine inter-rater agreement (e.g. using Cohen's kappa) include these details. Please indicate if articles in a language other than English will be translated. The reference management software you will be using to manage the records retrieved from searches of electronic databases should be named. The most commonly used software programs are Endnote and Reference Manager. It is common for included studies to refer to a process evaluation or other methodological detail that is published elsewhere in a separate paper. These additional papers must be obtained (when referred to in the primary paper) and considered as part of the included study. They are likely to contain important information needed to understand the implementation of the intervention and adequately assess the risk of bias of the study. Multiple reports of the same study must be collated, so that each study (rather than each paper) is the unit of interest in the review. Data extraction and management This section should provide an adequate description of methods used to collect data from included studies. You should mention how many people will conduct data extraction and how any disagreements will be resolved. It is highly desirable that that study characteristics are extracted in duplicate (i.e two review author s extract data and a third review author be consulted where there is disagreement), but not essential. It is essential that outcome data is extracted in duplicate. You also need to identify the data extraction form/methods you will be using. Appendix 4 provides a sample data extraction form that you can modify to meet the needs of your review. It is recommended that you pilot your data extraction form, to ensure that all participating authors are retrieving comparable results, and this should be noted in the protocol. Inclusion of the data extraction form that your author team has developed would be useful to include as additional information in a figure or additional table. In the text of the protocol you should include a brief description of the categories of data you intend to collect. In order to provide essential detail for decision makers, information about context, implementation factors, equity, cost and sustainability must be collected from included studies where available (in addition to data on effectiveness). As mentioned in the above section, included studies may refer to a process evaluation or other methodological detail that is Last updated: 24 November

11 published elsewhere. These additional papers must be obtained and relevant information extracted, as a minimum requirement. In this section of the protocol you should state that you will collect this information (i.e. context, implementation, cost, sustainability etc) and what you will do with it thereafter. i.e. at a minimum, this information should be reported in the Characteristics of included studies table. If you plan to use this information to help synthesise your findings (for example, grouping studies by measure of intensity or level of implementation), you should state this in your protocol (with the caveat that this would only be done if adequate data was available). You may also choose to conduct additional searches for contextual information, such as implementation factors, cost and sustainability, beyond that linked with included studies. Any plans to do this should be noted in your protocol. Within your data extraction form, you should identify studies that report on socio-demographic characteristics known to be important from an equity perspective. For this process, use the PROGRESS (Place, Race, Occupation, Gender, Religion, Education, Socioeconomic status, Social status) framework and, at a minimum, report for each included study which of the eight PROGRESS factors were reported for participants at baseline and which were reported at endpoint. In addition to the PROGRESS framework, also collect whether or not interventions included particular strategies to address diversity or disadvantage. We strongly recommend that you incorporate the Cochrane-Campbell Methods Group Equity Checklist in your data extraction form and you should note this in your protocol ( This may help you to identify the impact of interventions on equity later on in your analysis, if this is an important consideration for your review. In this section you should state that all potential moderators/confounders of study outcomes will be included in the extraction form (even if some of these characteristics are not expected to be formally tested or discussed in the final review). Consider potential confounders and adjustment processes. When extracting data, include this information. It can be helpful to refer to The Quality Assessment Tool for Quantitative Studies ( developed by The Effective Public Health Practice Project (EPHPP), to be used in conjunction with their Quality Assessment Tool for Quantitative Studies Dictionary ( to check for items that you may wish to consider including as part of your data extraction form. If you use items from this tool, you need to state this in your protocol. Document how you intend to handle instances in which a single study of effectiveness provides data on multiple measures of the same or similar outcomes and you may need to choose what to report (e.g. when several variations on an outcome are measured (e.g. weight and BMI) or when the same outcome is measured at multiple points in time). An explanation of the criteria used to determine whether multiple outcomes from the same or related studies are independent data points should be included. Authors of primary studies should be contacted where information is missing or clarification is needed. You should note this in the later section on Dealing with missing data. You should outline Last updated: 24 November

12 how you will attempt to avoid duplicate publication bias and state how multiple reports and if publications of the same study will be assembled and compared for duplication, completeness and possible contradictions. Please note in this section if you will use RevMan to manage data storage and analysis. If you intend to use alternative software you must discuss this with the CPHG, and note your intention in the protocol. Resources Cochrane Handbook Chapter 7 Chapter 21 in Cochrane Handbook on Reviews in Health Promotion and Public Health ( Refer to Cochrane Health Equity website for information on assessing issues relevant to equity, along with a checklist for authors that can be used and submitted along with the draft protocol ( Assessment of risk of bias in included studies In this section, you should provide an adequate description of the tool(s) you will use to assess the risk of bias of included studies. You should also describe how the tool(s) will be implemented and the criteria used to assign studies, for example, to judgements of low risk, high risk and unclear risk of bias. The risk of bias assessment must be conducted in duplicate with a clear process for resolving disagreements. Supporting information to justify all risk of bias judgements must be included in the risk of bias tables. You can consider including the source of the information, for example, direct quotes from the study paper. If you are only including randomised controlled trials, we recommend that you use the Cochrane Collaboration s Risk of Bias (RoB) tool. This includes selection bias, performance bias, attrition bias, detection bias and reporting bias. With regard to the assessment of blinding, we recommend that you consider separately the risk of bias due to lack of blinding for (i) participants and study personnel (performance bias) and (ii) outcome assessment (detection bias). It is also often appropriate to consider the risk of bias due to lack of blinding separately for different types of outcomes. When assessing attrition bias, it is recommended to consider the impact of missing data separately for different outcomes. If you are including non-randomised studies we recommend you use the Effective Practice and Organisation of Care (EPOC) RoB Tool for studies with a separate control group instead. This can be used for randomised controlled trials as well as controlled before and after studies and other nonrandomised designs that include a control group (with the exception of interrupted time series studies). This tool includes the standard Cochrane RoB tool items as well as an additional item to consider the likelihood of contamination. Importantly for nonrandomised studies, it also includes additional items to assess the risk of selection bias and subsequent confounding ( were baseline outcome measurements similar? and were baseline characteristics similar? ). We recommend supplementing this with another additional item, did the study authors appropriately adjust for important confounders in their analysis?. Last updated: 24 November

13 If you are including ITS studies we recommend that you assess the risk of bias of these studies using the EPOC RoB tool for ITS study designs which includes four items from the Cochrane 'Risk of bias' tool to assess performance, attrition, detection and reporting bias as well as the following additional items relevant for ITS studies: was the intervention independent of other changes?, was the shape of the intervention effect pre-specified? and was the intervention unlikely to affect data collection?. Both of the EPOC RoB tools mentioned above can be found on the EPOC website: f%20bias%20criteria%20for%20epoc%20reviews.doc as well as guidance for how to prepare a RoB table: %20a%20risk%20of%20bias%20table%20for%20reviews%20that%20include%20more%20than%2 0one%20study%20design.doc. Note that for some items in the RoB tool, for example blinding, it may make sense to provide a different assessment for different outcomes (for example outcome assessors may be blinded for some outcome measures and not others, or some measures may be more objective/subjective than others). Where appropriate, add additional items into the RoB table to allow for this. Alternative approaches should be discussed and agreed with your contact editor. See Appendix 3 for general information on Risk of bias tables (referenced from the Cochrane Handbook). This is an area of methodological development and any new tools will be available on the CPHG website. In this section you also need to describe the method by which you will summarise the risk of bias assessments. Rather than at the study level, it is recommended that you do this at the outcome level. This is due to the fact that the risk of bias may be different for different outcomes within the same study. To do this you can provide an overall risk of bias assessment for the main outcomes within each study, then provide an overall risk of bias assessment for relevant outcomes across studies, so that outcomes will be judged overall as Low, Medium or High risk of bias given overall considerations of the study designs, and the potential impact of the identified risks noted in the table for each study that contributed results for that outcome. Also, consider how you will address risk of bias in the synthesis of your results, considering how potential study biases might affect your conclusions. Measures of treatment effect In this section you should state how outcomes will be reported (e.g. dichotomous data) and how you will analyse and compare them (e.g. using Risk Ratios). It is likely that a number of quantitative outcome measures may be identified. You need to firstly identity the types of data you will obtain from your included study designs. For controlled studies, with continuous data, we recommend reporting means or changes in mean scores. Weighted mean difference can also be reported for continuous outcomes. Standardised mean differences should be reported when different studies use different scales to report the same outcome (e.g. quality of life scales). Where a number of outcome measures are identified, authors should use the ratio of means method (Friedrich 2008). Dichotomous (or binary) outcomes can expressed as relative risks (RR), odds ratio (OR) or risk difference (RD), however the CPHG recommends using RR (Deeks 2002). We also recommend using RR for categorical data (e.g. outcomes reported on a short Likert Last updated: 24 November

14 scale). Unit of analysis issues It can be difficult to identify the unit of analysis issues that may emerge from your included studies. However, it is important to consider and document in this section what these might be, based on your included study designs. Special issues in the analysis of studies with non-standard designs, such as cross-over trials and cluster-randomised trials, should be described here. Where these studies are not analysed appropriately this may result in unit of analysis error, for example where cluster-randomised trials are analysed at the individual participant level without taking into account the effect of clustering. Effects can be recalculated using the approximately correct analysis or by inflating standard errors which is equivalent to calculating new sample sizes. If you plan to include these studies you should consult a statistician to identify how unit of analysis errors will be treated, and note this in the protocol. If you find studies with multiple intervention groups, you will need to consider and document how you will deal with multiple groups in your analysis. Report outcomes for all groups of interest in your review, however only include groups that meet the eligibility criteria. If you identify more than one intervention group of interest, you may need to divide your analysis into pair-wise comparisons (e.g. Group A vs control, Group B vs control, Group A vs Group B) and conduct a meta-analysis for each comparison, if appropriate to do so. Be careful not to include a group of participants twice in the same meta-analysis. One way to include two pair-wise comparisons against the same control group in one meta-analysis is to simply halve the number of participants in the control group. Document your intentions regarding these issues in this section. Resources Cochrane Handbook - Chapter 7 and 9 (data extraction and methods of analysis by type of outcome) Chapter 21 in Cochrane Handbook on Reviews in Health Promotion and Public Health ( Non-standard designs are discussed in detail in Chapters 9 and 16 of the Cochrane Handbook for Systematic Reviews of Interventions, including cluster-randomised trials (Section 16.3), cross-over trials (Section 16.4), and studies with multiple intervention groups (Section 16.5) Dealing with missing data Strategies for dealing with missing data should be described in this section. This will principally include missing information on the methods used in included studies, missing participants due to drop-out (and whether an intention-to-treat analysis will be conducted), and missing statistics (such as standard deviations or correlation coefficients or where data is reported at baseline but is not reported at outcome). It may be necessary for you to contact authors where data are missing. Your strategy for contacting authors (e.g. using addresses on the study s publication or accessing phone directories from author s documented affiliated organisation) should be outlined in the protocol. Section of the Cochrane Handbook outlines the options available to review authors if missing data is not found. Refer to Section of The Cochrane Last updated: 24 November

15 Handbook which outlines the options available to review authors if missing data is not found and state in your protocol which option you will use. The option chosen will depend on what is practical and whether or not it can be assumed that the data is missing at random. Resources Cochrane Handbook - Chapter 16 (Sections 16.1 and 16.2) Assessment of heterogeneity This section should outline how you plan to assess the heterogeneity (or differences) between your included studies. It is helpful to consider and refer in this section to methodological heterogeneity (e.g. how similar or different your included studies are in terms of study design, types of participants, interventions and outcomes), as well as statistical heterogeneity (i.e variability in the intervention effects being evaluated in the different studies) which is a consequence of methodological heterogeneity. If you plan to conduct a meta-analysis you should consider and document how you will assess statistical heterogeneity. You should state that this analysis will be used to determine whether it is suitable to conduct a meta-analysis or to analyse your studies qualitatively. The CPHG recommend that you should use the I 2 statistic to quantify the level of heterogeneity present. Review authors may also consider using the Chi square test for heterogeneity (p<0.10). RevMan will calculate these statistics for you. If there is a high level of heterogeneity between studies, it may be inappropriate to conduct a meta-analysis. Meta-analysis should only be considered when a group of studies is sufficiently similar in terms of participants, interventions and outcomes to provide a meaningful summary. If the studies are comparable in these respects, but there remains a very high level of heterogeneity, you may decide not to conduct a meta-analysis, and to present your results in a qualitative analysis. If you plan to use a heterogeneity threshold above which you will not conduct a meta-analysis, you should note this in the protocol. A rough guide to interpretation of the I 2 statistic is given in the Cochrane Handbook, however note that thresholds for the interpretation of I 2 can be misleading, due to the range of factors that may contribute to heterogeneity. For this reason, statistical assessment of heterogeneity is not a substitute method for exploring the causes of variation between your studies. You should outline the statistical methods you will use for conducting subgroup analyses and investigation of heterogeneity in the section below. Resources Cochrane Handbook Chapter 9 Assessment of reporting biases Assessment of reporting bias is often difficult in public health reviews where the numbers of studies can be small. If you have a small number of studies (<10) you should follow methods described in the Cochrane Handbook. If, however, a larger number of studies (>10) are available for review, it may be useful to investigate reporting bias using a funnel plot. You should identify both approaches in your protocol as you will not know how many included studies you will find at protocol stage. Resources Last updated: 24 November

16 Cochrane Handbook Chapter 10 Data synthesis The protocol should outline the procedures you intend to use to analyse and summarise the study results, including whether or not you intend to carry out meta-analyses. This section should contain a clear rationale for any choices, considering the potential impact of each choice on the outcomes of the review. Meta-analyses should only be undertaken if participants, interventions and comparisons are judged to be sufficiently similar to ensure an answer that is meaningful. It may be useful to organise this section by study type and/or by type of outcome data (e.g. continuous or binary). More specifically, if the intention is to carry out a quantitative analysis of results, you should outline in the protocol: the software package that will be used to conduct the analyses (this should be RevMan and alternatives should be discussed with the CPHG); how statistics describing the overall literature will be presented; why a particular effect size metric is to be used; if adjustments to effect sizes will be made for any reason; techniques to be used to combine results of separate tests; techniques to be used to assess and then analyse variability in findings across tests. As a default option, the CPHG recommend using the random-effects model to incorporate heterogeneity into your meta-analyses (as opposed to the fixed-effect model). The randomeffects model allows for a greater level of natural heterogeneity between studies, assuming that each study is estimating a unique true effect applicable to the time, population and context in which the study was conducted. Fixed effects meta-analysis may be included in subsequent sensitivity analysis or when significant justification is provided in the protocol. For qualitative syntheses (with or without a meta analysis), you need to provide a rationale for how you intend to organise studies/findings to arrive at reasonable conclusions and present information in a useful way for end-users of the review. For example, you may like to synthesise studies grouped by the type of intervention, the length of the intervention, the type of outcome, or by study design. It may be difficult to identify what is most appropriate at protocol stage but we encourage you to report your anticipated approach or to identify the options for analysis and how the decision on synthesis structure will be reached. You must include information about how you will summarise the findings of the review. Use the five GRADE considerations (study limitations, consistency of effect, imprecision, indirectness and publication bias) to assess the quality of the body of evidence for each outcome, and to draw conclusions about the quality of the body of evidence within the text of the review. Note that this does not mean you have to provide a GRADE rating for the quality of the body of evidence (high, moderate, low, very low), however you must consider the five factors mentioned above in your review. Be sure to justify and document all assessments so that they can be included in your review. These factors must be addressed irrespective of whether your review will include a Summary of Findings Table (SoFT). If you plan to include a formal SoFT in your review, you should include this information in this section of your protocol. It is good practice to list the Last updated: 24 November

17 outcomes (up to seven) that you plan to include within the SoFT, as well as which comparisons and subgroups will be covered, if appropriate. A SoFT should include the number of participants and studies for each outcome, summarise the intervention effect and include a measure of the quality of the body of evidence addressing the considerations listed above. Subgroup analysis and investigation of heterogeneity In this section you should list your proposed sub-group analyses, selecting key factors that may differ between or within your included studies, and that you anticipate will be associated with differences in the effect observed, such as differences in the population, intervention or context of the studies. These should be restricted in number with a rationale provided for each. Issues of equity may be a key consideration. Include information about any formal statistical tests that will be used to compare subgroups, in the event that there are sufficient studies for this to be meaningful. It may be possible to undertake a meta-regression. Please note if this is an option for your review. Resources Cochrane Handbook Chapter 9 Campbell and Cochrane Equity Methods Group Equity Checklist: Sensitivity analysis You should document in this section under what circumstances a sensitivity analysis will be undertaken. Sensitivity analysis is often confused with subgroup analysis although they are different. Sensitivity analysis is undertaken to identify whether review findings are dependent on the decisions made during the review process, such as about study inclusion/exclusion, the selection of data to analyse, the analysis methods used, imputed data, and the impact of studies at high risk of bias. Some examples of decisions that may indicate a need for sensitivity analysis are listed in The Cochrane Handbook (Chapter 9, section 9.7). Where it is known in advance that decisions have been made that may affect the results of the review, sensitivity analysis should be planned and noted in the protocol. During the course of the review, other study characteristics may also be identified for inclusion in the sensitivity analysis. This is acceptable and should be stated as a possibility in the protocol. Resources Cochrane Handbook Chapter 9 Acknowledgements Acknowledge any individuals or organisations who may have made a sufficient contribution to developing the protocol, (e.g. secretarial support, protocol referees, review advisory group members) but are not included in the Contributions of Authors section. Please ensure you have obtained permission to name these people in the protocol, otherwise a general acknowledgment based on roles ( our advisory group members, external peer referees of the protocol ). It is not necessary to acknowledge the support of specific people within the CPHG team who have helped Last updated: 24 November

18 you in developing the protocol. A broad acknowledgement of the CPHG team is quite adequate if you wish to acknowledge the group. Contributions of authors Identify who has or will perform each of the following tasks (authors initials only): Draft the protocol Study selection Extract data from studies Enter data into RevMan Carry out the analysis Interpret the analysis Draft the final review Disagreement resolution Update the review Potential conflict of interest You should report any conflict of interest of any of the authors, which may influence their judgments in conducting the review, at protocol stage (refer to the summary of the Collaboration s policy on conflicts of interest in Chapter 2 (Section 2.6) of the Cochrane Handbook for Systematic Reviews of Interventions for more information). This will include any present or past affiliations or other involvement in any organisation or entity with an interest in the review that might lead to a real or perceived conflict of interest. Areas of uncertainty should always be discussed with the CPHG. Situations that might be perceived by others as being capable of influencing a review author s judgements include personal, political, academic and other possible conflicts, as well as financial conflicts. Authors must state if they have been involved in a study that may be included in the review. This section should reflect the information contained in authors Declarations of interest statement within the Disclosure of Potential Conflicts of Interest Forms. You can access these forms from within your Revman file by clicking File -> Reports -> Declaration of Interest (you will receive a reminder from the editorial office to fill out electronically during the course of completing your protocol if you have not done so already). If there are no known conflicts of interest, this should be stated explicitly, for example, by writing None known. References - Other references Additional references References cited in the text should be listed here. Other reference categories (such as Included Studies) are generally not used in a protocol, but will be used in the completed review. References are referred to throughout the review using a Reference ID, usually comprising the first author s surname and year of publication (e.g. Smith 2001), and all references must be linked Last updated: 24 November