National Collaborating Centre for Healthy Public Policy METHOD FOR SYNTHESIZING KNOWLEDGE ABOUT PUBLIC POLICIES

Transcription

1 National Collaborating Centre for Healthy Public Policy METHOD FOR SYNTHESIZING KNOWLEDGE ABOUT PUBLIC POLICIES REPORT SEPTEMBER 2010

2

3 National Collaborating Centre for Healthy Public Policy METHOD FOR SYNTHESIZING KNOWLEDGE ABOUT PUBLIC POLICIES REPORT SEPTEMBER 2010

4 AUTHORS Florence Morestin National Collaborating Centre for Healthy Public Policy François-Pierre Gauvin National Collaborating Centre for Healthy Public Policy Marie-Christine Hogue National Collaborating Centre for Healthy Public Policy François Benoit National Collaborating Centre for Healthy Public Policy LAYOUT Madalina Burtan National Collaborating Centre for Healthy Public Policy ACKNOWLEDGEMENTS The authors would like to thank: our colleagues at the NCCHPP, the Public Policy Team at the, Valéry Ridde, John Church and Donna Ciliska for their comments on earlier versions of this document. Production of this document has been made possible through a financial contribution from the Public Health Agency of Canada through funding for the National Collaborating Centre for Healthy Public Policy (NCCHPP). The National Collaborating Centre for Healthy Public Policy is hosted by the Institut national de santé publique du Québec (INSPQ), a leading centre in public health in Canada. The views expressed herein do not necessarily represent the views of the Public Health Agency of Canada. This document is available in its entirety in electronic format (PDF) on the Institut national de santé publique du Québec website at: and on the National Collaborating Centre for Healthy Public Policy website at: La version française est disponible sur les sites Web du Centre de collaboration nationale sur les politiques publiques et la santé (CCNPPS) au et de l au Reproductions for private study or research purposes are authorized by virtue of Article 29 of the Copyright Act. Any other use must be authorized by the Government of Québec, which holds the exclusive intellectual property rights for this document. Authorization may be obtained by submitting a request to the central clearing house of the Service de la gestion des droits d auteur of Les Publications du Québec, using the online form at or by sending an to droit.auteur@cspq.gouv.qc.ca. Information contained in the document may be cited provided that the source is mentioned. LEGAL DEPOSIT 2 nd QUARTER 2011 BIBLIOTHÈQUE ET ARCHIVES NATIONALES DU QUÉBEC LIBRARY AND ARCHIVES CANADA ISBN: (FRENCH PRINTED VERSION) ISBN: (FRENCH PDF) ISBN: (PRINTED VERSION) ISBN: (PDF) Gouvernement du Québec (2011)

5 ABOUT THE NATIONAL COLLABORATING CENTRE FOR HEALTHY PUBLIC POLICY The National Collaborating Centre for Healthy Public Policy (NCCHPP) seeks to increase the expertise of public health actors across Canada in healthy public policy through the development, sharing and use of knowledge. The NCCHPP is one of six Centres financed by the Public Health Agency of Canada. The six Centres form a network across Canada, each hosted by a different institution and each focusing on a specific topic linked to public health. In addition to the Centres individual contributions, the network of Collaborating Centres provides focal points for the exchange and common production of knowledge relating to these topics. National Collaborating Centre for Healthy Public Policy I

6

7 TABLE OF CONTENTS LIST OF TABLES AND FIGURES... V INTRODUCTION FRAMEWORK FOR ANALYZING PUBLIC POLICIES Dimensions Related to the Effects of Public Policies Effectiveness Unintended Effects Equity Dimensions Related to the Implementation of Public Policies Cost Feasibility Acceptability Relationships Between the Six Dimensions for Analysis TYPES AND SOURCES OF DATA TO BE CONSIDERED KNOWLEDGE SYNTHESIS METHOD Inventory of Policies and Selection of the Subject of the Knowledge Synthesis Exploration of the Grey Literature Survey of the Scientific Literature Selection of the Subject of the Knowledge Synthesis Explication of the Intervention Logic of the Public Policy Being Studied Synthesis of Data Drawn From the Literature Documentary Search Appraisal of the Quality of Data Data Perusal and Extraction Synthesis of Data Drawn From the Literature Enrichment and Contextualization of Data / Deliberative Processes Synthesis Integration of the Different Types of Knowledge Gathered BIBLIOGRAPHY APPENDIX 1 APPENDIX 2 LIST OF SELECTED WEBSITES OF RELEVANCE TO PUBLIC POLICY AND HEALTH LIST OF DATABASES PROPOSED BY THE COCHRANE HEALTH PROMOTION AND PUBLIC HEALTH FIELD National Collaborating Centre for Healthy Public Policy III

8

9 LIST OF TABLES AND FIGURES Table 1 Dimensions for analyzing public policies... 4 Table 2 Summary table: dimensions and criteria for analyzing public policies Table 3 Sample data extraction table Figure 1 Generic logic model... 5 Figure 2 Relationships between the six dimensions for analyzing public policies Figure 3 Steps in the knowledge synthesis process Figure 4 Logic model (nutrition labelling) National Collaborating Centre for Healthy Public Policy V

10

11 INTRODUCTION Intended readership This document is intended for public health actors who, given their role as health promoters, are interested in how public policy can act as a lever for action that affects population health and its determinants. More specifically, this document seeks to meet the needs of public health actors acting as expert advisors to decision makers during the promotion, adoption, and implementation of public policies. Fulfilling this role requires them to synthesize knowledge about how effective public policies are at promoting health (including how effective they are in a given context) and about the issues surrounding the implementation of these policies. Objective The objective of this document is to propose a knowledge synthesis method that is applicable to public policies and takes into account not only data linked to their effectiveness, but also data on issues related to their implementation, with the aim of identifying the policies that are most likely to succeed in the specific context in which their implementation is being considered. Overview The proposed knowledge synthesis method draws inspiration from political science, policy analysis, literature on evidence-informed decision making in public health, literature on evaluation, and theoretical developments related to deliberative processes. Having integrated these various foundational elements, the proposed methodological approach: applies an analytical framework that takes into consideration not only the effectiveness of public policies, but also their unintended effects, their effects on equity, and the issues related to their implementation (cost, feasibility, and acceptability); and considers a range of quantitative and qualitative data from scientific and non-scientific sources. Our knowledge synthesis method includes four steps. The first involves compiling an inventory of public policies that could address the targeted health problem, and choosing the policy on which the knowledge synthesis will focus. The second step is devoted to making explicit the intervention logic (logic model), that is, the sequence of effects expected to link the policy under study to the targeted problem. The third step, carried out through means of a literature review, involves synthesizing data on the effects of this policy in contexts in which it has already been implemented (effectiveness, unintended effects, effects related to equity) and on the issues related to its implementation (cost, feasibility, acceptability). Finally, the fourth step aims at enriching and contextualizing the data drawn from the literature, through deliberative processes that bring together actors concerned by the targeted health problem and working within the context in which implementation of the policy National Collaborating Centre for Healthy Public Policy 1

12 is being considered. The aim of the deliberative processes is to have these actors discuss the data drawn from the literature, enrich analysis of the data with their own knowledge, and assess the extent to which the data apply to their own context. To illustrate the use of this method and to verify its relevance, our team tested it by applying it to a public policy option aimed at addressing obesity. Because of this case study, some of the methodological references used refer to obesity; nevertheless, they are equally applicable to public policies concerned with other issues. Note: because the production of knowledge syntheses on public policies is still a relatively uncharted area of endeavour and due to the complexity of the subject, our approach should be considered indicative rather than prescriptive. Thus, persons using our method may choose to apply only certain steps and/or to adapt other steps as they see fit, based on the issues at play in their own contexts. Indeed, each step, taken separately, constitutes an interesting tool. This variability does not imply the sacrifice of methodological rigour: it is important to be as explicit and as transparent as possible about the methodological decisions one makes, so that the results (in terms of the knowledge gathered and synthesized) can be assessed with reference to the method that produced them. This document begins by describing the analytical framework proposed for examining the various dimensions of public policies. Next, it specifies the types and sources of data to be considered in a synthesis of knowledge about a public policy. Finally, it describes the proposed synthesis method, step by step, in sufficient detail to allow public health practitioners to apply it. Alongside this document, we are publishing a sister document that will describe the application of our knowledge synthesis method to public policies in the area of nutrition labelling on food packaging and in restaurants (Morestin et al., in press). The reader is invited to consult this document for a concrete example of the type of knowledge synthesis that can be produced using the method proposed here. 2 National Collaborating Centre for Healthy Public Policy

13 1 FRAMEWORK FOR ANALYZING PUBLIC POLICIES The knowledge synthesis method that we are proposing applies to public policies. In this context, the National Collaborating Centre for Healthy Public Policy defines public policy as, An action or group of strategic actions carried out by a public authority with the aim of attenuating or promoting particular phenomena occurring in the population. A public authority here refers to a legislative, executive or judicial authority belonging to or derived from a federal, provincial, regional or municipal public administration. The public authority may act alone or in partnership with other public and/or private organizations. The action or group of actions may concern the whole population or one or several of its constituent sub-groups. (National Collaborating Centre for Healthy Public Policy, 2010). What factors should be considered when assessing the likelihood that a public policy will succeed? Effectiveness is generally considered to be the principal indicator of the success of a policy (Salamon, 2002). In the field of health, in particular, the movement toward evidence-based decision making places emphasis on the effectiveness of the options being considered. This movement has its counterpart in the field of public policy, where various governments have made it their purpose to analyze policies and programs to see what works. It is very clear that decision makers expect to be presented with data on the effectiveness of interventions (McQueen, 2007). However, it does not suffice to examine the effectiveness of a public policy. As many authors have shown, it is also necessary to take into account the context in which a policy will be adopted and implemented (Banta, 2003; Potter & Harries, 2006; Salamon, 2002; Rychetnik et al., 2002). In reality, the implementation context has an impact on the results observed: the same policy implemented in two different contexts will not necessarily produce the same results (Pawson, 2006; Rychetnik et al., 2002). Moreover, information about contextual factors is indispensable to forming judgements about whether a policy implemented among a given population will be applicable to others (McNeil & Flynn, 2006; Rychetnik et al., 2002). Finally, decision makers are influenced by considerations that go beyond effectiveness (Peters, 2002; Hood, 2007), and which must be taken into account in the information they are provided: syntheses that present evidence in a manner that is divorced from the realities of policy implementation are of little use to decision makers (Jewell & Bero, 2008). Therefore, many groups that are recognized for their expertise in the area of knowledge synthesis go beyond the use of data on effectiveness. For example, the Task Force on Community Preventive Services affiliated with the Centers for Disease Control and Prevention in the United States includes, in its systematic reviews, not only data about the effectiveness of the interventions studied, but also data on their applicability to other populations, on their economic impact, and on observed obstacles to their implementation (Briss et al., 2000). As for the Cochrane Public Health Review Group, it plans, beginning in 2009, to conduct a series of systematic reviews that will extract from primary studies not only data on effectiveness, but also other information that meets the needs of decision makers, such as data about context, the implementation process, equity, the views of stakeholders, cost, and sustainability (Waters, 2009). National Collaborating Centre for Healthy Public Policy 3

14 Thus, like many authors (Pineault & Daveluy, 1986; Dobrow et al., 2006; Petticrew & Roberts, 2006; Heller, 2005; McNeil & Flynn, 2006), we are proposing a two-pronged analytical approach, which takes into consideration, on the one hand, dimensions related to the effects of public policies and, on the other hand, dimensions related to their implementation while at the same time recognizing, naturally, that the two categories are interconnected. In defining the policy dimensions to be studied, we drew on Salamon s analytical framework, derived from the traditions of policy analysis and political science (Salamon, 2002), and on a related analytical framework that was developed with reference to policies aimed at addressing obesity (Swinburn et al., 2005) and applied to such policies (Victorian Government Department of Human Services, 2006), but which is a generic framework that can be used to study other public policies. We are proposing the following six dimensions for analysis: Table 1 Dimensions for analyzing public policies Effects Implementation Effectiveness Unintended effects Equity Cost Feasibility Acceptability What effects does the policy under study have on the targeted problem? What are the unintended effects of the policy? What are the effects on different groups? What are the financial costs of the policy? Is the policy technically feasible? Do the relevant stakeholders view the policy as acceptable? Effectiveness, unintended effects and equity relate to the effects produced by public policies. Cost, feasibility, and acceptability relate to the implementation of policies. A seventh element brought forward by some authors (Swinburn et al., 2005; Victorian Government Department of Human Services, 2006; Waters, 2009) is sustainability, but this was not included as a separate dimension for analysis because it cuts across the other six dimensions. Concretely, in terms of the sustainability analysis, one must gather data on whether and how adopted public policies remained active and maintained their effects over the long term; these data on sustainability are to be attached to the corresponding analytical dimension. Because we are considering policies from a public health perspective, our analytical framework focuses on technical effects: the effects on health obtained by means of the public policy under study. Cost and feasibility are also considered in technical terms. However, in addition, our approach takes into account political issues (political motivations, effects and costs ) related to the promotion, adoption and implementation of public policies. These elements are addressed, in particular, under the dimensions of Unintended effects and Acceptability. The elements that should be considered under each dimension are specified below. 4 National Collaborating Centre for Healthy Public Policy

15 1.1 DIMENSIONS RELATED TO THE EFFECTS OF PUBLIC POLICIES Effectiveness The first element used to assess the success of a public policy is its effectiveness or, in other words, the degree to which it has achieved its objectives (Salamon, 2002; Potvin et al., 2008): there may, however, be an absence of effects, or even negative effects that will aggravate the targeted problem. We choose to consider these neutral or negative effects under the "Effectiveness" dimension, along with positive effects, because they are measured against the objective being pursued by the policy under study. On the other hand, all other effects (positive or negative) that are produced by the policy, but that do not relate to the objective pursued, are classified under Unintended effects (see below). The analysis of effectiveness comes up against a problem. It is often very difficult to judge the ultimate effectiveness of public policies (Salamon, 2002; Banta, 2003; Potter & Harries, 2006). This difficulty is compounded by the fact that analysts are frequently confronted with a lack of literature on the links between policies and their ultimate effect on the health problem they target. In response to these issues, our method opens analysis to other types of data on effectiveness: those focused on the link between a public policy and its intermediate effects and on the link between these intermediate effects and the ultimate effect on the targeted problem. To identify these data on effectiveness, it is first necessary to deconstruct the chain of effects expected to link the policy under study to the targeted problem. This process has been borrowed from program evaluation methods, in which (with terminology varying from one author to another) many authors suggest detailing the chain of effects, or intervention logic, and representing it graphically in the form of a logic model as shown below (Champagne et al., 2009b; Weiss, 1998). Public policy Intermediate effect on the determinants of causes Intermediate effect on causes of the problem Ultimate effect on the problem Figure 1 Generic logic model Construction of the logic model of the policy being studied is the second step in the knowledge synthesis method we are proposing and we will return to it later. Analysis of the empirical data on the effectiveness of the intermediate links in the chain of effects is particularly relevant in public health, which is concerned with the determinants of health: measuring the effect of an intervention on a determinant (an intermediate effect) can be as important as measuring the (ultimate) effect on the health problem targeted (McNeil & Flynn, 2006). Conversely, it is often unnecessary to gather data on the relationship between the problem targeted and its most proximal determinants: for example, between food intake/energy expenditure and obesity; or between smoking and lung cancer. Generally, this link has already been solidly established in the literature, a brief reference to which suffices National Collaborating Centre for Healthy Public Policy 5

16 (Banta, 2003; Briss et al., 2000; Champagne et al., 2009a); the main interest is in determining what factors, further upstream, a public policy seeks to influence. When analyzing the effectiveness of a public policy, it is always useful to examine whether its intervention logic, as represented by its logic model, is plausible. Plain common sense allows for an initial assessment; ideally, the opinions of experts either confirming or invalidating the intervention logic should be gathered (from the literature or through direct consultations). Solid logic points to the potential effectiveness of a policy; this effectiveness may or may not be achieved, depending on the context and the quality of the implementation (Weiss, 1998). It is necessary to differentiate between these two levels of success or failure, by comparing the plausibility of the intervention logic with the empirical data on the level of effectiveness achieved by the public policy under study (Rychetnik et al., 2002); while at the same time taking into account the data on how the implementation context influenced the policy s effectiveness (Rychetnik et al., 2002; Pawson & Tilley, 1997). Finally, in cases where there are no empirical data on the effectiveness of a policy, certain authors hold that the plausibility of its intervention logic, in itself, constitutes evidence of effectiveness (Swinburn et al., 2005; Haby et al., 2006). Examples of Key Questions: Effectiveness Is the intervention logic of the public policy under study plausible? What are the effects of the policy (positive, neutral, negative) on the problem targeted? How effective is the policy in terms of its intermediate effects? What impact does the implementation context have on the policy s effectiveness? Unintended Effects Consideration is given here to all the effects that are produced, but not directly pursued within the context of the public policy under study; in other words, effects that are unrelated to the objective(s) pursued. The fact that they are unintended implies nothing about their value: these effects may be positive or negative. Moreover, they may or may not have been anticipated. Unintended effects occur because public policies are applied within complex systems characterized by multiple interrelated processes that interact in a nonlinear manner and adapt to changes (Morell, 2005). Put more simply: given the complexity of the real world, it is impossible to control a public policy so fully as to ensure that it produces only the desired effect, and no other. A public policy can have unintended effects in all sorts of areas: effects on health that are unrelated to the health problem targeted, economic, political, or environmental effects, effects on social relations, etc. Example of a Key Question: Unintended Effects Does the policy under study produce unintended effects (whether these be positive or negative, anticipated or unanticipated)? 6 National Collaborating Centre for Healthy Public Policy

17 1.1.3 Equity There are as many definitions of equity as there are conceptions of the meaning of social justice (Potvin et al., 2008) and we make no attempt here to cover that debate. Of interest here are the implications of public policies in terms of equity. In this regard, we uphold a distinction established by Mooney that is often made in the health field (Culyer, 2001; Mooney, 1983; Starfield, 2001). Two facets of equity are distinguished: horizontal equity, which calls for similar treatment of individuals with similar needs; and vertical equity, which calls for different treatment of individuals with different needs, in proportion to the differences that exist between them (concretely: those with greater needs receive more, and the reverse). In terms of public policies, the concept of equity is related to that of redistribution: the extent to which policies mainly benefit groups with greater needs (Salamon, 2002). Thus, we are examining effects on equity; that is, the differential effects of the public policy under study on different groups (categorized by age, gender, socioeconomic status, ethnicity, residence in certain zones, etc.), or the probability that the policy will affect the unequal distribution of the targeted health problem (Swinburn et al., 2005). It is very important to take into account equity and not only effectiveness, because it is recognized that, often, public policies which improve population health in terms of the overall average nevertheless increase health inequalities (Potvin et al., 2008). For example, nutrition labelling aimed at promoting healthy eating habits may have effects that are less than equitable: if the Nutrition Facts table is too complicated to interpret, its information will be exploited mainly by the most highly educated groups, and only to a lesser degree by less educated groups, which are already more affected by the problems of overweight and obesity. Because of its different effects on groups with varying levels of education, this labelling policy could increase inequalities linked to weight problems. Examples of Key Questions: Equity What are the effects (pursued or unintended) of the policy under study on different groups? Does the policy provoke, reinforce or correct social inequalities in health? 1.2 DIMENSIONS RELATED TO THE IMPLEMENTATION OF PUBLIC POLICIES Cost Consideration is given here to the financial costs associated with applying a public policy, which may be negative (expenses) or positive (gains). It should be noted that a policy may also have symbolic or political "costs" (for example: loss of votes during the next election for the party in power if it implements an unpopular public policy); these are not costs in the strict sense of the word, but rather a type of unintended effect, and are thus classified under the corresponding dimension. When studying a public policy, one thinks first of all of the financial cost assumed by the government in its application; but it is also necessary to take into account the cost to other actors affected by the policy (Salamon, 2002): those who implement it as well as those directly or indirectly affected (Rychetnik et al., 2002). Costs can be considered in an absolute National Collaborating Centre for Healthy Public Policy 7

18 manner or in a relative manner, by comparing the costs of the policy under study with the costs of other potential policies (Pineault & Daveluy, 1986; Drummond et al., 2005). Analysis of the costs of public policies can also be cross-referenced with data on their effectiveness, to estimate their efficiency (Salamon, 2002; Pineault & Daveluy, 1986; Drummond et al., 2005). Finally, costs can be analyzed in terms of their distribution over time (immediate or deferred costs, short- or long-term investments) (Pineault & Daveluy, 1986) and in terms of their visibility, that is, the degree to which the positive and negative costs associated with a policy are apparent (Salamon, 2002; Peters, 2002). Examples of Key Questions: Cost What are the implementation costs for the government? For other actors (private, community, individual, etc.)? How do the costs of the policy under study compare to those of other potential policies? What is the cost-effectiveness of the policy under study, for the government, for society? Are the costs of implementation immediate or distributed over time? To what extent are the costs of implementation apparent? Feasibility Under this dimension, the technical feasibility of the public policy under study is considered. Aspects of political feasibility are considered under the Acceptability dimension. The level of feasibility is tied to a series of factors. The feasibility of the public policy under study depends first on its conformity with all applicable legislation (Pineault & Daveluy, 1986); attention should be paid, in particular, to the distribution of powers among the various levels of government. The pre-existence of pilot programs to which the public policy is a follow-up is both an indication of the feasibility of the policy and a facilitating factor, if the policy can draw on the experience and structure of these programs (Swinburn et al., 2005). The policy s feasibility can be enhanced by what Salamon refers to as automaticity, that is, the extent to which a public policy s implementation is managed by pre-existing administrative mechanisms (Salamon, 2002); however, Salamon acknowledges, along with others (Sabatier & Mazmanian, 1995), that existing mechanisms have their own objectives, which may not completely coincide with those of a public policy a problem which does not arise when a new structure is created specifically for the implementation of a policy. Regardless, it is interesting to collect data on the degree of automaticity involved, to see whether or not, in a given context, it increases the public policy s feasibility. Application of a policy tends to be simpler in contexts of greater directness, that is, when the entity that authorizes, finances or launches the policy is also involved in its implementation (Salamon, 2002). Inversely, it tends to become more complicated as the number of actors involved in its implementation rises; although this complexity can be tempered, according to Sabatier and Mazmanian, through a form of hierarchical integration: that is, the extent to which those spearheading the public policy guide the activities of the 8 National Collaborating Centre for Healthy Public Policy

19 other actors involved in its implementation, using an appropriate system of incentives and sanctions (Sabatier & Mazmanian, 1995). The quality of the cooperation between the actors involved in implementation is fundamental to feasibility (Salamon, 2002; Pineault & Daveluy, 1986; Swinburn et al., 2005). With inverse effects, the ability of opponents to interfere is an equally important factor; especially since the opponents of a public policy are often more active, over a longer period, than its partisans (Sabatier & Mazmanian, 1995). It should be noted that cooperation and interference are considered in relation to the feasibility dimension because of their practical implications for the implementation of a public policy; however, these factors are influenced by how stakeholders view the policy, which is an aspect of the "acceptability" dimension. To conclude, at the operational level, feasibility depends on the availability of the personnel (Pineault & Daveluy, 1986; Swinburn et al., 2005), material resources and technology (in the broad sense) required (Dobrow et al., 2004; Sabatier & Mazmanian, 1995). Examples of Key Questions: Feasibility Does the policy under study fall under the jurisdictional power of the authority that wishes to adopt it? Does it conform to existing legislation? Is the policy a follow-up to a pilot program? Can the policy be administered by pre-existing mechanisms? Is the authority promoting the policy also the one applying it? How many different actors are involved in implementing the policy? To what extent are their activities being guided by the policy s promoters? Do they cooperate well? Do the policy s opponents have the ability to interfere with its adoption or implementation? Are the personnel required to implement the policy available? Are the required material and technological resources available? Acceptability Acceptability refers to the way in which a public policy is judged by stakeholders (Swinburn et al., 2005). Stakeholders are actors concerned with the objectives and/or implementation of a policy (Rychetnik et al., 2002); these may include: groups directly targeted by the policy, the wider public, ministries, other decision makers, professionals from the relevant public sectors (for example, health or education), funding agencies, industry, the media, political organizations, etc. (Swinburn et al., 2005). The term acceptability is not to be understood in the moral sense. Rather, it refers to the balance of power between the actors concerned by a public policy. Regardless of how effective the policy is expected to be, if it does not have enough political support (including the support of public opinion, of those with economic and financial power, etc.), it will be difficult to see it adopted and implemented, and thus producing the desired effects (Salamon, 2002; Dobrow et al., 2004). National Collaborating Centre for Healthy Public Policy 9

20 Acceptability is probably the most complex dimension of the analysis. First, it involves subjective elements (the judgement of actors). Secondly, it is influenced by all the other dimensions of the public policy being studied, among other things. Finally, the acceptability of a policy also depends on factors that are external to it: the position of each actor is determined by that actor s knowledge, beliefs, values and interests (political, economic, symbolic, etc.) (Peters, 2002). In analyzing the acceptability of a public policy, it is necessary to consider stakeholders judgements not only of the intrinsic characteristics of the policy, but also of the conditions surrounding its adoption and implementation. a) Stakeholders judgements of the intrinsic characteristics of a public policy must themselves be deconstructed (Peters, 2002; Barry et al., 2009): First, there is the acceptability of acting on the targeted problem: do the relevant actors think that the problem merits public intervention; or, using the terminology of political science, is it on the political agenda? Next, there is the acceptability of the public policy proposed, compared with that of other potential policies aimed at addressing the problem: in fact, all actors have their own way of framing the problem (that is, of interpreting the necessarily complex reality, to construct their own definition of the problem and its causes), which influences which solutions they judge suitable for addressing the problem (Rein & Schon, 2005). With regard to the acceptability of a public policy, stakeholders reactions are based on their assessment of the other dimensions: do they think that the policy is effective (in particular: do they subscribe to its intervention logic), that its unintended effects are acceptable, that it is equitable, that it is costly (and this perception depends largely on the distribution of costs over time and their visibility (Salamon, 2002; Peters, 2002)), and that it is feasible. Often, the judgement formed is not explicitly articulated in these terms; but ultimately, stakeholders are reacting to these factors. Since it is based on their perception, their judgement might not correspond to the objective data on these dimensions. The perceptions of actors are not any less important. On the contrary, these perceptions often carry more importance for political decision makers than objective evidence (Sabatier & Mazmanian, 1995). Another important factor affecting how public policies are judged is the degree of coercion they involve. In concrete terms, the question raised in relation to public health is: how far can we go to change behaviour? (Pineault & Daveluy, 1986). Public policies can make use of a wide range of instruments, from the least coercive (for example: information campaigns), to the moderately coercive (for example: subsidies or other incentives), to the most coercive (for example: regulations prohibiting or making mandatory certain behaviours) (Salamon, 2002). Because coercive policies restrict individual liberty, they are poorly tolerated by some actors; decision makers are aware of these reactions, and often choose the least coercive option, or at least try to combine coercive measures with informative measures or incentives (De Leeuw, 2007; Sabatier & Mazmanian, 1995). 10 National Collaborating Centre for Healthy Public Policy

21 b) Stakeholders judgements concerning conditions related to the adoption and implementation of a policy. Stakeholders reactions to decisions depend on the extent to which they acknowledge the legitimacy of the decision maker(s) and the decision-making process (Singer et al., 2000). They form judgements about the legitimacy and the capability of actors implementing a public policy (actors who may include themselves). Finally, planned accountability related to a policy is an important factor in stakeholders assessment of its acceptability (Salamon, 2002). To document the acceptability of a public policy, one can draw information from the literature, such as, the results of opinion polls, the stated positions or publicly declared views of certain actors, etc. Because acceptability can change over time, given ongoing changes to socio-economic, political and technological conditions (Sabatier & Mazmanian, 1995), it is important to document acceptability not only when a public policy is being discussed, adopted and implemented, but also in the medium- and long-term during its application. Examples of Key Questions: Acceptability Which actors are or will be affected by the public policy being considered? Is the problem the policy aims to address considered to be a social issue that merits intervention? Is it on the discussion agenda? What are stakeholders reactions to the idea of intervening to address this problem? What type of intervention do stakeholders propose for addressing this issue? What do they think of the proposed policy? Of its effectiveness, its unintended effects, its effects on equity, its costs, its feasibility? Of the degree of coercion it involves? What do they think of the conditions surrounding the adoption and implementation of this policy? 1.3 RELATIONSHIPS BETWEEN THE SIX DIMENSIONS FOR ANALYSIS The six dimensions for analysis described above may be considered separately, but it is interesting to keep in mind that they are interrelated. All of the dimensions, without exception, influence the Acceptability dimension, because stakeholders judge a public policy on the basis of their assessment of the other dimensions. Inversely, a public policy s degree of acceptability can have a bearing on its feasibility: this is the case, for example, if certain actors assessment of a policy is unfavourable ( Acceptability dimension) and, consequently, they decide to take action to impede its implementation ("Feasibility" dimension). In addition, feasibility can influence the cost of implementing a public policy: the more compromised its feasibility, the greater the risk that its implementation will entail additional costs. Finally, the three dimensions of implementation (cost, feasibility and acceptability) collectively influence a policy s effects (effectiveness, unintended effects, and effects related to equity), because they determine its ability to produce those effects. National Collaborating Centre for Healthy Public Policy 11

22 It should be noted that we are not taking temporality into account here, that is, the fact that some dimensions come into play before or after others. The proposed knowledge synthesis process first adopts a completely a posteriori approach (review of the literature to gather data about the six dimensions of a public policy already implemented elsewhere); and then a completely a priori approach (deliberative processes aimed at anticipating what would occur in terms of these six dimensions, if a similar public policy were to be applied in one s own context). IN BRIEF Framework for Analyzing Public Policies To identify the public policies that are most likely to succeed in a given context, it is necessary to study their effectiveness, but study cannot be limited to this. The analytical framework proposed here also takes into consideration other dimensions related to a policy s effects: its unintended effects and its effects on equity. In addition, it takes into account dimensions related to a policy s implementation, because these influence the effects produced and because they are of interest to the decision makers and actors concerned: the cost, feasibility, and acceptability of a policy. The figure below illustrates the six dimensions of the analysis framework and their relationships. Effects Effectivenes Unintended effects Equity Implementation Cost Acceptability Feasibility = influence Figure 2 Relationships between the six dimensions for analyzing public policies Next, let us examine the types and sources of data to be considered when documenting the different dimensions of public policies. 12 National Collaborating Centre for Healthy Public Policy

23 2 TYPES AND SOURCES OF DATA TO BE CONSIDERED The traditional approach to synthesizing evidence gives precedence to data on effectiveness drawn from scientific literature and, in particular, to data from experimental studies (randomized controlled trials). However, this approach, designed for interventions targeting individuals, seems limited as an approach to assessing public policies, which are far more complex and apply to entire populations in varied contexts, which are themselves complex (McQueen, 2002; Briss et al., 2000; Lomas et al., 2005). In the field of public health and public policy, it is very difficult to obtain effectiveness data from experimental studies because carrying out such studies is often technically and ethically unfeasible (Banta, 2003; McNeil & Flynn, 2006; De Leeuw, 2007; McQueen, 2007). Moreover, assessment of the very complex socio-political processes at play during decision making and policy implementation is often carried out by other research groups guided by different values and methods (Rychetnik et al., 2002). Thus, there are two serious problems associated with using the traditional approach to collecting evidence when gathering evidence to guide public health decision making: There is a risk that promising policies will not be taken into consideration because of the absence of evidence (in the restrictive classical sense of the term) to support them (Rychetnik et al., 2002). The traditional approach fails to take into account data on issues related to implementation, despite the fact that these issues are highly important to decision makers and influence the effectiveness of public policies (Salamon, 2002; Waters, 2009). Many experts have broadened their definition of the concept of evidence, so that decisions can be informed by the best available data and not by the best possible data (Swinburn et al., 2005; Banta, 2003; Lomas et al., 2005). For example, subsequent to work carried out by the European Advisory Committee on Health Research, the World Health Organization s Regional Office for Europe adopted a more exhaustive definition of evidence, which includes not only the findings from research, but also other knowledge that may serve as a useful basis for decision-making in public health (WHO Regional Office for Europe, 2009). Mays and his colleagues, in a methodological article on systematic reviews aimed at informing decision makers and managers, argue that the more the authors of a knowledge review seek to support decision making, the more the review must consider context and the more open it must be to different forms of evidence (Mays et al., 2005). This openness implies including quantitative and qualitative data, research data, and also other types of data. More specifically, certain authors have suggested that a whole range of data should be drawn on when producing a knowledge synthesis aimed at informing public health decision making (Haby et al., 2006; Swinburn et al., 2005): not only data traditionally referred-to within the context of the evidence-based movement, drawn from experimental studies (controlled trials); and quasi-experimental studies (case-control, cohort studies); but also data drawn from descriptive studies; effectiveness modelling; National Collaborating Centre for Healthy Public Policy 13

24 economic evaluations (for example, cost-effectiveness analyses); process evaluations; monitoring & surveillance data; "parallel evidence (on a strategy that is similar, but targets another public health problem; for example, data on the taxing of tobacco, which provide indications about what could be achieved by taxing junk food); the intervention logic (the chain of effects expected to link the policy under study to the targeted problem); and expert opinion. Our method for synthesizing knowledge about public policies adopts this openness toward data, going beyond the exploration of the scientific literature, to include exploration of the grey literature as well (documents produced by governments or non-profit organizations, statements by professional associations, opinion polls, etc.). Moreover, a synthesis of knowledge about a public policy should extend even beyond the literature. Data from the literature are often complex, incomplete, inconclusive, subject to scientific controversy or non-contextualized; and, consequently, uncertainty remains as to a public policy s chances of succeeding (Lomas et al., 2005). Also, when used for decision making, these data must, inevitably, be interpreted in the light of, and even supplemented by, colloquial evidence drawn from local experience. One way to combine evidence from the literature and colloquial evidence in a transparent and explicit manner is to organize deliberative processes (Lomas et al., 2005). By deliberative processes, we mean processes through which dialogue can be established between experts, decision makers and other actors, for the purpose of critically examining an issue. The organization of deliberative processes to gather information for knowledge syntheses is a recent trend, but one in which interest has been spurred by the movement promoting evidence-informed policy making. A deliberative process can fulfill at least three roles within the context of a knowledge synthesis (Lomas et al., 2005; Lavis, 2006; Lavis et al., 2009): Combining different forms of evidence : The results of knowledge syntheses compete with many other factors that can influence decision making. Consider, for example, the impact of values, interests, institutional arrangements, the political or administrative feasibility of an intervention, etc. Thus, deliberative processes can be useful as a way of drawing out data on these factors and combining them in a transparent manner with data found in the literature, to guide decision making. Contextualizing data drawn from the literature on the subject: It is sometimes difficult to determine if data are generalizable or transferable from one context to another. Deliberative processes involving experts, decision makers and other actors allow data from the literature to be co-interpreted in the light of the respective knowledge of these actors about the context in which implementation of the public policy under study is being proposed. 14 National Collaborating Centre for Healthy Public Policy

25 Generating new knowledge: By promoting dialogue between experts, decision makers and other stakeholders, deliberative processes can generate new insights from the cross fertilization of knowledge. This can lead to creative and innovative solutions to collective problems. Thus, knowledge syntheses become much more complete and likely to inform decision making. Consequently, we recommend that deliberative processes be organized to gather data and that this be a specific step in the public policy knowledge synthesis process. IN BRIEF Types and Sources of Data to be Considered The traditional approach to synthesizing evidence, focused on effectiveness data drawn from experimental studies, does not apply to the synthesis of knowledge about public policies. When reviewing public policies, it is necessary to expand the definition of evidence: quantitative, qualitative, research and other data should be included. Sources to be considered include not only the scientific literature, but also the grey literature and (through the organization of deliberative processes) actors working within the context in which implementation of the public policy under study is being proposed. Having made these general points about the approach required by the specific nature of public policies, we can now describe the proposed knowledge synthesis method in detail. National Collaborating Centre for Healthy Public Policy 15

26

27 3 KNOWLEDGE SYNTHESIS METHOD The proposed knowledge synthesis method involves four steps: 1) Compilation of an inventory of public policies that could address the targeted health problem (by means of a preliminary exploration of the literature), followed by the selection of the policy on which the knowledge synthesis will focus. 2) Explication of the intervention logic of the policy under study (construction of the logic model). 3) Synthesis of the data on effects and on the issues related to implementation of the selected policy, carried out through means of a literature review. 4) Enrichment and contextualization of the data drawn from the literature, through deliberative processes that bring together actors concerned by the targeted health problem and working within the context in which implementation of the policy under study is being considered. STEP 2 Explication of the intervention logic STEP 4 Enrichment and contextualization of data STEP 1 Inventory of policies and selection of subject of synthesis Preliminary exploration of the literature Construction of logic model STEP 3 Synthesis of data drawn from the literature Literature review Deliberative processes SYNTHESIS Integration of data drawn from the literature and from deliberative processes Figure 3 Steps in the knowledge synthesis process With an eye to simplicity, these steps are being presented as part of a linear process. However, in reality, the proposed method is open to iteration: one may, at any time, return to a previous step to rework it in light of new understanding of the subject under study, acquired while carrying out a subsequent step. Before even embarking on the knowledge synthesis, the targeted health problem must be described (in terms of the problem s characteristics and its scope in the particular context in which a public policy s adoption is being considered as a way of addressing it). We will not dwell further on this preliminary step, since once actors are at the point where they have decided to carry out or to mandate the production of a knowledge synthesis of public policy options, this step has already been carried out. In the document presenting the knowledge synthesis, this description of the problem should be brief and included in the introduction. It should be informed by material already provided by those mandating the knowledge synthesis and, if necessary, supplemented by some of the data gathered during the knowledge synthesis process. National Collaborating Centre for Healthy Public Policy 17