EVALUATING LONG-TERM TRANSITION PROGRAMS ON A SHORT-TERM BASIS Towards a six-step transition program evaluation tool

EVALUATING LONG-TERM TRANSITION PROGRAMS ON A SHORT-TERM BASIS Towards a six-step transition program evaluation tool Tom Creten, Sander Happaerts & Kris Bachus HIVA KU Leuven Research paper 9 Leuven, September 2014

The Policy Research Centre Transitions for Sustainable Development brings b together researchers from KU Leuven, Ghent University, VITO and Erasmus University Rotterdam. Please refer to this publication as follows: Creten T., Happaerts S. & Bachus K. (2014), Evaluating long-term transition programs on a short- Research Centre on Transitionss for Sustainable Development. For more information kris.bachus@kuleuven.be term basis. Towards a six-step transition program evaluation tool (Research Paper 9). Leuven: Policy This publication was sponsored by the Flemish Government, under the t 2012-20155 Policy Research Centre Programme. This publication reflects the views v only of the author, and the Flemish Government cannot be held responsible for any use which may be made of the information contained therein. 2014 POLICY RESEARCH CENTRE ON TRANSITIONS FOR SUSTAINABLE DEVELOPMENT p.a. Administration Policy Research Centre on Transitionss for Sustainable Development HIVA - Onderzoeksinstituut voor Arbeidd en Samenleving Parkstraat 47 box 5300, BE 3000 Leuven, Belgium This publication is also available via www.steunpunttrado.be Evaluating long-term transition programs on a short-term basis i

Contents Voorwoord (in Dutch) Executive summary v vi Introduction 1 1. Theoretical framework 2 1.1 Evaluation in the transitions literature 2 1.2 Environmental evaluation literature 5 1.3 A first glance at the transition program evaluation tool 9 2. The transition program evaluation tool elaborated 10 2.1 Step 1: Evaluation of internal program consistency 10 2.2 Step 2: Evaluation of crucial process activities 13 2.3 Step 3: Evaluation of crucial policy activities 16 2.4 Step 4: Evaluation of policy instruments implementation 19 2.5 Step 5: Evaluation of outcome effects: decreasing regime resistance 21 2.6 Step 6: Exposing causality 24 2.7 The tool revisited 28 3. Reflections on the use of the tool 31 4. Research agenda 32 Bibliography 34 Evaluating long-term transition programs on a short-term basis iii

Voorwoord (in Dutch) Deze paper stelt een ontwerptool voor die kan dienen om transitiebeleidsprogramma s te evalueren. De tool is voornamelijk bedoeld voor de managers of coördinatoren van transitieprocessen (in de eerste plaats opgezet vanuit het Vlaamse beleid). Maar hij kan evengoed gebruikt worden door andere betrokkenen in gelijkaardige processen of door externe evaluatoren. De toepassing van de tool moet kunnen aantonen of het transitieproces goed verloopt en of de overheid wel de juiste beleidsondersteuning biedt om de vooropgezette doelstelling te kunnen behalen. Hij moet kunnen aangeven welke goede stappen er al gezet zijn, wat nog ontbreekt en hoe de overheid kan bijsturen. Op die manier zorgen we ervoor dat de transities, en vooral het beleidsprogramma dat opgezet is om ze te sturen, op korte termijn en pragmatisch geëvalueerd kunnen worden, hoewel ze verandering op lange termijn beogen. De tool focust op de niches die gepromoot worden om een bepaalde transitie te versnellen of in de juiste richting te sturen. Hij kan daarom toegepast worden om één, meerdere of alle verschillende niches te evalueren die in een bepaald beleidsprogramma aan bod komen. Dat komt tegemoet aan de vraag om een rollende evaluatie te kunnen doen, zonder meteen een hele transitie (waar de overheid slechts in beperkte mate controle over heeft) te moeten beoordelen. Zes verschillende stappen moeten doorlopen worden om tot die evaluatie te komen. In een eerste stap gaan we de interne samenhang van het transitiebeleidsprogramma na. Zijn de geselecteerde niches wel de juiste? Hebben ze, ongeacht hun uitwerking, de mogelijkheid om de langetermijndoelstellingen van het transitiebeleid (ten dele) te bereiken? De tweede stap kijkt naar een aantal cruciale procesacties die moeten gebeuren om een transitiebeleidsprogramma te doen slagen, bijvoorbeeld op het vlak van visievorming en samenwerking. De derde stap doet hetzelfde voor een reeks cruciale beleidsactiviteiten die, in verschillende domeinen, de mechanismen van transitiebeleid in gang moeten zetten. Om de beleidsactiviteiten uit te voeren, moeten een aantal instrumenten ingezet worden. Als vierde stap gaan we dan ook de goede implementatie van die beleidsinstrumenten na. Het beleid kan immers wel goed bedacht worden, maar moet ook nog correct uitgevoerd worden. Ten vijfde proberen we te achterhalen welke van de verwachte effecten er al gerealiseerd zijn. Dat doen we door te meten in hoeverre het gangbare regime al dan niet weerstand biedt aan de niches die gepromoot worden door het beleidsprogramma. Als laatste stap bestuderen we het oorzakelijke verband tussen de effecten enerzijds en het beleidsprogramma anderzijds. Een transitie kan immers in een bepaalde richting evolueren, zonder dat die evoluties een noodzakelijk gevolg zijn van overheidsbeleid. De tool biedt daarom wel de mogelijkheid om de rechtstreekse invloed van het beleid in te schatten. Het finaliseren van deze tool is het einddoel van onderzoekslijn 2 van TRADO. Op basis van deze eerste versie zullen we de tool daarom in 2014 en 2015 toepassen in twee transitieprocessen van Vlaanderen in Actie, nl. Slimme Mobiliteit en Flanders Care. Op basis van de ervaringen van die twee casestudies zullen we de tool verder verfijnen en aanvullen waar nodig. Op het einde van het project leveren we een afgewerkte tool met een gebruikershandleiding af aan de Vlaamse overheid. Evaluating long-term transition programs on a short-term basis v

Executive summary Despite a growing body of literature on transitions, surprisingly little has been published on the evaluation of policy programs that aim to enhance ongoing transitions in society. Possibly, such is due to the inherent paradox in the evaluation of transition policy. While policy-makers are interested in rather short-term evaluations to know when and how to adapt ongoing policy programs, transition policy by definition aims at achieving system change only on a very long time horizon. To tackle that paradox, we develop in this paper a hands-on six-step evaluation tool for transition policy programs that can be used both by civil servants during the coordination of a transition program and by external evaluators performing ex-post evaluations. The tool is based upon an innovative combination of the transitions literature with the knowledge from the field of traditional (environmental) evaluation. Such an original combination of two strands of literature also forms an academic added value to the transitions field of study. Specifically, we make use of the distinction made by environmental evaluation scholars between output, outcome and impact effects (Gysen et al., 2006b). In the case of transition programs, the ultimate goal of system change is the impact effect that is hoped for, but which is almost impossible to evaluate given its long time horizon. Therefore, we only investigate the potential impact of the program and otherwise focus upon the intermediate output effects (direct policy results) and outcome effects (influence on target groups) that should be visible on a shorter term, next to the important aspects of process evaluation that we also take into account. Apart from that, we also make use of the program theory evaluation methodology which is based upon the idea that policymakers have in mind a well-defined theory about how their policy will influence reality. In the case of transition programs, the assumptions that form the basis of that policy program theory come from the transitions literature. Our evaluation tool therefore investigates three possible problems: 1. The evaluator investigates the possible occurrence of an application deficit. In that case, the transitions literature was not well applied in the development of the policy program theory. Essential elements that should be common to any policy theory based upon transitions thinking are missing in reality, which explains why success is impossible. 2. The evaluator should examine the possibility of implementation deficits. Apart from the common elements that should be present in each transition policy theory, the policymakers will also at some point make a choice for specific policy instruments. Those specific instruments are needed to carry out in practice the general crucial policy activities that are common to all transition programs, but will themselves be different for each transition program. Whatever the chosen instruments however, they should be used and implemented in an appropriate way, according to the spirit of the instrument. 3. Finally, there is the possibility of a policy theory failure that should be evaluated. In that case, it is the transition program theory itself that proves to be invalid, at least for the policy problem under consideration. It means the policy-maker s assumptions and expectations (based for a large part upon the transitions literature, but also on assumptions on specific policy instruments) about the consequences of the chosen policy were wrong. vi Tom Creten, Sander Happaerts & Kris Bachus

For the latter problem, use will be made of a very specific methodology called the modus narrandi (Gysen et al., 2006b) to investigate causal chains between the transition program on the one hand and the outcome effects on policy target groups on the other hand. In the first step of the evaluation tool, the internal consistency of the program is investigated. Specifically, we focus on the question whether or not the chosen niches of the program can contribute to the final aim of system change for the system under consideration, i.e. we investigate the potential impact, of the transition program. In case all scientific literature aims in the same direction, namely that the chosen niches are irrelevant or pernicious for the long-term objective of the program, then we can conclude the crucial policy theory assumption that the chosen niches will contribute to the wanted system change is false and a theory failure occurs. Also, any further evaluation in fact becomes obsolete. That is why we start off with that first step. In the second and the third steps, we look for possible application deficits. The second step identifies a number of crucial process activities that should be present in any transition program. The third step does the same for crucial policy activities (output effects) that should be witnessed. In case those crucial process and policy activities do not take place, the transitions theory was not well applied in the development of the policy theory. The crucial policy activities can be carried out by making use of a several different policy instruments. Those instruments have to be implemented well in case any outcome results are to be expected. That is the research object of step 4. In step 5 and 6 we finally evaluate the policy program theory itself. We evaluate whether or not the expected outcome effects from the transition program have occurred. Such is investigated by monitoring indicators that point towards a decrease of regime resistance against the chosen niches. In case the indicators do go in the expected direction, we still need to evaluate to what extent such can be assigned to the transition program. As such, we need to tackle the issue of causality and that is exactly what we do in the last step when reconstructing causal chains using the modus narrandi methodology. For all six steps we give very concrete evaluation questions that should be posed by the evaluator and/or clear indicators that can be used. Table 8 in the paper provides a visual overview of the total six-step evaluation tool and is shown at the end of this summary as well. In the paper, many concrete examples are provided from the transition policy programs in the Belgian region of Flanders, in the context of which this research project has been set up. In the continuation of this project, we will apply the evaluation tool on existing transition programs so to make further improvements and refinements. In particular, we will attempt to improve the evaluation questions and indicators so to make the tool even more widely applicable to all kinds of transition programs (for technological as well as social transitions) and we will investigate the possibilities to evaluate also the role of other partners in the transition arena except for the government. Finally, while we focus strongly upon transition programs that nurture niches in this paper, the practical examples throughout the paper already clarify that niches are to be defined in a wide sense. In further research, we will investigate whether the evaluation tool can be used for transition programs in a broader sense. Evaluating long-term transition programs on a short-term basis vii

Evaluation step Evaluation criteria Evaluation method Type of evaluation Transition phase Evaluation questions: Program theory failure? 1 Internal program consistency - What are the long-term policy objectives of the transition program? - What are the chosen niches in the transition program? - What is the potential impact of the chosen niches and are those consistent the long-term policy objectives of the program? Potential impact Stabilization Checklist: Application deficit? 2 Crucial process activities - Setup of a transition arena for a specific socio-technical system - Development of a long-term vision on the system within the arena - A multi-actor, multi-level and multidomain approach is chosen, but with a single point of coordination - Setup of learning moments Process Predevelopment Checklist: 3 Crucial policy activities - Stimulating R&D for the chosen niches - Levelling and turning the playing field - Shifting public resources - Avoid regime reinforcement Output Predevelopment Evaluation questions: Policy instruments implementtation Implementation deficit? 4 - What is the spirit or the aim of the policy instrument? - What are the essential elements for good implementation? - Are those elements present? - If not, wherein lies the problem? Process Predevelopment viii Tom Creten, Sander Happaerts & Kris Bachus

Program theory failure? 5 Decreasing regime resistance Indicators: - Powerful actors have embraced the niches - Niche players are becoming powerful actors - Increasing market share for the niches - Increasing efficiency (performanceprice ratio) of the niches - Decreasing complexity of the niches for end users - Emergence of network organizations for the niches - Increasing public support Outcome Take-off & Acceleration 6 Causality between program and outcomes Reconstruction of causal chains by using the Modus Narrandi method Causality assessment From predevelopment to take-off & acceleration Table 1: Transition program evaluation tool extended version Evaluating long-term transition programs on a short-term basis ix

Introduction Sustainability transitions are becoming more common as a framework for sustainable development policies, encompassing but not limited to environmental issues. In transition programs (policy programs inspired by transitions), the paradigm of transitions thinking is used by policy-makers to decide upon policy interventions in socio-technical systems (e.g. the housing, food or energy system) to solve the persistent problems of sustainable development. In the Belgian region of Flanders, for instance, the Flemish government has formulated a sustainable development strategy based on the transitions framework, and it has additionally identified 13 transversal transition programs to achieve sustainable change in systems as diverse as energy, care, mobility, etc. (Vlaamse overheid, 2012d). 1 This paper is part of a research project, commissioned by the Flemish government, on the evaluation of the transition approach. It is a first attempt to develop a transition program evaluation tool based on a thorough analysis of different strands of literature. In next papers, the tool will be improved after a concrete application on a number of Flemish transition programs. After completion, the tool is intended to be used in the first place by the Flemish administration in the evaluation of its transition programs, and second as a contribution to the scientific literature on the evaluation of transition policy that could be useful for external evaluators. We build upon the program theory evaluation literature, in which it is taken as an assumption that policy-makers have in mind a well-defined theory about how their policy program will influence the real world (Crabbé et al., 2006). In the transitions literature the process of system change is described step by step and the way policy can have an influence on those transitions is well described in certain streams of the literature, such as the transition management approach (Loorbach, 2007). Taking this literature as a basis, there usually is indeed a clear policy theory in policy program and they can for that reasons be considered to be transition policy programs in the way programs are perceived in the literature on program theory evaluation. We assume transition (policy) programs to include at the minimum aspects that are intended to stimulate niches on the one hand and destabilize the existing regime to enhance the possibilities of those niches to break through on the other hand. Such a strategy for the program should also fit within a long-term view on system change, concretized by a number of long-term objectives for the system under consideration that ought to be reached by the program. Despite a large body of literature on sustainability transitions, surprisingly little attention has been given to the evaluation of such comprehensive transition programs. We attempt to fill that research gap by proposing a six-step evaluation tool for transition programs. The tool should be suitable for both extended external evaluation of transition programs, as well as for more modest internal evaluations and learning activities performed by the civil servants and 1 The Flemish sustainable development strategy puts forward seven transitions, namely in 1/ housing and building 2/ material use; 3/ energy; 4/ mobility; 5/ food; 6/ health care and 7/ knowledge and learning. The thirteen transversal transition programs were identified in the context of the most important Flemish policy strategy called Flanders in Action. Those thirteen transition programs are: 1/ New Industrial Policy; 2/ Gazellesprong (on the growth and internationalization of enterprises); 3/ Streamlining a directed innovation policy 4/ iedereen mee, iedereen actief (on the match between education and labor market); 5/ The fight against child poverty; 6/ Flanders Care (on the health care system); 7/ Renewable energy and smart grid; 8/ sustainable housing and building; 9/ sustainable material use; 10/ Ruimte voor morgen (on spatial planning); 11/ Smart mobility; 12/ Speeding up investment projects; and 13/ Towards a sustainable and creative city. Evaluating long-term transition programs on a short-term basis 1

other actors involved in the programs. The tool is developed through the combination of the limited literature on evaluation in transition theory and the more elaborated environmental evaluation literature. Using program theory evaluation as an overarching paradigm, that combination makes it possible to investigate three different potential policy problems. First, we can discover so-called application deficits that indicate a policy theory (i.c. the program theory based on transitions thinking) was not well applied by the policy-makers. Next, there could be an implementation deficit, meaning that the specific policy instruments chosen to stimulate transitions were not well implemented. Finally, we might identify a policy theory failure, which means the transition program theory itself is not the best approach for the policy problem under consideration. The first and second problem can be discovered mainly by a thorough process evaluation, which is the most common type of evaluation in the transitions literature. The third problem of a theory failure (but also partly the application deficit problem) is only possible to investigate through an evaluation of the (intermediate) results of the transition program. The latter is rather rare in transition studies and this is where the combination with concepts and methods from the traditional environmental evaluation literature constitutes a genuine added value for the evaluation of transition programs. While the tool is unique in the fact that it is able to evaluate a transition program comprehensively, it could nevertheless also be used to evaluate parts of the program. In particular, it could be useful to investigate the effects of the stimulation of one specific niche rather than all niches that are nurtured within the program. The evaluation of that part would still be a full or comprehensive evaluation nonetheless, given that the tool for that part still makes it possible to evaluate both the process aspects and the results of nurturing that specific niche and to discover both the problem with application and implementation deficits and the problem with theory failure. The paper is structured as follows. Part 1 elaborates on the theoretical framework. We give an overview of the most inspirational literature in the field of transition studies and in the field of environmental evaluation. After that, we introduce a first compact version of the transition program evaluation tool. Part 2 continues with a thorough discussion of all six steps of the tool and ends by presenting a more elaborated version of the tool. In part 3 we reflect upon the generalizability of the tool to any kind of transition program. Finally, part 4 draws a number of conclusions and formulates an agenda for further research. 1. Theoretical framework 1.1 Evaluation in the transitions literature In the literature on transitions, there are four main strands of theory that dominate the work of scholars (Markard et al., 2012). First, there is the Multi-Level Perspective (MLP), strongly influenced by the work of Geels (2002), that analyzes transitions of socio-technical systems (e.g. the housing, transport or food system) as the consequence of an interplay between a macrolevel (landscape), a mesolevel (regime) and a microlevel (niches). Second, the transition management approach focuses on the role of policy-makers in transition processes and emphasizes the need to set up multistakeholder collaborations to find long-term solutions to solve persistent sustainability problems in socio-technical systems (Loorbach, 2007; Rotmans & Loorbach, 2009). A third strand of literature is known as strategic niche management and concentrates on the importance of developing and nurturing experiments on the microlevel as a way to strengthen system innovation 2 Tom Creten, Sander Happaerts & Kris Bachus

(Kemp et al., 1998). Finally, the Technological Innovation Systems (TIS) literature is concerned with informing policy-makers on how to encourage the development and diffusion of new technologies for system change (Bergek et al., 2008; Hekkert et al., 2007). As Mazur et al. (2014) point out, the attention in the different transitions literature streams for the evaluation of policy interventions in the context of transitions has been limited (see also Bussels et al. (2013). The MLP has mainly focused on developing ideal-typical transition pathways in which the interplay between the three levels happen in different ways and which can explain transitions taking place in the present and the past, but with very limited attention for drawing policy lessons (Geels & Schot, 2007; Van Bree et al., 2010; Verbong & Geels, 2010). Strategic niche management has given some attention to evaluating the success of policy interventions in encouraging the breakthrough of certain niches. Ieromonachou et al. (2004), for instance, have formulated the Strategic Policy Niche Development (SPNM) methodology to explain the success of the introduction of the Durham road access charging scheme and to make recommendations for introducing comparable schemes or other (policy) niches elsewhere. Albeit interesting, the approach remains limited to evaluating policies in stimulating specific niches and also does not evaluate the effect of such a policy on actual system change. The same is true for the TIS literature that focuses on drawing policy lessons for developing specific technologies (e.g. solar panels), but again does not provide evaluations of the results of complete transition programs and their effects on system change (Bergek et al., 2008; Hekkert et al., 2007). Both streams of literature also have limited attention for the transition policy process. The transition management literature then again is limited in its scope for evaluation exactly because it focuses almost exclusively on the evaluation of process aspects. Indeed, in that strand of literature there are long lists of prescriptions on what policy-makers should and should not do to support a transition process (e.g. setting up a transition arena, developing a long-term vision together with the transition stakeholders, working with back- and forecasting, etc.) (Loorbach, 2007). To evaluate whether or not all of those prescriptions are fulfilled, transition management stresses strongly the importance of reflexive learning among the actors involved as a way to achieve ex durante self-evaluation about the process (Bussels et al., 2013). Hence, there is a great deal to learn from the transition management literature when it comes to process evaluation and we will integrate this in our work. The transition management literature brings us however little insight in the evaluation of results. As such, for as far as transition scholars have given attention to policy evaluation, it has been limited in scope. There are few evaluations of complete transition policy programs. However, with transition governance becoming more and more common as a framework for government policies on sustainable development, there is a growing need for evaluation tools that are able to evaluate complete programs. In that context, there are two noteworthy attempts that we will discuss below. Commonly interesting to both methods is that they try to evaluate to some extent both the policy process and the results of the policy program. Obviously, it is taken into account that the real impact of transitions can only be witnessed in the long run, but the authors note that some first other results should be observed already in the predevelopment and take-off phases, i.e. the first two out of four phases according to the transition literature. The predevelopment phase is mainly characterized by experimentation, which could be witnessed and should be actively encouraged by government. In the take-off phase the first indications of system change should be visible. On the longer term the transition will be speeded up during the acceleration phase when changes in the system accumulate and finally come to a new (dynamic) equilibrium in the stabilization phase (Loorbach & Rotmans, 2006). Such a massive system turnover and the emergence of a new Evaluating long-term transition programs on a short-term basis 3

equilibrium usually take a very long time and as such it is a reasonable approach only to evaluate the effects of transition policy in the first two phases. First of all, Mazur et al. (2014) have identified four steps to evaluate policy interventions for system change. More specifically, they analyze which of the ideal-typical transition pathways developed by Geels & Schot (2007) are the most likely to occur for the system under consideration (i.c. the transport system) and they investigate to what extent current policy interventions are encouraging those pathways. Perhaps the most interesting part of their analysis is the fact that they do not attempt to decide upon themselves what a sustainable transport system should look like in 25 years. Rather, they choose to examine the long-term sustainable transport policy objectives that the government has put forward and take those as a given. Next, they investigate whether or not current policy interventions are likely to lead to those policy objectives. That approach avoids a tricky discussion on what sustainability is and what a sustainable system should look like, which is in fact a normative and political debate. For that reason, we will operate in the same manner. However, the four ideal-typical transition pathways are in our opinion too compelling. As a consequence, policy interventions that actually encourage a transition can be evaluated negatively if they do not fit into the pathway that should have been chosen according to the authors. On the other hand, discouraging policy interventions can be legitimized as long as they fit into the preferred pathway. For instance, the attempts of Germany to weaken EU CO 2 emission limits for vehicles is considered by Mazur et al. (2014) not as a German attempt to block the transition towards electrical vehicles, but actually as a logical step in the incremental transformation pathway that Germany is following to adapt its regime. Arguably that way any policy intervention could be considered to be beneficial for transitions. That is why we do not copy that method. The second four-step evaluation tool was developed by Ros et al. (2006) in the context of the fourth Dutch National Environmental Policy Plan (NMP4) which brought transitions to the forefront in the Netherlands. In that approach, there is an assessment of specific policy programs that aim to stimulate innovating system options (strongly comparable to niches) so to make specific systems more sustainable. Although strongly focused on the microlevel, the approach is very much in line with what we need when it comes to evaluating comprehensive policy programs. The tool is specifically interesting for its flexibility in evaluating the programs. Rather than working with predefined transition paths, a number of crucial policy activities are defined that are commonly necessary during the predevelopment phase for any transition pathway taken. Those crucial activities include the development of a long-term vision among the relevant actors, the stimulation of R&D for niches and the set-up of experiments (Ros et al., 2006). Next, it is investigated (using a number of indicators) whether or not those policy interventions decrease regime resistance against the niche innovations under consideration. Decreasing regime resistance indicators (or in other words indicators that show niches are breaking through) can be expected to evolve positively during the take-off phase and/or acceleration phase. The concepts of crucial policy activities and regime resistance are interesting and flexible to use for any kind of transition program evaluation and we will integrate those two steps in our evaluation tool. We will however adapt and broaden the identified crucial activities and indicators for regime resistance, as we believe a number of essential transition policy elements are missing and some of the regime resistance indicators need refinement. Also, as mentioned before, we prefer to take policy objectives as a given and we do not attempt to determine ourselves how a sustainable system should look like as Ros et al. (2006) do. Summarizing, the development of our evaluation tool in this paper builds upon the integration of several concepts and ideas from the existing transitions literature. One step in our tool will be on process evaluation and will be strongly based on the policy prescriptions from the transition 4 Tom Creten, Sander Happaerts & Kris Bachus

management strand of literature, complemented with insights from earlier phases of this project (Bussels et al. 2013; Bussels and Happaerts 2014). From Mazur et al. (2014) we learn not to get lost in the discussion on what a sustainable system looks like and we take over the idea of testing policy interventions against long-term sustainability goals formulated by government itself (possibly together with transition stakeholders) within the system under investigation. Finally, we integrate the concepts of crucial activities and indicators of regime resistance developed by Ros et al. (2006). Nevertheless, we still do not have enough elements to perform a comprehensive evaluation of a policy program aimed at achieving a transition. Most importantly, the matter of causality has not been tackled. Ros et al. (2006) claim that certain policy interventions could have had an influence on specific indicators of regime resistance. Nevertheless, their claims are highly speculative. Causality between policy interventions and results is however of crucial interest to policy-makers. To find an appropriate methodology that can elucidate causal links, we should turn to more traditional environmental evaluation literature. 1.2 Environmental evaluation literature In the environmental domain there is a very broad literature on evaluation. In this section we will not provide a comprehensive overview of the environmental evaluation literature, but we will touch upon a number of elements that we use in the development of our evaluation tool. First of all, the literature on environmental evaluation can provide us with arguments that the evaluation of the results of policy programs for transitions is indeed possible. Some transition scholars have neglected the evaluation of results, justified by claiming that evaluating the impact of government actions on the sustainability of a socio-technical system is a mission impossible given the long-term framework and complex circumstances (Bussels et al., 2013). Yet, as Ros et al. (2006) or Mazur et al. (2014) have demonstrated, policy interventions aimed at encouraging a transition can have other (more short-term) effects in the predevelopment or take-off phases that are not yet a full proof of impact in the sense of system change, but that are at least showing that the probability of a transition taking place is increasing. Here it is relevant to point out the traditional environmental policy evaluation distinction between impact, outcome and output effects of a policy program (Gysen et al., 2006b): The impact effects refer to the final policy objectives, the reason why the policy program is set up in the first place. In the case of transition programs such would be the aim of system change or the effects of the policy program on the sustainability of the system. A concrete example could be the improvement in air quality or even the improved human health that is realized thanks to a transition program on mobility. Typically, transitions need time and system change can only be witnessed on a long term, thereby making it indeed almost impossible to evaluate the impact of transition programs on a short term. The outcome effects occur one step earlier and refer to the effects of the policy program on the behavior of target groups within the system. In transition programs those could occur on a short to mid-term basis. As such, we might be able to grasp the first outcome effects in a short-term evaluation. An example could be the increased number of people that buy electrical cars and the resulting increasing market share of electrical vehicles. Evaluating long-term transition programs on a short-term basis 5

Finally, the output effects take place in a very early stage and refer to the short-term direct policy actions that are taken in the context of the program. The output effects are easily witnessed and form a first step in investigating causal chains that link a policy program to outcome and final impact effects. For instance, the amount of tax cuts given within a transition program to car users that choose to buy an electrical car would be an output effect. Now, the three types of effects also fit perfectly into the framework of program theory evaluation. The idea behind program theory evaluation is that policy-makers have a very clear theory about how their policy program will influence the real world (Crabbé et al., 2006). It is the task of the evaluator to check whether or not that theory makes sense and as such the right approach is taken to achieve the desired results. In our case, we know the transitions literature is the basis of the transition program theory. More specifically, the transition management literature stream provides assumptions on how certain process and policy activities will lead to certain results. Apart from that, there are also assumptions in the minds of the policy-makers and civil servants on how (within the general transition framework) specific policy instruments will have an effect on the behavior of target groups (outcome effects) and finally on the sustainability of the system (impact effects). The mixture of both forms the policy theory (i.e. the transition program theory) which drives the decisions that are taken within the program. Although the specific program theory will be different in each transition program, there is a common basis to all transition program theories given that all are to a large extent based upon the transitions literature. In all transition program theories, the three types of effects described above are the results of a certain policy input, which are the resources and energy that are put into the transition program. The program should consist out of a number of crucial process aspects and that process should lead again to a number of crucial policy activities to stimulate niches. The specific ways in which that happens, will depend upon the specifics of the program theory, but all transition programs should carry out those activities in some way as those are crucial to the transition framework. The output effects then refer to which extent those crucial policy activities took place. Next, the outcome effects are the effects on target groups who are progressively taking up the chosen niches, indicating that the regime resistance is decreasing and niches are breaking through. Finally, the impact is the system change towards a more sustainable character which is the final objective of the program. The transition framework states that in case a transition process is set up in the right way and the right crucial activities are undertaken to stimulate niches, then automatically there should be a decrease in regime resistance against the niches and finally this should lead to sustainable system change. That causal chain is part of each transition program theory and is presented in figure 1. 6 Tom Creten, Sander Happaerts & Kris Bachus

Input Output Outcome Impact Policy process Transition program Crucial policy activities Decreasing regime resistance System change Application & implementation deficits Theory failure Figure 1: Theory aspects common to all transition program theories Obviously, we would like to check whether or not the theory that forms the basis of the transition program is valid or whether the theory fails to yield (sufficient) results. However, to identify a theory failure, we first need to know whether or not the theory that the policy-makers put forward was converted to practice as foreseen. If not, it is not the theory itself that fails but rather the translation of the theory into actual policy deeds. Given that the policy theory in transition programs is developed from two elements, there are also two possible problems with conversion of the theory that should be investigated. As such, there is triple challenge for the evaluator 2 : 1. The evaluator should investigate the possible occurrence of an application deficit. In that case, the transitions literature was not well applied in the development of the policy theory. Essential elements that should be common to any policy theory based upon transitions thinking are missing in reality, which explains why success is impossible. It corresponds to an evaluation of the crucial process activities and crucial policy output activities that should be present in each transition program. 2. The evaluator should examine the possibility of implementation deficits. Apart from the common elements that should be present in each transition policy theory, the policy- 2 Typically, there is only a double challenge identified in program theory evaluation, namely to detect on the one hand implementation deficits and on the other hand theory failures (Crabbé et al., 2006). Usually, the policy theories evaluated are rather simple theories assuming that one specific policy instrument will lead to a number of specific effects. For instance, a new tax will lead to less polluting behavior and finally will improve environmental quality. In such a case, the implementation deficit would refer to a bad implementation of the specific policy instrument, namely the tax (e.g. no enforcement policy). In transition programs however, the policy theory is much more elaborated. Apart from the effect of very specific policy instruments, there is also the general assumptions about the effects generated by carrying out the crucial process and output prescriptions that are common to all transition programs. Therefore it is also necessary to investigate the possible occurrence of an application deficit. Evaluating long-term transition programs on a short-term basis 7

makers will also at some point make a choice for specific policy instruments. Those specific instruments are needed to carry out the more general crucial policy activities in practice and will be different for each transition program. In other words, it will be evaluated whether the actual policy is implemented in the right way. It corresponds to an evaluation of the policy process during the phase of policy implementation. Here it is not the idea to evaluate the presence of process aspects that should be common to all transition programs, but rather to evaluate the process of implementation of the program specific policy deeds. 3. Finally, there is the possibility of a policy theory failure that should be evaluated. In that case, it is the transition program theory itself that proves to be invalid, at least for the policy problem under consideration. It means the policy-maker s assumptions and expectations (based for a large part upon the transitions literature, but also on assumptions on specific policy instruments) about the consequences of the chosen policy were wrong. Such corresponds to evaluating the outcome effects and (if possible) the impact that the process has yielded and also evaluating the causal relation between those latter effects and the crucial policy activities undertaken within the transition program. As such, it becomes clear why evaluation of both the process and the results yielded by that policy process (in terms of output, outcomes and impact) becomes indispensable for policy-makers. The first is important to reveal implementation deficits and to adjust the program if necessary, while the second is necessary to know when to switch to another general approach in case the transition program theory does not yield sufficient results. To investigate the application deficit, it is only a matter of monitoring whether or not the crucial process and crucial output elements have been carried out within the transition program and the examination of implementation deficits can be done by thorough process analysis in the phase of implementing the policy instruments. The tracing of a theory failure is probably the most complex. Fortunately, within the context of theory based evaluation, a methodology has been developed which allows us to investigate causality between the crucial government activities in the predevelopment phase and indicators of niches breaking through (or decreasing regime resistance) in the later phases of the transition. Despite the fact that there is no such thing as proving causality, the modus narrandi is a method developed by Gysen et al. (2006b) that attempts to clarify causal mechanisms and shares some similarities with methods like theories of change and realistic evaluation (Blamey & Mackenzie, 2007). The idea behind the method is to reconstruct causal chains and identify in each step both the causal proximity with the next step and the causal contribution of the preceding step in the next one. The causal proximity refers to what extent the first step logically can influence the next one theoretically. The causal contribution is an indication of the strength of that influence in reality (Gysen et al., 2006b). How the modus narrandi works in practice will become clear when applying it to an example in later sections. The modus narrandi methodology has already been used in several Flemish and Belgian environmental and sustainability policy evaluation studies (e.g. Gysen et al (2006a)) Given that in transition programs the impact (system change) can only be investigated on the long term, it is already a big step to elucidate the causal relation between the policy outputs on the one hand and the outcome effects on the other hand and it is exactly our intent to do that by using the 8 Tom Creten, Sander Happaerts & Kris Bachus

modus narrandi. Nonetheless, we will not fully neglect the impact effects, as our evaluation tool will start of by reflecting upon the potential impact of the policy program on system change and more specifically on the predetermined policy goals of the program. 1.3 A first glance at the transition program evaluation tool The combination of the transitions literature with the environmental evaluation literature leads to the development of a new transition program evaluation tool that uses the program theory evaluation approach as an overarching paradigm. The tool is useful for the evaluation of policy programs that are based on the transition literature and that include at the minimum aspects that intend to stimulate niches on the one hand and destabilize the existing regime on the other hand in order to reach a number of long-term system objectives. In Table 2 we introduce briefly the six steps of the evaluation tool that we have developed. In the next part, we will elaborate more thoroughly on all six steps. Evaluation step Evaluation criteria Type of evaluation Transition phase Main source Program theory failure? 1 Internal program consistency Potential impact Stabilization Mazur et al. (2014) Application deficit? 2 Crucial process activities Process Predevelopment 3 Crucial policy activities Output Predevelopment Transition management Ros et al. (2006) Implementation deficit? 4 Policy instruments implementation Process Predevelopment Policy theory evaluation 5 Decreasing regime resistance Outcome Take-off & Acceleration Ros et al. (2006) Program theory Failure? 6 Causality between program and outcomes Causality assessment From predevelopment to take-off & acceleration Modus narrandi (Gysen et al., 2006b) Table 2: Transition program evaluation tool a first glance Evaluating long-term transition programs on a short-term basis 9