EVALUATING AND MONITORING TRANSITION INITIATIVES Lessons from a field scan

Transcription

1 EVALUATING AND MONITORING TRANSITION INITIATIVES Lessons from a field scan Matthias Bussels, Sander Happaerts & Hans Bruyninckx HIVA KU Leuven Promotor: Hans Bruyninckx Research paper 5 Leuven, may 2013

2 The Policy Research Centre Transitions for Sustainable Development brings together researchers from KU Leuven, Ghent University, VITO and Erasmus University Rotterdam. Please refer to this publication as follows: Bussels M., Happaerts S. & Bruyninckx H. (2013), Evaluating and monitoring transition initiatives. Lessons from a field scan, Research paper 5, Policy Research Centre TRADO, Leuven For more information matthias.bussels@kuleuven.be This publication was sponsored by the Flemish Government, under the Policy Research Centre Programme. This publication reflects the views only of the author, and the Flemish Government cannot be held responsible for any use which may be made of the information contained therein POLICY RESEARCH CENTRE ON TRANSITIONS FOR SUSTAINABLE DEVELOPMENT p.a. Administration Policy Research Centre on Transitions for Sustainable Development HIVA - Onderzoeksinstituut voor Arbeid en Samenleving Parkstraat 47 box 5300, BE 3000 Leuven, Belgium This publication is also available via

3 Contents Managementsamenvatting iv Introduction 1 1. Transitions for sustainable development Sustainable development as a persistent issue Sustainability transitions & transition thinking 3 2. Evaluation and monitoring in the context of sustainability transition initiatives The necessity and utility of monitoring and evaluation Challenges to M&E in STIs Result based evaluations Evaluating the STI process 9 3. Examples of M&E approaches for STIs Reflexive monitoring The Applied Systemic Programme Evaluation framework Contribution analysis Conclusions Lessons & suggestions for evaluating transition initiatives Evaluating STIs: existing practices in Flanders 19 Further research 22 Bibliography 23 Evaluating and monitoring transition initiatives iii

4 Managementsamenvatting Deze eerste paper van onderzoekslijn 2, Evaluatie en monitoring van transities, bespreekt huidige praktijken en uitdagingen van het evalueren van transitieprocessen, met het doel om aandachtspunten te formuleren. Duurzaamheidstransities, een kernwoord binnen het Vlaamse middellange en langetermijnbeleid, kent doorgaans een invulling in de vorm van het oprichten van arena s waarin verschillende stakeholders samenzitten om visies en strategieën uit te denken voor een gewenste transitie van het beoogde socio technisch systeem: duurzaamheidstransitieinitiatieven. Dergelijke processen bestaan uit intense leerprocessen, waarbij de verschillende partners nadenken over hun positie in het systeem, de hardnekkige problemen waarmee het systeem worstelt en voornamelijk hoe andere partners daarover denken. Samen poogt men dan te komen tot een gemeenschappelijke actiebasis. Dergelijke processen zijn echter moeilijk, intensief en onvoorspelbaar. Dat maakt dat gebruikelijke evaluatiebenaderingen (waarin evaluatie gezien wordt als een ex antebeschouwing van wenselijke verandering, die lineair verloopt en waaraan productindicatoren en strakke trajecten gekoppeld kunnen worden) minder geschikt zijn voor het opvolgen van die processen. Zulke benaderingen verhinderen de flexibiliteit en openheid die nodig zijn bij dergelijke exploratieve processen gericht op complexe of hardnekkige problemen: enkel tijdens het proces wordt het duidelijk (mits voldoende reflectie) waarop ingezet zou kunnen worden. Toch is het nodig om dergelijke processen te monitoren en te evalueren. Enerzijds moeten de deelnemers verantwoording afleggen voor hun betrokkenheid en voor de middelen die via hen in de initiatieven geïnvesteerd worden. Anderzijds hebben de deelnemers verschillende noden, verwachtingen en handelingsmogelijkheden. Doorheen het interactieproces wordt er voortdurend geleerd, maar dat leren blijft vaak informeel en onuitgesproken. Het uitvoeren van monitoring en evaluaties helpt om dergelijke zaken naar buiten te brengen, en kunnen het proces voeden en kunnen ook aan de vraag naar verantwoording tegemoetkomen a.d.h.v. een notie van vooruitgang. Daarom vermelden we een aantal aandachtspunten voor evaluatie inspanningen gericht op duurzaamheidstransitieinitiatieven, waarbij we ons baseren op een bespreking van drie evaluatiekaders ontworpen met het oog op complexe processen. De voorbeelden en de bespreking maken duidelijk dat een procesgebonden focus noodzakelijk is, waarmee bedoeld wordt dat evaluatoren zich niet blind moeten staren op het behalen van vooraf bepaalde uitkomsten of deadlines, maar de participanten aan het proces zelf moeten bijstaan bij het verschaffen van procesgerelateerde inzichten, sterktes en werkpunten. Een tweede aandachtspunt betreft het behouden van flexibiliteit en aanpasbaarheid in het proces dat, zeker in de exploratieve beginfasen, de ruimte moet krijgen om te reflecteren en te zoeken een strak tijdsschema en productindicatoren waarop gerapporteerd moeten worden kunnen dat in de weg staan. Ten derde vermelden we de noodzaak om voortdurend aandacht te hebben voor het evalueren. Voorbeelden tonen aan dat evaluaties, als ze al gebeuren, vaak te laat komen en op een ad hocbasis. Evaluaties dienen het proces net te voeden wat wijst op een ex durante benadering van evaluatie. Daarmee raken we ook aan de positie van de evaluator, die eerder met de rol van een procesbegeleider dan van een verslaggever overeenkomt, aangezien hij of zij een belangrijke rol speelt bij het aanvoeren van procesgerelateerde informatie. Ten slotte benadrukken we dat, bij afwezigheid van directe resultaten en de moeilijkheid om causaliteit te bewijzen iv Matthias Bussels, Sander Happaerts & Hans Bruyninckx

5 in het geval van dergelijke complexe processen, een focus op leren een houvast en focus biedt voor evaluaties: leren leidt idealiter tot handelen, en het is dan ook cruciaal om te kunnen vaststellen waar mogelijke handelingsruimte gecreëerd wordt aan de hand van het proces een onderwerp dat verder uitgewerkt zal worden in het verdere onderzoek. Evaluating and monitoring transition initiatives. Lessons from a field scan v

6

7 Introduction This paper forms the output from the first work package within the second research stream evaluation and monitoring of sustainability transitions of the Policy Research Centre on Transitions for Sustainable Development (TRADO). It explores the possibilities and challenges that present themselves when one tries to evaluate initiatives modelled on the transition governance framework. In doing so, it takes a look at various theories and approaches to evaluation, and combines these with, or rather sets them off against the principles of transition thinking. The information gathered from the literature study is deepened and put into perspective through interviews with experts on both transition studies and evaluation and monitoring. The goal of the paper is therefore to infer current knowledge gaps and correspondences, leading to the proposal of some preliminary recommendations that need to be kept in mind when evaluating sustainability transitions. The structure of this paper is as follows: in first instance we will elaborate on the concept of sustainability transitions and transition governance, after which we will introduce the concepts of sustainability transition initiatives as exploratory multistakeholdernetworks functioning as a first step in pushing transition governance. After that, the importance of monitoring and evaluating such processes is emphasised and contrasted with the conventional approach to evaluation, i.e. ex ante assessment using result based indicators and assuming a linear change model. Thirdly, three evaluation approaches that provide an alternative to that view are discussed, feeding into a fourth section bundling suggestions and summarizing the arguments spelled out in the paper by way of practical illustrations. A final section on further research concludes this paper. Evaluating and monitoring transition initiatives. Lessons from a field scan 1

8 1. Transitions for sustainable development 1.1 Sustainable development as a persistent issue Scholars focusing on sustainable development as a concept denoting the awareness of limits to our model of societal development, increasingly recognise issues of unsustainability as systemic flaws, i.e. problems that are deeply rooted in the functioning of our societal systems (Dresner, 2002; Kemp & Grin, 2009; Kemp & Rotmans, 2001; Rotmans et al., 2001). Consider for example the societal mobility system, providing possibilities for transport for people and goods. In fulfilling its systemic function, it produces very diffuse and collateral, yet grave risks. Merely the issue of congestion poses infrastructural demands resulting in environmental pressure, pressure on family cohesion as working parents are caught up in traffic and get to spend less time with their family, etc. Another example can be found in the food and agriculture system: in aiming to provide healthy and sufficient nutrition for consumers, it favoured a meat intensive diet (in Western societies). However, the increase in meat consumption globally induces an increase in the demand for land, livestock feed, shorter production processes and concomitant health hazards, increased carbon emissions etc. Sustainable development therefore envelops problems that are systematically produced (Beck et al., 1994, p. 6). Such processes are defined as persistent (Loorbach, 2010) issues because they are - Complex: involving multiple levels of governance, multiple scales of causality, and requiring multiple actors for their effective resolution - Interdependent, e.g. the multiple applications of land use (energy, agriculture, infrastructure), the link between transport, energy and climate change, etc. - Uncertain: how will the future play out and what do we want, where are thresholds and tipping points, what is the chain of causality underlying policy interventions - Deeply embedded in the fabric of society: the problems are linked to our dominant patterns of production and consumption, our way of life and everything that we put in place to maintain that (Loorbach, 2007, p ). Consequently, without addressing the very way society is modelled, reassessing its core values, the challenges will multiply and increase in size. Sustainable development should therefore be aimed at, and based on, systems and system change. Such change at the level of systems transcends so called system improvement (doing things better, without reflecting upon the desirability of those things themselves) (Dresner, 2002). A radical system innovation, or transition, is needed (Kemp & Rotmans, 2005, p. 34). Consequently, the term transition has gradually (since the late 90s) found its place beside sustainability, 1 to denote the latter s indispensable system transformative character. 1 We will use the concept of sustainable development interchangeably with sustainability. For example, the sustainability of a societal system derives from its development along the lines of a sustainable development model. 2 Matthias Bussels, Sander Happaerts & Hans Bruyninckx

9 1.2 Sustainability transitions & transition thinking When referring to systems in transition thinking, the objects of analysis are so called socio technical systems: a cluster of elements (technology, science, regulation, consumer practices, markets, cultural symbols, infrastructure, and networks of production and supply) and their interconnections, satisfying a societal need (mobility, food, energy, housing,...) (Paredis, 2009, p. 15). Socio technical systems are considered to be made up of three pillars, i.e. cultures, structures and practices, 2 that converge into fulfilling said societal need, i.e. its function (such as transporting people or providing food) (Frantzeskaki & de Haan, 2009). Acknowledging that societal system transformations are at the heart of sustainable development, the concept of transitions is used to explain the dynamics of socio technical system change as well as how they might be stimulated and nudged into a desirable direction. As such, the discourse on transitions can be divided into two main groups. The first group deals with historical analyses of (societal or sectoral) transitions (often termed the sociotechnical approach, e.g. Geels, 2002), in an attempt to understand the dynamics of socio technical system change or transitions. The second group, which could be labeled as approaches of transition governance, focuses on how to bring about, or at least influence societal transitions they are future oriented (Kemp & Grin, 2009; Paredis, 2009, p ). The underlying conviction is that it is possible, and therefore desirable, to accelerate the shift (Sterling, 2007, p ) towards a sustainable society in order to prevent major crises or system collapses. Approaches to transition governance often suggest the organisation of what we have come to call sustainability transition initiatives (STIs): small scale networks, creating an interface between actors from various institutional realms, and in doing so fostering learning through participation and interaction (Loeber et al., 2007, p. 92). Those initiatives comprise a wide variety of actors that each hold a stake in the current system, and engage in a discussion on how they envision the current system flaws, how they might collaborate to support a sustainability transition, and how they perceive of their role and position in the future system. Such exploratory visioning exercises are expected to lead to changes in the practices of the network members, and feed into or spur ongoing changes at the societal level (van den Bosch & Rotmans, 2008, p. 22). Possible (more tangible) outcomes of such explorative, strategic and multistakeholder exercises are experiments that build on the acquired insights, or new collaborative networks 3 or organisations. A third expectation in terms of expected results that might engender socio technical system transitions, can be described as the oil spill effect, where STI participants translate the acquired insights into practice within their own environment, network or organisation (Loorbach, 2007, p. 99). The objective of STIs can be framed within the multi level perspective (MLP), a heuristic developed by Rip and Kemp (1998) in order to analyse transitions in socio technical systems. The MLP asserts that transitions come about as an interplay between different societal levels, i.e. niche, regime and 2 3 In which cultures refer to the appraisal and legitimacy of the functioning of the societal system (how it is perceived) ; structures refer to the infrastructures and institutions related to the functioning of the societal system (how it works) (De Haan, 2010, II p. 42); and practices stand for the routines, habits, formalisms, procedures and protocols by which actors, which can be individuals, organisations, companies, etc., maintain the functioning of the societal system (de Haan & Rotmans, 2011, p. 92). An example that can serve to illustrate this point, is the consolidation of the ongoing collaboration between STI participants to the Flemish transition programme on waste/materials management ( Plan C ) through the formation of a nonprofit organisation. Evaluating and monitoring transition initiatives. Lessons from a field scan 3

10 landscape level. While various transition trajectories are possible (de Haan & Rotmans, 2011), most approaches to transition governance focus on niche initiatives, such as STIs, to exert pressure on the regime (establishing the conventional cultures, structures and practices) in order to eventually effect a socio technical system transition towards sustainability. 4 The following graph can elucidate the dynamics at play, and again emphasise the importance of learning processes. Figure 1 The MLP and the position and functioning of STIs Source: based on Geels and Schot (2007) In brief, STIs - are purposefully created explorative platforms (by interested parties, be they governmental actors, societal or civil organisations, grassroots citizen consumers networks,...); - explicitly target systems (that might be an economic (sub )sector cf. the Flemish transition process on sustainable housing and living ( Duurzaam Wonen en Bouwen ), a city (cf. Leuven s strategy for climate neutrality) for transformation (the threshold to be able to speak of a true transition to be decided upon through participation and discussion); - have/display the ambition to achieve, in the long term, a sustainable system state; - and engage the participants in processes of social learning. 4 The concept critical mass is used in theories on technological diffusion (Rogers, 2003) to denote the tipping point, that will lead to a rapid increase of the adoption of novel technologies (or, in transition terms, an accelerated transformation of the socio technical system). 4 Matthias Bussels, Sander Happaerts & Hans Bruyninckx

11 In that fashion, STIs refer to those multistakeholder processes that strive to fundamentally alter the constitution of society. Those might be grassroots initiatives, such as transition towns, that are made up of concerned citizen consumers and collaborate to think of ways of anticipating on the crises that result from, for example, peak oil. On the other hand, STIs might refer to multistakeholder processes set up by governmental actors in order to stimulate macro societal change processes. Examples abound in current Flemish policy making: attempts are made to organise (government led) STIs in the energy, agriculture, mobility, health care sector etc., while two such STIs, in the waste/materials and housing sector, have been running for over 6 years. As stated however, STIs can be born from the actions of civil society actors (cf. Transitienetwerk Middenveld looking at the role such civil society organisations can play in the upcoming sustainability transition), and focus on the city or provincial level instead of the macro social scale (e.g. Leuven and Ghent s climate neutrality projects (by 2030, respectively 2050), Limburg s ambition to become climate neutral by 2020,...). 2. Evaluation and monitoring in the context of sustainability transition initiatives 5 The concept of sustainability transitions features strongly in current Flemish policy ambitions. The Flemish government has adopted the transition management model 6 as a framework to be applied to mid to long term strategies. The concept was initially introduced in the second Flemish Sustainable Development Strategy (FSDS) 7 and eventually embedded as the modus operandi to achieve the objectives stated in the socio economic development programme Flanders in Action 8 (ViA). Within those strategic documents, the ambition is unveiled to stimulate transitions by organizing various socalled transition arenas (or STIs), composed of stakeholders from a specific sector (or socio technical system), with the objective to explore (and learn about) potential transition paths and interventions to seize opportunities towards the required sustainability transitions. In our research, such government induced or supported STIs make up the object of analysis. Now, since STIs require the investment of money, personnel and time, thinking about how progress or success of such initiatives can be assessed, becomes a prominent issue. In other words, monitoring and evaluating (M&E) STIs is a central point of concern in both policy domains as well as discourse on transitions, as we shall now point out As concerns methodology, beyond an extensive literature review and the consultation of web pages, the following discussion and the remainder of the paper is based on mail correspondence and multiple interviews (10 in total) on the subject of transition governance, evaluation and monitoring in complex settings, systems thinking etc. The type of respondents interviewed ranged from academic experts (5), over process facilitators (3), to evaluators in the field of developmental support (2). Transition management (TM) is an approach to transition governance, focusing explicitly on policy and policy actors as crucial in stimulating system innovation at the societal level, while explicating tools and instruments to do so (see Loorbach, 2007). The Strategy entertains the ambition to realise transitions in 6 socio technical systems (Flemish Government, 2011a, p ). Flanders in Action aims to organise transition arenas in 13 socio technical systems of which some overlap with those present in the FSDS: Evaluating and monitoring transition initiatives. Lessons from a field scan 5

12 2.1 The necessity and utility of monitoring and evaluation Monitoring, or the extraction and documentation of information, and evaluation, or the framing and judging of that information, are tools that nurture learning and support the making of decision about the process or programme they bear on. First of all, monitoring and evaluation (M&E) is awarded a central position within the academic discourse on transitions. That is illustrated in the following picture, which displays the cycle of transition management and how each successive spiral is fed by monitoring and evaluating the steps taken (internal dynamics) and the interaction with the environment (the way the context impacts on and reacts to the STI). Figure 2 The transition management cycle & learning through monitoring and evaluation as the engine of the transition process Source: adapted from Loorbach and Rotmans (2010) and Nevens et al. (s.d.) The time based arrow (brown) illustrates the progress towards a sustainability transition (and reflects the green arrows from figure 1), contained (to some extend) and nudged into a desirable direction through consecutive cycles of TM (explained in more detail in the right hand corner), in which learning through monitoring and evaluation practices (the blue area of the rotating arrow) functions as the engine of each successive TM cycle. However, although M&E appears to be limited to the fourth and final phase of each cycle, due to the complex nature of systemic change (which we will get back to later on in the paper) each step in a cycle requires a thoughtful and constant reflection on what has happened, which actions are being promoted and how to they align with a constantly changing environment. 9 In other words, complex processes of change elicit various pertinent questions on a constant basis, which in turn affirms the potential of M&E to contribute to the process at all stages. In this paper (and by extension, the 9 i.e. how does the niche initiative relate to its surrounding, external system and how does the regime respond to the niche pressure, how can windows of opportunity be used to the niche s advantage,...? 6 Matthias Bussels, Sander Happaerts & Hans Bruyninckx

13 research performed for the policy research centre), we will limit our scope to the initial, exploratory phase of transition processes, i.e. the STIs. Our analysis will therefore pertain to how the STI characteristics affect and challenge (conventional) M&E approaches. From a policy perspective on the other hand, the utility of M&E is obvious: evaluation is concerned with determining the worth or merit of an object (Scriven, 1980) and can respond to questions of accountability, legitimacy, efficiency, efficacy,... Those questions take the shape of evaluation criteria (De Peuter et al., 2007, p. 38) and are central to modern policy making processes. The importance and reliance on M&E in the policy documents advancing transition thinking as the way forward on the mid and long term, i.e. ViA and the (second) FSDS, becomes immediately obvious after a scan of those documents. We shall discuss those practices shortly. 2.2 Challenges to M&E in STIs As stated, evaluation is concerned with determining the worth or merit of an object (Scriven, 1980), in this case an STI. Such a question would translate into an evaluation of the achievements of the STI or, in transition terms, its success in stimulating a sustainability transition at level of the socio technical system. However, such an endeavour is highly problematic, as shall be pointed out hereafter (2.2.1), which is why a focus on programme effects (or presumable results) should be substituted for a process focus (2.2.2) Result based evaluations Monitoring in ViA and the FSDS With regard to those STIs that are included in the Flemish cross sectoral policy programme ViA, monitoring activities (performed by an inter departmental working group) are focused on two levels: one the one hand, they focus on the policy programme itself, looking at which activities have been initiated/delayed/cancelled or otherwise as an aggregate indication of the success rate of the programme itself. 10 On the other hand, indicators are set at the level of the separate transition initiatives. Those involve output, outcome and impact indicators (for a discussion on output outcomeimpact indicators or product indicators, see Gysen & Bachus, 2003, p. 13). Here, apart from those processes which have been operational from before ViA has been conceived (namely Plan C & DuWoBo) and that can therefore rely on concrete realisations, indicators have often been developed even before the processes themselves have commenced. Furthermore, most of the suggested indicators in ViA are taken from the previously drafted collection of indicators on the overall progress of the strategy, as each transition theme is linked to an existing policy line within the strategy. That could lead to a lock in of options by channelling investments down paths that might reveal themselves undesirable in the face of unexpected change, while the indicators themselves might prove inadequate as the transition unravels. 10 Based on policy document VR DOC.0628 (Flemish Government, 2011b) and the following sites: & Evaluating and monitoring transition initiatives. Lessons from a field scan 7

14 Monitoring practices for the Flemish Strategy for Sustainable Development (FSDS) consist of an annual document listing various product indicators portraying the sustainability of Flanders along the three pillars of sustainable development (people, planet, profit) as viewed by the Flemish government (Flemish Government, 2011a, p. 53; Studiedienst van de Vlaamse Regering, 2012). However, as reported by Happaerts (2011, p. 130), those indicators are not meant as an evaluation instrument, but mainly to track Flanders s progress in time and in comparison to others (...) [and] support internal discussions within the working group. As such, the information deriving from those indicators do not directly provide input for evaluations. Attention is paid to integrate the monitoring of the FSDS with other result based indicator sets, such as those listed in ViA. Mention is further made to strengthen the M&E practices by performing regular internal evaluations in order to boost performance on those indicators The difficulty of evaluating the STI results Judging the merit of an STI implicitly refers to what it is expected to achieve, and how that can be assessed. In other words, it is a question of causality. Traditional positivist strands of evaluation aim at inferring causality in (semi )controlled settings, preferably by applying counterfactual and quantitative methods, and generalisation to other settings. That is achieved by breaking down the intervention into phases, assuming a linear connection between cause and effect, the progression of which can be predicted and traced rationally, and consequently analysed through the identification of criteria and indicators (cf. the ViA practice to list indicators even before STIs have been set up). The obtained information can then be transposed to other settings, allowing for rational planning of policy intervention and the prediction of outcomes (Marra, 2011, p. 329; Van Ongevalle et al., 2012, p. 3). Many, if not most evaluation approaches and/or frameworks, especially those nested in public policy, depart from such positivist assumptions (Howlett et al., 2009, p. 187). While such assumptions are well adapted to analyzing structured and simple problems, they become difficult when considering that the ultimate aim of STIs is to nurture a sustainability transition (Kurtz & Snowden, 2003; Taanman, 2012). That ambition should ideally provide for an assessment of the success of an STI, yet appears impractical due to the persistent nature of the issues it tries to address (cf. supra). Change processes occurring within socio technical systems are not linear: causes are multiple and produce effects at variable time intervals, there is no relation between the size of cause and effect (large changes in causative factors might only engender small scale changes in the affected dimensions, and vice versa), a relation that is further distorted and obfuscated through the interference of contextual factors outside of the system (Forss & Schwartz, 2011, p. 11). By way of an example, imagine an NGO aiding farmers in developing countries to increase their income and bargaining position vis à vis other actors in the chain of production and distribution of agricultural goods and services. While the objective might be clear, it remains unclear how it might be reached. The process towards the objective might envelop several years, during which much of the participating actors and relevant institutions, policies and other important societal structures (both within and outside of the country) might change. It is therefore deemed of little use to establish a predevised change strategy with fixed intermediate objectives that are monitored, while a positive end evaluation of the (supposed as it is completely grounded in the data collected through the monitoring of rigid and possibly redundant indicators) impact is expected to allow for a generalisation of the process and concomitant transposition to other roughly similar contexts. From a complexity perspective the assessment that an improvement of the position of the farmers, and an increase of income, has 8 Matthias Bussels, Sander Happaerts & Hans Bruyninckx

15 been achieved, does not allow for an interpretation of (exclusive) causality. Acknowledging the probability of multiple causes and multiple (and unintelligible) sources of distortion pertaining to measurement and individuation of such factors, caution is advised with regard to the interpretation of impact. The above illustration points to the impracticality of result based evaluations when dealing with complex change patterns: the long term character, the multiplicity of intervening factors (actors at multiple levels, unexpected events such as crises), the difficulty of establishing a threshold of sustainability... increase the overall uncertainty about the future validity of current objectives. In case of STIs however, the complexity that has to be dealt with is even more elevated: while in the previous example, the problem definition was more or less clear and agreed upon, in STIs (and by extention any attempt to engender sustainability transitions) such a focus is lacking. As sustainable development is a contestable (in the sense that no authoritative, universally valid definition can be formulated), normative (implying the inclusion of processes of value judgment beyond mere fact finding ) and revolutionary concept (Loeber et al., 2007, p. 84), different actors in the socio technical system might advance different objectives (and indicators) that they think should be focused on. Another item that warrants attention, when discussing the utilisation of output focused evaluations in STIs, relates to the question of attribution: granted that there is a causal connection between the intervention and the recorded change, to what degree is the intervention responsible for the change or is it the result of other interventions and/or external factors (Mayne, 2011, p. 53)? In terms of the MLP depicted in figure 1, there are many arrows pointed at the regime level. The question of attribution is about which of those arrows are a result of the STIs operations, a question severely predicated by the fact that STIs do not operate in a vacuum, but interact with an in itself responsive (and usually change aversive) environment. Both Plan C and DuWoBo have encountered the need to re establish their position vis à vis a changed context (see Paredis, 2011a & 2011b). Indicators such as those listed in the various transition initiatives within the ViA framework, are therefore commendable, but are not necessarily related to the initiated initiatives, while they might end up being undesirable as new developments in the socio technical system emerge. It is therefore difficult to assess the impact of STIs on a general socio technical transition. Nevertheless, attempts to exercise M&E of STIs are useful and even recommended. They are useful insofar as they can make (partial) realisations explicit, which can in turn provide for the requisite motivation to continue participation and support in the process. A suitable M&E approach is furthermore recommended (and often required) considering the decision to financially support STI usually hinges on a notion of progress and potential success. We can therefore look at the process characteristics of STIs and their implications for M&E approaches, as the second dimension of M&E within the transition governance framework (Loorbach & Rotmans, 2006) Evaluating the STI process Why STIs would benefit more from a process focused evaluation approach We argue (and shall point out hereafter) that evaluating STIs would benefit more from a processbased rather than an output based approach (Faber et al., 2004, p. 1; Loorbach, 2007, p. 80; Marra, 2011, p. 315 & 322; Meadowcroft, 2005; Voss & Kemp, 2005), due to the paramount importance of Evaluating and monitoring transition initiatives. Lessons from a field scan 9

16 learning as an engine of the STI process (Loorbach, 2007; Rotmans et al., 2005; van den Bosch & Rotmans, 2008; see figure 2). Our conceptualisation of an STI builds upon that insight, describing a multistakeholder forum in which learning is stimulated through interaction and participation, i.e. social learning referring to the social character of the process by which individuals learn (Glasser, 2007). Learning implies a shift in the framing of the (...) problem and of the perceived solution (...) [and] a shift in the principal approaches to solving the problem and in the weighing of choices between desirable yet competing objectives (Szejnwald Brown et al., 2004, p. 194), or selection and integration/reintegration of new factual, causal, contextual, methodological, theoretical, or epistemic knowledge, skills and/or attitudes to one s existing knowledge, skills, and attitude stocks (Kouévi et al., 2013, p. 17). 11 Building on the notion of sustainable development as a persistent issue, the centrality of learning becomes obvious. Loeber et al. (2007) mention three characteristics of sustainable development that establish learning as its meta narrative (Glasser, 2007), while highlighting the concept s tight connection with system failures. Sustainable development, they argue, is a contestable concept, in the sense that no authoritative, universally valid definition can be formulated. Secondly, it is a normative concept, implying the inclusion of processes of value judgment beyond mere fact finding. Thirdly, sustainable development advocates a radical system transition, as it is a revolutionary concept. Achieving a truly sustainable development means dislocating existing routines, rules, values and assumptions embedded in the institutions that have co evolved with earlier, unsustainable modes of socio technological development, which produce the risks mentioned earlier (Loeber et al., 2007, p. 84). If approached from a complex systems perspective, as does transition thinking, the paramount importance of learning for sustainable development is self evident. Analysing, and affecting, system change after all means to incorporate variability, adaptations, uncertainty and nonlinearity while heading for improved understanding of how co evolutionary processes and dynamic patterns emerge and interact across hierarchical levels and across different spatial, temporal and social scales (Rammel et al. 2007, p. 10; own emphasis). The concept s complexity fosters processes involving multiple stakeholders from various sectors that each (might) have their own preconception and definition of the issue(s) to be dealt with, and preferences regarding to the way to do so. A first and crucial step towards a sustainable path of development, then involves the creation of some sort of common ground between the actors involved, i.e. learning through interaction. Through the centrality of learning, the focus on the STI process as a fertile ground for eventual evaluations becomes self evident. Various characteristics of the STI itself strengthen the call for such a focus. - Emergence: due to many intervening, contextual factors, change is not (and therefore should not be deemed) predictable (Forss & Schwartz, 2011). That uncertainty is only compounded by the normative, revolutionary and contestable character of sustainability transitions, requiring a multiactor process in defining the problem and establishing both the (long term) objectives and possible solutions (strategies). Desirable patterns of social change will surface throughout the process, 11 Learning is here conceptualised at the level of the individual. Szejnwald Brown et al. (2004, p. 194) add a third element to learning, i.e. a shift in the relationship among the participants in the experiment, including mutual convergence of goals and problem definitions, [while] these shifts can occur among the participants of an experiment and their professional networks as well as in the broader social sphere, indicating learning at the meso level of network or organisation. 10 Matthias Bussels, Sander Happaerts & Hans Bruyninckx

17 and should be captivated as they emerge (Van Ongevalle et al., 2012, p. 4 5). Therefore, ex ante evaluation (or planning, cf. Vedung, 2010). - Open networks: as STIs are open, meaning that the constellation of participants to the process might vary through time, as certain individuals leave or other might join as roles and interests shift. 12 That makes it hard to trace changes and delineate boundaries in which to assess change (Julian et al., 1995, p ). - In other words, it is hard to indicate concrete, tangible results. On the other hand, an indication of success is crucial in order to maintain the dynamic of the process as it is often a time, energy and therefore money intensive process, while learning processes themselves benefit from re inforcement (Bandura, 1977, p ). If such re inforcement, which can be obtained through a processoriented evaluation for example, is not present, participants might choose to leave the STI Process evaluations in ViA and the FSDS The importance of a process based orientation for evaluating the STIs has not gone unnoticed by the FSDS. In its first drafting, the initial idea was to include process indicators to evaluate the Strategy, in addition to indicators on progress towards sustainable development. The former set of indicators has however not been retained (Happaerts, 2011, p. 130). Nonetheless, the new FSDS has expressed the importance of evaluating the process, inscribing it in its objectives (Flemish Government, 2011a, p. 43 & 53). The document explicitly states that the process might be more important than the result. After all it mentions the possibility that, in cross sectoral multistakeholder exploratory actions (or STIs ) suffering from a lack of clear results, the process might have yielded positive outcomes in terms of learning. Judging from an unpublished document listing the progress on the numbered actions within the FSDS, the learning mentioned concerns learning about the transition governance model as put into practice instead of actor based learning within STIs. For those reasons, alignment is sought with the learning platform established within the ViA framework. 13 It is a forum in which the transition managers, the individuals responsible for keeping track of the various ViA transitions, meet to discuss their experiences and actions. The aim is to stimulate learning about and from the transition governance approach (modelled strongly on the transition management model). In other words, the ambition of the learning platform is to refine the structure of the processes based on experiences across other initiatives, creating a network in which practical and theoretical experiences and questions can be shared, instead of strengthening the ongoing process within STIs by documenting learning therein. The ISSI project 14 to establish new methods and (evaluation) tools for Plan C and DuWoBo, falls into that category as well and is used to feed into the envisioned network. 12 As mentioned in a discussion on stakeholder mapping and involvement on the ViA learning platform: stakeholder mappings are never finished as boundaries fade, forms of cooperation change and power constellations shift. 13 See for example the annual policy statement by the Flemish prime minister Peeters (2012, p. 6) propeller.com/our clients/issi. Evaluating and monitoring transition initiatives. Lessons from a field scan 11

18 3. Examples of M&E approaches for STIs In order to illustrate the above points of discussion, and pave the way for the final section on lessons and suggestions, we shall now introduce and describe some M&E approaches that have been constructed and tested in the context of transition programmes, as well as some techniques suggested when attempting to assess the impact of programmes aimed at complex processes of social change, notably from the sector of international developmental support. 3.1 Reflexive monitoring A specific approach to monitoring is the so called reflexive monitoring approach, in which the focus is exclusively on learning processes within the setting under study. The underlying question that such processes aim to address is whether the activities of the, in our case, STI stimulate precisely those learning processes that induce a change in practices and their institutional embeddedness. Two approaches are worth mentioning in this regard: the Interactive Learning and Action evaluation approach (ILA) as developed by Regeer (2010), and a corollary monitoring approach, namely reflexive process monitoring (developed by Van Mierlo et al., 2010). Both approaches are part of the broader category of responsive and adaptive evaluations. In so far that they are proponents of the responsive evaluation philosophy, they too propose a qualitative and participatory method concerned with the facilitation of dialogue, learning and improvement in practice among stakeholders in intervention contexts characterised by ambiguity and power differences (Kouévi et al., 2013), such as STIs. The adaptive nature corresponds to a process focus, and the recognition that each process might develop along different lines, depending on factors such as power relations in the system, the constellation of participants and their (historical) interrelations, unforeseen events, the level at which STIs are organised (e.g. the agricultural sector vs. the poultry sector), etc. For that reason, no blueprint to assess the success rate of an STI can be set up (Loorbach, 2010; van Mierlo, Regeer et al., 2010, p. 15). Instead, general principles, potentially useful tools and role divisions are suggested, that are to be used according to the developments encountered and the judgment of the monitor. The ILA approach focuses on the inner workings of the project (or STI) and looks at the interplay of actor based theories of action and actions with the purpose of assisting in making timely adjustments (Regeer et al., 2009, p. 521). Such an internal focus accounts for the incompatibility of performance measurements of STI outcomes because they envision long term change process in which many actors and institutions play a role and effect changes beyond the scope of the evaluated project. That is clearly consistent with our earlier plea for a process focus instead of an orientation on output. Another important dimension of the approach is the intent, and role, of the monitor 15 to stimulate reflection of the participants upon their actions and theories. To the extent that the ILA approach is an adaptive evaluation approach, it has inspired other scholars to devise practical monitoring frameworks and tools, such as the reflexive process monitoring framework and the reflexive monitoring in action toolkit (developed by Van Mierlo et al., 2010) Our use of monitor refers to the person carrying out the monitoring activities, instead of a monitoring tool as another frequent application of the word. 16 See also 12 Matthias Bussels, Sander Happaerts & Hans Bruyninckx

19 Van Mierlo and her colleagues use theoretical insights and empirical experience to construct a monitoring framework aimed at stimulating reflexivity in such settings, using second order learning 17 emerging from such reflexive practices as an indication of the success of the STI. Acting as an information manager and process facilitator, the monitor feeds the process with information about events in the STI s environment and pushes the participants to reflect on their own position and role in the current and future system, as well as in relation to the other participants. In that sense the action component becomes obvious: instead of a separate practice, it is thoroughly entwined with the process and in constant dialogue with the internal proceedings of and external occurrences surrounding the STI. Such a take on M&E practices differs from established, conventional approaches in that it does not stipulate detailed milestones and deliverables, but evolves along the way in correspondence with developments in the field. Considering the difficulty of maintaining a high ambition and stimulating the motivation to continue participating, the reflexive monitoring approach(es) is (/are) equipped and devised to do just that: engage the participants to learn from each other, learn about the system, learn about themselves and not shy away due to the confrontational character of such learning. 3.2 The Applied Systemic Programme Evaluation framework An interesting approach to assessing the impact of a transition programme is proposed by Bressers (2011). In her applied systemic programme evaluation (ASPE) framework she focuses on programmes intent on stimulating system innovations through the development of knowledge and innovation so called Knowledge and Innovation Programmes (KAIP). As she defines system innovations as innovations for changes in the entire architecture or structure of socio technical systems, developed through multi actor processes, with substantial effects on both supply and demand actors, that are thus of great importance when dealing with persistent and complex problems (2011, p. 10 & 22 24), her analysis has considerable potential in contributing to the evaluation of sustainability transition initiatives. Similar to what we have argued earlier, the ASPE framework departs from the assumption that it is impossible to objectively assess impact when dealing with persistent and complex issues, disallowing standard impact measurements (2011, p. 13 & 21 22). However, despite the issues raised, and spurred forward by the high demand for impact evaluation (2011, p. 13), it asserts that it is possible to establish a notion of impact. Nevertheless, in order to do so, it proposes an unconventional take on the notion of impact, i.e. not as an outcome at the end of a process. Instead, it is more interactive, as is illustrated by the definition used by Bressers: impact is an evaluation criterion of the KAIPs, where effectiveness is assessed under the assumption that impact is a social process, hence non linear, and impact may therefore occur in both planned and non planned ways, at targeted and non targeted stakeholders. Impact has occurred when a degree, any degree, of change at the stakeholders is perceived to exist by the participants or members of both KAIPs themselves as well as the stakeholders, due to the actions of the KAIPs (2011, p. 33). She further argues that impact is thus unpredictable, and while intentional, goal directed action can produce some effect, that impact is probably to a certain extent due to other intervening factors as well (2011, p. 33). A large part of the 17 The notion of second order learning, referring to learning beyond instrumental learning (or learning how to achieve predefined objectives more successfully) or learning at the level of world views and paradigms, will be discussed more in depth in the following phases of this research. Evaluating and monitoring transition initiatives. Lessons from a field scan 13

20 achieved impact might well reside outside of the scope (i.e. intentions) of the programme, as a spreading oil spill. Consistent with the constructivist perspective on knowledge as a social construct (cf. the fourth order evaluation approach by Guba & Lincoln (1989)), the ASPE framework shifts the focus to learning and interaction occurring within and without the KAIP as the most adequate dimensions to focus on when trying to determine some degree of impact. The idea of learning, and the potential for the KAIP to learn from both internal and external (relative to the KAIP) evolutions, derives from the programme s and environment s reciprocal receptivity (2011, p. 31), i.e. the extent to which they are adaptive towards each other and are therefore able to adopt innovations or change derive from each camp. Analysing receptivity, as an intervening variable affecting the (largely unknowable, in the conventional sense of the word) impact of the KAIP on the socio technical system, says much about the potential impact of the KAIP (2011, p. 32). Along the same lines, Howlett et al. (2009, p. 192) emphasise the importance of the absorptive capacity and the openness to the context of the participants in the system in order to stimulate learning. Another central aspect of the ASPE framework regards the position of the evaluator. Siding up closely to research approaches such as action approach and mode 2 science, 18 the evaluator is involved actively in the object under evaluation (Bressers, 2011, p. 45). While specific attention was paid to retaining a necessary degree of independence between the evaluation practice and the research itself, the observations were carried out in a participative manner nonetheless (2011, p ). Such is recommended as a central part of the evaluation focuses on learning, involving changes in mental structures, world views, value and action frames, etc. Being able to establish change at those inner levels requires a close proximity of the evaluator to the participants of the process. One of the remarks that can be directed at the ASPE framework is its apparently predominant focus on technological innovation as a way of inciting social change. Although it might be due to a large extent from the specific KAIPs the research was based on, the framework does however appear to build largely on ideas deriving from product and technological innovation. That becomes clear in sentences such as the receptivity towards innovations and knowledge is an important determinant of innovation adoption (2011, p. 32), or the explicit focus on the application of innovations (2011, p. 41). The question might then be posed whether the same ideas underlying receptivity hold true when the scope of social innovation is extended. Various other scholars have pointed at the same attention bias transition theory appears to favour, however deeply that focus may be embedded in society s values and assumptions (Paredis, 2009; Shove, 2003). More crucially though, the ASPE framework is not equipped to deal with the main objective of STIs in another way. The main reason is that it does not incorporate any notion of sustainability into its evaluation focus. While growing receptivity might be established, such an assessment does not allow for a claim of change towards a sustainable direction. Additionally, a thorough screening of the framework uncovers the neglect of the issue of power (imbalances), and related issues such as public support. Processes in which dominant, authoritative actors shape the agenda and push growing alignment (one dimension of receptivity ) are easy to imagine. Even leaving the issue of sustainability out of this discussion, such a successful programme 18 According to Bressers own description, mode 2 science concerns the co production of knowledge, in which the research objects take part in the research to learn from their opinions (2011, p. 45). 14 Matthias Bussels, Sander Happaerts & Hans Bruyninckx