Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks

Transcription

1 Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks VT/2012/104 Request for services in the framework of the Multiple Framework Contract "Provision of evaluation and evaluation related services to DG EMPL Final Report, 3rd July 2014

2 Contents Abbreviations... i Executive summary... ii 1.0 Introduction Introduction to the report and report structure Context: Background to and operation of the ESF Learning Networks Evaluation aims and key evaluation questions Theoretical framework for the evaluation Theory of change / intervention logic Evaluation question and indicator framework Evaluation methodology Objectives and relevance Introduction The Learning Networks thematic focus and objectives Development of the Learning Networks themes and activities Relevance of Learning Networks themes and activities to ESF and national priorities Effects of the economic crisis Complementarity between Learning Networks Summary of key findings Effectiveness and efficiency: processes and inputs Introduction Effectiveness of Learning Network reach and participation Motivations for participation Effectiveness of Learning Network organisation and leadership Effectiveness and quality of Learning Network activities Efficiency of delivery and use of resources Summary of key findings Effectiveness and efficiency: dissemination and outcomes Introduction Effectiveness of dissemination... 30

3 4.3 Extent and nature of individual learning outcomes Extent and nature of organisational learning and change Extent and nature of policy transfer and learning Additional outcomes including broader co-operation between partners and Calls for Proposals Effectiveness in meeting Learning Network aims Meeting the aims of individual Learning Networks Meeting the aims of the Learning Networks as a whole Sustainability of Learning Network activities and outcomes Efficiency in generating outcomes and impacts Summary of key findings Support and monitoring Introduction The role of the Commission in supporting the Learning Networks Support offered by the Learning Network contractor and their role Summary of key findings Implications for the future of transnational co-operation through the ESF Introduction Stakeholder views on the key strengths, weaknesses and success factors relating to the Learning Networks Success factors concerning Member State engagement Key learning points from Learning Network activity Implications for future policy and activity around transnational co-operation (virtual Policy Delphi) Summary of key findings Conclusion and recommendations Concluding remarks and revisiting the theory of change Summary answers to the evaluation questions Recommendations Annex One: Additional detail on the theoretical framework... 1 Annex Two: Research tools... 6

4 Annex Three: Learning Network case studies Annex Four: Member state case studies

5 Abbreviations ACB AEIDL BFSE COP COPIE CRF DG EMPL ENYE: EMPOWERMENT EQ ESF ExOCoP GenderCoP IMPART MS NGO OECD OP PES RBM SaviAV WP Administrative Capacity-Building Network (ESF Learning Network) European Association for Information on Local Development Better Future for the Social Economy Network (ESF Learning Network) Community of Practice on Partnerships (ESF Learning Network) Community of Practice on Inclusive Entrepreneurship (ESF Learning Network) Common Reference Framework Directorate-General for Employment, Social Affairs and Inclusion European Network on Youth Employment Empowerment and Inclusion (ESF Learning Network) Evaluation Question European Social Fund Ex-offender Network (ESF Learning Network) Gender Mainstreaming Network (ESF Learning Network) Increasing the participation of migrants and ethnic minorities in employment (ESF Learning Network) Member State(s) Non-Governmental Organisation Organisation for Economic Cooperation and Development Operational Programme Public Employment Service Network on Results-based Management (ESF Learning Network) Social inclusion and vocational integration of Asylum seekers and Victims of human trafficking (ESF Learning Network) Work Package i

6 Executive summary Introduction This summary presents the findings of an evaluation aimed at Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks. Findings are based on research undertaken by Ecorys between May 2013 and April The methodology for the evaluation combined desk research, key stakeholder interviews, Learning Network case studies involving in-depth assessment of selected networks, Member State case studies investigating the Learning Networks from the Member State perspective, an online survey with Learning Network participants and other key ESF stakeholders, and a virtual Policy Delphi engaging a selection of high level stakeholders in considering the implications of the findings that emerged from the study. Reflecting the structure of the evaluation Final Report, key findings relating to a series of aspects concerning the Learning Networks operation and outcomes are presented below. The summary concludes by addressing the key evaluation questions set in the study Terms of Reference (ToR). Recommendations arising from the study are presented in the main body of the report at section 7.3. Objectives and relevance It is evident that the 13 Learning Networks assessed as part of the evaluation demonstrate significant breadth and scope in terms of their objectives. These objectives tended to focus broadly on one of three main areas (although there is some overlap): a thematic / policy focus, a concentration on ESF target groups, or operational concerns relevant to the delivery of the ESF. This breadth of focus was generally viewed as being a positive aspect of the overall Learning Network approach, though a minority view was expressed over the spread of thematic focus and objectives being too wide. The establishment of the Learning Networks thematic focus and planned activities can be said to have been broadly effective in most cases. In part this reflects the fact that a number of the networks emerged from pre-existing work and collaboration which was felt to have provided a solid basis for the networks development. Commonly the development process involved selecting a broad theme and subsequently refining this through collaborative working. This was widely seen as a positive way to engage multiple perspectives and ensure that the development of the networks thematic focus was effective through being inclusive and democratic. Another aspect commonly cited as effective involved the use of a two tier organisational model, using a steering group to develop a network s thematic focus and objectives supported by working groups charged with further developing and refining particular activities. This was cited as a significant success factor by a number of those involved in the networks when reflecting on the link between organisational structure and the overall effectiveness and outcomes of network activity. The evidence collated also suggests that the processes used to develop activities were broadly effective in ensuring that they reflected and supported the objectives of the networks. However, it was not always evident that a structured approach, ensuring that activities linked to and would be likely to support objectives, had been followed in all cases. In the main, those involved with establishing the networks also felt that the right partners had been engaged in their development. However, there was some variation in the extent to which the development process was largely restricted to ESF Managing Authorities, or involved a wider selection of stakeholders. It is clear that the objectives and themes selected by the Learning Networks have a high degree of relevance to ESF objectives, along with other high level EU policy drivers such as Europe As might be expected, a high degree of relevance was also apparent in respect of national priorities of participating ii

7 partners, particularly in the case of partners leading networks. This relevance was confirmed through consultations at the Member State level as being apparent both in respect of policy concerns and in relation to operational issues facing ESF MAs and other organisations. Evidence from the online survey undertaken similarly demonstrated that all the main network activities were considered to be relevant or very relevant on all policy levels (ESF, national and EU). While there were some clear potential linkages between the thematic focus and activities of the different Learning Networks, it is clear that the extent of collaboration between them varied, but was overall limited. In some cases this was related to the fact that co-ordination of, or involvement in, a network required significant inputs, hence reducing any time that might be available for collaboration. In others, those involved with a network felt that there would be limited value from engaging in such cross network activity. Finally, the study also gave the opportunity to explore whether and how the economic crisis had affected the thematic priorities or activities developed. In general the main impacts reported were practical and logistical, particularly in terms of reduced ability to travel on the part of some partners from Member States facing significant austerity measures. Some of the network leads and members felt that other external factors (that is, change of government or elections) had had more of an impact than the crisis itself. Effectiveness and efficiency: processes and inputs Participation in the Learning Networks and taking the role of lead partner was unevenly distributed across Member States. A group of Member States were represented in networks six times or more while several countries participated in only one or no networks. While a more even spread of participation at the Member State level can be seen as desirable, clear explanatory factors for this unevenness were evident. These included the lack of a specific requirement for Member State engagement in the networks and the differential level of resources available to ESF MAs to support their participation. Suggestions for encouraging a more even spread of participation included a greater promotional role on the part of the Commission in articulating the importance of transnational co-operation, more compulsion to co-operate for ESF MAs, and enhanced technical assistance to help address issues of limited resources or transnational expertise / experience. In terms of participation at the level of specific organisations and individuals, it is evident that several networks did take a pro-active and inclusive approach by opening up their activities to NGOs, policymakers, experts and practitioners. This spread of engagement can be identified as a key positive aspect of the networks in several cases. Equally, in general, the degree and balance of representation across the networks appeared to be appropriate and effective. However, the quality of the individual participants in the Learning Networks was more varied and it is clear that some networks experienced variations in the level of commitment by members. The evidence gathered also demonstrated that Member States and organisational / individual actors within them to engage in Learning Network activity due to a combination of individual interest, organisational interest and the desire to share learning or learn from others. The available evidence also indicated that the effectiveness of organisational processes within Learning Networks varied significantly. In addition, these processes clearly represent an important factor in determining the success or otherwise of particular networks. Generally speaking, it seems that different organisational modus operandi seemed to work for different networks. In turn this indicates that the organisational structure required to effectively implement network activities needs to reflect the size, particular composition and focus of the networks. In cases of effective practice, common factors included effective planning and leadership, clearly delineated steering and working groups with specific and clear remits, clear lines of governance and accountability and effective project management. Weaknesses in these organisational aspects appeared iii

8 to have been a key factor in particular networks not achieving their intended outcomes or doing so to a lesser extent than had been anticipated. In respect of organisational and process related challenges, work load intensity and the level of resource requirements (particularly for network leads) was clearly a key challenge. As noted by stakeholders, this indicates that an enhanced level of technical assistance would be beneficial in supporting network organisation in any future iterations of networks or similar transnational activity. Despite the variations in organisational effectiveness which were apparent, on the balance of evidence it appears that the activities taken forward by the Learning Networks were implemented in a broadly effective fashion in the majority of cases. They were also generally effective in the sense of reflecting and helping to support the achievement of the networks aims. In particular, the importance of activities which were interactive, enabling knowledge exchange and mutual learning, were emphasised in this context. In cases where activities were evidently less effective or successful, a common reason for this appeared to be that the goals outlined initially were too ambitious given the time available and resource constraints faced by those involved. Finally, judgements over the efficiency of the networks in terms of their implementation and use of available resources link closely to the above discussion over the effectiveness of organisation and leadership. In general, while the Learning Networks can be judged to have operated efficiently in terms of allocating resources to implement activities, some variations were apparent at the level of individual networks. However, it is certainly the case that many of the activities arranged by the networks were carried out on a limited budget which depended on notable in-kind resources being allocated by those involved in delivering them. This can be viewed as a key factor in supporting efficient implementation from the point of view of outcomes achieved against inputs in terms of resources used. Effectiveness and efficiency: dissemination and outcomes The dissemination approaches adopted by the Learning Networks proved to be mixed in terms of their effectiveness. In some cases dissemination products were effective in transmitting the results of network activity but this was not universally the case. Equally, while some dissemination strategies developed by offered a useful structure for sharing results, in respect of others evidence suggests that such strategies were either under-developed or inappropriately targeted. Where dissemination worked well, the use of partners as ambassadors to share learning within Member States was widely seen as a significant success factor. Similarly, building links with key players at EU and Member State level also appeared to be important to ensure effective dissemination. In terms of specific mechanisms, passive approaches such as printed materials or websites appeared less effective than more active forms of dissemination through events, meetings or on-going liaison with relevant stakeholders. The clearest and most consistently positive outcomes from the Learning Networks emerged at the level of individual learning amongst those participating. Whilst this might be expected, the evidence gathered illustrated the degree to which the vast majority of network members felt that notable positive effects had resulted from their engagement. These included new contacts, enhanced individual relationships, and a deeper level of knowledge about particular thematic areas. In the minority of cases where individuals felt that their involvement with the Learning Networks had been less positive, this generally related to the focus and activities of the network concerned in the sense of these not meeting expectations. In many cases organisations also clearly gained from the knowledge exchange involved in the networks, along with the opportunity to develop partnership and cooperation with related actors across Member States. On balance, positive examples of organisational learning and impacts outweighed cases where networks were felt to have had less of an influence at the organisational level. iv

9 Evidence of the degree of policy learning and transfer at the organisational, Member State and EU levels was, however, mixed and perhaps more limited than stakeholders involved with the Learning Networks had anticipated or hoped for. In particular, the concept of direct policy transfer stemming from network activity was questioned by a number of Learning Network and Commission stakeholders in terms of the achievability of this objective. Common reasons offered for this were the long term and complex nature of policy development, difficulties in achieving consistent engagement amongst policy makers, and the lack of direct transferability of policies stemming from the different political, social and economic contexts within Member States. Partly as a result of these factors, where policy effects could be identified these were more in the form of policy influencing as opposed to direct policy transfer. Within this, policy influence as a result of network activity was more evident in the sphere of the ESF itself, whether in terms of influence over particular OPs (particularly in the context of planning for the new programming period) or in respect of ESF guidance and regulations at the EU level. However, while concrete examples of policy transfer appear relatively limited, it is evident that more subtle effects concerning the transfer of policy ideas and options for operational adaptation occurred within the context of the networks. Moreover, such effects have the potential to influence policy and operational change over a longer timeframe. In terms of broader outcomes, in the majority of cases evidence suggests that the Learning Networks contributed to improving ESF transnational cooperation between Member States. However, there was little evidence of network activity influencing Calls for Proposals, certainly not in the sense of directly leading to such calls. As well as the broader outcomes noted, the evidence assessed indicates that the Learning Networks varied in the degree to which they met their specific stated objectives, but that in most instances the majority of or all objectives were met to a high degree. There was also a trend in the types of objectives that proved more difficult to achieve. Typically these related to the production of recommendations and influencing policy. In terms of sustainability in respect of the Learning Networks this can be considered at several levels: firstly, in relation to the sustaining of contacts or relationships; secondly, in terms of continuation of network activity or successor networks; and thirdly, in terms of the sustainability of effects and outcomes generated through Learning Network activity. While for the last of these levels it is perhaps too early to make a judgement, it seems likely that such effects will continue but might prove diffuse and difficult to identify as time passes. In respect of the other levels of sustainability noted, evidence was mixed but broadly positive in relation to the continuation of relationships and activity begun under the networks. Finally, in respect of the efficiency of the Learning Networks, it appears that the majority of Learning Networks were broadly efficient when judged from the perspective of what they achieved relative to the resources available to them. Key factors underpinning this judgement relate to the fact that the networks benefitted from considerable in-kind resources contributed by partners, had to implement their activities carefully given overall resource constraints, and in a number of cases produced outputs that would have cost significantly more had they been produced by consultants or other external contractors. Support and monitoring Perspectives on the support offered by the Commission to the Learning Networks by those leading or coordinating the networks were divided fairly evenly between positive views and a feeling that the support could have been better and / or could be improved in future. Despite this some very positive comments on the support and its contribution to network functioning and outputs were offered. In cases where views were less positive, the key issues raised concerned high levels of turnover in Commission staff and a concern that the capacity available to support the networks was insufficient. Potential improvements suggested by network leads included: greater assistance with identifying and working with external v

10 experts, facilitators and/or external evaluators; enhancing the support and guidance provided on running effective networks; providing standardised tools for running and co-ordinating a network; assistance in developing links with relevant Commission units; and reducing reporting and administrative requirements. Views on the role of AEIDL in monitoring the Learning Networks also varied. While a number of network leads and participants appreciated the supportive role that individuals from AEIDL had offered, noting that this went beyond their monitoring remit, in other cases views were less positive. The main issues raised concerned a perception that in some cases AEIDL staff did not possess the thematic knowledge that network participants felt was required for their role, that the quality of the monitoring function was variable and that some meetings and events were not organised effectively. Views on the network co-ordination meetings facilitated by AEIDL were similarly variable; while most stakeholders felt that the concept of the meetings was good, it was sometimes noted that the content and organisation of the meetings was less effective. Finally, stakeholders commonly felt that the role of AEIDL might have been clearer, particularly in terms of their remit and the balance between monitoring and support functions (whilst acknowledging that the latter element was not a formal part of their remit). Implications for the future of transnational co-operation through the ESF Stakeholders provided a range of views on the key strengths, weaknesses, and success factors emerging from the Learning Networks which can inform future Learning Network type activity and transnational cooperation more broadly. Key strengths cited tended to relate to the core purpose of the networks, in terms of transferring learning and sharing good practice, along with the quality of some of the networks outputs and perceived achievements. Weaknesses cited tended to relate to organisational and process issues in addition to a lack of impact in terms of directly influencing policy or policy transfer. Drawing on these views, a series of success factors relating to the networks can be summarised as follows: The importance of developing a clear and consistent thematic focus with clear and realistic objectives being developed to reflect these themes Producing a clear implementation plan which sets out the relative responsibilities of individual partners, steering groups and working groups Effective network composition in terms of ensuring that appropriate and committed partners (and an appropriate range of partners) are engaged in network activity The development of clear governance processes and lines of accountability Effective organisation and leadership on the part of those co-ordinating network activity A well developed dissemination strategy, supported by high quality tools and outputs which are aimed at specific audiences. Examination of the effects of the Learning Networks at the Member State level also serves to highlight some key implications for how Member States might best engage in transnational co-operation. These include the need to ensure that Member States develop a clear strategy to guide their participation in such activity, the development of a supporting infrastructure able to transfer learning to relevant stakeholders in the domestic context as well as feeding in learning effectively to the European level, and ensuring consistency of participation as far as possible (both in respect of operational and policy actors). Running a virtual Policy Delphi to explicitly consider the future of transnational co-operation served to highlight several key considerations in this area. These include the importance of effective technical assistance to support the engagement of Member States and actors within them in transnational initiatives, the significant role the Commission can play in promoting transnational co-operation and its value whilst actively guiding such co-operation in the context of the ESF, greater engagement of Commission policy units, greater realism over the both the potential and limitations of transnational co- vi

11 operation on the part of both the Commission and those engaging in it, and the importance of Member States / ESF MAs acting as an effective conduit for transferring learning to domestic actors and back to the European level. Summary answers to the evaluation questions Responses to the evaluation questions set out for the evaluation in the study ToR, based on the analysis presented in preceding chapters, are summarised below. Where applicable, the reader is signposted to the specific sections of the main report from which the evidence to inform these responses can be found. EQ1 - To which extent have Learning Networks activities corresponded to the overall aims and policy objectives of the ESF? There is a clear relationship between the activities developed by the Learning Networks and the overall policy aims and objectives of the ESF. A high degree of relevance between network activities and these policy objectives and overall aims was apparent in the majority of cases (section 2.4). EQ2 - To which extent, under what circumstances, from whom and by whom learning has taken place during the duration of the Learning Networks? Learning deriving from the activities of the Learning Networks has taken place at a range of levels and amongst numerous actors during their duration. Whilst such learning is clearest and most apparent in respect of individual network participants (section 4.3), it is also widely apparent in respect of participating organisations, whether ESF MAs or other partner bodies (section 4.4). The ways in which learning took place varied between the networks, reflecting their different focus and the range of activities they undertook. Typically, however, peer reviews, study visits, events and conferences were cited as effective mediums through which to share practice and generate learning (section 3.4). There is also evidence of influence on policy learning, although evidence of direct policy transfer is limited (section 4.5). EQ3 - To which extent the activities of the Learning Networks have contributed to the achievement of the Learning Networks objectives? As reflected in the response to EQ.2, there was some variation in the specific activities which led to particular objectives being met or otherwise across the Learning Networks. In turn, this reflects the variation in the scope and focus of the networks themselves. Overall, however, a clear relationship between the activities networks engaged in, and the high level of achievement of the objectives they set themselves, was apparent in most cases (sections 3.4, 4.7). The key exceptions to this involved some networks failing to produce particular planned outputs and less evident achievement of objectives around influencing policy development (sections 4.5, 4.7). EQ4 - What are the achievements of the calls for proposals organised in the context of the Learning Networks (where applicable)? The assessment revealed that Calls for Proposals did not generally emerge from Learning Network activity. While some examples of activities influencing the design or development of Calls for Proposals in the context of Member State and regional ESF OPs were offered, the linkages between these and network activity were unclear. Overall, therefore, there was little evidence of network activity influencing Calls for Proposals, certainly not in the sense of directly leading to such calls (section 4.6). EQ5 - To which extent have the dissemination activities reached relevant stakeholders? The dissemination approaches adopted by the Learning Networks proved to be mixed in terms of their effectiveness. In general, ESF MAs were reached to a greater extent than other actors, as were ministries of labour though to a lesser degree. NGOs, representatives from other ministries, and other actors such vii

12 as employers and trade unions appeared to have been less well reached by dissemination activity (section 4.2). viii

13 1.0 Introduction 1.1 Introduction to the report and report structure This final report presents the findings of an evaluation aimed at Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks. The report builds on two previous reports submitted as part of the evaluation, an Inception Report submitted in July 2013 and an Interim Report submitted in December The final report is based on research undertaken by Ecorys between May 2013 and April The remainder of the final report, including annexes, is structured as follows: Chapter one details the context for the evaluation, along with the evaluation aims and methodology; Chapter two addresses questions relating to the objectives and relevance of the Learning Networks, in particular the relevance of the networks themes and activities to ESF and national priorities; Chapter three assesses the effectiveness and efficiency of the Learning Networks in relation to key processes around their operation, including network participation, organisation and activities; Chapter four assesses effectiveness and efficiency in terms of dissemination and outcomes from the Learning Networks, including organisational learning, and policy transfer; Chapter five examines the effectiveness of support and monitoring processes linked to the operation of the Learning Networks; Chapter six considers implications for the future of transnational co-operation through the ESF arising from the experience of the Learning Networks and the findings of the evaluation; Chapter seven concludes the report by addressing the key evaluation questions established for the study and providing recommendations on the basis of the evaluation findings. Annex one details the theoretical framework guiding the assessment through mapping evaluation questions to indicators and methodological tasks; Annex two presents the research tools used in the study; Annex three presents the Learning Network case studies which formed one element of the methodology; Annex four presents the Member State case studies that formed another element of the approach. 1.2 Context: Background to and operation of the ESF Learning Networks To promote access to new ideas, innovative approaches and new skills the European Commission supports transnational co-operation between Member States of the European Union. One route to supporting such transnational co-operation involves channelling ESF funding to Member States and their regions to support transnational priorities in national and regional ESF Operational Programmes (OPs), the mechanism through which ESF objectives and activities are agreed with the Commission. In order to provide additional support and incentives to work transnationally, the Commission also undertook a number of actions in the ESF programming period including the funding of transnational 'Learning Networks'. The Learning Networks are intended to facilitate and strengthen transnational exchange and cooperation between ESF managing and implementing bodies and strategic stakeholders. The Learning Network approach draws on the experiences of the nine Thematic Focus Groups (TFGs) implemented within the context of the ESF Community Initiatives ADAPT and EMPLOYMENT during the 1

14 programming period, and the two rounds of European Thematic Groups (ETGs) implemented within the five thematic pillars of EQUAL Community Initiative during the programming period. As part of the process of establishing the networks, ESF Managing Authorities (MAs) were invited to respond to restricted calls for proposals in order to obtain funding for the management of Learning Networks, including the provision of facilitators and any other professional expertise required for their successful management. Calls were launched in 2008 (VP/2008/018) and 2009 (VP/2009/012), and more recently in 2012 (VP/2012/005). As a result of the 2008 and 2009 calls for proposals, thirteen Learning Networks were established which are the subject of this evaluation. Each Learning Network covered a specific thematic or governance related area as follows: COPIE: Community of Practice on Inclusive Entrepreneurship BFSE: A Better Future for the Social Economy EMPOWERMENT: Empowerment and Inclusion ENYE: European Network on Youth Employment IMPART: Increasing the Participation of Migrants and Ethnic Minorities in the labour market ExOCoP: Ex-offender Community of Practice Partnership: Community of Practice on Partnerships in the ESF TNC: Transnational Cooperation in the ESF CoP RBM: Results Based Management in the ESF Gender CoP: Community of Practice on Gender Mainstreaming ESF AGE: ESF AGE SaviAV: Social inclusion and vocational integration of Asylum seekers and Victims of human trafficking ACB: Administrative Capacity Building - Facing the Challenge How to effectively support public administration reforms by ESF funds. The Learning Networks were each led by representatives of MAs from within Member States, although each adopted different management, implementation and operational approaches and undertook different activities related to their thematic focus and objectives. Common elements included the engagement of a set of Learning Network members or participants drawn from different participating Member States, the defining of a set of objectives for the network to work towards, and the identification and selection of a set of activities through which to meet these objectives. Further detail on the rationale, objectives, activities and intended results and impacts of the Learning Networks is provided in section 1.4 below as part of setting out a theory of change relating to Learning Network activity. In terms of monitoring and support for the networks, as part of its approach to transnational co-operation in 2009 the European Commission launched a tender for a contractor to provide monitoring services for transnational and innovative projects, and to synthesise and disseminate relevant results (VT/2008/090). This tender was awarded to the European Association for Information on Local Development (AEIDL). The contract included the provision of a monitoring and support role in respect of the ESF Learning Networks. As part of their role, AEIDL participated as an observer in all EU Learning Networks and provided regular reports to the Commission on their performance. In order to support the work of the ESF Learning Networks, the European Commission sought to provide support throughout their implementation. In particular, Policy Officers from DG EMPL were assigned to specific Learning Networks in order to support their work in a variety of ways: attending certain meetings or events, seeking to link activities or outputs with other relevant policies/stakeholders, and providing inputs as appropriate. The Commission also set up "support groups" composed of policy officers from 2

15 relevant DGs. Finally, the Commission asked the monitoring contractor, AEIDL, to organise Coordination Meetings of the Learning Networks in order to share experiences on a regular basis. 1.3 Evaluation aims and key evaluation questions The assessment of the Learning Networks is significant in determining not only their success to date in the current ESF programming period, but also in identifying recommendations and areas of improvement in the field of transnational co-operation for the programming period. This balance of backward-looking evaluation and forward-looking assessment is reflected in the Terms of Reference (ToR) for the study. The ToR outlined three key aims: 1. To assess the effectiveness, efficiency and relevance of the Learning Networks. 2. To assess their [the Learning Networks] contribution to enhancing capacity building and mutual learning between ESF managing and implementing bodies and with strategic stakeholders. 3. To formulate recommendations on the basis of the outcomes of this evaluation, which support the discussion on the role of the Learning Networks in the next ESF programming period , especially in the area of promoting transnational co-operation. The ToR also set out six key evaluation questions (EQs) as follows: EQ1 - To which extent have Learning Networks activities corresponded to the overall aims and policy objectives of the ESF? EQ2 - To which extent, under what circumstances, from whom and by whom learning has taken place during the duration of the Learning Networks? EQ3 - To which extent the activities of the Learning Networks have contributed to the achievement of the Learning Networks objectives? EQ4 - What are the achievements of the calls for proposals organised in the context of the Learning Networks (where applicable)? EQ5 - To which extent have the dissemination activities reached relevant stakeholders? EQ6 - What are the successful examples of mutual learning coming out of the Learning Networks activities and successful projects funded under the Call for Proposals? 1.4 Theoretical framework for the evaluation At the outset of the work, a theoretical framework was developed to guide the study and ensure that the above evaluation aims and key evaluation questions were addressed. The core components of this framework were: The development of a theory of change approach in respect of the Learning Networks through which to provide a structured means of assessing their operation and outcomes. Mapping of the key evaluation questions to sub-questions, indicators and methodological tasks to ensure the evaluation aims were met and to guide the assessment process. 3

16 The overall approach to the assessment was informed by the recognition that multiple factors are likely to have influenced the relevance, effectiveness and efficiency of the ESF Learning Networks. In order to determine the contribution of the networks the assessment process therefore focused on: 1. Outlining the intervention logic / theory of change for the ESF Learning Networks to include learning processes and other outcomes. 2. Through this intervention logic, identifying effects we would expect to observe at each stage and determining the key success factors that explain how the desired effects have been brought about; if necessary, considering alternative explanations for any observed effects. 3. Undertaking case studies and an online survey to explore how the effects have been generated and that offer good practice examples for future expansion or replication. 4. Developing recommendations that draw on confirmed explanatory factors and key research findings Theory of change / intervention logic In order to undertake a structured assessment of the Learning Networks it was first necessary to define the intended rationale for the networks operation, along with their aims and objectives, inputs, activities, intended results and intended impacts. This theory of change or intervention logic is outlined below and presented in diagrammatic form in figure 1.1 overleaf. 4

17 Figure 1.1 Intervention logic / theory of change for the ESF Learning Networks Rationale Aims and objectives Inputs Activities Results Impacts Collaborating and sharing experiences across borders, whether national, regional or organizational, is an effective way to access new ideas, innovative approaches and new skills Use of Learning Networks as one mechanism by which transnational cooperation and exchanges of good practice and knowledge can be facilitated, hence supporting the implementation, delivery and outcomes of ESF Support the objective of: - improving the quality and efficiency of Structural Funds programmes and their impact on employment, social inclusion and training across the whole Union -contributing to policy development at EU level through: - exploiting opportunities for programme managers, strategic stakeholders, governmental departments in charge of policy supported under the ESF and practitioners for learning from one another, and with each other - capitalising good practice under the ESF Operational Programmes (in particular from innovative action and transnational cooperation) Financial support through the ESF Technical Assistance operational budget (maximum of 4 million under 2008 call for proposals and 3.6 million under 2009 call) Additional contributions through the Commission s role in facilitating good practice and acting as a catalyst for learning and change Additional (in-kind) Member State contributions (financial and human resources) Thirteen Learning Networks undertaking a range of activities including: - exchange events - peer reviews - learning seminars - policy fora - the development of toolkits, communication platforms and technologies - the transfer of competencies and experience through mechanisms such as training and coaching Programme level: - Fine-tuning of arrangements promoting transnational exchange and co-operation - Stakeholder and practitioner networks at OP level - Launching of action plans to base ESF support on agreed good practice - Introduction of common (management) tools - Common approaches to monitoring, evaluation or reporting Individual members: - helping each other in solving problems at operational and strategic levels - providing stimuli to reflect on practice and improve it - applying common, tested, working methods - providing opportunities for staff exchange - gaining professional skills and recognition Participating organisations: - keeping up with developments across Europe - getting easy access to a pool of competencies to respond faster to emerging policy needs; - using common successfully tested tools - developing and assuring professional competence - speeding up the use and integration of good practice gained elsewhere - developing a common voice on the issues at stake Learning Networks making a contribution to: - strengthening the capacity to innovate - modernising and adapting institutions to new social and economic challenges - identifying and assessing issues and solutions for reforms in policy and delivery - improving the quality of governance of public policies, programmes and actions. 5

18 Rationale The presumption behind the Learning Networks was that collaborating and sharing experiences across borders, whether national, regional or organisational, is an effective way to access new ideas, innovative approaches and new skills. In line with this the ESF Regulation No 1081/2006 (Art. 3) invited Member States and regions to support transnational cooperation in all policy areas identified for ESF interventions. The development of ESF Learning Networks was intended to form one mechanism by which transnational cooperation and exchanges of good practice and knowledge can be facilitated. Aims and objectives The overall aims and objectives of the Learning Networks involved supporting the wider purpose of transnational cooperation under the ESF; that is, contributing to employment policy and delivering reforms through learning from other countries and regions by bringing together people, knowledge and practice, activities, networks and fora in the field of employment, social inclusion and training. 1 As outlined in the 2006 EC Action Plan to support Transnational Cooperation at EU level and the 2008 Call for Proposals (VP/2008/018), the Learning Networks were intended to yield important multiplier effects to: Strengthen the capacity to innovate Modernise and adapt institutions to new social and economic challenges Identify and assess issues and solutions for reforms in policy and delivery Improve the quality of governance of public policies, programmes and actions. 2 The 2009 Call for Proposals (VP/2009/012) 3 supporting the objective of: defined the purpose of the ESF Learning Networks as Improving the quality and efficiency of Structural Funds programmes and their impact on employment, social inclusion and training across the whole Union Contributing to policy development at EU level. These objectives were intended to be achieved through: Exploiting opportunities for programme managers, strategic stakeholders, governmental departments in charge of policy supported under the ESF and practitioners for learning from one another, and with each other; and Capitalising good practice under the ESF Operational Programmes (in particular from innovative action and transnational cooperation). Inputs Financial support for the implementation of Learning Networks was through the ESF Technical Assistance operational budget, budget line It is not easy to fully quantify the totality of the financial inputs to the Learning Networks overall. However, as detailed in the Call for Proposals (VP/2008/018), a maximum of 4 million was allocated for commitments in The equivalent indicative amount for the 2009 call was 3.6 million. 1 European Commission, Learning for Change: Setting up learning networks under the ESF , Call for Proposals VP/2008/018 2 Ibid. 3 European Commission, Learning for Change: Setting up learning networks under the ESF , Call for Proposals VP/2009/012 6

19 Activities The inputs outlined above supported the range of activities implemented through the Learning Networks. At the overall programme level thirteen Learning Networks were established in response to the calls for proposals (VP/2008/018 and VP/2009/012) in 2008 and Within this a range of specific activities were taken forward by individual Learning Networks. While these varied, there were also a number of common elements which characterised the activity of the networks. These included exchange events; peer reviews; learning seminars; policy fora; the development of toolkits, communication platforms and technologies; and the transfer of competencies and experience through mechanisms such as training and coaching. Results Each of these activities was envisaged to generate added value for the ESF programmes and the policies which the ESF supports, and for the people and organisations participating in its activities. 4 These include, at the individual level in terms of professional development: Helping each other solve problems at operational and strategic levels Providing stimuli to reflect on practice and to improve it Applying common working methods tested in other countries or regions Providing opportunities for staff exchange Gaining professional skills and recognition. 5 Added value through the results of Learning Networks is expected at the level of participating institutions and organisations in terms of capacity building for managing ESF programmes effectively. This is anticipated to be evident through: Keeping up with developments across Europe in the policy or governance field in question Gaining a pool of competencies and experience which allows to respond faster to emerging policy needs Using common tools successfully tested across Europe Developing and assuring professional competence Speeding up the use and integration of good practice gained elsewhere in Europe Developing a common voice on the issues at stake. 6 Finally at the programme level: Mutual fine-tuning and orchestration of arrangements to promote transnational exchange and cooperation under ESF programmes, in order to develop synergies and complementarities Setting up networks of stakeholders and practitioners at OP level contributing to and taking up results of the network Launching action plans to base ESF support on commonly agreed good practice or approaches Introducing common (management) tools Common approaches to monitoring, evaluation or reporting on ESF activities in the fields of cooperation. 7 4 Ibid. Also detailed in the 2008 Call for Proposals. 5 Ibid. 6 Ibid. 7 Ibid. 7

20 Impacts Beyond the more immediate and short term results from Learning Network activity, the intervention is intended to contribute to several impacts which reflect the aims and objectives of the Learning Networks and the Commission s support for them. The impacts involved are intended to flow from the more immediate results of the programme and may only be evident in the longer term. These impacts are summarised in terms of the intention for the networks to contribute to: Strengthening the capacity to innovate Modernising and adapting institutions to new social and economic challenges Identifying and assessing issues and solutions for reforms in policy and delivery Improving the quality of governance of public policies, programmes and actions Evaluation question and indicator framework The second core component of the theoretical approach to the assessment involved a process of developing sub-questions relating to the main evaluation questions detailed in section 1.3, then mapping these evaluation questions and sub-questions to a series of indicators that could be used to underpin the assessment. The full framework, which also maps the evaluation questions and indicators to the methodological tasks involved in the study, is included at Annex 1. This approach ensured that the evaluation remained focused on the key questions under consideration, whilst also informing the methodological approach outlined below and the design of the research tools included in Annex Evaluation methodology Informed by the theoretical framework outlined above, the methodology for the evaluation involved a series of interlinked work packages (WP) as follows: WP1: Kick-off meeting A kick-off meeting for the evaluation took place at DG EMPL on 23 May The meeting focused on agreeing the study approach in terms of the theoretical framework, methodology, research tools, data sources, work organisation, day-to-day project management and client liaison. WP2: Desk research A programme of desk research was undertaken to examine available information about the Learning Networks. The analysis laid the foundation for the remaining work packages in terms of developing a deeper understanding of the networks on the part of the evaluation team, in addition to providing evidence through which help address the Evaluation Questions specified for the assessment. Specifically, the desk research involved a targeted and structured review of the following (where appropriate): Interim and final reports of the Learning Networks submitted to the Commission Monitoring reports prepared by AEIDL, including quarterly reports for each Learning Network produced during the duration of the grant agreements between the Commission and the networks Reports, evaluations, tools and other outputs produced by the Learning Networks Reports of the major events organised by the Learning Networks 8 Ibid. 8

21 Policy literature regarding the ESF, Learning Networks, Europe 2020 and EU policy more generally Additional relevant literature, including that form academic sources, identified over the course of the assessment. WP3: Evaluation scoping An evaluation scoping phase was undertaken to develop research tools for the assessment of the Learning Networks. The scoping stage also involved the further design and development of the case studies which formed a core element of the methodological approach (see WP6 and WP8 below). The research tools included semi-structured topic guides for use in the key stakeholder interviews (WP5) and case studies, along with templates for analysing and reporting the results of the case studies. In terms of informing the design of the case studies, the scoping phase involved the development of criteria with which to select a sample of Learning Networks for more in-depth assessment in WP6. The criteria used were as follows: Type: a balanced sample of Learning Networks focused on governance or thematic issues Theme: at least one network selected from each thematic cluster defined Geographical coverage: as far as possible, a selection of networks representing the inclusion of as many MS as possible, as well as diversity in ESF Managing Authority leads Longevity of Networks: as far as possible, a mix of Networks which started in the current ESF programming period and some that are continuations of earlier work (for example, EQUAL) Preliminary indications of degree of impact: examples of successful, but also less successful, practice identified through the desk research in WP2. Application of the criteria resulted in agreement with DG EMPL that the following networks would be selected as case studies for WP6: BFSE: A Better Future for the Social Economy EMPOWERMENT: Empowerment and Inclusion ExOCoP: Ex-offender Community of Practice Gender CoP: Community of Practice on Gender Mainstreaming CoP RBM: Results Based Management in the ESF TNC: Transnational Cooperation in the ESF ENYE: European Network on Youth Employment. The scoping work package also involved a selection process for the Member State case studies (WP8). This process was based on three main criteria, as well as on insights gained through the stakeholder interviews (WP5) and desk research (WP2). The criteria used were as follows: Geographical/welfare regime balance: aimed at ensuring coverage, as far as possible, of Nordic, Anglo-Saxon, Continental, Southern European, Central and Eastern European Countries Degree of involvement in transnational cooperation: involving the selection of Member States based on the levels of engagement (using two sub-criteria: participation in Learning Networks and the extent of promotion of transnational projects under Member States own funding) Degree of centralisation of ESF management: aimed at ensuring a balance of countries where ESF is managed at national/state/regional level. Application of the criteria resulted in the following Member States / OPs being selected in agreement with DG EMPL: Spain, Czech Republic, Belgium (Flanders), and the United Kingdom (England and Gibraltar 9

22 OP). 9 It was also agreed to conduct interviews with representatives of the ESF MAs in Portugal and Greece, not as full Member State case studies but specifically to investigate reduced participation in the Learning Networks that were the subject of the evaluation. WP4: Inception report A draft Inception Report was submitted on 2nd July 2013 providing an updated study methodology based on the evaluation scoping phase, a progress update, evaluation work plan and initial findings from the desk research. An inception meeting to discuss the report took place at DG EMPL on 7 July After minor revisions, a revised report was submitted on 16 July 2013 and subsequently approved in July WP5: Key stakeholder interviews Between August and November 2013, in-depth interviews were conducted with nineteen key stakeholders with a strategic role in the design and operation of the ESF Learning Networks. The interviews were used to gather an initial strand of evidence as part of addressing the specified Evaluation Questions, to identify additional contacts and data sources for the ESF Learning Network case studies, and to contribute to selecting Member States for the Member State case studies. The interviews included consultations with lead representatives of the 13 Learning Networks under consideration, along with a selection of DG EMPL representatives with roles relating to the networks. WP6: Learning Network case studies On the basis of agreed criteria, seven Learning Networks were selected to be the subject of case studies (see WP3 above for those selected and the criteria used). The purpose of the case studies was to: Examine Learning Network members opinions regarding the quality and relevance of activities Assess the extent and impact of dissemination activities Determine the extent, nature and depth of learning benefits Identify the extent and nature of increase in personal networking and contacts Identify instances of organisational learning and change; for example, increased capacity, new/improved relationships, networking or co-operation with other relevant stakeholders Identify instances of policy influence and change to which the networks selected contributed. Each case study involved several interlinked research tasks as follows: In-depth review of relevant documentation relating to the Learning Networks, including outputs produced by the networks, AEIDL monitoring reports, and other policy and contextual documentation relating to the thematic focus of the network in question Development of intervention logics for each network, drawing on the above review of documentation, through which to structure and frame the assessment Between six and ten in-depth interviews with stakeholders including, as appropriate, network leads and co-ordinators, AEIDL monitoring expert, European Commission support person, Learning Network members and experts used by the networks Analysis of the evidence emerging from the documents and interviews allowing a comparison of findings in light of the intervention logic 9 Given the way in which the ESF is managed and implemented in Belgium and the UK a decision was made, for reasons of consistency and being able to better focus the case studies, to focus on Flanders in the Belgian context and the England and Gibraltar OP in the United Kingdom context. 10

23 Case study reporting, involving written summaries of around pages each (the individual reports can be found at Annex three. WP7: Interim report An Interim Report, drawing on the above work packages, was produced and submitted to DG EMPL on 6 December The report included a progress update, revised work plan, and findings based on the research undertaken to that point. An interim meeting was held at DG EMPL on 12 December 2013 and a revised version of the report with minor amendments was submitted on 17 December WP8: Member State case studies On the basis of agreed criteria, four Member States were selected as case studies (see WP3 above for those selected and the criteria used). The Member State case studies were used to add to our understanding of the effectiveness of the Learning Networks by looking at how Member States engaged across a range of Networks and the impact of such engagement. The case studies were also used to draw conclusions relating to the process aspects of the Learning Networks in terms of: The most effective ways for Member States to engage with the networks The ways in which learning, policy transfer/adaptation, and the improvement of the quality of ESF management and capacity-building takes place. The Member State case studies were undertaken through: Interviews with key individuals at the applicable Member State / regional / OP level, including those involved in the networks, those with a co-ordinating role with respect to the networks within the Member State, and those from organisations such as Government Ministries that had potentially benefited from their activities Reviewing relevant policy and operational documents (including those suggested by interviewees) relevant to any learning or other impacts in the Member State context; Analysis of evidence and subsequent reporting through written summaries of around 5-7 pages each (included at Annex four). WP9: Online survey of Learning Network stakeholders To add to the key stakeholder interviews and case studies, and further develop the evidence base for the assessment of the Learning Networks, an online survey was conducted with stakeholders connected to the Learning Networks using the web-based CheckMarket tool. The survey was carried out via an online questionnaire sent out to ESF Learning Network participants and other key stakeholders including Commission and ESF MA representatives and a small number of wider stakeholders with a link to the work of the networks such as social partners. Multiple channels were used to obtain the contact details of the respondents: For the ESF Learning Network members and related stakeholders, the participant lists in the AEIDL Learning Network monitoring and final reports were used. Contact details were updated through internet searches and, where necessary, by consulting the lead partners of the Networks (yielding 195 potential respondents in total). The European Commission sent out an invitation to the Technical Working Group of the ESF Committee (yielding 62 potential respondents in total). 11

24 To reduce the time needed to respond and hence encourage participation, the survey involved a maximum of 36 mainly closed questions, with 13 questions for those respondents who were not Learning Network members. Given that many respondents participated in multiple Learning Networks, the survey was designed to prioritise information gathering on the networks with which the individual was most involved. The survey was administered between 30 January and 28 February Of the 195 Learning Network members invited to participate, a total of 93 (48%) opened the invitation to participate in the survey, 68 (35%) started the survey and 61 (31%) completed the survey. Of the 62 ESF Committee Technical Working Group members (and alternates) invited to participate, seven started the survey and six completed it. In total, 70 respondents either completed the survey or answered a significant number of questions, forming a sufficient basis for analysis. This translates to an overall response rate of 32%. Responses to the survey were analysed using Statistical Package for the Social Sciences (SPSS), a software package for analysing quantitative data. For many of the survey questions, respondents were asked to fill in answers for a minimum of one and a maximum of three Learning Networks (if applicable). This method provided a total of 98 responses derived from Learning Network participants for most questions. For questions where there was only one answer (that is, responses for more than one Learning Network were not provided), a lower number of responses was available and this number is stated where these results are presented. Survey responses were integrated with the findings emerging from the key stakeholder interviews, Learning Network case studies and Member State case studies to inform the chapters that follow. Along with the desk research (WP2), these elements collectively formed the evidence base for the assessment of the Learning Networks. WP10: Virtual Policy Delphi Drawing on analysis of the evidence gathered through the previous work packages, the final element of the evaluation involved a virtual Policy Delphi. This mechanism was used to promote discussion of a set of key research findings amongst a group of stakeholders with a role and interest in transnational cooperation through the ESF, and to encourage participants in the Delphi process to consider the key implications or considerations arising from these research findings. To address the evaluation objective around examining the role of the Learning Networks and transnational co-operation in the next ESF programming period ( ), the Policy Delphi was designed with a forward-looking focus to gather views on future approaches in respect of transnational co-operation. The process for running the virtual Policy Delphi involved identifying, in collaboration with DG EMPL, a sample of representatives from the Commission, ESF MAs, selected Member States and AEIDL to form the Policy Delphi panel. An initial discussion paper, including 10 key research findings and matching issues for consideration, was then produced for circulation to the panel (the paper is included at Annex two). The panel were invited to provide written responses to the issues for consideration, with these responses then being analysed and a second summary paper produced. This summary paper was then circulated to the panel to invite any further comments or views. The results of this process were used to feed into the discussion in chapter six of this report around implications for the future of transnational cooperation arising from the experience of the Learning Networks and the findings of the evaluation. WP11: Final report Through a process of analytical synthesis, bringing together the evidence gathered from the preceding work packages, the final stage of the evaluation involved the production of this Final Report. A draft Final Report was submitted to DG EMPL on 9 April Following a meeting with DG EMPL and requested amendments the final version of the report was submitted. 12

25 2.0 Objectives and relevance 2.1 Introduction This chapter examines issues relating to the objectives of the Learning Networks along with the relevance of their focus and activities to ESF and Member State priorities. It first outlines the networks thematic focus and objectives, including considering stakeholder views related to this, by way of contextualising the analysis that follows. The chapter then assesses the development process for establishing the networks thematic focus and objectives. The relevance of the Learning Networks themes to ESF and national priorities is then examined, prior to considering the relevance of network activities to ESF and national priorities. The chapter then briefly examines the degree of complementarity between the networks in terms of their operation, prior to concluding with a summary of the key findings from the preceding analysis. 2.2 The Learning Networks thematic focus and objectives Taken as a whole, it is evident that the 13 Learning Networks under consideration demonstrate significant breadth and scope in terms of their objectives. Table 2.1 overleaf summarises the strategic objectives and thematic focus of each network. As the table demonstrates, the objectives of the individual networks tended to focus broadly on one of three main areas (although there is some overlap in the case of certain networks and specific objectives within them). These areas can be defined as: Promoting knowledge-sharing and policy / organisational development around particular policy themes or areas such as gender equality, the social economy or the empowerment of disadvantaged groups Developing approaches to supporting particular ESF target groups such as ex-offenders or economic migrants Developing more horizontal, structural, aspects relevant to the delivery and governance of the ESF such as administrative reform, results based management (RBM), partnership working and transnational co-operation. The scope of the networks in terms of adopting a thematic / policy focus, concentrating on ESF target groups, and examining operational concerns relevant to the delivery of the ESF was referenced by several stakeholders as representing a positive aspect of the overall Learning Network approach. However, a minority view was also expressed concerning the possibility that the spread of thematic focus and objectives was too wide. From this perspective, there would have been benefit in having a smaller number of networks, less divergence in terms of thematic focus and a more limited and targeted number of objectives per network. This theme is reflected further in some of the analysis that follows in subsequent chapters, and in particular was a source of debate when considering the future of transnational co-operation (see chapter six). 13

26 Table 2.1 The objectives and thematic focus of the Learning Networks Learning network Objectives Themes ACB: Administrative Highlight how ESF funds can effectively be used to modernise public Strategic planning Capacity Building - Facing administration and to improve the quality of activities undertaken in the area E-government the Challenge How to of good governance Business-friendly administration effectively support public Partnership principle for better regulation administration reforms by Local government ESF funds. ESF AGE: Age Make more effective use of ESF opportunities for Age Management Sustainable employability and work ability management network programmes and projects through networking and mutual learning Transitions from (un)employment to work Link the world of ESF funding in Member States and regions with the state of the art, but scattered, knowledge on effective age management strategies BFSE: A Better Future for the Social Economy Partnership: Community of Practice on Partnerships in the ESF COPIE: Community of Practice on Inclusive Entrepreneurship EMPOWERMENT: Empowerment and Inclusion Exchange knowledge and information on the social economy Community law and social services of general interest Increase professional skills Measuring social added value and quality standards Develop new measurement instruments Socially responsible public procurement and public social Influence authorities partnership Social franchising Financial instruments and mechanisms of the ESF allocation to the social economy Stimulate more creative partnership thinking and implementation in the Governance Structural Funds, especially in the ESF Sustainability Transnationality Innovation Work together to put policy changes into practice in partners country or Quality management of enterprise advisers and programmes region that will widen participation in enterprise Enterprise education Integrated start up support Access to finance Empower beneficiaries, assist intermediary agencies to create an Empowerment measurement empowering environment and help employers to establish an inclusive workplace ExOCoP: Ex-offender Carry forward and expand the debate on the relevant policy and support Education 14

27 Learning network Objectives Themes Community of Practice strategies necessary for the reintegration of offenders Training Employment Resettlement Gender CoP: Community Facilitate better use of the gender mainstreaming strategy in improving the Knowledge and expertise of Practice on Gender implementation of the ESF priority axes Perspective change Mainstreaming New procedures IMPART: Increasing the Find out what works, what doesn t work and why in increasing the Validating migrants skills Participation of Migrants participation of migrants and ethnic minorities in employment Fostering anti-discrimination skills as a professional approach and Ethnic Minorities in Integrated territorial approaches the labour market CoP RBM: Results Based Promote Results Based Management (RBM) defined as the way an Strategic orientation and coherence Management in the ESF organization applies processes and resources to achieve targeted results." Stakeholders engagement Delivery planning Monitoring and evaluation SaviAV: Social inclusion and vocational integration of Asylum seekers and Victims of human trafficking TNC: Transnational Cooperation in the ESF ENYE: European Network on Youth Employment Enforce the human rights of trafficked people and contribute to a dignified Advice, education and vocational training standard of living for asylum seekers by identifying emerging issues and Forced labour and human trafficking policy gaps and influencing policy take-up and structures in this field. Prevention and integration support Identification and dissemination of good practice and policy lessons to help asylum seekers and victims of trafficking fully integrate into society Stimulate learning and build capacity for transnational cooperation among Good practice and experience ESF programme managers New tools and procedures Identify good practices through the execution of baseline studies, peer reviews and study visits Develop a common reference framework for the employment policies for the young Draw up proposals regarding national and European policies in the field of youth employment and Capitalise on this when ESF Operational Programmes revised Promotion of transnational cooperation Entrepreneurship Education and Vocational Counselling Mobility 15

28 2.3 Development of the Learning Networks themes and activities On the basis of the evidence gathered from the desk research, key stakeholder interviews, case studies and online survey, the initial development of the Learning Networks, and establishment of their thematic focus and planned activities, can be said to have been broadly effective in most cases. In part this appears to reflect the fact that a number of the networks emerged from pre-existing work and collaboration, including that undertaken as part of the EQUAL initiative in the previous ESF programming period. In the view of network leads in particular, this provided a solid basis for the development of the networks and supported the effective identification and development of network themes and activities. In some cases, networks sought to respond to specific recommendations emerging from previous work under EQUAL which provided an immediate focus to their work, as was the case with the EMPOWERMENT network for example. In other cases, Learning Networks developed out of other forums or initiatives around transnational working. Whether networks can be traced back to EQUAL, or emerged through other routes, a process of selecting a broad theme and subsequently refining it through collaborative working was a common way for networks to develop. It was also seen as an effective way of engaging multiple perspectives and ensuring that the development of the networks thematic focus was effective through being inclusive and, as one network lead put it, democratic. While, in a limited number of cases, those consulted felt that they had been unable to influence the direction of a network, in the main this aspect of the Learning Networks development was viewed positively. This impression can be reinforced with reference to the results of the online survey, wherein over 90% of responses indicated that it was possible to influence the thematic priorities of the Networks in one or more stages of their implementation. In addition, as figure 2.1 shows, a significant proportion of respondents felt that they were able to influence thematic priorities through contributing to the proposal for the network, and / or through specific discussions on such priorities shortly after the network started. Figure 2.1 Influence of participants in setting the thematic priorities of the Learning Networks Source: Online survey: 98 responses In some instances, experts were also involved in the developmental process for the Learning Networks and this was generally viewed positively where it occurred. In other cases, a process of undertaking research or developing baseline studies to inform the focus and direction of a network was used. This process was likewise seen as important and effective by most stakeholders in helping to ensure that the networks concerned started from a solid base. 16

29 Another aspect commonly cited as being effective involved the use of a two tier organisational model, using a steering group to develop a network s thematic focus and objectives, supported by working groups charged with further developing and refining particular activities. In general, partners were able to select the working group that they felt best suited their priorities or expertise. In addition, in several cases, those developing activities within a two tier structure reported basing their approach and focus on existing research or undertaking research to further refine it. While, in a limited number of cases, those consulted felt that they had been unable to influence the direction of a network, in the main this aspect of the Learning Networks development was viewed positively. The approach of refining a broad overall theme through partner dialogue was also cited by network leads and participants as being effective or important in the perceived success of a network. For example, in the case of the Gender CoP network, partners were brought together to work through the participative World Café approach to refine the focus of the network, with a series of working groups subsequently being established to develop and take forward particular activities. This process was cited as being a significant success factor by a number of those involved in this network when reflecting on the link between organisational structure and the overall effectiveness and outcomes of network activity. More broadly, the evidence gathered suggests that the way in which the Learning Networks developed and formulated their implementation plans is linked to their relative success, or otherwise, in meeting their objectives and achieving positive outcomes. As discussed in detail in section 3.3 on the organisation and leadership of the networks, such organisational factors are clearly significant in influencing outcomes. Evidence from the desk research and case studies also suggests that the processes used to develop activities were broadly effective in ensuring that they reflected and supported the objectives of the network in question. However, it was not always evident that a structured approach, ensuring that activities linked to and would be likely to support objectives, had been followed in all cases. While, in a small number of instances, networks used a log frame approach to map activities to objectives, in others these linkages were either more implicit or developed in a less structured fashion. As noted by one of the Commission stakeholders interviewed, in retrospect it may have been beneficial to provide more direction to the networks on the necessity of using a more structured, log frame based, approach. In the main, those involved with establishing the networks felt that the right partners had been engaged in their development. However, there was some variation in the extent to which the development process was largely restricted to ESF Managing Authorities, or involved a wider selection of stakeholders including, for example, relevant social partners, experts or organisations sharing the networks thematic focus. Where the latter were included they were generally seen as having been important given that the network members from Managing Authorities were not necessarily experts as such in the theme under development. However, in the cases where such wider stakeholders were not involved, the process was nonetheless seen as being effective in that the Managing Authorities in the networks concerned did have the required understanding and expertise. 2.4 Relevance of Learning Networks themes and activities to ESF and national priorities Evidence from the desk research, stakeholder interviews, case studies and online survey indicates that the objectives and themes selected by the Learning Networks have a high degree of relevance with ESF objectives, along with other high level EU policy drivers such as the Europe 2020 strategy. The nature of this relevance varies according to the broad areas of network focus as outlined above: thematic policy relevance, relevance to objectives around supporting particular marginalised or disadvantaged groups, 17

30 and links to supporting the planning, delivery and implementation of the ESF. Often these linkages are made explicit in Learning Network documentation, although the extent to which they were made clear varied across the networks. Interestingly, despite this, a small number of interviewees did feel that the focus of the networks derived more from preceding initiatives, notably EQUAL, rather than undertaking a thoroughgoing analysis of what would be appropriate in light of the specific ESF objectives for the programming period. Nonetheless, it was generally acknowledged in these cases that network objectives and themes were still relevant to a large extent. However, several stakeholders (representing both the Commission and Learning Networks) felt that the Commission might play a stronger role at the outset of any future calls for proposals on Learning Networks to ensure maximum relevance to, and support for, the ESF. As might be expected, a high degree of relevance was also apparent in respect of national priorities of participating partners, particularly in the case of partners leading networks. For example, in respect of the BFSE network, the relevance of the social economy to both the Polish ESF Managing Authority and to the country s economic strategies and policies was referenced. It was also noted that most of the network members were drawn from Member States with a high level of interest in this issue. However, in common with a number of other networks, the point was also made that partners came from different contexts in terms of the level of development and focus on the themes under consideration. Thus, while the participation of some Member States related to the high level of policy visibility or importance of the issue concerned, for others participation was more about learning in an area which may develop or become more of a priority in future. Relevance in this sense was also discussed with reference to contextual factors common to many Member States, such as difficulties in supporting particular marginalised groups to access the labour market. Examples include the point raised by some of those involved in the ExOCoP network that, in many countries, specific support programmes for ex-offenders are rare and/or are often one of the first to be cut in times of budget reductions. Similarly, in respect of the IMPART network, it was noted that the issue of integrating migrants is one that most Member States face to some degree, whilst interviewees connected to the ENYE network noted that youth employment is one of the largest challenges facing some Member States in the wake of the economic crisis. In a small number of cases, Member States that initially elected to participate in networks subsequently withdrew, in some cases as they had decided that the issues under consideration were not particularly relevant to their national context. Equally, in a couple of cases, AEIDL monitoring reports make reference to members of networks being asked to leave as it was felt to be clear they had no real interest or commitment to the issues under discussion, or that they were not relevant to the national priorities concerned. In other instances, shifts in national policy or priorities, sometimes resulting from elections and changes in government, were referenced by stakeholders as leading to a reduced level of commitment or participation by some partners. Nonetheless, as noted, it appears that in the vast majority of cases the themes and focus of the network were felt by partners to be highly relevant to their national context. The Member State case studies served to further confirm the relevance of the thematic focus of the Learning Networks to concerns at Member State level. In general, the majority of interviewees consulted felt there was a clear linkage between the thematic focus of the networks and key policy concerns at the Member State level. While the level of this relevance and linkage varied between the Member States concerned, and across the different networks, in all cases particular networks with a high degree of relevance were identified. For example, interviewees in the Spanish context noted that the ENYE network was clearly related to a key national priority in terms of addressing youth unemployment. Similarly, in the 18

31 Flemish context, it was noted that the BFSE network was timely given that the Flemish Government was in the process of developing a new strategy around the social economy. The relevance of particular networks to more operational concerns on the part of ESF MAs and other organisations in the national context was also confirmed through the Member State case studies. In each of the cases concerned, examples were offered as to why engagement in the networks was of direct relevance. In the case of the Czech Republic, for example, engagement in the RBM and ESF AGE networks was seen as directly supporting the development of the administration of the ESF programme (in the case of RBM) and informing the development of initiatives for older workers (in the case of ESF AGE). In the Spanish context, an interviewee involved with SaviAV stressed the relevance of study visits conducted to reception centres for asylum seekers and refugees in other countries. It was noted that such activity offered the opportunity to see concrete examples of good practice and exchange information between reception centre officers. While those interviewed in the Member State context generally gave positive views concerning the networks relevance, in a small number of cases individuals noted that the presumed relevance of a network did not prove to be the case in practice. Generally this was due to the precise focus of the network concerned, or the particular activities it developed, not proving to be particularly relevant even though the broad theme involved did have relevance. However, this was generally viewed as an inevitable function of networks being able to define and shape their focus over time and that it was unrealistic to expect all aspects of networks to be directly relevant to national contexts. Equally, activities which did not prove to be as relevant as anticipated were seen as being uncommon in comparison to activities which did prove relevant. Evidence also emerged from the online survey regarding the relevance of the activities of the Learning Networks to policies and programmes. As figure 2.2 below shows, almost all main activities were considered to be relevant or very relevant on all policy levels (ESF, national and EU) by at least 60% of responses received. The highest degree of relevance was, however, in relation to the ESF. On all three policy levels, the learning seminars were considered to be most relevant, whereas the baseline reports were considered least relevant. Figure 2.2 Relevance of Learning Network activities for different policy levels Source: Online survey: 98 responses 19

32 Further evidence in relation to relevance was provided through the opportunity provided by the online survey to investigate motivations for participation in the Learning Networks on the individual level. As figure 2.3 shows, the most common reasons for participation are that individuals involved felt that the networks and their focus were relevant to their own work (85%) and had relevance from the perspective of informing policies in their Member State (80%). Other common motivating factors included the opportunity to share good practice and the potential to inform the work of their organisations. As figure 2.3 also shows, promotion of Member State policies or thematic priorities were less common motivating factors. Figure 2.3 Reasons for participating in Learning Networks Source: Online survey: 98 responses Effects of the economic crisis Undertaking the key stakeholder interviews and case studies also gave the opportunity to explore whether and how the economic crisis had affected the thematic priorities or activities developed. In general the main impacts reported were practical and logistical, particularly in terms of reduced ability to travel on the part of some partners from Member States facing significant austerity measures. In some cases, this was reported as hampering or delaying planned activities and leading to additional financial pressures on the network. Other effects reported included the crisis leading to increased interest in the cost effectiveness of models and approaches for integrating asylum seekers, in the case of the SaviAV network, and enhancing the importance of maintaining the visibility of gender considerations in the case of the Gender CoP network (given the potential for these considerations to be downgraded while Member States focused on their economic situation). Despite these effects, none of the networks reported significant changes to thematic priorities or activities, though in some instances the crisis gave a slightly different focus to particular planned conferences, peer reviews or other activities. It was also noted in one or two cases that the crisis raised the importance of the network s activities; this was seen as being the case for the ESF AGE network for example. Interestingly, some of the network leads and members interviewed also raised other external factors as having had more of an impact than the crisis itself. Most commonly these factors related to the electoral cycle within participating Member States, with changes in government altering the priorities of network members and, in some cases, leading to a changed or reduced commitment to network activities. 20

33 This theme of the impact of electoral cycles and the effect of changes in Government also emerged strongly in respect of some of the Member State case studies, particularly in relation to the UK and the Czech Republic. Again, such effects were seen as more significant than the crisis itself. They were also linked on the part of those interviewed for the Member State case studies to effects on Learning Network participation and outcomes, as is discussed further in subsequent chapters of this report. 2.5 Complementarity between Learning Networks The desk research undertaken at the outset of the evaluation highlighted that there were some clear potential linkages between the thematic focus and activities of the different Learning Networks. In turn this suggests that potential synergies and examples of collaborative working might have been expected to emerge. However, it is clear that the extent of collaboration between Learning Networks varied. In a number of cases, network leads or members noted that there had been little collaboration or contact beyond the co-ordination meetings facilitated by AEIDL and the Commission. In some cases this was related to the fact that co-ordination of, or involvement in, a network required significant inputs, hence reducing any time that might be available for collaboration. In others, those involved with a network felt that there would be limited value from engaging in such cross network activity given (in their view) the particular nature of the network not having any obvious crossover with others. In one or two instances those involved in networks did recognise potential links with the activities undertaken by other networks and had held some meetings, although nothing concrete was felt to have emerged in these cases. In other cases those involved with some of the networks noted that there had been collaboration and this was felt to have led to some positive effects or outcomes. Examples included collaboration between the ExOCoP network and the network on transnational co-operation (on dissemination materials and the publication of articles) along with some collaboration between the ACB and RBM networks given the overlap in their thematic focus. Interestingly, some of the networks found it more beneficial to focus on making linkages with thematic networks outside of the ESF Learning Networks themselves. This was the case, for example, with the BFSE network where successful links were made with complementary European networks and organisations in the field including REVES (the European Network of Cities and Regions for the Social Economy), DIESIS (European Research and Development Service for the Social Economy), and Social Economy Europe. 2.6 Summary of key findings It is evident that the 13 Learning Networks assessed as part of the evaluation demonstrate significant breadth and scope in terms of their objectives. These objectives tended to focus broadly on one of three main areas (although there is some overlap): a thematic / policy focus, a concentration on ESF target groups, or operational concerns relevant to the delivery of the ESF. This breadth of focus was generally viewed as being a positive aspect of the overall Learning Network approach, though a minority view was expressed over the spread of thematic focus and objectives being too wide. The establishment of the Learning Networks thematic focus and planned activities can be said to have been broadly effective in most cases. In part this reflects the fact that a number of the networks emerged from pre-existing work and collaboration which was felt to have provided a solid basis for the networks development. Commonly the development process involved selecting a broad theme and subsequently refining this through collaborative working. This was widely seen as a positive way to engage multiple 21

34 perspectives and ensure that the development of the networks thematic focus was effective through being inclusive and democratic. Another aspect commonly cited as effective involved the use of a two tier organisational model, using a steering group to develop a network s thematic focus and objectives supported by working groups charged with further developing and refining particular activities. This was cited as a significant success factor by a number of those involved in the networks when reflecting on the link between organisational structure and the overall effectiveness and outcomes of network activity. The evidence collated also suggests that the processes used to develop activities were broadly effective in ensuring that they reflected and supported the objectives of the networks. However, it was not always evident that a structured approach, ensuring that activities linked to and would be likely to support objectives, had been followed in all cases. In the main, those involved with establishing the networks also felt that the right partners had been engaged in their development. However, there was some variation in the extent to which the development process was largely restricted to ESF Managing Authorities, or involved a wider selection of stakeholders. It is clear that the objectives and themes selected by the Learning Networks have a high degree of relevance to ESF objectives, along with other high level EU policy drivers such as Europe As might be expected, a high degree of relevance was also apparent in respect of national priorities of participating partners, particularly in the case of partners leading networks. This relevance was confirmed through consultations at the Member State level as being apparent both in respect of policy concerns and in relation to operational issues facing ESF MAs and other organisations. Evidence from the online survey undertaken similarly demonstrated that all the main network activities were considered to be relevant or very relevant on all policy levels (ESF, national and EU). While there were some clear potential linkages between the thematic focus and activities of the different Learning Networks, it is clear that the extent of collaboration between them varied, but was overall limited. In some cases this was related to the fact that co-ordination of, or involvement in, a network required significant inputs, hence reducing any time that might be available for collaboration. In others, those involved with a network felt that there would be limited value from engaging in such cross network activity. Finally, the study also gave the opportunity to explore whether and how the economic crisis had affected the thematic priorities or activities developed. In general the main impacts reported were practical and logistical, particularly in terms of reduced ability to travel on the part of some partners from Member States facing significant austerity measures. Some of the network leads and members felt that other external factors (that is, change of government or elections) had had more of an impact than the crisis itself. 22

35 3.0 Effectiveness and efficiency: processes and inputs 3.1 Introduction This chapter examines issues relating to the effectiveness and efficiency of the Learning Networks in terms of key processes and inputs surrounding their development and implementation. Effectiveness is first examined from the perspective of participation, both in relation to the spread of Member State participation in the networks and in terms of the networks individual composition. Motivations for participation are also briefly explored. The chapter then considers the effectiveness of the Learning Networks in terms of their organisation, including the effectiveness of the network leadership role. The effectiveness and quality of Learning Network activities is then examined, prior to an assessment being made of the networks efficiency in terms of their delivery and use of resources. The final section then summarises the key findings from the preceding analysis. 3.2 Effectiveness of Learning Network reach and participation Based on desk research undertaken at the outset of the evaluation, figure 3.1 illustrates the way in which participation as both member and lead partner is unevenly distributed across Member States. A group of Member States were represented in networks six times or more, although it should be recognised that, within each Member State, different organisations may be the participant body. This grouping included Belgium (Flanders), Czech Republic, Germany, Greece, Italy, Poland, Spain and Sweden. Conversely a number of countries participated in only one or no networks Austria, Bulgaria, Cyprus, Denmark, Ireland, Latvia, Luxembourg, Malta and Slovakia. Lead partner status is even more concentrated, with Germany having led four networks, Belgium (Flanders) three, and Poland two. Figure 3.1 Country participation in the ESF Learning Networks While a number of those interviewed in the key stakeholder consultations and through the Learning Network case studies felt that a more even spread of participation at the Member State level would have been desirable, some explanatory factors were also noted for the pattern of uneven participation that is apparent. For example, it was noted that the lack of a specific requirement for Member State engagement 23

36 at a particular level meant that inevitably an uneven pattern of participation would result. Equally, the differential level of resources available to ESF MAs was offered as an explanatory factor. This was confirmed in the interviews conducted with Member State representatives of countries that had only limited participation in the Learning Networks. In the case of Portugal, for example, a lack of human resources and time to participate was cited as a key constraint. As discussed further in chapter six, a number of stakeholders also offered suggestions for encouraging a more even spread of participation. These commonly included a greater promotional role on the part of the Commission in articulating the importance and benefits of transnational co-operation, more compulsion to co-operate for ESF MAs, and enhanced technical assistance to help address issues of limited resources or transnational experience on the part of some Member States. In one or two cases, the need for Learning Networks and similar transnational initiatives to be more pro-active in seeking a spread of representation was also cited. In addition, while the Commission took an active role in helping with the constitution of some of the networks, supporting with their composition and suggesting additional members or experts, the absence of such a role in assisting with this aspect was noted in a minority of cases. In these instances it was felt by network leads in particular that greater support or assistance in this aspect would have been helpful. In terms of participation at the level of specific organisations and individuals, when it came to recruiting participants and partners, it is evident that several networks did take a pro-active and inclusive approach by opening up their activities to Non-Governmental Organisations (NGOs), policy-makers, experts and practitioners. This spread of engagement was highlighted as a key positive aspect of networks in several cases and, in general, the degree and balance of representation across networks appeared to be appropriate and effective. For example, when considered from the national perspective on the basis of the Member State case studies, while levels and types of engagement varied in the contexts assessed, in general it is clear that a wide range of types of organisation contributed to and participated in the Learning Networks. Typically, these included the national ESF MA (and regional MAs where applicable), Labour Ministry, Public Employment Service (PES), other government departments and ministries, and NGOs. The evidence reviewed indicates that the quality of the individual participants in the Learning Networks was more varied and it is clear that some networks experienced variations in the level of commitment by participants. For example, about half of the network leads interviewed for the Learning Network case studies expressed that they were happy with the quality and commitment of their network members, while the others reported that the quality of participants was either not satisfactory or varied significantly. In such instances it was noted that a small committed group drove the majority of activities. While this was seen as having some advantages in terms of ensuring focus, the limitations and negative effects of this variable commitment were also often acknowledged. This aspect was also seen as a particular issue when key network members dropped out for a variety of personal or work-related reasons, leaving a gap that was difficult to fill. In addition, in one or two cases, the high turnover of network members was noted as problematic and as affecting the overall effectiveness and success of the networks concerned. A minority of those participating in the networks also felt that membership could have been expanded further to ensure that, for example, those working on the ground, such as ESF project managers, could be included to a greater extent. This was felt to have a potential benefit in terms of ensuring that the work of networks was grounded more in real world issues, hence protecting activities and outputs from becoming too abstract or theoretical. Equally, though, arguments against such an approach were raised, with some of these emerging through the Member State case studies. For example, several interviewees in the UK context felt that the most appropriate way to engage the ESF project level was through a coordinating infrastructure at the Member State level. This infrastructure was seen as the most effective and 24

37 appropriate mechanism for then channelling lessons on practice through to higher and more strategically focused transnational forums such as the Learning Networks Motivations for participation The Member State and Learning Network case studies also served to illustrate the motivation for Member States and organisational / individual actors within them to engage in Learning Network activity. This revealed a typical pattern of a combination of individual interest, organisational interest and the desire to share learning or learn from others. While, as noted in the previous chapter, the Learning Networks were generally considered to have a high degree of relevance to Member State policy considerations, this did not always appear to be a primary motivating factor for engagement. In contrast, a more general desire to share learning in an area of thematic interest (whether individual or organisational), without necessarily being driven to directly inform policy development or transfer, was more commonly discussed by interviewees. Interestingly, in the Spanish context, it was also noted that engagement with some networks (e.g. ESF AGE, Gender CoP) enabled some topics such as older workers and gender to be kept on the agenda, whereas otherwise they may have lost priority due to the challenging economic context in which the country found itself. This was also cited as a factor in the decision of some organisations to engage with the Learning Networks for example the Instituto de la Mujer, for whom engagement was partly driven by maintaining the visibility of the Institute and women s issues at a time of declining Government support. 3.3 Effectiveness of Learning Network organisation and leadership The available evidence indicates that the effectiveness of organisational processes within Learning Networks varied significantly. In addition, these processes were generally cited by network participants as an important factor in determining the success or otherwise of particular networks. In cases where there was a consensus that the organisational structures and processes within a particular network were effective, commonly cited factors included effective planning and leadership, clearly delineated steering and working groups with specific and clear remits, clear lines of governance and accountability and effective project management skills on the part of network leads or co-ordinators. The role of an effective Secretariat in organising events and addressing logistical issues was also commonly noted. In the case of networks such as Gender CoP, factors such as these were seen by network partners and other stakeholders as being key to their perceived success. Conversely, it is clear that in several cases organisational processes can be identified as a key issue which negatively affected participants views of the network in which they were involved. In particular, overly complex organisational or governance processes were cited, with examples including the view that the networks concerned had too many working groups or were not streamlined enough. In some instances, the lack of leadership or organisation on the part of those leading a network was also cited, along with a lack of project management and delivery expertise. In such cases, interviewees often felt that more guidance and support for network leads in terms of actually developing and running networks would have been helpful. Typically, these process or organisational weaknesses appear to have been a key factor in particular networks failing to achieve their intended outcomes or doing so to a lesser extent than had been anticipated. Such issues were also commonly cited when network members and other stakeholders were asked to reflect on the main weaknesses of the network or networks they were involved with, or had knowledge of. Additionally, in one or two cases, networks being overly ambitious in what they sought to 25

38 achieve was also raised as a contributory factor in not achieving intended outcomes, irrespective of the effectiveness or otherwise of their organisation and leadership. It is also clear that, in several cases, linkages between organisational structures and network composition were important in ensuring that these two aspects reinforced each other to support effectiveness in delivery and the production of outputs. The composition, skills and attitude of network members were seen as an important success factor by a number of stakeholders. Equally, in some cases, network leads emphasised that it was their inclusive and transparent approach, featuring a flat organisational structure, which had proved especially successful. Moreover, the involvement of external experts was highlighted by some network leads as being especially successful with regards to the outputs produced and the provision of support. Equally, it was acknowledged that the composition of network members could also pose a challenge; several networks, for example, found it difficult to bridge the gap between policy-makers and practitioners and to balance the involvement of both. In respect of organisational and process related challenges, a significant number of network leads cited work load intensity and resource requirements as the key challenge. This may be due to the fact that much of the administrative work was centralised within the lead partner and lead organisations were largely responsible for outputs to the Commission. However, when work was shared more widely with network partners, several stakeholders identified a lack of capacity and skills at Member State level as challenge. In several instances, these factors were cited as indicating that an enhanced level of technical assistance would be beneficial in supporting network organisation in any future iterations of networks or similar transnational activity. As discussed further in chapter six, such a theme also emerged strongly from the virtual Policy Delphi process arranged so that selected stakeholders could reflect on potential future improvements and developments in the context of transnational co-operation. Generally speaking, it seems that different organisational modus operandi seemed to work for different networks. This can be exemplified by the fact that, while one network identified the operational split between steering and working groups as a success factor, another claimed that it was exactly the fact that the entire steering committee went through all activities together that made their network successful. This may indicate that the organisational structure adopted for such activities needs to reflect the size and particular composition of the networks, in terms of the individuals involved and the available resource that they can offer, in addition to the focus and nature of activities. This is further illustrated by the fact that one network lead, whose network tried to involve all members in all working groups, stated that this did not work well and that members should only be involved in the working groups most relevant for them in the future. Again, these contrasting views illustrate that internal processes have to be tailored to the size, objectives and activities of each network and that there is no one-size-fits-all solution. However, as noted above, it is also clear that certain factors such as effective and organised leadership are key common indicators of likely effectiveness or otherwise. 3.4 Effectiveness and quality of Learning Network activities As illustrated by desk research at the outset of the evaluation, the type of activities implemented by the networks varied considerably. Similarly, the views of stakeholders from the Commission and Learning Networks shared little commonality in terms of the type of activities which were considered to have been successful or unsuccessful. In general, however, the majority of Learning Network representatives emphasised the success of their final results or outputs, be it a final publication, event or toolkit. Equally, there was a broad consensus that activities such as study visits, peer reviews and seminars were generally successful. 26

39 Other forms of activity commonly cited as effective and successful included: Activities which combined policy and practice and produced tailored outputs for these different target groups Activities which were embedded in other relevant events and implemented in collaboration with external stakeholders, for example Commission Directorate-Generals Presentation and discussion of good practices, in particular when these were discussed practically and scrutinised collectively with regards to their transferability The development of practical tools with the potential for use such as common standards, handbooks or practical guides. In cases where activities were evidently less effective or successful, a common reason for this appeared to be that the goals outlined initially were too ambitious given the time available and resource constraints faced by those involved. Certainly, in the case of several networks, it was clear that not all activities could be implemented successfully. Equally, however, the implementation of some activities did not occur as planned due to other factors, such as changes in context or it becoming clear that a particular activity was not needed or was unlikely to work well. This indicates that it was not necessarily the absence of effectiveness in activities that led to them being unsuccessful in terms of being realised. Indeed, the flexibility of networks in deciding that, for different reasons, activities initially planned should not be taken forward was seen as a positive thing by some of those involved. In addition, it is clear that the lack of effectiveness or quality of particular activities was related to factors specific to a particular network in several cases. As such, network partners reported a variety of activities to be unsuccessful in the specific context of that network, whether these were voluntary study visits, events, tools, or activities to share good practice. Given the specificity of these issues, there appears to be few common reasons for this perceived lack of effectiveness, though the overly ambitious nature of some activities was commonly cited as discussed above. Linked to the above discussion of the importance of organisational structure and effective leadership and its interplay with effective implementation, in several cases a lack of effective planning in identifying likely time and resource requirements of activities was referenced as a factor in their lack of success. Equally, it is clear that more general issues around the lack of effective planning or organisation played a role in particular activities being less effectively implemented or proving to be of a lower level of quality than they might have been. For example, this factor was cited in respect of events and conferences not working as well as expected in several instances, along with audiences not being clearly identified or effectively engaged, Despite these issues arising in some networks, on the balance of evidence it appears that the activities taken forward by the Learning Networks were implemented in a broadly effective fashion in the majority of cases. They were also broadly effective in the sense of reflecting and helping to support the achievement of the networks aims. In particular, the importance of activities which were interactive, enabling knowledge exchange and mutual learning, were emphasised in this context. This reflects the view of a number of those interviewed who stressed the importance of activities being implemented in a way that was engaging for participants and which encouraged their commitment. 3.5 Efficiency of delivery and use of resources Judgements on the efficiency of the networks in terms of their implementation and use of available resources link closely to the above discussion over the effectiveness of organisation and leadership. In 27

40 general, while the Learning Networks can be judged to have operated efficiently in terms of allocating resources to implement activities, there were some variations apparent at the level of individual networks. In some instances a lack of effective planning, or the inadequate development of activities, were cited by those involved in networks as representing inefficiencies. As such, in some cases, activities such as conferences or events could be judged as not providing a reasonable output relative to the inputs (costs and human resources) involved in their production. Conversely, it is certainly the case that many of the activities arranged by the networks were done on a limited budget which depended on notable in-kind resources being allocated by those involved in delivering them. In most cases this led network leads to strongly argue that the use of resources to support activities had been broadly efficient or, in some cases, very efficient. It should also be noted that no instances of resources being significantly wasted appeared in the evidence gathered and reviewed. Rather, as reflected above, the general view of stakeholders was that some individual activities could have been more efficiently implemented but, when viewed as a whole, the networks should be seen as being relatively efficient from a resource input standpoint. 3.6 Summary of key findings Participation in the Learning Networks and taking the role of lead partner was unevenly distributed across Member States. A group of Member States were represented in networks six times or more while several countries participated in only one or no networks. While a more even spread of participation at the Member State level can be seen as desirable, clear explanatory factors for this unevenness were evident. These included the lack of a specific requirement for Member State engagement in the networks and the differential level of resources available to ESF MAs to support their participation. Suggestions for encouraging a more even spread of participation included a greater promotional role on the part of the Commission in articulating the importance of transnational co-operation, more compulsion to co-operate for ESF MAs, and enhanced technical assistance to help address issues of limited resources or transnational expertise / experience. In terms of participation at the level of specific organisations and individuals, it is evident that several networks did take a pro-active and inclusive approach by opening up their activities to NGOs, policymakers, experts and practitioners. This spread of engagement can be identified as a key positive aspect of the networks in several cases. Equally, in general, the degree and balance of representation across the networks appeared to be appropriate and effective. However, the quality of the individual participants in the Learning Networks was more varied and it is clear that some networks experienced variations in the level of commitment by members. The evidence gathered also demonstrated that Member States and organisational / individual actors within them to engage in Learning Network activity due to a combination of individual interest, organisational interest and the desire to share learning or learn from others. The available evidence also indicated that the effectiveness of organisational processes within Learning Networks varied significantly. In addition, these processes clearly represent an important factor in determining the success or otherwise of particular networks. Generally speaking, it seems that different organisational modus operandi seemed to work for different networks. In turn this indicates that the organisational structure required to effectively implement network activities needs to reflect the size, particular composition and focus of the networks. In cases of effective practice, common factors included effective planning and leadership, clearly delineated steering and working groups with specific and clear remits, clear lines of governance and accountability and effective project management. Weaknesses in these organisational aspects appeared to have been a key factor in particular networks not achieving their intended outcomes or doing so to a 28

41 lesser extent than had been anticipated. In respect of organisational and process related challenges, work load intensity and the level of resource requirements (particularly for network leads) was clearly a key challenge. As noted by stakeholders, this indicates that an enhanced level of technical assistance would be beneficial in supporting network organisation in any future iterations of networks or similar transnational activity. Despite the variations in organisational effectiveness which were apparent, on the balance of evidence it appears that the activities taken forward by the Learning Networks were implemented in a broadly effective fashion in the majority of cases. They were also generally effective in the sense of reflecting and helping to support the achievement of the networks aims. In particular, the importance of activities which were interactive, enabling knowledge exchange and mutual learning, were emphasised in this context. In cases where activities were evidently less effective or successful, a common reason for this appeared to be that the goals outlined initially were too ambitious given the time available and resource constraints faced by those involved. Finally, judgements over the efficiency of the networks in terms of their implementation and use of available resources link closely to the above discussion over the effectiveness of organisation and leadership. In general, while the Learning Networks can be judged to have operated efficiently in terms of allocating resources to implement activities, some variations were apparent at the level of individual networks. However, it is certainly the case that many of the activities arranged by the networks were carried out on a limited budget which depended on notable in-kind resources being allocated by those involved in delivering them. This can be viewed as a key factor in supporting efficient implementation from the point of view of outcomes achieved against inputs in terms of resources used. 29

42 4.0 Effectiveness and efficiency: dissemination and outcomes 4.1 Introduction This chapter examines issues relating to the Learning Networks dissemination activity and the outcomes they achieved. The effectiveness of dissemination strategies and activities are considered first prior to Learning Network outcomes. Outcomes are examined in respect of individual learning and organisational learning and change, before assessing the degree to which the networks led to policy transfer and learning along with the nature of this. The chapter then considers additional outcomes stemming from network activity before assessing the effectiveness of the networks in meeting their aims, both individually and collectively. The sustainability of the Learning Networks and their effects are then examined before finally assessing the efficiency of the networks in generating outcomes and impacts. The chapter concludes by providing a summary of key findings from the assessment of dissemination and outcomes undertaken. 4.2 Effectiveness of dissemination The evidence gathered indicates that the dissemination approaches and strategies adopted by the Learning Networks proved to be mixed in terms of their effectiveness. While, in some cases, dissemination products were cited as being effective in transmitting the results of network activity by those consulted in the Learning Network and Member State case studies, this was not universally the case. Equally, while in some instances stakeholders cited that networks had produced a dissemination strategy that offered a useful structure for sharing results, in others it was acknowledged that such a strategy was either under-developed or proved to be inappropriate. In the main, the latter factor was cited where outputs, whether in the form of reports or other media, were not tailored or suited to the audience in question. Typically, the lack of short, concise and recommendation-focused outputs suitable for policy audiences was highlighted in this context. The role of individual network members was also seen as a factor affecting the success of dissemination, with the anticipated level of commitment in contributing to dissemination within Member States not having materialised in some cases. Equally, a lack of consistent follow-up where dissemination was initially through events, websites or mailing lists was seen as reducing the potential effectiveness of approaches at times. Conversely, where dissemination was cited as being more effective, the use of partners as ambassadors to share aspects of network learning within Member States was cited as a significant success factor. Drawing on the Member State case studies, in the case of the UK, the infrastructure developed at the Member State level was also seen as a supporting factor in terms of role of the Innovation, Transnationality and Mainstreaming (ITM) unit established in part to co-ordinate the two-way sharing of learning in respect of initiatives such as the networks. Similarly, building effective and on-going links with key players at EU and Member State level was cited as a key pillar of a successful dissemination strategy. While, as noted, the quality and suitability of the tools and outputs developed for dissemination appears to have varied, in some cases the perceived good quality of materials and tools for supporting dissemination was also commented on. The good quality of materials was cited by some interviewees in the Member State case studies, for example, as being helpful in sharing learning amongst their domestic networks. In particular, clear and concise reports 30

43 or the use of e-newsletters and regular briefings were highlighted as tools that had worked well in different contexts. More generally, the specific mechanisms used for dissemination were seen as a key factor in determining the success or otherwise of the approaches taken. In most cases, more passive forms of dissemination such as printed materials or those only hosted on websites were judged to be less effective than more active forms of dissemination through events, meetings or on-going liaison with the targets of dissemination activity. In particular, where Learning Networks had used websites as a key dissemination mechanism, perspectives on its effectiveness varied. In some cases limited effectiveness was cited in terms of the amount that materials were downloaded or accessed and in respect of the more passive nature of this medium. However, for some networks, web hosting was seen as valuable, although mainly as an addition to wider dissemination activity. In some cases, dissemination and its effectiveness were discussed by those involved in the networks and consulted through the Member State case studies in terms of levels where, for example, first order dissemination mechanisms such as reports, conferences, or web-hosting were felt to have worked well, but more ambitious plans for dissemination had been less effective. Typically, interviewees making this point linked dissemination to mainstreaming and felt that, while results and findings had been disseminated, more thoroughgoing dissemination in terms of materials, ideas or approaches being adopted by partner organisations or ESF MAs had worked less well. Such a perspective represented the general view of those interviewed for the Flemish and UK Member State case studies, for example. While some influence on the ESF MAs was noted in these instances, some interviewees commented that there was some resistance on the part of other Government departments to taking on board learning or even sometimes being open to discussing it. Evidence from the online survey appears to confirm this pattern, certainly in respect of ESF MAs. As figure 4.1 below illustrates, while results were generally felt to have reached ESF MAs to a great extent or somewhat in over 90% of cases, responses indicate that ESF MAs were felt to have acted on the results disseminated to a great extent or somewhat in only 55% of cases. Figure 4.1 Reach and use of Learning Network results by ESF MAs Source: Online survey: 91 responses However, there were also exceptions to the picture presented above, particularly in cases where networks had been successful in engaging with policy-makers and higher level officials, whether at the EU or Member State level, and / or where such individuals were directly engaged in networks. In the case of Spain, for example, the participation of the Head of the PES in the ESF AGE network was seen as significant in influencing a greater commitment to addressing the needs of older workers in the country s employment policies. Other examples of the effectiveness of dissemination in this deeper sense in the Spanish context included the manual for entrepreneurial guidance developed by COPIE being used by 31

44 Galicia as part of their self-employment orientation programme. Likewise, following an example from a study visit to Finland in the context of the SaviAV network, the Rede Ariadna (an NGO working with asylum-seekers) produced an information leaflet including practical information on which basic needs of asylum-seekers are covered and how. Interestingly, there was wide variation amongst stakeholders in terms of the level at which dissemination and learning was felt to have been most effective across the networks (that is whether dissemination most effectively reached the EU, national or regional / local actors and which organisations within each of these levels had been reached). In some cases, learning was commonly cited by those involved in the networks to have occurred at each of these levels, albeit to varying extents. In others, the view of such stakeholders was that dissemination and learning had worked particularly well at one or other of these levels, but not at others. Equally, variation was noted in respect of the types of organisations engaged through dissemination activity and the relative success of their involvement. Typically, and as would be expected given the focus of the networks, labour and/or social ministries were engaged more frequently than other government departments, along with ESF MAs and intermediate bodies. Attempts at engaging with other ministries were generally cited as being either less common or less successful. Evidence available from the online survey reflects this variation in terms of the reach of dissemination at Member State level and also suggests that, in general, ESF MAs were the organisations reached to a much greater extent than other actors. As figure 4.2 below shows, in most cases ESF MAs were felt to have been reached to a significant extent, as were ministries of labour, although to a lesser degree. NGOs, representatives from other ministries, and other actors such as employers and trade unions were generally cited as having been reached by dissemination activity to a lesser extent. Figure 4.2 Type of organisations reached within Member States through dissemination Source: Online survey: 95 responses In part, these variations in the level and types of organisation reached by dissemination activity relates to the focus and aims of each of the networks that is, whether the intent was to effect change at the EU, Member State, or regional levels and in respect of which actors. However, other factors were also seen as playing a role in where dissemination or learning proved most effective or apparent. These included the nature of links built by the network for example, a greater impact at EU level had been observed 32

45 where networks had linked in with key individuals or parts of organisations at EU level. The same pattern held true where such links were stronger at Member State or regional levels. A related factor was the degree to which actors at particular levels were felt to have been receptive to the work of the networks. Also, in one or two cases, stakeholders discussed the context in which actors at these levels were operating given that, for example, factors such as how busy they were or whether they were dealing with the effects of the economic crisis influenced the level of receptiveness encountered. It should also be acknowledged that, in some cases, the stakeholders interviewed, along with final network reports, cited that it is too early to fully assess the overall effectiveness of dissemination. This was particularly the case where a core part of a network s dissemination approach involved the distribution and take-up of particular toolkits or products, since the real effectiveness of this activity was likely to only emerge over time. In the case of the Gender CoP network, for example, one of the key outputs a common standard for gender mainstreaming was only at the stage of being piloted in selected Member States at the time of the evaluation fieldwork. As such it was felt that a full assessment of effectiveness would be premature. A number of suggestions were also made by stakeholders in terms of how they felt dissemination and learning might have worked more effectively or could be improved in similar contexts in future. Ideas for such improvements included: Network leads discussing with members in a more systematic fashion about how they are going to address dissemination in their own country and encouraging them to develop plans for this Developing an infrastructure at Member State level, whether through the ESF MA, Labour Ministry or ESF Intermediate Body, focused on transnational working and disseminating lessons from and to the EU level Better links with, and more support from, the geographical desks in the Commission for disseminating learning and outcomes within their respective Member States More of a focus on developing detailed strategies for mainstreaming outcomes, such as identifying key targets and more consistently following-up to check on, and encourage, incorporation of products and toolkits in mainstream policy and practice. 4.3 Extent and nature of individual learning outcomes The clearest and most consistently positive outcomes from the Learning Networks emerged at the level of individual learning amongst those participating. Whilst this is to be expected given that participation offered network members a direct link into the activities of the networks, and the learning accruing from them, the evidence illustrated the degree to which the vast majority of members felt that notable positive effects had resulted from their engagement. In particular, participation was cited by those consulted as providing them with an increased network of contacts, enhanced individual relationships with those working in similar fields, and deeper knowledge about particular thematic areas and available approaches relevant to their work. In addition, more intangible benefits were commonly cited by those involved in the networks. These included, in particular, the feeling that participation engendered a greater sense of commonality with partners from different Member States and contexts in terms of working towards shared goals. A number of stakeholders also cited that they benefited in terms of their own professional development. 33

46 Stakeholders tended to cite a number of common issues as being the most positive aspects of their participation. The transfer of knowledge and sharing of ideas or good practice was most often referenced, with this seen as reflecting the core aims and value of such transnational co-operation. Equally, in the case of several of the networks, the outputs produced were viewed in this light, while the role of the networks in raising awareness of particular issues such as migrant integration or gender mainstreaming was another common theme discussed. In one or two cases, the commitment of individual members to the network and its aims was also noted as one of the most positive aspects of the experience on the part of those leading or co-ordinating networks. In the small minority of cases where individuals felt that their involvement with the Learning Networks had been less positive, this generally related to the focus and activities of the network concerned in the sense of these not meeting their expectations. In some cases, the less positive experience was due to the network having developed in a different direction from that which the individual expected. In other instances, a perceived lack of quality of particular activities individuals were involved with, such as peer reviews or study visits, led them to reflect less positively on individual learning outcomes. Equally, in one or two cases, individuals felt that the quality of debate or interaction amongst network members was less positive than they had hoped, which in turn was seen as lessening the value of any individual learning. 4.4 Extent and nature of organisational learning and change In many cases organisations clearly gained from the knowledge exchange involved in the networks, along with the opportunity to develop partnership and cooperation with related actors across Member States. Numerous examples of such organisational learning were provided through the Learning Network and Member State case studies in particular, with these also translating into more concrete organisational change in some instances. On balance, such positive examples tended to outweigh cases where networks were felt to have had a limited impact on organisations with the potential to benefit from their work, or where, in particular Member State contexts, it was felt that little organisational change had occurred as a result of network activity. Examples of positive effects at the level of organisational learning and change can be illustrated through reference to the ExOCoP network. In Germany, the five Länder that were partners in the network acted effectively as ambassadors to push forward the offender resettlement agenda at regional and national levels. The partners involved succeeded in persuading both the ESF MA and the German Federal Administration to sign the Berlin declaration stemming from the network s activity, lobbied the ESF MA to adopt ExOCoP ideas, and now run spin-off projects from ExOCoP implemented at the national level. A number of examples of such organisational learning and change also emerged through the Member State case studies. In the case of Spain, for instance, the Andalusian PES adopted a Dutch example of Mobility Centres encountered in the ESF Age network to support older workers to establish their own Early Assistance Units. These centres aim to advise workers on opportunities to find a new job at an early stage when they are at risk of redundancy and before they become unemployed. The same service also drew inspiration from practice in Flanders to set up a one-stop shop. Similarly, in the Spanish context, it was noted that a number of effects relating to organisational learning can be identified at the regional level. These include, for example, the use of tools produced by COPIE by personnel in regional administrations, or planning for the development of microcredit programmes in certain localities based on the work of the network. It was also clear that in some contexts the work of the networks was felt to have had less of an impact on the organisational level. Sometimes this was linked by stakeholders to their perception that results, 34

47 tools or other outcomes from network activity were either not relevant to particular Member State contexts, and the organisations within them, or that these outputs had not proved to be of the requisite quality or usefulness. Inconsistent or ineffective dissemination was also seen as a factor in this as discussed above. Finally, in some of the contexts examined through the Member State case studies, a perceived reluctance of organisations to engage with the networks and their outputs was noted. However, it was less clear whether this was linked to a lack of time or resource, or simply a lack of interest on the part of the organisations concerned. Again, this confirms the importance of receptiveness to network activity, combined with effective engagement, for positive outcomes to occur at the organisational level. 4.5 Extent and nature of policy transfer and learning Evidence on the degree of policy learning and transfer at organisational, Member State and EU levels was mixed and, in general, was perhaps more limited than stakeholders involved with the Learning Networks had anticipated. In particular, the degree to which the concept of direct policy transfer stemming from network activity is realistic was questioned by a number of Learning Network and Commission stakeholders. While some felt that new policies had been at least influenced as a consequence of Learning Network activity, others clearly pointed to what they perceived as being the limited impact of Learning Network activities at the policy level. The pattern of limited concrete impacts in respect of policy transfer emerged in respect of the Member State case studies in particular. Although individual learning was felt to be likely to influence policy development over the longer term, those consulted acknowledged that direct policy transfer was unlikely. The most common reasons offered for this were: The long-term and complex nature of policy development, involving multiple actors at different governance levels being influenced by a range of factors. Such a context was viewed as not being amenable to simple transfers of policies. Difficulties in achieving consistent engagement amongst policy-makers at Member State level (this was cited in the UK context in particular as a result of the way the Civil Service is organised and the rotation of personnel within it). Lack of direct transferability stemming from the different political, social and economic contexts of Member States. Similar evidence and perspectives emerged from the Learning Network case studies and key stakeholder interviews. For example, just under half of the network leads interviewed stated that they were not aware of any new policies or significant policy change as a consequence of network activities. It was also pointed out that it was difficult to attribute policy change to network activities alone, and that many networks activities aimed at bringing attention to certain issues and introducing new mind-sets, rather than occasioning concrete policy or organisational change. This was referenced in particular in relation to networks whose focus was on cross-cutting or horizontal governance themes, along with those concentrating on informing practice at lower governance levels. Within the research carried out for the Learning Network case studies, a significant number of interviewees also stated that there were instances where activities had shown that certain policy practices could not be transferred to other countries. Barriers to transferability were seen to be posed by Member States differing administrative and legal structures, budgetary constraints, a lack of political will and also the geographical size of the country. However, it is also worth noting the view of one 35

48 interviewee who argued that Member States tended to hide behind these differences, rather than constituting real constraints. From this perspective, the issue of political will was seen as the real constraint on transferability in policy or operational terms. Despite these considerations, effects were reported in terms of influencing policy or, as noted above, operational change at the organisational level in respect of several of the networks. Examples included the perceived impact of the SaviAV model on Spanish social policy through introducing new training networks for social workers and making changes to reception centres. Evidence gathered for the RBM network case study also indicated that organisational and / or policy change occurred in several contexts including influencing the development and use of RBM in the Wallonia ESF Agency, while in Sweden the ESF Council is now preparing for the introduction of RBM in the ESF Managing Authority in At Member State level, a number of stakeholders also felt that participation in the Learning Networks had led to adaptation or adoption of new policies and initiatives. However, interviewees did not always give concrete examples about where exactly Member States policies had been influenced. Examples that were offered included the effect of the work of the BFSE network in informing the Polish National Strategy for the Development of the Social Economy, which included examples of good practice taken from the network. Similarly, it was reported that the Czech Republic intended to introduce new financial instruments as a result of their participation in the BFSE network. Again, however, several stakeholders felt that it was unrealistic to change policies at the Member State level through Learning Network activities, pointing out that it would require much greater involvement of influential policy-makers, time, financial resources and, crucially, windows of opportunity in terms of timing or the receptiveness of policy makers and institutions to change. As discussed, it was also noted in this context that the objectives of the networks were generally more constrained than seeking to influence policy; for example, being focused more on horizontal issues such as supporting operational or administrative reform in the planning and delivery of the ESF. In general, it appears that policy influence as a result of network activity was perhaps more evident in the sphere of the ESF itself, whether in terms of influence over particular OPs (particularly in the context of planning for the new programming period) or in respect of ESF guidance and regulations at the EU level. Examples of the former impacts included the view that the treatment and integration of migrant issues as a cross-cutting theme in the new Berlin OP (and similarly the new Swedish OP) was influenced by the work of the IMPART network, in addition to the role of the Gender CoP network in influencing the drafting of the gender dimensions of the new OP in Sweden. Influence at EU level was cited where there was close contact between individuals involved in some of the networks and those overseeing or drafting the new ESF regulation. For example, those involved with the IMPART network felt that they had some influence over the way migrant integration plays an important part in the new programming period, particularly in terms of its territorial dimension and the plan to target efforts on a spatial basis. It is also worth noting that, while concrete examples of policy transfer appear relatively limited, it is evident that more subtle and diffuse effects concerning the transfer of policy ideas and possibilities for operational adaptation and development have certainly occurred within the context of the networks. Evidence from Learning Network monitoring and evaluation reports, allied to that from the stakeholder interviews and case studies, shows that there were certainly impacts achieved in this sphere at the level of participating partners. Such impacts encompass the transfer of learning and the gaining of new insights in a number of thematic contexts. While these effects sometimes appear to relate more to learning achieved between individuals with a role in policy or ESF delivery, rather than occasioning broader 36

49 structural change or impacts, they can nonetheless be considered to have the potential to influence such change over a longer timeframe. It is also important to note that, in several cases, intended outcomes in terms of influencing policy and operational delivery in the ESF context relate to the forthcoming programming period. As such, as a number of the monitoring and evaluation documents note, and as echoed by several stakeholders, assessing the achievement of these aims at present is premature. In some cases, the widely acknowledged quality and usefulness of toolkits and other products suggests that some of the results of several networks will have value in the future. Equally, in some instances, those involved with the networks gave examples of Managing Authorities taking note of findings in preparation for the next ESF period, again suggesting a potential for longer-term impacts and sustaining the results of Network activity. Lastly, Member States were seen to additionally benefit from the fact that the networks were able to increase the profile of some topics on the national agenda, in addition to the transfer of learning into national institutions. However, as noted above in respect of policy learning and transfer, benefits for Member States were in general harder to concretely identify than those at the individual or organisational level. In part this was seen as a timing issue in that such benefits would take time to filter through as well as being relatively dispersed and therefore difficult to fully pin down. 4.6 Additional outcomes including broader co-operation between partners and Calls for Proposals In the majority of cases, the evidence suggests that the Learning Networks have contributed to improving ESF transnational cooperation between Member States, at the very least at an individual or organisational level. Such effects were commonly discussed by stakeholders in the context of the role of the Learning Networks in either newly connecting people or allowing for existing relationships to intensify. This is illustrated by the fact that most Learning Networks resulted in the creation of formal or informal networks after their lifetime, albeit to different degrees and with varying intensity. The benefits of transnational linkages, in the sense of contributing to a feeling of shared endeavour and European identity, were also referenced by some interviewees. Despite these factors, other potential outcomes from the Learning Networks in terms of leading to specific Calls for Proposals emerging from their activities did not, in general, appear to occur. Some examples of activities influencing the design or development of Calls for Proposals in the context of Member State and regional ESF OPs were offered, though the linkages between these and network activity did not generally appear to be clear. Overall, therefore, there was little evidence of network activity influencing Calls for Proposals, certainly not in the sense of directly leading to such calls. 4.7 Effectiveness in meeting Learning Network aims Meeting the aims of individual Learning Networks Evidence indicates that the Learning Networks varied somewhat in the degree to which they met their stated objectives. In the majority of cases however, as illustrated by quarterly and final monitoring reports, most or all objectives were met to a high degree. Where objectives related to process aspects, such as study visits, peer reviews and the holding of seminars or conferences, these were generally achieved. On the basis of the evidence collated, there appear to have been only a few exceptions where issues in the management or organisational aspects of delivery negatively affected the operation of networks and / or achievement of core objectives. 37

50 Although most objectives were met, there appears to be a trend in the types of objectives that proved more difficult to achieve. Typically these objectives included the production of recommendations and, as discussed above, wider dissemination to policy-makers along with objectives to actually influence policy. In some cases, producing effective recommendations was reported as being problematic due to issues with the process of recommendation development itself for example, recommendation working groups not functioning effectively or a lack of communication between partners. In these instances, issues with producing recommendations were also offered as an explanation for the level of dissemination and policy influence being more limited than had been anticipated. In several cases, Learning Networks also failed to produce particular products that were defined as part of their initial aims and objectives. These included, for example, handbooks summarising good practice, databases and other online or printed tools. A number of factors were advanced to explain this. Typically, these included a lack of time and resource, discovering that the planned products were impractical or too difficult to produce, partners deciding that the products concerned would no longer be of relevance, and / or partners being unable to develop the products due to unforeseen circumstances. In most instances, however, the lack of production of particular products was seen as not having compromised the overall outcomes or success of the networks concerned Meeting the aims of the Learning Networks as a whole The network lead interviews and Leaning Network case studies also offered the opportunity to consider the degree to which the networks had contributed to the following overarching aims of the Learning Networks as a whole, namely: a. Strengthening the capacity to innovate b. Modernising and adapting institutions to new social and economic challenges c. Identifying and assessing issues and solutions for reforms in policy and delivery d. Improving the quality of governance of public policies, programmes and actions. In most cases, network leads tended to discuss outcomes in terms of identifying and assessing issues and solutions for reforms in policy and delivery more than the other aims stated. This was seen as reflecting the focus of most networks on defining issues or areas for investigation and developing new approaches to those areas based on sharing practice, research and analysis of the issues concerned. Where this was a particular focus of the networks, activities such as peer reviews were often cited as a mechanism through which such issues were addressed and solutions developed. The IMPART network offered a good illustration of this with its focus on utilising a peer review and benchmarking methodology, oriented around particular issues which were identified through prior research and partner discussion. Other examples of providing solutions for reforms in policy and delivery were often also seen as being instances of improving the quality of governance or public policies, programmes and actions, indicating the close relationship and overlap between these high level aims. Such examples discussed in this context included: The contribution of the Better Future for the Social Economy Network to reforming policy and delivery through informing the development of the National Social Economy Strategy in Poland and influencing policymakers and policies linked to this strategy. The IMPART network influencing the treatment of the integration of migrants as a cross-cutting theme in the new regional OP in Berlin along with, at a higher level, those involved with the network citing some influence on the development of the new ESF regulation in terms of its approach to migrants. 38

51 Similarly, the work of the SaviAV network on encouraging the development of new policies to support the integration of asylum seekers and victims of trafficking in several contexts. In terms of modernising and adapting institutions to new social and economic challenges, there was some variation in the degree to which networks were felt by stakeholders to have contributed to this aim. For example, some network leads felt that such an outcome was overly ambitious given the nature and scope of their network s activities. In one instance, for example, it was argued that such modernisation and institutional change would require the involvement and input of higher level individuals than those typically involved in the networks. However, those involved with some networks did feel that their activity had at least made a contribution in this area. For example, the work of the Gender CoP network in influencing the support structures being established for the new OP in Sweden, and their focus on / support for gender being integrated across the programme as a cross-cutting theme, was cited in this context. Where strengthening the capacity to innovate was discussed, examples tended to relate to the development of products that had the potential to contribute to wider innovation or innovative practice if taken on by Member States or Managing Authorities. Examples cited included the common standard for gender mainstreaming produced by the Gender CoP network, the toolkit for assessing practice in supporting migrants produced by the IMPART network, and the common reference framework produced by the ENYE network. In general however, as was the case to an extent with the other high level aims, network leads and other stakeholders often also noted that it was relatively early to judge whether innovation would result given that the influence hoped for from these products would take time to filter through. Accepting this, it was also pointed out that, where Member States or OPs were starting from a relatively low base in respect of particular issues, for example gender mainstreaming or supporting economic migrants, even if only some of the work of the networks was to be taken on, this in itself would represent innovation in many contexts. 4.8 Sustainability of Learning Network activities and outcomes Sustainability in respect of the Learning Networks can be considered at several levels: firstly, in relation to the sustaining of contacts or relationships, whether at individual or organisational levels; secondly, in terms of continuation of network activity or successor networks; and thirdly, in terms of the sustainability of effects and outcomes generated through Learning Network activity. While at the last of these levels it is perhaps too early to make a judgement, most stakeholders connected to the networks felt that such effects would continue but might prove diffuse and difficult to identify as time passed. In respect of the other levels of sustainability noted, evidence was mixed but broadly positive in relation to the continuation of relationships and activity begun under the networks. More specifically, in respect of sustaining of contacts or relationships, around half of the Learning Network members interviewed through the key stakeholder interviews and Learning Network case studies stated that they continued bilateral and multilateral exchanges informally after the lifetime of their networks. Such exchanges were particularly apparent at the operational level between individuals with a related thematic focus or similar roles within Member States. In some instances, interviewees felt that the development and on-going nature of these relationships was one of the most positive aspects to come out of their involvement with the networks. Similarly, examples of on-going contacts or relationships at the organisational level were cited, though less commonly than in respect of links between individuals. 39

52 In terms of continuation of network activity or successor networks, in a number of cases the work of networks supported through the 2008 and 2009 Learning Network Calls for Proposals has continued beyond their initial lifespan. In these instances, the success of network activity was cited as contributing to the establishment of follow-on networks, some of which were supported under the 2012 Learning Network Call for Proposals (including Gender CoP, ACB, RBM under RBM+, and the EMPOWERMENT network in the shape of the new Learning Network on Active Inclusion). In the case of networks such as Gender CoP, this was seen as important in facilitating the continuation of work commenced under previous networks and in enabling the networks concerned to move closer to achieving their full range of objectives and ambitions. The continuation of work begun under the Learning Networks was also referenced by some interviewees in terms of activity leading to applications for other programmes or funding streams on the part of key partners. For example, the SaviAV network was seen as important by those involved in helping to facilitate and lead to a joint bid by Germany and Greece for funding under the European Programme for Integration and Migration (EPIM) initiative. However, there are also a number of networks which have not led to any follow-up cooperation and, more broadly, the issue of sustainability was raised as a concern by both stakeholders involved in the networks and those from the Commission. Some stakeholders pointed to specific barriers to greater continued cooperation, amongst them the fact that network participants had changed frequently and that links were only created at a high level while not spreading more widely. The issue of resources, both financial and in terms of time, was also commonly noted, linking to difficulties faced in transferring learning within organisations and Member States on the part of network members. 4.9 Efficiency in generating outcomes and impacts On the basis of the available evidence it appears that, in the majority of cases, the Learning Networks were broadly efficient when judged from the perspective of what they achieved relative to the resources available to them. In terms of gathering evidence by which to assess efficiency, network leads and other interviewees consulted through the Learning Network case studies were asked to consider whether the results of the networks justified the resources allocated. While some stakeholders found this difficult to assess, the majority felt that the outputs and outcomes of the network with which they were involved justified the inputs (both in financial terms and in respect of the time given by network members). Indeed, in some cases, interviewees argued strongly that the network concerned offered excellent value when the scope of activity and results were considered. Only in a minority of cases did stakeholders feel that the resources allocated did not justify the results of networks. In these instances, the fact that the networks in question did not fully achieve their intended outcomes influenced this view, while in one case it was considered that the goals the network had set were too ambitious and would have required greater resources to achieve. Equally, inefficiencies or lack of effectiveness in project management were cited in one or two cases as an explanatory factor for why outcomes were not felt to have reflected resource inputs. Where outcomes were felt to justify resources, several common themes were cited. The fact that network members gave significant time and in-kind inputs was referenced in several cases. One network lead, for example, pointed out that the cost of creating the products produced would have been significantly greater if external consultants had been hired to produce them. In several cases, network leads also noted that the network had not used its available budget, while it was noted by others that the (in their view) limited resources available meant that the network had to be managed carefully and efficiently. The 40

53 view that outcomes were in effect multiplied, through network members cascading learning and products within their own Member State, was also commonly cited as important in the achievement of a good level of outcomes when balanced against inputs Summary of key findings The dissemination approaches adopted by the Learning Networks proved to be mixed in terms of their effectiveness. In some cases, dissemination products were effective in transmitting the results of network activity, but this was not universally the case. Equally, while some dissemination strategies offered a useful structure for sharing results, in respect of others evidence suggests that such strategies were either under-developed or inappropriately targeted. Where dissemination worked well, the use of partners as ambassadors to share learning within Member States was widely seen as a significant success factor. Similarly, building links with key players at EU and Member State level also appeared to be important to ensure effective dissemination. In terms of specific mechanisms, passive approaches such as printed materials or websites appeared less effective than more active forms of dissemination through events, meetings or on-going liaison with relevant stakeholders. The clearest and most consistently positive outcomes from the Learning Networks emerged at the level of individual learning amongst those participating. Whilst this might be expected, the evidence gathered illustrated the degree to which the vast majority of network members felt that notable positive effects had resulted from their engagement. These included new contacts, enhanced individual relationships, and a deeper level of knowledge about particular thematic areas. In the minority of cases where individuals felt that their involvement with the Learning Networks had been less positive, this generally related to the focus and activities of the network concerned, in the sense of these not meeting expectations. In many cases organisations also clearly gained from the knowledge exchange involved in the networks, along with the opportunity to develop partnership and cooperation with related actors across Member States. On balance, positive examples of organisational learning and impacts outweighed cases where networks were felt to have had less of an influence at the organisational level. Evidence of the degree of policy learning and transfer at the organisational, Member State and EU levels was, however, mixed and perhaps more limited than stakeholders involved with the Learning Networks had anticipated or hoped for. In particular, the concept of direct policy transfer stemming from network activity was questioned by a number of Learning Network and Commission stakeholders in terms of the achievability of this objective. Common reasons offered for this were the long term and complex nature of policy development, difficulties in achieving consistent engagement amongst policy-makers, and the lack of direct transferability of policies stemming from the different political, social and economic contexts within Member States. Partly as a result of these factors, where policy effects could be identified these were more in the form of policy influencing as opposed to direct policy transfer. Within this, policy influence as a result of network activity was more evident in the sphere of the ESF itself, whether in terms of influence over particular OPs (particularly in the context of planning for the new programming period) or in respect of ESF guidance and regulations at the EU level. However, while concrete examples of policy transfer appear relatively limited, it is evident that more subtle effects concerning the transfer of policy ideas and options for operational adaptation occurred within the context of the networks. Moreover, such effects have the potential to influence policy and operational change over a longer timeframe. In terms of broader outcomes, in the majority of cases evidence suggests that the Learning Networks contributed to improving ESF transnational cooperation between Member States. However, there was little evidence of network activity influencing Calls for Proposals, certainly not in the sense of directly leading to such calls. 41

54 As well as the broader outcomes noted, the evidence assessed indicates that the Learning Networks varied in the degree to which they met their specific stated objectives, but that in most instances the majority of or all objectives were met to a high degree. There was also a trend in the types of objectives that proved more difficult to achieve; typically, these related to the production of recommendations and influencing policy. Sustainability in respect of the Learning Networks can be considered at several levels: firstly, in relation to the sustaining of contacts or relationships; secondly, in terms of continuation of network activity or successor networks; and thirdly, in terms of the sustainability of effects and outcomes generated through Learning Network activity. While for the last of these levels it is perhaps too early to make a judgement, it seems likely that such effects will continue but might prove diffuse and difficult to identify as time passes. In respect of the other levels of sustainability noted, evidence was mixed but broadly positive in relation to the continuation of relationships and activity begun under the networks. Finally, in respect of the efficiency of the Learning Networks, it appears that the majority of Learning Networks were broadly efficient when judged from the perspective of what they achieved relative to the resources available to them. Key factors underpinning this judgement relate to the fact that the networks benefited from considerable in-kind resources contributed by partners, had to implement their activities carefully given overall resource constraints, and in a number of cases produced outputs that would have cost significantly more had they been produced by consultants or other external contractors. 42

55 5.0 Support and monitoring 5.1 Introduction This chapter examines the support available to the Learning Networks from the European Commission, in addition to the role played by AEIDL as the Commission s appointed contractor in monitoring network activities, implementation and outcomes. Stakeholder views concerning the role of the Commission in supporting the Learning networks are examined first. The chapter then examines the role played by AEIDL, prior to concluding with a summary of key findings. 5.2 The role of the Commission in supporting the Learning Networks Perspectives on the support offered by the Commission to the Learning Networks by those leading or coordinating the networks were divided fairly evenly between positive views and a feeling that the support could have been better and / or could be improved in future. For some networks the support offered was described in highly positive terms, with one interviewee describing it as a very positive experience. The benefits of receiving on-going and consistent support, along with the view that officers from the Commission were receptive to discussing issues as required, were noted in particular. Those with less positive views tended to cite issues such as too great a turnover of Commission staff, leading to a lack of consistency in support for the network, and / or a concern that the relevant unit in the Commission did not have the required capacity to offer a full level of support. This issue of limited capacity was acknowledged by some Commission stakeholders who noted that changes in organisational structure and focus had made it difficult to maintain the level of support available. In particular, restructuring and the loss of a specific unit focused on transnationality were highlighted in this context. In general, there was little use of, or awareness reported in relation to, the Commission Support Groups established for the networks. While this was acknowledged by Commission stakeholders, it was also noted that the onus was on networks to be proactive in their engagement with Commission officials. In cases where this engagement was evident, the view of some of those involved in the networks was that the support groups could be effective. For example, such a view was offered by both Commission stakeholders and network leads involved in the Gender CoP network. In particular, the pro-active approach taken to this engagement by the GenderCoP network was referenced as representing good practice in this area. It was also noted that, in some cases, network leads had established relationships with individual Commission officers, hence effectively bypassing the support teams. In part the limited engagement with the support groups was also viewed by Commission stakeholders as reflecting their more informal status, with this being recognised in the decision to more formally align such groups with networks in future. In terms of additional support that might be provided by the Commission or its contractors to facilitate the work of the networks and enhance the outcomes achieved, network partners made a number of suggestions. Greater clarity in terms of the role of the Commission and contractors in supporting the networks, and what the networks might expect from this support, was raised by several network leads. More assistance with identifying and working with external experts, facilitators and/or external evaluators was also raised as a potential development in several instances, as was enhancing the support and guidance provided on good practice in building effective Learning Networks. Suggestions here included guidance on the design and development of effective activities and potentially the provision of some 43

56 standardised tools for running and co-ordinating a network. Finally, a small number of network leads also felt that the administrative burden in terms of reporting requirements could be reduced. Other issues raised by those involved in the Learning Networks tended to be specific to the nature of an individual network itself or the experience that network leads had in running it. These included: Improving the consistency of the support offered in terms of the personnel providing it and the level of engagement involved. Creating more two-way dialogue in situations where the Commission is developing policy or producing papers or reports on topics linked to the network (this was seen as important to ensure that the network is kept up to date on developments that may impact their work and, equally, so that relevant findings or insights can be provided to the Commission). Ensuring a greater focus on supporting the mainstreaming of outcomes on the part of the Commission (for example, through facilitating links with relevant units or individuals). 5.3 Support offered by the Learning Network contractor and their role Views on the role of AEIDL in monitoring the Learning Networks also varied. Some network leads in particular gave a positive perspective, noting that the individual attached to their network had been helpful and played a useful facilitation role at network meetings and events. It was also noted that the role of the individuals concerned went beyond a mere monitoring role which was appreciated by those commenting on this. In some cases, network leads also clearly appreciated the wider contribution to their network made by AEIDL staff, for example at events and meetings, typically describing this in terms such as supportive and helpful. However, in other cases the perception of those involved in the networks was less positive. For example, some interviewees felt that the individuals attached to their networks had limited thematic understanding of what they sought to achieve. This was seen as compromising their monitoring role and meant that they were able to offer little of benefit beyond this core remit. Other interviewees similarly noted that there had been some variation in the quality of individuals they had been in contact with. In addition, in a couple of cases, the level of organisation displayed by AEIDL was felt to be lacking, while other issues raised included not receiving monitoring reports for comment in a timely manner and poor facilitation skills. More generally, even amongst those with positive views on the individual(s) from AEIDL they had worked with, the role of AEIDL overall was felt to be unclear or not adequately explained to the networks. In particular, the balance between AEIDL staff having a supporting and/or monitoring role was highlighted as being unclear and / or variable. This issue was acknowledged by Commission stakeholders as partly reflecting the fact that AEIDL s remit was one of monitoring, but that some individuals attached to the networks had usefully played a more supportive role. Along with some network leads, some Commission stakeholders felt that any future contract should explicitly include a support element in addition to monitoring as a result of these issues. Some network leads also felt that any similar contractor role should in future focus more on supporting the development of linkages between networks or similar bodies, for example through more actively facilitating the transfer of learning between them. Stakeholders similarly offered a range of views on the usefulness of the network co-ordination meetings which were facilitated by AEIDL and brought together members of the different Learning 44

57 Networks. In general, the concept of the co-ordination meetings was well received and the meetings themselves were often seen as a useful opportunity for liaison. Some network leads, for example, cited that the meetings had been valuable in terms of sharing experiences and ideas for enhancing the effectiveness of activities, in addition to offering practical support around administrative and financial procedures. Equally, examples were given of how attending the meetings had proved to be a catalyst for further collaboration between particular networks. However, a number of stakeholders also offered less positive views on the meetings and, in particular, their organisation and content. In a couple of cases, this related to a view that the agendas for at least some of the meetings had not been well thought through. In addition, it was noted that there was a general lack of organisation apparent in relation to some of the meetings. Equally, some network leads felt that the content and supporting materials for the meetings were of limited depth. Ideas for improvement tended to mirror these issues. For example, one network representative felt that the content of such meetings might be improved by ensuring more (policy-related) input from Commission units other than those directly related to the Learning Networks. Another felt that there should be a greater focus on enhancing the quality of the preparation, organisation and supporting materials produced. 5.4 Summary of key findings Perspectives on the support offered by the Commission to the Learning Networks by those leading or co-ordinating the networks were divided fairly evenly between positive views and a feeling that the support could have been better and / or could be improved in future. Despite this some very positive comments on the support and its contribution to network functioning and outputs were offered. In cases where views were less positive, the key issues raised concerned high levels of turnover in Commission staff and a concern that the capacity available to support the networks was insufficient. Potential improvements suggested by network leads included: greater assistance with identifying and working with external experts, facilitators and/or external evaluators; enhancing the support and guidance provided on running effective networks; providing standardised tools for running and co-ordinating a network; assistance in developing links with relevant Commission units; and reducing reporting and administrative requirements. Views on the role of AEIDL in monitoring the Learning Networks also varied. While a number of network leads and participants appreciated the supportive role that individuals from AEIDL had offered, noting that this went beyond their monitoring remit, in other cases views were less positive. The main issues raised concerned a perception that in some cases AEIDL staff did not possess the thematic knowledge that network participants felt was required for their role, that the quality of the monitoring function was variable and that some meetings and events were not organised effectively. Views on the network co-ordination meetings facilitated by AEIDL were similarly variable; while most stakeholders felt that the concept of the meetings was good, it was sometimes noted that the content and organisation of the meetings was less effective. Finally, stakeholders commonly felt that the role of AEIDL might have been clearer, particularly in terms of their remit and the balance between monitoring and support functions (whilst acknowledging that the latter element was not a formal part of their remit). 45

58 6.0 Implications for the future of transnational co-operation through the ESF 6.1 Introduction This chapter examines some of the key implications arising from the study for the future of transnational co-operation through the ESF. In doing so, it draws on the key stakeholder consultations and case studies undertaken, along with the virtual Policy Delphi held towards the end of the research process. To inform the discussion of future implications that follows, the chapter first analyses stakeholder perspectives on the key strengths, weaknesses and success factors relating to the networks, along with views on the key learning points emerging. Through reference to, and based on, the virtual Policy Delphi, a series of implications for future policy and activity around transnational co-operation are then discussed. The chapter concludes by summarising key findings from the preceding analysis. 6.2 Stakeholder views on the key strengths, weaknesses and success factors relating to the Learning Networks Stakeholders consulted through the key stakeholder interviews and Learning Network case studies tended to cite a number of common issues as being the most positive aspects of the networks. The transfer of knowledge and sharing of ideas or good practice were most often referenced, with this seen as reflecting the core aims and value of such transnational co-operation. Equally, in the case of several of the networks, the outputs produced were viewed in this light, while the role of the networks in raising awareness on particular issues such as migrant integration or gender mainstreaming was another common theme. Some network leads also saw the most positive effects of their network as influencing debate on, and preparation for, the forthcoming ESF programming period. In contrast to the tendency of these stakeholders to discuss outcomes as being the most positive aspects of the Learning Network experience, the weakest aspect of the networks were often cited as relating to processes and operational considerations. Many of these have been discussed in detail in chapter three but, in summary, these involved: overly complex organisational or governance processes; a more general lack of organisation; lack of project management expertise amongst those leading or co-ordinating networks; and insufficiently clear definition of partner roles and remits. In terms of weaknesses relating to outcomes, inadequate or misguided dissemination approaches allied to a lack of impact on the policy level were most often cited. The main success factors identified by stakeholders, including those offered through the open questions in the online survey, closely reflected the above strengths and weaknesses. These factors can be summarised as follows: Development of a clear and consistent thematic focus with clear and realistic objectives being developed to reflect these themes A clear implementation plan which sets out the relative responsibilities of individual partners, steering groups and working groups Effective network composition in terms of ensuring that appropriate and committed partners (and an appropriate range of partners) are engaged in network activity The development of clear governance processes and lines of accountability 46

59 Effective organisation and leadership on the part of those co-ordinating network activity A well developed dissemination strategy, supported by high quality tools and outputs which are tailored to specific audiences Success factors concerning Member State engagement Undertaking the Member State case studies illustrated that the way in which Member States choose to engage with and participate in Learning Networks can also be a factor in the success or otherwise of such transnational co-operation. For example, the case studies showed that the nature and degree of coordination on the part of ESF MAs as regards engagement at the Member State level varied significantly; this in turn had an impact on the perceived benefits or otherwise of involvement with the Learning Networks. Likewise, the case studies can also be used to identify success factors, and equally approaches to avoid, in order to generate positive outcomes from transnational co-operation. In the case of Flanders and the UK, involvement with the Learning Networks was co-ordinated through the ESF MA (Flanders) and a dedicated Innovation, Transnationality and Mainstreaming (ITM) Unit functioning within an Intermediate Body with delegated authority from the MA (UK). Conversely, in the case of Spain and the Czech Republic, it was noted that there was less centralised co-ordination on the part of the ESF MA. In these contexts, case study interviewees tended to cite that engagement with the Learning Networks was largely driven by the interest and commitment of individuals, rather than being planned or strategically co-ordinated by the MA. In respect of the Czech Republic in particular, interviewees linked the lack of central co-ordination and oversight as being a factor in engagement with the Learning Networks having less of a positive impact than it might have done when considering outcomes at Member State level. From this perspective, a more strategic and co-ordinated approach was thus recommended for future transnational co-operation. Conversely, the presence in the UK of a dedicated co-ordination function in the shape of the ITM unit was seen as a positive factor in ensuring that, for example, good practice evident in the Member State context could be collected and fed into Learning Networks and other transnational activities at the European level. The role of the unit in being able to feed learning from transnational co-operation back into the Member State context in a co-ordinated way was similarly remarked upon positively. In the Flemish context, the development of a horizontal policy committee, containing representatives of various departments of the Flemish Ministry of Employment and social partners, to validate the choices, priorities and representation of the ESF MA at the start of each Learning Network was seen as a beneficial element of the more strategic approach to engagement that was reported to have taken place. From the perspective of informing how Member States might best engage with Learning Networks and similar transnational co-operation to gain the most benefit, it is also worth noting that turnover of staff at the domestic level and inconsistent representation emerged as a common theme across the Member State case studies. In the Czech Republic context for example, it was noted that such turnover in the ESF MA and Ministry of Labour meant that effective engagement was more difficult. Similarly, in the UK case, it was acknowledged that inconsistency of representation from policy-makers in Government departments sometimes made it difficult to ensure the transfer of potentially useful learning, whether from or to the European level. In general therefore the potential to enhance effective engagement on the part of Member States through a strategic and co-ordinated approach, combined with consistency of representation, emerged as key learning points. 47

60 6.3 Key learning points from Learning Network activity When asked about key improvements that could be made to Learning Networks or similar approaches in future, stakeholder responses drawn from the Learning Network case studies and key stakeholder interviews varied between those focusing on internal improvements, relating to the network or networks with which they were involved, and broader external improvements relating to, for example, the networks as a whole or the role of the Commission in supporting them. Again, these suggested improvements closely reflect the success factors outlined in section 6.2 above, though at the level of more specific issues and ideas for improvement as opposed to broader themes. Whether the key improvements were internal or external, suggestions tended to reflect the experience of those consulted and the particular issues that had arisen with the networks they were involved in. Suggested internal improvements included: Improving websites and dissemination platforms Limiting the number and / or scope of activities to ensure that they are realistically achievable Ensuring that there is real interest and a willingness to contribute amongst Member States that sign up for involvement in networks Ensuring more flexibility in responding to changes in context Use of a facilitator to enhance the efficiency of meetings and network events More of a focus on risk assessment and contingency planning so that, for example, individuals leaving the network or not performing their roles can be more effectively dealt with Support to undertake a formative evaluation of the network. Suggested external improvements included: Greater engagement on the part of the Commission, for example through linking in policy / thematic experts in the Commission with relevant networks and / or enhancing links with those with a remit around mainstreaming network results Clearer articulation by the Commission of the expectations and role of the networks, along with more clearly tying this into ESF goals and those of the EU2020 strategy More explicitly building on what has gone before in terms of linking the work of networks and transnational co-operation across programming periods Greater highlighting of transnational co-operation as important along with its benefits on the part of the Commission, potentially even to the extent of forcing Managing Authorities to engage in activity With the move towards integration of the Structural Funds ensuring that membership can be expanded to include more than just ESF actors, in particular including those involved in ERDF Seeking to ensure that Member States only participate in those networks where there is a realistic chance of a supporting infrastructure focused on the theme concerned within that Member State; that is, that the learning gained has a route to be cascaded within Member States and that the polity involved is open to this. 6.4 Implications for future policy and activity around transnational co-operation (virtual Policy Delphi) To inform discussion of the implications for future policy and activity around transnational co-operation in a targeted manner, a virtual Policy Delphi panel was organised involving a small group of twelve high level stakeholders from the Commission, ESF MAs, selected Member States and AEIDL. This Policy 48

61 Delphi panel was provided with 10 key findings and matching issues for consideration arising from the assessment of the Learning Networks (see Policy Delphi paper at Annex 2 for full details). Responses were then analysed and further comments invited on the basis of this analysis. The final results of the Policy Delphi process are summarised below, organised around ten issue areas reflecting the initial issues for consideration provided to panel members. It should be noted that the Policy Delphi process was undertaken at a point when the ESF Regulation for the programming period had been published. For this period, Article 10 of the ESF Regulation allows Member States to implement transnational cooperation in three different ways: On the basis of common themes proposed by the Commission and endorsed by the ESF Committee, whereby MS benefit from an EU-level platform established and operated by the Commission (Common Framework) Without central coordination (flexible approach) Through a combined approach of the above. The final results of the Policy Delphi process are summarised below, organised around ten issue areas reflecting the initial issues for consideration provided to panel members. Enhancing the spread of participation in Learning Network type activity and transnational co-operation Some respondents made the point that the ESF regulation for the forthcoming period makes transnational co-operation (TNC) compulsory, albeit with an exception for Member States (MS) with only 1 OP. In the view of some, if implemented correctly and effectively, the Common Framework and Technical Assistance (TA) platform should enhance the aim of encouraging more even participation within transnational activity across MS. From this perspective TA is seen as important in reducing the administrative burden previously cited as a barrier to MS participation. However, the potential for unbalanced representation between those MS involved from a desire to learn, and those involved from a motivation to share learning, was raised as a potential concern and something that may need to be actively managed. The need to support the objectives of the Common Framework by conscious promotion of the benefits of TNC on the part of the Commission through a number of routes, including the ESF Committee, was also noted. A further suggestion involved organising some activity on a more concentrated geographical basis, through facilitating groups of Member States such as those around the Baltic Sea to come together. Promoting collaboration and cross-working between different transnational activities and initiatives The potential for the Common Framework and TA function were again referenced as important in increasing the extent of cross-working and collaboration between different networks, activities and initiatives. The ability to have an overview of different activities underway was seen as important in this, offering the opportunity to identify common themes and potential synergies. Networking events and the promotion of joint activities through the TA function were noted as more specific mechanisms for enhancing collaborative working within and across wider TNC, provided that they were well organised and that other mechanisms and routes to encourage collaboration were also used. From this perspective, the need for additional support and impetus to encourage collaboration and synergies was noted, whether this came from the Commission or the anticipated TA function (though capacity constraints on this Commission in this context were also raised as an issue). As part of this, raising the status and visibility of 49

62 TNC was also noted as an additional requirement with the Commission being seen as having a promotional role in this sense. Ensuring transnational activity reflects the aims of the ESF while remaining flexible A more active guiding role on the part of the Commission in terms of the focus of TNC activities was the most widely referenced view of panel members in this area. However, it was also noted that such activity need not be wholly top-down in that participative methods could be used to identify key issues within the overall context of the ESF as it develops over the period. In some instances the potential for the Commission to facilitate links between TNC activity and policy units / leads was also cited, though the responsibility of those involved in TNC to push for this was also acknowledged by some respondents. The potential for a push-pull approach in this area was thus highlighted. In one or two cases the need to be clearer on the remit and mandate of those engaging in TNC was also cited; that is, the Commission should be clearer on the goals which TNC activity should aim to achieve, in terms of both policy and practice. In terms of ensuring flexibility and responsiveness to external events, the likely limitation of the Common Framework in having only two calls for proposals was noted as a concern. However, a number of ideas for promoting such flexibility were also offered. These included the potential for use of the sort of participative mechanisms noted above to identify emerging issues from within and between MS, along with greater use and strengthening of intelligence, research and analysis on the part of those defining such issues. Supporting transnational activity to ensure effective organisation and implementation A stronger TA function offering more organisational support was widely seen as a pre-requisite for ensuring that TNC activity would be better organised and more effectively implemented in future. This was seen as particularly important given the limited resources that MS are able to offer in terms of supporting, and certainly co-ordinating, TNC activity. It was also seen as having the potential to lead to more consistency in terms of outcomes and results from activity through addressing organisational and capacity limitations on the part of MAs involved in co-ordinating activities. Consistent monitoring of activity on the part of the TA function was also cited as important so as to identify which activities face organisational or implementation challenges at an early stage, thus allowing more support to be provided. Enhancing policy learning and transfer in the context of transnational and Learning Network type activity In terms of policy learning and transfer at the European level, a greater role and focus on the part of Commission civil servants in policy units was seen as important in incorporating more learning from Learning Network type activity in the future. Likewise, for some respondents, those involved in TNC should also have a responsibility to seek out contacts in the Commission to engage in their work and discuss its results. Again, therefore, the potential for a push-pull approach was highlighted. The need for more clarity and realism in terms of the precise nature and degree of intended policy influence in was also cited. In particular, the need to be realistic over the amount of influence that can be achieved and to be precise over what is intended were seen as important factors in enhancing the success of policy learning and transfer activities. This was seen as applicable to all likely levels of policy influence, whether within MS, between ESF MAs or at the European level. 50

63 The role of the Commission in supporting Learning Network type activity and transnational co-operation A combination of more effective support from Commission officials with an enhanced and better defined TA role was widely cited as an important future development in terms of the support function around TNC activity. Specific support on the part of the TA function and the Commission to help disseminate and implement learning from TNC activities was seen as important in this area, as was an openness and willingness on the part of the Commission to take note of the results of such activity. In particular, clarity over the role of experts under the TA function, enabling them to play a full and supportive role in respect of TNC activity, was noted as being required. Promoting learning transfer at the ESF project level When considering learning from and for the ESF project level there was a consensus that MS must play an effective role as a conduit between TNC at the pan-european level and ESF projects implemented at the local level within MS. In several instances respondents felt that this can best be ensured through MS organising thematic networks within their own countries that mirror those at European level. This was seen as enabling relevant representatives within ESF MAs to act as a pivot, validating good practice or policy learning emerging from the national level to be shared at the European level and likewise validating learning from TNC to be used in the MS context. Supporting mechanisms for publicising policy learning and good practice such as events, targeted papers, reports and so on were also raised as being significant in this context. The key focus of transnational and Learning Network type activity in the period Respondents tended to divide into three broad groups in terms of their views on the key focus of TNC and Learning Network type activity in the period. The first group felt that the focus should be on operational level collaboration between ESF MAs and MS, in part as this was felt to be more achievable and realistic as opposed to focusing on ensuring policy transfer. The second group held that policy learning and transfer should be maintained as a key focus, partly on the grounds that the results of such activity do have relevance at this level, and partly as a way of convincing policy stakeholders that TNC does have a role and can add significant value. The third group argued that while the operational level should be a key focus, this can be effectively combined with policy transfer if the mechanisms for this were better designed and supported. Of all the themes / issues for consideration included in the exercise, this difference of opinion was perhaps the most notable. Enhancing the effective dissemination of outcomes from Learning Network type activity and transnational co-operation A number of suggestions for enhancing dissemination activity were commonly cited by respondents. Those most often noted included the following: Inclusion of potential audiences for the results of TNC in the development and focus of activities from the outset; Developing tailored dissemination methods with a stronger focus on the audience in mind; Moving beyond brochures and similar approaches to more innovative dissemination methods The development of clearer and more precise dissemination strategies that are then monitored by the Commission / TA function; Ensuring a stronger role for, and expectations of, MAs in ensuring effective dissemination within their MS; 51

64 Persistence in developing and maintaining contacts and possible recipients of outputs on the part of those involved in TNC. Key improvements to transnational co-operation and Learning Network type activity The most common suggestion for improving TNC in the future period related to the role of the Commission and the TA function around such co-operation. The need to enhance the profile of, and commitment to, TNC on the part of the Commission was commonly noted, as was the need for an effective, professional and well resourced TA function to support MAs and MS to effectively engage. Other suggestions included: Ensuring that the themes chosen for TNC have a high degree of relevance to the most pressing issues facing labour market policies across MS; More effective dissemination at multiple levels covering the European, MS and local / regional level; Greater geographical concentration focused on strengthening links between older and newer MS and between east and west Europe. 6.5 Summary of key findings Stakeholders provided a range of views on the key strengths, weaknesses, and success factors emerging from the Learning Networks which can inform future Learning Network type activity and transnational cooperation more broadly. Key strengths cited tended to relate to the core purpose of the networks, in terms of transferring learning and sharing good practice, along with the quality of some of the networks outputs and perceived achievements. Weaknesses cited tended to relate to organisational and process issues in addition to a lack of impact in terms of directly influencing policy or policy transfer. Drawing on these views, a series of success factors relating to the networks can be summarised as follows: The importance of developing a clear and consistent thematic focus with realistic objectives being developed to reflect these themes Producing a clear implementation plan which sets out the relative responsibilities of individual partners, steering groups and working groups Effective network composition in terms of ensuring that appropriate and committed partners (and an appropriate range of partners) are engaged in network activity The development of clear governance processes and lines of accountability Effective organisation and leadership on the part of those co-ordinating network activity A well developed dissemination strategy, supported by high quality tools and outputs which are aimed at specific audiences. Examination of the effects of the Learning Networks at the Member State level also serves to highlight some key implications for how Member States might best engage in transnational co-operation. These include the need to ensure that Member States develop a clear strategy to guide their participation in such activity, the development of a supporting infrastructure able to transfer learning to relevant stakeholders in the domestic context as well as feeding in learning effectively to the European level, and ensuring consistency of participation as far as possible (both in respect of operational and policy actors). Running a virtual Policy Delphi to explicitly consider the future of transnational co-operation served to highlight several key considerations in this area. These include the importance of effective technical assistance to support the engagement of Member States and actors within them in transnational initiatives, the significant role the Commission can play in promoting transnational co-operation and its 52

65 value whilst actively guiding such co-operation in the context of the ESF, greater engagement of Commission policy units, greater realism over the both the potential and limitations of transnational cooperation on the part of both the Commission and those engaging in it, and the importance of Member States / ESF MAs acting as an effective conduit for transferring learning to domestic actors and back to the European level. 53

66 7.0 Conclusion and recommendations 7.1 Concluding remarks and revisiting the theory of change The assessment of the ESF Learning Networks has demonstrated that using this mechanism for transnational co-operation can lead to a number of positive outcomes at different levels. Reflecting on the intervention logic behind the networks (p.5 of the report), the assessment shows that the core rationale for the Learning Networks, that of facilitating access to new ideas and skills through collaboration and sharing practice across borders, has been achieved in a range of contexts. Through this sharing of practice, individuals, ESF MAs, organisations, the EU and Member States have benefited from the work undertaken. In terms of the main aims and objectives of the networks, a number of examples of improving the quality and efficiency of Structural Funds programmes are also apparent, although fully assessing the impact on employment, social inclusion and training across the Union is beyond the scope of this evaluation. The contribution of the Learning Networks to the aim of contributing to policy development at the EU level is perhaps less apparent, although this reflects the ambitious nature of such an objective along with the inevitable barriers to influencing and informing policy development in this way. Despite this, a number of the intended results of the networks have been achieved. Examples include the introduction of common management tools and approaches to monitoring and evaluation, such as the common standard for gender mainstreaming developed by the Gender CoP and the outputs of the RBM network, the development of stakeholder and practitioner collaboration at the OP level, and the further development of existing arrangements promoting transnational exchange and co-operation. Achievement of the intended results of the Learning Networks for individual members, as summarised in the intervention logic, also represents a clear area of success and strength of the Learning Network approach. Members clearly helped each other to solve problems, shared reflections to improve practice, applied and tested common working methods, exchanged skills, and gained in terms of their own professional development. At the level of participating organisations, again reflecting on the theory of change behind the networks, there were clear benefits for such organisations in terms of keeping up with developments across Europe, gaining access to new and common tools, and developing a common voice on some of the key issues faced. As discussed in section of this report, numerous examples are also available of the ways in which the networks contributed to the intended impacts of strengthening the capacity to innovate, modernising and adapting institutions, identifying and assessing issues and solutions for reforms in policy and delivery, and improving the quality of governance of policies, programmes and actions. Such impacts may have been greater had some of the weaknesses identified through this assessment not been present, particularly those relating to internal organisation and the effectiveness of dissemination. However, such issues and challenges that the networks faced also represent opportunities to learn from the experience by way of informing future transnational co-operation. In this sense there are some clear lessons from the assessment that inform the recommendations presented in section 7.3. Prior to turning to this, section 7.2 provides summary responses to the key evaluation questions included in the study ToR. 54

67 7.2 Summary answers to the evaluation questions Responses to the evaluation questions, based on the analysis presented in preceding chapters, are summarised below. Where applicable, the reader is signposted to the specific sections of the report within which the evidence to inform these responses can be found. EQ1 - To which extent have Learning Networks activities corresponded to the overall aims and policy objectives of the ESF? There is a clear relationship between the activities developed by the Learning Networks and the overall policy aims and objectives of the ESF. A high degree of relevance between network activities and these policy objectives and overall aims was apparent in the majority of cases (section 2.4). EQ2 - To which extent, under what circumstances, from whom and by whom learning has taken place during the duration of the Learning Networks? Learning deriving from the activities of the Learning Networks has taken place at a range of levels and amongst numerous actors. Whilst such learning is clearest and most apparent in respect of individual network participants (section 4.3), it is also widely apparent in respect of participating organisations, whether ESF MAs or other partner bodies (section 4.4). The ways in which learning took place varied between the networks, reflecting their different focus and the range of activities they undertook. Typically, however, peer reviews, study visits, events and conferences were cited as effective mediums through which to share practice and generate learning (section 3.4). There is also evidence of influence on policy learning, although evidence of direct policy transfer is limited (section 4.5). EQ3 - To which extent the activities of the Learning Networks have contributed to the achievement of the Learning Networks objectives? As reflected in the response to EQ.2, there was some variation in the specific activities which led to particular objectives being met or otherwise across the Learning Networks. In turn, this reflects the variation in the scope and focus of the networks themselves. Overall, however, a clear relationship between the activities networks engaged in, and the high level of achievement of the objectives they set themselves, was apparent in most cases (sections 3.4, 4.7). The key exceptions to this involved some networks failing to produce particular planned outputs and less evident achievement of objectives around influencing policy development (sections 4.5, 4.7). EQ4 - What are the achievements of the calls for proposals organised in the context of the Learning Networks (where applicable)? The assessment revealed that Calls for Proposals did not generally emerge from Learning Network activity. While some examples of activities influencing the design or development of Calls for Proposals in the context of Member State and regional ESF OPs were offered, the linkages between these and network activity were unclear. Overall, therefore, there was little evidence of network activity influencing Calls for Proposals, certainly not in the sense of directly leading to such calls (section 4.6). EQ5 - To which extent have the dissemination activities reached relevant stakeholders? The dissemination approaches adopted by the Learning Networks proved to be mixed in terms of their effectiveness. In general, ESF MAs were reached to a greater extent than other actors, as were ministries of labour, although to a lesser degree. NGOs, representatives from other ministries, and other actors such as employers and trade unions appeared to have been less well reached by dissemination activity (section 4.2). 55

68 EQ6 - What are the successful examples of mutual learning coming out of the Learning Networks activities and successful projects funded under the Call for Proposals? The assessment indicated that a wide range of successful examples of mutual learning emerged from the operation of the learning networks. These examples were evident at the individual, organisational and Member State levels (sections 4.3, 4.4, 4.5) and were achieved through knowledge exchange, along with the development of Learning Network products and outputs. 7.3 Recommendations Key recommendations for future Learning Network type activity in the context of ESF transnational cooperation which have emerged from the research findings are set out below. In order to facilitate reading, they have been grouped in relation to key aspects of the set-up, implementation, dissemination and support of such networks. Developing relevant and achievable network objectives Networks should address both policy-related themes (i.e. youth employment, social economy) and horizontal operational issues related to delivery and governance (i.e. results-based management, transnational co-operation). Sharing knowledge on policy-related themes can promote a mutual and deeper understanding of thematic priorities, including national specificities, and stimulate ideas through learning about new or different approaches to common challenges. Mutual learning and exchange on operational issues can allow ESF stakeholders to improve implementation through sharing good practice and developing common tools. The scope of themes covered by networks should remain wide, in order to allow choice, as well as the exchange of experience on themes likely to concern both a majority of Member States (i.e. youth employment) and on those which may only interest a smaller minority of Member States (i.e. social economy) or are ground-breaking in nature (i.e. social innovation). However, it may be beneficial to reduce the thematic spread of networks running over the same period, for example by organising a rotation of themes on a bi-annual basis throughout the programming period. Within broad thematic areas, specific objectives of individual networks should be defined and refined through a variety of mechanisms including: baseline analyses (to determine the current state of play), building on existing work and recommendations (i.e. from former ESF thematic groups, other European networks), input of thematic experts, involvement of wider stakeholders (i.e. NGOs, social partners) and in particular (see following recommendation) - an inclusive process of dialogue with network members. The benefits of long-term and on-going exchange between network members (sometimes over a series of network lifecycles), leading to increased mutual trust and understanding, can lead to further benefits in terms of defining relevant, targeted objectives. Networks should not be over-ambitious in defining the range of objectives: a greater focus on a smaller number of objectives is likely to focus energies and be more effective. A democratic and participative bottom-up approach involving network members in defining the precise, concrete focus of network objectives has been shown to increase the relevance of network objectives to Member State priorities, foster long-term commitment of members and develop a network identity. To ensure the relevance of network objectives and activities to ESF priorities and wider EU policy drivers (such as the EU2020 strategy), this approach should, 56

69 however, be combined with a stronger top-down role of the European Commission in guiding the broader development of objectives. The objectives of networks should not only focus on producing outputs (i.e. events, publications) but also focus on and allow space for the less tangible goals of learning, sharing and collaborative working. Network participants have a variety of motivations for joining networks producing new tools, sharing experiences, exchanging good practice, mutual support, learning which should all be considered and accommodated within network design. ESF thematic networks should not become another form of expert network, but retain their specificity in relation to promoting learning between key ESF stakeholders. The development of network objectives and strategic plans should be carried out through a structured approach, for example through the use of the logical framework methodology ( logframes ). Such a structured approach provides a framework for identifying the key problems/issues and defining overall and specific objectives, results/outputs and activities to be implemented to achieve the aims of such activity. It also allows the structured development of indicators to measure progress, and assumptions or pre-conditions required in order to achieve these aims (i.e. availability of resources, long-term commitment of members). While developing such a structured approach, it will however be important to build in a degree of flexibility to network aims and activities, to allow adjustments over time in order to react to external factors (i.e. change in policy priorities, changes in governments, economic downturn/upturn, withdrawal of network members), including a range of contingency measures. Increasing participation and balance of participation in networks Networks should be encouraged and supported to widen and diversify Member State participation. Some ESF thematic fields have been taken forward in networks by a core group of Member States over a series of thematic networks and, while this has had value in terms of building expertise, developing efficiencies and forging strong links, it is also important that other Member States are encouraged to join in order to extend learning opportunities and inject a fresh perspective as well as new ideas/approaches. Wider participation can be encouraged for example through: greater promotion of the value and benefits of transnational exchange through ESF networks by the European Commission (i.e. via the ESF Committee) with support from technical assistance, including proactively approaching certain Member States in certain thematic fields; introducing criteria or recommendations on geographical spread or minimum numbers of Member State members into terms of reference for networks; encouraging network leads to be more proactive in opening up networks to new members; or, using technical assistance to support Member States with less experience in transnational working of this nature to participate. While recognising that such networks are primarily learning tools and mechanisms for ESF stakeholders, it should be ensured that, where relevant, the participation of other types of actors (i.e. NGOs, national and regional policy-makers, academic experts, social partners) is encouraged and facilitated, in order to increase learning, enhance relevance to policy developments and ensure a balance and complementarity of perspective. Such participation can take a number of potential forms, for example: direct participation of a limited panel of external actors as core members of networks (i.e. participation of REVES in the current social economy network); development of a stakeholder steering group of other actors who would participate in certain meetings and liaise on a regular basis with the network leads (this would an extension and improvement of the mechanism of support groups tested in the period under assessment which involved only Commission policy representatives and worked only to a limited degree); or, opening up wider events, such as seminars, peer reviews or conferences, to relevant other actors. In any of the above scenarios, both the Commission and their technical assistance should 57

70 play a more proactive role in identifying relevant stakeholders in specific fields and securing their participation. In order to bridge the gap between policy-makers and practitioners, future iterations of networks should better reflect ESF projects on the ground and at an appropriate level involve ESF project managers. Due to the non-existence of a common transnational programme/calendar in the programming period, transnational exchange between ESF-funded projects was much more limited, as well as their visibility in the Learning Networks, even including events. Future networks should seek to have a greater focus on good practices implemented by ESF-funded projects, in order to increase cross-border learning and exchange of promising practice, but also to reinforce links between networks and concrete implementation, a link which was significantly weakened in the current programming period. The benefits of, and learning from, networks should be felt more strongly throughout the ESF implementation chain, not only at management level. ESF project managers could be invited to participate in certain events (i.e. seminars, peer reviews, conferences). Where relevant, Member States should also be encouraged to set up shadow thematic networks at national level, bringing together ESF project managers at national events to share good practice and approaches to tackling common issues, serving both as a source of bottom-up input to transnational thematic networks and also a top-down vector to convey developments and findings from the transnational working groups. A future technical assistance may be able to support and facilitate these processes on behalf of the European Commission. On a concrete level, the importance of on-going and long-term commitment of network members should be reinforced in communication from the European Commission, supported by proactive follow-up by technical assistance, facilitated via the availability of funding to attend meetings, and encouraged by network leads. Our assessment repeatedly encountered the disruptive effects of changing levels of participation or withdrawal of members. While such change is unavoidable to some degree, contingency measures should be implemented to limit its impact through, for example, naming two participants per Member State (a permanent and backup member) or introducing some sort of guidelines to expected levels of participation in letters of commitment signed by network members. Improving network organisation and leadership Measures should be taken to build capacity and support effective network leadership and management, and reduce the workload on network lead Member States. The European Commission should provide clearer guidelines on what network leaders are expected/not expected to do, using where relevant examples of good practice and providing practical tools (i.e. checklists, simple guidance). Technical assistance should allow the provision of much stronger support to network leads including training in network management skills (i.e. logical frameworks, facilitating and organising meetings) and process support including planning, logistical support and advice on contents for network meetings, facilitation of meetings, and follow-up to meetings including minutes. Support for developing websites or interactive platforms could also be provided in order that network leads focus on content rather than technical issues. Networks should be encouraged to name a dedicated network coordinator, with appropriate network management skills and/or experience, who may not necessarily be within the ESF MA. Networks should be encouraged and assisted to develop appropriate organisational structures to ensure effective network management, including clear lines of governance and accountability. Our evaluation has shown that there is no one-size-fits-all approach to the networks under evaluation, with some adopting a flat management structure and others a two-tier or concentric circle approach involving a core steering group of more committed Member 58

71 States/other actors (i.e. content experts) and then the wider network members. As such, it will be important to allow a diversity of approaches to network organisation within a broader framework. The appropriate structure will be dependent on the size of the network, as well as its objectives, activities, scope, membership reach, relationship between members and duration. Networks should be supported by technical assistance in developing appropriate structures, and ensuring clear definition of roles and remits for all actors (i.e. steering groups, working groups, and individuals) from the outset, ideally agreed and discussed in a democratic manner. Core groups should seek not only to include the most experienced Member States, but also seek to involve a sample of those newer to this type of transnational work. Increasing the quality of network activities Improving the quality of network activities should be supported by sharing success factors, lessons from evaluations and examples of effective practice. The lessons from this evaluation, as well as individual Learning Network evaluations (whether self-evaluations or external evaluations) should be pooled and shared with network leads, then reinforced through process support provided to new networks by the technical assistance. Success factors in terms of the quality of activities which should be taken into account by future networks include: effective planning and follow-up; clearing identifying target audiences at the outset; tailoring outputs to different target groups; developing realistic and focused (as opposed to wide and over-ambitious) goals for each activity, taking time and resource constraints into account; implementing activities which combine both policy and practice, seeking to bridge the gap between policy-makers and practitioners; introducing flexibility to modify/cancel activities planned in line with changes in context; ensuring that activities are interactive, enabling active input from participants to facilitate mutual learning and knowledge exchange; linking and embedding activities within other relevant events and policy developments; where relevant, implementing activities in collaboration with external stakeholders such as other DGs of the European Commission, other European agencies or institutions (i.e. Eurofound, European Parliament) or NGOs; activities involving the sharing of good practices, in particular when scrutinised collectively in terms of practical implementation and potential for transferability. Improving efficiency Efficiency of the use of financial and human resources allocated to networks should be increased significantly through improved network management, organisation, monitoring, guidelines from the European Commission, and process support from technical assistance (as set out in the preceding sections). These measures should avoid in particular resources which should be dedicated to thematic and content exchange being spent inefficiently on process and organisational issues, by individuals and organisations poorly equipped to do so. Going forwards into the next phase, it is important to note that efficiency of networks should not be measured in terms of quantity (or indeed, to some degree, quality) of outputs, remembering that the key and less tangible and measurable goals are those of transnational networking, learning and mutual exchange. Networks could be more technically efficient in delivering high quality outputs in shorter timeframes by employing experts to produce outputs; however, this approach (although relevant potentially for specific parts of network activity) defeats the object of using the process of producing outputs as a vehicle for learning for all network participants. Notwithstanding, networks need to consider carefully the likely degree of efficiency of certain types of activities in delivering expected aims and contributing to network goals. In certain specific cases, our evaluation has called into question whether specific activities, including 59

72 for example conferences or the production of leaflets or publications, were the most efficient means of achieving positive effects. There has sometimes been an implicit assumption that events or specific outputs will be useful, involving devoting significant resources to these activities, without sufficient consideration as to whether it may have been more efficient to pursue other types of activity to achieve goals more effectively. Network resources and efficiency can be significantly enhanced through in-kind contributions (from ESF MAs, policy-makers and other network partner organisations) and strategic linkages of network activities with relevant external events and activities. Targeted outreach and smart partnering with external partners (i.e. relevant NGOs, other networks or academic experts) can be highly effective as these actors may be interested in participating in and contributing to certain network activities as part of their official remit, thus developing synergies as well as increasing impact and quality with limited cost and resource implications. Efficiencies in terms of overall transnational networking could be increased by significantly developing synergies and joint working across different transnational networks. Networks should be provided with improved mechanisms for the sharing of processes, results and good practice through, for example: improved and more dynamic network coordination meetings, the provision of an interactive and regularly updated common web platform for sharing experiences and progress, top-down direction and twinning of relevant networks by the European Commission (supported by technical assistance), involvement of representatives from different networks in relevant events and activities, and more regular joint dissemination activities (i.e. in ESF Committee meetings and other relevant exchange fora). Both the European Commission and network leads need to take greater responsibility for ensuring that such synergies happen, supported by a future technical assistance. Improving dissemination and learning outcomes All networks should develop a comprehensive dissemination strategy setting out the key target groups, the different levels of dissemination (EU, national, regional), potential multipliers, dissemination activities and tools, roles and responsibilities for dissemination, and follow-up activities. Technical assistance could provide support, training, facilitate mutual learning and provide templates for network leads in developing effective dissemination strategies. Individual network members should also be requested to produce national dissemination strategies setting out how they intend to share the results of the network in their own Member State. At EU level, the Commission, with support from technical assistance, should also set out an EU dissemination strategy for the networks results. There should be a much greater focus on dissemination of results at national and regional level, with the issue being discussed in a more systematic manner both within individual networks and in EU network coordination meetings. Within the framework of the national dissemination strategy, network members should become ambassadors to share network learning within Member States, with the aim of developing wider and organisational learning. Member States should also consider the establishment of shadow thematic networks or groups of key thematic stakeholders at Member State level to disseminate results. There should be better links with, and more support from, the geographical desks in the Commission for disseminating learning and outcomes within their respective Member States. Ideally an infrastructure should be established at national level (whether in the ESF MA, labour ministry, IB or elsewhere) to support transnational working more generally, which could also facilitate the dissemination and sharing of results. 60

73 Networks should learn from the lessons in good practice in dissemination from previous networks in developing their own strategies. Key success factors include: tailoring outputs to different audiences (i.e. format, style, length, degree of complexity), for example ensuring concise and recommendation-focused outputs for policy audiences; building effective and on-going links with key players at EU and Member State level; moving beyond brochures to more innovative dissemination methods, making use of the latest technologies where relevant; ensuring the quality and suitability of the tools and outputs developed for dissemination; producing regular concise reports for wide dissemination such as e-newsletters; and, ensuring follow-up to dissemination carried out for example through events or mailing lists. Overall, the networks should favour more active forms of dissemination such as events, meetings or on-going liaison with the targets of dissemination activity, as opposed to more passive forms of dissemination such as printed materials or those only hosted on websites. They should also actively seek to extend the reach of dissemination beyond easier targets such as ESF MAs, to reach wider stakeholders such as social partners, NGOs and the academic community. Networks should seek to go beyond dissemination to developing more detailed strategies for mainstreaming successful outcomes including tools, publications, ideas or approaches. This process could be supported for example by identifying key targets and more consistently following up to check on, and encourage, incorporation of products and toolkits in policy and practice. Network activities relating to good practices should actively consider transferability issues to other contexts and other Member States. Debates around transferability should seek to focus on positive aspects of how solutions and approaches may be transferable, rather than focusing on reasons why they may not be, as is more frequently the case. Such discussion may thus facilitate organisational and policy learning. It is important that networks consider and plan the sustainability of relevant impacts and outcomes. At an individual level, this may involve establishing mechanisms (i.e. mailing lists, LinkedIn groups) to continue personal contacts beyond the lifespan of networks. At the level of networks themselves, it may include consideration of the benefits of establishing a more sustainable network, or developing spin-offs on specific themes at either European, national or regional level, or even joining other existing networks to continue exchanges within another forum. In terms of the sustainability of learning outcomes and policy lessons, it will be important to, among others: ensure that outcomes are recorded (reports, publications, websites); ensure that results are disseminated in an appropriate manner (see recommendations above); actively consider transferability issues; develop and follow-up on appropriate mainstreaming strategies; empower individual and organisational network members to be ambassadors for on-going learning beyond the life of the network, including through the provision of appropriate tools and skills development. Improving support and monitoring Representatives of the European Commission should show a visible interest and, where possible, presence in the work of the thematic networks. This visible engagement is crucial to the commitment of network members, to the sense of the value and relevance of their work, to ensuring the on-going relevance of network activity within the EU policy context and to facilitating mainstreaming. Support to the network should ideally be provided by a consistent Commission representative (where possible with a second back-up representative in reserve to ensure continuity in the absence of the main representative), ideally reinforced by a wider Commission support group involving representatives not only from the ESF unit, but also from relevant policy units in the Commission. When developing policy or holding events in parallel thematic areas, the 61

74 Commission should ensure that the networks are given the opportunity to participate and input contributions. In order to increase their effectiveness and efficiency, networks should be provided with increased support, for example by a future support contractor. In close cooperation with the European Commission, this support could cover both content and process issues including: guidance in developing network objectives and planning network activities; support in facilitation and logistical organisation of network meetings; provision of guidance and tools for effective network management; capacity-building and guidance for dissemination and mainstreaming; identification of relevant external stakeholders and organisations, both for participating in network activities and as targets for dissemination and mainstreaming; support for administrative and financial management of the networks; identification of good practice; locating appropriate experts, facilitators and evaluators; feeding back to the Commission on progress; and organisation of network coordination meetings. It will thus be essential for the future contractor to have knowledge and experience in, among others, the key thematic priority areas, in network management, in ESF and in transnational co-operation. Overall, there must be clarity and transparency in the roles and responsibilities of all stakeholders involved with the networks and their support. In order to foster a process of continuous improvement, each network should include a process of on-going evaluation, providing constructive feedback on key achievements and areas for improvement. On-going evaluators should provide regular reviews and suggestions for improvement in relation for example to content, internal organisation and processes, relevance of objectives and activities, and degree of attainment of network objectives. Improved support mechanisms should be put in place to facilitate cross-fertilisation and exchange of ideas between networks. In particular, network coordination meetings should be continued, but developed into more dynamic events, involving the sharing of good practice and developments in networks. In between meetings, exchanges could be facilitated by an interactive and moderated web platform, and/or regular (monthly or bi-monthly) newsletters on network activities, featuring both network activities (i.e. events) and good practices at the level of ESFfunded projects. Content could include both top-down policy updates in relevant fields, and bottom-up exchange of experience and sharing of network results. The overall organisation of meetings should facilitate both formal and informal exchanges. 62

75 Annex One: Additional detail on the theoretical framework A1

76 Evaluation Framework Drawing on the research questions underpinning the evaluation part of the theoretical approach involved determining a series of indicators to allow us to identify and gauge the level of impacts observed. This model of indicators is implicitly applied at all levels of the evaluation, from activities undertaken by ESF Learning Networks to the ESF Learning Networks themselves, as demonstrated the below. As the summary table shows, the evaluation framework also mapped the evaluation questions to sub-questions and the methodological work packages, thus providing a guide for the overall assessment process regarding the learning networks. A2

77 Evaluation framework in tabular form Evaluation Questions (EQ) addressed Sub questions Indicators Work packages to provide evidence EQ1 - To which extent have Learning How were the themes for Learning Network Degree of innovation in the way themes were Desk research Network activities corresponded to the activities selected? What is the added value elaborated Key stakeholder interviews overall aims and policy objectives of the of the Learning Networks to ESF objectives? Degree of relevance of themes to ESF and Learning Network case studies ESF? To what extent have the Learning Networks Member State priorities (correspondence Member State case studies activities been complementary to the with, importance to) Web-based survey network stakeholders activities of other initiatives at EU and Degree of input into thematic prioritisation national level? (members and wider stakeholders Existence of explicit linkages to other networks (partnerships/joint activities) Degree of (in)novation of activities in comparison to those undertaken by other initiatives at EU and national level EQ2 - To which extent, under what Provide examples of policy practices that Credibility and pertinence of messages Desk research circumstances, from whom and by whom have been (in part) transferred to other emerging from activities and captured in Key stakeholder interviews learning has taken place during the duration countries? reports (e.g. based on opinion of participants Learning Network case studies of the Learning Networks? Provide examples of peer learning activities or experts) Member State case studies demonstrating that certain policy practices Individual participants reporting learning cannot be transferred to other countries? benefits (number, nature and depth) What are the short- and long-term impacts of Ministries and other bodies reporting the LNs results? organisational learning (number, nature, How did policies or projects subject of depth) Learning Networks activities develop over Evidence of mutual fine-tuning and time? orchestration of transnational learning arrangements and their valuation by stakeholders Existence of action plans to base ESF A1

78 Evaluation Questions (EQ) addressed Sub questions Indicators Work packages to provide evidence support on learning from Learning Networks Introduction of common management tools as a result of Learning Networks Common approaches introduced as a result of Learning Networks Reports by stakeholders on various forms of mutual assistance Application or testing of common working methods during or after the Learning Network s existence Representatives of ministries and other bodies reporting increased capacity for policy-making and policy implementation in their organisation, particularly in light of current challenges Ministries and other bodies reporting improved working relationships with other relevant stakeholders (including DG EMPL, other ministries, social partners, etc.) within and across Member States Ministries and other bodies reporting increase in organisational knowledge and competencies relating to the LN s theme, or general professional competence Ministries and other bodies reporting the use of outputs from the Learning Networks DG EMPL incorporating innovations and other learning into ESF and other EU policies and initiatives A2

79 Evaluation Questions (EQ) addressed Sub questions Indicators Work packages to provide evidence Changes in Learning Networks over time, reasons for these changes and their significance in terms of success / failure factors EQ3 - To which extent the activities of the What success/ failure factors of policies/ Success/ failure factors mentioned by Desk research Learning Networks have contributed to the projects did the Learning Networks activities stakeholders and their occurrence Key stakeholder interviews achievement of the Learning Network identify? Increase in personal networking and contacts Learning Network case studies objectives? Which informal networks resulted out of across different Member States according to Member State case studies Learning Network activities? Who stakeholders Policy Delphi participates in these networks? Number, nature and depth of new/improved Have these informal networks been used for structured networking and co-operation policy reform? across different Member States and at EU Are financial, time and human resources level, during and after the Learning Network s allocated to Learning Networks proportional existence to the achievements of the activity? Valuation of resources allocated to the networks by its stakeholders, in view of its objectives and its achievements Global intra-network comparison of resources, reach and outputs EQ4 - What are the achievements of the To what extent did these projects yield Number, nature and depth of new/improved Key stakeholder interviews calls for proposals organised in the context partnerships or networks for the selection and structured networking and co-operation Learning Network case studies of the Learning Network? (where applicable) transfer of policy practices? across different Member States and at EU Member State case studies To what extent did these projects promote level as a result of calls for proposals mutual learning and effective dissemination of Member States and other actors adjusting knowledge among and within EU Member policies, programmes or systems in response States? to organisational learning, e.g. adopting To what extent were authorities and innovations or good practice from other A3

80 Evaluation Questions (EQ) addressed Sub questions Indicators Work packages to provide evidence stakeholders from different levels of countries, as evidenced by content or governance involved? testimony of stakeholders What are factors of success and failure of Degree to which authorities and stakeholders these projects? from different levels of governance were involved Member States and other actors better aligning national/regional programmes with priorities set out in the ESF according to stakeholders Number, nature and depth of new/improved structured networking and co-operation across different Member States and at EU level, as illustrated by case studies Success /failure factors as reported by participants EQ5 - To which extent have the To what extent have Learning Network Number and type of participants/members of Desk research dissemination activities reached relevant activities been followed-up by participants to networks and activities therein Key stakeholder interviews stakeholders? disseminate good practices to authorities and Balance and regularity of the participation of Learning Network case studies stakeholders from different levels of Member States and key EU-level Member State case studies governance? stakeholders in terms of type of stakeholders Web-based survey network stakeholders Are current dissemination activities adequate Level / appropriateness of representation for dissemination purposes? within networks Which are cases of LNs results that were Number / type of actors reached by follow-up successfully transferred? dissemination activities Instances of disseminated results referenced by policy makers and practitioners A4

81 Evaluation Questions (EQ) addressed Sub questions Indicators Work packages to provide evidence EQ6 - What are the successful examples of What are successful examples of mutual Examples of policy transfer and exchange Key stakeholder interviews mutual learning coming out of the LN learning resulting from LN activities and Examples of policies and projects being Learning Network case studies activities and successful projects funded projects funded under Call for Proposals? developed as a consequence of Learning Web-based survey of network under the Call for Proposals? What are the success factors contributing to Network activity stakeholders the success of the examples of mutual Success factors identified by evaluations or Policy Delphi learning in the context of LN activities? stakeholders, and their occurrence What are the success factors contributing to the success of the examples of mutual learning in the context of projects funded under Call for Proposals? Which of these success factors should be taken into account when planning future activities? A5

82 Annex Two: Research tools A6

83 Evaluation of the ESF LN Topic Guide Network coordinators/leads Name Organisation Job title Date of interview Role in relation to the ESF LN 1. Please describe your role. 2. What is (or has been) your involvement with the ESF LN and what does it involve? Objectives and relevance 3. The ESF LN s (name of specific network) main strategic objectives are (insert objectives of specific ESF LN). How has the theme been relevant to: 1. The ESF? 2. Participating partner priorities? 4. How was the theme for the LN selected? Was it selected in the most effective way? Were the correct stakeholders involved in selecting and suggesting the theme? 5. Do you feel the activities undertaken by the LN were relevant to the objectives of the LN? Why? 6. What evidence was used to set up the activities how was the need for particular activities established? 7. Have thematic priorities and activities adjusted to respond to any changes in context, such as the impact of the economic crisis? 8. Has your LN worked with any other LN? If so why and in what way? Effectiveness 9. Were specific LN activities particularly successful or unsuccessful? If so, which and why? 10. How effective have activities been in achieving the Network s aims? 11. Do the correct people/partners participate in LN activities? Correct partner representatives, on-going and stable membership, commitment to input into activities? A7

84 12. Has the format and quality of events (such as seminars and conferences) been appropriate for achieving the aims of the LN? Preparatory documents, moderators, papers, language, time for debate? 13. What were in your view the main benefits of participating in LN activities to: a. Individual participants? b. Organisations? c. Member States/regions? 14. Are there instances where new policies or changes have been developed at any organisational level as a consequence of the LN activities? At which level do you feel were the main benefits? 15. To your knowledge, has participation in the LN led to any adaptation of policies and/or interventions on key themes, or adopting new policies/practices in Member States/regions or at EU level? Have new approaches to key challenges been introduced? 16. Are there instances where activities have shown certain policy practices cannot be transferred to other countries? 17. Are there any internal processes (such as management/coordination procedures) that have been particularly effective or challenging and why? 18. To what degree has the LN contributed to improved ESF transnational cooperation between Member States? 19. What are or will be the short- and long-term impacts of the LN s results? 20. Are you aware of any informal networks or other spin-offs created following the LN s activities? Who is taking part in these? Are these networks subsequently used for specific policy-related collaboration? Dissemination 21. Overall, has the dissemination/learning strategy of the LN been effective? 22. What have been the main mechanisms to allow participants to follow up learning events or activities? 23. Have participants in activities been given the right tools for dissemination? Were outputs of activities conducive to effective dissemination (reports, website )? 24. Which levels has dissemination most effectively reached (EU, national, regional/local, other actors )? Were you able to disseminate to other LNs effectively? A8

85 Support mechanisms 25. What is your view on the support provided by the Commission to ESF LNs? Are structures and procedures adequate? What more could or should be done? 26. Have you used the European Commission Support Group to gain assistance? If so what was the purpose and did you get the assistance you needed? 27. What is your view on the role of AEIDL in monitoring and supporting the ESF LNs? Are structures and procedures adequate? What more could or should be done? 28. What works well with regards to support and monitoring? 29. Were the Network coordination meetings useful? Were you given sufficient opportunity to liaise with other Networks? Calls for proposals 30. Are you aware of any domestic ESF calls for proposals being launched as a result of/influenced by the activities of the ESF LN (gain contacts etc. if yes)? 31. Were these coordinated by particular LN members? 32. What were the aims and objectives? General overview 33. To what degree do you feel that the LN contributed to the following overarching aims of the LNs: a. Strengthening the capacity to innovate? b. Modernising and adapting institutions to new social and economic challenges? c. Identifying and assessing issues and solutions for reforms in policy and delivery? d. Improving the quality of governance or public policies, programmes and actions? 34. Do you feel that the resources allocated to the LN are justified by the results? 35. What have been the most positive aspects of the LN (processes, outcomes )? 36. What have been the weakest aspects of the LN (processes, outcomes )? 37. What key improvements could be made to the LN in a future iteration to ensure a greater degree of relevance to, and impact on, EU and national policy-making? 38. What additional support could have been provided by the EC/EC contractors in order to facilitate the work and/or improve the impact of the LN? A9

86 Evaluation of the ESF LN Topic Guide DG EMPL Representatives Name Organisation Job title Date of interview Role in relation to the ESF LN 1. Over which period have you managed/did you manage the ESF LN at DG EMPL? 2. Have you participated in activities of all LNs during the period you have managed the programme? 3. What has been your specific role in relation to the ESF LNs? Support to ESF LNs 4. What is your view on the support provided by the Commission to ESF LNs? Are structures and procedures adequate? What more could or should be done? 5. What is your view on the role of AEIDL in monitoring and supporting the ESF LNs? Are structures and procedures adequate? What more could or should be done? 6. What works well with regards to support and monitoring? Objectives and relevance 7. Have there been ESF LNs with objectives particularly relevant to: The ESF? Participating partner priorities? 8. Do you feel that LN themes were selected in the best way? Were the correct stakeholders involved in selecting and suggesting the theme? 9. Do you feel the activities undertaken by LNs were relevant to the objectives of the LN? Why? Are there particular good examples or challenging cases? 10. Do you believe that evidence was used to set up the activities was the need for particular activities established? A10

87 11. Have thematic priorities and activities of ESF LNs adjusted to respond to any changes in context, such as the impact of the economic crisis? Effectiveness 12. Were specific LNs and in turn LN activities particularly successful or unsuccessful? If so, which and why? 13. Has the format and quality of events (such as seminars and conferences) been appropriate for achieving the aims of LNs (preparatory documents, moderators, papers, language, time for debate )? Are there specific examples? 14. Do the correct people participate in LN activities (correct partner representatives, on-going and stable membership, commitment to input into activities )? 15. What were in your view the main benefits of participating in LN activities to: a. Individual participants? b. Organisations? c. Participating Countries? 16. To your knowledge, has participation in the LNs led to any adaptation of policies and/or interventions on key themes, or adopting new policies/practices? Have new approaches to key challenges been introduced? 17. Broadly speaking how effective have activities been in achieving the Network s aims? 18. Are there any internal processes (such as management procedures) that have been particularly effective or challenging in ESF LNs and why? 19. Are you aware of examples of policy practices being transferred to other countries? 20. Are there instances where activities have shown certain policy practices cannot be transferred to other countries? 21. Are there instances where new policies have been developed at any organisational level as a consequence of the LN activities? 22. Are there instances where organisational changes have been implemented at any organisational level as a consequence of the LN activities? 23. What do you feel are or will be the short- and long-term impacts of the LNs results? Dissemination A11

88 24. Overall, has the dissemination of LNs been effective? What have been the main mechanisms to allow participants to follow up learning events or exercises? Which LNs were most successful and why? 25. Have participants in activities been given the right tools for dissemination? Were outputs of activities conducive to effective dissemination (reports, website )? Which LNs were most successful and why? 26. At what levels do you feel has dissemination been effective in reaching (EU, national, regional/local, other actors )? Calls for proposals 27. Which domestic ESF calls for proposals were launched as a result of/influenced by the activities of the ESF LN (gain contacts etc. if yes)? 28. What were the aims and objectives? 29. Are you aware of any impacts or outcomes stemming from the Calls? General overview 30. Do you feel that the resources allocated to the LNs are justified by the results? 31. What have been the most positive aspects of the LNs (processes, outcomes )? 32. What have been the weakest aspects of the LNs (processes, outcomes )? 33. What key improvements could be made to LNs in a future iteration to ensure a greater degree of relevance to, and impact on, EU and national policy-making? A12

89 Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks Topic Guide AEIDL Name Organisation Job title Date of interview Role in relation to the ESF LN 1. Over which period have you been/were you involved in the monitoring of the ESF Learning Network? 2. Have you participated in activities of LNs? If so which and why? 3. What has been your specific role in relation to the ESF LNs? Objectives and relevance 4. To what degree have the objectives of the Networks been relevant to the overall ESF objectives? Can you highlight examples of Networks which have been particularly relevant? To what degree have the objectives of networks been relevant to participating partner priorities (ESF MAs, other partners)? 5. Do you feel that Network themes were selected in the best ways? Were the correct stakeholders involved in selecting and suggesting the themes? 6. Do you feel the activities undertaken by Networks were relevant? Why? Are there particular good examples or challenging cases? 7. In your experience was the need for particular activities established through a needs analysis or other evidence? 8. Have thematic priorities and activities of any Networks adjusted to respond to any changes in context, such as the impact of the economic crisis? 9. To what extent is there mutual exchange between ESF Learning Networks and other fora of exchange in the field of European employment policy (for example, Social Dialogue and PES- PES dialogue exchanges)? A13

90 Effectiveness 10. Were specific Networks - and in turn activities - particularly successful or unsuccessful? If so, which and why? 11. Broadly speaking how effective have activities been in achieving the Networks aims? 12. Did the correct people participate in Network activities (correct partner representatives, ongoing and stable membership, commitment to input into activities )? 13. Was the format and quality of events (such as seminars and conferences) appropriate for achieving the aims of Networks (preparatory documents, moderators, papers, language, time for debate )? Are there specific examples? 14. What were in your view the main benefits of participating in activities to: a. Individual participants? b. Organisations? Are there instances where new policies have been developed at any organisational level as a consequence of the activities? Are there instances of organisational change? c. Member States/regions? 15. To your knowledge, has participation in the LNs led to any adaptation of policies and/or interventions on key themes, or the adoption of new policies/practices in Member States, regions or at EU level? Have new approaches to key challenges been introduced? 16. Are there instances where activities have shown certain policy practices cannot be transferred to other countries? 17. Are there any internal processes (such as management procedures) that have been particularly effective or ineffective in Networks and why? 18. What are or will be the main short- and long-term impacts of the LNs results? Consider the impact both at national and EU level. Dissemination 19. Overall, has the dissemination undertaken by Networks been effective? What have been the main mechanisms to allow participants to follow up learning events or exercises? Which LNs were most successful in terms of dissemination and why? 20. Have participants in activities been given the right tools for dissemination? Were outputs of activities conducive to effective dissemination (reports, website )? 21. Which levels has dissemination effectively reached (EU, national, regional/local, other actors )? A14

91 Support mechanisms 22. Were the Network coordination meetings useful? Were Networks given sufficient opportunity to liaise with each other? 23. What is your view on the support provided by the Commission to Networks? Are structures and procedures adequate? What more could or should be done? 24. What is your view on the European Commission Support Group and its assistance to the Networks? 25. What is your view on the role of AEIDL in monitoring and supporting the Networks? Are structures and procedures adequate? What more could or should be done? 26. What works well with regards to support and monitoring? General overview 27. To what degree do you feel that Networks (or specific a specific Network) contributed to the following overarching aims: a. Strengthening the capacity to innovate? b. Modernising and adapting institutions to new social and economic challenges? c. Identifying and assessing issues and solutions for reforms in policy and delivery? d. Improving the quality of governance or public policies, programmes and actions? 28. Do you feel that the resources allocated to the Networks are justified by the results? 29. What have been the most positive aspects of the Networks (processes, outcomes )? 30. What have been the weakest aspects of the Networks (processes, outcomes )? 31. What key improvements could be made to Networks in a future iteration to ensure a greater degree of relevance to, and impact on, EU and national policy-making? 32. What additional support could have been provided by the EC/EC contractors in order to facilitate the work and/or improve the impact of the Learning Networks? A15

92 ESF LN case study guidance and template Aim The purpose of the case studies will be to: Examine Network member satisfaction and opinion regarding: quality and relevance of activities, as well as "messages" emerging from Network activities; Assess the extent and impact of dissemination activities; Determine the extent, nature and depth of learning benefits; Identify the extent and nature of increase in personal networking and contacts; Identify instances of organisational learning and change, e.g. increased capacity, new/improved relationships, networking or co-operation with other relevant stakeholders; Identify instances of policy change, to which the Learning Networks have demonstrably contributed. Method The case studies should be conducted using a mix of desk and interview research. Travel for interviews is not foreseen in the budget. Deadline for all case studies is XX. 1. Desk research The case studies will commence with a more in-depth analysis of documentation gathered in the Inception Phase. The following types of document will be reviewed in a structured manner: Interim and final reports of the Learning Networks submitted to the Commission; A16

93 Monitoring reports prepared by AEIDL, quarterly reports for each Learning Network, produced during the whole duration of the grant agreements concluded between the Commission and the Learning Networks; Reports, evaluations and tools produced by the Learning Networks; Reports of the major events organised by the Learning Networks. 2. Intervention logic Based on the information available, an intervention logic will be developed for each case study network. This should be an adapted version of the following model (Please tailor all sections to the specificities of the Learning Network in question): A17

94 Rationale Aims and objectives Inputs Activities Results Impacts Collaborating and sharing experiences across borders, whether national, regional or organizational, is an effective way to access new ideas, innovative approaches and new skills Use of Learning Networks as one mechanism by which transnational cooperation and exchanges of good practice and knowledge can be facilitated, hence supporting the implementation, delivery and outcomes of ESF Support the objective of: - improving the quality and efficiency of Structural Funds programmes and their impact on employment, social inclusion and training across the whole Union -contributing to policy development at EU level through: - exploiting opportunities for programme managers, strategic stakeholders, governmental departments in charge of policy supported under the ESF and practitioners for learning from one another, and with each other - capitalising good practice under the ESF Operational Programmes (in particular from innovative action and transnational cooperation) Financial support through the ESF Technical Assistance operational budget (maximum 4m under 2008 call for proposals and Financial support through the 3.6m ESF Technical under Assistance 2009 call operational budget (maximum of 4 million under 2008 call for proposals and Additional 3.6 million under 2009 contributions call) through the Additional contributions through Commission s the Commission s role in role facilitating in good practice and acting as a practice and acting catalyst for learning and as a catalyst change for learning and Additional (in-kind) Member change State contributions (financial and human resources) AEIDL input in monitoring the ESF Learning Networks Additional (in-kind) Member State contributions (financial and human resources) Thirteen Learning Networks undertaking a range of activities including: - exchange events - peer reviews - learning seminars - policy fora - the development of toolkits, communication platforms and technologies - the transfer of competencies and experience through mechanisms such as training and coaching Results Programme are achieved level: at a series - Fine-tuning of interrelated of arrangements levels. promoting transnational exchange Individual outcomes: and co-operation - Stakeholder and practitioner networks at OP level - Launching - Increased of action exchange plans to base and ESF support networking on agreed good - Instances practice of individual - Introduction learning of common (management) tools - Individuals reached by - Common approaches to monitoring, evaluation dissemination or reporting Organisational Individual members: level: - helping Improved each other organisational in solving problems at operational and strategic knowledge levelsand capacity - - providing Improved stimuli linkages to reflect to on other practice organisations and improve it - applying - Increased common, tested, knowledge working methods - providing Programme opportunities level: for staff exchange - gaining professional skills and - Development recognitionof common tools Participating - Policy organisations: transfer - keeping up with developments - Convergence of MS across Europe - getting easy access policies to a pool of competencies - Strengthening to respond of faster ESF to emerging programme policy needs; - using common successfully tested tools - developing and assuring professional competence - speeding up the use and integration of good practice gained elsewhere - developing a common voice on the issues at stake Learning Networks making a contribution to: - strengthening the capacity to innovate - modernising and adapting institutions to new social and economic challenges - identifying and assessing issues and solutions for reforms in policy and delivery - improving the quality of governance of public policies, programmes and actions. A18

95 3. Interviews A minimum of between 6 and 10 interviews must be undertaken, including: ESF Learning Network lead (Initial stakeholder interview will be used for this purpose) Selection of ESF Learning Network Members Representative grouping according to Member State vs. Region/geographical balance and other factors Any external contractors including facilitators, evaluators, content experts, etc. Depending on the complexity of the Network and access to interviewees, more than 10 can be undertaken. 4. Sample of activities As part of the case study there will be a more detailed review of specific activities. The purpose is to contribute additional information and insight into relevance and effectiveness, and findings will feed into the overall case study template. All case studies should look at common activities and outputs such as steering groups and meetings, websites, management structures etc. These are the activities that are shared across all Learning Networks and understanding them will be intrinsic to effective reporting. However, each case study should look beyond these common/generic activities to analyse Network-specific activities. These are those activities and outputs that define the productivity and impact of the Network. For example, thematic conferences, peer reviews, policy paper production etc. Between one and three of these Network-specific activities/outputs should be analysed depending on the complexity of the Network. The more complex the Network, the more activities to be analysed. In this instance complexity refers to, for example, management structures, number of activities, number of outputs, and number of partners. Having reviewed the activities of the Network please confirm which of the activities were most relevant to the Network s work. It is important that we sample all types of activities across the case studies, for example, peer reviews, conferences, the production of manuals etc. Please make every effort to coordinate with colleagues to ensure differentiation between activities across different case studies. 5. Reporting Please use the template below for reporting. A19

96 Section 1 General Information Learning Network name: Lead partner: Network coordinator (name): Other members: Lifespan of Network (start and if appropriate, end date): Network budget: Commission support group/person: AEIDL monitoring contact: Section 2 Summary This section describes the aims and objectives of the Network as well as detailing wider, general information available. Please provide information on: Aims and objectives of the Network: Include why the theme was chosen EU policy context: Describe links to ESF and wider EU policy Activities undertaken by the Network: Outputs of the Network: Include numbers of recipients, target audience etc. Section 3 Relevance of the Network A20

97 This section includes initial information on and insight into the relevance of the Network and its activities. Please provide information on: The relevance of the Network aims to ESF objectives and wider EU policy goals: The relevance of Network activities to Network aims: How were the themes for LN activities selected? What is the added value of the LNs to ESF objectives? To what extent is the network s theme and objectives complementary to other networks and organisations? Section 4 Effectiveness of the Network This section provides an overview of the quality and effectiveness of specific activities and outputs (for example, conferences) of the LN were to the objectives of the Network. Please include information on: Achievement of Network aims: Include the effectiveness in delivering outputs and outcomes and how? Development of policies or subjects: How did the policies or projects of the ESF LN evolve? Quality of outputs: Effectiveness of processes: Include internal organisation etc. Instances of learning: Include learning took place at an individual, organisation, and policy level. Policy transfer: Describe examples of policy practices being transferred to other countries. Also highlight instances where activities have shown certain policy practices cannot be transferred to other countries. Impacts of the Network: Include the short- and long-term impacts of the LN s results Section 5 Success of LN activities A21

98 This section provides greater detail on the specific activities of the network. Reach and dissemination: Did activities and outputs reach wide audiences? What were the target groups? Lessons learned from activities: What specific success or failure factors for policies/projects did the activities discover? Networking: Did formal or informal networks result from activities? How have these been used? Section 6 Efficiency of the LN This section provides an overview of the Network s efficiency subject to available information. Please provide information on: Level of resources allocated: Include financial, time and human resources Proportionality of resources to achievements of activity: Changes to resources throughout the life of the network: Section 7 Likely sustainability This section outlines initial perspectives on the sustainability of the network. Please provide information on: Sustainability of Network activities: Sustainability of impacts and outcomes: New policies developed at any governance level due to network: Section 8 ESF Calls for proposals Details of national calls: What national calls for proposals (ESF) were inspired by the network? Include resources Call for proposal projects: What projects were launched? What were their aims and objectives? Impacts of calls: What outcomes and impacts did the projects lead to? A22

99 Section 7 Strengths and weaknesses This section is to give an overview of both the positive and negative aspects of the Network and its work. Please provide information on: The Network s key strengths: Include activities within the analysis The Network s key weaknesses: Include activities within the analysis Examples of good practice: Particularly focus on mutual learning and policy transfer Success factors underpinning learning and transfer: A23

100 Evaluation of the ESF LN Topic Guide Member State Case Studies Name Organisation Job title Interview Date Role and involvement with the Learning Network(s) 1. Please describe your role within [insert organisation]. 2. Please describe your involvement with the ESF Learning Networks and what this has entailed. - Probe on whether involvement was direct (e.g. membership of LN(s)) or more limited (e.g. attending events, receiving LN reports / outputs etc.) - Use this involvement as the basis for the questions in subsequent sections 3. More broadly, to what extent did [insert Member State / region] engage with the Learning Networks and in what ways? - Probe on whether this engagement was organised / facilitated (e.g. through the MA, relevant ministries etc.) and how co-ordinated this was Processes and relevance of the Learning Networks 4. How relevant do you feel the focus of the Learning Network(s) was to the operational and policy concerns of the following and why (ask as applicable for the LNs as a whole and / or individual LNs)? a) the ESF Managing Authority/Authorities in [insert Member State] b) relevant ministries within [insert Member State] c) other institutions e.g. NGOs, social partners within [insert Member State] d) [insert Member State] as a whole? 5. How might this relevance be improved in terms of the above contexts? 6. What were the main motivations for actors to become involved in the Learning Network(s)? - What might have encouraged more actors to get involved? 7. How straightforward was becoming involved with the Learning Network(s) and why? - How might this be improved? A24

101 8. To what extent was there a plan or strategy guiding the involvement of [insert MS / Managing Authority] in the Learning Network(s) it was involved in? - How effective or otherwise was this and why? - How might such a strategy be improved in future? 9. Were any tools or structures put in place to cascade leaning from the involvement of individuals from the Member State / Managing Authority to other relevant individuals within the MS/MA or other bodies? - How effective were these and why? - How might this aspect be improved in future? 10. How would you rate the quality of materials produced by the Learning Network(s) to support dissemination and any cascading of learning? - How might such materials be improved? 11. How far would you say the outputs and results from Learning Network activity have reached relevant stakeholders in [insert Member State] and why? - How might such outputs be more effectively disseminated in future? Outcomes from engagement with the Learning Networks 12. What would you say were the main benefits of engaging with the Learning Network(s) for: a) the ESF Managing Authority/Authorities in [insert Member State] b) particular ministries within [insert Member State] c) [insert Member State] as a whole? 13. To what extent did involvement with the Learning Network(s) lead to any operational or policy influence or change in the following contexts (ask as applicable and cover a) ESF governance, b) thematic learning, c) new/improved policies, d) new/improved (operational) practices, e) new links/networks between MS or within MS, f) other a) the ESF Managing Authority/Authorities in [insert Member State] b) particular ministries within [insert Member State] c) other institutions e.g. NGOs, social partners within [insert Member State] c) [insert Member State] as a whole? 14. (if not covered) Please outline the nature of this influence or change and what benefits this has led to in each of the above contexts. 15. To what extent would you say the Learning Network(s) have led to improved ESF transnational co-operation between Member States and why? -To what degree do you feel that this is an important benefit of the Learning Networks? 16. Were there any other outcomes from involvement with the Learning Network(s) that you would like to highlight? - What were the benefits or otherwise of this? A25

102 17. From your experience, what lessons can be drawn in terms of the most effective way for Member States and ESF Managing Authorities to engage with: a) Learning Networks? b) ESF transnational co-operation more broadly? 18. Do you have any wider recommendations about how the Learning Networks could be improved in the future to better correspond to the needs of MS? A26

103 Evaluation of the ESF LN Topic Guide Greece and Portugal Member State interviews Name Organisation Job title Interview Date Introduction 1. Please describe your role within [insert organisation]. Relevance of the Learning Networks and engagement with them 2. How relevant do you feel the focus of the Learning Network(s) was to the operational and policy concerns of the ESF in [insert Member State] and why? 3. Equally, how relevant were the learning networks to the policy concerns of [insert Member State] and why? 4. To what extent did [insert Member State] engage with the Learning Networks and in what ways? - Probe on the reasons for this level and nature of engagement (e.g. did they consider leading any networks? What encouraged engagement / non-engagement in networks?) 5. What would encourage greater engagement on the part of [insert Member State] in future in terms of: a. learning networks b. transnational co-operation more broadly? 6. Are there any barriers to such greater engagement? If so, what are these and how might they be addressed? 7. How might the Commission encourage greater engagement on the part of [insert Member State] in future? 8. To what extent was there a plan or strategy guiding the involvement of [insert MS / Managing Authority] in the Learning Network(s) it was involved in? - How effective or otherwise was this and why? - How might such a strategy be improved in future? 9. How far would you say the outputs and results from Learning Network activity have reached relevant stakeholders in and why? - How might such outputs be more effectively disseminated in future? A27

104 Outcomes from engagement with the Learning Networks (ask as applicable depending on above) 10. What were the main benefits of engaging with the Learning Network(s) for the ESF Managing Authority in [insert Member State] 11. Equally, what were the benefits for [insert Member State] as a whole? 12. Did involvement with the Learning Network(s) lead to any operational or policy influence or change in [insert Member State], either for the Managing Authority or more widely? 13. (if not covered) Please outline the nature of this influence or change and what benefits this has led to. 14. To what extent would you say the Learning Network(s) have led to improved ESF transnational co-operation between Member States and why? -To what degree do you feel that this is an important benefit of the Learning Networks? 15. Were there any other outcomes from involvement with the Learning Network(s) that you would like to highlight? - What were the benefits or otherwise of this? 16. From your experience, what lessons can be drawn in terms of the most effective way for Member States and ESF Managing Authorities to engage with: a) Learning Networks? b) ESF transnational co-operation more broadly? 17. Do you have any wider recommendations about how the Learning Networks could be improved in the future to better correspond to the needs of Member States? A28

105 Assessing the Effectiveness, Efficiency and Relevance of the ESF Learning Networks Member State case study guidance and template Aim The purpose of the case studies is to add to our understanding of the effectiveness of the Learning Networks by looking at how Member States engaged across a range of Networks and the impact of such engagement. In this way, we will seek to demonstrate tangible outcomes of Learning Networks. We also anticipate being able to draw conclusions that particularly relate to the process aspects of the Learning Networks: The most effective ways for Member States to engage with the Networks; and The ways in which effective learning and, ultimately, policy transfer/adaptation, improvement of the quality of ESF management and capacitybuilding takes place. Case study selection and allocation Confirmed case study selections and allocations are in the table below. This also briefly summarises the key features of the MS, involvement in transnational co-operation and nature of ESF governance. Note that the case studies also include two Member States where the focus of the exercise is to investigate the reasons for limited participation in the LNs under consideration. Proposed Member State Case study lead Key features (geography, size, impact of crisis ) Involvement in transnational cooperation including ESF LNs Nature of ESF governance Czech Republic xx Geographical area/ Welfare regime: Central & Eastern Europe Population size: medium Newer EU Member State (post 2004) Impact of economic crisis: medium Lead of 1 ESF LN Participation in 6 ESF LNs Strong commitment to ESF transnational cooperation (lead of LN on Transnational Cooperation) Separate ESF priority axis for transnational cooperation + eligible expenditure under all axes National ESF governance with additional urbancentric management element (Prague) United Kingdom (England and Gibraltar) xx Geographical area/ Welfare regime: Western Europe/ Anglo-Saxon Participation in 4 ESF LNs Indications of strategic approach to participation National ESF governance at the level of England and A29

106 Proposed Member State Case study lead Key features (geography, size, impact of crisis ) Involvement in transnational cooperation including ESF LNs Nature of ESF governance Population size: large in ESF LNs Gibraltar 1 st wave enlargement Member State (accession 1973) Impact of economic crisis: medium Dedicated strand for transnational cooperation (innovation and mainstreaming) in England OP; no transnational expenditure under other strands Spain xx Geographical area/ Welfare regime: Southern Europe Population size: large 3 rd wave enlargement Member State (accession 1986) Impact of economic crisis: high Participation in 9 ESF LNs No ESF LN lead role but most active Southern European Member State At national level, one of the 3 OPs is on "transnational and trans-regional cooperation"; no transnational expenditure eligible under other OPs/axes Combination of national and regional ESF governance system At regional level, 10 regions initially chose a distinct priority on transnational cooperation but many have abandoned Belgium (Flanders) xx Geographical area/ Welfare regime: Continental Europe Population size: small-medium Original EU Member State (1957) Impact of economic crisis: low Lead of 1 ESF LN Participation in 8 ESF LNs Separate ESF priority axis/measure for transnational cooperation; no transnational expenditure eligible under other measures Regional ESF governance system (Flanders level) A30

107 Method We will complete the Member State case studies through: Interviews (mainly face-to-face) of key individuals; those immediately involved in the Networks and/or instances of impact as a result of Learning Network activity; Reviewing relevant policy documents (including those suggested by interviewees) that capture the impacts in the Member State and comparing them to the relevant Network materials, in order to demonstrate linkage; Following up any claimed impacts at Member State level that have emerged from the research with Learning Networks to date; and Analysis of evidence and reporting, which will take the form of discrete written summaries of around 5-7 pages each. The approach will involve firstly identifying the most relevant initial contact or contacts (likely to be those with a remit for transnational co-operation in the relevant Managing Authority) to arrange an interview with alongside discussing and identifying other potential interviewees. These additional interviewees are likely to have a role in particular Ministries (or potentially other organisations such as social partners, regional bodies, other institutions and/or NGOs in certain cases) which have been directly or indirectly involved in the work of the Learning Networks and who may have benefited from them (influence on policies/practice, individual learning, etc.). The aim is thus to use a form of snowball sampling to identify the most relevant interviewees and to trace through the Member State level involvement with, and impacts from, the Learning Networks. A topic guide has been developed for use in the interviews (supplied separately). Given the different roles of likely interviewees, the expectation is that this will be tailored appropriately to the interviewee in question. Reporting Please use the template below (copied to a separate word document) for reporting A31

108 Member State: [Enter MS] Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks Member State case study report Section 1 Overview of engagement Outline the nature of MS / region / Managing Authority engagement with LNs (how involved, which Networks principally engaged with, engagement through MA and/or ministries, regional bodies etc.) Section 2 Relevance of the Learning Networks Outline the nature and extent of the relevance of the LNs to a) ESF MAs, b) relevant Ministries, c) Member State / region as a whole d) other institutions (e.g. social partners, NGOs etc.) Any ways LNs and their work / focus could be made more relevant? Section 3 Engagement with the LNs and key processes to support this What were the main motivations for the involvement of actors from the MS in the LNs? What might have encouraged other key actors to engage with the LNs? How straightforward was it to engage with the LNs and their work? How might this be improved? Was there a plan or strategy guiding engagement at the level of a) ESF MA, b) Member State / region? How effective? Improvements? Structures/tools to support dissemination / cascade learning. What were these (if any), how effective, improvements? A32

109 Quality of materials provided by the LNs to support dissemination and improvements? How far stakeholders were reached and reasons for this Improvements? Section 4 Outcomes and impacts What outcomes and impacts can be identified at the MS / regional level? Look to cover impacts relating to a) ESF governance, b) thematic learning, c) new/improved policies, d) new/improved (operational) practices, e) new links/networks between MS or within MS, f) other Where are such outcomes and impacts evident particular Ministries, ESF MA, other? What benefits were there at these levels and for the MS as a whole? Extent of operational and / or policy influencing and policy transfer and factors behind this Role/influence of LNs on transnational co-operation on ESF between MS Section 5 Lessons learned and possible future improvements Summarise the main improvements suggested by stakeholders / from the literature re. the above aspects Specific lessons re. the most effective way for Member States and ESF Managing Authorities to engage with Learning Networks / ESF transnational co-operation more broadly. Any other recommendations for improvements to ESF Learning Networks in the future A33

110 Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks Policy Delphi on the future role of Learning Networks and transnational co-operation Relevant findings and issues for panel members to consider 1. Introduction The aim of this paper is to help consider options for the potential future role and operation of Learning Network type activity and transnational co-operation. For the programming period, Article 10 of the ESF Regulation allows Member States to implement transnational cooperation in three different ways: On the basis of common themes proposed by the Commission and endorsed by the ESF Committee, whereby MS benefit from an EU-level platform established and operated by the Commission (Common Framework) Without central coordination (flexible approach) Through a combined approach of the above. Lessons from the current evaluation of the ESF Learning Networks ( ) will be considered within this context. To support this process, key findings drawn from the research undertaken to assess the Learning Networks are highlighted below. For each finding, a supplementary issue or implication for consideration is presented. Respondents are invited to provide their thoughts on these issues / implications in the grey boxes provided. These responses will be analysed to help inform the study in relation to the future of Learning Networks and transnational co-operation. Following this analysis, a further paper summarising responses will be produced. Respondents will then have another chance to comment on this as part of the Policy Delphi process. All responses will be treated in strict confidence and we very much appreciate all thoughts offered. 2. Research findings and issues / implications to consider Finding 1 In the ESF programing period, levels of participation of both network leads and network members within the Learning Networks were unevenly spread between Member States. Issue for consideration 1 Should promoting a more even spread of representation across Member States in respect of Learning Network participation and transnational co-operation more broadly be an aim in future? If so, how might this be achieved? Response 1: Finding 2 The extent of collaboration between Learning Networks in the period varied and, in some cases, potential gains from such collaboration were not maximised. Issue for consideration 2 How might such cross-working and collaboration best be maximised between different Learning Networks and other transnational activities in the ESF programming period? Response 2: A34

111 Finding 3 Some of those engaged by the research felt that the focus of the networks derived more from preceding initiatives, notably EQUAL, rather than thoroughly analysing what would be appropriate in light of the specific ESF objectives for the programming period. Equally some stakeholders felt that the Commission might play a stronger role at the outset of any future calls for Learning Network proposals to ensure maximum relevance to, and support, for the ESF. Issue for consideration 3 In the ESF programming period, how might the Commission best ensure that the work of Learning Networks or other transnational co-operation organised within the context of the ESF best reflects and supports its aims? How can such transnational co-operation remain flexible to changing policy needs and priorities? Response 3: Finding 4 The effectiveness of organisational processes within Learning Networks appeared to vary notably, with stakeholders reporting some as being well organised and others less so. Issue for consideration 4 Are there ways in which Learning Networks and similar transnational initiatives might be better supported and organised to ensure that activities are well organised? If so, how might this be achieved? Response 4: Finding 5 The degree of concrete outcomes in terms of policy learning and transfer appeared to be variable between the networks and in some cases was reported to have been limited. Issue for consideration 5 How might the extent of concrete policy learning and transfer be enhanced in future through Learning Network and related transnational activity? Response 5: Finding 6 There were varied views on the role and effectiveness of the Commission in supporting Learning Network and transnational co-operation in the context of the ESF. Issue for consideration 6 How might the Commission best support Learning Network type activity and transnational co-operation more broadly in the programming period? Response 6: A35

112 Finding 7 A common view was that Learning Network activity would benefit from an enhanced focus at the level of ESF projects and on sharing practical good practices. Issue for consideration 7 How might future transnational co-operation activity best incorporate and promote learning transfer at the ESF project level? Response 7: Finding 8 There were varied perspectives on which activities and outcomes Learning Networks and similar transnational activity should focus on e.g. promoting more collaboration between Managing Authorities, sharing good practice, promoting policy learning, promoting policy transfer etc. Issue for consideration 8 What should be the key focus of Learning Network and other transnational activity in the ESF programming period and why? Response 8: Finding 9 The success of dissemination activities was reported to be variable across the Learning Networks. Issue for consideration 9 How might the effective dissemination of outcomes from Learning Network type activity and transnational co-operation more broadly be encouraged in future? Response 9: Finding 10 Stakeholders expressed a wide range of views on how Learning Networks and transnational co-operation within the context of the ESF could be improved in future. Issue for consideration 10 What are the key ways in which Learning Network activity and transnational co-operation could be improved in the context of the ESF in the programming period and why? Response 10: A36

113 Annex Three: Learning Network case studies A37

114 TNC case study Section 1 General Information Learning Network name: Transnational Co-operation Network (TNC) Lead partner: Czech Republic: Ministry of labour and social affairs Network coordinator (name): Markéta Pěchoučková Other members: UK England & Gibraltar: Department for Work and Pensions, delegated to Birmingham City Council Germany: Federal Ministry of Labour and Social Affairs Greece: Managing Authority for the Equal Principles Integration in the ESF OPS and for Human Resources OP actions, Ministry of Employment and Social Protection Poland: Ministry of Regional Development Spain: Ministry of Labour and Immigration Sweden: ESF Council Slovenia: Government Office for Local Self-Government and Regional Policy Lifespan of Network (start and if appropriate, end date): 1 December 2009 and 30 November 2012 Network budget: 320,000 Commission support group/persons: Luca Pirozzi and subsequently Marie-Anne Paraskevas AEIDL monitoring contact: Katalin Kolosy A38

115 Section 2 Summary Aims and objectives of the Network: To build the capacity of ESF programme managers to successfully implement and promote transnational actions in the ESF in the programming period. The network s objectives, as outlined in its grant application, were: To stimulating learning and building capacity among ESF programme managers to ensure successful implementation and promotion of transnational actions under ESF ( ). by: Exchanging good practice and experience in designing, implementing, monitoring, disseminating and mainstreaming transnational actions; Jointly developing new tools and procedures to make transnational cooperation work; and Promoting transnational cooperation as a tool to achieve the Lisbon objectives and to improve the delivery of policy by sharing experiences and learning from other countries/regions. The theme was chosen due to the desire on behalf of Member States to ensure effective transnational working within ESF, even in the absence of a specific strand within ESF Regulations. EU policy context: The network addresses the cross-cutting theme of transnationality that has been an important dimension of structural fund policy and activities for many years. Within the current ESF Regulations ( ) there is no dedicated transnationality stream, but the importance of sharing information, enabling effective national calls for transnational working, and increasing knowledge on the governance of transnational working beneath the EU level remains. During the life of the network significant work has been done within the Commission and outside by stakeholders to develop new approaches for transnationality post This network has played a full and active role in such discussions through policy proposals and knowledge sharing. A39

116 Activities undertaken by the Network: The network undertook a wide range of activities including publications, policy development, peer review exercises, and assisting in finding partners for transnationality calls. These were entirely in line with the original application and workplan developed prior to the launch of the network. Importantly the network combined knowledge sharing (exemplified by working papers) with more outward facing activities (such as drawing up policy proposals for new ESF Regulations). This provided members with a high degree of added value from their membership and participation. The scale of activities can be argued to have been overambitious though. Whilst activities were most often completed as envisaged, network member interviewees have pointed to challenges around capacity and expertise required to complete some tasks such as working papers. As time progressed the network moved more and more to external-facing activities based on influencing the shape of ESF Regulations. Representatives presented results of work at a number of European and national conferences and seminars, for instance in Warsaw in December 2010, and in Belfast and Madrid in October This shift was based on clear dialogue within the network, and represents a sensible adaptation to external policy context. Outputs of the Network: According to the final report of the network, the work programmes led the anticipated outputs detailed in the initial planning stages. These included: Publications o o o o o Baseline study on the needs, current situation and resources on transnationality in Member State/Regions and other EU programmes. 8 issues of the TNC Newsletter, at 4-monthly intervals. The 6-10 pages newsletters contained articles reporting on transnational projects, European seminars concerning transnationality, partner fora and conferences of other Learning Networks. 29 editions of the monthly 2-page Flash News newsletter, mainly consisting of links to new online documents. Internal Steering Group News that updated partners, the European Commission and AEIDL on progress. 8 Working Papers based on real-life experience and practice from around the EU. The papers covered: Promotion of transnational cooperation Identifying and building transnational partnerships Selection of themes and actions for TNC projects A40

117 Eligibility of expenditure Designing and implementing calls for proposals Project selection Monitoring and support to transnational projects Evaluating the added value of TNC. 2 training modules. The first is on the concept of transnationality and why it should be promoted, and the second outlines how to effectively support project promoters throughout the project cycle; each module contains a PowerPoint presentation for trainees and a guide for trainers. Events o o 4-monthly steering group meetings 3 annual Peer Learning Seminars: Prague 2010, Berlin 2011, London The seminars provided the opportunity to present and explain the tools developed or identified by the Network, to share information on specific issues in workshops and to present and discuss good practices. o 5 meetings of the working group on transnationality post Website and partner search Forum Identification, drawing-up and promotion of good practice regarding the approaches and implementation of transnational co-operation at all levels (18 good practices on website) Policy position papers on: preliminary ideas for implementation of TNC post 2013; possible implementation modalities for ESF in the new programming period; and information on possible ways of TNC implementation post Nonetheless some anticipated outputs were not achieved: A41

118 A proposed working paper on Implementation of TNC in the next programming period was not written as it overlapped with activities of the established Working group on TNC post A handbook had been envisaged as a precursor to a series of working papers. However, the Commission and Learning Network representatives agreed that a series of Working papers would be more appropriate to suite the needs of Member States. Work on a Toolkit on partner searching was abandoned in the wake of no progress from either Italy or the commission. Instead network members developed the Partner Search Forum. Section 3 Relevance of the Network The relevance of the Network aims to ESF objectives and wider EU policy goals: The network enjoyed a high degree of relevance in part because of the lack of transnationality within ESF Regulations. This meant that the network s theme and activities served to fill a gap vacated by the Commission itself. As the network progressed so its relevance changed from supporting implementation to the equally relevant set of activities aimed at influencing the shape of future ESF policy. The relevance of Network activities to Network aims: The activities undertaken and resulting outputs are highly relevant to the aims of the network. Training modules present a highly transferable way to enhance capacity and knowledge on transnationality, integral to its effective implementation in future periods. The themes of papers and other outputs are also highly relevant to the thematic structure of the network s aims. How were the themes for LN activities selected? The themes for network activities were based on a comprehensive assessment of need and opportunity. First, a baseline study assessed the challenges facing Member States and their needs for support and collaborative actions. Second, experienced network members (from the first iteration of the network pre-2009) assessed the potential gains from a variety of types of activities and themes. This led to a list of potential themes and actions that were discussed between active members. To what extent is the network s theme and objectives complementary to other networks and organisations? The theme of transnationality is a cross-cutting theme that is potentially applicable to all Member States, depending on the status of transnationality within Operational Programmes. This led to a high degree of complementarity with other more thematic (not cross-cutting or governance-focused) A42

119 Learning Networks as the sharing of knowledge on transnationality extended across all 13 networks. Section 4 Effectiveness of the Network Achievement of Network aims: Highly linked to impacts and outcomes, the network s work has been argued to have led to a broad consensus among Managing Authorities and the Commission on the shape of transnationality in the ESF This means that the overarching objectives of the network appear to have been achieved. Learning and capacity building have been achieved by the joint working throughout the network s life on products especially working papers that share knowledge on central elements of transnationality. The promotion of transnationality has been supported not just by outputs but also the partner search forum which has succeeded in supporting the development of partnerships under ESF. An underlying aim of the network which became more solid as the network progressed was to secure transnationality in the ESF Regulations. Although not an explicit aim of the organisation, interviewees have nonetheless confirmed the importance of this issue. As such, the reorientation of activities towards policy development increased the effectiveness of the network and in turn allowed the achievement of additional, unplanned (at the beginning of the network s life) aims. Development of policies or subjects: An important aspect of the network s evolution was this gradual move from focusing on tools and outputs for the use of members and other Managing Authorities, to the production of policies and recommendations for the future of ESF. According to the network lead and members who were interviewed for this case study, the switch was seen as a natural progression as opposed to an unexpected move. However, the initial workplan and proposal for the network did not include significant influencing work in their scope. The on-going discussions that underpinned the functioning of the network allowed members to talk about the potential impact of negotiations for ESF Regulations, and in turn were able to agree on the need for more work on strategic issues in the latter part of the network s life. Despite this shift the network still delivered according to existing work plans and achieved key aims and outputs. Quality of outputs: The website has proved to be a very high quality, accessible tool for use beyond membership of the network. The information stored is relevant to ESF activities across Europe, and stakeholders are free to access documentation via a sign up procedure. Importantly the website contains information from other ESF Learning Networks, and so has become a central hub that compensates for the lack of European Commission website. Working papers are well written and are of particular use. These outputs provide practical and technical information, guidance and recommendations for effective working. They are written in a broadly accessible style and are freely available. Importantly the papers are highly practical, and show A43

120 examples of current practice and lessons learned. In a sense they can be seen as case studies, where examples are provided to address the theme together with comments, pros and cons. Interviewees have supported this assessment, arguing that working papers have given a detailed insight into new and better ways of working that have influenced if not entire policy change domestically, then at least implicit thinking and processes. The newsletter is useful not only for thematic updates but also as a gateway to information such as calls for proposals elsewhere. The quality is good but frequency can be argued to be too high, leading to overload from those receiving the materials. This was a concern for the network lead but measures were taken to ensure that Flash News were bite-size newsletters aimed at signposting important information as opposed to containing all info. The newsletters were again a tool for all networks, and not just TNC. Other networks were able to share news in the letters, whilst the letters were disseminated across all 13 networks. This again points to the network having taken on a central role that could have been expected from the European Commission. Effectiveness of processes: The network had an externally provided technical assistance programme that covered the preparation of the baseline study, the organisation of meetings and seminars and the running of the website. This enabled relatively clear management and effective delivery of main activities. There were more challenging issues relating to process however: The preparation and production of each deliverable was the responsibility of one partner, working on an individual basis. This meant that people were sometimes working after hours within their Managing Authorities indicating an ineffective process for output delivery. Interviewees all highlighted the challenge of producing high quality outputs when the experience, capacity and abilities of members was quite diverse. The role of the network lead was repeatedly flagged up as being vital to ensuring outputs were achieved. It appears that the scope of the work programme allied with the delegated management of tasks to require almost full time dedication of the lead partner to the network. There was also confusion over the position of RACINE in the network. Sometimes, partners and stakeholders (including the EC) assumed that the French MA was represented by RACINE. However, RACINE was only present as service provider, while the French MA was unwilling to participate in the network. This did lead to some scepticism from network participants about the validity of RACINE s membership although it must be stressed that there was no adverse impact on end outcomes. A44

121 In addition, analysis of the management structures indicates unnecessary complexity: Monitoring progress & agreeing changes to the work programme were done by the steering group: core partners + stakeholder + service provider; Operational decisions were delegated to the management committee: lead partner + stakeholder + service provider; Task forces were set up to manage actions such as each working paper and each learning seminar. A working group was established address the issue of TNC post Core members were LN participants, though meetings were open to other MSs and interested bodies. The result is a complex network of related groups within a small network with the potential for confusion and overlap. This was emphasised in interviews when network members argued that it was difficult to get a clear overview of how the network was managed, or at least how lines of governance functioned and related to each other. Instances of learning: For the most part learning appears to have been an implicit process, based on the gradual sharing of knowledge between peers as opposed to more immediate instances of revised domestic practices because of network activities. Numerous interviewees argued that the working papers were the most useful tool for learning. These were frequently disseminated by network members across their own personal and professional networks, leading to a much wider impact for the network itself. Although peer review-style activities had the potential for deep learning, these appear to have had less impact than expected due to constraints on time and inputs. Policy transfer: Good practice examples were not thoroughly progressed, nonetheless peer reviews were important arenas for potential policy transfer, through the sharing of knowledge on a specific example. Members of the network argued that working papers were the most valuable source of information that whilst not leading to direct policy transfer, did contribute to a sharing of policy ideas and approaches. Two particular working papers stand out as being of particular use for Managing Authorities in their everyday work due to their practical nature: working Paper on Monitoring and Support to TNC projects and the working Paper on Designing and Implementing Calls for Proposals. Both were also highlighted as useful for effective policy sharing by interviewees. A45

122 Representatives also presented the network results at a number of European and national conferences and seminars, for instance in Warsaw in December 2010, and in Belfast and Madrid in October These instances may have led to secondary policy transfer although there is no evidence. Impacts of the Network: Interviewees have pointed to numerous intangible impacts such as individual learning and increased understanding of practices across Europe. More tangible impacts appear when we look at the strategic level though. The culmination of the network s activities were a series of recommendations on how transnationality should be addressed in Regulations. These have been highlighted as high quality and important by network members, AEIDL and the European Commission itself. There is reported consensus among Managing Authorities and the Commission on the shape of transnationality in the ESF , indicating a high degree of impact barring external factors. The partner search forum has also had an important impact in helping ESF project promoters to find partners. This was achieved through the successful creation of the Forum. Hailed by several interviewees as a tool that should have been set up by the European Commission, this has indeed been taken up in the ESF Regulations for Section 5 Success of LN activities Reach and dissemination: The TNC network undertook significant dissemination activities that were in fact standalone work programmes. Flash reports were a high quality, wellreceived aspect of this. These reported the latest news about the Network, its partners, transnationality in the EU and new posts on the website. Importantly it extended scope beyond merely network members by including links to related initiatives, such as those undertaken by other Learning Networks. The Flashes were concise and monthly. Every four months there was an online magazine which showcased the network activities in more detail. Members provided short articles and again, the output was distributed beyond just network members. Individual network members were encouraged to circulate the magazine to their own contacts domestically. According to the final report of the network, subscribers for this comprehensive magazine included ESF programme managers, other LN members, Ad-hoc group members, Commission officials, as well as other interested parties. Lessons learned from activities: The importance of finding a transparent way to involve all Network members in Network activities. This was solved by making members leaders of activities. However, this in turn led to the challenge of differing levels of expertise and capacity which at times threatened the success of key activities such as individual working papers. The result is the learning point that rather than assuming blanket engagement, A46

123 a series of smaller dedicated smaller work groups is a more effective way to manage tasks and inputs. A clear communication strategy is vital if multiple activities are to be undertaken. Having a strategy with set points for discussions between core members allows important issues such as amendments to work plans or revision of individual activities to be addressed collaboratively. Beyond this, the communication of practical and thematic issues on a regular basis to all members was said to be important for ensuring continued buy in. An important lesson was that the network website need not be a closed, inward looking platform. The true value of the network s website is as an outward facing use ( to showcase and promote other Networks, i.e. have links and shortcuts. This provides one common place where people can access information regarding all Networks. It is also a forum to share cross Network results and achievements. Section 6 Efficiency of the LN The network proved itself to be highly efficient when the financial investment is weighed against the activities and outputs. The amount of budget is proportionate to the number of work programmes and activities undertaken. Indeed it is apparent that a larger budget would have made the ambitious set of activities more manageable. Significant staff time was needed to help members finalise outputs and to manage what was a large workplan and set of activities. Although financial efficiency was achieved, it is clear that organisational and operational efficiency would have been heightened with increased resource and increased dedicated staff time. Section 7 Likely sustainability All products and tools developed by the Network are made available on-line, and the majority of these are planned to be made available to programme managers in the next programming period especially the Working papers, training tools and implementation modalities for transnational cooperation in programming period. Importantly, the partner search forum has helped ESF project promoters to find partners, and in turn increased the effectiveness of their projects. The success of this tool may lead to on-going sustainability. Network members decided to continue some of the Networks activities, i.e. not those already fully finished, and submitted a grant proposal to the EC under the call for proposal VP/2012/005. This proposal was successful and from 1st of March 2013 a Learning Network on ESF Transnational Cooperation will start its two years running. Section 8 ESF Calls for proposals A47

124 Details of national calls: There are no reported national calls for proposals coming from the network s activities, or directly inspired by them. This is because the network s theme is a cross-cutting governance theme as opposed to a more content-related theme. However, several interviewees from Managing Authorities did state that the way in which they set up and manage transnational calls has been influenced by learning from within the network. A common argument was that certain governance procedures were shared, allowing participants to slightly adapt their own ways of working. In addition the partner search forum was described as a useful tool which helped facilitate effective transnational calls and subsequent projects by linking prospective project participants together in a clear and accessible format. Section 9 Strengths and weaknesses The Network s key strengths: Strengths include a committed core membership that has readily taken responsibility for the development of the network and delivery of outputs. The fact that members took responsibility for tasks is to an extent a positive, allowing distribution of work. However, the uneven capacities and abilities of members did mean that this approach did not preclude the need for intervention from the network lead. Importantly the network retained a high degree of relevance throughout its life. First, its activities were based on a clear needs analysis. Second, when EU policy discussions moved to the new programming period, the network was able to refocus on influencing the Regulations through collaboratively developing policy positions. The Network s key weaknesses: The final report for the network outlines a series of challenges and weaknesses. These include: Problems with inadequate access to English language skills in partner organisations which hampered the development of working papers; Lack of budget in some partners post-crisis to attend, with the network budget unable to fill the gap resulting in a two-speed network; The network s communication strategy was slow in being developed, and as such significant time was taken to reach decision-makers who were not already supportive of transnationality. Envisaged good practice examples have not been very convincingly carried through or presented. A48

125 Examples of good practice: Working papers and their themes were tailored to needs identified through the baseline study and its subsequent analysis; Use of external contractors to support the management and delivery of core elements such as meetings but without using such contractors so much that they do all the work ; Concise electronic newsletters as opposed to longer papers which may not be read as easily; Provision of non-network developed documents such as the Poland-produced handbook for project promoters; and The use of task forces to drive forward specific aspects of the network s work as opposed to relying on ad-hoc capacity. Success factors underpinning learning and transfer: The development of a suite of tools, rather than simply one or two methods for sharing information has been vital. The website and associated electronic information sharing and knowledge platforms allows on-going learning and development beyond the confines of meetings. Perhaps most importantly however, the leadership of the network has driven activities forward, ensuring outputs were delivered. The match funding through staff time allowed permanent, dedicated resource that was used to plan activities, manage technical aspects of the network, and finalise outputs. Desk research and interviews indicate that without this soft resource it would have been very difficult for the network to achieve its aims. A49

126 Annex: Intervention logic for the TNC network Rationale Aims and objectives Inputs Activities Results Impacts Transnationality is an important aspect of ESF activities, which can be strengthened through the sharing of knowledge and good practice. In the absence of prescribed transnationality in ESF Regulations, a learning network can perform an important role in ensuring collaboration and coordination between Member States. Overall Objective Stimulating learning and building capacity among ESF programmes managers to ensure successful implementation and promotion of transnational actions under ESF ( ). Specific Objectives Exchanging good practice and experience in designing, implementing, monitoring, disseminating and mainstreaming transnational actions; Jointly developing new tools and procedures to make transnational cooperation work; Promoting transnational cooperation as a tool to achieve the Lisbon objectives and to improve the delivery of policy by sharing experiences and learning from other countries/ regions. Financial Inputs: Financial support through the ESF Technical Assistance operational budget ( 320,000) Additional financial support through the network member co-financnig ( 34,000 from Czech Republic and 15,000 from Germany) Human resource and in-kind inputs Technical Secretariat (one Czech Republic official ) Network Member s participation in network meetings Contributions by European Commission through staff time Main activities: - Mapping exercise to understand needs and wishes of Member states - Working Papers on transnationality for programme managers (9) - Peer learning seminars (3); - Newsletters (8) and Flash news (29) - Good practice identification and presentation -Development of training tools on transnationality - Preparation and presentation of papers on the future of transnationality and ESF Programme level: - Contributed to securing transnationality as a core element in ESF Regulations - Partner search mechanism adopted by Commission as tool within Regulations Individual members: - helping each other in solving problems at operational and strategic levels - applying common, tested, working methods - gaining professional skills and thematic knowledge Participating organisations: - Increased understanding of how transnationality is applied in other Member States allowing progressive changes in working practices - Improved access to good practice that allows modification of domestic governance procedures - increased ability to find partners for transnational activities - developing and assuring professional competence - speeding up the use and integration of good practice gained elsewhere - Input into a coordinated and collaborative voice on the issues surrounding the future of ESF Learning Network making a contribution to: - strengthening the capacity of Managing Authorities to engage in transnationality; - identifying and assessing good practices in the field of transnationality (governance); - strengthened transnationality in the ESF Regulations; and - facilitating transnational partnership working through partner search tools A50

127 BFSE Case Study Section 1 General Information Learning Network name: The Better Future for the Social Economy (BFSE) Network Lead partner: Poland: Ministry of Regional Development, Department for ESF Management Network coordinator (name): Network Coordinator: Malgorzata Lublinska (Department for ESF Management, Polish Ministry of Regional Development) Network Facilitator: Dorotea Daniele (Diesis Coop) (since July 2011) Other members: Belgium Flanders: ESF Agency Czech Republic: Ministry of Labour and Social Affairs, ESF Managing Authority Finland: Ministry of Employment and Economy, ESF Managing Authority Finland: Tampere Region Cooperative Centre Italy: ESF Managing Authority - Lombardy Region Italy: Consorzio Sociale Light - Lombardy Region Sweden: ESF Council, ESF Managing Authority Sweden: Tillvaxtverket (Agency for Economic and Regional Growth) Sweden: Coompanion UK: Office for Civil Society, Cabinet Office UK: Birmingham City Council, ESF Innovation, Transnationality and Mainstreaming Unit - England and Gibraltar A51

128 In addition, the Network (as well as individual Members) also appointed external European and national experts to assist with the baseline study, seminars, the development of tools and writing the final publication. These experts included among others: Patrick de Bucquois, Dorotea Daniele, Francesco de Rosa, Keith Richardson, Christophe Guene, Samuel Barco, Sven Bartilsson, Bosse Olsson, Daniella Gatti, Jeremy Nichols, Gordon Hahn, Gianluca Pastorelli. Lifespan of Network (start and if appropriate, end date): October 2009 June 2012 Network budget: 483,000 (actual expenditure: approximately 80%). Original grant request was for 746,967. Commission support group/persons: Commission contact person (DG EMPL, ESF Directorate) changed 3 times in 3 years: Rafal Zawada, Lucas Pirozzi, then Anne-Marie Paraskevas Links in DG EMPL: o Sue Bird (Social policies, Innovation and Governance expert on CSR and author of Buying Social ) o Concetta Cultrera (Unit D2 Social Inclusion and Poverty) Links in DG ENT: o Representatives from Unit E3 (Crafts, small businesses, co-operatives and mutuals) attended some of the initial planning meetings o Apostolos Ioakimidis (D1 - Entrepreneurship and Social Economy) o Kiril Nikolov (C1 - Internal Market and its International Dimension, Policy Officer on market surveillance for products) Some links to representatives of DG COMP and DG MARKT AEIDL monitoring contact: Toby Johnson Section 2 Summary Aims and objectives of the Network: The Network theme was chosen because of the potential of the social economy not only in economic terms, but also in creating social added value. Social enterprises complement the welfare function of the state, enable active integration, create new jobs for disadvantaged groups, address the challenges of modern society and counterbalance commercial enterprises. However, the development of the social economy sector varies greatly: in A52

129 some Member States, the sector is very strong with a lot of initiatives exist to support social enterprises, whereas - in others - the sector does not have a long tradition and its importance is not sufficiently recognised. As such, the members felt that it was crucial to establish a network which would enable the exchange of experiences and points of view at European level between these different countries. The Network builds on and continues the work of the European Thematic Group on Enterprise for All (ETG 2) under the EQUAL programme, which concluded in As stated in the grant application 10, the overall objective of the Network was to enable learning from and with each other about different aspects of social economy and social enterprises, as well as to use the experience gained within the Network in the activities undertaken at ESF Operational Programme level. The specific objectives were to allow Network members the possibility to: Share information, participate in the discussion at wider level, exchange experiences; Implement good practices in the ESF OPs; Work out a new instrument on measuring social added value; Increase the effectiveness, quality and transparency of social inclusion measures; Increase their knowledge and skills; Enhance their professional networks; and Influence public authorities at European, national, regional and local level 11. These original objectives were streamlined and operationalized at a seminar in Warsaw in March The objectives were set out in a logical framework 12 in order to guide the work of the Network across the five thematic strands. The common objectives set out were as follows: Impact objective: Creation of an enabling environment for the social economy; Result objective: Delivering common, ready to use and user friendly way of coping with social economy; Activity objectives: Analysing legal interference between practice and community law, existing tools, practices, solutions, fund allocation, social franchising; Drawing up recommendations; Creating a set of solutions; Testing and/or refining the set of solutions; Determining the way the model may be financed; Engaging other stakeholders; Redesigning and delivering the final set of solutions; Building the communication strategy and mainstreaming results to policy level. Each strand also has its own specific objectives. EU policy context: The work of the Network links to a number of recent and ongoing EU policy initiatives. The ESF promotes entrepreneurship through financial and business support services. Other specific funding mechanisms at EU level include: the European PROGRESS Microfinance Facility (which helps stimulate self-employment and the creation of micro-enterprises), the ERDF (which supports entrepreneurship via INTERREG 10 Grant application, Network for Better Future of Social Economy, Annex D1, p.2 11 The Network for Better Future of Social Economy, Final Report, 2012, p.4 12 AEIDL, Learning Network of Better Future of Social Economy: Quarterly Report no.11 June 2012, Annex 3: Logframe and work programme, p.40 A53

130 projects such as ENSPIRE.EU) and the JASMINE and JEREMIE initiatives in the field of micro-credit and credit for SMEs. The new PSCI (Programme for Social Change and Innovation) programme will set aside 95.5 million for social enterprises. The Network was also implemented at the same time as the Commission s Social Business Initiative, which includes an initiative to set up a new European Social Entrepreneurship Fund label, so that investors can recognise funds that invest in social business. In addition, the work of the Network has linked into the reform of European public procurement regulations and work undertaken by DG EMPL on socially responsible public procurement. Activities undertaken by the Network: The network was divided into five strands, each led by one of the partners: Community law and social services of general interest Flanders; Measuring social added value Lombardy Region; Socially responsible public procurement and public social partnership Poland; Social franchising Sweden; Financial instruments and mechanisms of fund allocation to social economy Czech Republic. During the preparatory phase of the Network, a baseline study was conducted by Diesis Coop, covering all five strands. The work programme for the Network was then established, through 3 meetings of the core members of the Network (the Founding Fathers ) and then a two-day workshop in Warsaw in March 2010 to build the logical framework (facilitated by PCM experts). One key aim was to ensure that the activities and timings for each of the five strands were consistent through a work programme consisting of a series of common steps for each strand. The key activities undertaken were as follows: Conducting a survey among partner countries and regions, as well as willing ESF MAs, to identify examples of good practice. Four strands validated the good practices through a sort of peer review process; Presenting preliminary conclusions and recommendations for further work at a learning seminar in Warsaw (March 2011); Developing solutions (tools/products/recommendations) and testing/refining them; Producing good practice guidance and policy recommendations aimed mostly at ESF Managing Authorities; Disseminating/mainstreaming the final results via the BFSE website, the publication (A Better Future) and the final conference. Outputs of the Network: The key outputs of the Network were: The final publication A Better Future, targeted mainly at ESF Managing Authorities at national and regional level. This 100-page publication consists of 2 sections: part I addresses the role which the social economy could play in relation to the 18 investment priorities A54

131 for the ESF , while part II presents thematic recommendations (per strand). It is available to download on the Network website ( and was translated into Polish; The Network website: European tool for the evaluation of the socio-economic value of social enterprises : an innovative 46-page handbook measuring the social added value of social enterprises, available in English and in Italian. The aim was to provide public authorities (as well as other potential public/private investors) with a practical tool for making decisions concerning funding allocation; Recommendations for SRPP (socially responsible public procurement) and PSP (public social partnership) in the light of social economy development (20 pages); Report on the application of SRPP and PSP in support of social economy development (50 pages); Examples of initiatives with the social economy sector significant from the perspective of sustainable public procurement (14 pages) (all available in Polish and English); A report and recommendations on challenges and solutions in terms of socially responsible public procurement and social public partnership, as well as contributions to the consultation process for the European Commission s new public procurement regulations; An Overview of attitudes and financial instruments for social economy in BFSE countries and case studies of selected practice of support (68 pages); Recommendations for the implementation of financial instruments (33 pages); Reports on Social Franchising financing its growth (13 pages) and Social Franchising obtaining higher returns from investments for jobs in social enterprises (30 pages); a directory of social franchisors and case studies of a number of social franchises; A Report on Community law and SSGIs (social services of general interest) in the light of social inclusion (6 pages); ESF and state aids checklist ; The baseline study: survey of the state of the social economy in Europe. The key events were: Learning Seminars: Warsaw (15-16 March 2011), Milan (5-6 December 2011) Final Learning Seminar (stakeholder forum): Warsaw, 2-3 April participants Final conference (targeted at ESF policy-makers): Brussels, 26 June participants Strand-specific events included: Socially responsible public procurement: peer review in Krakow (September 2011) Social franchising: workshop in Genoa (April 2011), conference in London (October 2011) Financial instruments: peer reviews in Prague (November 2011, March 2012) Section 3 Relevance of the Network The relevance of the Network aims to ESF objectives and wider EU policy goals: The aims of the Network clearly have relevant links to ESF objectives, both under the current and future programming periods, at both EU level and in Member States. By creating an enabling environment for the A55

132 social economy to flourish, the Network contributes to ESF goals around entrepreneurship, promotion of employment and social inclusion. It also contributes to wider EU policy goals linked to the promotion of entrepreneurship for all, corporate social responsibility, social business (Social Business Initiative) and socially responsible public procurement. The relevance of Network activities to Network aims: The activities correspond well to both the general and specific objectives set out by the Network in terms of sharing information, exchanging good practices, enabling learning on the social economy and increasing knowledge and skills, including by the development of a tool on measuring social added value. These activities are well designed to improve ESF policy-making and guide content in ESF OPs, as well as to influence public authorities and decision-making. The final publication captures the outputs of these activities and corresponds to the overall aims set out by presenting the findings from the strands, and aiming to influence future ESF Operational Programmes by highlighting the connections between the social economy and ESF investment priorities for How were the themes for LN activities selected? The themes of the five strands were selected by the core Network partners ( Founding Fathers ) on the basis of gaps in knowledge identified in particular fields during the work of the European Thematic Group on Enterprise for All (ETG 2) under the EQUAL programme. The types of activities to implement were determined through a process involving different steps: the experience of the previous ETG; a series of preparatory/exploratory meetings between the core partners; the baseline analysis; and finally the two-day PCM (project cycle management) workshop in March 2010 in Warsaw where the logframe and common work programme for all strands were drawn up. To what extent are the Network s theme and objectives complementary to other networks and organisations? In terms of other ESF Learning Networks, the work of the BFSE Network was most closely akin to that of COPIE (the Community of Practice on Inclusive Entrepreneurship) and the Network on Youth Employment, which included a focus on youth entrepreneurship: however, only very limited attempts were made by the BFSE coordinators to involve these Networks in the work of BFSE (simply invitations to conferences). The Network successfully reached out to and involved a number of complementary European networks and organisations in the field including REVES (the European Network of Cities and Regions for the Social Economy), DIESIS (European Research and Development Service for the Social Economy), and Social Economy Europe. In different strands and Member States, further contacts were established. In Poland, the Network Coordinator was for example invited to speak every year at the event of the National Network of Social Economy. For the socially responsible public procurement strand, involved stakeholders included the Scrutiny Association of over Social Cooperatives, the National Office of Public Procurement, SKES (permanent group for the social economy) and individuals in regional social policy agencies. Section 4 Effectiveness of the Network Achievement of Network aims: The Network has overall been effective in achieving its aims, particularly in terms of building capacity among A56

133 Managing Authorities to design and implement ESF programmes in the field of social economy. Via peer reviews and surveys, it has enabled the transfer of expertise and good practice between countries notably within the strands of financial instruments and social franchising. New tools have been developed, particularly the tool to measure social added value, although there is little evidence to date that it is being widely used. It has enhanced the confidence and professional networks of the ESF MAs in dealing directly with representatives and organisations from the social economy sector, to such an extent that some of these organisations are now members of the successor network. It has taken a strategic approach to influencing the contents of ESF programmes for by tailoring its final product to the ESF investment priorities and highlighting the way in which a focus on the social economy can alleviate blockages to delivering EU 2020 targets. It has had some success in reaching out to other stakeholders/networks, participating for example in in the Active Europe conference in September 2011, and working with the REVES network. It is not however clear to what extent its efforts at dissemination/communication have been successful beyond the more captive audience of the ESF MAs and the social economy organisations who have been directly involved. The specific aims of most strands were globally met, with the exception of the strand on social services of general interest and state aid which delivered less than planned and had very limited transnational exchange of experience, due to overstretched capacity at the lead ESF MA (Belgium-Flanders). Development of policies or subjects: The main aims of the Network and its strands were established during the preparatory meetings, building on the experience from the previous EQUAL European Thematic Group, and did not evolve substantially over time. The results of the baseline analysis allowed fine-tuning, but did not alter the key goals. In order to maintain consistency and learning across the five strands, a common programme of activities and steps was established at the PCM workshop in March The Network was not significantly affected by the onset of the economic crisis, apart from the need for the Network to fund travel for some national representatives to participate in events. Network activities were however delayed at the start by the reduction to the budget requested, as well as a longer time taken to carry out the surveys on each strand. Quality of outputs: The main positive outcomes of BFSE was the achievement in the aim of capacity-building among ESF MAs, and demonstrating the role which social economy measures could play in the future ESF ; as such, the quality and nature of the outputs was appropriate to achieving these outcomes. As individual tools, events and publications, the outputs were not cutting-edge in terms of state-of-the-art tools or research in the field, but were clearly useful in achieving the key aims of the Network. The final publication is of a good quality, with useful policy context, strategic demonstration of how the social economy is relevant to the upcoming ESF investment priorities and recommendations from each strand. The tool developed on measuring the social added value of social enterprises is an innovative and original tool; the checklist and guidance on state aid are helpful tools for public procurement officials. The peer reviews provided a good opportunity to identify, share and review good practice, while the final dissemination events allowed the findings to be communicated to a wider, if still relatively restricted, audience. The main weakness in terms of quality of outputs was the Network website which served simply as an (incomplete) repository of information, which could have played a much bigger role in disseminating the work of the Network to a wider audience. A57

134 Effectiveness of processes: The work of the Network got off to a slow start due to the reduction in the grant received ( 483,000 as opposed to the requested amount of 747,000) which required the adjustment of the work programme and budget. After the PCM workshop on March 2010, a uniform structure was imposed on the five strands, which simplified management and improved the accountability of the partners activities. After an initial phase which was hesitant and overly process-driven, there was a change from 2011 when all the Network actors had grown into their roles and a Network Facilitator was finally appointed. After this point, the delivery of the activities generally worked well, although some weaknesses in processes subsisted. In particular, the steering group did not meet often enough (due to difficulties in finding common dates and the differing internal rules of the participating national ministries about travel) to be an effective decision-making entity. The Network found it difficult to find EU-level social economy experts in social economy, as opposed to those with purely national expertise, which slowed some of the work. The ESF MAs in charge of the strands found it difficult to dedicate sufficient time to the Network activities, and Belgium Flanders in particular had too little capacity to carry out all the work foreseen on their strand, in particular at transnational level. The Network Coordinator would have welcomed much more guidance/support in terms of processes from the European Commission; while the AEIDL expert provided very valuable support and expertise, his role was only supposed to be to monitor the Network. There was little attempt to link to other relevant ESF Learning Networks, such as COPIE. No evaluation of BSFE was undertaken, which could also have facilitated ongoing learning throughout the life of the Network. The division of the Network into five strands also had positive and negative elements: on the positive side, it shared the workload, forced other ESF MAs to take more responsibility and allowed more in-depth work on the five themes; however, on the negative side, there was insufficient coordination between strands (one interviewee described them as 5 different projects ) and hence a lack of an overall Network esprit de corps, as well as exposing one strand where the leader had insufficient capacity (Flanders). Processes were also challenged by the different levels of knowledge/development of the social economy in different Member States - the social economy in the Czech Republic is for example much less developed than in Italy or the UK meaning that some partners came to learn, whereas others came to share or even build new tools. Finally, some members were also more passive than others, and also (i.e. Finland) could not participate in all meetings. Overall, the nature of participation, in which Network member organisations are represented by one individual, created an over-reliance on the availability, knowledge and dynamism of those individuals, necessarily leading to weaknesses and differing levels of engagement. Instances of learning: As set out above, the main impact of the Network has been the learning and capacity building, in particular of ESF policymakers and Managing Authorities. The activities of the Network have considerably enhanced the knowledge, expertise and confidence of the participating ESF MAs in the field of the social economy/social entrepreneurship. As such, the learning impact for participating individuals and their organisations has been important, particularly in terms of: knowledge of how the social economy works in practice, how to overcome specific regulatory barriers (state aid, public procurement), understanding of the necessity to adopt a holistic approach, and knowledge of how to use European funding (ESF and ERDF) to support the social economy. In Poland for example, the participation of individuals from the Ministry of Employment meant that they gained a much greater understanding of the social impacts and importance of this economic sector, and are now involved on the work on the social economy (whereas it had previously been led mainly by the Ministry of Regional Development). There have however also been instances of policy learning, both at national and EU level. At EU level, staff from DGs MARKT, ENTR and EMPL dealing with the Social Business Initiative (SBI) attended certain events, which allowed a broader perspective to be taken into account in the SBI, A58

135 considering other areas than those originally foreseen such as accessing public markets, impact measurement and social franchising. As a result, the SBI is now taking up some of these issues, such as impact measurement. The results of the Network also fed in, though in a more minor way, to ongoing reform of European public procurement regulations. Furthermore, the Network s work on measuring social added value fed into the Commission s work on cost-benefit analysis of socially responsible public procurement, even though this work was later abandoned. At national level too there were instances of policy learning. Methods were transferred between countries (i.e. Italy to Poland), and there was significant bilateral learning from Poland to the Czech Republic. Within Poland, the National Strategy for the Development of the Social Economy included examples of good practice taken from BFSE. The Czech Republic intended to introduce new financial instruments as a result of their participation in the Network. The social economy experts selected at national level also contributed greatly to the learning of national organisations and networks, as they were able to translate the EU-wide findings into the national context. The most important policy learning at national level is however in terms of using ESF (and ERDF) to support the social economy in the next programming period, through examples of good practice and greater understanding of its policy fit. Policy transfer: As discussed above, there were several examples of policy learning, but less evidence of wholesale policy transfer. One instance was the transfer of methods from Italy to Poland. Interviewees highlighted that, due to differing national rules and regulations, it was not possible to transfer good practices in their entirety from one Member State to another, but rather to look at adaptations which could fit the national context. Impacts of the Network: The main short-term impact of the Network highlighted by interviewees and in reports is the new cooperation at national level between stakeholders involved in the social economy, both at EU and national level. Poland in particular highlighted the new collaboration between Polish organisations working in the field of social franchising, and the possibility to measure social added value. In Sweden, as a result of the BFSE work, the ESF Council has accepted a project proposal on social franchising which has allowed the development of further work on this issue. A broader perspective has also been introduced to the Social Business Initiative, for example on impact measurement, through the influence of the Network. The work of BFSE has also led to the funding of the new ESF Learning Network on social economy (Social Enterprise Network SEN) which will allow further work in this field at EU level. In the long-term, there is potential for a significant impact through the integration of possibilities for supporting the social economy within the new ESF OPs, allowing organisations to develop activities recognised as important and producing added value. Section 5 Success of LN activities Reach and dissemination: The results of the Network in terms of reach and dissemination were mixed. While a dissemination and communication plan was successfully drawn up, not all the targets and activities set out were achieved. On the positive side, the Network did have some success in reaching out to relevant stakeholders, including relevant DGs (ENTR, EMPL, MARKT), EU and national social economy experts (REVES, Diesis Coop) and relevant national networks and actors in some Member States and regions (such as Poland, Lombardy and Sweden). Representatives from the A59

136 Network were invited to present at the Active Europe conference. Some events, such as the Social Franchising Conference in London in 2011, were successful in bringing together the right actors to discuss ideas and disseminate findings. The final publication is of a good quality, and was translated into Polish to aid dissemination in Poland. In addition, Network members were asked to act as ambassadors of BFSE, attending events on its behalf: Lombardy region for example attended an event in Berlin on the social economy in this capacity. However, there were also some shortcomings. The BFSE website was a lost opportunity for dissemination of results to a wider audience, serving simply as a basic repository of information, with limited animation and content. Although the final events were generally considered to be of good quality, participation by non-partner ESF MAs was rather limited (only 12 of the 117 ESF MAs participated, many of which were Italian regions), which was disappointing since ESF decision-makers were the main target group. No evidence was found of dissemination or usage of the tools developed to date. Lessons learned from activities: The activities allowed in particular the identification and analysis of good practices, and a greater understanding of the social economy by the participating ESF MAs. Networking: A key outcome of the work of BFSE was the successful application for a follow-up ESF Learning Network: the Social Entrepreneurship Network (SEN). SEN involves 11 ESF MAs (more than BFSE), but has also innovated by involving EU representative bodies of social enterprises (such as REVES) as partners, on an equal footing with ESF Managing Authorities. This new Learning Network will allow the work to develop further by focusing on five key elements of a comprehensive support environment for social enterprises including outreach, start-up, finance, consolidation and governance. In Italy, the Network led to exchanges between Italian regions on social enterprise issues and, in Poland, links were made to NGOs working in the social economy sector. Section 6 Efficiency of the LN Level of resources allocated: The grant allocated to BFSE by the European Commission was 483,000. The original grant request was for 746,967. Significant time and human resources were input from ESF Managing Authorities, in particular Poland and the other strand leaders. A large number of content experts were also paid to provide their expertise at different points of the project, in particular: the baseline analysis, peer reviews, network facilitation role, and writing the final publication. Proportionality of resources to achievements of activity: Approximately 80% of the budget was used overall. Given the range of activities implemented (conferences, meetings, peer reviews) and outputs delivered, these resources seem globally appropriate. Changes to resources throughout the life of the network: Due to the reduction in the original budget, the work programme and planned expenditure had to be reworked in the preparatory phase. It was generally considered that this refocusing of activities served to streamline the work of the Network and make it more manageable. The engagement of a Network Facilitator in June 2011 also gave an important boost to human resources available, A60

137 allowing the Network to progress more efficiently, and for ESF MAs to focus on their strand work. Section 7 Likely sustainability Sustainability of Network activities: On the basis of the results achieved by the BFSE Network, the partners applied for more funding under the new call for proposals (VP/2012/0005) to further develop their work. Their proposal for a Social Entrepreneurship Network (SEN), with a budget of o 515,000 for the period was submitted and approved. The involvement, in this new call, of networks such as REVES (with equal partner status) may also ensure that some of the activities are continued by these networks after the end of SEN s lifespan. The new website will also be a much better basis on which to sustain activity beyond the ESF funding period. The national thematic group established in Poland may also ensure that the activities continue. Sustainability of impacts and outcomes: As noted above, evidence points to long-term impacts on the development of support for the social economy, particularly through the inclusion of social economy priorities in the upcoming ESF OPs, thus opening the door to funding opportunities. The tool on measuring social added value is useful, and has potential to be used in an ongoing manner, but will depend on uptake (which is not in evidence yet). New policies developed at any governance level: There have been policy impacts and learning from the Network s activities on policies at EU and national level, as discussed in section 4 above. At EU level, its work has had an impact in particular on the Social Business Initiative, and fed into work on socially responsible public procurement. At national level, its work has fed into the National Strategy on the Social Economy in Poland. Section 8 ESF Calls for proposals Details of national calls: No national calls for proposal were inspired by the Network. Call for proposal projects: In Sweden, a project focusing on social franchising was funded under an ESF call for proposals, which has allowed further development of work on this theme. Impacts of calls: The project in Sweden has allowed national actors to work further on the theme of social franchising. Section 9 Strengths and weaknesses The Network s key strengths: Strengths of the Network included: the peer reviews which allowed the identification and review of good practice from A61

138 across the EU; certain events such as the social franchising event in London; the involvement of national and (in the end) European experts and networks on social enterprise; the creation of a national thematic group in Poland to complement the European work; the tailoring of the final publication (A Better Future) to the upcoming ESF investment priorities; the development of an innovative and useful tool on measuring social added value. The Network s key weaknesses: Weaknesses of the Network included: the delay in starting up the Network activities; the slowness in appointing a Network facilitator and in contracting experts; the lack of any evaluation (formative or summative); the lower level of (particularly transnational) activity on the strand on social services of general interest and state aid; the lack of collaboration with other ESF Learning Networks; the weakness of the steering group as a decision-making entity; the limited number of key target groups (ESF MAs) who attended the final event; the poor quality and level of usage of the Network website; the final product, although of good quality, is more a review of how things are than how things should be.. Examples of good practice: Good practices included: the development of a coordinated work programme across the five strands using a logframe; the leading of strands by ESF MAs which enhanced their own learning; the use of a survey and peer review methodology to identify and analyse good practice across themes; the social franchising event in London for bringing together key actors in this field from across the EU; the tailoring of the final publication (A Better Future) to the upcoming ESF investment priorities in order to increase the relevance of recommendations; the tool on measuring the social added value of social enterprises. Success factors underpinning learning and transfer: Real engagement and lead on activities by ESF Managing Authorities, providing active rather than passive learning opportunities; tailoring outputs to ensure their relevance to the key target audience (i.e. in the final publication, providing examples of good practices in the fields of the upcoming ESF investment priorities); peer review methodology which allows the identification of good practice, as well as a review of its key features with a view to transferability issues; inputs by high-level European and national thematic experts in order to enhance learning and ensure the relevance of outputs; clear political will of some Member States (i.e. Poland, Sweden, Czech Republic) to be involved and to develop the social economy. A62

139 Annex: BFSE intervention logic Rationale Aims and objectives Inputs Activities Results Impacts The social economy has strong potential, not only in economic terms, but also in creating social added value. Social enterprises complement the welfare function of the state, enable active integration, create new jobs for disadvantaged groups and counterbalance commercial enterprises. As such, its contribution is highly relevant to ESF objectives. However, the development of the social economy sector varies greatly: in some Member States, the sector is very strong with a lot of initiatives exist to support social enterprises, whereas - in others - the sector does not have a long tradition and its importance is not sufficiently recognised. As such, the Network was established to enable the exchange of experiences, knowledge and points of view at European level between different Member States in order to increase visibility of the social economy and promote greater ESF involvement in its support. Support the objective of: Enabling learning from and with each other about different aspects of social economy and social enterprises, as well as to use the experience gained within the Network in the activities undertaken at ESF Operational Programme level. Through: - Sharing information, participating in the discussion at wider level, exchanging experiences; - Implementing good practices in the ESF OPs; - Working out a new instrument on measuring social added value; - Increasing the effectiveness, quality and transparency of social inclusion measures; - Increasing knowledge and skills of Network members; - Enhancng their professional networks; - Influencing public authorities at European, national, regional and local level Financial inputs Financial support from the European Commssion (ESF technical assistance) via the call for proposals for ESF Learning Networks of 483,000 In-kind inputs - Technical Secretariat of the Lead Member State (Polish ESF MA); - Travel, subsistence and time for participation in Network meetings and activities by members; - Engagement of national experts in social economy by ESF Managing Authorities; - Participation in events and meetings by staff from DG EMPL, DG MARKT, DG ENT, as well as expert networks and social economy NGOs. The network was divided into 5 strands, each led by one of the partners: - Community law and social services of general interest; - Measuring social added value; - Socially responsible public procurement and public social partnership; - Social franchising; - Financial instruments. Each strand undertook a common set of activities: - Conducting a survey to identify examples of good practice; - Presenting preliminary conclusions and recommendations for further work at a learning seminar; - Developing solutions (tools/products/ recommendations) and testing/refining them; - Producing good practice guidance and policy recommendations; - Disseminating/ mainstreaming the final results via the BFSE website, the publication (A Better Future) and the final conference. Programme level - Common understanding of the social economy and its role in ESF; - Influence on future OPs: concrete examples of how the social economy is linked to upcoming ESF investment priorities; - Recommendations targeted at ESF policymakers. Individual members - Enhancing confidence in dealing directly with representatives and organisations from the social economy sector; - Developing professional networks in the field of the social economy, and access to experts at national and EU level; - Knowledge of how the social economy works in practice; - Knowledge of how to use European funding (ESF and ERDF) to support the social economy Participating organisations - Keeping up with developments on the social economy across Europe; - Gaining easy access to a pool of competences and good practices to respond faster to policy needs; - Using common successfully tested tools and practices; - Developing a common voice on social economy. The Learning Networks has made a contribution to: - Strengthening the capacity to innovate in terms of finding new solutions via the social economy to problems addressed by the ESF ; - identifying and assessing issues and good practices improving social economy and ESF policies; - Influencing wider EU (and national) policy developments (i.e. Social Business Initiative, public procurement regulations) by introducing a broader perspective; - Providing the potential to develop and modernise social economy organisations through increased and more focused future ESF funding. A63

140 COP RBM case study Section 1 General Information Learning Network name: Community of Practice on Results Based Management (COP RBM) Lead partner: Louis Vervloet, Director of the Flemish ESF Agency Network coordinator (name): Benedict Wauters, deputy director Flemish ESF Other members: As of Dec. 2012: 121 registered people, 78 are indicated as Premium members. The latter are delegates from partner organisations. Countries and institutions include: Czech Republic, Ministries of Labour and Social Affairs and Regional Development Greece, Special Service Health & Social Solidarity Italian Region of Molise Italian Region of Piedmont Italy, Tecnostruttura Lithuania: Ministry of Finance (Managing Authority), Ministry of Social Security and Labour (Intermediate Body) and the European Social Fund Agency Poland, Cooperation Fund Foundation Slovak Republic, Ministry of Labour, Social Affairs and Family Sweden, ESF Council Wallonia, ESF Agency EIPA Lifespan of Network (start and if appropriate, end date): May 2009 to December The follow up LN (CoP RBM+) was initiated in 2013 and will run until March A64

141 Network budget: According to the factsheet, the Commission contribution foreseen amounted to 419,999. According to the final report the actual total cost of the action amounted to , consisting of a Union grant of and the beneficiary contribution of In addition, the network s premium partners committed to allocate at least 1.5 days per month for CoP activities. They also pledged to provide in-kind commitments such as providing meeting facilities, etc. Commission support group/person: DG EMPL Unit E1, Dirk-Jan-Robert Reyntjens AEIDL monitoring contact: Yves Boisselier Section 2 Summary Aims and objectives of the Network: The origins of the network can be traced to the EQUAL initiative ( ). At that time a working group for Member States on RBM existed. The network s manager was involved in this working group as Commission official. It was part of activities related to the development of Project Cycle Management and the more general desire to understand what works and how much is achieved. A large conference on this subject was organised at the start of the current ( ) ESF programmes. After his return to the VDAB, a small group of MAs was formed, most of them involved in monitoring and evaluation. This can be considered the precursor to the RBM learning network. From the interviews it became clear that the network coordinator was and remains a linking pin between potentially interested people or organisations and the network. A positive culture towards participation in EU initiatives is a further positive factor in the decision to participate in the network, according to interviewees. To decide on the choice of themes and focus of the network s activities in general, at the start a workshop with management level participants was held. It served to identify common themes and questions on which the LN should focus. The website and outer outputs, such as the indexation documents, reflect the outcomes of this workshop. As can be seen from this approach, the network aimed to operate in a demand-driven manner. However, it should be noted that many people do not have questions or have never given any thought to the subject. The network also inventorised what was already being done by others. This (also) served to identify input for the seminars in terms of good practices. This Learning Network has been designed as a Community of Practice (CoP). According to its final report (p. 2) its overall aim is to integrate Results Based Management (RBM) in to the EU Structural Funds (SF). For the purpose of this network the following operational definition of RBM has been chosen: "Results Based Management is the way an organization applies processes and resources to achieve targeted results." For the evaluation a more precise definition of objectives has been reconstructed. The final objectives are related to the working of EU employment A65

142 policy. These are (Network description, p. 21): o o o Efficiency, effectiveness and relevance of the European Social Fund for the European citizen Progress to the Lisbon strategy objectives Accountability to policy-makers and the general public The promotion of RBM contributes to these aims in two ways. These intermediate objectives are derived from the long term vision described in the Network description (page 2). The impacts concern both the organizational and individual levels and contribute to: o o Strengthening (human) capacities to manage the ESF programmes with a focus on RBM practices; Identifying and introducing necessary tools and programming mechanisms which promote RBM in relevant organizations. The intermediate objectives are achieved through the results of a learning process that consists of four dimensions: o o o o Development of professional competences Development and sharing of knowledge Dissemination Development and introduction of tools EU policy context: When the current network was being conceived, the delivery structures for employment policy within the EU had been increasingly subject of investigation and change with a view to increasing their effectiveness and efficiency. The modernisation of Public Employment Services became an integral part of the European Employment Strategy starting with a specific guideline in the integrated guidelines for employment and growth. In its wake the implementation of ESF and the functioning of Managing Authorities attracted increasing attention too 13. One of the four thematic objectives distinguished in the ESF Regulation for is the enhancing institutional capacity and an efficient public administration. Activities undertaken by the network: 13 See e.g. European Commission, Evaluation of the capacity of the ESF delivery systems to attract and support OP target groups, Brussels, March 2011 A66

143 The Work programme and time schedule proposed by the learning network show that both activities directly related to the aims and objectives of the network, as well as some activities designed to produce input for these activities were foreseen. During the first phase, the learning network planned to undertake interviews with the potential premium members to identify key topics and issues of interest. Together with the report on the state of the art of RBM, both activities were supposed to provide thematic direction for activities in later stages and contribute to the final set up of the network. Starting with the second stage, the learning network planned activities directly related to the learning of its members as well as wider dissemination of the findings. Most of the objectives were planned to be achieved through basecamp meetings and scouting teams. The expected results of these activities were envisaged to be different at each stage. In addition, the learning network planned study visits, to publish an updated Sourcebook, both in electronic and hard copy versions, and publish a DVD with the final version of the COP website. For the dissemination element, the network periodically held lunch presentations for the DG EMPL staff, circulated the newsletter, and organised a learning seminar. The focus in the remainder of this report will be on the latter type of activities. An overview of the activities can be seen in Annex I. This annex shows the intervention logic for the first RBM network, as reconstructed for the purpose of the current evaluation. Outputs of the network: According to the addendum of the final report, all planned activities (as of the interim report and rescheduled with the request for extension) were carried out, except for a final advisory group meeting. Basecamp meetings The network organised eight basecamp meetings on different topics such as How to link finance and performance or How to ensure that the results can be attributed to the projects. External experts were invited to the meetings. In addition, study visits were organized which included self-assessment workshop and examining Lithuanian practices concerning communication, indicators, risk management, etc. On average 21 persons participated in a basecamp meeting, the lowest attendance being 14, the highest 26. Scouting teams Two supporting scouting teams were launched: On self-evaluation ( for meetings, materials and final output which was a list of common principles) and on appraisal criteria" for project assessment ( At the Stockholm meeting of September 2011 it was decided to discontinue the scouting teams for lack of participation and interest. Study visits Two study visits were organised, to Lithuania and to the Czech Republic. A one-day RBM self-assessment using the tool developed by the COP RBM was undertaken in Lithuania on 22 September 2011 with representatives from the Ministry of Finance, the Ministry of Social Security and Labour and the ESF Agency. The follow-up two day study visit took place on January 2012 and had COP RBM partners looking at Lithuanian practices concerning communication, indicators and targets, risk management and presenting practices concerning mission, vision and values and piloting a new policy action via calls for proposals. A two-day self-assessment workshop with the Czech Ministry of Labour and Social affairs in Prague on March In addition, a trial run of the study visit methodology was A67

144 undertaken in the Flemish Managing Authority ( for a first discussion and report on the method). DG EMPL lunch presentations Two sessions were held with DG EMPL: a session on the COP RBM was organized with DG EMPL on and at the ESF Committee Technical Working Group 24/10/2012 Newsletters Two newsletters were published ( and Learning seminars The final seminar in Maastricht on 5-6 November 0212 attracted participants managing Structural Funds programmes from Greece, Poland, France, Belgium, Slovenia, Czech Republic, Lithuania, Portugal, Germany, Bulgaria, Sweden, Italy and Hungary. See Sourcebook A completely revised version of the Sourcebook was issued. Furthermore, guides on RBM for OP managers and for organisations concerned by governance reform/change, as well as an instruction kit on the identification and presentation of a good practice, were developed. The latter includes guidance for practice description and assessment criteria for peer evaluation of relevance/interest of the practice to the LN members. Training or Lesson Packs were offered through the knowledge base with access to free learning opportunities, presentations, reports and good practice cases. 15 partner practice descriptions and a self-assessment tool were produced ( COP website The website was developed, including a discussion centre, a knowledge platform, the community members and the various events of the LN and others related to RBM and ESF. All the results of and working documents used for the Basecamp meetings are available on line although some may required a registration to the website. DVD Members were provided with a download of the website on DVD Report on RBM A baseline study was conducted. The report (90 pages) provides an overview of the approaches and tools to tackle the challenges inherent in making results based management a reality. Section 3 Relevance of the Network The relevance of the Network aims to ESF objectives and wider EU policy goals The network s aims are highly relevant, in the sense that they the address the implementation or capacity issues that have gained a quite prominent role in ESF and EU employment policies (see section 2 policy context). It in particular reinforces the programmatic approach to policy making such as project cycle management (PCM) and the practices of monitoring and evaluation and the tools and analytical frameworks developed to this end by the Commission. RBM has inspired much of this work, but is in fact a far more encompassing and sophisticated approach. While RBM was being much debated and had been adopted by major multilateral agencies in the world such as the UN and the World Bank, its application in EU employment policy and ESF in particular was still very limited before One of the few related activities was the ESF Sourcebook on Sound Planning and two communities of practice which explored two of its tools project cycle management (PCM) and Common Assessment Framework (CAF). With regard to RBM EU policy makers in the field of employment were therefore lagging behind the rest of the world. In order to catch up first of all knowledge and information needed to be collected and disseminated. Policy makers needed to learn about RBM and explore its potential in their daily policy development A68

145 practices. These needs were to be addressed by the RBM COP learning network. The relevance of Network activities to Network aims: The network activities were coherent with the objectives of the network. The activities focused on building, exchanging and disseminating knowledge, which formed the basis for the further effects and impacts to be achieved. Basecamp meetings e.g. helped to discuss and develop tools and methodologies as well as exchange knowledge. Thematic meetings helped to structure the process. How were the themes for LN activities selected? What is the added value of the LNs to ESF objectives? Considerable effort was made to gear the themes to not just the members but also the organisations these members were serving, notably policy makers in their country involved in the preparation and implementation of employment policies. In addition to the process described in section 2 above, it can be mentioned that a stakeholders analysis was conducted detailing for each type of- stakeholder the relevant expertise and experience, as well as the actions, and events that would be relevant for the work of the network. These stakeholders were approached in the first phase of the project to explore the opportunities for and interest in collaboration. Section 4 Effectiveness of the Network COP self-evaluation The network conducted a self-evaluation using the concept mapping methodology also used for the preparation of the study visits. Self-evaluation methodology (Final report p.5) It was conducted by asking all members of the advisory and premium membership groups to input their ideas regarding the key statement The COP will be widely acknowledged as very successful if Once all ideas had been gathered, they were asked to rate the ideas according to importance, actual performance and actionability. In the mean time, the ideas were clustered. This led to the identification of 8 clusters. Using the ratings, it was possible to prioritise three clusters were performance was lacking and one where performance was good and needed to be sustained. A brainstorm for actions was then conducted via the web platform. Some of the ideas were incorporated into the new COP RBM + proposal. Achievement of network s aims Results The results of the network to be attained related to: development of professional competences, development and sharing of knowledge, dissemination, and A69

146 development and introduction of tools. These results have by and large been achieved. The professional competences of the participating MAs, network members and activity participants in general have benefitted from the activities of the learning network. Individual partners of the Learning Network indicated clear added value for personal development (in terms of skills and competences which have been developed, and as well in terms of experience exchanged between colleagues of different countries/services) though no official certificates were issued. Of course, these results depended on the efforts made by the participant. Also, participants already interested in RBM were more likely to learn but their final professional competence was also fed by other sources than the learning network. The network s activities have generated much knowledge that was shared, actively processed and further elaborated and geared towards the ESF practice. This is notably true for the basecamp meetings and the study visits. The most effective activity in terms of learning was the basecamps. They were also the most expensive ones. With regard to the success of scouting we already had our doubts beforehand. The intention was that members would themselves actively seek out and gather knowledge and subsequently share this with each other. This is the basic idea behind creating a community such as our RBM community. One of the scouting team has been successful (on evaluation, led by Benedict Wauters), the other two teams went under and participants did not continue the work. The reason why the base camps were more successful than e.g. the study visits may be that the latter were voluntary and depend entirely on the interest of members of the network. As a result, only two study visits were organised. The base camps were organised by the VDAB. In principle though study visits are more useful for partners, as in many case seminar participants are not debriefed afterwards. Dissemination was actively pursued, but generally limited to the primary circle of the network. An exception in this respect was the Italian member, whose organisation was also tasked with supporting Italian regions in their management of ESF. The Italian member used short papers, meetings and presentations to disseminate the knowledge gained through the network and the events that were organised. They advised the regions on tools they could use and sources they could consult. Regions took up RBM to different degrees and in line with their needs and level of advancement. This Italian example can be considered a good practice for dissemination of learning gained in a learning network. There are some indications that internal meetings, rather than written information, are the most effective way to disseminate the experience gained through the network. The network has developed a coherent and rich collection of RBM tools, including some very practical ones for further learning. In addition, the final conference did not just provide a learning opportunity to participants. Many of them (67%) were triggered to move forward, although participants did feel in need of further support. Almost one quarter felt challenged and ready to move forward. Quality of outputs: The quality of the outputs was generally high. Some examples: Website The website is rich in content and features many prominent authors and articles, as well as offering various functionalities facilitating exchange of information and views between members and providing information on relevant events and contacts. Interviewees considered the website to be on equal par with other websites and fit for tis purpose. The website requires a certain degree of familiarisation and is less accessible for outsiders. However, the latter was not the intention of the site. Time constraints rather than site characteristics kept other members from inputting themselves. The website was actively used for finding information and valued very highly in this respect by all interviewees. A70

147 Basecamp meetings Internal evaluations as well as the interviewees conducted for this evaluation showed a great appreciation and extremely positive judgement for the basecamp meetings. They were considered to far better than the average seminar people attended, for the following reasons± o o o o The time set aside for interaction Intensive preparation and follow/up, notably also by participants Quality of the speakers and quality f the lectures Active involvement of participants and higher learning impacts, due to the above Study visits and preparatory workshops (self-assessment) The workshop organised intended to prepare a visit to the Czech Republic received rather low scores in its own evaluation. This is likely to be related to the feedback concerning the complexity of RBM and criticisms of the tool used for the self-assessment. This tool has been adjusted since. In Lithuania participants were much more positive and the study visit was executed. In fact, participation from other countries was rather limited. It remains somewhat unclear whether this was due to lack of interest or to unawareness of the event. Perhaps the name study visit is confusing as from the interviews it would seem that the study visits were particularly useful to the host country and the fact that several stakeholders from this country discussed intensively together. Final conference On 26 November 2012 the final conference of the learning network took place. It was held in Maastricht and featured lectures on RBM, evaluation tools, RBM tools and the possibilities to apply the former under ESF. The organisation of the event was rated very highly by its participants, who awarded average ratings from 8.7 to 9.2 for the various dimensions. The quality of the speakers was on average rated much lower (7.7), but this difference is not uncommon for conferences. Also, six speakers were awarded scores higher than 8 and only four of them scored below 7. Interestingly among the highest rated presentations were those given by network members. An indicator for quality may be the popularity of the outputs on the website. The most popular sources on the website are (23 October 2013): 1. Sourcebook on results based management in the EU Structural Funds download(s) 2. COP RBM self-assessment tool download(s) A71

148 3. Collection: results oriented self assessment tools download(s) 4. Knowledge base on Results Based Management download(s) 5. COP RBM self-assessment tool vs download(s) 6. Conceptual Framework (CH 3 in Qualitative Research Design) download(s) 7. Challenges and Lessons in Implementing Results-Based Management download(s) 8. Definitions of results download(s) 9. Results chains and Strategy Maps download(s) 10. RBM management system and framework download(s) The sourcebook and the basecamp seminars were judged to be the most useful tools by the interviewees. The Sourcebook for the quality of its content but also because it greatly facilitated the dissemination of RBM. As indicated already, RBM is not an easy subject, requiring serious study before it can be applied. The Sourcebook helped explaining and allowed people to answer further questions once they encountered them. The principle of the Sourcebook can be labelled a good practice and it would be worth to examine if it could be applied in other networks also. The basecamp seminars for the quality of their speakers and the time available for discussion. The latter is what distinguished these seminars from ordinary seminars. Development of policies or subjects Have not occurred. Effectiveness of processes o The network has four decision making processes or levels: o o o Advisory group sets the direction of priority issues; Management group monitoring the progress; CoP coordinator daily management and logistics; o o Premium members decide learning agenda for each basecamp The learning network has three types of membership: o o Premium managers of ESF programmes Expert representatives of DG EMPL, OECD, etc. o o o Regular Input was mainly provided by the LN Coordinator Benedict Wauters. Members should be encouraged to directly input content on the site to simplify task and bottlenecks. Actors to be recruited as Premium Members may not have fully grasped the results they will obtain from the Network. For instance, the Polish A72

149 o member has been taken out (unclear by whom) because this member clearly indicated that they would never apply or be interested in applying RBM approaches but were simply interested in enabling some of their people be able to travel to the different meetings. In addition, a Romanian member was also removed because there was no effective participation (AEIDL report). Despite some delays and a subsequent extension for the network, it seems it was organized in an efficient manner. It also seems that the project coordinator was deeply involved and acted as project champion. Skills are key to the success of a network. Not everyone is able to lead a team; some lack the intellectual capacity or are discouraged many programme managers are focused on implementation and do not consider it within their powers to play a more independent role. The final evaluation pointed out that the Commission has limited interest in the networks. This is related to regular staff changes in DG EMPL, but according to some of the interviewees it also reflects an attitude of file management focusing on formal obligations. Instances of learning The most useful tool for learning according to the interviewees, were the basecamp meetings (See also under quality of outputs for details). For individuals learning included concrete skills such as setting relevant and realistic objectives, conducting a self-assessment, and formulating indicators. Learning occurred though the external experts, the network coordinator, and the presentation of practices in other countries. A large majority of the participants of the final conference also felt that the conference had increased their understanding of RBM as an overall approach. A smaller number, though still over 70 and 80%, felt they learned about RBM tools and methods and about the application in their own context. This confirms feedback from the study visits, signifying that RBM is a subject with a highly theoretical foundation that is demanding of its potential users. Also, the ambitions of RBM are to changer organisations, again a challenging task. The second objective considered achieved by a large majority of the participant is the expansion of their professional network. This indicates that a foundation for further learning was also achieved. Interviewees underlined the fact that RBM is a tough subject and that it takes time for learning to occur and to achieve a level that enables dissemination and, subsequently, application. Interviewees also underlined the fact, that while the theory of RBM is good, it really needs to be sold to others. An instance of learning at the organisational level was found in Lithuania, following the self-assessment and study visit held in this country. Not only did the three participating Lithuanian institutions study visit all achieve a better understanding of key concepts of RBM, they also developed a joint language and understanding and learned to work together better cooperate closer in this area. Finally, in various countries a more immediate impact, or spin-off, concerned the knowledge gained on monitoring and evaluation. Policy transfer In Lithuania, after the study visit, the vision or lookout on programme management changed. So far a very hierarchical structure prevailed, with the Ministry of Finance at the top and line ministries and then MA below it. The two lower levels were expected to listen to the level above them, with the Ministry of Finance as chief executive. After the study visit the MA felt they was supporting the line ministries and they were part of the key process, with a distinctive role to play. The feedback on the tool used for the self-assessment led to a number of changes in the tool to render it less complicated and allow more time A73

150 for explaining concepts and definitions used. In Sweden, the ESF Council is now preparing the introduction of RBM in the ESF Managing Authority next year. Benedict Wauters is expected to come and conduct a seminar for key people in the agency. At the Council, they are planning to change the way they work with the ESF programmes as a whole. Among the first steps, they will develop their mission and vision. This is the first time such a discussion will be held in the MA. For the above process, at least at its beginning, Benedict Wauters is a key person. Other consultants might be also contracted. Also a person responsible for quality management will be involved. The support from other stakeholder is considered important: PES, social insurance agencies, municipalities and employment organisations. In Italy, the active dissemination to regional authorities by the network s member (see above) has helped these regions apply some of the BM principles in their planning and monitoring & evaluation work. It seems the impact was higher in regions and organisations already more predisposed towards change. In addition, the member increasingly applies RBM in its own organisation and for its own work, e.g. in the way objectives are defined for the coming programming period. The case of the French speaking part of Belgium illustrates how fertile ground increases the chance that a learning network will engender changes in organisations and policies. When the Wallonia organisation joined the network, they had been reflecting on how to improve their working practice. They had already adopted a Common Assessment Framework and had made Project Cycle Management compulsory for all projects. To them, RBM was a logical extension of this work. They in particular looked forwards to the opportunity to exchange views and experiences with other countries. Much of the learning was small : it helped to solve some concrete challenges and made work easier in practice. Rather than generating massive changes, it engendered a series of smaller steps that all worked towards better management and further application of RBM. This development is continuing under the follow-up project (RBM+). Examples of the steps undertaken are: the application of RBM principles in a new project on gender and the introduction of Mr and Mrs Result, two colleagues who are tasked with learning about RBM and supporting the further introduction of RBM in the organisation. This effectively doubles the RBM support or promoting team in the organisation. It is furthermore worth mentioning that in Wallonia the introduction of RBM is explicitly applied at both the project and the organisational level. The Managing Authority in Greece was operating in a change demanding environment when joining the network; they were themselves on the verge of redefining e.g. their mission and of course the Economic Adjustment Programme and all it was accompanied by exerted strong pressure on increasing and measuring the results of amongst others spending under ESF funds. Participation in the network increased the awareness that a move from compliance to control was needed. As a result, they introduced new working processes that were generally well received by staff. The thinking about RBM also changed the management style in the organisation, leading to more room for experimentation and increased flexibility in general. In addition, hearing the Belgian and Polish experience with the simplified cost options led to management of the Greek MA giving the green light to its application in their country too. The impact of RBM thinking was not limited only to one organisation and is gaining more visibility in Greece. For example, the National Academy of Public Administration introduced RBM in its curricula. Also, the Institute for Public Administration dedicated one day to RBM approach in its five day trainings on project cycle management. As a result, around people heard about RBM. Right now, there are some efforts to persuade the Institute to organise seminars exclusively dedicated to RBM. The interviewee herself promotes RBM while organising training on other topics. In Lithuania, the self-assessment and the study visit created also a basis for further policy making. The activities produced concrete overviews of good/best practices in the country, promising alternative practices and inputs for improving the strategic planning system in terms of formulating results oriented goals, and indicators for measuring their achievement. After the study visit, the three institutions established a common working group on RBM to discuss further on its applications. In addition, as a part of RBM+ efforts, the internal RBM working group within ESFA was established. One of the other concrete results is that now that a new Operational Programme is being prepared, all three institutions work together in producing indicators that are results based. Furthermore, the MA will itself install an internal working group on RBM to further its introduction within the MA. The agency is going to promote RBM by organizing a meeting in January where it will invite its stakeholders. Also it managed to secure political support from the Cabinet. All in all, since 2002 ESF A74

151 management in Lithuania has gradually been moving towards results-oriented indicators and more results-oriented management. As one of the interviewee noted: The change does not happen overnight often 5 10 years are necessary to change mind set and thinking. Impacts of the network Intermediate impacts The intermediate impacts foreseen related to: strengthening (human) capacities to manage the ESF programmes with a focus on RBM practices, and identifying and introducing necessary tools and programming mechanisms which promote RBM in relevant organizations. The COP self-evaluation identified three weak clusters of factors important to success of the network: EU level pressures to manage for results, shift in programme management focus from compliance to results, and practical use of RBM concepts on the ground. Plotting the clusters on two dimensions furthermore reveals that the weak clusters are situated where success factors across the EU are concerned and where the issue is to put RBM into concrete practice. The strongest cluster is somewhat more closely related enabling conditions and measuring success factors at COP level. The strongest cluster was: COP members are agents of change. This supports the view that while results were achieved, the intermediate objectives have not been achieved to the same degree. The capacity to manage ESF programmes with a focus on RBM practices has increased in as much as staff participated in the network. At the organisational level the impact is weaker, but in a number of countries changes at this level could be observed and could be directly traced to the participation in the network. See the practices described under policy transfer above). The network has been strong in identifying tools and programming mechanisms. While their actual introduction in MAs and other organisations takes time, this has been taking place already. The analysis of these occurrences shows though, that rather than a massive one-off launch, a gradual introduction is the likely, and perhaps best, way to proceed. Of course, many organisations traditionally have a low degree of the results-orientation and this will hamper rapid introduction of RBM. This is illustrated by the views of participants of the Czech workshop that was meant to prepare a study visit. The workshop applied the RBM self assessment. Almost all participants afterwards felt that the workshop had helped to draw attention in the organisation towards important issues not on the management agenda before. However, only a minority expected the workshop to lead to changes in the way programmes are being managed in the Czech Republic 14. It is important to remember here that when network started, gaining knowledge was paramount as so little was known at the time in the EU. All the other objectives, further down the line or further up the objectives tree were defined in a very global manner as so much groundwork was to be done first. In fact, the follow-up network now running has been designed amongst others to take impacts to the next level. The network s activities have also led to the conclusion that RBM can work, but that it requires that someone makes a start. This should happen at management level. But many of the people that participate in the network s activity hesitate to bring their experience to the management level. When asked about the future application to the tools they learned about during the conference, participants indicated they were likely to apply those related to evaluation, but far less so those related to RBM itself. This can also be seen from the assessments on the website: people will state that the practice has been good, but when asked they also state that they are not going to propose this to their management. Or they state it would take management another year to decide on implementing such a practice. The role played by the managerial level is best illustrated by the experience of one of the members, who became a manager in the course of his participation in the network. According to him, in the beginning it was difficult to transfer knowledge while he had 14 In fact, the minority amounted to 40% which is in itself not a bad result. With only eleven participants conclusions here should be drawn with care, A75

152 not had the management position. It was too difficult to introduce it to a boss who was looking for more quick to implement solutions. After having obtained that position himself he actively set out to introduce RBM. He tried to reshape the organisation according to what he had learned, but found that RBM was not easy to explain to others. People also have difficulties to see the links between RBM and what they are doing in their daily work. A solution to this is to introduce RBM thinking into the organisation without explicitly referring to RBM. This proved to be a more effective approach. This does require a certain pacing, and gradual introduction. Nevertheless, the examples mentioned in the previous section show that there is potential for effecting changes in organisations and at policy level with networks such as the RBM COP. Time will have to tell how substantial and how lasting these changes are. Final impacts Given what has been concluded regarding the intermediate impacts, it is clear that achieving the formally set- final objectives was not realistic and has not been achieved. A number of factors can be mentioned that further explain the absence of wider impacts of the network: o o o Duration and budget: two or three years is too short a period to effect such changes in organisations. Also a far larger budget would be required if substantial changes are to be effected in all participating countries. Tardiness of organisations: one does not change organisations by a couple of activities and a can as such not be expected to have such an impact. It needs the co-existence of a policy or political drive towards change, coming from within the organisation, within the Member State or perhaps from an EU level agreement and commitment, e.g. in the framework of the EES. Volatility of the conditions for success: little things may have a great impact on the success of a network. Also, things need to move in the same direction, in different Member States, at the same time. It also requires a high degree of sensitivity to make use of the window of opportunity when it -briefly- presents itself. Section 5 Success of LN activities Reach and dissemination The actual reach of the project is the twelve partners. This is intentional -setting something in motion in those organisations is considered to already constitute a tremendous achievement. Also the other activities, including the conference and the website aimed to reach the partners. As people had to register themselves, only people with a serious interest would do so. The activity on the website was limited, as partners often preferred . Lessons learned from activities The COP self-evaluation (see above, under results) showed that with regard to cluster 3 (EU level pressures to manage for results) priority should be given A76

153 to influencing the European Commission in exhibiting both pressure and support for creating more Results Based Management oriented OPs in the next programming period 15. The most important assets of the COP related to the members being agents of change (cluster 8). Key strengths to reinforce or sustain include: support from the advisory group to new ideas on RBM, and COP partner organisations taking stock of the lessons learned and moving forward to innovative ways of managing. For the other two clusters it was decided that further brainstorming was required. These concerned the shift in management focus from compliance to results and the practical use of RBM concepts on the ground. An important lesson coming from the study visits is the importance of involving senior management in events. Without their participation the impact of events and of the network in general on organisations in Member States will not come about. Changing organisations starts with changing the attitude and of management and enlisting their commitment. In Lithuania the study visit was held with managers. In the Czech Republic a broad arrear of staff and functions participated. Here participants realised something needed to change, a more results oriented approach was needed, but this needed to be put in motion by management and they were absent from the meetings. Finally, several examples have shown that Introducing RBM takes time as it involves a serious amount of studying in order to grasp the concept and understand the tools and their application. In addition, countries, organisations and people move at different paces. This is something to be accepted and taken into account. The speed of introducing RBM and perhaps any kind of learning is highly dependent on the starting point and the capacity for change of those concerned. Networking The network reaches a wider audience through cooperation with others. Through this they amongst others reached people active in development activities. The network coordinator actively seeks out and participates in other networks such as evaluation networks. The learning network has been successful in establishing links with other international organisations and networks active in the field of RBM. This has amongst others yielded an alliance with the OECD. Section 6 Efficiency of the LN Level of resources allocated: The Flemish Managing Authority provided all the co-financing; o o o The CoP covered the travel costs for the Premium members to participate in meetings; Premium members had to commit to a minimum time contribution (at least 1.5 days/month); A framework contract was established to deliver the CoP secretariat role; and The learning network was granted one extension. It considered applying for a second extension, but instead started CoP RBM+. 15 Measured as: statement having more than average importance and lower than average performance. A77

154 The network is centrally managed. This has been a deliberate choice on the part of the network coordinator. He expected that initiatives from partners would be limited. A more active role of partners and more sharing in the network would have been preferable, but as it turned out this was not possible. The aim though is to institutionalise joint activities in the network. Therefore in the new network two co-coordinators have been appointed besides the coordinator. The network made as little use as possible of sub-contractors. The RBM network only used them for specific content inputs, notably speakers, and tasks for which not enough time was available or that were not important, e.g. editing the English texts produced by non native speakers. Proportionality of resources to achievements of activity: The activities and the outputs generated by the network seem ample in view of the resources spent. Changes to resources throughout the life of the network: The budget was underused. Section 7 Likely sustainability Sustainability of Networks activities The sustainability of the networks activities can be considered high, as a follow/up learning network aims to go forward where the first network had to stop. This means that the knowledge gathered, the tools developed, the learning by key individuals etc. has a fair chance of being put to use to produce the impacts desired in terms of institutional and policy change. In addition, the changes already effected in participating organisations seem well embedded in their regular operating systems. Sustainability of impacts and outcomes Not applicable, since impacts were limited during the fist term of the network. For results see previous point. New policies developed (at any governance level): This was not the case. A78

155 Section 8 ESF Calls for proposals Not applicable. Section 9 Strengths and weaknesses The Network s key strengths: o o o Motivated members and network co-ordinator (although further research might also conclude this to be a weakness) Centralized organizational structure The learning process structured around thematic basecamp meetings The Network s key weaknesses: o o o o It did not recruit right members from the beginning. As the Polish and Romanian cases suggest not all the initial partners were aware of the network goals and ready to commit to them; Too much workload on the network co-ordinator who already had daily duties at his organisation. Thus, there would be also risks that in extreme situations daily duties could take over the commitments to the network and, as such, result in a slower process. Dissemination website independent of the project and accessible to Integration into ESF or MA practice: adopt in overall rules regarding M&E, task definition of MAs Examples of good practice: o o o Basecamps and testing new tools during study visits, although further research is needed to determine whether the expectations raised were met in practice. For study visits the partner organisation first undertakes a self-assessment, then, on the basis of the assessment, defines interesting practices it wants to present to others as well as practices it wants to learn about from others. Tool for assessing good practices Success factors underpinning learning and transfer: o The highly motivated network coordinator A79

156 o The flat network structure where all network members were attending the same meetings Annex: Intervention logic Results Based Management COP Rationale Aims and objectives Inputs Activities Results Impacts The learning network makes a contribution to: RBM improves the decision making process and drives change It has been adopted by major multilateral agencies in the world Although, RBM is not adopted in the ESF policy area, the interest has been growing Overarching aim: - efficiency, effectiveness and relevance of the European Social Fund for the European citizen; - progress to the Lisbon strategy objectives; - accountability to policymakers and the general public through: - describing existing RBM practices and their usefulness; - updating the Sourcebook with findings by the means of standard templates. Financial inputs: Financial support from the European Commission equal to 419,999 The beneficiary contribution of 38, Human resources and inkind inputs: Additional contributions in-kind such as meeting facilities as well as a commitment of each premium member to allocate 1.5 days per month for CoP activities Main activities: Basecamp meetings Scouting teams Study visits Sourcebook DG EMPL lunch presentations Newsletters Seminar for ESF programme managers Seminars for ESF committee members DVD Sub-activities: COP website DVD Report on RBM (baseline study) Programme level: - development and introduction of new tools - Wider dissemination and support for RBM within ESF Individual members: - development of professional competences Participating organisations: - knowledge development and sharing - Strengthening (human) capacities in the relevant organizations to manage the ESF programmes with a focus on the best RBM practices; - Identifying and introducing necessary tools and programming mechanisms which promote RBM in the relevant organizations - Efficiency, effectiveness and relevance of the European Social Fund for the European citizen - Progress to the Lisbon strategy objectives - Accountability to policymakers and the general public A80

157 A81

158 Empowerment case study Section 1 General Information Learning Network name: European Learning Network on Empowerment and Inclusion Lead partner: UK-Northern Ireland: Department for Employment and Learning (DEL), the NI ESF MA Day-to-day running was subcontracted to PROTEUS (national support structure) Network coordinator (name): Brita Terpe and Pat Donelly Other members: UK Northern Ireland: Belfast City Council Belgium Flanders: ESF Agency Spain: Ministry of Labour and Immigration (Managing Authority) Greece: Ministry of Employment and Social Protection Sweden: Arbetsmiljöforum Slovenia: Government Office for Local Self-Government and Regional Policy Lifespan of Network (start and if appropriate, end date): 1 September 2009 to 31 August 2012 Network budget: 445,000 Commission support group/person: Elisavet Platsouka supported by Peter Saztmari, subsequently Sonia de Melo Xavier A82

159 AEIDL monitoring contact: Allen Mercer Section 2 Summary Aims and objectives of the Network: The learning network had a very practical oriented objective; it set out to create and offer tools to promote an empowerment and inclusion approach to employment issues in the Member States based on existing good practices. This objective is the result of work carried out previously by two EQUAL Communities of Practice Equalising the workplace diversity in action and Diversifying the workplace strategies for empowerment and inclusion where Northern Ireland took a leading role. The objectives of the network included: 1. to improve the quality of actions to be financed under the ESF Operational Programmes; 2. to monitor the extent of the implementation of objectives related to Empowerment and Inclusion outlined in National Action Plans and the ESF Operational Programmes; 3. to provide a platform for debate on issues related to Empowerment and Inclusion; 4. to enhance the effectiveness of existing policies dealing with Empowerment and Inclusion; 5. to undertake research and benchmark current best practice in the areas of Empowerment and Inclusion across the Member States. The theme was suggested by the European Commission. After the launch of proposals for the Learning Networks in 2008, EU Commission staff approached Northern Ireland with a request to lead the network. Due to the national priorities and positive experience with the network members under the EQUAL period, the UKNI agreed to lead the network. Within the network (preparatory phase) it was agreed that the recommendations from the EQUAL period should be leading, including to focus on the individual, employment agencies but also the employers themselves. During the EQUAL period, the tools in Northern Ireland and the Netherlands had amongst others been identified as good practices. The network aimed to build on these to develop a benchmark and identify other good practices and develop practical tools for individuals, employers and intermediary agencies. EU policy context: The focus and approach to empowerment by the network specifically focused on increasing and improving access to employment. In that sense, the network s objectives can be related to the EU s goals, policies and objectives for increasing employment as well as on improving social inclusion. The EU2020 objectives of A83

160 - ensuring that 75% of the population aged is active in the labour market and - having 20 million people less at risk of poverty and social exclusion are relevant for the network s objective. The latter target is reinforced by the flagship programme European Platform against Poverty and Social Exclusion, which (amongst others) aims to provide strategic actions to make better use of EU funds to support social inclusion and combat discrimination. The objectives of the network are also in line with the purpose of the European Social Fund. Furthermore, in the final conclusions of the EQUAL events, it was recommended that the concepts of empowerment and diversity should be developed within the ESF Operational Programmes and within the wider framework of national and EU social inclusion, anti-discrimination and employment policies. The network aimed to fulfil this recommendation. Activities undertaken by the Network: The learning network worked towards producing three tools: Beneficiary Tool a questionnaire designed to capture the sense of empowerment of any trainee within an ESF project; Intermediary Agency Tool based on the beneficiary questionnaire, as by mirroring these questions, the answers from agencies such as training providers, social economy organisations, career services, job centres and NGOs will provide a 360 degree perspective on empowerment and inclusion within these organisations; Employers Tool This has been conceived as a system of recognition and applause for employers who recognise and implement measures to empower their workforce. All activities were geared towards the fine-tuning of these tools. The activities included peer reviews, study visits, workshops, piloting/testing exercises in a number of number of Member States. Eventually the network also decided to develop a handbook. The activities were decided upon and adjusted by the Steering Group which met 10 times throughout the duration of the network. The network followed (or at least aimed to) a set work plan with a specific and detailed course of action. A84

161 The preparatory phase (approximately 6 months) consisted of the following main activities: Familiarisation with the network s members, objectives and existing tools Fact finding exercises (including meetings with leaders of potential best practices) Design of the work plan for the implementation phase The implementation phase consisted of the following main activities: 1. Meetings (network meetings in BE, ES) 2. Research (Study visits, policy context reviews in Member States, background paper) 3. Peer reviews (1) on the definition of empowerment, 2) on the beneficiary tool 3) the employer recognition tool 4. Development of the tools (identifying, agreeing, piloting & testing incl. training delivery in use of Empowerment & Inclusion measurement tool in all core group Member States) 5. Dissemination (development and maintenance of the network s website, final conference, production of brochures etc.). Outputs of the Network: The goal of the LN was to make a practical contribution to the learning from MS practices by developing tools and indicators for measuring empowerment and inclusion. The network produced the following outputs: Beneficiary Tool Intermediary Agency Tool Employers Tool The Empowerment Handbook, designed to help the European Social Fund Managing Authorities and the promoters of ESF-funded projects to use empowerment approaches in their work. To facilitate the dissemination of these outputs, a website dedicated to the learning network was launched containing the tools as well as other useful documents. However, as of August 2013, the website contains many broken links, thus limiting its usability. The network targeted employers, intermediates agencies, managing authorities and beneficiaries. A85

162 Section 3 Relevance of the Network The relevance of the Network aims to ESF objectives and wider EU policy goals: The network was linked to ESF and wider EU policy goals through the core goals of the Europe 2020 strategy of ensuring smarter, more sustainable and inclusive growth. Relevant EU 2020 headline targets linking to empowerment and inclusion are to bring 75% of the population aged into employment and to have 20 million fewer people at risk of poverty and social exclusion. The European Platform against Poverty and Social Exclusion, which supports social inclusion and anti-discrimination, is also related to the learning network since empowerment is embedded in employment measures and, as such, they provide the way out of poverty and social exclusion. In terms of ESF, in the final conclusions of the EQUAL events, it was recommended that the concepts of empowerment and diversity should be developed within the ESF Operational Programmes and within the wider framework of national and EU social inclusion, anti-discrimination and employment policies. The network aimed to fulfil this recommendation. The relevance of Network activities to Network aims: The Network activities reflected its objectives, especially by narrowing the activities to focus on developing and testing appropriate tools and indicators to promote Empowerment and Inclusion. Activities were mostly practical exercises which resulted in the concrete outputs for its target audiences, namely beneficiaries, intermediary organizations, managing authorities and employers. However, it may have been useful if from the beginning a) all network members understood and agreed on the practical focus b) there had been discussions on the consequences of this practical focus, which partners should be involved, what dissemination methods would be most appropriate etc. In the end there are four concrete tools, but the use is very much limited to (some of) the partners of the network, when it could have been useful for a much wider audience. How were the themes for LN activities selected? What is the added value of the LNs to ESF objectives? The Learning Network (LN) on Empowerment and Inclusion traces its origins to the EQUAL Mainstreaming Platform. One of its main recommendations was to continue developing the concepts of empowerment and diversity within the ESF Operational Programme. The outputs of the learning network are practical tools which have the potential to enable ESF Programmes to empowerment and inclusion challenges; they are oriented towards the users and projects of ESF. A86

163 To what extent is the network s theme and objectives complementary to other networks and organisations? There were a number of possible synergies that were not fully explored or built upon. The learning network could have cooperated with the network on ex-offenders (EXOCOP) in removing the barriers to access mainstream employment. Potential synergies also existed with the COPIE Network, IMPART and TNC learning networks. However, these synergies were not fully explored. The network did explore cooperation with the European cooperation network among Managing Authorities (ESF CoNet) since the later network had set improving social and labour integration for people with greater risk of being unemployed as one of its priorities. The discussion with ESF CoNET was organized after the suggestion of a member of both networks. However, the results of the discussion did not lead to any substantial developments. Section 4 Effectiveness of the Network Achievement of Network aims: The aims of the network to develop tools and indicators for measuring empowerment and inclusion were reached. The achievement of the key objectives as stated in the original application was also highlighted as positive by the network partners in the final technical implementation report and subsequent interviews. Moreover, an additional deliverable the ESF handbook was produced during the course of the implementation phase. The initially allocated budget was underspent despite additional activities such the handbook and its translations into several languages. On the other hand, the website with the network material is not user-friendly, hard to locate and contains many broken links, thus limiting its potential impact. The website represents one of the missed opportunities of the network; despite producing its intended outcomes, the network has done little to follow through and disseminate widely, follow up with the target groups etc. There appeared to have been a relief when the network finished, rather than an opportunity to develop the next stage with the selected tools and recommendations. Development of policies or subjects: The network started its work with the fieldwork in order to explore the extent to which the notions of empowerment and/or inclusion were embedded in the OPs of the partner Member States. Based on the findings, the Baseline study was drafted. Equipped with the Baseline study and its own elaborations on the notions of empowerment and inclusion, the network undertook the task of developing the necessary tools which, after a number of workshops, peer reviews and piloting exercises, were produced. Finally, after a year of the LN s operation, the production of a Handbook for the ESF Managing Authorities was proposed by a network member and, in the autumn 2011, agreed and approved by the Commission (AEIDL Final report). It is noteworthy that, despite the presence of a work plan, it took quite some time before it was fully developed and agreed upon by the network A87

164 (preparatory phase reports). Quality of outputs: The network produced five main outputs: the beneficiary tool, the intermediary agency tool, the employer s tool, the empowerment handbook and the network website. All tools have undergone a period of testing and the final technical report points out that they have proven to be good diagnostic tools. Having involved the target groups and piloting amongst the different organisations (and not just ESF managing authorities) was especially recognized as adding quality to the outputs. Several of the members of the network confirmed that, despite the organisational issues in the network, they were proud of the outcomes and did see ways it would be used in their work. The empowerment handbook appears to be comprehensive, touching upon issues of definition of empowerment, the role of empowerment in the ESF and how empowerment approaches could be used in the ESF. It focuses on the right target group addressing those who are responsible for the OPs or Joint Action Plans for the programming period. As suggested previously, the Handbook should be helpful to ESF MAs, IBs and projects, whilst the Beneficiaries/Empowerment tool should be useful to projects. In addition, the experience of working in the network seems to have been valuable to members. However, both the final report as well as some of the interviewees criticised the lack of ambition in establishing the award scheme strand of the employers tool and the final commitment to mainstreaming the results of the network. Indeed the dissemination part of the network appears to be rather weak all was done by the book but with little extra effort to ensure the actual use or follow up of the tools. As the tools targeted three very different groups and more attention could have been spent to ensure that they would be reached. The low quality of the website and the difficulty of finding it through commercial search engines confirm this point. The website could have been one of the ways to disseminate the outputs of the network, but the information is difficult to find. Effectiveness of processes: The network consisted of a small number of members, but that was not considered an obstacle as the members felt it provided focus to the network, in particular during the testing and piloting phase. The Steering Group was the overall decision-making body, although there are some suggestions in the reports (confirmed through interviews) that not all participants were able to be involved in the decision making regarding the direction of the network from the start, leading to some friction, as well as an issue with lack of ownership. The day-to-day coordination of the network was subcontracted to a non-governmental organisation (Proteus), serving as the general secretariat. They were however, much more the leading organisation (in terms of focus, decision-making and pushing the network forward) than perhaps is expected A88

165 from a secretariat. This is indeed related to the ownership question in this network. The secretariat did organise all the travel arrangements and prepaid expenses for all participants. This reduced the burden on the members and was considered as a positive experience by all parties involved. There were however, suggestions that the discussions on VAT and the cost breakdown of the accommodation (separating bed from breakfast) caused a significant issue for the secretariat. The network did involve and collaborate with non ESF authorities, particularly NGOs, employer organisations, and local authorities. Based on the interviews these collaborations appeared both important as well as difficult to manage. It required a lot of extra work to ensure the involvement of additional stakeholders. External experts were also hired and became increasingly more important to ensure the finishing of the work towards the end of the agreement than initially foreseen. Instances of learning: According to the Final Technical Implementation Report, members of the learning network recognized that learning process over the course of the project was positive and helped to improve their work practices in their own Member States. Indeed, the interviews confirmed that the network did allow for individual and organisational learning of the members. Learning took place in a variety of situations. For example, drafting the Beneficiary Tool required developing and testing the questionnaire twice which resulted in analysing several hundred responses. Feedback sessions on the results of the pilot runs helped to improve the quality and response rate of the questionnaire. Moreover, testing the Empowerment Tool also facilitated mutual learning. All interviewees confirmed that the most valuable learning moment was during the testing/piloting workshops in each Member State with all the participating (and diverse) organisations involved. Policy transfer: None identified. Impacts of the Network: The learning network considered developing practical tools and recommendations for the Member States as one of its top priorities. According to the AEIDL Final report and its subsequent Legacy Report, the tools had been tested by the ESF Managing Authorities and were likely to be applied in practice in home countries / regions of the LN members. It appears that all network members use at least some of the tools. The interviews confirmed that the reason for not using all tools was either the quality or the relevance. In addition, in some instances it was considered that the use of the tools would increase with the next round of ESF ( ). Beyond the participating network Member States there is little knowledge/use of the tools. A89

166 Within the Member States the dissemination has taken place (though the extent varies strongly between the members). It is, for example, expected that the Beneficiary Tool will be used in the new programming period in Northern Ireland. Furthermore, Sweden indicated that it would use the handbook as guidance. The Employers Tool is used in Belgium by an organisation Time4Society where the tool helps to assess the CSR activities of companies which dedicate some time to help community groups and NGOs. Interviewees did mention other organisations (especially NGOs) having used the tool, but were unsure to what extent they were not already using versions of the method beforehand. Section 5 Success of LN activities Reach and dissemination: The outputs of the network allow for a longer-term success; the tools and handbooks can be used and applied at various project levels and within different types of organisations and do not require intensive training. Within the participating Member States it has had a wider use than outside of the partners. The Final Technical Implementation Report gives the following overview of the target groups and stakeholders important for the success of the network: Managing Authorities and their agents for the ESF Programmes; Employers and Employer Organisations, particularly those with a clear policy of engaging with the wider community, and those whose CSR policies would be models of good practice for other employers; Local authorities; Chambers of Commerce/Employer Organisations/Trade Unions; HR managers who are the first point of contact for job seekers and who deter-mine company recruitment policies; NGOs representing participants/trainees; Champions both from the EQUAL programme and from current ESF Operational Programmes who can evidence the participant/trainee experience; and Bridge builders and policy makers who have a role to play in the empowerment/ inclusion process. The learning network managed to reach a number of these target groups through the design phase. For example, HR managers participated in employer focus groups as well as being involved in testing the Employers tool. In addition, 19 ESF-funded projects were invited into the two-day Employment Workshop which contributed to the drafting of the Handbook. The leaning network from the beginning strived to include the stakeholders described above, but experienced difficulty in getting them on board. It did cooperate with Belfast City Council, Red Cross, Time4Society to name a few. Stonewall and the O2 Ability Awards shared their experience in their employer recognition schemes. The involvement of stakeholders helped to A90

167 refine the initial ideas and facilitated testing of the network s tools. The network presented its outputs during its final event in Northern Ireland in November 2012 where around 40 participants (outside North Ireland) took part. Moreover, the Handbook was distributed during the 7th Meeting of the ESF Committee Ad-hoc Group on Innovation and Transnational Cooperation in Brussels (December 2012) and the Final Event of the Learning Network on Gender Mainstreaming (January 2013). The Handbook was translated into several languages which might help to improve its reach and impact. The network also produced a DVD which contains the tools developed over the course of the project as well as video testimonials. (However, the tools lacked corresponding manuals and instructions, thus limiting their applicability as it would be complicated for a network outsider to correctly use them this is our own observation from looking at the DVD). Finally, the network used its website to disseminate its tools and other outputs to a wider audience. Though the website was supposed to remain operational until the end 2014, as of August 2013, many documents are not accessible online because of the broken links. Lessons learned from activities: The Network had difficulty retaining its members and continuing to keep them focused and involved. This was partly due by the origin of the network (too much reliance on previous objectives/goals, not enough own interest taken on board) and partly due by the management of the network. A changeover in tasks and responsibilities within the lead organisation further complicated the strained relationship. In regards to ensuring involved and committed members, the lead partner suggested that the EC could facilitate a platform to allow member States to pitch their network ideas to potential partners. This would allow them to find motivated partners as well as ensure they are less likely to drop out over the lifespan of the network. Whether this would have solved all problems for this network is unclear. The network also recommended to simplify the application procedure and the financial schedules (this would help facilitate the handling of VAT). Networking: Members of the network had worked together before and were familiar with each other s priorities and working methods. This network did not change that relationship but instead strengthened existing contacts where appropriate. This has taken the form of post-network study trips and continued consultation on methods and approaches. A91

168 Section 6 Efficiency of the LN Level of resources allocated: Over the lifetime of the network 10 steering group meetings took place. Participation levels differed strongly between partners only Northern Ireland and Flanders attended all 10 meetings, while for example Greece attended only half of all meetings. The network also retained an outside expert Mike Morrissey (who actually works for PROTEUS) as adviser for many of its outputs, which was not planned in advance. One coordinator at PROTEUS was most involved in the organisation this was a different person than originally foreseen- and received additional support from another senior advisor within PROTEUS. The budget supplied by the Commission in support of the network was 445,000. The network underspent the budget. In an effort to make use of the allocated budget towards the end of the operational period, resources were spent on translating the handbook into Dutch, Spanish, Greek and Swedish. The budget was also used to assure Slovenia s continued participation by providing financial assistance after national-level budget cuts. Proportionality of resources to achievements of activity: The underspend was the result of a reduced participation of partners. There are also suggestions that the budget was underspent due to insufficient focus and involvement, impacting the quality of the final products. Changes to resources throughout the life of the network: After Latvia and Malta withdrew from the network because of the budget cuts, the network decided to cover the costs for Slovenia s attendance. Moreover, some financial resources were diverted to the production of a Handbook and its translation into four languages which was not planned from the beginning. Additional resources were also allocated to the expert Mike Morrissey. Section 7 Likely sustainability Sustainability of Network activities: The Active Inclusion network is expected to build on many thematic objectives and activities carried out by the Empowerment Network and includes the lead partner, Northern Ireland. It already used the ESF Handbook published by the LN to submit its bid for funding. Representatives from Belgium and Northern Ireland also indicated that they would continue use of the tools produced. A92

169 Sustainability of impacts and outcomes: Though it is too early to assess whether the network achieved its goal to influence the new programming period, the outputs of this network will likely feed into national programmes in most of the partner countries. In addition, the translation of the Empowerment Handbook into several languages might facilitate the use of the tools in other countries. On the other hand, the majority of members of the Network were a relatively small countries / regions, thus limiting the chances of scaling up outcomes. Sweden indicated that it had disseminated the tools through its own national learning network amongst ESF managers and expects this to be picked up. New policies developed at any governance level due to network: No evidence found Section 8 ESF Calls for proposals Details of national calls: None of the interviewees were aware of the network s influence on calls for proposals. NI, SE, BE, EL did expect to see the results of the network return in the next funding period. Call for proposal projects: None found Impacts of calls: N/A Section 9 Strengths and weaknesses This section is to give an overview of both the positive and negative aspects of the Network and its work. Please provide information on: A93

170 The Network s key strengths: The key strength of the network was the focus on practice and the development of practical tools. The network also involved a large number of stakeholders in its processes and tested the tools produced with very relevant organisations. All interviewees indeed confirmed the testing to be the most valuable part of the networks activities. In general, the network was mostly practice oriented and the designing and testing the tools facilitated knowledge exchange among members. The final network event itself focused on study visits and the use of tools. The tools can be used by organisations at project level but are also very much designed to support the programme level. A second strength of the network was that the financial resources were managed centrally by the lead partner travel, accommodation and facilities for meetings were arranged and directly paid by PROTEUS. This allowed the members of the network to more easily attend the meetings. The Network s key weaknesses: The network experienced several organisational issues. It is clear that these issues (related to ownership, leadership and disagreements between the network and the EC) had a negative effect on the commitment and interest of network members. The work plan was also too optimistic and, as a result, the network could not keep up with the intended activities. For example, the final evaluation could not be performed because of the time constraints. The organisational issues overshadowed the network s potential and quality. Examples of good practice: Using the tools and experience of the network to further develop a handbook useful for project and programme managers. Success factors underpinning learning and transfer: Since the majority of the activities were geared towards the development and piloting practical tools, members of the network were exposed to a numerous possibilities for mutual learning. Also, Managing Authorities were directly involved in testing the tools with beneficiaries (employers, intermediary agencies, individuals), thus potentially facilitating wider adoption of the network outputs. A94

171 A95

172 Annex: Learning Network intervention logic Rationale Aims and objectives Inputs Activities Results Impacts The reintegration processes of unemployed people back into the labour market is intrinsically linked to empowerment of the individual, especially of those experiencing social exclusion. Reintegration measures that include and consider empowerment can be more effective in the long-term. ESF can be and often is used for reintegration measures to support individuals experiencing (or at risk of) exclusion. The LN aimed to bridge the knowledge gap as there was insufficient information on the best approaches to empowerment in employment measures. Overarching aim: To create and offer tools for promoting an empowerment and inclusion approach to reintegration into the labour market through: - Learning from existing good practices -Developing a tool for employers -Developing a tool for intermediary agencies -Developing a tool for individuals Financial Inputs: Financial support through the ESF Technical Assistance operational budget ( 445,000) Human resource and in-kind inputs Technical Secretariat (1 full-time, 1 part-time) Lead Network Member coordination Network Members (5) Contributions by DG EMPL Contributions by NGOs, employer organizations and other professional organisations Main activities: - Meetings - Research (study visits, policy context reviews, background paper) - Peer reviews -Development of the tools (piloting & testing ) - Dissemination (website, conference, brochures etc.) Programme level: - All tools have become available for all ESF managing authorities and stakeholders - Additional (unplanned) Handbook Individual members: - Experience in further developing ideas and using practical experience with empowerment tools Participating organisations: - increased collaboration with NGOs and employer organisations on empowerment in the participating MS Learning Network making a contribution to: - Awareness raising on empowerment in employment - Accessibility to practical tools for implementing an empowerment approach A96

173 ENYE Network case study Section 1 General Information Learning Network name: European Network on Youth Employment (ENYE) Lead partner: Cyprus, DG European Programmes, Coordination and Development in collaboration with the Greece: Ministry of Employment and Social Protection Network coordinator (name): Akis Nicolaides, Officer A, ESF Unit, Ministry of labour and Social Insurance, [email protected] Other members: Belgium Flanders: ESF Agentschap Vlaanderen VZW Italy: Area d, Coordinamento Orientamento Istruzione Germany: Federal Ministry of Labour and Social Affairs Lithuania: Ministry of Social Security and Labour Spain: Ministry of Employment Sweden: ESF rådet Lifespan of Network (start and if appropriate, end date): September 2009 to June 2012 Network budget: Originally it was foreseen that the network would have 429,961 at its disposal but at the end they used less financial means ( ). Commission support group/person: Kimon Pappas, Unit B4/ Transnationality, EGF (European Globalisation adjustment Fund), DG Employment, Social Affairs and Equal Opportunities, [email protected] A97

174 AEIDL monitoring contact: Jyostna Patel eventually switched to Toby Johnson Section 2 Summary Aims and objectives of the Network: The main aim of the European Network on Youth Employment (ENYE) was to identify and mainstream good practices in youth entrepreneurship, mobility and educational and vocational guidance. These good practices would then form the basis of a Common Reference Framework. The ultimate objective was that the findings and recommendations of the ENYE would inform the ESF Operation Programmes in the subsequent programming period. In addition, it sought to bring together all national and European bodies handling Youth Employment issues and promote transnational cooperation. Specifically, the ENYE had the following over-arching objectives16: 1. To provide networking opportunities for pertinent national and European bodies 2. To exchange information and knowledge as well as innovative approaches regarding youth employment issues 3. To identify and transfer best practices with regard to youth employment 4. To design and develop common products suitable for dissemination 5. To create synergies between relevant national and European policies 6. To draw up proposals regarding national and European policies and practices in the field of youth employment. EU policy context: Combating youth unemployment goes to the heart of both ESF and EU employment policies. High unemployment rates amongst this group have put this theme once again on the top of the social policy agenda. Moreover, the LN activities promote youth employment and, therefore, are linked to the ESF objectives (CRF). Activities undertaken by the Network: 16 Preparatory phase report A98

175 Steering group meetings, Peer Reviews (2), study visits (3), baseline studies, benchmarking report, a common reference framework, a dissemination conference and a final evaluation. The ENYE drafted and presented the Common Reference Framework which was the main output of the network. It gave an overview of findings and recommendations of the network. The CRF was based on the benchmark report and three baseline studies which were carried out by the experts within the thematic working groups. To assess the quality of the CRF, two peer review sessions took place. Moreover, three thematic study visits were organised between November 2010 and January To promote the results of the network, a dissemination conference for 117 participants from 20 countries was organised in May In addition, the representatives of the LN participated in a number of other conferences including Hungarian Presidency Conference on Youth Employment, Entrepreneurship Education Conference (Finland), 5th International Conference on Entrepreneurship, Innovation and Regional Development (Sofia, Bulgaria 1-2 June 2012). The ENYE also contracted an external expert to carry out the evaluation exercise. Outputs of the Network: The LN sought to produce recommendations and findings which would inform the policy makers responsible for the ESF operation programmes in the new period. Following this logic, the target group of the Final Dissemination Conference was defined as people writing the programmes and EU2020 policy-makers, including people ERDF (AEIDL, Quarterly Report N 11). The conference was attended by 117 participants from 20 countries. More specifically, the outputs of the Network include: 1. Three baseline studies: Educational & Vocational Counselling, Mobility, and Youth Entrepreneurship; 2. The benchmarking report; 3. The Common Reference Framework; 4. CD with the baseline studies, the CRF, and the benchmarking report; 5. Two peer reviews with 10 reviewers, including two from non partner countries; 6. A substantial contribution was made by a policy officer responsible for youth employment in the EC. As a result of these contributions, the CRF was re-written (AEIDL, Quarterly Report N 11). 7. A website (no longer active) At the same time, the ENYE decided to scrap the CRF training manual for policy-makers and newsletters only the first issue was prepared. A99

176 Section 3 Relevance of the Network The relevance of the Network aims to ESF objectives and wider EU policy goals: Youth (un-) employment is a widely recognized issue and, in the Council Resolution of the European Cooperation Framework of 27 November 2009, Member States pledged to make employment the overall thematic priority for the period of January 2010 June 2011 (Baseline Study. Entrepreneurship). Youth employability plays an important part in the successful implementation of the Lisbon Strategy as the European Council identified promotion of employment together with the economic growth as two priority areas (Baseline Study. Educational & Vocational Counselling). ESF supports the employment of young people. Youth are in fact an important target group of ESF across the Member States. According to SFC data youth (between the ages of 15-24) constituted 29% of participants in ESF projects across the EU between 2007 and It is noteworthy, however, that both of the leading Member States, Greece and Cyprus did not reach a large group of youth through their ESF activities during this period (less than 15% of ESF participants were youth in both countries). Amongst the network members, Germany (with just over half of its participants), Spain and Belgium (both approximately a third) do have a significant share of youth as participants in their ESF projects. Participation in the network is not, however, solely based on the reach of ESF participants but also on the national need and priorities. With the exception of Germany, all network members experienced a youth unemployment rate higher than 10% between 2008 and In fact, all except Germany and Belgium experienced an unemployment rate of youth above 20% from 2009 onwards and were thereby well above the EU average. In that sense, all network members had good reason to be involved in the network. Considering the high participation of youth in ESF and the high levels of unemployment rates across the MS is indeed surprising that not more MS joined this network. The establishment of the network grew out of EQUAL activities. A 2012 conference in Cyprus and Corfu with the participants of the network on youth led to the decision to put forward an application for the network. During these conferences it was decided to provide specific focus to the network, to avoid having too broad of an objective. The three thematic areas are: Educational & Vocational Counselling, Mobility, and Youth Entrepreneurship. Each of these areas is closely related and complementary to the overarching theme of the LN as well as reflects current EU policies. The preparatory report of the network highlights that the three areas are indeed relevant to Members national priorities and experience and how they are therefore able to derive best practices on the subject. Youth entrepreneurship is a cornerstone for the implementation of European Youth Pact and Europe Youth Strategy and Europe 2020 strategy (Baseline Study. Entrepreneurship). Promoting the entrepreneurial mind-sets and initiative of young people has long been a priority for the EU. The 17 Evaluation of ESF support to lifelong learning p Eurostat lfsa_urgan A100

177 resolution of the Council (Official Journal C 196 of 12 July 2001) and the Communication from the Commission of 13 February 2006: Implementing the Community Lisbon Programme: Fostering entrepreneurial mind-sets through education and learning [COM (2006) 33 final] both stress the need to promote entrepreneurial culture among young s people. It is noteworthy that the majority of entrepreneurship related EU priorities and actions are closely related to education and fostering an entrepreneurial mind from a young age onwards. This would imply that cooperation with DG EAC or education related organisations and networks would have been relevant for this particular theme. The ENYE and its focus on youth mobility are closely related with several Europe-wide initiatives as well as national discussions. The mobility scheme for young entrepreneurs, for example, as well as the overall Youth on the Move Programme proposes series of actions to increase the employability and the access to the labour market by encouraging young people to study / train in other Member States (CRF). The EC treats mobility as a way to promote employment and reduce poverty (A Commission Action Plan for Skills and Mobility). Lastly, in the context of the European Qualification Framework and the need to support young people in developing the skills to adapt to the labour market needs, the vocational guidance theme has been recognized by the EC as an important provision. The 2000 Memorandum on lifelong learning, for example, identifies high quality vocational guidance counselling to be an important feature of sustainable growth. The relevance of Network activities to Network aims: The ENYE sought to influence policy makers responsible for drafting Operational Programmes. As such baseline studies, benchmark reports and the introduction of a Common Reference Framework correspond to the initial objectives of the learning network. The dissemination of information and work produced by the network took place through a conference where members of the target group from all Member States participated. A website was designed and initially updated to provide access to the studies and Common Reference Framework to those who did not participate. The activities as such were therefore not out of line with the priorities; however, they were not sufficient to reach the objective of the network. In addition to influencing policy, the network aimed to increase transnational cooperation. It therefore opened up the network for others to join and actively invited non-participating Member States to do so. However, this was unsuccessful. Several participating members also dropped out or reduced their participation. It may therefore be considered whether the activities to increase transnational cooperation were indeed the relevant or if other activities should have been chosen instead. Lastly, the network aimed to collect good practices. However, the activities have a rather small scope and focus on the organisations in the network, rather than look for wider EU networks on youth, entrepreneurship, mobility etc. A strong collaboration with existing EU NGOs, DG EAC, DG ENTR and units within DG EMPL tasked with mobility and entrepreneurship could have enhanced the quality and quantity of outputs related to the good practices and eventually impacted the policy recommendations. The good practices were selected during peer reviews entailing study visits. It can be A101

178 debated whether study visits are the most relevant activity for identifying good practices. How were the themes for LN activities selected? What is the added value of the LNs to ESF objectives? The ENYE is a continuation of a network originating from the EQUAL initiative and, thus, aims for the same objectives as its predecessor. To what extent is the network s theme and objectives complementary to other networks and organisations? The external evaluation suggests that the ENYE shared synergies with the Learning Networks Inclusive Entrepreneurship (COPIE), Empowerment & Inclusion and Age Management. This is confirmed through the interviews, although admittedly, not used to further develop such synergies. Section 4 Effectiveness of the Network Achievement of Network aims: According to the lead partner, the major vision and objectives of the learning network were achieved within the (restricted) time and budget. Indeed, baseline reports were formulated for all three focus areas and a comprehensive Common Reference Framework has been designed and disseminated. Several working groups took place (within each of the themes), study visits were organised and the reports were produced by external experts that were hired within each thematic area. The CRF was the result of consultations and expert facilitation. There were, however, significant delays in the start up of the network, particularly with the selection of experts, which hampered the outcomes. The network did not manage to develop the originally foreseen handbook for policy makers which aimed to train them on how to use the CRF. However, the decision to cancel this output was based on the recommendations from the peer review sessions rather than some other major obstacles. The network also planned to publish newsletters, which it eventually did not. It was realized that it took too much time and, therefore, the newsletters were dropped. AEIDL noticed that the network suffered from the lack of ambition to mainstream their findings and that it missed an opportunity for not having organised a meeting of relevant stakeholders in Brussels to present their findings and recommendations. According to the external evaluation, the network did not grow interest in youth employment issues among actors in the countries involved. Several of the interviewees also indicated a disappointment with the outcome of the network, particularly in terms of the quality of some of the outputs and the lack of involvement of other MS and relevant networks. There is indeed little evidence of the network s legacy. A102

179 Development of policies or subjects: The development of the CRF started with the preparation of three baseline studies covering each thematic learning area. Later these studies were synthesized into a benchmark report. Finally, after two peer review sessions, the final version of CRF was drafted. Quality of outputs: Despite generally reaching the goals of the network, there are some disputes about the quality of the products. Firstly, it appears that there is a difference between the three thematic areas, where one produced more and better quality products (not surprisingly also more engagement) than the others. According to the external evaluation report, hired by the lead partner, ENYE was successful through peer learning, exchange of good practices and knowledge, recommendations for the programming of relative interventions, interaction and a better understanding of what is happening in other countries on youth employment issues. The network leader, however, also acknowledged that one of the thematic areas was more successful than others, although they were satisfied with the three experts teams as well as the CRF expert in terms of the quality and providing valuable and considerable input. The experts were of special value in terms of brining a critical perspective to the work of the ENYE, and helped to improve the text of the CRF. For example, some pre-planned activities were dropped following the recommendations from peer experts. One of the baseline reports was not able to use a large enough sample size in the survey they used for the baseline, putting seriously into question the quality of the output. The report literally reads as follows. This refers to the fact that the data presented in this section should only be viewed as a very general and probably not very representative estimate of the views of experts employed in the field of career guidance. The reason for the aforementioned statement lies in the undersized sample that was utilized for the qualitative analysis. Furthermore, the network s website, which would be able to distribute the outputs to a larger audience, is no longer active. When it was active it was found to lack certain valuable characteristics, namely: 1) Poor aesthetic design low attractiveness, 2) Incompatibility issues in different browsers, 3) No mobile version available, 4) Various mistakes as far as HTML Validation is concerned (no HTML, CSS valid), 5) No accessibility by people with disabilities (W3C Accessibility Standards), 6) No contact form, 7) FAQ and HELP not in operation and Searching is only very basic, 8) (source: evaluation report). Interviewees further confirmed these findings and indicated that they were disappointed by the quality of some of the working groups and that it had been a reason to reduce their involvement/ not apply in the future in the network. Effectiveness of processes: The network was structured around three pillars: 1. The Steering Committee; 2. Three Working Groups: A103

180 a. Entrepreneurship b. Mobility c. Education and Vocational Counselling 3. A general Secretariat Each network partner could chose which working group they would be involved in and one of the members in each led the working groups. This decision further removed the lead partner from the activities, thereby playing a bigger role in the administrative organisation rather than in leading the way and maintaining momentum. This was left to the working group members, who were not equally involved and motivated and were not necessarily aware of the importance of their leadership. In essence, the structure resulted in an unintended waiting game. The network also had a website and an ing list. Three external experts were responsible for the learning areas in each working group. As mentioned previously, there was some delay at the start of the network, which resulted in the cancelation of meetings. Also, the peer review resulted in the cancellation of the handbook and the newsletters were not started. Considering that the sustainability of the network and the general outcome of the network is less than originally anticipated, there may have been some improvements in the processes that could have enhanced this network. It is however, not evident which changes could have taken place, other than perhaps, more frequent contact/meetings (phone/skype/ ) to ensure the engagement of all partners and increase the participation of relevant MS and other NGOs and networks. Little opportunity was taken to collaborate with other associated networks. There were some attempts to cooperate with other networks such as COPIE and ENSPIRE EU, yet no concrete plan was laid down. Representatives of both networks were invited to the final dissemination conference and made presentations. This could have potentially improved the outcomes of the network. The role of the monitor has taken on a bigger role for this network than perhaps others, in particular in developing such relations. Instances of learning: The learning strategy of the Network focused on the development, dissemination and adoption of a Common Reference Framework and, in the process, the participating members and experts did individually find areas of learning (content expertise). Although the external evaluation indicates that the network was successful in transnational learning through the peer learning, study visits, exchange of good practices and knowledge, recommendations for the programming of relative interventions, interaction and a better understanding of what is happening in other countries on youth employment issues, there is little evidence that this has actually enhanced policies or organisational decisions. A104

181 Policy transfer: There are no examples of policy transfer. In fact, despite the work that the network produced on youth mobility, for example, the baseline report and findings of the network s work were not used in the design of Your first EURES job, the youth mobility scheme for employment which was in the process of being designed whilst the network was conducting its work. There could have been synergies and transfer of learning and real discussions that could potentially have impacted the design of the programme. Impacts of the Network: The ultimate objective of the network was to inform the ESF Operational Programmes in the next programming period and, as such, it is difficult to assess its impact at this stage. All of the interviewees indicated that youth employment will be addressed in the next ESF OPs, and several indicated that they would take the CRF into consideration when formulating and assessing projects. It is most likely that from all the areas of work, the entrepreneurship recommendations may be picked up in the next programming period. The network produced a policy paper Initiative: Promoting Learning Mobility for Young People with Fewer Opportunities As a Support Tool Within the Transnational ESF Measures ( ) calling the network partners for their support by campaigning for the inclusion of mobility in the next programming period as a fundamental tool to increase the employability of young people. Attention for mobility is indeed growing in the Member States, with programmes such as Mobipro in Germany and similar schemes. However, there is no evidence that these activities are the result of the network. Section 5 Success of LN activities Reach and dissemination: The ENYE targeted policy makers and ESF managing authorities. The main findings and recommendations were presented during the Final Conference which was attended by 117 participants from 20 Member States. In addition, primary and secondary findings were presented at a number of other occasions: Hungarian Presidency Conference on Youth Employment (Hungary, 4 5 April 2011); The Entrepreneurship Education Conference (Finland, January 2012); ESF Learning Network Coordination Meeting and Exhibition (Northern Island, May 2012); 5th International Conference on Entrepreneurship, Innovation and Regional Development (Bulgaria, 1 2 June 2012). Moreover, the network was covered by the European Commission s Magazine on employment and social affairs Social Agenda (Issue No 28, February A105

182 2012). On the other hand, the external evaluation concludes that the ENYE did not raise national interest and awareness in the issues of youth employment and the network itself. Generally, the communication strategy was one of its weaknesses. Lessons learned from activities: The network produced a set of recommendations in all three areas which could be interpreted as success factor for policies within the three fields. Networking: The networking within networks did not occur or was temporary. Section 6 Efficiency of the LN Level of resources allocated: The total budget for the network was 477,735.00, which was intended to be used for the travel and subsistence of the Steering Group meetings, the hiring of the experts and the delivery of the outputs (reports, website). By the end of the network, 152,000 euro remained unspent due to less travel costs than required and less expert costs than foreseen, but also due to the cancellation of some activities (handbook on training the CRF). The network was managed by a secretariat of 1 and three coordination experts were hired to facilitate the working groups. Additional experts were recruited to work in learning areas, to produce the CRF and to evaluate the network. 10 experts were contracted for two peer review sessions. Proportionality of resources to achievements of activity: There were sufficient amount of experts involved to achieve the results, but insufficient manpower for the overall leadership and coordination of the network. This may also have contributed to the gaps in the results of the network. Changes to resources throughout the life of the network: The main change was less budget spent than foreseen. A106

183 Section 7 Likely sustainability Sustainability of Network activities: Few of the partners continue to collaborate on the three areas of work. The website has been turned off and the dissemination has been minimised (no training as originally foreseen). Links with other networks and organisations were minimised. The final conference may have the best opportunity for creating a sustainable outcome; the reach was relatively high and the types of participants could use the outcomes of the network to create change within their own organisation. There is however, no information available from the network members as to whether this proved to be the case. Sustainability of impacts and outcomes: Sustainability in this area is likely to be limited as outlined above. New policies developed at any governance level due to network: None identified. Section 8 ESF Calls for proposals Details of national calls: What national calls for proposals (ESF) were inspired by the network? Include resources There are no examples of ESF calls for proposals known by the network members or identified in any of the reports. Call for proposal projects: What projects were launched? What were their aims and objectives? N/A. Impacts of calls: What outcomes and impacts did the projects lead to? N/A Section 9 Strengths and weaknesses A107

184 The Network s key strengths: The members generally appreciate the quality of the external experts. It was also considered that the focus on three themes was a good choice, although not carried out to the end. By having such focus, it avoided discussions on what was most useful/relevant to youth and allowed the teams to delve into the details of three selected subjects. The Network s key weaknesses: The origin of the network was very much based on the outcomes of previous activities and less so on the internal motivation and drive of the participating members. Whilst the ideas for activities and objectives were shared by the members, the network members lacked the drive to carry them through. Much of the work was outsourced to experts and there was quite some delay between meetings without much communication in between, which resulted in a loss of momentum. The Steering Committee Meetings were therefore too infrequent. There were also no specific result indicators, which could have given the process more guidance. One of the frequent critiques of those commenting on the network is that the voice of young people was missing in the CRF. The lack of involving other organisations who could have represented the voice of young people is not just a critique of the CRF but also of the final conference and the baseline studies beforehand. Having achieved more involvement (not just of youth NGOs but also DG EAC/ENTR other units in DG EMPL) could have also produced more validation, for example for the surveys which did not generate sufficient response. Although the network members themselves could have been more proactive on this front, this is also an area where the Commission could have played a stronger role in terms of suggesting contact people, highlighting relevant conferences or facilitating contact in other ways. Examples of good practice: The good practice and success factors are closely linked for this LN. They were able to produce relatively high quality outputs under one of the thematic focuses because they had organised along three different themes. This meant that if work on one thematic area was slowed down or encountered obstacles, it did not strongly impact the other thematic focus areas. Success factors underpinning learning and transfer: Involving external experts, particularly in the peer review sessions and the focus brought to the three subjects from the beginning of the LN (although they were still too broad to be covered within the available time) were the main elements supporting the successes that were achieved. A108

185 A109

186 Annex: ENYE network intervention logic Rationale Aims and objectives Inputs Activities Results Impacts Combating youth unemployment goes to the heart of both ESF and EU employment policies. High unemployment rates amongst this group have put this theme once again on the top of the social policy agenda. Youth between the age of constituted 29% of participants in ESF projects across the EU between The LN focused on the three areas which provided the most potential for improving the situation of young people, namely Educational & Vocational Counseling, Mobility, and Youth Entrepreneurship. Overarching aim: To identify and mainstream good practices in youth entrepreneurship, mobility and educational and vocational guidance. through: - Learning from existing good practices -Developing a Common Reference Framework -Influence policy makers Financial Inputs: Financial support through the ESF Technical Assistance operational budget ( 429,961) Human resource and in-kind inputs Technical Secretariat (1 full-time) Network Members (8) Contributions by DG EMPL and DG EAC Main activities: - Meetings -Peer Reviews - Study visits -Baseline studies - A common reference framework - A dissemination conference - Final evaluation Programme level: - Benchmark information -Baseline studies - A common reference framework Individual members: - Gaining in knowledge, especially through the external experts Participating organisations: - a base to build on, especially on mobility and entrepreneurship Learning Network making a contribution to: - Knowledge and information through a common reference framework - A benchmark and baseline for the three areas of focus A110

187 EXOCOP Case study Section 1 General Information Learning Network name: Ex-Offenders Community of Practice (ExOCoP) Lead partner: Senate for Labour, Women, Health, Youth and Social Affairs of the Land of Bremen (Germany), but delegated to Senate of Justice and Constitution of the Land of Bremen Network coordinator: Jurgen Hillmer, Senate s Desk Officer for European Affairs of the Bremen Prison Regime and Criminology Other members: Network partners came both from ESF Authorities and Justice Departments: ESF authorities participated in Belgium (Wallonia and Federal), Germany (Federal level and Bremen), Greece, Hungary, Italy (national level and the regions of Emilia Romagna, Lombardy, Lazio and Tuscany), Netherlands, Romania, Slovenia, Spain (national level and Catalonia), UK (England and Northern Ireland). Justice Departments or Prison Administrations participated in Belgium, Germany (Federal level, Berlin, Schleswig Holstein, Bremen (representing the Northern Alliance), Hessen, Rhineland-Palatine and Saarland (representing the Southwest Region Cooperation)), Greece, Hungary, Italy, Poland, Portugal, Slovenia, UK (England) 19. The network also included European Professional Organisations as partners: EOEF - European Offender Employment Forum, EPRF - European Prison Regime Forum, ICPA - International Correctional & Prison Association, EPEA - European Prison Education Association, CEP - European Probation Conference, OIJJ - International Observatory on Juvenile Justice, EFK - Europäisches Forum für angewandte Kriminalpolitik, EuroPris The European Organisation of Prison and Correctional Services Ex-Offender Network self-presentation, EC website, (accessed ) 20 ExOCoP (2012): Lessons Learned no. 1, The ExOCoP Network, European Good Practice, the Berlin Declaration, June 2012, available here: (accessed ) A111

188 Lifespan of Network (start and if appropriate, end date): Network budget: The Community contributed 559, Financial resources were reallocated internally as a result of the financial crisis and in support of the participation of struggling Member States. Commission support group/person: Support group included DG EMPL, DG EAC and DG JUST AEIDL monitoring contact: Allen Mercer Section 2 Summary Aims and objectives of the Network: The overarching aim of the network was the development of a common strategy for the reintegration of exoffenders into society. This topic had been identified as important in preceding EQUAL activities and existing community initiatives. It was felt that there was a practical need to build an evidence base, learn, exchange and harmonise approaches in this policy area and develop a more standardised approach across Europe, given that previous programming periods had seen a number of projects for ex-offenders under ESF/ EQUAL funding. The objectives of the network were threefold 22 : Promoting mutual learning in a series of 18 workshops feeding into five larger seminars, leading to the final policy forum in Berlin. Here, the focus was on exchange, transfer and standardisation of expertise and the development of a European learning network. Developing a coherent European-wide strategy on the reintegration of ex-prisoners and ex-offenders, building on previous experiences of the EQUAL and NEON networks. Raising awareness for the importance of these strategies and actions to promote active labour market policies for vulnerable groups, by working closely with decision makers, stakeholders and NGOs across the areas of justice, labour and education. EU policy context: The EU has only limited competences in the area of prison matters, yet the network is relevant for the EU policy objective on social and labour market inclusion of marginalised groups. The Europe 2020 strategy aims to deliver inclusive growth and translates these objectives into targets in the areas of employment, education and poverty reduction. Prisoners and ex-offenders are strongly affected by social exclusion and face severe barriers to inclusion beyond their criminal history, such as detachment from the labour market, lack of employment history, qualifications and 21 Ex-Offender Network self-presentation, EC website, (accessed ) 22 ExOCoP (2012): Lessons Learned no. 1, The ExOCoP Network, European Good Practice, the Berlin Declaration, June 2012, available here: (accessed ) A112

189 often even basic education. Although there is no European re-integration strategy for ex-offenders, the ESF is the most important funding agency for innovations in the field of prison education and resettlement 23. Three ESF funding priority areas have a strong connection to the network theme: Improving the social inclusion of less-favoured persons, Improving human capital and Improving access to employment and sustainability. It is used to provide education, develop skills and facilitate the transition to work for ex-offenders. Previous funding periods have explored the strategies and structures of reintegration under EQUAL and NEON. Activities undertaken by the Network: The networks main activity was the organisation and implementation of a series of thematic workshops and seminars, built around face-to-face contact, the exchange of experiences and best practice cases and feeding into the development of recommendations on the reintegration of offenders and ex-offenders. These recommendations were then discussed and endorsed in the final policy forum in Berlin. Other sub-activities included the development of a European-wide knowledge management system/ platform and the production of a report on evaluation evidence and practice in the area of employment for ex-offenders. Outputs of the Network: The network delivered 20 workshops (two more than initially planned), 5 seminars and one final policy forum. The series of events followed a bottom-up approach: workshops were generally attended by experts and practitioners who discussed specific topics (more than 350 people attended in total), seminars were attended by higher level experts, practitioners and stakeholders and summarised aspects of several workshops (more than 250 people attended in total), while the final policy forum targeted high-level decision makers and presented the final recommendations and results (more than 110 participants). Additionally, the network released several publications summarizing the outcomes of its work: Lessons Learned 1 (report on network aims and activities, good practice examples and recommendations for the future), Lessons Learned 2 (report on Workshops), Lessons Learned 3 (report on seminars) Evaluation Sub-project Report Knowledge Management Sub-Project Report and knowledge management platform eu.exocop.org Section 3 Relevance of the Network The relevance of the Network aims to ESF objectives and wider EU policy goals: The network theme was relevant to the objective of promoting inclusive growth through the EU2020 strategy. Three ESF funding priority areas have a strong connection to the network theme: Improving the social 23 Ibid: page 6 A113

190 inclusion of less-favoured persons, Improving human capital and Improving access to employment and sustainability. Several interviewees emphasised the relevance of the network theme to ESF objectives in discussing the provision of education and employment opportunities for marginalised groups such as ex-offenders. It was felt that in times of economic crisis and continued downward pressure on public budgets, the ESF support had become a crucial source of funding for employment programmes for ex-offenders. It was also emphasised that the level of ESF expertise in the group was high, as all participating partners had to have experience with ex-offending projects within the ESF to be admitted to the network. Additionally, the network contributed to the overall aims and objectives of the Learning Networks in fostering transnational cooperation under the ESF. It discussed education, skills and employment policy and reforms by bringing together people, knowledge and practice in the field of employment, social inclusion and training. The relevance of Network activities to Network aims: The network activities constituted a clearly structured pathway to reach the network s main aim to develop a common strategy for the reintegration of ex-offenders into society and were very relevant. It approached its aim from the bottomup by discussing specific topics in 20 workshops with practitioners, then summarizing their findings with medium-level experts in 5 seminars and by finally formalising and presenting the strategy in the final Berlin declaration. The sub-activities on evaluation and knowledge management provided relevant inputs to the networks aims through broadening the evidence base and dissemination. How were the themes for LN activities selected? What is the added value of the LNs to ESF objectives? The choice of themes built both on prior developments and experiences (e.g. in the EQUAL network) and consultation with network partners on which themes were felt to be most relevant across a number of Member States. It added to ESF objectives by sharing knowledge and developing strategies on the integration of ex-offenders into European labour markets and societies, which is in line with ESF objective to help disadvantaged groups to get into jobs, to enhance skills and access employment. To what extent is the network s theme and objectives complementary to other networks and organisations? Although a relatively niche topic, the LN overlapped with and participated in activities of other networks for dissemination purposes, e.g. youth employment or transnational cooperation. The close cooperation of the network with other European Bodies also enabled participation and dissemination in a number of European conferences. However, several interviewees were not aware of the cooperation and dissemination activities of the network. Section 4 Effectiveness of the Network Achievement of Network aims: The network achieved its main aim, which was to develop a common strategy for the reintegration of ex-offenders into society. The Berlin declaration was discussed and signed in the final policy forum, which was attended by more than 110 high-level European policymakers. The network also facilitated mutual learning, knowledge transfer and exchange in a series of 20 seminars and 5 workshops, which were A114

191 attended by more than 350 and 250 people respectively. This success was made possible through its structured bottom-up approach, which effectively worked towards the delivery of the final output. Although the network successfully delivered its final output, it is felt the Berlin declaration has yet to be implemented at Member State and European level and there is little policy impact of the network s work. Development of policies or subjects: The delivery plan set out at the start of the network was followed through. Although the economic crisis hampered the mobility of the network members and reduced the number of participants at some events, it did not change the nature or focus of the activities. Quality of outputs: The quality of outputs was generally high. The activities are comprehensively documented, their findings summarized (in lessons learned 1-2, the evaluation sub-project report and knowledge management sub-project report) and reports are easily accessible through the network website. According to AEIDL reports and interviewee accounts, the quality of workshops varied due to the fact that there was initially no common structure and template and they were organised by different people. The AEIDL report also points to flaws in the recommendation production process. Effectiveness of processes: Most interviewees felt that the network was centralised, very well organised, while demands on network partners were clearly communicated. However, the large and diverse network structure was seen as a challenge and significant time was spend to support administration (also financially by rearranging the budget in response to the financial crisis). One steering group participant pointed out that the work was challenging at times due to personnel changes and leadership was at times missing at steering group level, due to the coordinators weaknesses (centralisation of activities and lack of English communication skills) and later absence. Instances of learning: Learning took place at individual and national/ regional policy level. From an individual perspective, all interviewees felt that the participation in the learning network fostered a sense of commonality, increased the understanding of common challenges and gave important insights into what is working and not working in practice across Europe. It was also felt that individuals profited with regards to professional development due to knowledge and experience gained at the workshops. Additionally, instances of learning could be seen at national level 24, where they mainly related to an intangible increased understanding of good practice in the re-integration of ex-offenders. Some descriptions of these impacts are rather vague, others are more specific on what was learnt, like the following ones: In Poland, ExOCoP has shown the importance of focusing on cooperation with NGOs and other institutions which can help with the Education, Training and Employment of offenders. 24 AEIDL (2012), Learning Network Ex-Offenders Community of Practice (ExOCoP), Final Report A115

192 Such impacts are mainly at policy level, however the following one points to the organisation and individual levels: In Portugal, the prison services acknowledged that sharing common viewpoints regarding the approaches to specific problems and the integration of experiences and results contributed to modernising the intervention models and, consequently, to increasing the system s effectiveness. They have also contributed decisively to stimulating the interest of professional members of staff, and consequently to their higher commitment and better performance at work. One interviewee pointed out that learning at European level had taken place, in particular through demonstrating that it was possible to run such a large network in an inclusive and effective manner. Policy transfer: The policy area the network operated in constituted a challenge to policy transfer. Legal frameworks and penal systems vary greatly in Europe and practices are strongly embedded in these contexts, which makes transferability difficult. Consequently, the number of concrete examples of transferred policy practices is low. However, ExOCoP made countries more aware of holistic approaches to re-integrating ex-offenders and enabled them to gain insights into other practices, which may have an influence on policy design in the future. Sometimes, findings of ExOCoP were integrated in national strategies, for example in the Romanian national reintegration strategy. Impacts of the Network: Short-term impacts were felt at European, national and individual level: EU: The network contributed to a greenbook by the European Commission in the frame of the Stockholm Program, which proposes a standardised European prison education programme and the continuation of a European Learning Network. Interviewees also emphasised that the network lead to greater understanding of the complexity of resettlement at European level and amongst European stakeholders. National/ Regional: Positive short-term impacts are claimed by AEIDL 25 for eight of the countries or regions that were members of ExOCoP and also especially in Germany, where the two main networks continue to grow and jointly develop their activities. Some of them are described in rather vague terms and most revolve around general information sharing and intelligence boosting. Individual: According to interviewees, short-term impacts mainly referred to the acquisition of contacts and networks: It also had a positive impact on the experience and knowledge level of individual participants and was important for their professional development. Most long-term impacts of the network were formulated in hypothetical terms, as long-term impacts can not yet be observed and it was felt that lessons learnt have yet to be incorporated into policy making. It was felt that the network should have long-term impacts on the development of specific programmes and funding for the sub-population of ex-offenders. The network was also believed to lead to greater long-term cooperation between countries and the use of synergies. 25 ibid A116

193 Section 5 Success of LN activities Reach and dissemination: Through the bottom-up structure of the LN s main activities different specialist target groups were effectively included and reached: workshops brought together a large number of practitioners, seminars gathered administrators and mid-level policy-makers, while the final policy forum addressed high-level European and national policy makers. Additionally, the network cooperated with European Institutions and Professional Organisations in conferences, where ExOCoP was able to disseminate its work and findings. One example is in the European conference on Prison Education Pathways to Inclusion Strengthening European Cooperation in Prison Education and Training ( ), were ExOCoP was able to reach more than 200 participants from 30 countries in cooperation with DG Education and Culture. Apart from these activities, the most effective dissemination strategy seems to have been the dissemination via national partners who publicised the network findings and activities through their own channels (mailing lists, websites, and national networks). The dissemination through ExOCoP s own channels, such as the knowledge management system, was rather limited. Lessons learned from activities: The most important message network participants took-away was on the crucial importance of cooperative and networked approaches to resettlement 26. It was understood that the successful reintegration of ex-offenders relies on collaboration and inputs of different departments and institutions. Moreover, the bottom-up approach allowed participants to understand the breath of topics involved in resettlement and see reintegration as a step-wise process from gaining education and skills, over work experience and the assisted transition into employment. Networking: A number of network participants continue to participate in the new LN on Active Inclusion led by NOMS UK. One of the three thematic strands revolves around marginalised groups, such as ex-offenders. There are also discussions on a prison focused LN, but this has not materialised yet. Several countries also cooperate in Europris (The European Organisation of Prison and Correctional Services), founded in Spain has established a national learning network, which took inspiration from ExOCoP. According to a number of interviewees, a number of countries are cooperating in bilateral of multilateral informal settings, with contacts and partners facilitated through the network. Portugal and NOMS for example are running a twin-project with the beneficiary Turkey. Yet, some interviewees felt that there remained an open question over the sustainability of the network and follow on networking activities. 26 ExOCoP (2012): Lessons Learned no. 1, The ExOCoP Network, European Good Practice, the Berlin Declaration, June 2012, available here: (accessed ) A117

194 Section 6 Efficiency of the LN Level of resources allocated: The network was allocated 559,920 27, less than initially requested. It was felt that the network could have achieved more with larger financial resources. The input of time and human resources is difficult to estimate and no data is available. The main inputs were as follows: The network was coordinated by one full time staff with the support of two part-time members of staff. Members of the Steering Committee met nine times, with meetings lasting between one and two days. Additionally, 20 workshops, five seminars and the final policy forum were organised and attended. More than 350 people attended the workshops, more than 250 the seminars and around 110 the final policy forum. Time and resource input was made to the two subprojects. Proportionality of resources to achievements of activity: The network reached its main objectives and delivered a tangible final output with the Berlin declaration. It also financed the travel of a significant amount of practitioners and facilitated their knowledge exchange. Overall, the network delivered good value for money, yet more could have been achieved with additional resources. Changes to resources throughout the life of the network: The financial crisis put pressures on the network budget and some budget had to be shifted to travel costs in order to support struggling Member States. Consequently, the sub-activities evaluation and knowledge management were deemphasised. Section 7 Likely sustainability Sustainability of Network activities: Part of the network s work is continued in the new learning network Active Inclusion and several participant countries cooperate in informal and formal networks, where they continue to exchange good practices and find solutions to common challenges. As a follow-up to the Policy Forum, the United Nations Office on Drugs and Crime approached ExOCoP to integrate its findings into a Handbook on Reintegration and a database. Sustainability of impacts and outcomes: The Berlin declaration was a somewhat static output, laying down an important fundamental strategy on the reintegration of ex-offenders in Europe, but lacking a concrete follow-up due to the fact that it was not legally binding. Although it was felt that there was a true buy-in of European and national stakeholders through the Berlin declaration, some interviewees felt that there was a lack of subsequent revision 27 Ex-Offender Network self-presentation, EC website, (accessed ) A118

195 and enactment of policies. As there was a focus on raising awareness and formulating a strategy, sustainable impacts might develop later when knowledge gained in the network is fed into the policy-making process. For example, the Portuguese prison administration is currently running a big change progress. This was not a consequence of the network, but findings of ExOCoP are now incorporated and show the possibility of a sustained impact of ExOCoP in the future. The mutual learning aspect of the network continues to exist. Several forms of cooperation have been established, bi-laterally, multilaterally or within one country itself and the participants profit from increased networks and partnerships. New policies developed at any governance level due to network: According to the network, it led to enhanced cooperation between justice, education and employment administrations at regional, national and transnational level 28. This can be illustrated by the cooperation of the EC Directorates Justice, EMPL and EAC in supporting the LN. Based on the AEIDL final report 29, the clearest signs of the project s effects having taken roots and the best perspectives for sustainability are in the country of the lead partner Germany. In Germany, the five Länder that were partners in the network acted effectively as ambassadors to push forward the resettlement agenda in regional and national policy and practice, with various events intensifying involvement. They succeeded in getting both the ESF MA and the German Federal Administration to sign the Berlin declaration, they lobbied the ESF MA to adopt ExOcoP ideas and they now run spinoff projects of ExOCoP implemented at national level. Moreover, the Länder administrations took over the funding of some activities, such as the E-Lis project. A national network inspired by ExOCoP has also been established in Spain. Section 8 ESF Calls for proposals Details of national calls: None Call for proposal projects: Impacts of calls: 28 ExOCoP (2012): Lessons Learned no. 1, The ExOCoP Network, European Good Practice, the Berlin Declaration, June 2012, available here: (accessed ) 29 AEIDL (2012), Learning Network Ex-Offenders Community of Practice (ExOCoP), Final Report A119

196 Section 9 Strengths and weaknesses This section is to give an overview of both the positive and negative aspects of the Network and its work. The Network s key strengths: The success of the network was grounded in a number of key strengths: Coverage: ExOCoP was one of the largest learning networks, with 40 partners having signed the Memorandum of Understanding/ Letter of Intent at the end of the preparation phase. It covered 13 European countries through the participation of their ESF MAs and Justice Departments directly and around 75% of the European prison population through the partnering of European professional organisations. Commitment: It is reported throughout that the commitment of the network participants was very strong. This is exemplified by a number of authorities supporting the network financially by paying expenses and hosting workshops, for example. Structured and goal-oriented activities: The main activities of the network (workshops, seminars, policy forum) were from the outset organised in a straightforward and purposeful way, building on each other and leading towards the final policy forum and declaration in Berlin. Bottom-up approach: The foundations of the network s final output and recommendations were laid in 20 topical workshops, where practitioners exchanged knowledge and practices. These were then summarized and discussed at an analytical level during five seminars. The final policy forum presented the work to high-level European and national decision-makers and formulated recommendations. This implementation strategy effectively brought together practitioner knowledge and policy-makers decision making power. Cooperation: The network s activities and findings were presented at several European conferences, opening its work to a wider audience beyond the network s members. This was facilitated through cooperation with European professional associations, the European Commission DGs. The Network s key weaknesses: Weaker aspects of the network included operational and strategic aspects: Late implementation of a standardised template for workshops: It was felt that some workshops were unstructured and that an earlier implementation of a common approach to the workshops and seminars would have helped the later synthesis of findings and the development of recommendations Lack of follow-up: It was felt that the network lacked strategic follow-up activities outside of the network s main activity strand (workshops, seminars, policy forum). There was no newsletter, for example. This might be due to the fact that the funding was shifted away from the knowledge management sub-project in the course of the project, which was to facilitate on-going knowledge exchange on a shared platform. Centralisation: Some functions should have been decentralised; for example the drawing of recommendations from the workshops. These A120

197 were developed by the Technical Secretariat on the basis of workshop reports. Moreover, it was felt that the rotation of the steering group chair should have been introduced earlier and not been centralised by the coordinator. Lack of policy impact: The goal of the network was to formulate standards in re-integration of ex-offenders in Europe and did not have a strong focus on the implementation of these recommendations. However, participants of the network felt that although ExOCoP reached its aims, it did not have a great enough impact on policy making and there was little measurable change since the Berlin declaration. Examples of good practice: Good practices of this learning network include purposeful activities, which logically build on each other, a large number of face-to-face meetings enabling mutual learning and real exchange and an inclusive approach integrating a large number of Member States and stakeholders. Success factors underpinning learning and transfer: The network profited from an inclusive and cooperative approach integrating a large number of Member States and relevant European stakeholders, providing a large knowledge base for the exchange of good practices. It also benefited from a large number of face-to-face exchanges of practitioners, experts and policy-makers (which it partially funded). The close cooperation with three Directorates at the EC enabled a truly cross-cutting approach to re-integration of ex-offenders and a broad base for cooperation and support. A121

198 Annex: ExOCoP intervention logic Rationale Aims and objectives Inputs Activities Results Impacts Prisoners and ex-offenders are strongly affected by social exclusion and face severe barriers to inclusion beyond their criminal history, such as detachment from the labour market, lack of employment history, qualifications and basic education. The ESF provides reintegration support to this target group and is the most important funding agency for innovations in the field of prison education and resettlement. The LN is used as a tool of transnational cooperation and exchanges of good practice, hence supporting the implementation, delivery and outcomes of ESF work on the re-integration of exoffenders. Overarching aim: The development of a common strategy for the reintegration of ex-offenders into society. through: - mutual face-to-face learning in a series of 18 workshops, 5 seminars and 1 final policy forum - formulation of a coherent European wide strategy on the reintegration of exprisoners and ex-offenders - raising awareness for the importance of these strategies and actions to promote active labour market policies for vulnerable groups Financial Inputs: Financial support through the ESF Technical Assistance operational budget ( 559,920) Additional financial support through the network members financing some travel and workshop costs Human resource and in-kind inputs Technical Secretariat (one full-time, two part-time staff) Network Member s participation in 9 Steering groups (1-2 days) Organisation of and participation in 20 workshops, 5 seminars and one final policy forum (Member States in-kind contributions) Contributions by DG Just, DG EMPL and DG EAC Contributions of European professional organisations Main activities: Organisation and Implementation of 20 workshops, 5 seminars and one final policy forum Sub-activities: 1. Development of an ExOCoP Knowledge Management Platform 2. Evaluation Sub-Project: Provide standards of evaluation for the work with ex-offenders Programme level: - Fine-tuning of arrangements promoting transnational exchange and co-operation - Stakeholder and practitioner networks at OP level - Launching of action plans to base ESF support on agreed good practice - Common approaches to monitoring, evaluation or reporting Individual members: - helping each other in solving problems at operational and strategic levels - providing stimuli to reflect on practice and improve it - applying common, tested, working methods - providing opportunities for staff exchange - gaining professional skills and recognition Participating organisations: - keeping up with developments across Europe - getting easy access to a pool of competencies to respond faster to emerging policy needs; - using common successfully tested tools - developing and assuring professional competence - speeding up the use and integration of good practice gained elsewhere - developing a common voice on the issues at stake Learning Network making a contribution to: - strengthening the capacity to innovate in the sector - identifying and assessing issues and solutions for reforms in this policy area - improving transnational cooperation and coordination between the education, employment justice administrations -raising awareness for the complexity of resettlement at European and Member State level A122

199 GenderCoP case study Section 1 General Information Learning Network name: European Community of Practice on Gender Mainstreaming ( GenderCoP ) Lead partner: Swedish ESF Managing Authority Network coordinator: Bengt Nilsson Other members: Austria: ESF Managing Authority Belgium: Flanders ESF Managing Authority Belgium: French speaking ESF Managing Authority Czech Republic: ESF Managing Authority Germany: ESF Managing Authority Germany: Agency for Gender Equality within the ESF Estonia: ESF Managing Authority Spain: ESF Managing Authority; Women s Institute Finland: ESF Managing Authority France: ESF Managing Authority Greece: ESF Managing Authority Italy: ESF Managing Authority Italy: ISFOL Italy: Province of Bolzano: ESF Managing Authority Poland: ESF Managing Authority Portugal: Commission for Citizenship and Gender Equality Sweden The Regional Development Fund in Sweden (Tillväxtverket) Sweden: Process Support in Gender Mainstreaming (ESF Jämt); Equality Ombudsman A123

200 Lifespan of Network (start and if appropriate, end date): 15 March January The network was also successful in the new call for proposals and received funding for a further two years (Feb 2013-Feb 2015). Network budget: Total budget 728,117 (including 75,139 of co-financed contributions from Sweden, Austria, Estonia, Finland and Portugal) Commission support group/person: Kimon Pappas, Marie-Anne Paraskevas, then Sonia de Melo Xavier AEIDL monitoring contact: Jyostna Patel Section 2 Summary Aims and objectives of the Network: The overall aim of the network was to create a solid community of learners dedicated to integrating the gender dimension into the ESF programmes ( ) and the related policy fields of the Lisbon agenda and EU2020. At the end of its lifespan, it intended to move towards making gender mainstreaming an integral part of ESF management, all the way from planning, programming, implementing to monitoring and evaluation. The theme was chosen to take forward and further develop work on gender mainstreaming begun under the EQUAL programme. Partners involved in that work recognised that that, despite the existence of principles on gender mainstreaming (GM), support and tools were required in order for them to be implemented. This led to the development of specific objectives for the network around: the achievement of: an ESF Standard (guidelines) on gender mainstreaming; a training and capacity-building programme; a community of learners on gender mainstreaming within the ESF; a network of contact points in ESF; increasing cooperation between ESF and other key actors; a communication platform on gender mainstreaming within the ESF (and beyond); baseline mapping of gender mainstreaming in the ESF. EU policy context: Gender equality is a key theme running through many EU level policies and strategies, whether it is the focus of those strategies or functions as a cross-cutting or horizontal theme throughout the strategy or policy area. The key document setting out the Commission s workplan in this area is the Strategy for equality between women and men The strategy outlines key actions for this period encompassing the economic independence of women; equal pay; equality in decision making; dignity, integrity and an end to gender based violence; gender equality in EU external policy actions; and promoting gender as a cross-cutting horizontal issue. Reflecting the focus of the GenderCoP network, the strategy articulates that Gender mainstreaming will be implemented as an integral part of the Commission s policymaking. A124

201 As noted, gender also functions as a key theme running through other high level EU policies and strategies in addition to forming part of the structural funds regulations. Key examples include: Art. 16 of the Regulation which lays down general provisions on the Structural Funds (No. 1083/2006) which states that The Member States and the Commission shall ensure that equality between men and women and the integration of the gender perspective is promoted during the various stages of implementation of the Funds. The preamble to the EC Regulation (No.1081/2006) which articulates that The Member States and the Commission are to ensure that the implementation of the priorities financed by the ESF [ ] contribute to the promotion of equality and the elimination of inequalities between women and men. A gender mainstreaming approach should be combined with specific action to increase the sustainable participation and progress of women in employment. The EU2020 strategy which states that Policies to promote gender equality will be needed to increase labour force participation thus adding to growth and social cohesion. Activities undertaken by the Network: Key activities undertaken by the network were: Baseline mapping of gender mainstreaming (GM) in ESF in the EU27 Establishment of a network of actors working on GM in the ESF/Structural Funds Establishment of 5 working groups: evaluation; training & capacity-building; awareness-raising; policy impact; sustainability Events: Gender pay gap (Tallinn, May 2011), GM in next funding period (Brussels, May 2011), GM support structures (Stockholm, October 2011), GM in evaluation (January 2012), Gendering the Flagships (Brussels, June 2012), Gender budgeting (Berlin, September 2012), Launch of the Standard (Leuven, January 2013) Steering Group meetings (including presentation of Member State GM strategies): Rome, Paris, Tallinn, Lisbon, Helsinki, Stockholm, Madrid, Warsaw, Brussels Position papers: EU2020 and Flagships, 5th Cohesion Report, new Structural Funds regulations Policy outreach to key actors: EIGE, QUING, FEMM Committee, EC (ESF and Gender Equality Units), EWL, European Parliament; A125

202 Communication platforms: on-line community of practice (Projectplace), website open to all ( Production of on-line guidelines for GM in the ESF policy cycles: the Standard Development of a Social Game available on Facebook ( Outputs of the Network: The key outputs of the network can be summarised as follows: The ESF Standard on GM was produced ( and provided a key focal point for learning throughout the network activities; A community of learners on gender mainstreaming within the ESF was established (online Community of Practice with 250+ members); A network of contact points in ESF was established via the network Steering Committee and wider contacts made at events; Increased cooperation between ESF and other key actors on GM was achieved through policy outreach to key actors (see above activities); A communication platform on gender mainstreaming within the ESF was set up via the online Community of Practice (CoP) + web site; Baseline mapping of gender mainstreaming in the ESF was achieved in 2010 (see above). The development of a training and capacity-building programme was the only aim not achieved: a decision was made to delay the development until the extension period ( ), using the Standard as a basis for a programme. Section 3 Relevance of the Network The relevance of the Network aims to ESF objectives and wider EU policy goals: The network is clearly highly relevant in terms of ESF and wider EU policy goals in terms of gender equality and gender mainstreaming, in particular: A126

203 The provisions set out in Strategy for equality between women and men (COM(2010) 491 final): Gender mainstreaming will be implemented as an integral part of the Commission s policymaking ; Art. 16 of the Regulation laying down general provisions on the Structural Funds (No. 1083/2006): The Member States and the Commission shall ensure that equality between men and women and the integration of the gender perspective is promoted during the various stages of implementation of the Funds ; The preamble to the EC Regulation (No.1081/2006): The Member States and the Commission are to ensure that the implementation of the priorities financed by the ESF [ ] contribute to the promotion of equality and the elimination of inequalities between women and men. A gender mainstreaming approach should be combined with specific action to increase the sustainable participation and progress of women in employment ; The EU2020 strategy: Policies to promote gender equality will be needed to increase labour force participation thus adding to growth and social cohesion. For partners involved in the network, its relevance principally stemmed from the objective to promote gender mainstreaming and the development, and hopefully subsequent adoption of a common standard, into ESF programmes. The common standard was seen as a key practical way to promote the achievement of gender mainstreaming and hence contribute directly to the objectives of the ESF and wider EU policy goals outlined. The relevance of Network activities to Network aims: Partners involved in the network felt that there was a clear and direct link between the overall aims of the network around creating a community of learners and promoting gender mainstreaming and the activities developed to advance these aims. The process of theme / activity selection (see below) and its basis in existing experience and research / baseline study development, allied to partners work in refining the activities, were seen as ensuring this connection. Certainly the focus on developing a common standard as a practical, implementable tool would seem to closely reflect the aims of the network, as would the activities aimed at promoting gender mainstreaming and creating a community of practice around it (in particular, the development of an online community of practice, the activities developed around policy outreach, and the development of the communication platform on gender mainstreaming and website). How were the themes for LN activities selected? What is the added value of the LNs to ESF objectives? A127

204 Originally a predecessor network led by Finland ran as part of the EQUAL programme under the previous programming period. The participating Member States felt that, despite the existence of principles on gender mainstreaming (GM), support and tools were required in order for them to be implemented. Plans to lead the successor network were originally taken forward by Portugal but the organisation concerned was unable to lead, due to its status as an intermediary body, so Sweden took the lead. A steering group to narrow the focus of the LN then was developed involving core partners. The final selection of the network themes and approach was influenced by research (including the development of a baseline study) and a meeting using a world café approach to discuss issues and refine the approach on the part of those involved. This was seen by both those involved in leading the network and partners as an effective approach to inclusively developing the LN, along with ensuring that it was based on evidence from the previous EQUAL network and the identification of existing shortcomings related to gender mainstreaming. Subsequent to the initial development phase, the network set up a series of 5 working groups to further refine and develop its activities. To what extent is the network s theme and objectives complementary to other networks and organisations? Given the horizontal, cross cutting nature of the network s focus (i.e. gender mainstreaming) there is a clear a priori link to all of the other learning networks and organisations working in the EU policy and delivery fields. Stakeholders involved in the network also outlined that the cross membership of some partners with other networks and organisations relevant to the fields of ESF and gender quality had helped to develop and confirm these linkages, though it was acknowledged that opportunities for directly working with other networks on these themes could not always be realised (for reasons of time and resources principally). Section 4 Effectiveness of the Network Achievement of Network aims: Documentary evidence and that drawn from the stakeholder consultations indicates that the network can be considered to have been effective in achieving the majority of its aims. As noted above, the exception to this concerns the decision taken to delay the development of a training and capacity-building programme on the grounds that the fully developed and finalised common standard would offer the best and most solid basis for this. Network partners consulted agreed with this decision and felt that it was logical, particularly in the context of the extension to the network to In terms of the factors lying behind this success in achieving network aims, stakeholders felt that a combination of good co-ordination and leadership, a solid organisational structure and effective organisational processes, committed partners, good relationships between those running the network and the Commission, and the nature of the activities themselves were key in supporting the achievements of the network. In respect of activities, the peer reviews and conferences were seen as enabling the discussion and sharing of learning in terms of mainstreaming gender considerations. It was also noted that that the common standard has definite potential to achieve the aim of encouraging the incorporation of A128

205 gender across planning and delivery but that this will depend on how well it is used / taken up (though indications on this are positive and four countries have agreed to act as pilots for the purpose of using / incorporating the standard). In this sense it was thus acknowledged that it is still relatively early to fully judge the extent of the achievement of all of the network s aims, but in terms of stated outputs the network should be viewed as being effective from this perspective. In terms of operational contributions to the achievement of network aims, the network was viewed by those involved as having achieved a good balance between Managing Authority representatives and experts along with good Member State coverage. High levels of commitment on the part of partners, with no-one dropping out, was also viewed as a supportive factor in achieving aims (though there were some issues in terms of representatives of countries affected by the economic crisis being able to attend meetings, events etc.). The development and organisation of the network was also highlighted from this perspective. Those running the network took what they call a democratic approach to involving partners as widely as possible in the development and planning of activities; this was seen as important in encouraging ownership and commitment and hence ensuring that objectives and aims were met. Good relationships developed with the Commission were also an important factor here and the network was pro-active in seeking the engagement of the Commission Support Group established. As noted, partners in the network also felt that an effective organisational structure, allied to good leadership and co-ordination on the part of the lead partner, was important in achieving aims. Development of policies or subjects: The development of activities and projects was seen by those involved in the network as both evidence based and inclusive, given the processes used to develop them. As outlined above, the development of a baseline study, use of the world café methodology, and development of working groups charged with the further development and refinement of activities were seen as key in this. Quality of outputs: Evidence from the monitoring reports and the network final report, allied to observation by the researchers and evidence gathered from stakeholders, indicates that the quality of outputs produced by the network was generally high to very high. In particular it appears that: The events were generally of a very high standard encompassing very relevant topics (linked to current policy priorities or concrete ESF processes (i.e. evaluation, GM support structures), achieving a high participation rate of relevant stakeholders (EU, national, regional), including high level speakers (EC, academic experts, good practice examples from ESF), and were generally felt that have had very good organisation. Position papers were seen as providing valuable inputs to key EC decision-making processes and building the case for better GM in, for A129

206 example, Cohesion Policy, EU2020, and the new ESF regulations. The common standard is viewed as a comprehensive and well-structured tool providing guidelines and examples of successful implementation of GM in the 4 main policy cycles of ESF. The website is attractive, well-structure and regularly updated with news. The Community of Practice has attracted 250+ users, although does not attract a lot of comment from and discussion between members. However, in respect of events there was some recognition by those involved in the network that it was often difficult to fit everything in in terms of the time available, despite the organisation of the events themselves being good. In addition, a couple of the wider partners consulted for the case study did note that the time taken to develop the common standard in part stemmed from the incomplete (or not of adequate quality) early drafts of the document. Effectiveness of processes: As indicated by both documentary research and the perspectives of those involved, the network can be considered to have set up a very good basis for the organising its work which in turn is widely viewed by partners as contributing to the successful achievement of aims and objectives. While allowing a large number of interested parties to become part of the wider community of practice (CoP) (notably via the online tool), the network leads introduced a system of "concentric circles" of involvement, with a core Steering Group of the dedicated Member States who signed a clear Memorandum of Understanding on their roles and input, and a CoP Secretariat composed of the Network Coordinators and 2 content experts. Furthermore, in order to avoid too many large meetings, five operational working groups were established at an early stage, corresponding to five target areas of the CoP (reflecting the core objectives outlined above), each with clear leaders and supporting members. A Reporting Committee was set up to bring together the leaders of the five working groups on a regular basis. There was also a clear calendar of meetings, and regular reminders from the CoP Secretariat. However, despite this positive basis, it is evident that certain processes did pose challenges throughout the life of the network. In particular, the Steering Group was acknowledged as being too big (involving over 30 members) to be an effective decision-making entity. As such, this role was adopted by the Reporting Committee (which only involved the CoP Secretariat + leaders of the 5 working groups). The working groups themselves however were also cited as being challenging since they varied greatly in terms of numbers of members, degrees of expertise, and resources (for example, some groups paid for experts from their own MS budgets). Despite these issues the overall perspective of those consulted was positive in respect of the effectiveness of processes as noted. A130

207 Instances of learning: Evidence suggests that the GenderCoP provided an important vehicle for learning at different levels. Individual learning was a key feature of the network and took place through peer learning of Steering Group members and wider CoP members (i.e. sharing of good practice and tools, brainstorming), inputs on specific topics (e.g. gender budgeting) from expert speakers and workshop exchanges at events, working group activities for the development of the network outputs, and linking to other key stakeholders in GM. One of the key major achievements of the network appears to be that it was effective in ensuring that learning took place for all network members, whatever their level of expertise on gender or ESF issues. Linked to this, organisational learning was targeted through the participation of wider stakeholders at events, and the development (and dissemination) of the common standard on GM. As a result of the network, events were organised in certain organisations such as a training day at the Belgium (French-speaking) ESF Managing Authority. A number of ESF Managing Authorities have also agreed to act as pilot countries for implementation of the common standard, including the Czech Republic, Spain and Sweden, which suggests that on-going organisational learning will accrue. At the policy level it is more difficult to identify concrete instances of learning per se, though the network members interviewed felt that such learning had undoubtedly occurred, and policy influence had resulted from it (see below), but that the nature of such effects was inevitably diffuse and difficult to pin down. Policy transfer: Improvement of policies on gender mainstreaming was the ultimate aim of this network, and was targeted through all its activities. However, it appears that effects in this area are more accurately described as policy influencing rather than occasioning policy transfer per se. Examples offered by stakeholders included the role of the network in influencing the writing of the new ESF programme in Sweden and in particular the gender dimensions of it. It was also noted that the activities of the working groups identified challenges, gaps and things that needed doing in terms of enhancing the gender focus and mainstreaming of policies which has influenced OP development elsewhere within the Member States and regions of participating partners. Equally, in respect of the common standard, the aim is that after piloting in 4 Member States the standard will be further refined / sharpened and then rolled out to all Member States through the Commission. At this stage therefore, while there are few concrete examples of policy transfer to date, it seems fair to conclude that the network has succeeded in some instances of policy influencing and that the potential to extend this in future is certainly apparent. Impacts of the Network: Key short term impacts of the network include: individual and organisational learning; influence on key EU level processes relating to the new Structural A131

208 Funds; raising key issues in relation to GM such as its integration into evaluation, the need for support structures on GM and gender budgeting; raising awareness of the need to implement GM and gaps in provision and mainstreaming in MS and at EU level; providing tools to support on-going GM; and creating a community of GM contacts on ESF in a majority of MS. Some concrete examples of short term impacts within participating Member States with the potential to lead to longer term effects were also offered by case study interviewees. These included the view that French speaking Belgium had gone further than it would have otherwise done in respect of developing a gender mainstreaming strategy through a decision to focus on capacity building. This has resulted in an expert being engaged to help with this capacity building and provide greater support to those designing and delivering projects in the next programming period, hence providing the potential for longer term effects to accrue. More concrete and longer term impacts in Member States were also cited by network partners as being likely to be achieved in the continuation phase of the network, with the implementation of the common standard in four pilot countries and the potential for this to be adopted more widely through the aegis of the Commission. It was noted that the longer term impact stemming from this is intended to be that gender is reflected in a more thoroughgoing way across OPs than is the case at present; however, partners again acknowledged that the degree to which this occurs will only be evident over a longer timeframe. Those leading the network also discussed impacts in terms of its contribution to transnational co-operation, facilitated through the liaison of partners involved in the peer reviews and in developing the common standard. In respect of this it was noted that there have been short and longer term spin-offs in terms of on-going contact between individuals working in different MS and MA contexts. Section 5 Success of LN activities Reach and dissemination: Evidence suggests that the network was successful in engaging significant numbers of individuals and organisations through its activities and in the sense of the dissemination of products. The perspective of those involved in the network was thus that its effects had successfully reached different levels, in the sense of the organisations and Managing Authorities directly involved but also those engaged in network activities and events more widely. A key effect was also cited in respect of the dissemination work of the network enhancing the visibility of gender and gender mainstreaming issues at the Commission / EU level, both in terms of the common standard and through policy / position papers. Some partners also referenced dissemination in terms of the cross fertilisation of ideas and outputs with other learning networks (in an informal rather than formal way through cross membership and so on). Lessons learned from activities: Lessons cited by those engaged in the case study included: A132

209 The significant role that capacity building can play in integrating gender in a more thoroughgoing, effective and sustainable way The need to ensure that participating Member States have the will and capacity to ensure impacts are transferred to their own contexts (In operational terms) the need to ensure streamlined decision making processes by way of enhancing efficiency and promoting effective results The importance of involving as many network members as possible in the production of outputs such as tools etc. to enhance and maintain commitment. Networking: While no formal networks as such have emerged to date, stakeholders stressed the large degree of on-going, bilateral and multilateral, contacts that have been made. This was seen as an important outcome and as reflecting the aims of the network around promoting a CoP and on-going collaborative working in the sphere of gender mainstreaming. It was also noted that the network itself has received funding to continue into Section 6 Efficiency of the LN Level of resources allocated: The grant allocated by the European Commission was 652,978, supplemented by 75,139 of Member State contributions (mainly in-kind contributions for staff time though stakeholders cited that the full extent of this was very difficult to quantify but nonetheless significant). Proportionality of resources to achievements of activity: Given the large number of outputs and wide-ranging impacts, the network can be considered to have been very efficient. Smart solutions were often adopted in order to organise so many high-quality events, including gaining additional funding for events from European sources or ESF Managing Authorities in the Member States concerned. The coordination of the network also appears to have been both effective and efficient, with a Coordination team (Bengt Nilsson and Anna Tengqvist) hired by the Swedish ESF Council. The association of two content (gender) experts and an on-going evaluator also contributed to efficiency. The perspective of those interviewed for the case study, both lead and other partners, was that the resources used were certainly justified by the results of the network in light of these factors. It was also noted that this judgement rests on a consideration of how much the work would have cost if it were delivered by consultants. The approach taken was thus argued to have offered an efficient way to draw on resources in kind to develop the outcomes produced in terms of their scale, level and quality. Changes to resources throughout the life of the network: No significant changes to resources were cited, though it was noted that, in some instances, contributions from MS and MAs were required to support A133

210 and fully achieve the outputs of the network. Section 7 Likely sustainability Sustainability of Network activities: The network has received additional funding to continue its work until February 2015, which should ensure further dissemination of outputs, continued activity and hence contribute to greater sustainability at least in the short/medium term. Stakeholders also noted that sustainability should be considered in terms of the CoP developed along with the range of on-going contacts and relationships inherent within this. Sustainability of impacts and outcomes: The network would certainly appear to have a high level of potential sustainability in terms of the common standard and the potential for this to offer a lasting legacy should the standard achieve the level of adoption anticipated. At the time of the interviews stakeholders gave a positive perspective of the potential for this, drawing on the piloting of the standard in four Member States and the potential for the Commission to support and facilitate its wider adoption. Other impacts of the network also appear to have the potential to occasion some effects that can be sustained into the future. These include the development of an established network of GM contacts on ESF, the online Community of Practice, the impact on the upcoming Structural Funds regulations which stakeholders cited had been influenced by the network s work, and a marked increased awareness of the need to ensure better GM which those interviewed argued was already apparent as a result of network activity. New policies developed at any governance level due to network: See above. Section 8 ESF Calls for proposals Details of national calls: None identified Call for proposal projects: N/A Impacts of calls: N/A Section 9 Strengths and weaknesses The Network s key strengths: A134

211 Dynamic and open coordination by expert network coordinators, with expert inputs from content experts; Wide and varied membership: participants from 13 Member States representing ESF Managing Authorities, gender institutions and experts; Solid upfront needs analysis and careful targeting of actions; Wide range of high quality outputs (events, position papers, online community of practice, website, social game, Standard ) produced in a short timeframe, ensuring impact at different levels; Ensuring learning for all members, irrespective of the initial level of expertise and aims of joining the network (learning, sharing, contributing or producing). The Network s key weaknesses: The Steering Group was too large for effective decision-making; Insufficient coordination of the working groups; Unclear impacts the wider online Community of Practice. Examples of good practice: Peer learning and exchange of good practice Relevant position papers on key EU policy developments Excellent policy outreach to key actors Using professional, dedicated Network coordinators Solid basis (baseline mapping) for definition of actions Smart use of resources to achieve high number of good quality outputs despite relatively limited budget A135

212 Recognition of the diversity of members Development of good relationships with the Commission Proactivity in seeking engagement with the Commission and in particular the Support Group Success factors underpinning learning and transfer: Involvement of all members in developing outputs and tools, especially the Standard Basing outputs on real needs identified Providing a wide diversity of outputs which may suit different actors, at different levels Providing a variety of solutions which may be suitable for different national/regional contexts (i.e. GM support structures, see above) Professional network coordinators, and input from gender experts (as content experts but also expert speakers at conferences) A136

213 Annex: Intervention logic for the GenderCoP network Rationale Aims and objectives Inputs Activities Results Impacts Gender Mainstreaming is not adequately implemented in all phases of ESF management / cycles. In addition there is a lack of coherence in the way that gender mainstreaming is implemented, along with a lack of know-how on how to move gender mainstreaming from the policy to project level. The LN aims to address these issues and to better integrate gender dimensions into ESF. Overarching aim to make gender mainstreaming an integral part of ESF management, all the way from planning, programming, and implementation through to monitoring and evaluation. Specific objectives around the achievement of: an ESF Standard (guidelines) on gender mainstreaming a training and capacitybuilding programme a community of learners on gender mainstreaming within the ESF a network of contact points in ESF increased cooperation between ESF and other key actors Provision of a communication platform on gender mainstreaming within the ESF (and beyond). Total budget of 728,117 (including 75,139 of cofinanced contributions from Sweden, Austria, Estonia, Finland and Portugal) In-kind support from network partners Baseline mapping of gender mainstreaming in the EU27 Establishment of a network of actors working on gender mainstreaming Establishment of 5 working groups: evaluation; training & capacity-building; awareness-raising; policy impact; sustainability Production of position papers: EU2020 and Flagships, 5th Cohesion Report, new Structural Funds regulations Policy outreach to key actors: EIGE, QUING, FEMM Committee, EC (ESF and Gender Equality Units), EWL, European Parliament; Development of communication platforms: on-line community of practice Production of on-line guidelines for gender mainstreaming in the ESF policy cycles: the Common Standard The ESF Common Standard on GM was produced A community of learners on gender mainstreaming within the ESF was established (online Community of Practice with 250+ members; A network of contact points in ESF was established via the network Steering Committee and wider contacts made at events Increased cooperation between ESF and other key actors on GM was achieved through policy outreach to key actors A communication platform on gender mainstreaming within the ESF was set up via the online Community of Practice (CoP) + web site Baseline mapping of gender mainstreaming in the ESF was achieved in 2010 Intended impacts are longer term and revolve around the incorporation of the Common Standard into ESF management from planning, programming, and implementation through to monitoring and evaluation. A137

214 Annex Four: Member state case studies A138

215 Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks Member State case study report - Spain Section 1 Overview of engagement Spain participated in the activities of nine ESF Learning Networks over the evaluation period: COPIE, Gender CoP, ExOCoP, ENYE, IMPART, E&I, AGE, SaviAV, TNC. None of them was led by Spain (but on the basis of prior experience, Spain is now leading the EUROMA network, out of scope of this evaluation). The officers who participated in network activities belonged to the national ESF Managing Authority, other government departments (e.g. General Subdirectorate of Immigrants Integration, Ministry of Labour; Women s Institute, Ministry of Justice), Regional Managing authorities (e.g. Aragon, Asturias, Andalusia, Catalonia, Extremadura, Galicia) and Public Employment Services (e.g. Andalusia). Non governmental stakeholders were also involved, for instance in the SaviAV network other members of the Rede Ariadna, the network of organisations working with asylum seekers and refugees, including reception centres and NGOs like Red Cross or Caritas, participated in study visits. The Incyde Foundation also participated as dissemination partner in COPIE. Section 2 Relevance of the Learning Networks Spanish MA representatives and other stakeholders felt that their participation in the different networks had relevance for various reasons. It was noted by case study interviewees that some of the networks are clearly linked to national priorities (e.g. youth employment), while others enabled subjects that would otherwise lose priority to be kept on the agenda (e.g. employment of older workers with ESF AGE). In addition, some networks allowed Spain to disseminate its own experience in respect of particular themes (e.g. Gender CoP). The types of activities were generally considered relevant to the needs of Learning Network participants. An interviewee involved with SaviAv, for example, stressed the relevance of study visits conducted to reception centres for Asylum Seekers and Refugees in other countries, as there was the possibility to see concrete examples of good practices and exchange information between reception centre officers. Likewise, the Andalusian Public Employment Service appreciated the opportunity to get in contact with other PES and learn their good practices. The Instituto de la Mujer found it relevant to participate in a working group on the sustainability of the Gender CoP at a time when achievements on gender mainstreaming appeared to be in danger in the country. However, in some instances interviewees felt that the relevance of the networks was affected by their focus or by the particular activities they chose to take forward. An interviewee involved in COPIE was more critical of the mere good practice exchange format that they felt the network had adopted. A139

216 COPIE 2 was considered more relevant than COPIE 1 because it focused on the production of concrete tools and deliverables, rather than the traditional exchange of good practice. An interviewee involved in ExOCoP argued that there was at times too much focus on the need to deliver all project outputs to comply with the grant agreement with insufficient attention being paid to the actual relevance and potential impact of these outputs. From this perspective the programme of ExOCoP was viewed as not always being relevant and too ambitious, primarily as it aimed to address a range of very detailed technical issues on which national authorities already have expertise, instead of focusing on ESF-related matters. It was the opinion of ESF MA representatives that, on the one hand, while the ESF Learning Networks should promote innovation and address new and unexplored themes, on the other hand it is important to keep the focus on ESF, producing tools that speak the language of ESF as this is also what differentiates the networks from other types of transnational projects. Section 3 Engagement with the LNs and key processes to support this The main motivation for involvement was interest in learning about other countries practices and to disseminate Spain s good practices. In the case of the Instituto de la Mujer, participating in a transnational network was an opportunity to maintain the profile of the Institute in a moment of declining government support. In terms of participation, however, an overrepresentation of the ESF MA and an underrepresentation of other types of entities among Spanish participants in the Networks was noted in the evaluation commissioned by Spain. 1 Accepting this, from the interviews it appeared that in some cases the engagement of pre-existing national networks helped in organising a more extended level of participation (e.g. as was the case of the participation of the non-governmental Rede Ariadna organisation in SaviAV). In general, it was noted by ESF MA members that participation was mainly driven by the interest and commitment of individuals. Since the 2008 crisis started, transnational cooperation lost priority both at national and at EU level as it was considered a luxury. It took a lot of personal effort by the members of the transnational group of the ESF MA to get things going as there was no institutional incentive. The evaluation commissioned by Spain on a number of transnational and interregional networks also pointed to the lack of prior agreement on the approach to the problem of asylum seekers (open and inclusive or rather defensive, according to the Member State) as an imbalance in the composition of the SaviAV Network 2. From the administrative point of view it was considered extremely easy to participate in activities. When there were study visits and conferences, the hosting partner took care of the organisation of trips and logistics. No language barriers were encountered as participants would normally speak 1 Estudio Sobre la Validez y Eficacia del Modelo de Redes Cofinanciadas por el Fondo Social Europeo y los Resultados de la Participación en las Mismas en el Actual Periodo de Programación Informe de Evaluación, p Estudio Sobre la Validez y Eficacia del Modelo de Redes Cofinanciadas por el Fondo Social Europeo y los Resultados de la Participación en las Mismas en el Actual Periodo de Programación Informe de Evaluación, p. 46. A140

217 English (and some of the tools were translated). The coordination role of the UAFSE was visible and appreciated. In terms of a strategy for engagement, the Technical Assistance Operational Programme of Spain sets out the objectives of transnational and interregional cooperation, which include mutual learning, exchange of good practices, improving communication, joint platforms for experimenting with new products, services and methods, and the development of new management models 3. However, no specific formal strategy was devised for participation in the ESF Learning Networks. Despite this an informal group of experts was established to coordinate transnational activities in the UAFSE. ESF Learning Networks are also mentioned in the Guide to transnational cooperation disseminated at the beginning of the programming period amongst relevant stakeholders (see below) 4. Tools developed in the Member State context to support dissemination and cascade learning included: a) Initiatives by the national ESF MA The first initiative consisted in organising conferences to launch and disseminate results of transnational activities among other government officials, the 22 Operational Programmes and all the Comunidades Autonomas. Two events were organised: 22 May 2008 National Information Day on transnational cooperation in ESF programmes. ESF MA staff presented a Guide to Transnational cooperation, including information on how to set-up a transnational project and the involvement in ESF Learning Networks. About 70 representatives of national Ministries, regional Managing Authorities and bodies, and intermediary bodies participated in the event. 30 October 2012 National Seminar on Good Practices from Transnational Cooperation; participants from the same target groups. The second initiative was to translate into Spanish the products of some networks (COPIE, IMPART, ENYE). ENYE circulated among MAs the Guide for young entrepreneurs and the Database whereby managing authorities can have access to young people who participate in programs. b) Other initiatives In terms of other initiatives, the Instituto de la Mujer regularly disseminated results of their participation in the Gender CoP among the members of the national network which includes officers responsible for gender mainstreaming in ESF and ERDF. In addition the Rede Ariadna organised a rotation 3 POAT FSE Apartado Objetivos específicos de la transnacionalidad e interregionalidad. EJE 4 Cooperación transnacional e interregional (p. 12), cit in Estudio Sobre la Validez y Eficacia del Modelo de Redes Cofinanciadas por el Fondo Social Europeo y los Resultados de la Participación en las Mismas en el Actual Periodo de Programación Informe de Evaluación. 4 Guía de Cooperación Transnacional para el nuevo período de programación FSE España Versión 2 (mayo 2008) A141

218 system to participate in study visits and other activities of SaviAV and every time all other members were informed. Also, the Andalusian Employment Services ensured that the newsletter of the ESF AGE network was sent to 30 key stakeholders in the region. Moreover, the Guide was disseminated to all officers from the relevant departments, some experts involved in technical assistance, the Andalusian Employers Confederation, the local University and the Chamber of Commerce. The link to the video on YouTube was also disseminated by and personal Facebook pages. Finally, the ENCYDE Foundation, a partner in the ENYE Network, widely used the methods and tools developed by the network in the context of their consulting and training activities to support youth entrepreneurship in the framework of the agreement they have with the Spanish National Youth Institute. The quality of materials provided by Learning Networks was considered to be generally good by those interviewed for the case study. In addition it appears that Learning Network products were disseminated among a number of key stakeholders and the existence of prior national networks was of help. The translation into Spanish of certain products was also crucial, though it was noted that having the ESF Age video subtitled in Spanish would also have been beneficial. Section 4 Outcomes and impacts Outcomes and impacts can be identified in the Member State context as follows: a) ESF governance An ESF microcredit programme at the country level is in the planning stage in part as a result of participating in the COPIE network and benefitting from its learning. The Spanish MA plans to launch the microcredit programme in the ESF programming period drawing on the lessons gained through participating in the COPIE network. It was also noted that the theme of inclusive entrepreneurship was in general much stronger in FSE programming after participation in COPIE. b) thematic learning The methodologies to support (youth) entrepreneurship developed in the related working group within ENYE were used by the Incyde Foundation in its activities in cooperation with the governmental Youth Institute. c) new/improved policies The National Employment Service participated in the ESF AGE Network (via its Head) and this was seen as being very important. A recommendation to do more for 55+ workers was included in the Employment Plan with the ESF MA interviewees believing that given the participation of the national Head of PES in the network might have had an influence on this. Also, it was noted that participation at regional level (Andalusian PES) facilitated inclusion of the topic of age management in the policy agenda and a Regional Law including a programme for 45+ is currently being developed. A142

219 In addition, the Ministry of the Interior and the Ministry of Employment and Social Security signed a final document produced by ExOCoP 5 as an attempt to give political visibility to its results. However this fact reported in the evaluation was not mentioned by the ESF MA officer involved. d) new/improved (operational) practices The manual for entrepreneurial guidance developed by COPIE was used by Galicia for their self-employment orientation programme. Also, following an example from a study visit to Finland in the context of the SaviAV network, the Rede Ariadna produced an information leaflet for asylum seekers and other people interested in knowing what type of assistance is provided. This leaflet is broader than the Ministry of Interior s note explaining asylum procedures and includes practical information on which basic needs of asylum seekers are covered and how. It has been translated into English, French and Arabic. The Andalusian Public Employment Service followed a Dutch example of a Mobility Centre encountered in the ESF Age network to support older workers to establish their own Early Assistance Units (Unidades te atención temprana). These units aim to advise the worker on opportunities to find a new job before he has lost his job, at an early stage. They also drew inspiration from a practice from Flanders to set up a one-stop shop. A number of micro-effects can also be identified at the regional level. These include, for example, the use of tools produced by COPIE by technical personnel within regional administrations, or planning for the development of microcredit programmes in certain localities 6. During the interviews for the case study it was noted that while the ESF MA does not have a clear overview of all these micro-effects they were certainly felt to be present. e) new links/networks between MS of within MS It was noted during the case study interviews that a few bilateral projects between detention centre authorities developed from contacts made in the ExOCoP network. Also, a Germany Galicia exchange programme (in the framework of the Mobility network 2013, led by DE) was cited as an impact of the ENYE network. In addition, the Spanish INCYDE Foundation (involved in the COPIE network) now works with a Greek Foundation on entrepreneurship. Moreover, they participate in two Leonardo da Vinci Projects - one on social entrepreneurship and one on entrepreneurship for rural women - due to transnational relationships developed in the COPIE network. As a result of Learning Network participation it was also reported that the Spanish PES had 2 meetings with the French PES on relations with social agents. Finally, from the perspective of those involved in the Rede Ariadna, participation enabled them to get in touch with international networks via the SaviAV network and this allowed them to join other projects (e.g. EPRA network, NARC Network). 5 Estudio Sobre la Validez y Eficacia del Modelo de Redes Cofinanciadas por el Fondo Social Europeo y los Resultados de la Participación en las Mismas en el Actual Periodo de Programación Informe de Evaluación, p Ibid. p. 99. A143

220 Overall impacts / benefits Overall, besides the ESF MA and the parent Directorate-general of the Ministry of Labour and Social Security, the Public Employment Service appeared to have been benefitted from the Learning Networks to the greatest degree. In contrast, involvement (and impact) was weaker in respect of the Ministries of Justice and Interior. One general effect also worth noting is that the ESF MA officers who participated in the networks are now regarded as experts of the relevant themes by their colleagues within the Ministry. In terms of the extent of operational and / or policy influence and policy transfer, according to ESF MA representatives transmitting results to the national administration is complicated. There is a huge gap between the technical and the decision-making level. For example, on the technical level COPIE was viewed by case study interviewees as being very successful, though much less so in respect of policy influence. In the case of ExOCoP, a gap was reported between the committed participants and associational entities managing training for the labour market re-integration of detainees, and relevant policy-makers, in a policy environment that is generally seen as being difficult and closed 7. Finally, participation was seen as having an effect on subsequent transnational co-operation. The ESF MA initiated the EuRoma Network drawing on their experience of involvement in the Learning Networks along with the impulse of the Director-General s contacts with EC officials who encouraged Spain to start such a network (this top-down input helped establishing it quite quickly). Section 5 Lessons learned and possible future improvements The main improvements suggested by stakeholders and drawn from the literature were: - Greater commitment to transnational cooperation by the EC and MS (a decline in attention was perceived in the later part of the programming period) - Improved technical assistance with clearer added value - Implementing the Common Framework for Transnational Co-operation and as part of this synchronising the publishing of calls for proposals for transnational cooperation (with a maximum of two times in the programming period). In terms of lessons regarding the most effective way for Member States and ESF Managing Authorities to engage with Learning Networks / ESF transnational co-operation more broadly, it appears that: 7 Estudio Sobre la Validez y Eficacia del Modelo de Redes Cofinanciadas por el Fondo Social Europeo y los Resultados de la Participación en las Mismas en el Actual Periodo de Programación Informe de Evaluación, p. 52. A144

221 - Dissemination is more effective when national networks are linked up to transnational networks; that is, participation can be organised according to a rotation mechanism so that several different entities can participate in study visits and exchanges and all are informed of outcomes. - Engagement is stronger when there is a theme where the MS either leads (and can showcase good practice) or has much to learn from other Member States. In terms of other improvements to ESF Learning Networks in the future that can be drawn from the case study: - It is important to have thematic experts supporting the individual Learning Networks, as it was the case for some networks in the current programming period. - The Commission could consider developing a communication strategy for ESF learning Networks, to support individual network dissemination activities. - One expert who was involved on behalf of a dissemination partner (INCYDE) raised the issue of the sustainability of ESF Learning Networks in the sense that there should be funding available to maintain the networks and their activities. - Finally, it was noted that it would be better to assess the impact of the networks at a later stage once OPs are finalised in all Member States, so as to better capture impacts, with interviewees noting that Spain for instance has not concluded its programming exercise yet. A145

222 Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks Member State case study report - Czech Republic Section 1 Overview of engagement Outline the nature of MS / region / Managing Authority engagement with LNs (how involved, which Networks principally engaged with, engagement through MA and/or ministries, regional bodies etc.) The Czech ESF Managing Authority (ESF MA) took the lead in the transnationality network. Other representatives participated in four Learning Networks, namely: 1. Age 2. BFSE 3. COPIE 4. RBM Representatives were from the ESF MA (all networks), other departments in the Ministry of Labour (BFSE, COPIE), an NGO (BFSE), and the Central Organisation Authority for Structural Funds in respect of their role in ensuring synergies between ESF and ERDF (RBM). Although Flanders led the RBM network, the Czech representatives were also heavily involved in this network. Section 2 Relevance of the Learning Networks Outline the nature and extent of the relevance of the LNs to a) ESF MAs, b) relevant Ministries, c) Member State / region as a whole d) other institutions (e.g. social partners, NGOs etc.) In terms of the content of the selected Learning Networks, all five networks which Czech representatives led or engaged in involved themes seen as highly relevant to both the ESF MA and the Czech Republic, either by giving new insights for policy or practices or validating existing practices. Engagement was also seen as being relevant to the participating individuals in the context of their organisational roles (because they were either dealing with the subject on a regular basis (Age, RBM) or it was foreseen that certain issues would eventually become important for the Czech Republic (COPIE, BFSE)). In addition, the NGO involved was invited to join, and participated in, the BFSE network as it collaborates closely with the Ministry of Labour and ESF MA on the social economy of the Czech Republic. It also brought content and practical knowledge, managing as it does the A146

223 national network of the social economy (TESSEA). In all cases the participants interviewed indicated that they had been keen to learn from other Member States, especially New Member States. Any ways LNs and their work / focus could be made more relevant? It was noted that in some Member States like the Czech Republic, the staff turnover in Ministries of Labour and ESF Managing Authorities, in particular, is very high. Out of all the interviewees consulted only two are still in the same position they were when they joined the networks; therefore looking to develop more consistent representation was seen as a possible improvement. Some interviewees also felt that it would be helpful if there was more of top down strategic selection made as to which networks to participate in and more of an influence from the policy side of the Ministry of Labour. Section 3 Engagement with the LNs and key processes to support this What were the main motivations for the involvement of actors from the MS in the LNs? The decision to lead the TNC network reflected the interest and commitment of the Czech ESF MA to transnational co-operation and participation in the Learning Networks. This, along with the relevance of the networks to national priorities, combined with individual commitment and interest in the themes they covered, served to drive participation at the MS level. In some cases engagement in networks was also linked to participation in previous EQUAL networks. For example, the AGE, BFSE and COPIE network had their roots in the EQUAL period. During that time the same representative for BFSE and COPIE participated because of his role as manager of ESF projects with related themes. While such projects were a rather small component of the ESF programme, the representative anticipated that this would evolve and grow over the next programming periods. He therefore remained involved to help develop results into policies at MS level, to help formulate themes or conditions in calls for proposals and to inform project promoters of the latest and best practices. At the level of particular networks, in the case of RBM participation related to individual interest combined with interest from the participants Ministry. At the Ministerial level, it was recognized that use of RBM in the Czech Republic was not particularly advanced or developed and engagement in this network was thus felt to represent a good opportunity. At the level of individual engagement, this was seen as involving a combination of individual interest, wanting to learn from other countries, being in an international environment, and relevance for work. Similarly for COPIE; there were some microfinance/credit facilities available through the Ministry of Trade and Investment, but these were often not suitable for disadvantaged groups. Involvement in the network was thus undertaken to get more good examples of how such facilities could work, with the aim of eventually lobbying the Ministry to utilize microfinance approaches through ESF. A147

224 What might have encouraged other key actors to engage with the LNs? There were some attempts to involve other (policy) departments from the Ministry of Labour and other national stakeholders in the Learning Networks. This was reported as having worked to some extent for BFSE, but not so much in respect of the other networks. It was acknowledged that it would perhaps have been beneficial if there had been a more developed and structured national strategy for participation, moving beyond the ESF MA, which could have made stronger links / incentives between departments. Social economy was a relatively new knowledge area for the Czech Republic and the representative concerned was convinced that this was a relevant and important topic to further explore. He tried to involve the Ministry and the most relevant NGO in the network to make sure it could be embedded into policy and practice. The participation of the Ministry was limited at first but the NGO ensured that the Czech participation was based on practical experience and allowed them to organize a workshop at home. It was noted that the combination of knowing that such opportunities existed, mixed with demonstrable success and relevance to policy development, helped encourage increased participation from the Ministry later in the process. How straightforward was it to engage with the LNs and their work? How might this be improved? On the whole the participants were positive about their ability to engage in the Learning Networks. They were, however, also very motivated themselves to do so. The interviewees did indicate that language could be an obstacle for some other participants, which could slow down or reduce the value of the sessions. A representative involved in the RBM network commented that trust played an important role in their engagement. It was seen as important that there was a trust between the participants, a commitment to do the work, and recognition of its potential benefits. The leader of the network had in general found the right people to want to participate and that was seen as helping to create trust. The network leader also frequently allowed participants to insert their own ideas and asked them what they wanted to get out of the network. Those involved in BFSE also highlighted the importance of personal relationships. The representatives in the transnationality network also added that leading a network is not easy, very time consuming, administratively heavy and often needs to be done in addition to an already high work load. This was seen as reflecting the importance of those taking this role really believing in the network and being able to guide the process. In this, it was seen as being important to remember the priorities for the EU rather than only those for the lead partner s own Member State. Was there a plan or strategy guiding engagement at the level of a) ESF MA, b) Member State / region? How effective? Improvements? A148

225 It was reported that there was no single structured strategy for engagement at the Member State level. Rather, participation was guided by the ESF MA but not in the sense of being through a formal strategy taking in wider actors at the MS level such as other Ministries. The potential to develop such a strategy was partly complicated by the high turnover of staff with the elections that took place in this period in the Czech Republic. It was acknowledged that there is significant room for improvement in this regard. Structures/tools to support dissemination / cascade learning. What were these (if any), how effective, improvements? While there was no structured dissemination plan developed at the national level (beyond the standard requirement for reporting to management), the materials provided throughout the Learning Networks were much appreciated, especially those concerning good practice, indicators and guidelines. As the use of the network outputs may come at a different time than the running time of the network, it was much welcomed that most of the networks had their own website (and that some are still up and running). The online community for RBM in particular was seen as useful during the network and is still so now. One of the interviewees commented that they consult the website for literature, information and contact details. The online training programme being developed was also cited as offering a new way to disseminate knowledge (this has been developed under the new Learning Network, but is a continuation of the work completed under the old Learning Network). Quality of materials provided by the LNs to support dissemination and improvements? The majority of interviewees indicated that their Learning Network did not focus significantly on dissemination but rather on just sharing practice amongst participants, with them subsequently being responsible for dissemination. They were hopeful that the current networks will be more useful on this aspect. Section 4 Outcomes and impacts What outcomes and impacts can be identified at the MS / regional level? Look to cover impacts relating to a) ESF governance, b) thematic learning, c) new/improved policies, d) new/improved (operational) practices, e) new links/networks between MS or within MS, f) other The outcomes and impacts of the participation for the individuals interviewed and for relevant organisations varied from little to greatly. One of the most notable impacts has been on the national action plan on active ageing which has been in place since The representative of the Age network is also part of the national working group for active ageing and has in that respect been able to directly influence the national action plan. Of course this action plan is not the direct result of the Learning Network, but rather originated from the European Year However, the view was that its content A149

226 and implementation took on board best practice and ideas from the Learning Network. It was also noted that learning from the Age network influenced some of calls for proposals, but more importantly, the design of the new OP which will have a section on support of employees in transition and can specify calls around older workers. There were also important outcomes and impacts at national level from the lead role taken by the Czech ESF Managing Authority in the Learning Network on Transnational Cooperation in the ESF, In particular, it provided the Czech Republic with a wide network of contacts with other Member States, which provides new links for other forms of cooperation in the ESF (other networks or wider issues). In addition, by learning from other Member States and co-developing tools at EU level on this topic, it helped to strengthen the processes used in the Czech Republic for managing their own transnational calls for proposals and projects. There is also evidence of a (slow) process to create policy change in the area of the social economy. It took two network periods to really get to this place, but now under the third network the policy departments of the Ministry of Employment are heavily involved in the BFSE network allowing for the results to be better embedded in policy making. It is also expected that the new programming period will demonstrate a greater focus on the social economy as well as a strategy on the subject. This was the result of persistence attending and learning from personal motivation and being convinced that the results were relevant to feed into the policy process. It was noted that it also helped having such an influential NGO involved. The head of the NGO for example wrote a paper on whether some of the methods were transferable to the Czech Republic and what different indicators were necessary. It was reported that cooperation has recently intensified and that there appears to be more buy-in from the Ministry on the subject. On a practical level, the NGO has indicated that through the network it found a way to intensify cooperation with people from the Polish network doing similar activities. There is a good opportunity that the next programming period will also see more of the results of COPIE, likely in the form of space for a micro-credit facility for disadvantaged groups and entrepreneurship. There are some evaluations related to COPIE on-going which will help provide input into this programme. The evaluation set up has also been taken from the COPIE network (the aim, target, intervention logic). More broadly, it was also noted that it does take time to transfer knowledge from an individual to an institution to a policy. In the case of the Czech representatives with their high turnover- this has sometimes been difficult to influence and achieve. The manpower simply wasn t always there. However, those involved were keen to stress that his does not mean that learning does not get used. One of the representatives has for example been seconded to the EC where she continues to keep in contact with the RBM network members and uses the material when there are discussions on the subject. Section 5 Lessons learned and possible future improvements A150

227 Summarise the main improvements suggested by stakeholders / from the literature re. the above aspects It was noted that leading a network involves a lot of work, often more than is expected, and this is key in terms of influencing the value of a network. It was also noted that currently the learning from such transnational engagement is mostly West to East and not enough East to West or even East to East. Those involved in the networks tended to feel therefore that it would be good to increase cooperation among the Eastern Member States, with Poland providing a good example of leading the way in many areas and having similar systems/historical legacies to that of the Czech Republic. It was also argued that there should be more clear formulations in terms of demand from policy makers around what would be beneficial to them by way of informing network activity. This was seen as having already improved a lot, with not just those implementing ESF involved but also policy makers, though more progress was felt to be possible. Finally, a further suggestion involved the fact that language skills are not always so great amongst those participating and that having translations or meetings in native languages might be considered. It may also be that a more strategic and structural plan to embed Learning Network activity and learning in the ESF MA and in policy making amongst wider Ministries may be beneficial, particularly in a context where there is high staff turnover. A151

228 Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks Member State case study report Belgium (Flanders) Section 1 Overview of engagement Outline the nature of MS / region / Managing Authority engagement with LNs (how involved, which Networks principally engaged with, engagement through MA and/or ministries, regional bodies etc.) The participation of Flanders in the ESF Learning Networks (Learning Network) was by and large coordinated through the ESF Managing Authority (ESF MA). The majority of participants were its staff members, with an occasional collaboration with other departments in the Ministry of Social Affairs, Ministry of Justice and local NGOs. Jointly Flanders was with various levels of intensity - involved in 11 of the networks, namely: AGE - Age Management Network BFSE - The Better Future for the Social Economy Network. COP - Community of Practice on Partnerships COPIE - Communities of Practice on Inclusive Entrepreneurship EMPOWERMENT - European Network on Empowerment and Inclusion ExOCoP - Ex-offender Network Gender CoP - Gender Mainstreaming Network IMPART - Increasing the participation of migrants and ethnic minorities in Employment RBM - Network on Results-based Management SaviAV Social inclusion and vocational integration of Asylum seekers and Victims of human trafficking ENYE - European Network on Youth Employment A152

229 Flanders was the leading partner for the Results-Based Management Network and co-lead the BFSE and to some extent the AGE network. Although it contributed to the COP initially, it eventually reduced its activity in this network. Section 2 Relevance of the Learning Networks Outline the nature and extent of the relevance of the LNs to a) ESF MAs, b) relevant Ministries, c) Member State / region as a whole d) other institutions (e.g. social partners, NGOs etc.) With the heavy role of the ESF MA in the Learning Networks, the relevance aspect is mostly associated with its organisation s objectives and priorities as well as through its position/role in wider government policies. For that purpose, a horizontal policy committee, containing representatives of various departments of the Flemish Ministry of Employment and Social Partners, validated the choices, priorities and representation of the ESF MA at the start of each Learning Network. Staff from the Ministry were strategically involved (such as RBM and BFSE) or invited on an ad hoc basis if a session was particularly relevant to Flanders. This was also the case for non-government or semi-government organisations (such as the PES- VDAB). Their participation was however, minimal. Only for ExOCoP was the participation completely organised via the Ministry of Justice. The Flemish participation was thus very much oriented around the role of the ESF Managing Authority. Within the Managing Authority, there is on the whole a sense of pride in having participated (and continuing to participate) in the ESF Learning Networks. Nearly every interviewee looked back on the networks having taken something out of it, whether it be new knowledge, ideas, contacts or general energy. Coming from that perspective, most interviewees could think of a good reason why Flanders joined the network in which they participated. There were two main reasons: 1. Working with others on practice or approach in a setting that was new or changing in Flanders (e.g. BFSE, RBM, Empowerment) 2. Wanting to share something, validate good practice and generate more awareness for things that worked well for Flanders (e.g. Gender, Age) In terms of the relevance of the content of the ESF Learning Networks selected for participation; this varied greatly from Network to Network. In the case of BFSE, the Flemish government was in the middle of developing a new strategy and therefore hoped to influence the focus of the network on those areas that were relevant to the new strategy (particularly state aid). This proved possible as Flemish representatives took the lead on that aspect within the Network. Similarly having the lead in RBM allowed them to focus the content to the interests of Flanders. In other networks, especially those where representatives mostly participated to share something, the relevance and potential use for Flanders was not always evident. This was seen as A153

230 being the case especially if the network did not focus enough on the aspects that could have been useful for Flanders (like awareness raising for Gender and Age), or if there was limited room to influence policy and participation was aimed at making minor changes to calls for proposals (like with Youth and SaviAv). It was a strategic decision on behalf of the ESF MA to select staff that were most involved in a certain theme to participate in the Network, rather than having a designated group of transnational staff. It was anticipated that these individuals would have the most influence on creating change within the ESF programme when new ideas were encountered. Some of these examples are; The representatives of the ESF MA to the Gender network were also in charge of gender mainstreaming issues for the ESF MA and also members of the Commission for Equal Opportunities in Flanders, thereby having access to a wider platform for disseminating the learning from the Network. The representative of the ESF MA to the SaviAv network was the main person responsible for the European Integration Fund and the European Refugee Fund The representative of the ESF MA to the ENYE network is also tasked with youth unemployment. These individuals expressed that they appreciated the opportunity to participate in the learning network because they wanted to learn from others, feel strengthened in their work and discuss their work with people with similar roles. Any ways LNs and their work / focus could be made more relevant? There was a large level of satisfaction about the relevance of the learning networks in which representatives from Flanders participated. When dissatisfaction was noted it was mostly due to an inability to sufficiently influence the focus of working groups within the networks. At the same time it was recognized that a level of compromise is required as each participant brings their own priorities with them. There were two aspects noted by interviewees relating to the potential to make the networks more relevant: 1. When the priority of a network is to make an inventory of best practice and then achieves this goal, the network should develop an enhanced focus on how to disseminate the results effectively. This should involve not just relying on a conference with a lot of attendants, but really developing a strategy for getting ESF promoters and the end users to pick it up. As one interviewee noted: You can inventorise as many tools A154

231 as you want, but if the users are not picking it up, you have to focus on awareness raising. 2. From the perspective of several interviewees, the success of the networks is highly dependent on the individuals participating in them. Even if everyone can understand all that is discussed (not a given), then it still cannot be assumed that understanding is equal to learning; that is, one can listen and comprehend, but seeing and thinking how it can apply to your own work, organisation or Member State requires more than the odd seminar. From this perspective, therefore, it was noted that if participation in the networks was focused from the start on the anticipation of bringing something back to the Member States, and continuously working towards that achievement, this would be beneficial. Section 3 Engagement with the LNs and key processes to support this What were the main motivations for the involvement of actors from the MS in the LNs? What might have encouraged other key actors to engage with the LNs? The previous section provides a good insight into the motivation for the involvement of the ESF MA as a representative of Flanders (with the exception of ExOCoP). Other departments of the Flemish Ministry of Employment joined those networks that could provide insights into new policies or practices like RBM and BFSE or that were well advanced like Gender. It was noted that at the time of the networks operation youth unemployment, demographic change and migrants were not as much of a concern, explaining the absence of Ministry staff in SaviAV, AGE and ENYE. The interviewees indicated that on the whole they found this separation appropriate. How straightforward was it to engage with the LNs and their work? How might this be improved? For the networks in which Flanders played a key coordinating role (RBM, BFSE, Age) it was by definition easy to engage with the work of the Learning Network but involvement still required a lot of preparation, investment and compromise particularly when Member State and ESF MA departments had very different priorities. With these and other networks, on occasion there was some disappointment when a Flemish priority did not gain enough interest from the other participants and was thus not sufficiently explored. In other instances a change in policies or other situations altered the relevance of the content of the Learning Network during its running time. When this occurred, the extent of continued participation depended on organisational needs and individuals interest in the themes emerging. As part of developing the networks, although it was important to make sure that everyone focused on the same objective and that time was invested in this, it was noted that this part of the process is rather tricky and can be the make or break component of the network. In some cases the start up did A155

232 take a very long time. There were also complaints that the repetition of the objectives in a plenary session at every meeting was long, drawn out and an old-fashioned way of working. This was not just a distraction but also impacted the effectiveness of the meetings, given that there was not that much time in them to begin with (1-2 days). In two of the networks, IMPART and RBM, it was also noted that these networks had a demanding methodology, requiring serious commitment and preparation from all those involved. This did lead to the reduced participation of those representatives from other Member States who were not prepared to make this commitment or for whom it was too complex or demanding. Adopting a centralised management structure for networks was also mentioned as an important success factor in enhancing clarity and ease of engagement. Conversely it was noted that in learning networks where each sub-group went off to do their own work focus was sometimes lost, or insufficient activity from participants resulted. This meant that some working groups produced higher quality and / or more relevant results than others. In general, interviewees noted that there was a big difference between the experience and ability of the participants. Three issues were identified, namely: 1. Some people have been involved in these networks for a long time and the learning process becomes less, which can become a problem when participants join with the idea that they have been doing it for years and already know what they are doing. 2. At the same time, sending inexperienced staff new to the ESF way of working also requires a significant investment on the individual to catch up and avoid re-inventing of the wheel. 3. In a few of the networks the English language skills of the participants were not up to standard, resulting in misunderstandings, slowing down of the processes and /or lack of interaction from those participants. Most interviewees referred to the economic crisis when citing the absence of some of their colleagues from new and Southern European Member States and were disappointed that this had resulted in such an impact. Several interviewees mentioned the contracting of external experts in the networks. For Flanders it was decided that the least amount of external contracting would be used for content development so that the staff themselves could benefit most from any content-building experience. This was a strategic choice, which in some cases paid off, but it was also found that there was often little time to actually do the preparation work and that assistance from experts would have enhanced the Learning Network. Having at least the support with the administrative and coordinating part of the Learning Network would have been helpful. When the networks worked well, it was noted that everyone contributed greatly and invested significant time and that there was an opportunity for the A156

233 development of informal relations. This was seen as having made a real difference. When the website concerned was also very good and all the products were kept there, it was found useful and easy to access and come back to/prepare for the next meetings. Was there a plan or strategy guiding engagement at the level of a) ESF MA, b) Member State / region? How effective? Improvements? As mentioned previously, there was a clear reason for selecting the individuals who participated in the network on the basis that they had some relationship to the themes concerned. Furthermore, the involvement of a horizontal committee in the selection of the type of Learning Network to participate in (and to what extent) was helpful for focusing engagement from the Flanders perspective. However, this structure only covered the extent to which Flanders was able to contribute to and participate in the Learning Network and not quite how it would also get something out of it. There was a structure of post-meeting reporting, which went to both the team manager as well as the horizontal committee, but there was not a solid structure for abstracting practices and implementing them. Indeed many of the participants commented that they participated in the networks on top of or besides their regular work. Embedding the work from the Learning Network in their daily work was not a given and indeed an obstacle for some, particularly as the better prepared one was, the more one would get out of the Learning Network but yet there was so little time to focus on the Learning Network work. Both the difficulty with individuals time as well as the lack of significant implementation of the Learning Network findings in the work of staff resulted in a re-adjustment of the strategy taken by Flanders. Between the previous networks and the current ones a new structural system of engagement has been implemented. The change entails appointing a person dedicated to structuring the involvement of Flanders in the networks. Their role also involves considering how to make most use of the networks for Flemish participants and the ESF promoters/project beneficiaries in the region. There is some indication that the group of promoters under the previous programming period have not benefited from/learned from the best practice available both from within Flanders as well as from the networks. This is seen as having been addressed with the new strategy, focused as it is on making the transition to the real work of the MA and better balancing the giving and taking in the Learning Networks. Structures/tools to support dissemination / cascade learning. What were these (if any), how effective, improvements? Dissemination and cascading of learning was reported to be one of the weakest aspects of Flemish involvement with the networks. In the Flemish case, dissemination can be considered at three levels, namely: A157

234 Within the ESF MA Within the policy making field/ministry Within the practice and implementation field/ project promoters, employers etc. Although the writing of reports ensured that information was at least shared to the first and second level, it did not necessarily lead to change unless the participant in the network was able to adopt learning within their daily work. The level of sharing within the practice and implementation level was indicated to be the lowest. There was a general consensus that when you participate in as many Learning Networks as Flanders did, that some will provide more useful results than others and that some may not be as relevant as originally expected. However, even for the networks that did run well and produced useful (validated) results, it was found that the dissemination and implementation aspect required more attention. Several of the interviewees indicated that if they had put awareness raising on the agenda as a priority within the network, the outputs may have been better used. In the case of Gender and Empowerment, there has been some dissemination through training for ESF project managers, but it was noted that avoiding a high administrative burden on such managers was also a concern. Also, it was noted that the dissemination of findings were sometimes met with resistance within Member State Ministries and the MA on the grounds that there was little time, or interest to invest resources and that regular work needed to be done. It was also noted in respect of dissemination that if there is not a direct result yet it does not mean that it will not happen in future. Equally, the point was made that the results of engagement do become useful to the work of the MA over time. This indicates that the alignment of the priorities of the Learning Network and the MA does not necessarily happen simultaneously, but can provide added value later on with this often depending on the individual involved. In the mean time more evident and immediate benefits were seen as being at the small-scale, practical level, with personal contacts, tips and tricks and so on being shared. Section 4 Outcomes and impacts What outcomes and impacts can be identified at the MS / regional level? Look to cover impacts relating to a) ESF governance, b) thematic learning, c) new/improved policies, d) new/improved (operational) practices, e) new links/networks between MS or within MS, f) other Despite the need to better embed learning into the daily work of the ESF MA, outcomes and impacts from involvement were identified by interviewees. In part these resulted from the RBM network which was designed and developed with the interests of Flanders in mind. Although interviewees felt that A158

235 this network met with certain obstacles, including low involvement of some members, the results of the network were positively viewed. The network duration was used to delve into the subject and come to serious conclusions on results based management. This was viewed as a success and is now moving into the next phase of disseminating and implementing across other Member States. Through this process those co-ordinating the network have been able to develop an online preparation training and built an alliance with the OECD. It was noted that this would not have happened without the findings from the network and the commitment of its key members. Also, within the BFSE network, the Flemish representation was able to influence a portion of the work, especially on state aid. The intended legal changes in Flanders that got them interested in this aspect did not go through as planned. As a result the outputs were less useful than expected, but did play an important contribution to knowledge building, which will also be useful for the other Member States. There has also been some talk of including new indicators in calls for proposals based on work undertaken in the networks. This was the case for Gender, Age Empowerment and SaviAV for example. At the same time there is a genuine concern for overburdening project promoters by demanding too many new indicators. This may change in the next programming period. Most of the interviewees highlighted their personal learning as the most valuable outcomes. This should not be considered insignificant as the relatively low turnover of staff in the Flanders ESF MA means that there is an opportunity for continuity on the themes that did work well and for using the results of the Learning Network with a more appropriate timing. In addition, there is a strong potential for further effects through the monthly meetings between the Minister for Employment and the head of the ESF MA. The participation of his staff in the Learning Network provides him with information about best practices in other Member States that can be brought to the attention of the highest level of policy making. This was seen as being a potentially powerful source to create changes over time and to continue to learn from abroad. It was also noted that to create change there is a need for cultural and structural adaptation. If the Learning Network provides all the good practices and the financing to implement it, but it does not actually happen, the focus should shift to culture and how with further awareness raising more buy-in can be created. In the case of the Gender Learning Network, it was noted that this also means not letting go of project results simply because the funding period is over. In this context it was also noted that the lack of immediate outcomes does not mean that what has been learned under the Learning Network is not useful or not important, but it needs to be placed within the daily tasks and overall strategy of the work undertaken within the ESF and that may be at a much later moment. A similar experience took place in the Age Network. During the Learning Network, every working group had its own method for determining good practice. It involved such a broad coverage of topics that it proved useful in some areas but not all. Because of what was determined and shared, the Flanders team decided it was useful and important enough to continue. They therefore lead the new network. Under the new Learning Network, it was argued that those involved were able to take the good parts and mould and improve the network, for example by focussing more on awareness raising and dissemination. It is now not just age as a theme but rather career & age, which from the perspective of those involved is much broader, A159

236 comprehensive and in line with the real issues facing employers. This has also been reflected by the way the teams have been organised internally in the ESF MA- namely by designating a focus on sustainable employment, HR and career perspectives. The goal of the previous AGE Learning Network was to validate practices and this partly worked and partly did not; the validation could have been better. The budget was too small to really do the validation process correct. Where are such outcomes and impacts evident particular Ministries, ESF MA, other? What benefits were there at these levels and for the MS as a whole? Transnationality has in general been high on the map for the ESF MA both in terms of their strategy as well as in the large number of staff that participated in the networks. When new links evolved, they were not new links between the Member States in many prior and different thematic settings there has been some contact with most Member States. However, new links were made between individuals who were in the right place at the right time or on specific themes or projects for example, between the Members of the RBM network (particularly within the Czech Republic and Spain). Section 5 Lessons learned and possible future improvements Summarise the main improvements suggested by stakeholders / from the literature re. the above aspects The main improvements identified from the case study research are as follows: 1. At the start of each Learning Network it should be clear what the participant intends to do with their participation; this should be monitored over time. 2. The time for the start up of the network should be proportional to the objectives; continuous re-iteration of the goals of the network should be avoided unless it appears people are forgetting it. 3. Participants should be selected on their affiliation with the subject, not just their interest in transnationality Specific lessons re. the most effective way for Member States and ESF Managing Authorities to engage with Learning Networks / ESF transnational co-operation more broadly. More specific lessons in respect of the most effective way for Member States and ESF Managing Authorities to engage with Learning Networks / ESF A160

237 transnational co-operation more broadly that emerged were: 1. ESF MAs should consider pulling their staff out of the Learning Network when priorities change and it is no longer relevant to their work. 2. There needs to be a balance between transnational and regular work so that it does not become an additional on the side job 3. Involvement in fewer Learning Networks and more focus could have enhanced the experience of the ESF MA. 4. In terms of managing the Learning Networks, the following recommendations were given: - Stay close to the path you originally set out - Know what you want to do and who wants to do it with you; who is in and out - Conduct a future search; which stakeholders will benefit most from the results and should therefore be involved from the start through a stakeholders panel - Use good experts - Be clear early on as to what kind of seminars are useful and appropriate and organise them well - Really do a thorough inventory of good practices and validate them with experts from other Member States. Any other recommendations for improvements to ESF Learning Networks in the future In terms of other recommendations for improvements to ESF Learning Networks in the future, interviewees noted the following: Real change may take 5-6 years and should be allowed to take that long; the themes of the Learning Network should also anticipate labour market changes and likely key issues for the future Using network participation as a form of tourism should be avoided at all costs, but informal contact building should be encouraged. A161

238 Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks Member State case study report UK (England and Gibraltar OP) Please note that the following case study analysis relates only to the England and Gibraltar Operational Programme (OP) and not to the other OPs present in the UK context. Section 1 Overview of engagement At the outset of the programming period, the ESF MA in the UK (Department for Work and Pensions DWP) made a strategic decision to devolve responsibility for transnational co-operation to a dedicated Innovation, Transnationality and Mainstreaming (ITM) Unit, hosted within an ESF Intermediary Body (IB) a Local Authority, Birmingham City Council. It should be noted that the ITM Unit acted as an IB for the England and Gibraltar OP only, not the other OPs operating in the UK. The idea was that this unit could wholly focus on transnational co-operation and offer a dedicated function linking ESF activity within the UK to transnational initiatives. The unit also acted as a body through which a Call for Proposals for innovative and transnational ESF projects managed from the UK could be issued, with the unit managing this activity. Alongside this role the unit was also given devolved responsibility for co-ordinating involvement with the ESF Learning Networks. A decision was taken by this unit to engage with some of the networks on the basis of their relevance either to the work of the unit, the ESF MA, or other UK Government Departments. The decision to engage with the TNC network was driven by the role the unit had in relation to co-ordinating transnational activity in the UK context. Involvement with the BFSE network was driven by the Social Economy programme developed as part of the units ITM programmes managed from the UK (Social Enterprise being one of the 6 themes that ITM projects were commissioned under within the Call for Proposals noted above). Involvement with the IMPART network was driven by migrants and ethnic minorities being a key theme within the England and Gibraltar OP. Engagement with ESF AGE was motivated from a different perspective; while not a key OP or ITM unit priority, individuals in the DWP felt that this was topical and relevant in the context of the work undertaken by the Extending Working Lives unit within the Department. Also, one of the ITM unit projects focused on demographic change so there was an additional link there. Beyond these direct reasons for involvement, interviewees noted that engagement with the Learning Networks was driven by a more general desire to learn from other initiatives and contexts as well as offering learning. The networks were also seen as a route through which learning could be captured and used to inform the work of the ITM projects operating in the UK, as well as learning from those projects being transferred to the EU level. Section 2 Relevance of the Learning Networks The general perspective from the UK was that the relevance of the Learning Networks varied between individual networks depending on their thematic A162

239 focus relative to the policy concerns amongst relevant Government Departments and the ESF OP itself. As noted above this influenced thinking on which Learning Networks to engage with, how and to what degree. It was also seen as varying according to whether interest and engagement in a network was motivated more by sharing learning practice from the UK or by learning from others (i.e. the balance between these). For example, the high degree of relevance in respect of the TNC network was due to the amount of learning and experience the UK felt it could share based on the ITM unit approach, the 31 ITM projects funded in the UK and the management / co-ordination of those, and the central role played previously through EQUAL. Other networks were seen as highly relevant due to their thematic focus overlapping with key policy concerns, whether in respect of the ESF (IMPART, BFSE) or Government Ministries (BFSE as a central concern of the UK s Cabinet Office and ESF AGE as a policy issue for the extending working lives agenda in the UK taken forward by DWP). In general, the majority of the networks that the UK did not engage with were still viewed as relevant to the ESF MA and UK Government policy concerns; however, it was judged that a better approach would be to engage more fully with a more limited number of networks. Relevance in respect of other social partners was also seen as significant, though participation amongst NGOs was noted as relatively limited. Given the above perspective, it was generally acknowledged that there would be little that could be done to make the networks more relevant. In contrast, it was felt that the Learning Network approach was a suitable one and an improvement on EQUAL in the sense that it was designed to be more strategic and operate on a higher level than simply involving lots of exchange on the level of ESF projects. Importantly, however, it was noted that for this to work effectively co-ordination at the Member State level was required to facilitate an effective 2 way exchange of learning which drew lessons and practice from the project level and transmitted new ideas back to this level. Section 3 Engagement with the LNs and key processes to support this Motivations for engagement largely related to the relevance of the Learning Networks to UK concerns, whether in the sphere of the ESF of more widely, and whether policy based or more operational. There was also a view amongst some stakeholders that there was an intrinsic value to such transnational co-operation. Barriers to wider engagement were seen as relating to resources. In respect of this, time was viewed as being as much of an issue as financial resource, certainly for those involved in the ESF administration and Government departments though it was acknowledged that financial restrictions may be more significant for NGOs. Individual interest and commitment was also noted as a motivating factor amongst some interviewees. In terms of getting involved with the networks this was seen as relatively straightforward and reflecting the existing links between the ITM unit and those involved in running the Learning Networks and in overseeing them on the European level. The strategy adopted, in terms of having a dedicated unit in A163

240 charge of transnational work under the ESF was seen as highly beneficial in a number of ways. This view is reinforced by an independent evaluation of ITM work under the ESF commissioned by DWP. 8 The main benefits of adopting what stakeholders viewed as a more strategic approach through the dedicated ITM unit were: Having a central co-ordination function with oversight of the range of transnational co-operation occurring through the ESF, both at the EU level and in respect of UK actors involved Being able to act as a conduit between transnational initiatives at the EU level and work at the domestic ESF project level, facilitating a twoway sharing of information Enabling greater focus on transnational co-operation and issues through their being a dedicated function for this rather than it just being merged with the remainder of the ESF MA s role and remit. The ITM Unit and its functions were also viewed as effectively providing an infrastructure to support dissemination and the cascading of learning through being visible as a central point of contact that domestic actors (whether ESF related or not) could interact with. The unit could thus act as a form of clearing house for potentially useful dissemination outputs from the Learning Network, passing these on to potentially interested parties through the networks and infrastructure built up. While such dissemination outputs were generally viewed positively, however, there was a concern that such materials were not always adequately designed bearing in mind their potential audience. More clear, recommendation focused outputs were thus cited as a way in which this aspect of Learning Network activity could have been improved. It was also noted that, irrespective of the quality of network outputs or the supporting infrastructure provided by the ITM unit, it had sometimes been difficult to ensure effective dissemination at the UK level. One key reason for this was seen as the tendency of UK policy makers, particularly higher level civil servants, to move around regularly. A lack of consistency in personnel was thus cited as a barrier to building up the kind of longstanding links and relationships that would have made dissemination in the domestic sphere easier and more effective. It was also noted that shifts in policy thinking and focus over the timeframe of the ESF programming period had affected the receptiveness of potential targets for dissemination. This was linked in particular to the 2010 General Election in the UK, the change of Government and the hiatus caused when the new Government came to power. Despite this, dissemination was felt to have reached actors directly engaged in ESF, whether at more strategic levels within the ESF MA, or more operational levels in terms of co-financing organisations and ESF projects, to a greater extent. This was seen in part as reflecting the direct relevance of such outputs to the day to day work of people engaged in ESF implementation and delivery. It was also noted that influencing policy through the kind 8 DWP (2012) Evaluation of the European Social Fund Innovation, Transnational and Mainstreaming projects, Research Report No A164

241 of outputs produced by the networks was perhaps too ambitious from the outset; stakeholders thus generally felt that the objectives of the Learning Networks should be more realistic and achievable with this being reflected in dissemination outputs. Suggested improvements in terms of dissemination included: Materials that were more suited to policy audiences i.e. more concise and providing clear policy relevant findings from activity Less of a focus on e-newsletters and leaflet type products which were felt to have little impact and not often being read by recipients Enhancing the effectiveness of dissemination events and conferences (these were seen by stakeholders who had attended them as varying in quality). Section 4 Outcomes and impacts It was noted by stakeholders involved with the ITM Unit that it was anticipated and hoped that impacts from the Learning Networks would occur in respect of three main levels: ESF projects, ESF programme and operational management, UK policy makers. The general view of these interviewees, backed up by others consulted, was that learning and positive outcomes had occurred mainly at the level of ESF projects and ESF programme / operational management. At the level of ESF projects it was felt that those involved in the ITM projects certainly benefited from the transnational activity involved in these, including where such individuals became involved with Learning Network events and activities. Specific examples included a project operating in Liverpool which learnt lessons around social financing from Swedish and Italian colleagues involved in the BFSE network. This learning has contributed to the first social franchise hotel being built in Liverpool. At the ESF programme and operational management level it was felt that engagement with the TNC network had sharpened the thinking and approach of those in the ESF MA and ITM Unit around transnational co-operation, how it can best work and its potential benefits. As noted, much less of an impact or positive outcomes were felt to have emerged at the policy level. In part this was seen as relating to the issues noted above around inconsistent engagement, policy shifts, lack of appropriate dissemination materials, and disruption caused by the change in Government. The fact that anything related to European co-operation was low down on the current Government s agenda was also cited by one interviewee as being a factor in this. The key point here was felt to be around the need for audiences to be receptive to the outcomes of the Learning Networks, irrespective of the quality or potential utility of these. As a result of these issues it was noted that there were few, if any, concrete examples of policy influence or transfer that emerged as a result of the Learning Networks. Section 5 Lessons learned and possible future improvements A165

242 A range of suggested improvements emerged from the case study. These can be summarised as follows: Ensuring events are well organised and engaging, particularly dissemination events (the ITM final event held in the UK was felt to be an exemplar of this in terms of bringing the impacts and positive work of projects to the fore in an engaging way) Greater support by the Commission for network activity and transnational co-operation more broadly (it was noted that the Common Framework offers the potential to address this) including senior actors with a clear vision of the benefits of co-operation and the ability to promote this Encouraging Member States to set up infrastructures to support transnational co-operation to act as a bridge between the policy and ESF project level, similar to the role taken by the ITM unit, and considering providing the resources to Member States (both financial and in terms of knowledge) to achieve this Having more realistic objectives for transnational co-operation in the sense that it should be acknowledged that wholesale policy transfer is impractical and that policy influencing can only be subtle and takes a long time Ensuring quality management of transnational networks in terms of the individuals charged with leading and co-ordinating them Become more systematic in recording the experiences of Learning Networks, their activities and their outputs More clarity on the goal of transnational co-operation in each instance in terms of what is it trying to achieve, how and why? A166

243 A167