Ms Éva Nádor, external expert

Project Title SEE Science Project Number SEE/B/0048/1.3/X Document Title FINAL EVALUATION Document Version 0.1 Document Origin Lead Partner Document Date 15 February 2014 Character of Document REPORT Related Document(s) (Origin Partner, Date) Target Group SEE Science Steering Committee Developed by Ms Éva Nádor, external expert 1

FINAL EVALUATION 1. April 2011-15. February 2014. February 2014. 2

CONTENT I. EXECUTIVE SUMMARY II. PROJECT BACKGROUND 1. Project objectives and methodology 2. Brief summary of the mid-term and interim reports 3. Objectives and methodology of the final assessment III. KEY FINDINGS 1. Management issues - project design - partners change - project management 2. Results regarding the defined objectives of the SEE Science project - Results in terms of outputs - Achievement of projected result indicators and targets - Budgeting issues - Preliminary assessment of outcome/impact IV. CONCLUSIONS APPENDICES SOURCES 3

I. Executive summary The main objective of the final assessment is to determine the extent to which the objectives as defined in the AF have been met and describe the extent project management helped project implementation to achieve project goals. The methodology of the final assessment was the combination of secondary and primary research. Secondary research was relying on the previous analyses and the documents elaborated since the interim report. The primary research also included personal interviews with the members of the LPMT and interviews with the project partners with the help of a structured questionnaire. Key findings 1.Management issues Project design: The SEE Science project was entirely in line with the Application Form. Both the methodology and the activities followed the revised implementation plan which was approved by the Steering Committee at its first meeting. The content related issues have been discussed by the PCT. The project design was mostly adequate to address the problems at hand however some issues have been raised. At the beginning there were lots of methodological discussions, partners needed a more pragmatic approach than first provided by the responsible partner. Work Package leaders and LPMT made lots of efforts to clarify the project design for the partners what made the project partners more comfortable with the methodology, the coherence of the different activities in the Work Packages. Partners change: The extensive partner change caused permanent difficulties to to follow the work plan of the AF. The replacing partners had to catch up with the tasks, what required more time than expected causing scheduling problems. By the end of the 4 th period most of the delays have been covered. However the partners change caused delays in activities, it did not influence the final outputs. The partners change primarily challenged the project management. Project management: In overall the project management of the SEE Science project was good. After the initial shortcomings significant improvement was noticeable during the implementation of the project. After the partners changes the activities were better synchronized. The change in LP s internal team was beneficial, both the cooperation of the partners and the communication among the partners improved. The recommendations of the mid-term and interim assessments helped the project 4

partners to improve the efficiency of their activities. The stronger internal relationships and the better understanding of the project had a positive impact on the quality and the delivery of the outputs. The partner level project management was satisfactory, although the competences of the project partners were different most of the project partners were active and creative. During the project valuable professional support was provided by PP3, PP4, PP12 and PP13. The external management team provided professional management support during the whole project. Due to the long procurement process and the internal management problems some of the partners had difficulties with the process of reporting and the related deadlines. SEE Science is characterised by significant underspending. The SEE Monitoring Committee has already made a decision about the withdrawal of 111.000 euros. Considering the total budget set in AF for the whole project, spending is at ~69% at the end of the 6 th period. Taking into consideration the forecasts given by the partners, SEE SC project will exceed the expected minimum (75%) level of spent budget. 2.Results regarding the defined objectives of the SEE Science project Results in terms of outputs : By and large the number of outputs was in line with the plan. Based on the partners analysis Transnational State-of-the Art was created, SC Service portfolio was developed. The network of SC agents has been developed, action plans for building or developing local/regional strategic partnerships have been designed, strategic partnerships have been established. SC agents coordinated small scale pilots targeted at children in early school age, students/ young adults and businesses (innovation agencies, research institutes, large companies, SMEs). SEE Science Centre as a virtual platform was developed, 3 successful Science Festivals have been organised. Achievement of projected result indicators: Small scale projects and the science festivals increased the project visibility and accessability. The small scale pilot activities were an opportunity for the general public and especially children and students to be actively involved in activities a science centre can offer for them. Partners agree that the SEE Science Festival series, and the small scale pilots helped all the partners to increase their visibility towards new public. Most of the other result indicators are close to the accomplishment, stakeholders have been involved in regional roundtable discussions, local/regional strategic partnerships have been built and public regional debate started on science, innovation and technology. The most problematic field is the IT, which expected to be sorted out only by the end of the project. 5

Elaboration of result indicators as Regional/ local policies improved or developed, Set of EU level policy recommendations formulated and Joint action plan for sustainable SEE cooperation is in progress, those papers will be discussed at the final event of the project and will be ready by the end of the 7th period. Preliminary assessment of outcome/impact: Based on the documents and the partners opinion the primary objective of the project seems to be achieved. All of the project partners feel that the SEE Science project brought important outputs for them. Most of the partners emphasized the importance of beeing a part of the SEE Science network. Project partners with existing SCs discovered new possibilities for their service portfolio, others without SCs feel stronger to establish a science centre. A wide range of the targeted groups have been reached by the project raising interest for science, innovation and technology. WP5 and WP3/3 were fully dedicated to ensure financial / institutional and political sustainability. This effort was supported by the project design. Strategic Partnerships have been established to increase the range of funding available to SCs in the longer term based on mutually beneficial cooperation with the business sector. Policy recommendations will be formulated both on national and international levels.these will sustain key activities that connect SEE Science with society in raising awareness and promoting a culture for innovation. The organised SEE Science Festivals enhanced visibility and accessibility. The joint work and cooperation of the partners envisage a good chance for the sustainability of the SEE Science network. The SEE Science Centre will provide virtual access to visitors via online and digital technologies. Partners are going to define a joint action plan to ensure the institutional and financial sustainability of the transnational pilots as well as the SEE Science network. The main guarantee for the sustainability of the project is the commitment of the project partners. At the 5th PCT meeting ideas were collected which were built in a document commissioned by the Lead Partner which show directions for further cooperation for SEE Science partnership. Partners are looking for further funding opportunities to continue the cooperation. 6

II. Project background II.1. Project objectives and methodology According to the unanimous opinion of the experts it is obvious that capacity for innovation is a key driver of economic growth. Key components of innovation are science and technology and promoting them in the economy and society offers a positive approach to generating innovation and improving human welfare. Due to several reasons weak culture for innovation, lack of proper framework conditions to bridge research, education and business innovation scores low in SEE. The overall objective of the SEE Science project was to contribute to the image formation of the SEE region as a place of innovation through increasing public awareness on the importance of natural sciences, technological progress and innovation. The SEE Science project investigated how Science Centres could improve in order to better serve the increase of public awareness on the importance of natural sciences, technology and innovation as key determinants of economic growth. The project concept was initiated by the Municipality of Debrecen as the Lead Partner inviting science centres, local authorities, research organizations and universities as PPs. The SEE Science project was a cooperation of 10 ERDF Partners and 8 Associated Strategic Partners. Among the ERDF partners there were 3 partners with functioning Science Centres ERDF PP3 MUSE, ERDF PP4 Science Center Network Austria, ERDF PP12 CSE. At this point it has to be mentioned that the planned contribution of the Associated Partners could only be realised to a very limitied decree. The project specifically aimed at waking the interest of young people for science, technology and innovation by creating new methods to increase visibility and accessibility through science centres. Implementation was broken down to 3 phases, with stakeholder involvement as a continuous overarching activity. 1. Joint analysis of operation & services of SCs; benchmarking & knowledge building by appointed SC Agents to create Transnational State of the Art Report and draft SC Service Portfolio. 2. Networking by SC Agents to develop cooperation with innovation actors; implement pilot actions to test jointly selected SC services and to create & test main transnational outputs of SC cooperation. 3. Finalise core outputs: build pilot results into final SC Service Portfolio; define joint action plan to ensure sustainability of transnational project outputs and the SEE Science network; finalize action plans for improved SC operation/ strategies for setting up SCs at PP level. 7

The activities to achieve the project objectives were organised in 5 Work Packages, such as - Work Package 1 Transnational Project and Financial Management - responsible - LP - Work Package 2 Communication activities - responsible ERDF PP1- UniDeb - Work Package 3 Improving the operational management and services of Science Centres in SEE- responsible ERDF PP 11 SCSTI (replacing LUDUS ) - Work Package 4 Improving the role of Science Centres as catalysts of innovation - responsible ERDF PP4 SCN - Work Package 5 Sustainable SEE cooperation of SCs for increased awareness on the role of technical sciences on innovation- responsible ERDF PP3 MUSE In the course of December-January 2013 a Mid-term assessment has been conducted. The main objective of the evaluation was to provide the Joint Technical Secreteriat (JTS) South East Europe Programme, the Lead Partner, the Municipality of Debrecen and the project partners an independent review of the status, relevance and performance of the project as compared to the project documents. The primary task of the Mid-term report was to determine the extent to which the objectives as defined in the Application Form have been met as of the date of the assessement, and estimate the likelihood of achieving them upon project completion. In September 2013 an Interim report has been prepared. The main objective of the evaluation was to provide the LP an interim review of the project progress based on the continuous monitoring activity. The recommendations aimed to help the LP and the partners to make the necessary measures to ensure the successful execution of the project. II.2. Brief summary of the Mid-term and Interim reports Methodology of the Mid-term evaluation In order to achieve the goals of the Mid-term evaluation integrated research methods such as - secondary and - primary researches were applied. The evaluation were based on the findings and factual statements identified from the review of the relevant documents; Application Form Boosting innovation through capacity building and networking of science centres in the SEE region, Implementation Plan, Progress reports/ Amended 8

Progress reports and AfR, Minutes of the Steering Committee and the Project Coordination Team meetings, Applications for Reimbursement, the project website. Besides secondary sources the evaluation also included field research. Visits and interviews with the LPMT; the Overall Coordinator, the Project Manager, the Project Communication Manager and the External Management Team members were interviewed personally. The project partners were interviewed by the help of a structured questionnaire. The aim of the methodology was to be able to gain opinions, expectations and vision of the project partners towards the achievement of its objectives. Evaluation criteria: - Operation of the Project Management; Lead Partner (Project Management Team, External Management Team), Steering Committee, Project Coordination Team - responsibilities, tasks, documentation - Implementation by the project partners; in terms of quality, timing, activity and spendings - Activities, outputs by Work Packages, deviations in the plan. Key findings of the Mid-term analysis The Mid-term evaluation pointed out that the Partnership is relatively strong. The cooperation with the LP and the LPMT is excellent, the cooperation among the project partners is good. The partnership can be described by a good team work. Taking into account the strong commitment of the LP and the expectations of the project partners the realization of the SEE Science project seemed to be feasible in case the necessary changes in managing the process and finances will be made as intended. By and large the Municipality of Debrecen, as Lead Partner, was managing the project efficiently. The internal and external management was working closely together both on strategic and operational level. The leadership of the project was in the hands of the Lead Partner s Internal and External Project Management Team having a wide range of competences and the capacity to guide the partners. The assessment revealed also the key problems the SEE Science project. Due to some delay in the official notification of approval from the JTS the actual project activities started 2 months later. 4 partners withdrew from the project after the kick - off meeting which caused a hiccup in the actual project start. The detailed analysis of the activities in the Work Packages showed that there were always significant deviations concerning timing compared to the timeframe of the AF. It caused temporary difficulties in the implementation of the project for the partnership. The replacing partners had to catch up with the tasks, what required more time than expected causing scheduling problems. 9

By the end of the 4 th period (30. November 2012) most of the delays have been covered. The performance of the Work Packages differed. The evaluation of the WP2 was contraversial. Although the delay of activities was obviously affecting the communication the intensity of communication could be improved. More proactivity and creativity was expected. WP3 management and operation was satisfactory, problems arised regarding the SWOT methodology what caused significant delay in implementation. Management of WP 4 and WP5 was strong. Major problem occurred in terms of financing. SEE Science was characterised by significant underspending. The SEE Monitoring Committee has already made a decision about the withdrawal of 111.000 euros. Some of the replacing partners could not validate their progress reports by the National Controller. Partner change has also contributed to the high level of underspending. Key findings of the Interim report Mid - September 2013 an Interim assessment has been conducted. The evaluation was based on the findings and factual statements identified from the review of the relevant documents elaborated between December 2012 and September 2013. Based on the Interim evaluation it was clear that the partnership is strong, project partners are committed to the project success. LPMT is managing the project efficiently. Although the major content related activities were going to be accomplished in the 6th and 7th period the realization of the SEE Science project seemed to be feasible in the period of the Interim report. The SEE Science project was still facing considerable underspendings and due to that another possible decommitment. The reason behind that was that some partners were not able to report costs in time or costs cannot be declared in time. In order to meet the project objectives further control of the activities in the Work Packages, based on personal relationships, communication and output monitoring was suggested. The Interim analysis also recommended the strict monitoring of the preparation of the interim financial report, keeping the deadline of submission. II.3. Objectives and methodology of the final assessment The main goal of the final assessment is to determine the extent to which the objectives as defined in the AF have been met and describe the extent project management helped project implementation to achieve project goals. The final assessment is focusing on the following issues: 1. Management issues- 10

Project design: Changes in context and review of assumptions: Is the project s design adequate to address the problems at hand? Does the project remain relevant considering possible changes in context? Changes in partnership: Has the partner changes any consequences on the partnership, on the project implementation, the final outputs of the project? Project management on project and partner level: To what extent the project management helped the project implementation to achieve the project goals? Have the project s activities been in line with the schedule of activities as defined by the work plan in AF? Have the project expenditures been in line with expected budgetary plans? What internal and external factors have influenced the ability of the LP and the partners to meet projected targets? 2. Results regarding the defined objectives of the SEE Science project Results in terms of outputs achieved vis-á-vis projected targets (efficiency): Has the project reached the expected number of outputs (i.e.. analysis, partner meetings, conferences, transfer visits to be implemented etc?) Achievement of projected result indicators and targets (effectiveness): What has the performance been like with respect to the projected result indicators? Have there been any unplanned effects? Preliminary assessment of outcome/impact (effectiveness): Has the project generated any results so far that could indicate that the implemented actions have had an impact on the waking the interest of young people for science, technology and innovation and that the visibility and accessibility of science centres were successfully increased in partner cities. Methodology The final assessment adopted both secondary and primary research. Secondary research was relying on the previous analyses Mid-term and Interim reports and the documents elaborated since the Interim report 5 th PCT meeting, 5 th Progress report and project outputs. The primary research included personal interviews with the members of the LPMT and interviews with the project partners with the help of a structured questionnaire. 11

III. KEY FINDINGS III.1. Management issues Project design The SEE Science project was entirely in line with the Application Form. Both the methodology and the activities followed the revised implementation plan. The content and administrative issues have been discussed at the PCT meetings while strategic decisions were made at the SC meetings. At the beginning there were lots of methodological discussions, partners needed a more pragmatical approach than first provided by the responsible partner. The project partners expressed that the project s design was mostly adequate to address the problems at hand however some issues have been raised. In the opinion of some project partners lower number of participating partners could ensure bigger transparency and a more effective communication among them. Another key issue is the selection of the project partners. In the opinion of some partners it is crucial to select committed partners. It was also mentioned that higher involvement of the partners in the project design and more meetings, conferences during the project could ensure better understanding of the project methodology and contribute to the project success. Partners change 4 partners withdrew from the project after the kick-off meeting which caused a hiccup in the actual project start. The extensive partner change caused deviations in timing compared to the AF. It caused temporary difficulties in the implementation of the project for the partnership. The replacing partners had to catch up with the tasks, what required more time than expected causing scheduling problems. By the end of the 4 th period (30 November 2012) most of the delays have been covered. However the partners change caused delays in activities, it did not influence the final outputs. The partners change primarily challenged the project management. LPMT had to make lot of efforts to handle the problem, to replace the former partners and to manage the administrative processes. The project partners emphasized that the delays in activities did not change the final outputs, the catch up of the new partners went smoothly, however they needed more time than expected. 12

Project level and partner level management Organisational structure TRANSNATIONAL PROJECT LEVEL LEAD PARTNER LEAD PARTNER s MANAGEMENT TEAM (LPMT) (certain changes in the internal team later) - Internal (Overall Coordinator, Project Manager (PM), Communication Manager (PCM), Financial Manager (PFM) - External (Project Manager, Financial Manager) PROJECT COORDINATION TEAM (PCT) TASKS Responsible for the coordination, supervision and management of the project. Reporting and setting up and maintaining good working relations with the programme authorities and the partners. Appointed by the LP is responsible for the daily management and coordination of activities. External experts assist the internal team in the efficient project management and implementation. LPMT and PPM form the PCT. PCT is discussing both professional and operational issues regarding implementation and results. PROJECT STEERING COMMITTEE (SC) PARTNER LEVEL PARTNER LEVEL PROJECT MANAGEMENT TEAMS (PPMT) - Project Partner Manager (PPM) - Project Financial Manager (PFM) SC members are appointed by the Project Partners. SC is a strategic decision making body ensuring that the project objectives are fulfilled. The major task is to approve basic project documents, to monitor implementation and take decisions on key issues and intervene, if needed. Responsible for the project implementation on partner level. In charge of project activities, reporting to the National Controller and the LP (Partner reports etc.) 13

Project level management Lead Partner s Management Team Municipality of Debrecen as Lead Partner, was responsible for the successful implementation of the SEE SC project. The management team was composed of internal and external project management team, working closely together. The ultimate performance of the project management was the result of this cooperation. Initially the Internal Management Team involved the overall coordinator, the thematic expert, the project manager, the project assistant and the financial manager. There were significant administrative changes at the LP. The project manager who was also in charge of communication left the project in February 2013, the new project manager was active since then. The overall coordinator left the project in June 2013. The new project manager successfully caught up to the project relying strongly on the external project management working closely together. Internal staff strengths: commitment, motivated employees, expertise in project management, good relationship with the project partners and the programme bodies (VÁTI, JTS). Internal staff weaknesses: heavy workload, other responsibilities, staff changes, lack of international experiences, weak leadership in terms of motivating the project partners. The work of the Lead Partner was supported by the External Project Management Teamand they worked closely together, management issues were thoroughly discussed. LP was delegating part of its responsibilities to the external consultant. The External Project Management Team members have a long record in project management as such and in EU Interreg programmes as well. The external advisor is characterised by professionalism, in aware of the methodologies and procedures. The SEE Science program faced lots of difficulties what could not be bridged without this problem solving capability. External Project Management Team Strengths: professionalism, experience, expertise, good management skills, good problem solving capabilities, good communication skills, efficient relationship management, language skills, effective cooperation with the internal staff and the JTS. The conclusion of the Mid-Term analysis was that the LPMT is highly motivated, the project partners are satisfied with the support provided by the LPMT however more direct contact was expected. The overall impression was that a stronger leadership by the LP could be more effective. The performance of the External Project Management Team was highly ranked by the project partners. Among others 14

the extensive knowledge of the SEE Programme and the good leadership qualities were highly appreciated. At the same time some project partners mentioned that the split of the leadership between the LP and Grants Europe is not explicit what is sometimes weakening the overall lead of the project. The Interim monitoring report showed certain improvement, the direct involvement of the LP was higher. The new project level manager established direct relationship with the project partners, mostly in form of e-mailing. Another improvement in the project management was the newly introduced output monitoring system what made the up-to-date monitoring possible. Lead Partner was able to follow the status of the Work Packages outputs and to intervene if necessary. Based on the PCT and SC minutes and the partners opinion the recommendations of the Mid-term evaluation contributed to the improvement of the project implementation. The project partners agreed that the recommendations have been taken into consideration. LPMT improved the communication with the project partners, the links among the WPs were better explained, the project partners got more explanation about the expectations regarding results and outcomes. Due to those changes the deadlines were kept better than before, Project Coordination Team (PCT) The Project Coordination Team consisted of the members of the LPMT and PPMT. The PCT is the overall operative, decision preparatory body of SEE Science, serving as a base for the whole implementation of the project. During the SEE Science project 6 PCT meetings were planned. The 6th meeting will be held in February 2014. At the 1st meeting PCT Rules of Procedures were discussed. The key content issues presented by the WP leaders and discussed by the PCT were the common SWOT methodology, the Communication Plan, the Knowledge sharing strategy, the task of the SC agents, SEE Science Centre start-up activities and the preparation for science festivals. The 2nd PCT discussed the revised Implementation Plan, the Financial Management issues, the ASP Management, the Science festivals methodology and the progress of SWOT analyses. The 3rd PCT meeting among others discussed further the SEE Science festival methodology, the benchmark visits, the upcoming events and included a website training a knowledge sharing session. Formulation of pilot teams, first ideas for the small pilot actions and management issues were also discussed. The 4th PCT meeting discussed the key issues of the project implementation in WP3, WP4 and WP5. Among others how the results of the TSAR are going to be utilised in the Operational Plans for Improved Operation/Strategies for setting up SCs, how findings of the TSAR will be integrated to Small Scale Pilots and Transnational Pilots, issues related to the project Policy Recommendations on local level (Municipality or region or national level) and EU level, SC Agents and their network and the operation of the Virtual Science Center. 15

On the agenda of the 5th PCT meeting there were general project and financial issues, among others the Interim report and the financial issues regarding the project closure. PCT members reviewed the final documents of the Work Packages and the tasks regarding SC agent network, strategic partnerships, SC Science Centre and Science festivals. Partners agreed on the methodology of policy recommendations and discussed the ways of sustainable cooperation. In general the PCT was functioning well, the meetings were efficient, discussing content related issues. Based on the minutes project partners participated on the PCT meetings and the activity of the members was sufficient. PCT meetings were pragmatic in the sense that the upcoming tasks, deadlines and responsible partners were always defined. Steering Committee (SC) The Steering Committee was the supreme decision-making body of the SEE Science Project. In principle it involved decision makers of the project partners representing the decision making level of the partner organisations. During the SEE Science project 3 SC meetings were planned, the 3rd SC meeting will be held in February 2014. The 1 st meeting was attended by all project partners. The partners approved the Rules of Procedures of SC, discussed and approved the Project and Financial Management Guidelines. The LP gave an overview of the SEE Science Project. Furthermore the SC approved the revised Implementation Plan. The 2 nd SC meeting discussed the financial issues and made recommendations for the partners. Furthermore the results of the Mid-term analysis making recommendations for the partnership have been presented and approved by the SC. Finally the partners discussed the possibility to join the European Science Events Association (EUSEA) in terms of sustainability of SEE Science Festival and the Virtual Science Center. Considering the whole SEE Science project the project partners expressed a very positive opinion regarding the LPMT. The efficiency of the internal team was improved with the help of the new team members. The LP and its team provided a big support for the project partners in the daily project management. They appreciated the professionalism, expertise and competence of the Grants Europe, particularly its leadership, efficient trouble shooting, successful cooperation, quick response and consultation on reports, spendings and management issues have been emphasized. Some negative comments have been made too: some partners would expect more meetings, conference conversations, better mediation of issues among the partners and stronger lead on content level. One partner was in the opinion that updating of the implementation plan would be more efficient than new schedules defined in the PCT memos. 16

All of the project partners agreed that significant improvement was noticeable during the implementation of the project. After the partners changes the activities were better synchronized. The change in LP s internal team was beneficial, both the cooperation of the partners and the communication among the partners improved. The recommendations of the Mid-term and Interim assessments helped the project partners to improve the efficiency of their activities. The stronger internal relationships and the better understanding of the project had a positive impact on the quality and the delivery of the outputs. Partner level project management In overall the partnership became a good team, the competence profile of the partners was beneficial regarding the projects objectives, balancing the different competences of the partners. However the competences of the project partners were different most of the project partners were active and creative. Valuable professional support was provided by PP3, PP4, PP12 and PP13. Due to the long procurement process, the internal management problems, the strict rules regarding the expenditures some of the partners had difficulties with the process of reporting and the related deadlines. Despite the ongoing deviations from the original implementation plan by the 5th period most of the activities have been in line with the schedule. Majority of the partners assessed the partner level project management good, regarding timeliness and contribution of the partners the number of good and satisfactory opinions were equal. Some of the partners mentioned the lack of support from their local representatives and their heavy workload as hindering factors. Problems occurred during the project However there was ongoing delay in the activities by the end of the 4 th period most of the outputs have been reached. There were both project and partner level reasons of deviations from the schedule defined in the AF. The delayed start of the SEE SC project, the extensive partners change and the long administrative process of the partner changes caused constant delays in the activities. The new partners had to catch up to the project what took more time than expected. Furthermore there were some methodological issues to be solved, rather pragmatic than academic methods have been expected. Less experienced partners needed some explanation on the project design, particularly on the coherence among the Work Package activities. 17

Budgeting According to the Progress Reports the SEE Science project generated substantial under spending during the whole project. Due to the under spending in October 2012 a decision was made by the MC about the de-commitment of 111.000 euros. Considering the total budget set in AF for the whole project, spending is currently only at ~69% at the end of the 6 th period. The main reasons of the under spending were the followings: - The actual start of the project was 2 months later than officially announced. - The extensive partner change caused delays in the activities, most of the events have been postponed. - Due to the problems related to National Controls some partners were not able to report costs in time or costs cannot be declared in time. - ASP participation in the SEE Science Project is far less active than planned. The travel cost of ASPs was supposed to be financed by the project partners. Lack of those activities is also decreasing the level of under spending. There are significant differences in the spending of the project partners. They mentioned several problems regarding budgeting. Some partners were not able to report costs on time in lack of clear assignments and staff member selection. Others mentioned the long procurement process locally. In other cases the different financial systems, strict national rules regarding staff costs made the financing more difficult. For the Greek partner the necessity of pre-financing resulted a limited spending. Taking into consideration the forecasts given by the partners, SEE SC project will exceed the expected minimum level of spent budget (75%). The only one partner not able to spend the budget due to lack of pre financing possibilities is CSE. LP has already given notice to JTS about the remaining 120 000 euros at CSE. The SEE Science project was entirely in line with the Application Form. In overall the project management of the SEE Science project is good. After the initial problems significant improvement was noticeable during the implementation of the project. The LPMT helped the project partners to feel more comfortable regarding the methodology of the project. The extensive partners change caused delays in activities. The partners change primarily challenged the project management. LPMT had to make lot of efforts to handle the problem, to replace the former partners, to manage 18

the administrative processes and help the new partners to catch up. Due to several reasons SEE Science project generated substantial underspending during the whole project. III.2. Results regarding the defined objectives of the SEE Science project The main objective of the SEE Science project was to wake the interest of young people for science, technology and innovation by creating new methods to increase visibility and accessibility through science centres. A detailed analysis of its implementation follows below. III.2.1. Results in terms of outputs The assessment of the results is based on the outputs of the content related Work Packages. (WP3-4-5) The activities in the WP3, WP4 and WP5 were closely interrelated. Work Package 3 Improving the operational management and services of Science Centres in SEE Responsible partner: ERDF PP11 - SCSTI This activity served as a base for the other content related Work Packages, therefore its role was crucial regarding the performance of WP4 and WP5. Science centres are the main pillars of SEE Science project due to their potential to become main communicators of innovation and technology development for the public and awaken interest of the young. The activities in WP3 were aiming to enable SCs in being more efficient in their operation and in fulfilling this communicator role, and intended to combine the complementary strengths of individual SCs in the SEE area. For the partners currently not operating SCs it gave a strategy how to set up a SC and how to ensure its efficient operation. Based on the conclusions of the Transnational State-of-the-Art Report (TSAR), the finalised SC Service Portfolio and the Strategic Partnership Action Plan the project partners prepared their Action Plans for Improved SC Operation (existing SCs) and Strategies for setting up SCs in case where SCs do not exist yet. This way SCs could fulfil their potentials as catalysts of innovation. Activity 3.1:State of the Art The work started with creation of SWOT and/ or Feasibility studies of each project partner and SEE Science Benchmark Studies on good practice in European science centres. Based on the regional data a SEE Science Transnational State-of-the-Art Report(TSAR) was created defining common development opportunities for the SCs in SEE. 19

Activity 3.2: SC Service Portfolio Project partners developed a SC Service Portfolio, what means an inventory of efficient services, awareness raising tools and methods for Science Centres collected from practices of other science centres and from the partners small scale pilot activities. Activity 3.3: Stakeholders and policy makers To successfully promote the role of the SCs in facilitating innovation and increasing public awareness roundtable discussions were organized by the project partners. The actual value of the stakeholders involved in regional roundtable events and consultations is close to the targeted. The roundtable discussions proved to be very efficient involving representatives from the fields of business, education and policy level such as enterprises, universities, schools, research institutions, NGOs, government bodies. One key result of the SEE SC project will be the policy recommendation what is based on the key messages of local SWOTs and the TSAR, the conclusions of small-scale pilot reports, the evaluation of Science Agent activities and the Science Centre Service Portfolio. All will be channelled into policy recommendations with the aim to result in effective policies at local and EU level which contribute to improved environment for functioning of the SCs so that they better fulfil their mission. Development of policy recommendations for policy makers and funding programme authorities on partner level is in progress, 4 policy recommendations have already been prepared. Based on the partner level recommendations a EU level policy recommendation will be elaborated in the 7th period. Work package 4 Improving Role of SCs as catalysators of innovation Responsible partner: ERDF PP 12 - SCN The key actors of WP4 were the SC Agents and their network. In line with the overall project objectives, the mission of SC Agents was to help SCs extend their competences and functions, strengthen linkages with regional SEE innovation actors and build up a network of SCs based in SEE and beyond, contribute to the visibility of SC activities. Activity 4.1 Knowledge building of SC agents SC Agents have been appointed by the project partners. After the SC agents have been appointed and the knowledge sharing strategy was finalised an in-house training of SC agents was organised. The SC agents gathered a solid knowledge base through benchmark visits, training sessions and sharing of experience within the transnational SC network. SC agents conducted more than 25 benchmark visits 20

to science centre institutions. The results fed partly into the SEE Science Service Portfolio, but also contributed to the internal profile development and improvement of partners operation with a focus on innovation. Activity 4.2. Knowledge sharing within the partnership and the SC agents network Internal knowledge sharing workshops in partners organisations - PP4 SCN, PP3 MUSE, PP13 SEERC, PP14 CRA - and a transnational knowledge sharing session were organised. Activity 4.3. Developing strategic partnerships to promote innovation One of the key tasks of the SC Agents was to establish strategic partnerships through networking. Strategic partnerships enable SCs to enhance the effectiveness of cooperation with innovation actors and to increase the efficiency and significance of SC services offered. Such partnerships are also envisaged increasing funding possibilities (business and public) for SCs, and the dialogue with stakeholders will contribute to creating supportive policy frameworks. Based on the methodology elaborated by PP13 SEERC SC agents were drawing up their Action Plans for building and/or developing local/regional strategic partnerships (PP13 SEERC, PP3 MUSE, PP4 SCN, PP8 BM, PP10 UniOr, PP11 SCSTI, and P14 CRA). In order to build capacity in the South East Europe as a whole developmental block, a Joint action plan for strategic partnerships has been devised in order to build strategic partnerships at the wider regional and transnational level towards increasing the powerful impact of the SEE Science project outcomes. Activity 4.4. Implementing small-scale pilots The idea of the pilots was to develop and test innovative science centre services and activities, and share experiences and advice within the partnership through peer visits. Small-scale pilots targeted 3 groups: children in early school age, students/young adults especially young women, innovator agencies, research institutes, large companies, SMEs. The small scale pilots were coordinated by the SC agents. 13 different pilot activities based on the state of the art analyses and good practices collected - were developed and implemented by the partners. About 4000 individuals have been reached through the pilot activities so far and some peer visits were accomplished too. The small-scale pilot was successful not only in terms of numbers but the new workshop formats and contents. New forms of science communication such as virtual workshops were tested and innovative information materials for potential strategic partners and stakeholders were developed as hands-on mini-exhibits etc. 21

WP5 Sustainable cooperation of SCs for increased awareness on the role of technical sciences on innovation Responsible partner: PP3 MUSE Activities within WP5 contribute to the sustainable cooperation of SCs in SEE for increased awareness on technical sciences to boost innovation by implementing two transnational pilot activities (SEE Science Centre and SEE Science Festivals) and developing a Joint Action Plan for sustainable SEE cooperation. Activity 5.1: Transnational pilot 1: SEE Science Centre The SEE Science Centre is a virtual platform that aims to strengthen existing relationships within the SEE Science consortium and to create new ones through networking and the sharing of resources. Inspite of some technical problems since the start of the platform in Spring 2012, the successful operation of the platform could be indicadted by figures below: 10 educational activities, shared among the partners through the library section of the portal; 16 video clips, 5 of which were specifically produced for the web portal, showing important Science events in the SEE region; 22 picture galleries, containing more than 2000 original pictures with benchmark examples of Science centre activities from all across Europe; 65 blog posts by all 10 SEE Science partners, covering a wide array of subjects from robotic road show to zombie lab; 1 online Science Centre services portfolio, serving as an interactive database for innovative Science Centre activities in the SEE region. During the last 12 months the portal received more than 7000 visits from almost 4000 unique visitors with an average visit time of 5 minutes. The portal served also as a hub and showcase for all the other activities of the SEE Science project, such as the SEE Science Festival series, the small scale pilot activities, and the Science Centre agent network. Activity 5.2: Transnational pilot 2: SEE Science Festival The SEE Science Festivals were supposed to contribute directly to raising public awareness on science and technology. These events created a more inclusive environment and made innovation more accessible to the wider public. The most important aim was to turn these events into a sustainable instrument to support the image formation of South-East Europe as a place of innovation and economic growth. 22

During the project 3 Science Festivals have been organised: 18. October 2012. in Thessaloniki, 22 March 2013 in Debrecen and 26-27 September 2013 in Burgas. The 4 th Science Festival will take place 21 February 2014 in Trento. All festivals had different main topics, presented in different zones. The festivals were succesful, with 1700, 3600 and 1000 visitors respectively. In the opinion of the project partners the number of outputs partner meetings, analyses, conferences, roundtable discussions, trainigs, festivals was in line with the plan. The only one shortage within this activity was mentioned regarding the benchmark visit to Scientival in Ljubljana Due to the fact that the study visit to Sciencetival (Ljubljana) could not been organised together with the kick-off meeting less project partner had the budget to finance the travel to the study visit. That resulted in that fewer participants attended the study visit (4 compared to 20 indicated in the AF). Achievement of projected result indicators However an entirely precise evaluation of the SEE Science project will be possible only after a couple of month after the project end, there are still some results we can rely on to predict it now. The previous point reviewed the activities of the Work Packages, evaluating the project efficiency in terms of outputs and numbers. Another issue is the effectiveness of the project, what has the performance been like with respect to the projected result indicators. The AF contained the following result indicators: - Public regional debate started on science, innovation and technology, - Regional/ local policies improved or developed, - Set of EU level policy recommendations formulated, - Stakeholders involved in regional roundtable events and consultations, - SC Centres with improved capacities raise awareness of and facilitate innovation, - Partners with improved capacities to set up SCs with innovation facilitator role. - Improved capacities of the staff through trainings and knowledge sharing, - Local/regional strategic partnerships built, - SEE Science festival tool to increase the project visibility, - IT infrastructure to increase the project visibility, - Joint action plan for sustainable SEE cooperation, - Individuals directly reached by the transnational SEE Science Festivals. 23

The project partners evaluated the achievement of the result indicators. Most of the partners agree that the following results have already been fully achieved by the SEE Science project: - Stakeholders involved in regional roundtable events and consultations, - Improved capacities of the staff through trainings and knowledge sharing, - Individuals directly reached by the transnational SEE Science Festivals, - SEE Science festival tool to increase the project visibility, - Partners with improved capacities to set up SCs with innovation facilitator role. Project partners emphasized the importance of small scale projects and the science festivals in the increase of project visibility and accessability. The small scale pilot activities were an opportunity for the general public and especially children and students to be actively involved in the life of the Science centres. Partners agree that the SEE Science Festival series, and the small scale pilots helped all the partners s science centres to increase their visibility towards new public. Most of the other result indicators are close to the accomplishment, local/regional strategic partnerships have been built and public regional debate started on science, innovation and technology. In the opinion of the project partners the most weak field is the IT, there are still lots of problems with the IT infrastructure. Elaboration of result indicators as Regional/ local policies improved or developed, Set of EU level policy recommendations formulated and Joint action plan for sustainable SEE cooperation is in progress, those papers are going to be ready by the end of the 7th period. Preliminary assessment of outcome/ impact Based on the documents and the partners opinion the primary objective of the project seems to be achieved. The science festivals attracted a great number of visitors, the roundtable discussions resulted expanding relationships with the stakeholders. The pilot projects generated interest for science and innovation and attracted new target groups as well. The project partners agreed that thanks to the dissemination of information about SCs, the efficient advertisement of the science festivals the visibility and accessibility of science centres were successfully increased. The information available on the Virtual Science Centre also helped to increase the visibility, giving information for students and stakeholders however there are still problems to be sold. 24