Final Evaluation of the Eurostars Joint Programme

Transcription

1 Ref. Ares(2014) /11/2014 Final Evaluation of the Eurostars Joint Programme Expert group Marja Makarow, Chair Georg Licht, Rapporteur Isabel Caetano Dirk Czarnitzki Sirin Elçi November 2014

2 Final Evaluation of the Eurostars Joint Programme Final report 24 November 2014 Marja Makarow (Chairwoman), Georg Licht (Rapporteur), Expert group members: Isabel Caetano, Dirk Czarnitzki and Sirin Elçi This document has been prepared for the European Commission however it reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein. November 2014 Page 1 of 139

3 Table of Contents Table of Contents... 2 Tables, figures and boxes... 5 Executive summary Introduction Historical and policy context Background Objectives of the final evaluation and evaluation criteria About Eurostars History of Eurostars Scope and objectives of Eurostars How does Eurostars work? A multilateral approach Selection and funding mechanisms of Eurostars projects Facts and figures Results and recommendations of the Interim Evaluation The Interim Evaluation EC Report on the Interim Evaluation Response of the EUREKA Secretariat to the recommendations of the Interim Evaluation The Agilis Report Eurostars Data Collection and Assessment Objectives Findings Data quality and data cleaning Assessment of the project documentation forms Assessment of the classification of technology area and market area Recommendations Methodologies and data sources Introduction Desk research and data sources Interviews with programme administrators and SMEs Information based on data from ESE Online survey Quantitative Analysis November 2014 Page 2 of 139

4 6 Evaluation findings Relevance, efficiency and effectiveness of Eurostars Relevance for target group Relevance within EUREKA Adequacy of funding level Efficiency of programme implementation Efficiency of programme governance Impacts of Eurostars Impact on employment growth: an econometric analysis Impact on innovative outputs and IPR Behavioural Additionality: Impact on R&D Strategy, R&D Management, and R&D collaboration Overall Added Value for an R&D performing SME Summary of programme impact study Programme-level effects Contribution of Eurostars to the European Research Area Economic and social impacts of Eurostars Sustainability of Eurostars Impact of financial and economic crisis Uptake of recommendations from Interim Evaluation Report Summary of findings Assessment of findings Target group and scope of the programme Governance Management and Operations Funding Outcomes and impacts of Eurostars Recommendations Target group and scope of the programme Governance Management and Operations Funding Annexes Supplementary tables Supplementary methodological information The expert group Members of the expert group November 2014 Page 3 of 139

5 9.3.2 Term of reference: mandate, scope and objectives of the Final Evaluation Working methods and meetings Interviewed programme authorities and programme management Interviewed SMEs List of Abbreviations List of Country Codes November 2014 Page 4 of 139

6 Tables, figures and boxes Table 3-1: Key figures for the Eurostars programme Table 3-2: Status of Eurostars projects by December Table 3-3: Funding by National Funding Bodies and the EU Commission Table 5-1: Combination of data sources and methods used in this report Table 6-1: Share of SMEs in total project cost requested Table 6-2: Funding requested and commitments for funded projects Table 6-3: Descriptive statistics for evaluation sample Table 6-4: Difference-in-difference regressions on average annual employment growth Table 6-5: Fuzzy RDD: IV regressions on average, annual employment growth Table 6-6: Probit regression on probability of subsidy receipt Table 6-7: Matching results Table 6-8: Difference-in-difference regressions on change in patent stock: subsample of firms ranked above threshold Table 9-1: Key figures for Eurostars per budget year Table 9-2: EUREKA projects and project cost in 2013 by country Table 9-3: Number of applicants by country and cut-off Table 9-4: Number of applicants above threshold by country and cut-off Table 9-5: Number of applicants with funding contracts signed Table 9-6: Table 9-7: Table 9-8: Share of applicants in projects above the threshold as % of number of applicants Share of applicants with signed funding contract in per cent of the number of applicants above threshold Share of applicants of broad technology categories by country (in per cent at applicant level) Table 9-9: Distribution of types of participants by country (in per cent) Table 9-10: Results of Cox-regression of determinants of time-to-contract Table 9-11: National funding rules for Eurostars Table 9-12: Response rates to the online survey by funding status and country group Table 9-13: Meetings of the expert group November 2014 Page 5 of 139

7 Figure 3-1: Total number of applications by status and cut-off Figure 3-2: Total number of applicants and funded applicants by cut-off Figure 5-1: Status of approved projects by cut-off by December Figure 6-1: Applicants and participants by organisation type Figure 6-2: Distribution of SMEs by age Figure 6-3: Broad technology categories of funded and not funded projects (at project level) Figure 6-4: International sales of SMEs applying for Eurostars Figure 6-5: Motives for choice of partner by SMEs Figure 6-6: Difficulties encountered in partnerships Figure 6-7: Evolution of EUREKA s portfolio Figure 6-8: Figure 6-9: National funding for Eurostars in relation to national GBAORD 2011 (%) National funding for Eurostars in relation to the estimated public funding demanded by projects above the threshold (%) 1) Figure 6-10: Initially earmarked budget, budget extensions and unspent budgets (in million Euro) Figure 6-11: Working of Eurostars virtual common pot vs. a real common pot Figure 6-12: Country of origin of technical experts in the Eureka database Figure 6-13: Cumulated probability for still waiting for funding contract Figure 6-14: Time-to-contract by country Figure 6-15: Scatterplot and regression of employment growth on initial firm size Figure 6-16: Hypothetical illustration of data conforming to a sharp RDD: RDD applied to Eurostars data Figure 6-17: Distribution of propensity scores in treatment and control group Figure 6-18: Histogram of differences in project start dates in application and survey Figure 6-19: Implementation of non-funded projects Figure 6-20: Non-monetary benefits of Eurostars funding Figure 6-21: Impact of the programme in award-winning firms Figure 6-22: Share of SMEs which will likely reapply for Eurostars Figure 6-23: Public budgets for R&D in countries participating in Eurostars November 2014 Page 6 of 139

8 Box 1: Country participation in Eurostars Box 2: Article 185 of the Treaty on the Functioning of the European Union (TFEU) Box 3: What is EUREKA? Box 4: What is an R&D-performing SME according to Eurostars definition? Box 5: Recommendations from the Eurostars programme Interim Evaluation Box 6: What is effectiveness and efficiency? Box 7: Funding rules Box 8: Harmonization of funding rules results of a survey Box 9: Results of the survey on staff turnover in ESE Box 10: Impact assessment study November 2014 Page 7 of 139

9 Executive summary The Eurostars Joint Programme has succeeded in accelerating the growth and innovative outputs of R&D-performing SMEs. However, several aspects of governance and managerial implementation need to be improved. The expert group studied the Eurostars interim evaluation of 2010 and the uptake of its recommendations. In contrast to the interim evaluation, the expert group was able, due to the considerably higher numbers of applications, applicants and funded projects, to make quantitative calculations and carry out extensive econometric analyses. The results of the final evaluation provide evidence that Eurostars is relevant for the growth of R&D-performing SMEs in Europe. The employment growth rate of R&Dperforming SMEs funded by Eurostars was nearly twice as high as that of applicant SMEs which were not funded. This can be causally attributed to Eurostars funding. Given the still relatively small number of completed projects, this estimate will become more precise in future. In addition, the market impact of Eurostars projects has not completely manifested itself yet. The programme has accelerated the development and roll-out of new and improved products, processes and services. The econometric evaluation established a positive and significant impact on the patent portfolio of funded firms relative to unfunded applicants. The programme has stimulated new cross-border collaborations that the members of the funded consortia intend to continue beyond the Eurostars funding period. At the same time, the final evaluation reveals the need for improvement in harmonising funding rules and synchronising national processes. The effectiveness of central governance, administration and operations also needs to be improved. National resources are still too scarce, though some countries have increased their Eurostars budgets significantly above the originally agreed levels. The maximum rate of funding for the same type of partners varies between countries. Insufficient resources in some of the consortia s partner countries have resulted in the exclusion of selected projects from funding. With the increasing number of applications, the success rate - measured as share of eligible project proposals which were approved for funding - has fallen from 48% in 2008 to 20% in The time elapsed from submission deadline to evaluation, signing of grant and consortia agreements and activation of funding have gone down significantly, but is still too long and varies between partner countries, causing delays in starting projects. The High-Level Group has repeatedly discussed these issues, but so far failed to translate them into binding decisions and actions. The quality of the central evaluation of the projects, managed by the ESE, is good: however, better gender and geographical balance amongst technical experts and more competence in market knowledge are still needed. Moreover, national evaluations in parallel with the central evaluation still prevail in some countries. The virtual common pot works well and provides incentives to national governments to increase their budgets for Eurostars projects. The ESE s Eurostars database, including information of the applicants, projects, and their progress and impact, contains errors and deficiencies preventing thorough November 2014 Page 8 of 139

10 impact analysis. The expert group is convinced that there is an urgent need to improve the information content as well as the quality of information included. The findings and results of the final evaluation have given rise to 28 recommendations, summarised below. Concerning the target group and scope of the programme, all R&D-performing SMEs need to be targeted, whether or not they have been involved in international collaborations before. The ESE should support successful Eurostars projects by providing links to the EUREKA network and to public and private financing instruments supporting follow-up development and commercialisation. Regarding governance, the High-Level Group s decisions should be implemented, and this implementation should be monitored. National parallel application and evaluation needs to be abolished, and the transparency and feedback mechanisms of the evaluation improved. Concerning management and operations, time-spans from submission deadline to evaluation outcomes, signed grant and consortium agreements, and to activation of funding, must be synchronized and shortened to binding deadlines. The Eurostars database should be improved, based on re-designed application and reporting forms, in order to serve the purpose of assessing the impact of the programme on employment as well as R&D and innovation activities of the SMEs, and for information exchange with national project coordinators. Regarding funding, a commonly agreed baseline of harmonized funding rules needs to be agreed, providing the same type of partners with the same maximum rate of funding in each country, and allowing universities and public research organisations (PROs) to participate as partners instead of sub-contractors of SMEs. The results of this final evaluation should be used for the benefit of the Eurostars 2 Joint Programme, which has the potential to step up the delivery of results if increased national resources are provided. November 2014 Page 9 of 139

11 1 Introduction The European Parliament and the Council decided in on the Community s participation in a programme supporting research and development-performing small and medium-sized enterprises (R&D performing SMEs). The programme, designated the Eurostars Joint Programme (hereinafter Eurostars ), was designed to contribute to the sustainable competitiveness of Europe through stimulating innovation, growth, employment and economic change. The objective of the programme was to create an easily accessible European support mechanism to encourage SMEs to engage in R&D-based economic activities resulting in accelerated production of new close-to-market products, processes and services, as well as to promote their technological and business development and internationalisation. The Community agreed to co-finance Eurostars within a ceiling of EUR 100 million for the period , amounting to the equivalent of a maximum of one-third of the effective contributions of participating Member States and other participating countries. The EUREKA Secretariat (ESE) was invited to act as Eurostars Dedicated Implementation Structure, charged with organising calls and the central evaluation of proposals, channelling the Community funds to the projects and monitoring their progress. According to the Decision of the European Parliament and the Council, Eurostars was to be subject to an interim evaluation in 2010 and to a final evaluation at its end. The interim evaluation focused on the quality of the processes, efficiency of implementation and the ability to complement and harmonize national provisions of support to R&D-performing SMEs. In 2010, the programme had run for two years and realized four calls. Only 14 projects out of the 264 funded had been completed. Therefore it was too early to assess the impact of the programme, nor outcomes of individual projects, and thus the assessment was based on a qualitative analysis of data obtained mainly from interviews and funding decisions. In 2013 the Commission appointed an expert group to conduct the final evaluation of Eurostars. This final report, Evaluation of the Eurostars Joint Programme, sets the stage by describing policy context, scope and objectives, working methods and governance structure of the Programme. The evaluation focuses on the impact of Eurostars funding and overall progress in the implementation of the programme. The final evaluation covers the years , covering ten calls. The numbers of applications (3,548), applicants (11,733), and funded projects (783) are about three times higher than those at the time of the interim evaluation. The number of completed projects has increased from 14 to 300. In the final evaluation, a combination of qualitative and quantitative methods was used to assess the relevance, efficiency, impact and sustainability of the programme. These methods included desk research, face-to-face and phone interviews of individuals involved in the governance and administration of the programme, as well 1 Chapter 1, Introduction, deliberately omits references to the documents mentioned, as they are provided in the following chapters of the report. November 2014 Page 10 of 139

12 as funded and non-funded SMEs. The countries hosting the interviewees were selected from large, small, strong and emerging innovation countries, representing a good geographical balance. The impacts of the programme on employment growth, innovative outputs and generation of protected immaterial property participating SMEs were analysed comparing funded projects to non-funded ones where appropriate. The participating SMEs spending, employment, strategy, management and collaboration concerning specifically their R&D activities were assessed to shed light on the degree of additionality of the programme. In addition, the development of companies receiving funding from Eurostars was compared with that of companies whose applications were not accepted, and also with that of a random sample of similar companies which did not apply for Eurostars. The quantitative data sources consisted of a dedicated on-line survey targeted on all SMEs who applied for the programme, together with the information content of all application and reporting forms filled in by applicants and kept by the EUREKA Secretariat. The Commission engaged an external consultancy, Agilis S.A., to support the final evaluation by preparing data collected by the EUREKA Secretariat during the application process and during the monitoring and reporting of funded projects. The consultancy s assessments and recommendations concerning the quality and relevance of the data on the applications, applicants and projects, the information content of the application and reporting forms, as well as the appropriateness of the technical and market area classifications, are summarized in this final evaluation report. Finally, the programme-level effects, such as the contribution to the European Research Area and to the economic growth of Europe, are also addressed, taking into account the impact of the European financial and economic crisis on the programme. The findings of the final evaluation lead to 28 recommendations concerning the project and programme levels, governance and administration, as well as data collection and management. The Eurostars participating countries of EUREKA decided in 2012 to launch a continuation of Eurostars Eurostars 2 - which the European Parliament and the Council decided in 2014 to co-finance. Although Eurostars 2 is beyond the scope of this final evaluation of Eurostars, the recommendations can be used, where relevant, to support Eurostars 2 in supporting the sustainable growth of European industries. November 2014 Page 11 of 139

13 2 Historical and policy context 2.1 Background Eurostars aims to support R&D-performing 2 SMEs 3 by co-financing their marketoriented research projects and providing them with the necessary legal and organisational framework. The programme was set up in With the Decision 4 of 9 July 2008 of the European Parliament and of the Council, the European Community 5 agreed to participate in Eurostars by making a financial contribution of up to one-third of the effective contributions of the participating Member States and other EUREKA countries, up to a ceiling of EUR 100 million, for the period The overall value of their participation was estimated to be a minimum of EUR 300 million for the proposed period of six years. Box 1: Country participation in Eurostars 33 countries (Member States and other participating countries) have participated in Eurostars ( ). The participation has been as follows: a) 25 countries from February 2008: 20 Member States: Czech Republic, Denmark, Germany, Estonia, Ireland, Greece, Spain, France, Cyprus, Lithuania, Hungary, Netherlands, Austria, Poland, Portugal, Romania, Slovenia, Slovakia, Finland, Sweden. Five other participating countries: Iceland, Israel, Norway, Switzerland, Turkey. b) 31 countries from November 2008: As a) plus Belgium, Croatia, Italy, Latvia, Luxembourg and United Kingdom. c) 32 countries from February 2010: As b) plus Bulgaria. d) 33 countries from March 2011: As c) plus Malta. Source: ESE According to the Decision of 9 July 2008, the Commission is required to conduct both an interim evaluation and a final evaluation of Eurostars. The interim evaluation was conducted in 2010, two years after the start of the programme. This final evaluation was conducted at the end of Eurostars: the results will be presented to the European Parliament and the Council EUREKA applies a specific definition of R&D performing SMEs, see Box 4. The current definition of SMEs is set out in a Recommendation of the Commission adopted on 6 May 2003 (Recommendation 2003/361/EC (OJ L 124, , p.36)). The main factors determining whether a company is an SME are number of employees (<250) and either turnover (< 50m) or balance sheet total (< 43m). The Expert Group uses Small- and Medium-sized Enterprises (SMEs) as defined in Recommendation 2003/361/EC. The Recommendation can be found on: Decision No 743/2008/EC of the European Parliament and of the Council of 9 July 2008 on the Community s participation in a research and development programme undertaken by several Member States aimed at supporting research and development performing small and medium-sized enterprises. (OJ L 201, , p.58). The Decision can be found on: With the entry into force of the Lisbon Treaty on , the European Union (EU) succeeded the legal personality of the European Communities (EC). November 2014 Page 12 of 139

14 On 22 June 2012, the EUREKA Ministerial Conference endorsed the Budapest Document 6 which stated the interest of the EUREKA countries to support the follow-up to Eurostars. The Budapest Document sets out a common vision for Eurostars-2 and the basis for addressing the recommendations of the interim evaluation. It also invites the EU to participate in Eurostars 2. On 15 May 2014, the European Parliament and Council decided on the participation of the Union in Eurostars 2 for the period Objectives of the final evaluation and evaluation criteria In Article 13(3) of the Decision of 9 July 2008 (Decision No 743/2008/EC) it is stated: At the end of the Eurostars Joint Programme, the Commission shall conduct a final evaluation of the programme. The results of the final evaluation shall be presented to the European Parliament and the Council. Under the Eurostars Delegation Agreement between the European Commission and the EUREKA Secretariat, namely in Article 24 Follow up and Evaluation and Annex 1, the final evaluation shall address three main aspects: programme level, project level and governance. A set of questions on these different dimensions is defined 8, and expected results and indicators are also indicated: 1. Leveraging investments and co-funding of Eurostars by participating States. 2. Programme efficiency. 3. Progress towards the integration of national programmes in what concerns scientific, management and financial axis. The Terms of Reference (ToR) for the, included in Annex of this report, cover these dimensions and their effects on the quality and efficiency of the programme, specifying, through more than 40 questions to be addressed, the main evaluation criteria and final evaluation issues Eurostars 2: , Budapest Document, The Budapest Document can be found on: b dc&groupid=10137 Decision No 553/2014/EU of the European Parliament and of the Council of 15 May 2014 on the participation of the Union in a Research and Development Programme jointly undertaken by several Member States aimed at supporting research and development performing small and medium-sized enterprises. The Decision can be found on: Eurostars Joint Programme, Delegation Agreement 30-CE /00-14 (signed 19/6/2009), Annex 1, Pages 11 and 12. November 2014 Page 13 of 139

15 3 About Eurostars 3.1 History of Eurostars Eurostars was designed to stimulate R&D in SMEs, targeting SMEs already having research and development activities (designated as R&D performing SMEs) and which can play a critical role in innovation in Europe. Under the initiative of EUREKA, in May 2004, the 15 th EUREKA Inter-Parliamentary Conference (IPC) stressed the importance of stimulating R&D and innovation in SMEs and welcomed the efforts made by the EUREKA member countries and the Commission in setting up an Article 169 based initiative, respecting the EUREKA principles and methods, initially for highly innovative R&D-performing SMEs. IPC urged EUREKA member countries, the European Commission and the European Parliament to support this initiative financially and to address this initiative actively in their respective national parliaments. 9 Box 2: Article 185 of the Treaty on the Functioning of the European Union (TFEU) Article 185 TFEU [ex Article 169 of the Treaty establishing the European Community (TEC)] states: In implementing the multiannual Framework Programme, the Union may make provision, in agreement with the Member States concerned, for participation in research and development programmes undertaken by several Member States, including participation in the structures created for the execution of those programmes. Source: Having in mind EUREKA s founding principle of civilian purpose 10, in June 2004 the EUREKA Ministerial Conference noted that coherent frameworks to support R&D activities by SMEs were lacking in Europe, an issue that needed an urgent approach considering its importance as an engine for job creation and growth. The conclusions of the Ministerial Conference also emphasised the need to develop combined actions between EUREKA and the Framework Programme of the EU and to explore the setting up of financing mechanisms, including for SMEs 11. Several other key messages reinforced the importance of encouraging SME participation in R&D and innovation programmes, in parallel with the need to forge public policies contributing to this goal. Resolutions from the European Parliament also stressed the importance of reinforcing ties with EUREKA Adapted citation from Final RESOLUTION OF THE 15th EUREKA INTER-PARLIAMENTARY CONFERENCE, The Hague, 26 and 27 May 2004 Hubert Curien, a former President of the European Space Agency and French Research Minister when EUREKAEUREKA was created commented: In Europe, we did not wanted a military programme but a civilian initiative giving technology a kick-start. We wanted to promote a bottom-up approach. In 20 th Anniversary Report, Two decades of support for European Innovation, EUREKA Secretariat, September 2005 Ref. EUREKA Doc. MC , EUREKA Ministerial Conference, Paris, 18 June 2004 Examples of this reiterated message can be observed in the Council Conclusion from 24 September 2004 and the Resolution of 10 March 2005 Guidelines for future European Union Policy to support research (OJ C 320 E, ) November 2014 Page 14 of 139

16 In June 2005, the EUREKA High Level Representatives discussed a proposal aiming to create an integrated programme for R&D-performing SMEs, applying common evaluation criteria. Twenty-two EUREKA Member States expressed their approval of the Eurostars programme and undertook to commit financial and human resources to it. The Eurostars Programme Document 13 was endorsed by the Eurostars High Level Group (HLG) in Como in October 2006, and revised following the HLG meeting in Venice in April This document provided a comprehensive framework within which objectives, submission rules, evaluation criteria and procedures, funding and monitoring of projects were to be considered. The decision of the European Parliament and of the European Council of 9 July 2008 regulates EU participation in Eurostars and establishes its links to the Seventh Framework Programme (FP7), falling within the field of the Research for the benefit of SMEs, part of the Specific Programme Capacities. Box 3: What is EUREKA? EUREKA was established as a platform for market-driven industrial R&D. It is a decentralized network facilitating the coordination of national funding for innovation, and aims to boost the productivity and competitiveness of European industries. EUREKA comprises three pillars for funding R&D projects driven by the private sector: Individual projects, Clusters, and Eurostars. 15 EUREKA Umbrellas are thematic networks to facilitate the generation of future EUREKA projects. Founded in 1985, EUREKA now unites over 40 countries and also includes the European Union (represented by the European Commission). As stated in the Hannover Declaration, envisaging already a cutting edge trajectory, it was assumed that EUREKA will enable Europe to master and exploit the technologies that are important for its future and to build up its capability in crucial areas 16 and foresaw possible supportive measures from the European Union 17. EUREKA s structure is composed of the following main bodies 18 : The EUREKA Secretariat (ESE) is an international association based in Brussels acting as the central support unit for the network. An Executive Board (EB) is responsible for the management of ESE. ESE acts also as the implementing body of the Eurostars programme Ref. Programme for Eurostars, The Companies of Tomorrow, EUREKA Doc. HLG 2367, Maastricht, June 2005 Ref. Eurostars Programme Document, Endorsed by the HLG in Como, on 19 October 2006 and revised according HLG meeting in Venice on April Table 9-2 in the Annex provide quantitative information on the relative size of these pillars at the country level 20th Anniversary Report, Two decades of support for European Innovation, EUREKA Secretariat, September 2005 General Conditions, number 4, from Hannover Declaration, in: EUREKA Governing Bodies which are not relevant to this evaluation are not included as it is the case of National Information Points (NIPs). November 2014 Page 15 of 139

17 Box 3 cont.: What is EUREKA? The EUREKA Chair rotates yearly between EUREKA s member countries, with each Chairs mandate running from July to June the following year. The Chair implements a three-year rolling programme in cooperation with the previous and future chairs (the 'Troika'). The Troika countries form an Executive Group (EG) to which the EU representative is invited. It is responsible for reporting, and implements decisions taken by the HLG as well as for debating key policy issues. The MINISTERIAL CONFERENCE (MC) is the political body of EUREKA, gathering every two years the ministers from EUREKA countries and an EU Commissioner. The INTER-PARLIAMENTARY CONFERENCE (IPC) raises public awareness of EUREKA s role. It is organized in alternate years with the MC The HIGH-LEVEL GROUP (HLG) is the key decision-making body of EUREKA. The ministry responsible for EUREKA in each member country names its High-Level Representative (HLR), who in turn endorses new EUREKA projects, takes decisions on the management of EUREKA, and prepares EUREKA policy discussions for the MC. The EXECUTIVE GROUP (EG) reports and implements the decisions taken by the HLG. The EG is also responsible for debating key policy issues, deciding on topics delegated by the HLG, and advising successive Chairs. The NATIONAL PROJECT COORDINATORs (NPCs) run the National EUREKA Offices at operational level and are responsible for project generation, national and international support and follow-up. They are the direct contact with project participants, facilitating the setting-up and running of a project. The NATIONAL INFORMATION POINTs (NIPs) provide industry and research institutes with an interface with EUREKA and facilitate participation in projects. Source: (information adapted from EUREKA site, sections Structure and History) As stated in the Interim Evaluation of Eurostars 19, the programme reflects a commitment from the EUREKA network to conciliate this type of programme with national support programmes for SMEs. In fact, one of its main goals was to contribute to aligning and synchronising national Research & Development and Innovation programmes in line with European Research Area (ERA) expectations, removing barriers, and constructing a multipurpose integration -scientific, management and financial. Due to the consequences of the economic and financial crisis in Europe, which led to changes in resources allocation, it is foreseen that EUREKA will respond to the challenges, mainly the high demand by SMEs for innovation support, advancing the Eurostars concept in cooperation with the EU and other stakeholders in the innovation eco-system. More recently, during the EUREKA Ministerial Conference in Budapest in June 2012, it was emphasised that EUREKA s vision is that Eurostars will become the scheme of 19 The Interim Evaluation can be found on: evidence-base/other_fp7_panel_evaluations/eurostars_programme_interim_evaluation.pdf November 2014 Page 16 of 139

18 choice for its target group in Horizon 2020, directly contributing to generate high value goods, services and positive, wider economic impacts Scope and objectives of Eurostars The Eurostars concept was developed taking into consideration the importance of R&D performing SMEs for Europe and focusing on their potential contribution to strengthen the ERA by stimulating the generation of new products, services and processes. At the origin of the programme, back in 2005, it was recognised that this group of R&D performing SMEs is faced with strong global competition, and need to raise its knowledge and research intensity, expand its business activities into larger markets and internationalise its knowledge networks. To become a dynamic, knowledge-based economy Europe needs these firms to grow more and to become significant continental and global players. 21 Box 4: What is an R&D-performing SME according to Eurostars definition? 22 An R&D-performing SME is an SME in which at least 10% of staff, in terms of full time equivalents (FTE), are occupied with R&D, or 10% of annual turnover is dedicated to R&D. Source: Eurostars Programme Document. Endorsed by the HLG in Como on 19 October 2006 and revised according HLG meeting in Venice on April 2007 ( Como Document ) Thus the main objectives of Eurostars 23 are linked to the general purpose of contributing to European competitiveness through a bottom up approach aiming to support R&D performing SMEs by: 1. Creating an easily accessible and sustainable European R&D support mechanism. 2. Encouraging these SMEs to create new economic activities based on R&D results and bring new products, processes and services to the market faster than would otherwise be possible. 3. Promoting technological and business development and internationalisation. 3.3 How does Eurostars work? A multilateral approach Eurostars is based on a central evaluation process organized by the EUREKA Secretariat (ESE) in Brussels and decentralized funding by the Member States, the other participating countries and the EU. This arrangement derives from a multilateral XXVI EUREKA Ministerial Conference, Budapest, 22 June 2012, Room Paper, page 2. EUREKA Doc. 2367, Programme for Eurostars. The companies of Tomorrow, Maastricht, June 2005 For Eurostar-2 this definition is modified and then states SMEs with a headcount of 100 FTE employees or fewer must have at least 5 FTEs dedicated to R&D activities or dedicate at least 10% of their FTE to R&D activities or dedicate at least 10 % of their turnover to R&D activities. SMEs with a headcount of more than 100 FTE employees but less than 250 FTE have at least 10 FTEs dedicated to R&D activities or dedicate at least 10% of their FTE to R&D activities or dedicate at least 10% of their turnover to R&D activities. Source: Eurostars for SME, Eureka Secretariat, Brussels, May 2014; EUREKA Doc , Venice, April 2007 November 2014 Page 17 of 139

19 approach to the strategic goal of achieving R&D cooperation while at the same time respecting the different participating countries national procedures and budgetary policies. Eurostars is implemented by ESE acting as the Dedicated Implementation Structure (DIS). ESE is mainly responsible for the organisation of the calls for proposals, verification of the eligibility, peer-review evaluation and selection of projects for funding, monitoring projects, and allocation of the EU contribution. The national funding bodies finance the national participants, which are selected centrally and also channel the EU s financial contribution to SMEs. The governance system of Eurostars involves four main bodies: the EUREKA High Level Group (EUREKA HLG); the Eurostars High Level Group (Eurostars HLG); the Eurostars Advisory Group (EAG) and the ESE. The Member States and the European Commission take part in governance through their representatives in the HLG and NPC bodies, not only supervising the functioning of Eurostars, but also ensuring its broader promotion, project monitoring and funding Selection and funding mechanisms of Eurostars projects Project proposals are submitted to ESE by applicants following centralised, joint calls for proposals, with 1 or 2 cut-off dates per year 24. Proposals are evaluated and selected centrally on the basis of common eligibility and evaluation criteria. ESE is responsible for organising and coordinating the evaluation and selection processes. The central evaluation process The evaluation process follows a bottom up approach through which applications are submitted and then evaluated centrally using a three-step procedure: 1. Eligibility Check: Carried out by ESE, this aims to verify whether the eligibility criteria are met by the proposals. 2. Technical Assessment: A technical assessment by two independent technical experts per application is conducted by a review of both technical and market aspects of the proposal. More specifically, the assessment covers the review of the technological innovation and economic impact, the consortium, and the management of the project. The technical experts rate the proposals using qualitative indicators (poor, average, good or excellent). As a result of this assessment, a Technical Expert Report of each project proposal is delivered by the technical experts to the Independent Evaluation Panel (see below). The three main areas of technical evaluation are the following: Basic assessment: the consortium, its participants capabilities, the project plan, and financial aspects. Technology and innovation: the R&D activities to be conducted in the project, the degree of innovation, and the technological profile. 24 Eurostars is open for funding applications on a continuous basis, but ESE sets deadlines for submitting applications, called cut-offs in EUREKA terminology. The central evaluation process starts after each cut-off. November 2014 Page 18 of 139

20 Market and competitiveness: the market opportunities (size, geography, potential, time and risk), return on investment, and strategic importance of the project. Each of the main areas of evaluation has equal importance. Technical experts produce detailed comments and explanations, also in cases of low scores, to facilitate accurate feedback to the applicants and to support the ranking conducted by the Independent Evaluation Panel (IEP). The technical experts are selected by ESE. Their role is crucial to guarantee the credibility and quality of Eurostars since, as stated in the Eurostars Application Assessment Guidelines, their opinions contribute to direct millions of Euros of taxpayer s money to R&D-performing SME for market-oriented research and development Independent Evaluation Panel (IEP): On the basis of the Technical Expert Reports, IEP recommends to the Eurostars HLG whether or not to select an application for funding. The retained applications are ranked at an IEP meeting dedicated to assess and overall review all eligible applications 26. The resulting IEP report is composed of the Evaluation Consensus Report, the ranking list, and the minutes of the IEP meeting. The final ranking list is submitted to the Eurostars HLG for approval or rejection in its entirety. Individual projects cannot be rejected by the HLG, except in specific cases of applicants financial problems. The members of the IEP are nominated by Eurostars HLRs to constitute a pool of experts. Based on the recommendation of ESE with the support of the Eurostars Advisory Group, the HLG approves a list of IEP members for each IEP session. Budget allocation and funding at national level As the projects retained for funding have been approved centrally by the HLG, the allocation of funding from the EU contribution and from the national budgets earmarked for Eurostars projects is binding. Nevertheless, in spite of being placed in the ranking list, a project may not have a guarantee of funding. Each country participating in Eurostars commits an earmarked budget to the programme for each cut-off date. Of the overall public budget of each project, 25% is contributed by the EU. The maximum contribution of the EU for the duration of the Eurostars programme was set at EUR 100 million and to a maximum of one-third of the effective contributions of the participating Member States Eurostars Application Assessment Guidelines, Version 2.0, EUREKA Secretariat, March In line with the Guidelines for Eurostars Independent Evaluation Panel Members, recently (10th cut-off, June 2013), due to a large number of applications, it was decided to follow a specific method fast tracking of poor quality projects in order to concentrate on high quality projects. Methodology summary states where there is no large discrepancy between the 2 experts and 2 of the major criteria receive an average score of less than 3 stars then (1) Feedback to applicants will be prepared based on the expertise results. (2) A nominated IEP may confirm this result and the feedback to applicants. (3) The applications will be considered as not sufficient quality and declared by the IEP Panel as below the threshold potentially without scoring from the IEP members. (4) Applications may be reintegrated into the bilateral discussion process should the IEP consider them to be of sufficient quality. Decision No. 743/2008/EC of the European Parliament and of the Council of 9 July November 2014 Page 19 of 139

21 On an annual basis, but preferably within a multi-annual framework, participating countries establish national earmarked budgets that will be used to fund participation from the respective countries. This funding mechanism is referred to as the virtual common pot. Eurostars participating countries allocate funds according to their rules and funding rates. However, it was stated that For each selected Eurostars project, the R & D performing SMEs should collectively contribute the larger part of the overall costs relating to the R & D activities of all participants. 28 As decided by the European Parliament and the Council, the funding rules for individual Eurostars participants were determined at national level: The financial contribution to the participants in these projects is calculated according to the funding rules of the participating national programmes 29. Recommendations and suggestions are transmitted to National Funding Bodies (NFBs), based on best practice, stimulating their adaptation and emulation. In terms of the funding procedure, participant countries are expected to use a dedicated scheme for Eurostars, fully synchronised with the Eurostars process. Several methods are used to fund Eurostars projects, such as grants, loans, structural or regional funds or pre-accession funds. Differences also exist between Member States concerning budget allocation. Some participating countries run out of funding before others. Nevertheless, the projects can be initiated only if funding commitments from all participating countries have been confirmed or alternatively if participants declare their willingness to autonomously fund the project (private funding). The EU financial contribution to the selected Eurostars projects is disbursed by ESE to the NFBs 30 which have been designated by the participating EU Member States and by the other participating countries, on the basis of bilateral agreements between each NFB and ESE. While the funded projects are running, ESE is responsible for the implementation of common operational procedures to manage the full project cycle and for monitoring the projects. Project follow up, including visits and audits, is executed at national level Facts and figures Overall, Eurostars received 3,548 applications comprising 11,733 applicants 31 of which 8,461 were R&D-performing SMEs or other SMEs. Hence, the overwhelming majority (72%) of applicants belong to the group of SMEs. The total budget demanded by the applicants was EUR 4,955 million which by far exceeded the available budget. This also holds if restricted to the 3,018 eligible applications. Even more, Eurostars attracted over the first 10 cut-offs an increasing number of proposals and an increasing number of companies seeking public support for their near-market innovation projects Decision No. 743/2008/EC of the European Parliament and of the Council of 9 July 2008, Annex I. Decision No. 743/2008/EC of the European Parliament and of the Council of 9 July 2008, Annex I. Decision No. 743/2008/EC of the European Parliament and of the Council of 9 July This is the number of applications contained in the ESE data. However, a closer examination of the data (see chapter 4) showed that several records contain incomplete and erroneous data so that these numbers overestimated the true number of applications and applicants. But, for the sake of consistency with publications from ESE these numbers are used here. November 2014 Page 20 of 139

22 Figure 3-1 and Table 3-1 show that the number of applications increased from 532 in to 659 in 2010 and 717 in In when there was only one cut-off - ESE received 594 applications. 33 In 2008, 1318 SMEs applied, 1545 in 2010, and 1734 in The project partners of SMEs comprise other SMEs, universities, PROs, and large firms. As shown in Table 3-1, the total number of applicants amounts to 1777 (2008), 2165 (2010), and 2409 (2012). 34 On the other hand, the number of project approved for funding and the number of SMEs involved in funded projects did not rise accordingly. Overall, 783 projects were approved for funding. The number of funded projects and applicants per cut-off slightly decreased between cut-off 1 and cut-off 9. Only for the final cut-off in 2013 are similar numbers as for cut-off 1 observed. This implies that the probability of receiving funding has gradually declined. The success rate for Eurostars, calculated as number of applications approved for funding divided by the total number of applications submitted, amounts to 22% for all ten cut-offs. But as shown in Table 3-1, the success rate declined rapidly from 42% for cut-off 1 to 17% for cut-off So, for the later years the success rate for Eurostars comes quite close to success rates in FP7 projects, which are on average 20%. If we focus on selected thematic priorities which correspond to Eurostars, an interim evaluation of SMEs in FP7 uncovered varying success rates between 16% for ICT and 25% for Health An application is complete if all partners provided the proposal and also all supporting documents in time. As the provision of supporting documents is done decentralised by each partner there is always a number of applications where not all documents needed are provided by all partners. As shown in Table Table 3-1 about less than 10% of the applications are incomplete. In addition, several of these applications might also comprise applications where the applicant finally decided not to apply. EUREKA provides guidelines for applicants to ease the application process. The latest version of these guidelines are available under Since 2010 there were two cut-offs per calendar year. Deviating from the cut-off date ESE map cut-off 1 to 2008, cut-off 2 to 2009, cut-off 3 and 4 to 2010, cut-off 5 and 6 to 2011, cut-off 7 and 8 to 2012, cut-off 9 and 10 to 2013 as the budget for participating countries is on a yearly basis. These numbers include double counting of firms, universities, PROs, etc. However, identification of multiply entries is not easy as ESE has not developed a unique identifier for organizations. There are different reasons for multiple entries: (1) Applications which were not successful can again be submitted in a later funding round. We identified 340 applications which were submitted twice, 34 applications which were submitted three times, 6 applications were submitted four times, and one application was submitted five times. Hence, out of 3534 applications we found 3104 unique applications with different acronyms. (2) Via the online survey we uncovered another source of multiple entries. The same applications with the same participants were resubmitted as project application under a different name. Finally, there were SMEs which also applied for funding for different projects (with a different team). The maximum number of applications by an SME amounted to 17 applications. So, resubmission of unsuccessful former proposals and parallel applications for different projects was significant. On average about 10% of applications are resubmissions. At the level of applicants the calculation of unique applicants is straight forward. The number of applicants given in Table 3-1 amounts to After removing of invalid records we identified 5840 unique SMEs, 1360 Universities and PROs, and 632 large companies or government agencies. If related the success rate to eligible project only, hence no. of approved to no. of eligible applications, the figure for cut-off 1 is 48% and 20% for cut-off10 and 26% for all ten cut-offs. Koos van Elk, Jacqueline Snijders, Yvonne Prince, Petra Gibcus, Sophie Doove, Paul Simmonds, Katharina Warta, Barbara Good, Sascha Ruhland, Sonja Sheikh (2014) Performance of SMEs within FP7. An interim evaluation of FP7 components. Volume II. Annexes to Main Report. The report can be found here: November 2014 Page 21 of 139

23 The success rate of cut-off 1 of Eurostars is clearly higher compared with the success rate of the first cut-off of the new SME instrument under H Figure 3-1 also shows that from the third cut-off onward, a semi-annual rhythm was established. The large number of proposals for the tenth cut-off might also mirror the fact that this cut-off was the last funding round in phase one of Eurostars and the successor programme Eurostars 2 could only take-off in spring Table 3-1, Table 3-2, and Table 3-3 provide key indicators for the first ten cut-offs of Eurostars. 38 Only the main features of Eurostars are summarized here. More detailed analyses can be found in Chapter 6.1. Figure 3-1: Total number of applications by status and cut-off Source: ESE (2013) Figure 3-2: Total number of applicants and funded applicants by cut-off Source: ESE (2013) See the press release of the Commission on the results of the first cut-off of the new SME instrument: Key figures per budget year are given by Table 9-1. November 2014 Page 22 of 139

24 The following points highlight the key characteristics of the programme: On average, 50% of eligible project costs are covered by public funds, consisting of national and EU funding. The EU contribution amounts to about 20% of national funds. The numbers of proposals approved for funding stayed more or less the same for all cut-offs. Hence, the success rate for proposals steadily declined as the number of proposals increased. The success rate, defined as the relation between the number of approved projects and the number of submitted projects, declined from 42% to 17%. If the success rate is measured as the relation between the numbers of approved applications and eligible applications, it amounted to 42% for cut-off 2 to 20% for cut-off 2. The declining success rate corresponded to the fact that the annual cost of the programme for the public budgets remained at the same level. However, with two cut-offs per year since 2010, the annual cost of the programme for the public budgets increased considerably. A Eurostars project typically comprises of 3-4 partners (3.3 is the average consortium size) including an R&D-intensive SME as lead applicant and other SMEs, as well as large firms, universities or PROs in about one third of the projects. The total number of applicants as well as the applicants which were SMEs in Table 3-1 overestimates the total numbers as this figure includes multiple applications of the same unit both as consortium lead and as member of consortia - and unsuccessful projects in previous rounds which were resubmitted (see footnote 34 for details). About 10% of all submitted application forms were incomplete, because the consortium finally decided not to apply, or because at least one of the partners did not manage to deliver the required supporting documents in time. The evaluation reports of the technical experts were mostly submitted in time to the IEP. For about 10% of the projects, the judgments of the technical experts differed to an extent that rendered the work of the IEP difficult. Not all projects which were approved by the HLG received funding, due to the withdrawal of one or more participants because of e.g. failure to pass the financial check performed at the national level. The top ranked 20 projects normally received funding. However, part of them could not be funded, because of the too low earmarked budget of one or more NFBs. Especially for some countries the number of applicants from the country was much larger than for other countries. Hence for those countries the number of not-funded projects was much larger and even excellent projects which were placed high in the ranking list could not receive funding. More than half of the funded projects are still running. Some projects of even the second cut-off (application date in 2009) were not completed by December One reason for this is that the time-to-contract is still significant in many countries, delaying the initiation of the projects. November 2014 Page 23 of 139

25 The median time-to-contract was significantly reduced between the first four cut-offs and later cut-offs. However, even then a small number of contracts exhibit a fairly long time before the final signature of at least one country was made. The total cost of projects approved for funding was less than one-fourth of total project cost of all applications. Total estimated EU funding amounted to EUR 100 million and was less than one -third of the estimated funding by participating countries, which accounted to EUR 372 million. This fulfilled the EU funding requirement that the EU funding should be less than one-third of total public funding for Eurostars. Similarly, funding released by participating countries (EUR 226 million) and by the EU (EUR 54 million) shows the same picture. November 2014 Page 24 of 139

26 Table 3-1: Key figures for the Eurostars programme 1 (2008) 2 (2008) 3 (2009) 4 (2010) Cut-off / Cut-off year 5 (2010) 6 (2011) 7 (2011) 8 (2012) 9 (2012) 10 (2013) TOTAL / AVERAGE Submission Number of applications submitted (projects) Number of applicants Number of R&D performing SMEs and SMEs % of R&D performing SMEs and SMEs 75% 74% 70% 71% 71% 72% 71% 73% 71% 72% 72% Total budget of submitted applications (M ) Number of complete applications in % of all applications submitted 89% 90% 91% 91% 95% 90% 86% 92% 93% 94% 91% Number of eligible applications in % of all applications submitted 88% 77% 85% 85% 90% 87% 80% 87% 86% 86% 85% Centralised evaluation Number of expertises by technical experts submitted to IEP Number of projects with differing expertises* Number of applications above the threshold % applications above the threshold vs eligible 70% 45% 47% 38% 36% 38% 39% 36% 35% 36% 40% Total budget of projects above threshold (M ) Approved projects Number of applications approved % application approved vs above the threshold 68% 81% 76% 63% 65% 56% 62% 61% 63% 55% 64% Number of participants Number of R&D performing SMEs and SMEs % of R&D performing SMEs and SMEs 74% 72% 69% 67% 68% 68% 65% 68% 73% 71% 70% Total budget of approved projects (M ) Success rate (approved/submitted applications) 42% 28% 30% 20% 21% 18% 19% 19% 19% 17% 22% Ranking of projects and funding allocation Share of 20 best evaluated projects approved 100% 100% 100% 90% 95% 100% 100% 100% 100% 100% 99% Share of 40 best evaluated projects approved 98% 100% 100% 93% 88% 90% 98% 88% 95% 90% 94% Share of 80 best evaluated projects approved 80% 93% 88% 70% 75% 74% 81% 71% 70% 73% 77% Annotation: * 2 or more differences in two or three criteria Source: ESE (2013) November 2014 Page 25 of 139

27 Table 3-2: Status of Eurostars projects by December (2008) 2 (2008) 3 (2009) 4 (2010) Cut-off / Cut-off Year 5 (2010) 6 (2011) 7 (2011) 8 (2012) 9 (2012) 10 (2013) TOTAL / AVERAGE Project Status Number of projects approved Number of projects not finished Number of projects completed Number of projects "on hold" or number of not started projects Number of project withdrawn by participants % completed and not completed projects divided by all projects approved 96% 88% 96% 94% 89% 93% 87% 84% 78% 0% 78% Final signed Consortium Agreements Number of CAs received by ESE Time to contract (in months) Median - Project level Median - Participant level Source: ESE (2013) November 2014 Page 26 of 139

28 Table 3-3: Funding by National Funding Bodies and the EU Commission 1 (2008) 2 (2008) 3 (2009) 4 (2010) Cut-off / Cut-off Year 5 (2010) 6 (2011) 7 (2011) 8 (2012) 9 (2012) 10 (2013) TOTAL / AVERAGE Funding approved Number of projects approved Total costs of projects approved ( M ) Funding committed Total funding estimated (M ) in % of total project cost 46% 44% 44% 43% 38% 44% 34% 44% 44% 38% 42% Total funding reserved by NFBs (M ) in % of total funding estimated 100% 100% 100% 100% 100% 100% 100% 95% 63% 27% 87% Total funding released by NFBs (M ) in % of total funding estimated 72% 76% 72% 69% 57% 50% 39% 19% 18% 2% 48% EU contributions EU funding estimated in % of national funding estimated 21% 25% 25% 26% 25% 25% 26% 14% 13% 14% 21% EU funding reserved in % of national funding reserved by NFBs 21% 25% 25% 26% 25% 25% 26% 15% 14% 13% 22% EU funding released in % of national funding release by NFBs 24% 26% 24% 25% 26% 23% 20% 13% 13% 100% 24% Source: ESE (2013) Annotations: The funding of participants is monitored according to 3 key indicators: 1) Funding estimated is based on the funding lists approved by HLG. The total public funds are calculated according to the observed funding rates per country and per type of participant 2) Funding reserved is based on the "Declaration of Commitments" from NFBs, which reflects the public funds committed by NFBs to the participants through funding contracts. Data are incomplete for the last cut-offs. 3) Funding released is based on the "Declaration of Expenditures" from NFBs, which reflects the public funds actually paid throughout the project lifecycle to the participants. Data are incomplete for the last cut-offs. November 2014 Page 27 of 139

29 3.4 Results and recommendations of the Interim Evaluation The Interim Evaluation A Group of Independent Experts conducted, between June and November 2010, an interim evaluation of Eurostars. 39 Article 13.2 of the Eurostars decision of 9 July 2008 stipulated that the interim evaluation should cover progress toward objectives set out in Annex 1 of the Eurostars decision, include recommendations on the most appropriate ways to further enhance scientific, management and financial integration, assess the ability of R&D performing SMEs to access Eurostars, and assess the quality and efficiency of its implementation. More specifically, the Group of Independent Experts of the interim evaluation were requested to address the following three issues: (1) whether the Eurostars Programme has been appropriately set up and whether the current structure, processes and procedures are appropriate; (2) whether Eurostars complements the current provision of R&D support for SMEs in Europe and whether it is able to complement to, and contribute to the harmonisation of, national-level provision of such support; (3) whether the programme has delivered a firm-level impact or is able to do so. The interim evaluation was carried out using qualitative evaluation methods, and mainly used archival data, survey and operational data, and qualitative interview data as gathered by members of the Group of Independent Experts. The final report of the interim evaluation was published in December It contains 24 recommendations (see Box 5) EC Report on the Interim Evaluation In its report to the European Parliament and Council on the interim evaluation of the Eurostars Programme, 40 the Commission stated that Eurostars had demonstrated potential as a programme of interest to R&D-performing SMEs, helping them to innovate. The Commission s report highlighted the need for improving Eurostars operational performance and reinforcing its integration with the national participating programmes. More specifically, the Commission stated that the majority of participating SMEs had already been involved in international research, and questioned whether this constituted the intended target group; encouraged the EUREKA Secretariat to develop impact indicators and collect this data, without, however, imposing a too-high reporting burden on SMEs; The Interim Evaluation can be found on: evidence-base/other_fp7_panel_evaluations/eurostars_programme_interim_evaluation.pdf Report from the Commission to the European Parliament and the Council, Interim Evaluation of the Eurostars Joint Programme, Brussels, 8 April 2011, COM(2011) 186. The report can be found on: ation_eurostars.pdf November 2014 Page 28 of 139

30 Box 5: Recommendations from the Eurostars programme Interim Evaluation Policy Recommendations beyond The Eurostars Joint Programme should be continued beyond In the next edition of Eurostars Programme, European Commission should prepare a proposal to expand the budget, using flexibility clauses, to respond to the increasing demand. Policy Recommendations until European Commission and the Member States should provide sufficient funds to make it possible to fund a greater share of projects that meet the Eurostars threshold criteria. 4. EUREKA Secretariat and NPCs should seek to increase its reach among SMEs that do not possess preexisting international links. 5. EUREKA Secretariat, in close collaboration with NPCs, should increase the number of technical experts in general and obtain a good balance between participating countries in terms of number of experts relative to the number of participants from a given country. 6. EUREKA Secretariat should establish procedures to ensure that there is a sufficient number of technical experts with good market knowledge. 7. When increasing the number of technical experts, EUREKA Secretariat should pay close attention to maintaining a high level of expertise amongst these. 8. The Eurostars IEP should be allocated more time so as to be able to analyse problematic applications in greater detail and possibly refine some of its judgements. This is recommendable particularly as the number of applications increases. 9. When the Eurostars IEP overrules the technical experts scores, there should always be a comprehensive and transparent explanation of the reasons for this. This explanation should be fed back to technical experts. 10. EUREKA Secretariat should ensure that systematic feedback is always given to technical experts, regardless of whether or not application scores are changed by the Eurostars IEP. 11. The voluntary Working Group for good practice dissemination should be made a permanent fixture of Eurostars implementation, and all participating countries should be encouraged to participate in it. 12. To expedite time-to-contract, participating countries should start the national evaluation phase (e.g. financial evaluation) well before the communication of the evaluation rankings. 13. The EUREKA Secretariat should initiate discussions with those countries that have persistently slow processes (in terms of time-to-contract) with a view of bringing their performance closer to the median performance among all countries. 14. There should be a global deadline for the signature of Grant Agreement. The failure of any given project to meet this deadline should mean that the project will not be funded, and the unallocated funds will be allocated to the highest ranked unfunded project in the ranking list. 15. The rules for providing feedback to project applicants should be further harmonised across countries, with an emphasis on high-quality feedback and with strict deadlines imposed for providing the feedback. 16. Harmonisation of national procedures should be strengthened further by EUREKA Secretariat and European Commission, and specific, measurable milestones should be agreed by HLG to facilitate this process. 17. Eurostars eligibility criteria should be universally established as the necessary and sufficient condition for any participating project to be funded in any participating country. No participating country should be allowed to impose eligibility conditions that go beyond the Eurostars eligibility criteria. 18. Dual reporting arising due to the use of different languages should be avoided. 19. The EUREKA HLG should assume an active role in monitoring the time-to-contract and in ensuring that countries that lag behind streamline their processes with Eurostars so as to avoid project delays. 20. The remaining bilateral agreements should be signed as a matter of priority. 21. Eurostars Joint Programme should continue to be organised around a Virtual Common Pot principle. 22. However, it is also recommended that Eurostars Joint Programme experiments with the approach of allocating 10% of total earmarked funds to a Real Common Pot, which would be used to finance highly ranked projects that risk not getting funded because of country-level complications. 23. EUREKA Secretariat should initiate an activity intended to explore the integration of EUREKA cluster initiatives in view of learning and implementing integration lessons that could be applied to promote the integration of Eurostars with national R&D support initiatives. 24. The post-hoc evaluation of the Eurostars programme, to be carried out in 2013, should emphasise quantitative output metrics to measure firm- and programme-level outcomes (the latter both Nationally and Centrally). The evaluation should use appropriate statistical methods to control self-selection bias and examine input-, output- and behavioural additionality effects at the firm level. November 2014 Page 29 of 139

31 agreed with the recommendation that the European Commission and the Member States provide sufficient funds to make it possible to fund a greater share of projects that meet the Eurostars evaluation threshold as far as it would allow the financing of all the best projects, in line with the principle of the virtual common pot; agreed with the recommendation to test a partial "real common pot" approach (to be accomplished by 2013) and invited the EUREKA Secretariat to propose other relevant measures towards the same end; encouraged the EUREKA Secretariat to undertake the necessary steps to improve the quality of the central evaluation, starting from the recruiting and selection of experts, and ending with the provision of feedback to technical experts and applicants Response of the EUREKA Secretariat to the recommendations of the Interim Evaluation In the 2012 Eurostars Annual Report, the ESE replied to the recommendations made in the interim evaluation report. In its reply, the ESE mentioned that efforts were being made to: ensure that the EU and the participating countries provide sufficient funds to make it possible to fund a greater share of projects that meet the Eurostars threshold criteria; improve the quality and number of technical experts; ensure that the Eurostars IEP is allocated more time to analyse problematic applications; provide comprehensive and transparent explanation for the cases when Eurostars IEP overruled the technical experts scores; provide systematic feedback to all technical experts; establish working groups for good practice dissemination; expedite time-to-contract (with a target to improve financial and administrative synchronization as well as to reduce or remove existing national legal and administrative barriers) include a global deadline for the signature of grant agreement in Eurostars-2; provide high-quality feedback to project applicants; strengthen the harmonization of national procedures; establish common project eligibility criteria; continue organizing Eurostars around the Virtual Common Pot principle. The recommendation about seeking to increase the reach among SMEs that do not possess pre-existing international links was not responded to directly by ESE. However, reference was made to two surveys launched in 2010 and 2011 in order to better identify pre-existing international links of SMEs. November 2014 Page 30 of 139

32 All Eurostars 1 Bilateral Agreements have been signed, as recommended in the interim evaluation report. Signing a Bilateral Agreement has been set as a pre-condition for participation in Eurostars 2. The recommendation to allocate 10% of total earmarked funds to a Real Common Pot was not implemented. However, in its reply, the ESE referred to the Budapest Document where it was agreed that the budget for each Eurostars participating country should reflect its potential funding weight. This revealed the importance of considering a balanced funding and the agreed targets for project funding (i.e. the top 50 ranked projects and from 50 to 75% of the applications ranked above the quality threshold) based on the experience observed in Eurostars 1. Finally, ESE mentioned its participation in the Inter-Cluster Meetings as a response to the recommendation to initiate an activity to explore the integration of EUREKA cluster initiatives. November 2014 Page 31 of 139

33 4 The Agilis Report Eurostars Data Collection and Assessment 4.1 Objectives In order to provide the expert group of the final evaluation of Eurostars with the best possible quantitative basis for their work, the Commission launched in October 2013 a call for tenders designated Eurostars Data collection and assessment. The main tasks of the contractor were to: (1) collect, systemize and present all data on Eurostars applications and funded projects covering the period , (2) produce statistics on the basis of the data, and (3) assess the quality of the data contained in all Eurostars project documentation forms, especially what regards the reliability and usefulness of the information therein for assessment and documentation of the impact of Eurostars funding. In addition, the contractor was to assess whether the classification used by the ESE concerning the Technological areas and Market areas conforms to the Standard Classification of Economic Activities in the European Community (NACE), and to suggest amendments if needed. The Commission selected the offer which was presented by Agilis S.A. 41 (hereinafter Agilis ), who submitted their final report 42 (hereinafter referred to as Agilis Report ), together with a technical report, in April Findings Data quality and data cleaning ESE delivered to Agilis the data from application forms, final reports and market impact reports submitted under Eurostars, as well as the guidelines provided to applicants and participants for filling out these forms. The applicant forms contain basic data about the main applicant and the member of the consortium such as sales, number of employees, and R&D activity, plus, data on total cost of the project, the technological field, and the expected start and end date. The final report form collects data on the achievement of the project, IP gained by the project, investment induced or additional employment generated. The market impact form asked for similar data but was issued about one year after the delivery of the final report form. In addition, data on the date of contract signature, time-to-contract, and the technical evaluations scores were provided. Agilis assessed the validity of all records contained in the project documentation forms, the validity and correctness of individual data entries, the logic of the forms and the way they asked for information, as well as the coherence of the corresponding Guidelines Agilis S.A. is a consulting company which is specialist in data preparation and data analyses (see For details see the final report from Agilis S.A. D3. Final Report for the contract Eurostars Data Collection and Assessment, Contract No. SC-RTD/DIRC/C$/S /Eurostars. November 2014 Page 32 of 139

34 The data in the records were found to contain many errors of different types. Mistakes include for instance erroneous use of dot and comma as decimal digit separator and indicator of thousands separator, % figures instead of absolute values, amounts declared in Euros instead of million Euros, and national currencies used instead of the Euro. Typos resulted in severe misunderstandings, such as negative turnover, fractional headcounts and confusion of R&D expenditure with R&D employment. Several double-entries and incomplete data entries by applicants were detected. Due to the removal of the invalid data records, the numbers of applications and applicants in the Agilis Report are smaller than those published by ESE, impacting the building of reliable statistics, such as on the share of successful applications. The mistakes and distortion of financial data retrieved from the forms are due to lack of logical checks of field content and systematic outlier detection. The invalid entries prevent meaningful analyses, development of reliable statistics and assessment of the impact of the Eurostars programme. Agilis embarked on corrections of the erroneous data. This, however, was possible only in cases where the correct figures were obvious from the internal logic of the data. Wrong entries could not be corrected when this would have implied Internet searching, company home page or company data provider (e.g. the AMADEUS database of Bureau van Dijk). Therefore the data provided by Agilis to the expert group still contain errors, especially in the case of financial variables. Agilis concluded that even after the data-cleaning efforts, the deficiencies in the quality and relevance of the data retrieved from the Eurostars forms, especially with regard to market impact and other outcomes of the projects, significantly reduces the usefulness of the collected information and limits its use for evaluation purposes and impact assessment Assessment of the project documentation forms Agilis assessed the project documentation forms together with the respective Guidelines from the point of view of the logic of the construction of the forms questions and the way in which information was requested. Agilis concluded that the design of the forms and the number of questions had led to a degree of misunderstanding and confusion, causing errors to be made by the applicants Assessment of the classification of technology area and market area ESE uses two broad classifications to provide information on the projects and applications: Technology Classification to provide information of the R&D fields supported by Eurostars, and Market Classification of the market relevant to the products, processes and services developed within the projects. Both classifications have been developed by ESE itself, and are used for Eurostars as well as the other EUREKA programmes. November 2014 Page 33 of 139

35 Agilis pointed out that the classification of the projects technology and market areas should allow linking the Eurostars statistics to other data in order to enable, for instance, comparisons of Eurostars data with other business statistics. Agilis examined the ESE classifications and the degree to which they conform with the NACE-rev. 2 classification of the European Statistical System used by Eurostat and national statistical offices. The technological and market fields were mapped into the NACE rev. 2 nomenclature at the two-digit level (division level). In addition, a correspondence between market areas and the NACE rev.2 nomenclature was developed at the threedigit level (group level). Agilis concluded that ESE s classification needs deep revision, as it suffers from significant heterogeneity in what concerns the degree of detail between the widely different technologies and markets typical for a bottom-up programme such as Eurostars. The heterogeneity creates an imbalance between the groups of technologies and markets of the participating projects. Agilis highlighted that the ESE classification does not allow a comparison of the Eurostars statistics with other relevant statistics nor a comparison of Eurostars data with the data of other SMEsupporting programs. 4.3 Recommendations Agilis main recommendations regarding the Eurostars application form, project progress report, final report and market impact report are as follows: (1) The application form and the project progress, final and market impact reports need reconstruction. This should comprise revisions of specific questions and headings, as well as reordering, deletion and addition of a number of questions to improve the logic of the forms. Correspondingly, the guidelines for these forms should be revised. This will serve stakeholders needs and avoid imposing unnecessary workload on the Eurostars participants. (2) The forms should be filled in on-line, allowing for temporary saving. For this purpose, a web-page based form would be more appropriate than the current fillable PDF form. (3) Some open-ended questions could be replaced by drop-down menu questions. (4) The final submission of the form should not be allowed unless all fields are completed. (5) Logical controls in fields requiring arithmetic completion should be introduced. (6) Concerning terminology (such as result and output being used interchangeably), a uniform approach should be adopted in the documentation forms and the Guidelines. The main recommendations concerning the technology classification and the market classification are: (1) The current market-area classification should be replaced with the NACE rev. 2 nomenclature, as this will allow applicants to provide more accurate information about the market that the project addresses. The NACE nomenclature can also provide a common basis for the comparative evaluation November 2014 Page 34 of 139

36 of different EU funding schemes and of different economic sectors regarding their R&D efficiency. An accompanying short version of the NACE manual, possibly appearing in a pop-up electronic form, can assist the applicant in locating the correct market area of their application. (2) The current technological area classification should be restructured to eliminate inconsistencies, and the BBS Technology Keywords of EURESEARCH should be accommodated correctly. The new classification should also take into account the IPC nomenclature, a technological innovation classification system developed to classify patent applications. Taking IPC into account in the design of the technological classification system is also useful for statistical purposes, for example for monitoring whether Eurostars projects follow trends in patent activity. November 2014 Page 35 of 139

37 5 Methodologies and data sources 5.1 Introduction The expert group used qualitative and quantitative tools for the final evaluation of the implementation and impact of Eurostars, as stipulated in the Terms of Reference in Annex The main methods used by the Group were: (1) desk research; (2) interviews with stakeholders and SMEs; (3) online survey of SMEs which applied for Eurostars; (4) quantitative analyses. Chapter 5 briefly describes the methodological basis of the evaluation. The table below shows an overview of which data sources have been used in combination with different methods in this report. The subsequent sections in this chapter describe each approach in more detail. Table 5-1: Combination of data sources and methods used in this report Qualitative Methods Quantitative Methods Desk research Descriptive statistics Interviews Econometric analysis Academic literature, policy and legislative documents and papers X OECD and Eurostat country data X X Data hosted by ESE: minutes of NPC and HLG meetings X Data hosted by ESE: Eurostars applicant data and administrative data X X Bureau van Dijk s Amadeus database EPO Worldwide Patent Statistical Database (PATSTAT) Interviews with representatives of the Commission and EUREKA Interviews with representatives of national governing and administrative bodies Interviews with SMEs which applied for the programme Online Survey of Eurostars SME applicants X X X X X X X 5.2 Desk research and data sources Through desk research, the expert group gathered and analysed documents containing data and information relevant for the Eurostars evaluation criteria from the various stakeholder groups. Desk research allowed the identification of major issues, challenges and questions on the implementation of Eurostars. The documents analysed are listed below. November 2014 Page 36 of 139

38 Legislative and cross-cutting documents These documents include regulations, decisions and assessment/evaluation reports issued by the European Commission, as well as assessments of other R&D and Innovation programme targeted for SMEs. Minutes of EUREKA High-Level Group meetings Minutes from meetings of EUREKA High-Level Group Representatives were analysed, covering the period In general, several topics, which can be directly linked to evaluation criteria, emerged as permanent agenda items. Statistical data Data from OECD and Eurostat were screened in order to illustrate country-specific analyses, e.g. on gross domestic expenditure and employment, and specific thematic indicators linked to R&D and Innovation activities. Data hosted by ESE ESE provided the minutes of the NPC and HLG meetings, the EUREKA Annual Reviews and Reports, Eureka 25-year country overviews ( ), the Eurostars Annual Reports, and data on the Eurostars participants and projects covering the period The latter data sets were quality-controlled by Agilis, as described in Chapter 4. Other sources Other information publicly available on the Internet was also consulted. In the case of specific evaluation questions, such as the impact of the financial and economic crisis on the Eurostars target group, academic, policy and legislative papers were studied in order to obtain different key actors perspectives and positions. In the context of the financial and economic crisis, the expert group also analysed OECD, the IMF and national studies on the impact of EUREKA. 5.3 Interviews with programme administrators and SMEs The expert group conducted in Brussels interviews of two representatives of the Commission, three representatives of EUREKA and the top management of ESE. The Members individually interviewed 20 representatives of the national governing and administrative bodies of Eurostars, such as NPCs, NFBs, HLG and EAG members (see Annex 9.4). The interviews were structured according to a guideline composed by the expert group and adjusted according to the function of the interviewees in the Eurostars programme. All interviews were documented in writing 1. The main topics raised in the interviews of the governing and administrative bodies were: The role of Eurostars in EUREKA and in the wider landscape of European public support for private R&D. Governance of Eurostars including harmonisation and synchronisation of funding as well as the centralized evaluation. November 2014 Page 37 of 139

39 The impact of the financial and economic crisis on Eurostars. The impact of public funding (by the EU and the national level) on their project. The impact of Eurostars on supported SMEs. The perspectives of Eurostars and expected changes of the programme. The expert group selected a set of countries which together would form a good geographical balance, including small and large countries, as well as old and new EUREKA member countries. 26 SMEs, operating in 12 different countries, were randomly picked from amongst these pre-selected countries (see Annex 9.5). The expert group members individually conducted face-to-face or telephone interviews with the CEOs of the SMEs. These interviews were also carried out according to a guideline developed by the expert group for this purpose, and were documented as above. The interviews of SMEs covered the following topics: Awareness and information about Eurostars (source of information, previous experience with EU and national R&D funding). Process of access to Eurostars (finding partners, support used for development of the application). Application phase (funding rules, eligibility criteria, and eventual delay in waiting for the funds). Evaluation phase (quality of technical evaluation, extent and quality of feedback). Output and impact of the project (network, market access, innovation, IPR, employment, growth, profitability). Alternatives to Eurostars funding. Suggestions for developing the Programme. Recommendations on funding rules and level, other issues. The results of the interviews of the programme administrators and the SMEs were triangulated in the light of the evaluation questions drawn from the Terms of Reference (see Annex 9.3.2). The triangulation was laid down in a working document, which provided the basis for the discussions of the expert group and the interpretation of the results. 5.4 Information based on data from ESE ESE supported the work of the expert group by providing information on all applications as well as applicants. In addition, ESE provided data from the final reports and the market impact reports for all projects where these reports are available. This data were checked and prepared for the analysis by Agilis (see chapter 4). The expert group uncovered some more invalid records in the application data and deleted them from the analysis. The total number of valid application records then amounted to 11,714 applicants (instead of 11,733 in the original ESE data) and 3,525 applications (instead of 3,548). November 2014 Page 38 of 139

40 ESE also provided data on the status of projects in December 2013 (e.g. whether and when the funding contracts between National Funding Bodies and national participants were signed. Finally, ESE made available full address details, including a phone number and an address, for all applicants. These data were entered into a database. This database was used to draw the random sample of funded and not-funded SMEs, as well as SMEs in running and completed projects which were used for SME interviews. It also formed the basis for the selection of SMEs for the online survey. This database included all corrections performed by Agilis. The final evaluation report refers to this database as ESE evaluation data (2014) in chapters 6 and 7. The Agilis report also showed that a significant number of projects were not finalized (see Figure 5-1). In addition, Agilis uncovered that the ESE data did not provide a unique starting and ending date for the projects as well as for the subprojects ( projects module ) of each participant. For this reason, it is hardly possible to see which of the projects were completed and should have provided a final report and a market impact report, which is due about one year after the project end. Finalized projects mainly belong to the early cut-offs. Overall, 922 participants out of nearly 2500 in approved projects delivered a final report. A market impact report is available from 377 participants. Figure 5-1: Status of approved projects by cut-off by December 2013 Source: ESE evaluation data (2014) In addition, a significant share of companies only partly responded to the final report form and the market impact form, so that the impact of Eurostars is insufficiently covered by the available data. One reason is that at the point in time when final and market impact report are due, the project results are not yet available. Given that the question for collecting information, e.g. on project inputs like R&D, was changed over time, no comparison is possible for, e.g. the R&D expenditure of firms. Similarly, such changes in the concrete formulation of the respective question also occurred for employment figures and sales figures, so that the growth impact cannot be analysed. November 2014 Page 39 of 139

41 5.5 Online survey As part of the evaluation, an online survey was conducted to ask participants about their experience with Eurostars. The target group of the online survey was restricted to R&D-performing SMEs and other SMEs for two reasons. First, both are the prime group of interest for public support and hence it was decided to focus on this group. Second, addressing other participants such as universities and public research organizations or large firms, would need separate questionnaires for those groups. Hence for the sake of simplicity, those groups were not included in the survey. Two different types of questionnaires were prepared. The first targeted funded projects, and the second addressed not-funded projects. Both identified separately firms with completed projects and firms with projects still underway. In order to compare data between funded and not-funded firms, the questionnaire addressed as similar topics as possible for both groups. The main topics of the questionnaire were: Characteristics of the project (e.g. start and end of the project). Characteristics of the applicant (e.g. age of the company, experience in international markets, total number of applications to Eurostars). The consortium (e.g. what determined the choice of partners). The impact of the project (e.g. human capital of employees hired in the context of the project). Input, output, and behavioural additionality. The importance of the Eurostars application in relation to the project portfolio of the SME. Endorsed by a letter from Peter Dröll, Acting Director for Directorate В - Innovation Union and European Research Area, from DG Research & Innovation, the online survey was operational from May 9 th till July 10 th The contact for the individual Eurostars project was selected as addressee. The addressees received a personalized . Several variants of s were used for completed, running and not-funded projects. In addition, four reminder s were sent to those applicants who had not responded. The initial as well as all reminder e- mails provided some information on the project, such as the name of the project and its cut-off and cut-off date of the project. When a project applied several times, only the last application date was selected. The ESE application data contained 8,440 valid project modules by SMEs. A project module refers to the number of applications by SMEs, no matter how many times an application was resubmitted. Out of these 8,440 project modules, 7,018 did not receive funding by Eurostars. Removing previous applications of resubmitted projects, the survey comprised 6,620 SMEs which received s with a link to an online questionnaire s were returned because the address was no longer valid. 193 SMEs were probably no longer in operation as neither the address nor the website were available on the internet and also the company was not found on November 2014 Page 40 of 139

42 the internet. Hence, a total response rate of 46% was obtained. The response rate amounted to 72% for funded and 39% for the not-funded firms. 43 Results based on the online survey were weighted to adjust for design of the survey and different response rates for fund and non-funded SMEs. In addition, the weights also took care of the fact that SMEs that applied in earlier cut-offs showed a lower response rate than those applied in more recent cut-offs. All results based on the online survey are extrapolated from the survey respondents to the population of 8,440 submitted project modules by SMEs. The online survey is referred to in this report by using Eurostars Online Survey (2014) as source. Non-funded firms were asked whether the project had been undertaken even without the Eurostars grant. According to the survey, about 32% (weighted) of the 7,018 not funded projects modules 2,218 were still undertaken - possibly in modified form. Out of the projects modules that were undertaken (i.e. funded and non-funded that were started even in absence of Eurostars financing), many project modules are still not finished. Where the project is not finished yet, the descriptive statistics reflect partly expectations of the SMEs. At the time of the survey s field phase (May 2014), 56% of the funded projects were completed, and 28% of the non-funded projects that were started were finished. 5.6 Quantitative Analysis As stated in the ToR, the Commission already foresaw "the use of data analysis techniques in order to compare SMEs that have participated in Eurostars with a control group of comparable SMEs that have not participated". In order to understand the extent of the Eurostars impact, an econometric counterfactual impact evaluation has been developed. The econometric study focuses primarily on job creation in participating SMEs and on innovation output as measured by patent applications. With respect to data gathering, this part of the expert report required that the ESE SME applicant data were matched to Bureau can Dijk s Amadeus database 44 from which information on employment figures were gathered. In addition, the ESE SME applicant data were matched with the EPO Worldwide Patent Statistical Database (PATSTAT) 45. These linkages across databases were established as follows: first, it was necessary to identify unique SME applicants within the 11,714 participants in submitted applications, as ESE did not create a company ID yet and firms could submit multiple applications. Such a company ID was created by the expert group using a text-field search engine analysing the applicants names and addresses. The text-field search engine uses a sophisticated n-gram 46 search algorithm so that spelling variants of the same firm can be identified. The proposed results were manually checked. After obtaining a unique ID, all participating SMEs were searched in A more detailed breakdown of response rates is provided in Table 9-12 in the Annex. See See A method used in text mining. Words are cut in fragments and these are searched in a text. 4-Gramms of the word Eurostars are, for instance, Euro, uros, rost, osta and so forth. These fragments are searched in order to capture spelling variants of firm names and addresses. November 2014 Page 41 of 139

43 both the Amadeus and the PATSTAT database using the text field search engine. Again, the results were manually checked. The resulting database brings together the applicant information and time series data on employment and patent applications which allowed conducting econometric treatment effects estimations. The expert group applies a variety of state-of-the art methods to this database: (i) difference-in-difference (DID) regression models, (ii) (Fuzzy) Regression Discontinuity Designs (RDD), (iii) nearest-neighbour matching method resulting in conditional difference-in-difference regressions (CDID). The application of treatment effects estimators allows conclusions about the effect of the Eurostars programme. With respect to job creation, for instance, the estimation of such models yields the conclusion on how many jobs would not have been created in participating SMEs if these had not gotten the Eurostars grant. Without such an econometric application, it could e.g. only be observed how many jobs a participating SME had before it had gotten the subsidy and how many jobs it had afterwards, but it could not be concluded what proportion of the employment change happened because of the fact that the firm got a subsidy and what has happened because of other reasons. The econometric models thus make causal inferences on programme effects possible. November 2014 Page 42 of 139

44 6 Evaluation findings 6.1 Relevance, efficiency and effectiveness of Eurostars Relevance for target group According to the governing and administrative bodies, Eurostars is a unique programme in that it is designed to target close-to-the-market projects of R&Dintensive SMEs and their partners. Overall, about 2 out of 3 applicants and participants were indeed R&D-performing SMEs, as shown by Figure 6-1. Scientific institutions (universities or public and private research institutes) represent the second largest group. Large companies or other SMEs are equally important when looking at participants. However, looking at unsuccessful applicants, non-r&d performing SMEs are more frequent than large companies and research institutes. From these figures, the most important partners of R&D-performing SMEs are clearly other R&D-performing SMEs; in fact 50% of all partners of the main applicant are themselves R&D-performing SMEs. In one out of three projects, SMEs partnered with universities, and in one out of four projects, they collaborated with other public research-performing organisations (PROs). According to the NPC interviews, the motivation for other participants, such as universities and PROs, was access to finance. However, some countries (such as Finland) aim to limit funding to universities in order to keep the programme focused on SMEs (see also Table 9-9 in the Annex). There are also countries, like Spain and Turkey, which reported that they only provide direct finance to participating SMEs under Eurostars, while universities and PROs can be funded as subcontractors. 47 Figure 6-1: Applicants and participants by organisation type Source: ESE evaluation data (2014) The ESE figures indicate that large companies were involved in 17% of project proposals. Other partners comprise cities, regions or industry associations. The 47 According to the funding rules of countries (as of February 2014), the following do not provide direct funding to universities and PROs: Belgium (Brussels and Flanders), Bulgaria, Ireland, Israel, Malta, Spain, Turkey and the UK. In addition, Switzerland does not finance PROs (although universities are funded). November 2014 Page 43 of 139

45 0 5 Percent Expert group participation of large companies is seen by NPCs as very beneficial, since they can provide established marketing channels or act as lead customers. An overwhelming share of total funds is allocated to SMEs, according to ESE data (Table 6-1). R&D-intensive SMEs, as the main target group of Eurostars, requested slightly more than 75% of total project funding. The decrease in SMEs share in more recent cut-offs is mainly due to the involvement of universities or PROs in projects. Their involvement resulted in a shorter time-to-contract, as the universities and PROs appeared to play a crucial role in project management and especially in negotiations with NPCs. Even more striking was the fact that Eurostars also reached quite young firms. Figure 6-2 demonstrates that more than 50% of the firms were less than 10 years old at project start. The median age of SMEs at project start was about 8 years. Figure 6-2: Distribution of SMEs by age Firm age in years at time of project start Source: Eurostars Online Survey (2014) Analysis of the interviews with participating SMEs also revealed that the programme meets the needs of R&D-intensive SMEs. It is recognised as fit for purpose and wellstructured by its participants. Almost all SMEs interviewed declared that they intended to apply to the programme again in future. Small companies experienced more difficulties when accessing Eurostars, mainly due to national rules: in several countries, funding rules impose a minimum size and other specific conditions for a company (e.g. age and business turnover). For this reason, in some countries public support does not target micro firms (<10 employees). Nevertheless, SME interviews indicated that Eurostars has also been instrumental in helping micro firms access to R&D funding. The main reasons identified by these firms for participating in Eurostars were access to funding, complementary knowledge, expertise and skills, and also to gain reputation and improve their image. Interviews with NPCs also supported this finding. For example, the Turkish NPC noted that micro firms in Turkey are not usually selected for support by national R&D programmes, November 2014 Page 44 of 139

46 while support through Eurostars, they have been able to prove that they have the capacity to implement a good R&D project together with a European partner. Table 6-1: Share of SMEs in total project cost requested Cutoff Source: ESE evaluation data (2014) Submitted projects Average share of R&D performing SMEs in total project cost Average share of all SMEs in total project cost Approved projects Average share of R&D performing SMEs in total project cost Average share of all SMEs in total project cost % 85.0% 76.4% 83.6% % 84.5% 75.3% 81.2% % 82.0% 74.6% 79.5% % 82.4% 73.2% 76.7% % 83.3% 75.4% 78.8% % 83.3% 71.6% 78.3% % 82.6% 73.1% 78.8% % 82.7% 69.1% 77.7% % 82.1% 65.2% 77.2% % 82.9% 44.1% 77.8% Total 78.4% 83.0% 72.6% 78.7% As shown by Figure 6-3, Eurostars addresses a wide range of technologies selected by SMEs in a bottom-up fashion (see also Table 9-8 in Annex 9.1). In fact, Eurostars covers a wide spectrum of technological sectors and research topics. Information technology was the most important field of technology. However, information technology (comprising both software and hardware) is losing the dominant role it had in the earlier cut-offs, which also reflected the traditional focus of EUREKA projects. Other fields of technology have steadily become nearly as important in later cut-offs. In addition, the selection of projects for funding also induced a shift from information technology, mainly towards biotechnology. The simplicity of the Eurostars application procedure, compared to that of other international programmes, and the relatively fast central evaluation process, are cited as important features of the programme by the governing and administrative bodies of Eurostars. Central evaluation is considered as crucial because it affects the length of time needed to carry out the projects by R&D-performing SMEs and finally how long it takes to get the product into the market. For this reason, the governing and administrative bodies of Eurostars think that Eurostars corresponds well to the needs and objectives of SMEs, as is also evident in their view, from the unexpectedly high number of applications. Furthermore, in some countries, like Norway, no national programmes exist for supporting R&D-intensive SMEs. This is why Eurostars appear to fill a gap in the research and innovation ecosystem of such countries. November 2014 Page 45 of 139

47 Figure 6-3: Broad technology categories of funded and not funded projects (at project level) Source: ESE evaluation data (2014) The results of interviews suggest that almost all SMEs have previous experience in national R&D programmes, and most of them have already applied either to Eurostars, EUREKA programmes or to EU Framework Programmes. The SMEs tend to think that participation in such programmes was an important learning experience and facilitated project preparation for Eurostars. It is mainly because of this past experience that almost none of the SMEs interviewed used intermediaries, such as consultancy firms, to help them in project preparation. Nevertheless, some SMEs recognise that intermediaries can contribute to simplify complex scientific project presentation, advise on the content and budget proposal presentation, and suggest an implementation plan, including a work plan and an overall approach to management. Figure 6-4: International sales of SMEs applying for Eurostars Source: Eurostars Online Survey (2014) As the overwhelming majority of R&D-performing and other SMEs applying to Eurostars were active in international markets before taking part in the programme, international market access was not in general a motivation to apply for Eurostars funding. In addition, interviews showed that access to a specific market, while expected, was not a crucial motivation for partner choice. Eurostars attracted a population of SMEs that had international sales experience despite their youth and their size. This is also reinforced by data from the online survey: the specific location of the partner did not play a significant role when looking for partners. However, the possibility to explore new markets with the help of a partner was a widespread motivation (see Figure 6-5 for a full set of motives for choice of partners). November 2014 Page 46 of 139

48 The SME interviews revealed that participating SMEs propose to connect in general with partners they already know and have partnered with before. Previous participation in other FP projects or other international programmes, such as EUREKA as well as in national funding programmes, supported them in developing a range of contacts and connections. These networks also helped the SMEs to learn about Eurostars: in general, SMEs obtained information about Eurostars from their partners and NPCs. Other channels, such as the Eurostars website, Internet search, national or international meetings/conferences and social networks, such as Twitter and Linkedin, were also cited as sources of information. The most important motivation for the choice of partners, however, was their technological know-how. The opportunity to exchange technological knowledge, either as a sender or as a receiver of technology transfer, turned out to be the most important motivation for partner selection. The online survey asked about the role of strategic choice of partners to maximise the probability of getting funded. For example, R&D-performing SMEs might collaborate only with partners from certain countries that provide significant funds to Eurostars, hence minimizing the risk of not receiving funding because a partner country runs out of money for the programme. In addition, another strategic motive for partner selection was not to collaborate with partners from countries known for long administrative procedures, which might delay the start of a project. It turned out that while both reasons were not very widespread for the applying SMEs, about 20% of SMEs agreed that those strategic reasons influenced their choice of partners. Figure 6-5: Motives for choice of partner by SMEs Source: Eurostars Online Survey (2014) The online survey showed that only a small minority of SMEs encountered significant problems except the delay in a project resulting partly from the administrative processes - with one or more partners (see Figure 6-6). No significant differences between funded and non-funded collaborations were observed. It is not surprising that November 2014 Page 47 of 139

49 about 91 % of SMEs with funded projects intend to work together with at least one partner in the future. What is more surprising is that partnerships of unsuccessful applications are likely to at least partly survive. Figure 6-6: Difficulties encountered in partnerships Source: Eurostars Online Survey (2014) Relevance within EUREKA Eurostars is referred to as being the flagship of EUREKA by the interviewees because of its advantages (central evaluation, availability of EU funding, near-to-market projects, etc.) to beneficiaries compared to other EUREKA instruments. Many interviewees underline that the EUREKA Member countries show higher commitment to Eurostars than EUREKA, mainly because of the availability of EU funding. Interviewees stated that because of its advantages, Eurostars is becoming more popular than EUREKA. Development of the EUREKA project portfolio has also been a permanent item in the NPC and HLG meetings agendas over the years. A regular analysis of the different project types and their relevance has confirmed the overall perception that Eurostars success has led to a decrease in the number of the other types of EUREKA projects (both individual projects and clusters). Since the Eurostars programme was launched, other projects have been decreasing in terms of the number of applications. Also, due to its different nature, Eurostars has stimulated mainly R&D-performing SMEs and this has created a change when compared with the original EUREKA participants, which were mainly from academia and large companies. As shown by Figure 6-7 Eurostars amounted to more than 40% of all projects of EUREKA. In addition, the traditional EUREKA individual projects became less important in recent years where as Eurostars projects get more and more important (see also Table 9-2 in the Annex for country level data). However, the majority of EUREKA s total project costs were in EUREKA clusters. November 2014 Page 48 of 139

50 Figure 6-7: Evolution of EUREKA s portfolio No. of projects Total cost of projects (in million Euro) Source: EUREKA: Annual Report (various years) Due to the characteristics of Eurostars projects and their impact on the EUREKA portfolio, EUREKA projects have become more SME-driven, involving small consortia and with shorter durations. A portfolio analysis of overall EUREKA projects demonstrates this change. ESE has applied two main indicators to monitor this trend, as stated at several HLG/NPC meetings: duration of projects: individual projects became shorter; participant s profile: projects are SME-driven, and so different from what was the case when EUREKA started, when the biggest participation came from large companies and academia Adequacy of funding level The amounts of funding requested and the amounts committed to funded projects are shown in Table 6-2. Due to the increase in the number of applications, the share of projects passing the quality threshold and receiving funding amounted to 66% of all projects above the threshold (see Table 3-2 or Table 9-1). However, the share of eligible applications that met the quality threshold went down from 70% for the first cut-off, and 45% for the second cut-off, to about 36% for the two latest cut-offs. The majority of NPCs emphasise that the full benefits of the virtual common pot have not yet been exploited in Eurostars, since some of the countries, in particular Germany and Spain, do not allocate sufficient funding for the programme. However, Germany, Denmark, Sweden and the Netherlands increased their contribution by 60% or more (see Figure 6-10 for details). Also, in countries affected by the economic crisis, such as Portugal, the programme is not run under the same conditions, due to the lack of motivation, promotion and funding. Other situations reported are related to national budgetary limitations, which can be considered as negative externalities imposed on other countries execution capacities. November 2014 Page 49 of 139

51 Table 6-2: Funding requested and commitments for funded projects Cut-off Total project cost of projects above threshold Total project cost of projects approved for funding Total funding for projects above threshold ** Total public funding for project approved for funding in million EUR in million EUR in % in million EUR in million EUR % % % % % % % % % % % % % % % % 9 * % % 10 * % % Total % % * For cut-off 9 and cut-off 10 funding is mainly based on estimates as NFBs were still in the phase of finalising and signing the funding agreements with participants at the time of data extraction (see Table 3-3 for details). ** funding is calculated as 42% of total costs Source: ESE Evaluation Data 2014 in % The NPCs and NFBs think that due to this situation some of the good projects selected dropped out of the ranking list, which in turn leads to frustration among applicant SMEs. Some of the NPCs interviewed stated that they notify their SMEs about the low level of funds allocated by certain countries and tell them that there is a high risk of not being funded in case they establish partnerships within these countries. It is proposed by some NPCs that if the level of funding by countries with a low Eurostars budget is not increased, then the selection criteria should be revised so that fewer projects pass the threshold level. Otherwise it will be difficult for NPCs to explain the situation to selected SMEs, and demand may decrease in time due to the dissatisfaction of applicants. 48 Apart from the above-mentioned challenge, some NPCs highlight the need for increasing the national funding for Eurostars by all participating countries, due to the increase in the number of applications. 49 It was recognised by the SMEs interviewed that funding was an enabler of their projects. Most of them considered that the level of funding was adequate, and if their projects had not been funded by Eurostars, they would probably not have been able to implement them. These claims can be validated by putting the national funding for Eurostars in context. First, the size of the national public funding for Eurostars can be compared to national NPCs mention that these shortcomings should be alleviated in Eurostars 2, as there will be new guidelines, which requires that the first 50 projects above the threshold should always be funded. In addition, Germany aims to increase its contribution for Eurostars 2 since Eurostars has become highly visible at national level, as explained in the interviews. As explained in Chapter 6.4 Eurostars 2 is designed with a much higher budget. November 2014 Page 50 of 139

52 public budgets for R&D. The relation between national contributions to Eurostars ( ) is compared to the size of government outlays and appropriations for R&D. This is depicted in Figure 6-8, which shows remarkable differences in the relative size of national public R&D spending for Eurostars. Figure 6-8: National funding for Eurostars in relation to national GBAORD (%) Figure 6-9: National funding for Eurostars in relation to the estimated public funding demanded by projects above the threshold (%) 1) Sources: ESE (2014; OECD MSTI ; EUROSTAT (2014) 1) Funds demanded to fund all projects ranked above the quality threshold is calculated as 75% of total public funding, which is calculated as 47% of total projects costs demanded by applicants from the respective country Sources: ESE (2014) Newer EU Member States such as Estonia, Latvia, Cyprus and Slovenia spend a much larger share of their GBAORD than older Member States with well-established R&D funding systems, like Germany and France. Large, R&D-rich countries are located at the bottom of the ranking, while most of the newer Member States, as well as non-eu 50 GBAORD = Government Budget Appropriations or Outlays for R&D by socio-economic objectives. November 2014 Page 51 of 139

53 Member States that participated in Eurostars (e.g. Norway or Switzerland), can be found at the top of this ranking. A second test for national differences compared the size of national funding (the supply side ) with public funds demanded by R&D-performing SMEs and their partners of the country concerned (see Figure 6-9). The latter was estimated as the size of the public funds needed to cover the public share of total project costs for national participants in projects that met the quality threshold set by the IEP. Hence this indicator highlights the relation of national supply and national demand for public R&D funding for SMEs via Eurostars. Again, large Member States devoted below-average amounts of money to Eurostars, while some of newer Member States and participating non-eu countries provided public funds above average. Both indicators thus confirmed the message coming from the expert interviews: at least in relative terms, the national funding of Eurostars differs quite significantly. Furthermore, statements by several national representatives that some countries provided relative low amounts of money to the programme are indeed supported by the data Efficiency of programme implementation Implementation efficiency with respect to the virtual common pot The Eurostars central funding mechanism is called a virtual common pot. Under a (real) common pot the contributions from all countries are put into one account to finance all projects starting with the best (based on the ex-ante evaluation) and ending with the marginal project for which funding is still available. Under a virtual common pot, an additional restriction is that the financial contribution of country x is used only for applicants from country x. If the financial contribution from country x is exhausted, the project is not financed, and the next project in the ranking list gets funded. For a collaborative approach, this implies that there are financial needs to be available from all home countries of the project applicants. Box 6: What is effectiveness and efficiency? Effectiveness: The extent to which the intervention's objectives were achieved, or are expected to be achieved, taking into account their relative importance. Also used as an aggregate measure of (or judgment about) the merit or worth of an activity, i.e. the extent to which an intervention has attained, or is expected to attain, its major relevant objectives in a sustainable fashion. Efficiency: A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted into results. Such a system gives an incentive to individual countries to provide enough money to finance the applicants from the countries. This is even more the case as the EU also puts additional money into the pot. But there is also a drawback. Suppose SMEs from countries "x" and "y" propose a joint project which gets an excellent ex-ante assessment. The project will only receive funding if there are sufficient funds from both countries. If one of the countries does not provide enough funding ( pot of country x or y is exhausted), the whole project will not receive funding. Hence the November 2014 Page 52 of 139

54 relative size of each country s contribution (compared to the relative size of the country or the demand by applicants from that country) to the common pot becomes crucial for the functioning of this type of funding system. This is highlighted in section above. Figure 6-10 provides some insight into the working of the incentive structure for national governments to provide a sufficient amount of national money for the programme. This figure shows, first, the initially agreed size of the national budget contributions for Eurostars, second, the extension of the national budget in the course of the programme. This figure also gives national money unspent with respect to national budget cuts of the Eurostars budget. The incentive effect of the virtual common pot allocation is demonstrated by Figure 6-10, as several countries extended their initial budget quite significantly. Germany, Denmark, Sweden or the Netherlands increased their contribution by 60% or more. On the other hand, there are also a significant number of participating countries which did not contribute the initially agreed amount to the common pot. A small number of Member States were unable to realise their original commitment, probably due to the impacts of the financial crises the on the national R&D budgets of Italy, Greece, Cyprus, Spain and Ireland. Figure 6-10: Initially earmarked budget, budget extensions and unspent budgets (in million Euro) Source: ESE (2014) To highlight the working of the virtual pot, the expert group compared the allocation of funding using the virtual common pot system, with a hypothetical common pot system (Figure 6-11). This figure shows that the allocation of projects following the virtual common pot system got more closely towards the first best allocation of the real common pot starting from cut-off 1 onwards. However, the deviation of the virtual common pot from the hypothetical real common pot increased following cut-off 3. In an increasing number of cases, funding requests from applications exceeds the November 2014 Page 53 of 139

55 available national earmarked funding for Eurostars. In addition, countries are not in a position to enlarge their contribution in parallel to the request for Eurostars funding from their partners. Here, we should also note that the economic and financial crisis hit the public research budget considerably, so that some countries were even not able to finance the initial agreed earmarked budgets. Therefore the crisis seems to limit the functioning of the common pot of Eurostars. It can be concluded from this analysis that: (1) for the vast majority of projects, it does not matter whether a real common pot or a virtual common pot rule was applied; (2) Eurostars has made considerable progress in cut-offs 2 and 3 towards a purely quality-based, real-pot allocation. In more recent years, the restriction from the availability of fund became more and more binding. Figure 6-11: Working of Eurostars virtual common pot vs. a real common pot Explanation: Under a real common pot allocation system, all projects would be financed according to the ranking list. Any deviations from the ranking list would represent deviations from the real common pot approach due to the use of a virtual common pot system. In Figure 6-11, the straight line represents funding allocation according to the ranking list (i.e. the real common pot approach). The dashed, coloured lines represent the cumulative number of projects actually funded according to their ranking and available contribution of member states. November 2014 Page 54 of 139

56 Instead of raw numbers, we use the share of cumulated number projects in order to be able to compare patterns across different cut-offs. The larger the deviation of the dashed lines from the straight line the larger is the deviation of the two allocation rules. Source: ESE Eurostars Evaluation Data (2014) Note: Data for cut-off 9 were not complete at time of data extraction. In interviews, NPCs, NFBs and members of HLG and EAG were quite satisfied with the way the virtual common pot worked. They agreed that the virtual common pot is the preferred funding mechanism since it attracts all countries to contribute to the programme and to increase their shares beyond the initially agreed level (see Figure 6-10). It is also easier for countries to raise funding from national sources for a virtual common pot than a real common pot. In that respect, interviewees state that the virtual common pot provides incentives to increase national funding. According to the NPCs, Eurostars was implemented quite successfully. The SMEs interviewed found that the programme is achieving its goals, even if there are still cases of projects in which partners are not financed at national level. SMEs think that this is the most important bottleneck of Eurostars that needs to be solved. However, some SMEs, despite an excellent evaluation of their projects, did not receive funding because of the restrictions coming from the virtual pot. Central evaluation Some NPCs stated that they found their involvement in the eligibility check unnecessary since it should be a straightforward process to control whether a project satisfies eligibility criteria and this could have been done easily by the ESE. The current process required participation of country representatives of NPCs in a session in Brussels, which added costs to the programme management. The possibility for applicants to automatically check for the eligibility of their proposal in Eurostars 2 is appreciated by the interviewees, since the applications will be received through an online system where an applicant will be warned automatically if any information entered does not satisfy the eligibility criteria. However, automatic eligibility checks will not substitute for detailed eligibility checks by a project officer. ESE project officers also need to check all applications to identify the potential experts most suitable to perform the technical evaluation of the application. The process of central evaluation in Eurostars is appreciated by all governing and administrative bodies involved in Eurostars, due to its speed and objectivity. November 2014 Page 55 of 139

57 The improvements in the process, such as the increase in the number of technical experts from two to three in Eurostars 2, and the involvement of experts with market expertise are considered as important developments. According to ESE data, the number of technical experts increased from 485 in 2010 to 1,736 in However, there is an imbalance in their geographical distribution (see Figure 6-12). By 2013, there were a large number of experts from UK, France, Italy and Spain. Also, a considerable large number of experts were listed from newer Member States such as Romania and Hungary. A small number stem from non-member states participating in Eurostars, like Switzerland or Norway, but also from Germany, at least when that number is compared to the size of the country or to the number of applications to Eurostars. The primary technological areas of expertise of the technical experts were in line with the technological fields covered by Eurostars projects. The majority of experts were in the fields of ICT and biotechnology and chemistry. The technical experts with an industrial background formed 38% of the whole population, whereas the share of experts with a consultancy background was 23%. These two types of expertise provided skills for evaluating the marketing aspects of Eurostars projects. Figure 6-12: Country of origin of technical experts in the Eureka database However, there are diverse views and experiences by NPCs on the sufficiency of technical experts with different expertise. While some interviewees mention that with the opening of the FP7 expert database to Eurostars and through the announcements of NPCs, there are now a significant number of technical experts who also have market knowledge, several NPCs stated that they observed, from time to time, that appropriate technical experts cannot be found by ESE. Those NPCs are of the opinion that the NPC network could be used by ESE to identify enough number of experts with the required qualifications. However, this has never been the practice in Eurostars. Source: ESE: Eurostars Annual Report 2013 Some NPCs stated that there are a number of actions which should be taken to improve the central evaluation process. First, the process should become more transparent, particularly by providing detailed feedback to the applicants in a more November 2014 Page 56 of 139

58 harmonised way as there are differences between countries. The NPCs also find the amount of information about the result of evaluation provided to them insufficient. This particularly leads to problems when SMEs with high levels of specialisation in their fields do not feel that their proposals are appropriately evaluated and contact NPCs for further explanation. Second, streamlining the process in order to decrease the time from application to evaluation is highlighted as an area for improvement. Third, further improvement of the expert database is emphasised as a requirement in order to ensure geographical and gender balance and increase the number of those from the business sector. It was also mentioned that the technical experts have not been provided with orientation training on how to evaluate Eurostars projects. As a result, there have been cases where completely different scores have been given to the same project by two experts, making the IEP s ranking process difficult. The SMEs interviewed are satisfied with the central evaluation process and the level of information received about the results of the evaluation. They see the evaluation process as clear, transparent and timely. In general, SMEs consider that information about evaluation is pertinent and they appreciate the feedback report. They think that the feedback on evaluation is important to facilitate the redrafting and improvement of the proposal, and a better comprehension of the overall programme objectives and evaluation criteria. Additional technical feedback is seen as an area where there is a potential for improvement. In some cases (and mainly in the case of unsuccessful applicants), SMEs noted the need for a more comprehensive analysis, not only based in a more detailed feedback report but also through personal interaction explaining the project application s deviations from evaluation criteria. Some of the interviewed SMEs said that though their applications to Eurostars were unsuccessful, the process was fair and credible, and the evaluation findings helped to further improvement of their projects. Finally, establishing a central database that can be accessed both by ESE and NPCs is deemed a necessity by interviewees. In this way they would be able to access information on the results of the evaluation as soon as it is completed, and the timeto-contract would decrease, as they might start earlier with their own procedures for the funding of the selected projects. 51 Synchronisation of funding The data available from ESE only permitted the expert group to look at the time-tocontract, defined here as the time-span between the cut-off date and the date when the funding contract was signed. As this differs significantly between countries, this question was analysed at the level of individual participants and not at the level of projects. ESE was not able to provide us information on time-to-funding, i.e., when the money first arrives at the participants. From interviews with SMEs, it is apparent that time-to-funding might crucially differ from time-to-contract (e.g. there were cases where the project was formally finished and most of the participants already received funding from their governments while there were participants which did not receive any financial support despite of an existing funding contract). 51 In the meanwhile ESE has established such a central database and NPCs have access to the detailed evaluation material include i.e. experts evaluation forms or IEP consensus reports. November 2014 Page 57 of 139

59 Synchronisation of time-to-contract, that is the time between the cut-off deadline and the signing of the funding contract, was considered as the most important challenge of Eurostars by the interviewees. Also, the interim evaluation recommended significant improvements. In addition, this was repeatedly a key topic of the annual HLG meeting. Hence the expert group examined by data analyses whether a general trend towards short time-to-contract were observed in general and whether the significant differences between participating countries in the time-to-contract were reduced. Finally, time-to-contract was regularly a key topic of talks with SMEs and programme governance and programme administration. Figure 6-13: Cumulated probability for still waiting for funding contract Figure 6-13 depicts the advances made to bring down the time-to-contract. It shows the estimated probability of the distribution of the time needed before the funding contract was signed for all 10 cut-offs (e.g. in the top chart of Figure 6-13 the red line of cut-off 2 indicates that the probability that about 70% of the funding contracts (=1-0.30) were signed within 400 days after the deadline; It took nearly 1600 days to sign all funding contracts). Figure 6-13 indicates a significant improvement in time-to-contract. NPCs were able to avoid extremely long delays in signing. However, in the initial cut-offs, a small number of SMEs had to wait more than three years before their funding contracts were signed in the initial cut-offs. November 2014 Page 58 of 139

60 Such long waiting times were avoided in later cut-offs. Even more important is that a growing share of contracts were signed with a period of less than 250 days from the cut-off deadline, 52 and an increasing number of funding contracts were signed within 200 days. Source: ESE Eurostars Evaluation Data (2014) Although overall improvements have been achieved, time-tocontract is still high in some countries. Figure 6-14 gives the average time-to-contract for cutoffs 1 to 5 and cut-offs 6-10 for participating countries. 53 The SMEs interviewed consider synchronisation as a key success factor for the future of the programme. They expect shorter time-to-contract with effective negotiation and faster transfer of funding, enabling them to accelerate and develop project tasks and achieve consortia objectives. In some countries, time-to-decision for Eurostars projects takes longer than that for projects for national R&D programmes. Figure 6-14: Time-to-contract by country Overall, ESE, NFBs and NPCs demonstrated their willingness and ability to shorten the time-tocontract. As a result, the average Source: ESE Eurostars Evaluation Data (2014) Table 9-10 in the Annex provides the ranking of countries by time-to-contract in a multivariate regression setting, accounting for complexity of projects, size of grants, technology sector and the number of projects above the threshold by country. For cut-off 9 and cut-off 10, not all contracts were signed in December 2013, and hence the time-tocontract for the second period was overestimated, as only signed contracts were used to calculate this figure. November 2014 Page 59 of 139

61 time-to-contract has gone down significantly from 458 days in cut-off 1 to 213 days in cut-off 10. The maximum delay in time-to-contract was even reduced much more, so that we can see significant progress between the first phase of Eurostars and the later funding rounds. However, there are still applicants who had to wait more than two years in cut-off 7, the last cut-off for which such an analysis is feasible. Synchronisation has also been a regular action item in the agenda of NPC and HLG meetings since the launch of the programme. In 2009, and following the first results of an internal working group on synchronisation, the EUREKA Ministerial Conference stated that constant and increasing attention must be given to synchronisation and rationalisation of national funding schemes and procedures to support EUREKA projects. As for the negotiations with NFBs, findings from the SME interviews indicate that the smoothness of the negotiation process differed from country to country. It is observed that interactions with NFBs reflect a diversity (as shown by Figure 6-11) which is not only linked to national regulations, but also to cultural aspects. Significant differences are also observed in time to funding, according to empirical evidence, as there are no such data available from ESE. Nevertheless, when asked to rank (on a scale from 1 (low) to 7 (high)) how effective the negotiations with NFBs were, the majority of SMEs selected 6 and 7, indicating that it was indeed effective. In general, the negotiation process in participating countries is characterised by diversity and heterogeneity. As evident from the SME interviews, different countries use different negotiation procedures (mainly deriving from national regulations and funding schemes), and it is not possible to identify a homogeneous Eurostars Negotiation Process. On the other hand, in many cases SMEs declare that the negotiation process went smoothly and was transparent. The NPCs stated that the target of achieving a time-to-contract of 7 months is feasible, and that all countries involved in the programme should take action to meet this target. Countries like UK and Romania report that they have met the target (6 and 7 months 54, respectively) by streamlining their processes. However, the lack of synchronisation in other countries put their projects at risk, since time-to-contract in some countries is twice and even four times longer than theirs. There are cases reported by SMEs about long time-to-decision in various countries, which created important obstacles in the implementation of projects. For instance, one of the SMEs reported that its partner applied for funding to its NFB. After two years of negotiations, the NFB decided not to fund the partner. As a result, the consortium had to be changed and the work-plan modified. It is observed that in some countries NFBs do not accept the project proposals forms submitted to Eurostars before giving a decision on funding, but require SMEs to submit a parallel proposal by completing their own application forms in their language. There are also cases where project proposals are evaluated by external experts of NFBs although they had undergone the process of central evaluation and selection by Eurostars. Double evaluation leads to prolonged time-to-contract, which not only 54 A time-to-contract of 7 months or shorter for all project participants is a specific milestone to be included, among others, in the bilateral agreements between ESE and NFBs under Eurostars 2 (Source: General Implementing Guidelines ) November 2014 Page 60 of 139

62 negatively affects their national SMEs, but also other project partners. Parallel evaluation of proposals can also increase vulnerability concerning the confidentiality of IP and information about business strategies. However, there are good practices observed in this area which can be adapted by others: Finland and Hungary, for example, note that in order not to increase time-tocontract, they accept the results of the central evaluation in their national support programme, and they neither request an application form in their language nor carry out an evaluation at national level. Hungary only expects the applicant to submit a budget plan with technical description of the work packages in the national languages. Italy, which has a long time-to-contract, abolished the legally demanded national evaluation to speed up the administrative processes. NPCs and NFBs were deemed to have a valuable proximity role, helping SMEs to better understand Eurostars approach, procedures and rules. SMEs stated that NPCs and NFBs should keep financial management adjustable to the needs of SMEs and projects. Aligning practices concerning eligibility date for expenses approval is also referred to as important, because in some countries project costs are eligible from the date of project approval, while in others NFBs accepts project expenses incurred before the date of approval. In other cases, SMEs reflect the need to speed up the processes by NFBs. SMEs argue that, considering that projects are international and collaborative, some management practices could be further harmonised, such as the eligibility date for project expenses. Harmonisation of funding rules Harmonisation of funding rules 55 is another critical issue, which still needs to be tackled for effective management of the programme, according to the interviews with the governing and administrative bodies of Eurostars. The members of HLG think that ESE cannot play an active role in harmonisation and synchronisation, and that this challenge should be addressed through a top-down process with an initiative of national ministries and the EU. Box 7: Funding rules Funding rules concern the following dimensions: eligible costs (i.e. what cost items, e.g. personnel, subcontracting, etc. are to be financed by NFBs), funding intensity (i.e. % of funding provided to SMEs and other participants in relation to their total costs of the project, maximum level of funds provided per project), and type of organisations funded (SMEs, universities, PROs, or large companies.) Details can be found in Table 9-11 in the Annex. According to the majority of the NPCs and NFBs interviewed, while the conditions such as the % of funding and the upper limit for finance per SME should remain countryspecific, steps should be taken to achieve a better harmonisation of funding rules. The funding rules per country demonstrate striking differences. For example, as noted 55 In financing Eurostars projects, NFBs follow their own funding rules. However, harmonisation of funding rules is required in order to decrease time-to-contract and provide a balanced funding within projects. November 2014 Page 61 of 139

63 above, universities and PROs are not funded directly in 8 countries, and one country (Switzerland) does not fund PROs at all. The maximum funding for SMEs by countries varies between 40% and 80% of their eligible project costs. This reflects maximum funding rates allowed for different types of R&D for different size classes of firm according to European State aid rules. Furthermore, 16 countries apply a maximum limit for the amount to be provided nationally for projects. 56 Box 8: Harmonization of funding rules results of a survey According to a survey conducted by ESE in 2011, 12 of 26 respondent countries implemented a new funding scheme for supporting Eurostars projects. The rest adapted an existing scheme. As seen in the figure below, of those countries which stated that they needed to adapt their funding schemes, o Ten countries declare that adaptation of funding rules is required (meaning, easy access to funding not only by industry but also by universities and PROs); o Eight countries state that adaptation is needed in procedures (time-to-contract, bureaucracy, flexibility); o Eight countries noted that both rules and procedures should be adapted. Those survey respondents that adapted their rules reported two types of change: o Five countries started to fund all types of participants; o Seven countries simplified their funding procedures. Harmonisation examples included the following: o use of English for the national application form; o more decision-making power for the NFB; o simplifying national funding documentation; o translating national funding guidelines into English; o Eurostars Application Form replacing national application form; o rescheduling the funding process to ensure a better time to contract. Source: ESE (2011) Eurostars National funding schemes survey 56 Countries applying a maximum limit for funding include: Austria, Bulgaria, Croatia, Cyprus, Denmark, Estonia, Greece, Iceland, Malta, Portugal, Romania, Slovenia, Sweden, Switzerland, the Netherlands, and the UK. Details are provided in Table 9-11 in the Annex. November 2014 Page 62 of 139

64 There are cases reported in NPC interviews where funding rules and procedure were adjusted in line with Eurostars, such as in Switzerland and the UK. Some countries, like Austria, introduced an upper limit for finance for projects in order to better estimate the national budget requirement for Eurostars. Furthermore, Turkey reported that the NPC carried out a study on harmonisation of funding rules and submitted a proposal to the top management of the NFB for approval. In Spain, funding programmes were adapted to Eurostars; for example, although Spain started to support national projects through loans instead of grants due to the financial crisis, grants are still used in the projects supported under Eurostars. In the context of the reform of the national research and innovation ecosystem, the NPC in Finland revealed that the NFB (Tekes) is undergoing a transformation. It will use more and more loans instead of grants as a funding instrument. In addition, towards the end of 2014, Tekes will become the main public venture capital (VC) provider for Finnish start-ups. Tekes funding will be directed to dedicated VC funds which then make investments into target enterprises. The way in which universities are funded in Eurostars is one of the issues mostly discussed by NPCs and NFBs. In some countries, NFBs can directly fund universities involved in Eurostars projects, while others can only fund universities as subcontractors. For an effective implementation, the need for a better harmonisation of subcontracting rules was raised by NPCs, since the total amount of work that can be subcontracted changes between countries. The SMEs interviewed also cited the lack of harmonisation in the type of organisations funded by NFBs as an important problem in Eurostars. The SMEs interviewed noted that differences in the practice of payments by countries also create problems. For instance, while some countries make-up front payment, others only pay based on invoices. Some countries reported that they have to apply double reporting: in addition to the technical reports sent by SMEs to ESE, NFBs ask for more detailed technical reports from them. It is suggested by some NPCs that, although there have been platforms created to exchange good practices on funding rules between countries, a stronger collaboration should be established by ESE between the NFBs in order to achieve a better harmonisation and synchronisation. ESE as dedicated implementation structure In general, ESE is found successful in the implementation of Eurostars both by the administrative and governing bodies, and SMEs interviewed. They state that ESE performed well and its implementation practices developed considerably since the beginning of Eurostars. It was also noted that ESE became literally the Eurostars Secretariat since the number of Eurostars projects is increasing while those of EUREKA is decreasing. ESE is described as more dedicated compared to the previous years. November 2014 Page 63 of 139

65 However, there is room for improvement both in relation to the strengthening of the ESE s structure and its operations, according to the interviews. The following list of improvement actions was proposed by some of the interviewees: Resources are scarce, and they had to put resources from other EUREKA programmes into Eurostars. In order for ESE to improve programme implementation, it should be provided with enough people and resources. There is a high turnover of staff in ESE, which affects effectiveness at operational level. A study has been carried out in ESE to understand the reasons for this high turnover rate. The NPCs emphasise that the result of the Report Survey on Staff Turnover (see Box 9 below) should be assessed and measures taken according to the results of the assessment. In line with the point above, interviewees noted that it is important to increase the number of staff in ESE and to communicate this need with all countries involved in Eurostars in order for them to support a staff increase. It is also required to evaluate and map technical competencies of ESE staff in order to improve the quality of the programme. As noted by NPCs and NFBs, there is a need to re-evaluate Eurostars in terms of its workload compared to the other EUREKA pillars in terms of balance. The EU should reinforce its support to include more people and devote more money to the management of ESE, according to the interviewees. They emphasise that Eurostars cannot jeopardise the success of other pillars of EUREKA. ESE staff should work in a way that they provide evidence for the organisations involved in the governance system of Eurostars, for them to take evidencebased decisions. In this respect, ESE should be able to communicate effectively with IEP members. This may require the introduction of strategic and competitive intelligence as a collective staff skill, according to the NPCs. Concerning programme implementation, ESE is required to improve its practices in collecting final and marketing reports from the beneficiaries, in monitoring ongoing projects, and in auditing projects. The NPCs also think that ESE needs to design an audit strategy to ensure that money is spent according to project objectives. ESE should establish a robust database to collect, store and process reliable data on projects and beneficiaries. The interviewees think that it is important to improve the IT system, provide access to the database for NPCs and NFBs, and develop summary tables including important data and statistics per country in order to increase and speed up coordination. These conclusions are also in line with the Agilis report summarised in Chapter 4. ESE should develop and implement a strategy for the proactive promotion, branding and communication of Eurostars. The interviewees suggest that the Eurostars label could be used to develop easier access to venture capital finance, new export markets, and other complementary tools. This should also include the dissemination of success stories at national and European levels. ESE is required to work more closely with each country and have in place a stronger coordination among NPCs and NFBs. Interviewees also note that it is important to improve knowledge management practices. ESE has tacit knowledge that could be translated into a network activity plan in order to November 2014 Page 64 of 139

66 improve knowledge-sharing between the actors involved in the governance of Eurostars. It is important to streamline ESE processes by minimizing bureaucracy. It is stated that ESE pays too much attention to formal governance processes, such as requiring stamps, manual signatures and postal delivery of declarations of commitments and expenditures, practices which were abandoned in some countries like Finland with the adoption of electronic processes. The lack of a central active mechanism assisting SMEs to find partners is an issue in promoting transnational collaboration. NPCs suggest that it would be very useful to design and implement a mechanism which helps SMEs find partners actively. The system in LinkedIn is not effective, according to the interviewees. One of the limitations of ESE is related to the transparency of the management of the top-up funding from the EU. For this purpose, regular updates on funding started to be provided to the HLG only towards the end of the programme. Box 9: Results of the survey on staff turnover in ESE The Survey on Staff Turnover was conducted in 2014 at the initiative of the incoming Head of ESE in order to understand the reasons for turnover between 2012 and former ESE staff whose employment periods ranged between 2 months to 11.5 years were surveyed. These staff represented more than half of the directly employed staff as of The findings from the survey revealed that staff motivation and satisfaction should be increased by improving the management and administration practices in ESE. The report on the survey produced by ESE listed the following as practical conclusions which needed to be addressed in the short term: Structural change: (a) a coherent organigram for the organisation, clearly separating functions, notably human resources and finance; (b) clear job descriptions relevant to the needs of ESE. Direct and indirect financial means of motivating staff: (a) a clear and published salary scale based on grades; (b) transparency in the allocation of salary-based benefits. Non-financial motivation: (a) training in communication in multicultural environment for all staff; (b) training to promote skills development and career progression. Career perspectives: encouragement for internal upward mobility for interested, suitable staff, prior to advertising a vacancy. Management: (a) training in people management in a multicultural environment; (b) training on organisation of tasks and workload monitoring. Source: ESE (2014) Eurostars Survey on Staff Turnover Leverage effect of EU funding and participation The interviews with the national organisations and high-level bodies in Eurostars indicate that EU funding has a leverage effect on the programme in that it persuaded the national policy-makers to have clear goals and allocate enough funds for Eurostars. As such, the participation of the EU is described as stimulating and as an important political argument by the NPCs, NFBs and members of the HGL. It is noted that thanks to EU funding, most Member States have mobilised significant national November 2014 Page 65 of 139

67 funding for a special target group (R&D-intensive SMEs) which would not have been available otherwise. EU funding has also led to a competition among countries, which resulted in an increase of national funding beyond originally agreed levels. The interviewees noted that EU funding has also created a strong marketing effect towards potential beneficiaries. The existence of EU support increased the prestige and attractiveness of the programme for SMEs. Because of this, the EU involvement helped NFBs reach new SMEs, which had not approached them before, and expanded their client-base. This is expressed as a clear benefit of EU involvement beyond its financial contribution. The support of the Commission for the continuation of the programme is also highly appreciated by the NPCs and NFBs interviewed. The interviewees underlined that the Commission s involvement was essential to move towards a higher level of synchronisation and harmonisation in Eurostars. They noted that Eurostars 2 sets ambitious goals (such as a time-to-contract limit of 7 months), which could only be reached with the involvement of the EU. One of the proposals mentioned in this context is that the EU should send an official letter to the high-level national bodies to which NFBs report, requesting that synchronisation and harmonisation is completed before launching Eurostars 2. It was also suggested that synchronisation and harmonisation is included as items in the bilateral agreements of Eurostars 2. As for the role of the EU, the NPCs note that the Commission s role was critical at the beginning of the programme when EU funding acted as glue. According to some of the NPCs, the level of Commission s involvement decreased over time, and this is seen as a healthy evolution. The Commission s role is mainly defined by the NPCs as supervising and controlling EU funding spent through the programme. While some interviewees considered this as a normal intervention, since the Commission needs to see whether taxpayers money is spent appropriately, others suggest that they expect the EU to act as a stakeholder, since Eurostars success will also become the EU s success, as it is the most successful programme implemented under Article 185. Having said that, the NPCs state that some of the practices of the Commission were felt to be overblown ; for example, unclearly defined and communicated requirements of NFB s auditing reporting stopped the payment of EU top-up funding to NFBs for a while. Some NPCs and NFBs mentioned that the EU imposes too much bureaucracy on an NFB, such as the evaluation of the internal control system and reporting, requiring Annual Financial Audit Reports, auditing visits to NFBs, etc. It was stated that the NFBs at the same time report to the EU about the same notified funding instruments they use in Eurostars, which may mean double reporting. It is noted by NPCs and NFBs that auditing is much harder in Eurostars compared to other EU programmes. Some of the interviewees suggest that the NFB auditing within the framework of Eurostars could be abolished, and that the EU can use the audit reports produced by national independent audit offices as they already audit the NFBs as government agencies. Some of the NPCs state that since the EU has already agreed to the procedures of NFBs by participating in Eurostars, the auditing rules of the NFBs should also be accepted. They also mention that the EU should not audit the national programme participants, since this creates too much bureaucracy. November 2014 Page 66 of 139

68 6.1.5 Efficiency of programme governance According to interviews with the NPCs, NFBs and members of HLG and EAG, the governance system of Eurostars functions properly and the interaction between the national and central administrative bodies is efficient. Interviewees also noted that the governance of Eurostars has improved over time. One good practice is that working groups/task forces are formed immediately to solve a particular problem. Recommendations developed by working groups/task forces are taken into account by the organisations involved in governance. In addition, the communication and partnership is well developed between different parties in the Eurostars governance mechanism. As noted in the interviews, the networking between countries works very well, and valuable information is shared continuously within the Eurostars network, which in turn increases the level of integration of the programme. The interviewees noted that the most intensive interaction takes place between the ESE and EAG, since they were involved in day-to-day management of the programme, and that cooperation between ESE staff and members of the EAG is very good. The interactions with the HLG are limited mainly to HLG meetings. The interviewees propose that the ownership of HLG towards the programme should become more significant for the facilitation of synchronisation and harmonisation, and therefore that communication with the HGL should be enhanced. The need for higher ownership is also evident in the HLG minutes, where a large number of issues are discussed at several meetings without concrete conclusions or solutions being reached. It is suggested that there should be much closer interactions among NPCs and NFBs. It is also underlined that the role of NFBs is critical in efficient implementation of Eurostars, and that therefore they need to be more involved in the programme (while they have started to interact more closely with the introduction of annual NFB meetings, it is not sufficient to create and sustain effective communication channels). Therefore it is proposed by the NPCs that the NFBs should be more involved in the governance of the programme in order to achieve better synchronisation and harmonisation. The proposal for the improvement of the project database by ESE is also underlined by the NPCs and NFBs as a means to enhance governance. It is expected that with the development of a reliable and accessible database, all parties involved will share information and they will have the same level of information about the programme. In this respect, interviewees highlighted the importance of avoiding the double entry of project-level data, eliminating coding errors and entering information on every aspect of projects and beneficiaries required for a sound monitoring and evaluation (such as an effective project start date, size of national funds at project level, etc.). Through content analysis of HLG Minutes, it is observed that HLG meetings reflected a very intense debate on several issues, such as synchronisation, harmonisation, funding or evaluation processes. In many meetings, a final remark indicates "The HLRs and NPCs took note of the Eurostars state of affairs". Nevertheless, there is no evidence of concrete objectives, towards future actions, associated with the debate established during the meetings, neither with the assessment presented by ESE, HLRs or external experts. HLG Minutes also state different MS positions, but there is not an November 2014 Page 67 of 139

69 overall conclusion for each topic that could crystallise an overall agreement on the decisions taken. In several HLG meetings, it has been proposed to introduce performance indicators related to Eurostars management (e.g. in February 2011 it is stated that a working group proposed a 7-months timeline after cut-off as best practice for time to contract, which was considered "ambitious"; in April 2011, it was suggested that the time to contract target should be 9 months), but in subsequent meetings there is no indication of progress made and measures taken to achieve the foreseen result. 6.2 Impacts of Eurostars Impact on employment growth: an econometric analysis This section of the report presents an econometric counterfactual impact evaluation, and thus revolves around the central question of every evaluation study, that is, the estimation of treatment effects. The treatment effect of interest in this study is the socalled Average Treatment Effect on the Treated (ATT), i.e. the effect of the policy on a certain outcome variable of the subsidised firms. Outcome variables of interest in the context of the Eurostars programme could be R&D spending, R&D employment, the number of new products and/or processes introduced, the number and quality of intellectual property rights (e.g. patents) obtained, total employment (growth), and sales (growth). As job creation through the programme is typically of central interest to policy-makers and also because data on firm-level employment are available in large databases, the first outcome variable considered in the econometric analysis is employment growth. Firm-level employment can be observed at the time of the project application and also after the project. Calculating a treatment effect of the policy amounts to the estimation of how many employees a subsidized firm would have had if it had not received the subsidy. This counterfactual situation can then be compared with the actual situation, i.e. observed employment after the project, and the difference between the actual and the counterfactual situation amounts to the treatment effect on the treated. Conceptual design of the econometric study In order to implement an econometric estimation of treatment effects, employment data of the Eurostars applicants were collected from the Amadeus database. At the time of data collection, the Amadeus database contained employment information of European firms up till the year The econometric study is carried out at the firm-level, but the Eurostars applicant data might contain multiple observations of a firm if a firm applied for more than one project. A first necessary step, therefore, is to disambiguate the applicant names in order to determine how many unique firms submitted project proposals. In total, the ESE application database contains 11,714 records of applicants in submitted project proposals. These 11,714 records correspond to 3,525 submitted projects, i.e. the average project has about 3.3 participants. November 2014 Page 68 of 139

70 The focus of the treatment effects study is on SMEs. Out of the 11,714 proposed project modules, 8,440 were submitted by SMEs. These records were disambiguated in order to identify unique SMEs within the submitted proposals. The disambiguation was executed with text-field search algorithms using the applicant s names and addresses, as well as manual checks. According to this routine, the 8,440 project modules were filed by 5,466 different SMEs. As will be outlined below, it is essential for the econometric evaluation that projects have gone through a complete peer review process. Consequently, out of the 8,440 project modules filed by SMEs, 1,336 were dropped. These are mainly part of incomplete, ineligible or withdrawn project applications. Out of the 7,104 retained project modules, almost half started only in 2012 or later. These observations were also dropped, as the most recent employment data are from 2013, and it may not be meaningful to expect any effects of the subsidy during a very short time-frame. Thus only project modules started before 2012 were retained for the treatment effects analysis, numbering 3,654. According to the disambiguation routines, these were submitted by 2,711 different SMEs, and correspond to 1,536 projects. The firms were searched for in the Amadeus database using the same search engine that had been used for disambiguating the proposal records. For 1,480 firms out of the 2,711, employment data could be found. However, some firm-level observations cannot be used, as the employment data were not available before the project started (but only for a later point in time), and therefore no reasonable analysis could be conducted for such cases. Furthermore, some large outliers in the data had to be removed. The final sample consisted of 1,313 SMEs. As some of these SMEs applied multiple times and were sometimes successful and other times not, some further data manipulations were made. If a firm ever won a grant, all proposals that were submitted by the same firm but did not win a grant were dropped from the analysis; this avoided that a firm was used as non-funded firm in a certain period, although it possibly had been funded before. Such a scenario would bias the estimated treatment effect towards zero. If a firm applied multiple times, only the first application for which the review process had been completed was used to estimate treatment effects. Although this constitutes a simplification of the actual situation, it is not more arbitrary than, e.g., averaging project characteristics in order to aggregate the applicant data to the firm level. Note that this does not change the number of observations that can be used for the analysis. The final sample can be split basically into three groups: observations referring to (i) winning projects, (ii) non-successful applications that did not cross the threshold of 400 points of the Eurostars peer-review score system, and (iii) non-funded applications that passed the threshold, but were not funded because of budget restrictions imposed by the virtual common pot. As the outcome variable, the average annual employment growth of a firm is considered. As common in the literature, this is approximated by the following logarithmic transformation: ln(annual empl. Change) = (lnempl_2013 lnempl_start) / (2013 START), November 2014 Page 69 of 139

71 where EMPL_2013 denotes total employment in the year 2013, and EMPL_START the employment in the year before the project started (or was supposed to start in the case of the non-funded projects). This difference was divided by the number of elapsed years since the project started to obtain a measure, in %, for the average annual employment change at the firm. Descriptive statistics for evaluation sample Descriptive statistics of the variables used in the econometric study are presented in Table 6-3. The sample was split into the three groups of firms described above: firms with at least one successful application, and firms that applied and were not successful, either because their proposal received a peer review score below 400 points, or because the national funds of one partner were exhausted although the project was evaluated as being above the quality threshold. Table 6-3: Descriptive statistics for evaluation sample Variable Granted projects Mean N = 435 Std. dev. Non-granted, above threshold Mean N = 220 Std. dev. Non-granted, below threshold Mean N = 658 Std. dev. Ln(Empl.growth) Score Empl start D(Large) D(PRO) The average annual employment growth rate of successful SMEs was 7.9%. For the two control groups, the growth rates were about 6.1% for applicants above the threshold but not funded, and 6.9% for those below the threshold; i.e. on average funded firms grew slightly more than non-funded firms. As the outcome variable in this case is a relative measure, that is, the employment growth is measured in %, the observed growth rate crucially depends on the initial firm size. The larger the firm before the application to Eurostars, the less it will grow in %. For this reason, the subsequent regression models control for initial firm size. As can be seen in the descriptive statistics, the funded firms had, on average, about 27 employees before they submitted an application, and were thus larger than the nonfunded firms, which had about 23 or 24 employees. Using a scatterplot and a nonparametric regression, as in Figure 6-15, shows that the relationship between the growth rate in % and the initial firm size very closely fits a quadratic modelling of the logarithmic initial firm size. Therefore the subsequent regression models will include ln(empl start) as well as the squared term of this variable on the right-hand side of the equation. November 2014 Page 70 of 139

72 Expert group Figure 6-15: Scatterplot and regression of employment growth on initial firm size log(initial employment) annual empl. growth (in %) non-parametric regression parametric regression As will be further elaborated below, using the peer review score in the regression models is important. The original overall technical evaluation score varies between 0 and 600. For easier interpretation of the subsequent estimations, the score was rescaled so that the quality threshold equals zero. It therefore ranges between -400 and 200. The funded firms, on average, received a score of 58, the non-funded above the threshold a score of 31, and the ones below the threshold scored about -61. Further control variables that were considered are time dummies, technology field dummies, and a EU15 and EU28 dummy. This means that it was taken into account that firms in different years, sectors and countries may show heterogeneous growth patterns because of macroeconomic shocks. In addition, some project characteristics besides the peer review score were used. The regression models include a dummy variable indicating whether a large firm participated as a project partner, D(Large), and a dummy whether a university or public research institution, D(PRO), was part of the consortium. Firms that maintain close connections to PROs and large firms may be able to enter a steeper growth path because of such established business connections than other SMEs. In the descriptive statistics, it can be seen that applications including large firms and/or PROs were somewhat more likely to be evaluated above the quality threshold: in the samples of funded and non-funded firms above the threshold, 55% and 54%, respectively, are applications which include at least one PRO in the consortium. This share only amounts to 46% in the applications rated below the threshold. With regard to the participation of large firms, the corresponding shares are 18% and 17% in the samples of funded and non-funded firms above the threshold, but only 9% for firms below the threshold. November 2014 Page 71 of 139

73 The following econometric evaluation will analyse whether these descriptively visible but small growth differences between funded and non-funded firms are statistically significant, and, even more importantly, whether these differences are caused by the Eurostars project grant or whether the better-performing firms were simply more likely to get the grant. The concept of Difference-in-difference estimation In order to test whether the growth differences between funded and non-funded firms are statistically significant, a difference-in-difference (DID) estimation was considered. The database basically consists of four major data blocks of interest. The data consists of the treatment group and the control group as well as two points in time. Each firm was observed in a year before the project was submitted and in a year after the project, or otherwise at the most recent year of the Amadeus dataset (2013). As this is panel data, that is, firms and their characteristics can be traced over time, DID estimators are applied in all applications that follow. The idea of the difference-in-difference (DID) estimator is based on exploiting the panel structure. The difference in employment can be calculated as follows for each observed firm over time, i.e. for both the treated firms and the control group; i T = ln(empl i,t1 ) ln(empl i,t0 ) j C = ln(empl j,t1 ) ln(empl j,t0 ) Here T denotes the treatment group, i.e. the firms that applied for Eurostars and also obtained a grant, and C the control group, i.e. firms that applied, were ranked, but did not receive a subsidy, either because their project was evaluated as being below the quality threshold, or because the national budgets of at least one project partner did not suffice to fund the project. The treatment group receives a treatment, here the project grant, in period t1, the growth of employment over time is calculated. As the growth of employment may well be subject to economic shocks that concern the whole economy, the growth of employment of the treatment group is therefore related to the growth of employment of the control group. An underlying assumption is that both treated and control group would be affected by economic shocks in the same manner. Thus the treatment effect, α, can be estimated as the difference in both differences: α DiD = E( i T ) E( j C ). The expected value would then be estimated as the sample average of employment growth in the treatment and control group respectively. A test of whether the treatment effect is positive in statistical terms, that is, the programme actually increases employment in the funded firms, could be implemented by a two-sample t-test on mean differences in this example. In a regression context, it would mean that the employment change is regressed on the treatment dummy variable, [D(TREAT)]. This would be numerically equivalent to conducting a twosample t-test. The regression, however, easily allows for the inclusion of other control variables that could affect employment, besides the Eurostars treatment. Such estimation should mainly be seen as a naive benchmark estimate of the treatment effect, as an underlying assumption is that the treatment is exogenous, i.e. November 2014 Page 72 of 139

74 it is randomly given to the applicants. This clearly does not hold, as the treatment is assigned based on the peer-review scores and the availability of funds in the virtual common pot. However, it is nevertheless instructive to estimate a DID model, as it delivers a first insight into the statistical significance of the descriptive findings. It should also be noted that the subsequent regressions are slightly adapted to the data: although two points in time are considered for the estimation, the elapsed time between the initial year, t0, and the last year of the period of observation, t1, varies across the observations. Therefore the outcome variable is not the growth rate (i.e. the differences in log employment) in the econometric application as illustrated in the example above; instead, the difference is scaled by the number of elapsed years, which yields an average annual growth rate over the observed period (see data description above). Furthermore, the DID estimation is carried out as a multivariate regression using the control variables described above. DID estimation In the first DID model, the employment growth rate was regressed on the treatment dummy and the controls. It was found that the funded firms grow on average 1.8 % more than the non-funded firms, and that this effect is statistically significant at the 1% level. The initial firm size has the expected negative effect, as shown in Figure Furthermore, it turns out that firms in the EU15 and EU28 areas respectively grew on average about 5.5 and 6.7 % more than firms in other EUREKA countries (the EU15 and EU16-28 are defined on a mutually exclusive basis, such that the EU28 dummy includes the 13 Member States which were not part of the EU15). Concluding that these significant effects are caused by the fact that the treated firms received a Eurostars grant is premature, however. As explained above, the DID method would be only valid under the assumption that the grants were given randomly to the firms. This is clearly not the case, as the selection of winners is based on the peer-review score. However, the assumption of an exogenous treatment could be fulfilled, if the nonfunded firms below the threshold are discarded from the sample. All the remaining firms were eligible for a grant, and whether they received it or not was exogenously determined by the availability of funds in the virtual common pot. When performing the regression with the subsample of firms above the threshold, the treatment effect amounts to 2.2 % and is significant at the 5% level (cf. Table 6-4). However, this should still not be seen as the final evaluation result, as the treatment is still not fully exogenous; the grants are more likely to go to firms that have a higher peer-review score even within the group of applications above the threshold. November 2014 Page 73 of 139

75 Table 6-4: Difference-in-difference regressions on average annual employment growth full sample N = 1,313 Coef. (Rob. std. err.) subsample above threshold N = 655 Coef. (Rob. std. err.) subsample above threshold and score as control variable N = 655 Coef. (Rob. std. err.) D(TREAT) 0.018*** 0.022** 0.031*** (0.007) (0.010) (0.011) SCORE * (0.0004) SCORE (0.001) lnempt *** *** *** (0.008) (0.011) (0.011) (lnempt0) *** 0.007*** 0.007*** (0.001) (0.002) (0.002) D(PRO) (0.006) (0.009) (0.009) D(LARGE) (0.008) (0.011) (0.011) EU *** 0.036* (0.014) (0.022) (0.021) EU *** (0.017) (0.025) (0.025) Intercept 0.094*** 0.110*** 0.128*** (0.021) (0.029) (0.030) R Note: All regressions include a set of time dummies and three technology field dummies. *** (**, *) indicate a significance level of 1% (5%, 10%). For this reason the DID regression was performed again, but this time the score was added as a control variable to the regression. By doing so, the treatment dummy can now be considered to be exogenous, as once it is conditioned on the score, the treatment only depends on the budgets in the virtual common pot which are exogenously determined (i.e. not under control by the applicants). The score has been added linearly and in a quadratic fashion. Note that other choices of functional forms made no difference in the final results. It is found that the funded firms grow by 3.1 % more, and this effect is statistically significant at the 1% level. The score variable is weakly negatively significant. Regression Discontinuity Design estimation As robustness checks, further econometric methods were applied to the data. The framework of the Eurostars programme is such that it is almost a textbook application of a so-called Regression Discontinuity Design (RDD). In such a setting, firms are selected into the programme because of a score that determines whether they receive November 2014 Page 74 of 139

76 funding or not. There are two variants of RDD; sharp and fuzzy. In the present case, the application of a Fuzzy RDD (FRDD) is the most appropriate. Example with simulated data for explaining the idea of sharp RDD In order to explain the logic of RDD, the idea of a sharp RDD is described for illustrative purposes. Suppose the budget constraints in Eurostars were absent. Then firms would be funded if they passed the threshold of 400 in the scoring model of the peer-review process, otherwise the submitted proposals would not be retained for funding. This means that the subsidy would not be given randomly to firms, as the DiD approach outlined above assumes, but strictly based on the scores. In such cases, one could expect a relationship between a certain target variable (here employment growth) and the scores, as depicted hypothetically in Figure The underlying idea in the simulated data is as follows: if one would compare simply the mean of the growth in the outcome variable (as has been done with the DID estimator), one would find that the values are -57 for non-funded firms, i.e. observation below the threshold, and 123 above the threshold, that is, for funded firms. Thus the DID estimator would estimate a treatment effect of 123 (-57) = 180. This would, however, be misleading, as the graph illustrates that firms which got funding differ in two ways from the other firms: first, their observations are shifted upwards because of the treatment, i.e. higher growth in Y. Second, when looking at the plot, it also shows that the relationship between outcome variable and score shows a steeper slope for the score in the group of funded firms. This reflects the possibility that the firms that obtain better scores are simply the better firms on all accounts, i.e. they might have grown more than the firms without a subsidy, even if they had not received the treatment. In this illustration, the simulated treatment effect actually amounts to 50 units of higher growth in Y, and not 180 as the DID estimator would suggest. The correct treatment effect could be estimated here in this example by regressing the growth in the outcome variable on a treatment indicator (i.e. a dummy variable taking the value 1 if a firm is ranked above the quality threshold, and zero otherwise), and the score, which is interacted with the treatment dummy. The correct treatment effect in the graph can be identified visually by the vertical difference between the two regressions lines at the quality threshold, that is, the discontinuity point created by the subsidy. Note that the RDD estimator thus identifies a local treatment effect around the threshold, whereas the DID estimator identifies an average treatment effect for all treated observations. November 2014 Page 75 of 139

77 Expert group Figure 6-16: Hypothetical illustration of data conforming to a sharp RDD: RDD applied to Eurostars data score The example above was used to introduce the logic of Sharp RDD estimation. The actual Eurostars data, however, do not correspond to a Sharp RDD, but a Fuzzy RDD, as there are also firms that submitted proposals which passed the quality threshold of a 400-point score, but could not obtain funding, as the national project budgets of at least one partner s country did not suffice. Here the funding is not strictly a function of the score, but also happens because of some further, typically unobserved, factors. In the fuzzy RDD framework, one can not only make use of the difference between funded and non-funded firms at the discontinuity point in order to derive an estimate of the treatment effect, but can also exploit the fact that observations of firms that passed the quality threshold but did not get funding are available. These firms might be the better firms when compared to the non-funded firms in terms of unobserved characteristics such as management quality, R&D capabilities, and so forth, but it could be assumed that they are similar in these unobserved qualities to the funded firms. Therefore one can make use of these additional observations in order to derive a treatment effect. Estimating the treatment effect is slightly more complicated compared to the sharp RDD. In the framework of a fuzzy RDD, an instrumental variable (IV) regression is estimated, where the treatment indicator is instrumented with a dummy that indicates whether a submitted proposal has been ranked above or below the quality threshold by taking into account the scores, as in the example above. The estimator can be implemented with the familiar Two-Stage Least Squares (2SLS) approach. In the first stage, the funding dummy is regressed onto the above/below-threshold dummy and the scores of funded and non-funded firms. The linear prediction obtained from this regression is used in a second stage, instead of the actual treatment dummy. November 2014 Page 76 of 139

78 Another decision that has to be taken is the choice of functional form regarding how the proposal scores enter the model, as the results of the RDD may be sensitive to this. For instance, one might choose a linear specification, and this may be appropriate closely around the threshold; but as the difference between the individual scores and the threshold increases, it can become imprecise. In practice, a nonparametric regression could be applied that makes no assumption about how the scores should be modelled. This requires a very high number of observations, though. As the present database is too limited for a fully flexible non-parametric approach, a second-best solution was chosen: namely, one estimates a parametric IV regression as described above, but restricts the sample to observations that scatter closely around the threshold. As, typically, there is no clear definition of closely in this context, researchers usually try different cut-off values to restrict the sample and check the results. The idea behind this is that the more the sample is restricted around the threshold, the function form assumptions that are made are more likely to hold locally rather than for observations that are distant from the threshold value. The trade-off is that an increasing chance that the model is correctly specified locally coincides with a loss in precision of the estimates as observations are discarded, i.e., the model might be more appropriately fitting the data, but the researcher may find no significant effect because of smaller sample sizes. The Fuzzy RDD model is estimated below. First the full sample was used, then the sample was first restricted to observations with a score +/- 100 points around the threshold, and subsequently to +/- 50 points around the threshold (cf. Table 6-5). For the first two versions, no statistically significant treatment effect was found, but for the final step using the +/-50 sample, a positive and significant treatment effect of 7.6% was estimated (cf. Table 6-5). Note again that this treatment effect is a local treatment effect for proposals close to the threshold. Comparing this result with the average treatment effect estimated with the DID regressions above suggests that firms close to the threshold benefit more from the Eurostars grant than firms with the best proposals. This could be an indication that the best firms (as measured by the peer-review score) may even have performed well in the absence of the treatment, whereas firms close to the threshold have benefitted a lot from the Eurostars grant. November 2014 Page 77 of 139

79 Table 6-5: Fuzzy RDD: IV regressions on average, annual employment growth full sample N = 1,313 Coef. (Rob. std. err.) subsample +/- 100 around threshold N = 1139 Coef. (Rob. std. err.) subsample +/- 50 around threshold N = 681 Coef. (Rob. std. err.) 2nd stage, dependent variable: average, annual employment growth D(TREAT) ** (0.024) (0.029) (0.037) SCORE_FUNDED * (0.0002) (0.0004) (0.0004) SCORE_NON-FUNDED (0.0001) (0.0001) (0.0002) lnemp t *** *** *** (0.008) (0.008) (0.011) (lnemp t0 ) *** 0.006*** 0.004** (0.001) (0.001) (0.002) D(PRO) (0.006) (0.006) (0.009) D(LARGE) (0.009) (0.010) (0.011) EU *** 0.052*** 0.041** (0.015) (0.015) (0.019) EU *** 0.062*** 0.050** (0.017) (0.018) (0.022) Intercept 0.095*** 0.092*** 0.112*** (0.021) (0.022) (0.030) R st stage, dependent variable: D(TREAT) D(ABOVE THRESH.) 0.457*** 0.490*** 0.472*** (0.026) (0.031) (0.042) SCORE_FUNDED 0.008*** 0.010*** 0.020*** (0.0004) (0.0004) (0.001) SCORE_NON-FUNDED *** *** *** (0.0001) (0.0003) (0.001) lnemp t (0.014) (0.014) (0.019) (lnemp t0 ) (0.003) (0.003) (0.004) D(PRO) (0.013) (0.013) (0.016) D(LARGE) (0.012) (0.021) (0.024) EU (0.023) (0.022) (0.024) EU *** 0.121*** 0.105*** (0.028) (0.028) (0.032) Intercept *** *** *** (0.036) (0.035) (0.044) R Note: All regressions include a set of time dummies and three technology field dummies. *** (**, *) indicate a significance level of 1% (5%, 10%). November 2014 Page 78 of 139

80 Conditional difference-in-difference estimation implemented via propensity score-matching A propensity score-matching method was implemented as a conditional difference-indifference estimator. This method is a non-parametric version of the difference-indifference estimation shown above. Instead of regressing the logarithmic employment growth on the treatment dummy and the peer review scores (and the other control variables), a nearest-neighbour matching was used to search for a control observation in the group of non-treated applicants, which is very similar in its characteristics to the treated firms. This means that for each subsidy recipient, its initial firm size (before the project start), the peer review score, and all other control variables were taken into account. Then the group of applicants was searched for a nearest neighbour, that is, an applicant who basically has the same characteristics, except that it did not get funding. In this way, the method tries to mirror an experimental setting where both firms had the same chance to get funding, but only one actually got it and the other one did not, based on statistically random criteria, or in other words, factors which were beyond the control of the applicants. Once such a matched control group has been formed, the successful applicants and selected, non-successful applicants having the same observable characteristics can be compared with respect to their average outcomes (here average, annual employment growth). Of course, the most important matching criterion in the Eurostars setting is the peerreview score, which strictly predicts whether the firm got a subsidy below the threshold value of 400 (i.e. probability of zero) and also has a high explanatory power above the threshold. The probability of getting a subsidy increases with the scores, but the score is not a perfect predictor, because of the limited availability of national budgets for the different project consortia. As further variables are also taken into account, a multidimensional matching problem is faced, and therefore propensity score-matching is applied. In a first step, the probability that a firm gets an Eurostars grant depending on the peer review score, the initial firm size, and all the other variables, was estimated This was done by applying a Probit model and by using its predicted probabilities as matching criterion, that is, to determine which non-funded applications can serve as nearest neighbour for the granted Eurostars projects. As the probability of receiving a Eurostars grant is zero if the peer-review score is below , these observations were omitted from the sample. Furthermore, in practice researchers limit the samples of treated and potential control observations to common support. Here, this means that treated firms with a high peer-review score from the sample of treated observations were dropped, as no close neighbour could be found for them. Inspecting the data shows that above a score of 520, almost no projects were rejected, so such observations on subsidised applications were discarded. The results of the probit regression model are shown in Table 6-6. It turns out that the peer review score is, as expected, the strongest predictor of a firm s success in the Eurostars programme. Besides the fact that firms in EU Member States are more likely to succeed with their subsidy application than those of non-eu countries, no other variable besides the score influences the subsidy decision. 57 As of cut-off 5 the threshold was set to 402 points of the overall score. November 2014 Page 79 of 139

81 Expert group Table 6-6: Probit regression on probability of subsidy receipt Variable Coef. Std. err. SCORE 0.021*** lnemp t (lnemp t0 ) D(PRO) D(LARGE) EU *** EU *** Intercept McFadden R Note: All regressions include a set of time dummies and three technology field dummies. *** (**, *) indicate a significance level of 1% (5%, 10%). After the regression was executed, the estimated probabilities could be obtained. Figure 6-17 shows the distribution of the funding probabilities before and after matching. Before the matching exercise, the distributions of the probability to receive funding differed considerably among the two groups of firms, which is of course not surprising; if they were not different, it would mean that the subsidy allocation is purely random. After the matching, the distribution of the probability to receive funding no longer differs between treated firms and the selected control observations. Thus for the treated firms, the appropriate nearest neighbours have been drawn from the pool of potential control observations. Figure 6-17: Distribution of propensity scores in treatment and control group before matching after matching Estimated Propensity Score Treatment Group Potential Control Group Estimated Propensity Score Treatment Group Selected control group In order to test whether the subsidy had any effect on the outcome variable of annual employment growth, one can compare the means of this variable and test if these significantly differ from each other. It was found that the estimated treatment effect amount to about 3.5 % and that this is positively significant (cf. Table 6-7). Note that the matching estimation constitutes the non-parametric version of the difference-indifference estimator using only the sample above the threshold, which yielded very similar results. November 2014 Page 80 of 139

82 % Expert group Table 6-7: Matching results Average, annual employment growth Mean (N=402) Std. err. Treatment group Matched control group Difference between treatment and matched control group (= treatment effect) t-test ** Note: ** indicate a 5% significance level. Standard error of t-test is corrected for sampling with replacement in the construction of the matched control group. The econometric results described are not free of limitations. Some potential caveats are discussed below. Project start- and end-dates A first caveat is that the actual project start and end dates are not known. ESE stores the dates stated in project applications in a database. Given the heterogeneous timeto-contract across EUREKA member countries, it may not be uncommon that the start date mentioned in the project application is not the actual start date of the project. Furthermore, firms may start projects at their own risk, that is, before a Eurostars decision is taken. To look for evidence of these phenomena, the online survey asked SMEs about project start-dates. These can be compared to the dates stored in the ESE database. For illustration, Figure 6-18 shows the discrepancies between the start-date given in the survey by the respondents, and the dates stored in the ESE database. Although it is unclear which values are more accurate (the survey Figure 6-18: Histogram of differences in project start dates in application and survey Difference of start date in application and survey Difference in months (bin width = 6) Source: Eurostars Online Survey (2014) or the application data), it makes clear that the treatment variable in the econometric study above may not be without measurement error with respect to timing. The results should not be crucially affected by this potential problem, though, as initial employment is measured one year before project start in the ESE application database, and the employment year used at the end of the observation period is If at all, the treatment effect may be underestimated, as initial employment may be measured too late in cases where the projects indeed started earlier than stated in the application. November 2014 Page 81 of 139

83 Ongoing versus completed projects Closely related to the fact that the actual start- and end-dates of the projects are not known exactly is the fact that it is somewhat unclear what the estimated employment effects measure in detail. On the one hand, employment might have increased on average because the firms got the Eurostars grant and hired additional personnel to implement the projects. On the other hand, additional hiring may have taken place because the projects were successfully implemented and, e.g., a new product was introduced to the market which resulted in an expansion of the production and sales units of the awardees. The former effect would refer to input additionality, an effect directly related to the financial benefit of Eurostars grant. The latter would refer to output additionality, benefits that are only accrued because of successful completion of the Eurostars project. The employment effects estimated above are a mixture of both aspects, and cannot be easily disentangled into input and output additionality. According to the online survey, about 56% of projects were completed at time of data collection (May 2014). When looking at the survey results only until cut-off 6, the data that are used for the econometric study above, about 81% were completed by May As the econometric analysis only takes into account employment numbers until the end of 2013, however, it becomes clear that the estimated effects are both a result of hiring because the Eurostars grant was awarded and also because some projects were already completed (successfully) and the firms expanded because of good project results, such as new products and services. Eurostars versus other funding Finally, it is also noteworthy that the reported effects are somewhat underestimated because of the nature of the control group, which is composed of firms that applied for Eurostars grants but were not successful. As Eurostars is not the only funding source for innovation projects, some firms may have been able to find other public funding or to undertake the project with other financial means. This was investigated in the survey. Among 7,014 non-funded project modules, 32% (2,218 project modules) were implemented by the applicants despite the fact that their Eurostars submission was not successful. These firms where asked which financial resources had been used to implement the project. Out of the 2,218 undertaken, but non-funded projects, most were implemented because the applicants mobilised their own, internal financial resources. Either more financial means were allocated to R&D in general (65% of the projects), or resources were shifted from other R&D projects (33%). In addition, however, it is reported that 44% got funding from public resources other than Eurostars, and 29% were successful with a later Eurostars application. 33% of the SMEs reported that they obtained other external funding (e.g. bank loans) for the project. Figure 6-19 illustrates these findings. November 2014 Page 82 of 139

84 Figure 6-19: Implementation of non-funded projects Source: Eurostars Online Survey (2014) These results show that the estimated treatment effect is possibly too low given that while Eurostars enables firms to implement a desired project, in the counterfactual situation this project is not at all implemented. As the survey results show, about onethird of rejected project modules were still undertaken. Searching for heterogeneous treatment effects In the analysis above, it was concluded that funded firms grow on average about 3.1% per year more in terms of employment because of the Eurostars grant. In order to uncover possible inefficiencies in the programme and thus to derive recommendations with regard to future budget allocation, it is instructive to screen the database for heterogeneous treatment effects. Several dimensions that could lead to a deviation of treatment effects from the estimated average treatment effect were considered, as explained below. Firm size: The average initial firm size of funded firms amounts to some 27 employees. The median firm size, however, is only 10 employees. The size distribution of awardees is thus skewed. Therefore, it was investigated whether the treatment effect varies significantly with the initial firm size: but no such evidence was found. Heterogeneity of consortia: As the Eurostars programme also allows for collaborations of SMEs with large firms and public research institutions including universities, it was checked if the estimated treatment effects vary with the composition of the consortia: but no evidence was found here either. Peer-review score and ranks per cut-off: It was also analysed whether the treatment effect increases or decreases systematically with the peer-review score and/or the ranks per cut-off. On the one hand, firms close to the quality threshold might crucially depend on the November 2014 Page 83 of 139

85 Eurostars subsidy and might not be able to achieve growth without the grant. Better-ranked firms might also find other channels to finance their innovation projects in case of unsuccessful applications. In this case, one would expect a decreasing treatment effect as the score or rank improves. On the other hand, it could be argued that the better the research proposal is ranked, the higher is the expected treatment effect. In that case, one would expect that the treatment effect systematically increases with the score or the rank. However, no such evidence was found. Monetary value of grant: In addition to the rank, the monetary value of the grant could play a role in the treatment effects. It could be hypothesised that a small grant might not have measurable effects on firm performance in any respect, but that very high grants might lead to inefficient spending patterns in the firm. Thus one would either expect that the estimated treatment effect increases with the size of the grant, or that an inverse U-shaped curve is obtained, meaning that if the grants are very high, funds are spent inefficiently without further performance benefits. However, neither of such effects turned out to be present in the data. Multiple grants: Related to the amount of the grant, it is often a concern in subsidy programmes that firms develop a routine to apply for grants regularly to source public money. Policy-makers fear that such firms may be subject to large crowding-out effects. It turns out that such concerns should not play a major role in the Eurostars programme. First, 87% of awardees received only a single grant within Eurostars. 45 firms received two grants, ten received three grants, and one firm got 4 grants. Second, the heterogeneous treatment effects analysis did not show lower treatment effects for firms that received multiple grants. In conclusion, no severe, significant inefficiencies could be identified with respect to the econometric treatment effects analysis Impact on innovative outputs and IPR Given the ambiguity about whether the employment effects are measuring the input or output additionality of the innovation process at the firm, the expert group also collected, additional to employment figures, patent data of Eurostars applicants. Patent applications are a measure of innovation output and thus an indicator of successfully conducted R&D activities, at least from the firm perspective. It can still be the case that many patents have a value close to zero, but at least the firms considered the results of their R&D projects worthwhile enough to apply for intellectual property rights in the form of patents. The patents were collected from the PATSTAT database. 58 This required a text-field search of the Eurostars firms and their addresses as patent applicants in the patent database. Patents were counted according to their priority dates, i.e. if a firm patented first in the UK, and then at a later point in time filed the same application at the 58 ESE also collects information about the creation of IP in the final project reports. However, final project reports are only submitted by firms that were funded. Thus no information on non-funded projects is collected, and these data cannot therefore be used for an econometric treatment effects analysis. November 2014 Page 84 of 139

86 European Patent Office (EPO), this patent family is counted as one patent. On this basis, the information collected accounts for the number of inventions and does not count the number of intellectual property rights (IPRS) one invention may have generated in different patent offices. The 2,711 SMEs that had projects starting before 2012 were again the starting point. Unlike for employment, these firms did not have to be found in the Amadeus database, as PATSTAT contains the population of all patents filed with more than 40 patent offices and all of 2,711 firms could be used for the analysis. If firms were not found in the patent database, their patent applications were set as zero. However, about 5% of the 2,711 firms were not used for the subsequent analysis, as the patent database contained records with very similarly written applicant names, so that it could not be unambiguously determined whether certain patents were filed by the applicants or by very similarly named firms (e.g. parent companies). These ambiguous cases are dropped. It was found that about half of the firms, 49%, have filed at least one patent since For the multivariate econometric analysis, the patent stocks of firms were considered. Patent stocks are the cumulative number of patent applications and are often considered as the IPR-protected knowledge stock of firms. When applying the difference-in-difference logic as before for the employment variable, the difference in patent stocks is equivalent to new patent applications filed between 2005 and On average, firms below the quality threshold filed five patent applications between 2005 and 2013 (1,362 observations). 483 firms were ranked above the quality threshold, but did not obtain funding. These filed on average 3.7 patent applications. There are 808 funded firms, and they filed 5.4 patent applications on average. In terms of methods, equivalent models as for employment were applied. For reasons of brevity, only the DID regressions using the subsample of firms above the quality threshold that include the score as control are shown. The other DID regressions yield similar results, however. In addition to linear, OLS regressions, Poisson count data regressions were estimated. In the case of patent application counts, such models are often preferred over OLS regressions, as the Poisson model takes into account the fact that the dependent variable is non-negative and takes integer values. Poisson count models are efficient estimators in this setting. Table 6-8 shows the results. According to the linear regression, the funded firms file on average 1.9 patents more than non-funded firms ranked above the threshold. According to the Poisson model, the corresponding treatment effect amounts to 42% more patent filings for the funded firms. When applying the Fuzzy RDD estimator, no significant results were found around the quality threshold value. The nearest-neighbour matching estimator yields similar results to the DID regressions, though. For 762 funded firms, nearest neighbours were drawn in the matching routine. As for the estimation of the employment effects, funded firms with a score above 520 were dropped as not sufficiently close neighbours, that is, non-funded firms with very high peer-review scores exist. The 762 funded firms filed on average 5.5 patents. The corresponding control group of nearest November 2014 Page 85 of 139

87 neighbours only filed 2.6 on average. Thus the estimated treatment effect using the matching estimator amounts to 2.9 patents, and is significant at the 1% level. These results should also be seen as preliminary treatment effects. As mentioned above, several of the projects in the sample are still ongoing, and thus not all envisaged patent applications may have been realised by the Eurostars beneficiaries. According to the online survey, about 23% of the funded firms stated that they have not yet filed a patent application but plan to do so in the next three years. 59 Table 6-8: Difference-in-difference regressions on change in patent stock: subsample of firms ranked above threshold (N = 1,291) Linear regression (OLS) Count data model (Poisson) Coefficient (Robust standard error) Coefficient (Robust standard error) D(TREAT) 1.911** 0.418** (0.810) (0.181) SCORE * (0.027)) (0.007) SCORE ** ** (0.0002) ( ) D(PRO) 1.233* 0.255* (0.720) (0.149) D(LARGE) (1.090) (0.188) EU (0.842) (0.227) EU (1.568) (0.306) Intercept (1.546) (0.367) R Note: All regressions include a set of time dummies and three technology field dummies. *** (**, *) indicate a significance level of 1% (5%, 10%) Behavioural Additionality: Impact on R&D Strategy, R&D Management, and R&D collaboration Beyond the direct effects of the Eurostars grant on employment and innovation (as measured by patents), it is also interesting to investigate whether other effects can be identified. Such less direct effects are often subsumed under the term behavioural additionality. Behavioural effects are often referred to as, for example, improved R&D management because of a subsidy receipt and the reporting duties that come along with the award. Furthermore, specific programme features could influence how the innovation process of a company is organised. Here the requirement of collaboration in Eurostars is such a feature. In addition, researchers often refer to so-called certification effects of public grants. The fact that the subsidy receipt is a result of a positive peer-review process is often seen as quality signal to potential investors, lenders and customers. 59 It would have also been interesting to analyse innovation inputs more directly through R&D spending or R&D employment. As the SME applicants are located in many different countries, it was not possible to collect such data, as no comprehensive pan-european database containing R&D expenditure and/or employment exists. November 2014 Page 86 of 139

88 Consequently, the expert group asked some questions about other dimensions of benefits beyond the monetary contribution of Eurostars. As the following can only be calculated for awarded projects and is based on self-assessments of the companies, the results should be interpreted with some care; these are no treatment effects derived from econometric, counterfactual estimations. The firms that had funded projects were asked how important the funding by Eurostars is with respect to the following dimensions: Easier access to external finance. Facilitated cooperation between project partners. Easier marketing of products due to acknowledgement by Eurostars / European Commission. Increased recognition and reputation at an international level. Improved project management skills. The results are shown in Figure About 50% of the awardees stated that the Eurostars grant had high or medium importance for getting better access to other external, financial resources. This could be interpreted as a certification effect, such that certain lenders place value on a Eurostars-approved research agenda. This finding points in the same direction as the fact that almost 80% of awardees indicated that the Eurostars grant had a high or medium importance with respect to increased recognition and reputation of the firm. Figure 6-20: Non-monetary benefits of Eurostars funding Source: Eurostars Online Survey (2014) With respect to firm-internal behavioural effects, the vast majority of awardees confirm that the Eurostars grant facilitated international cooperation among project partners. In addition, about 70% of the firms rated the importance of Eurostars with regard to improved management skills as medium or high. November 2014 Page 87 of 139

89 The least affirmative responses were given to the question on whether Eurostars contributed to easier marketing of products. Although about 12% of firms still rated the importance as high and about 37% as medium, this result may suggest that the benefits of Eurostars mainly accrue in the innovation process at the firm. Once a product is being introduced to the market, it may not, beyond having attributes that may have benefitted greatly from the Eurostars project, be much easier to be marketed because of the Eurostars grant. The latter, however, can also not be expected as clients will mostly value actual product features rather than how these were developed Overall Added Value for an R&D performing SME Finally, the overall added value of the Eurostars awards can be considered. This concerns effects generated because new products, services or processes were introduced to the market or the production process. As many projects are still ongoing and not yet finished, the description below partly reflect expectations of the awardees. Firms with funded projects were asked whether the project generated or is expected to generate an increased market share an increased profit margin decreased costs and whether it helped to reach new customers in general develop new geographical markets within Europe and outside of Europe. Figure 6-21 shows that the largest fraction of firms indicating Eurostars as highly important with respect to a certain impact can be seen for reaching new customers, with 50%. The high importance for developing geographically new markets is almost evenly distributed among markets within and outside Europe, with 28% and 25%, respectively. 32% of awardees rated the effect of Eurostars on increasing their market share as highly important, and another 36% indicated the importance as medium. These values are slightly lower with respect to profit margins. Thus Eurostars seems to help more with respect to enlarging the product portfolio rather than increasing profit margins per unit sold. This suggests that Eurostars contributes more to the growth of SMEs than to their profitability. Decreasing production costs is rated least important and especially least relevant by the awardees. This could be expected, as most projects will primarily involve product and/or service innovation. Process innovations may come along with product/service innovation, but are most likely not the primary goal of Eurostars applicants. November 2014 Page 88 of 139

90 Figure 6-21: Impact of the programme in award-winning firms Source: Eurostars Online Survey (2014) Summary of programme impact study In conclusion, the Eurostars programme clearly generated measureable effects on the recipient firms. By applying an econometric treatment effects analysis on employment growth, it can be shown that the fact of receiving a Eurostars grant leads to a 3% to 3.5% higher growth rate in the corresponding firm, on average. For firms close to the quality threshold, the regression discontinuity design methods even identify a higher growth rate, but this is not an estimate of the average effect of the Eurostars grant. The effect for projects that just received the treatment, i.e. firms being evaluated as just above the threshold, is possibly higher, as otherwise these firms would have had no other chance to implement their R&D projects. From the online survey, it became clear that for several firms that did not get the Eurostars grant they applied for, the Eurostars programme had non-negligible, positive effects for attracting other sources of finance including other public funding. Thus their qualification in Eurostars may serve as a signal with respect to the positive certification of the firms R&D for other financiers, even if the firms did not get the grant because the national Eurostars budgets were not sufficient to fund their projects in a certain cut-off. The expert group also searched for heterogeneous treatment effects in the employment growth analysis. This might have yielded indications of potential programme implementation inefficiencies, or in other words, projects that perform better than others. Any finding in these respects could have helped to optimise the design features of the programme. For instance, it was tested if certain compositions of consortia perform better or worse than others, e.g. the participation of large firms and/or public research institutions including universities. Furthermore, it was also investigated whether firms of a certain size benefit more of less than others, or whether the size of the grant plays an important role for the estimated effects. However, no significant heterogeneity in programme effects in any such dimension November 2014 Page 89 of 139

91 could be found. Thus it can be concluded that there is no urgent need to change any of the programme design features with respect to the granting process. An analysis of the firms patenting behaviour also revealed that Eurostars has positive effects on the successful implementation of R&D projects and thus also on the creation of intellectual property rights. The online survey makes clear that the magnitude of these findings should be considered as preliminary, as several of the projects in the sample are still not completed, and about a quarter of the awardees have not yet filed a patent application but intend to do so in the coming three years. There is also evidence of behavioural additionality from the Eurostars programme, as in the online survey, many firms stated that, among other effects, the Eurostars application made collaboration with project partners possible, and the firms improved their project management skills, for instance. According to the online survey, the Eurostars programme has important effects not only on employment and innovation, but also on firms eventual market performance. About half of the awardees stated that the Eurostars grant was highly important for reaching new customers with the firms products and services, and about a third of the firms reported a high importance of the Eurostars grant for increasing their market shares. 6.3 Programme-level effects Contribution of Eurostars to the European Research Area The EUREKA Strategic Roadmap identified the importance of assessing the impact of EUREKA projects regarding their added value for other EU initiatives, such as the European Research Area (ERA). As recommended in the EUREKA Impact Assessment , an Eurostars impact analysis, depending on when it would be carried out, could follow an econometric assessment, thus promoting a better understanding and contributing to an evidence-based analysis of Eurostars effects. According to interviews with the NPCs and NFBs, Eurostars fills an important gap in ERA on industrial R&D, as it is a unique programme in that it is market-oriented and bottom-up. The NPCs think that it fits well with the ERA objectives and that its impact on ERA is extremely positive. It is noted that Eurostars contributes to ERA objectives by promoting synchronisation and harmonisation of national funding programmes with Eurostars, and by pooling national financial resources in order to support the common goal of strengthening the competitiveness of the European economy by giving targeted support to R&Dperforming SMEs. Certain ERA principles, such as using independent experts and conducting an international evaluation of project applications, were also applied in Eurostars The document can be found here: afce96a5e167&groupid=10137 EUREKA Impact Assessment , Working Group Summary Report, Jerusalem, June 2011 November 2014 Page 90 of 139

92 The British NPC stated that they use Eurostars as an example in their ERA-NETs, where they define partnered projects in such a way that no one partner of a country can carry out more than 75% of the work. In the UK, in ERA-NETs, this approach is being adopted as best practice. Box 10: Impact assessment study According to the EUREKA HLG Minutes (Jerusalem, 22 June 2011), the first results of the global Impact Assessment Study were presented by Daniel Wasserteil, from Applied Economics, Ltd. The presentation included the study s final results and its recommendations, some of which were included in the Minutes. These are as follows: EUREKA should conduct impact assessment studies on a regular basis. These should be overseen by a permanent expert taskforce. Data collection from firms should be mandatory upon application and for three years after the end of the project. It should be a systematic, real time accumulation and coordinated at a single entry point, preferably the ESE. Full access to the Amadeus database should be acquired to help estimate the EUREKA effect for firms that participated in EUREKA projects up till The Amadeus database should be matched to patent data sources to improve preparticipation matching and the dimensions of the databases. Source: HLG/NPC Minutes, Jerusalem, 22 June 2011 Furthermore, France has proposed that a new programme targeting less R&D intensive companies ( innovative SMEs ) that represents a large portion of SMEs can be developed and implemented alongside Eurostars based on the experience gained in the contribution of Eurostars to the ERA Economic and social impacts of Eurostars A back-of-the-envelope calculation of total employment effects can be derived from the econometric treatment effects analysis. As the estimated treatment effect of Eurostars with respect to job creation amounts to a 3.1% higher average annual employment growth-rate when compared to the counterfactual situation of no Eurostars grant, these numbers can be used to derive a total programme effect. This exercise, however, is subject to a number of assumptions that cannot be verified. It should thus be interpreted with care: The sample of the employment regressions contained 435 observations on SME awardees. According to the name disambiguation routine carried out, 1,406 different SMEs were subsidised within the Eurostars programme. This number includes projects that were committed by January 2014, but not necessarily started yet, and it accounts for the fact that some firms received multiple grants. For the subsequent exercise, it was assumed that the 435 observations in the sample are representative of the 1,406 SMEs that were funded or where funds were committed. The initial firm size before the grant receipt amounts to employees in the sample. It is assumed that this average employment of the 435 firms in the sample is representative of the 1,406 firms in the Eurostars population of awardees. November 2014 Page 91 of 139

93 For firms in the sample, on average 6.05 years elapsed between observing initial firm size and the firm size at the end of the current observation period. The following calculation now assumes a scenario where this is the case for all projects. Thus it makes the assumption that the projects that are currently not in the sample, for instance because they only started very recently, are also evaluated after a similar time-frame and that future economic conditions would lead to the same treatment effects estimations as the current econometric application. Created jobs in the average firm are obtained by calculating the initial employment multiplied by the growth factor of to the power of 6.05 elapsed years minus the initial employment. By multiplying this number by the 1,406 SMEs that received a grant, the total employment creation of the programme is obtained. Total employment creation is then: Employment Creation = (27.34 * ( ) 6.05 amounts to approximately 7798 jobs created ) * 1406 = 7798 which The back-of-the-envelope nature of this calculation means that the final number of 7798 jobs should only be taken as a rough approximation, and that it is based on strong assumptions; among others, that projects only started recently, and also that ongoing projects will have similar employment effects as the projects in the sample. It should also be understood that the current observation is just a snapshot of jobs created in sampled firms until It is by no means clear that these are all jobs that are created by marketing new products and services. The number could rise as time elapses and the new products reach established market positions. The number could also shrink if the jobs were only created during the innovation phase but are not sustainable in the production and marketing phase of the products and services, or if some products and services turn out to be unsuccessful on the market. To show how much sampling variation can affect this number, the back-of-theenvelope calculation can also be carried out within the bounds of the 90% confidence interval of the estimated treatment effect. The 3.1% average treatment effect on the treated is a point estimate. This coefficient is significantly different from zero. However, if one takes the confidence interval into account, one would, rather than the point estimate of 3.1%, state with a confidence of 90% that the actual population parameter lies in the interval between 1.3% and 4.9%. Using these confidence bounds for the back-of-the-envelope extrapolation from the sample to the 1,406 project modules carried out by SMEs, the following total employment effects are obtained: at the lower bound of 1.3% additional annual growth because of the Eurostars grant, the total employment creation would amount to about 3,124 jobs; while at the upper bound of 4.9%, the created jobs would approximately total 12,902. Thus it becomes apparent that the 7798 jobs derived from the point estimate of the treatment effect should be interpreted with caution when concluding on overall programme-level effects. A more precise estimate could be established after more projects are completed and more time has elapsed Sustainability of Eurostars The NPCs and NFPs consider that both the synchronisation of national funding procedures and the harmonisation of funding rules will have a lasting effect. Similarly, November 2014 Page 92 of 139

94 the exchange of knowledge and experiences between NFBs will also have a lasting effect in programme management practices in Europe. Finally, Eurostar supported new partnerships between SMEs in Europe, which 90% of SMEs expect to be continued after the project and even more that will be continued even if funding was not gained. Figure 6-22: Share of SMEs which will likely reapply for Eurostars An indication of the sustainability of the programme is also the finding that the majority of SMEs will likely reapply for Eurostars. At the level of projects and beneficiaries, the NCPs think that there is a need for complementary mechanisms to support the commercialisation of project results through coaching and finance. One proposal is to create another scheme, with similar funding principles to Eurostars, to support the Source: Eurostars Online Survey (2014) commercialisation phase of successful projects. Another suggestion is to make available the Access to Risk Finance facility in Horizon 2020 for projects supported by Eurostars. According to the NPCs and NFBs, Eurostars projects were managed reasonably efficiently by the participants, and the relatively light reporting requirements under the programme did not put excessive pressure on SMEs. The SMEs interviewed also considered that the workload for reporting and monitoring was appropriate, although some identified the elimination of double reporting as a necessity (for NFB and ESE). Furthermore, the cost statement requirements, and especially time sheets, were considered difficult to adopt by a small company due to scarce human resources and inadequate information systems tools. It was also indicated by some SMEs that reporting periods should be clearly documented, perhaps directly in the contract. On the side of SMEs, Eurostars is seen as an enabler for gaining competitive advantages and sustaining business development. A view from a participant company expresses this vision, stating that due to Eurostars project results, we will have a chance to compete in the drug development market for the new interferon-free hepatitis C virus therapy, whose size is estimated to be 20 billion USD. 6.4 Impact of financial and economic crisis The financial and economic crisis 62 hit participating countries just when Eurostars was started in It brought to an end a period of rapid expansion of public and private 62 The financial crisis, which started in 2007 in the USA in its banking sector, triggered the debt crisis in Europe. European banks that had invested heavily in the American mortgage market were hit hard. In an attempt to stop some banks from failing, governments came to the rescue in many EU countries, such as Germany, France, the UK, Ireland, Denmark, the Netherlands and Belgium. But the cost of bailing out the banks proved very high. As Europe slipped into recession in 2009, a problem that started in the banks began to affect governments more and more, as markets believed that some countries could not afford to rescue banks in trouble. As a response to the crisis, European countries have pulled together to create the world s biggest financial assistance funds. By working together, the European Commission, the International Monetary Fund and the European Central Bank, help governments in need November 2014 Page 93 of 139

95 R&D spending in Europe. The public R&D systems in transition in the new Member States and the R&D systems in Southern Europe, which were on a convergence track, were particularly shocked by the crisis, which affected both private-sector R&D spending as well as public-sector support for private R&D. At the EU level, the average growth rate of R&D in the private sector (BERD) 63 between 2008 and 2012 was 2.9%. The growth rate of nominal R&D expenditures of the public sector 64 also amounted to 2.9% for this period. However, the average growth rate for government budget appropriations and outlays for R&D (GBAORD) for 2008 to 2012 for EU Member States amounted to 0.29%, and for only 0.15%. Government budgets for R&D varied dramatically during the crisis, as can be seen in Figure 6-23 which depicts the inflation-adjusted budgets of governments in countries participating in Eurostars. In some participating countries, government R&D budgets shrank dramatically, while others countries were able to maintain further R&D growth. Looking at public R&D investments, one of the striking features was the increasing gap between Scandinavia and some central European countries on the one hand, and southern EU Member States on the other. A drop in public R&D was particularly significant for countries like Spain and Greece, where the consolidation of government budgets meant that R&D expenditures had to be cut, whereas Switzerland, Estonia, Poland, Slovakia and Turkey experienced significant increases in government R&D budgets. Private R&D spending was also affected by the crisis. Especially SMEs investment and R&D suffered in particular, as financial restrictions became binding and expectations of future growth dropped. However, the overall development of government R&D spending did not translate into a reduction of the budget available for Eurostars. Significant reductions in the budget spent for Eurostars were observable in Italy, Greece, Cyprus, Bulgaria, and Ireland, but also for Finland or Slovakia (see Figure 6-10). Although there is always a need for private co-funding of Eurostars projects, the number of applications declined only in Cyprus, Greece, Portugal and Hungary, but was probably not affected in countries like Italy and Spain (see Table 9-3). Interviews with the NPCs and NFBs indicate that the financial and economic crisisin Europe had a negative effect on national funding for Eurostars in several countries. The number of applicants with funding contracts signed decreased considerably due to a lack of funds, and for this reason, some countries (e.g. Greece or Italy) did not participate in the last calls of Eurostars to devise assistance programmes to stabilise fragile economies and address deep-rooted economic problems. (Source: The Financial and Economic Crisis: happen/index_en.htm) BERD = Business Enterprise Expenditure on R&D. Based on the sum of R&D spending of the government sector and the higher education sector. November 2014 Page 94 of 139

96 Figure 6-23: Public budgets for R&D in countries participating in Eurostars Note: Switzerland Source: OECD: Eurostat (2014) NFBs that did not suffer from a shortage of funds (such as those in Austria, Finland, Flanders and Turkey) also reported that they were negatively affected by the crisis, since the SMEs they support partnered with their counterparts in countries having funding difficulties. Among the NPCs which reported a negative effect, Hungary mentioned that it had problems in securing the earmarked budget, and hence contracts could only be concluded with the Hungarian project partners after significant delays. Hungary also had to reorganise its national support programme due to the crisis, and this created difficulties in the support processes of Eurostars projects. On the other hand, the UK reported that the crisis did not have a negative influence on their Eurostars support budget and the number of projects was actually increased in the country. The financial situation in many SMEs contributed to their cyclical innovative behaviour, 65 accompanying an overall investment decrease in countries affected by 65 In an academic paper published in 2011, the authors argue that the economic crisis affected innovation not only because firms could not expand their investment in innovation but also because a significant number of firms have even decreased it. It is also stated that this situation has not the same expression in all Member States, affecting mostly modest or moderate innovator countries or those countries which recently adhered to the EU. As stated by the authors, (...) firms which exhibit a cyclical behaviour are the major innovators. In fact, firms are more likely to reduce investment in response to the crisis are characterised by (i) larger innovation intensity (in terms of share of turnover invested in innovation); (ii) larger size. The crisis also highlighted major differences in national innovation systems and their robustness, urging for more integration and the need for the development of collaborative initiatives at the European level not only in good times but especially in bad times, as the authors admitted. November 2014 Page 95 of 139

97 the crisis 66. SMEs had thus also decreased or maintained their budgets dedicated to R&D, following a more cautious approach. In this uncertain environment, in some countries SMEs suffered severe difficulties to allocate resources to R&D, which reduced not only their possibilities to follow a more proactive innovation strategy, but also their growth. As illustrated in Table 9-3 in the Annex, the number of applicants from the most affected countries, namely those under EU-IMF financial assistance, shows a decrease, as is the case in Ireland, Portugal and Greece. An illustrative case from an SME participant in Eurostars highlights that the Eurostars project helped to change its funding situation. The credibility and reputation of the programme helped to restart a negotiation process, with its shareholders, guaranteeing a more robust financial structure. 6.5 Uptake of recommendations from Interim Evaluation Report The Interim Evaluation Report issued 24 recommendations to be implemented by 2013 and beyond. The uptake of the recommendations by the stakeholders was studied using data sources and methodologies described in Chapter 5. The findings are presented below under each of the numbered Interim Evaluation Report s recommendations. Policy recommendations beyond 2013 (1) The Eurostars Joint Programme should be continued beyond (2) In the next edition of Eurostars programme, European Commission should prepare a proposal to expand the budget, using flexibility clauses, to respond to the increasing demand. The implementation of these two recommendations was realised in the Commission s Proposal for a Decision of the European Parliament and of the Council on the launch of Eurostars 2 for the period of The maximum financial contribution of the EU to Eurostars-2 will be EUR 287 million. The EU contribution to the projects will be one-third of the total. Two-thirds of the contributions should come from public and private sources of participating countries. (Source: Archibugi, D and Filipetti, A., Innovation in Times of Crisis, National Systems of Innovation, Structure and Demand, Research Policy (40), 2011) 66 In a study about the EU IMF assistance, it is admitted that in case of Portugal: The main problem is that, so far, the programme has not succeeded in boosting investment. On the contrary, investment performance has been extremely disappointing with a contraction of more than 10 % in 2011 and again in 2012, and a further contraction of 8 % now expected in Portugal s investment performance is far worse than Ireland and only slightly better than Greece. It is also far worse than anticipated by the programme, with an accumulated loss of investment growth compared to expectation of nearly 19 %age points for the period (Source: Ferry, J., Sapir, André and Wolff, G., EU-IMF assistance to euro-area countries: an early assessment, Bruegel Blueprint Series, 2013) 67 Decision No 553/2014/EU of the European Parliament and of the Council of 15 May 2014 on the participation of the Union in a Research and Development Programme jointly undertaken by several Member States aimed at supporting research and development performing small and medium-sized enterprises. The Decision can be found on: November 2014 Page 96 of 139

98 The average cost of each Eurostars-2 project is estimated to be EUR 1.4 million. With an average public funding share of EUR 0.56 million (40%) per project and an overall programme budget of EUR million (EUR 287 million EU contribution plus EUR 861 million from national sources), about 2,050 projects can be funded, as compared to 783 funded projects in Policy recommendations until 2013 (3) European Commission and the Member States should provide sufficient funds to make it possible to fund a greater share of projects that meet the Eurostars threshold criteria. Thirteen of the 33 participating countries markedly increased their funding from their originally earmarked volumes to a total of EUR 139 million of additional funds. Seventeen countries committed less than the amount they had originally earmarked, and 3 countries committed no funds (Table 6-2). In spite of the increased resources, a greater share of eligible projects could not be funded, due to the increase in the number of applications and projects meeting the criteria. The share of the funded projects above threshold declined after cut-off 3 and evened out to 60%, since cutoff 5. (4) EUREKA Secretariat and NPCs should seek to increase its reach among SMEs that do not possess pre-existing international links. The vast majority of the applicant SMEs had pre-existing international links (Chapter 6.1.1). No specific efforts by the ESE or the NPCs to increase the reach amongst SMEs lacking international links could be identified. (5) EUREKA Secretariat, in close collaboration with NPCs, should increase the number of technical experts in general and obtain a good balance between participating countries in terms of number of experts relative to the number of participants from a given country. The ESE has been able to increase the number of technical experts from 485 in 2010 to 1,736 in 2013 (see Figure 6-12). Some of the interviewed NPCs stated that they had not been invited to cooperate in expanding the number of technical experts (Chapter 6.1.4). The progress towards geographical balance of the technical experts has been slow, as there are still many countries from which no or very few technical experts have been recruited (see Figure 6-12). (6) EUREKA Secretariat should establish procedures to ensure that there is a sufficient number of technical experts with good market knowledge. No direct evidence for an increase of experts with market knowledge could be found. However, the share of experts from industry rose from 26% in 2010 to 38% in 2013 (Chapter 6.1.5). Amongst these, market knowledge is assumed to be higher than amongst scientific experts. November 2014 Page 97 of 139

99 (7) When increasing the number of technical experts, EUREKA Secretariat should pay close attention to maintaining a high level of expertise amongst these. The majority of SMEs interviewed and all interviewed NPCs were satisfied with the level of expertise of the central evaluation (Chapter 6.1.5). The IEP members have the opportunity to monitor and comment on the quality of the technical experts. The ESE recruited an Expert Database Officer in (8) The Eurostars IEP should be allocated more time so as to be able to analyse problematic applications in greater detail and possibly refine some of its judgments. This is recommendable particularly as the number of applications increases. This recommendation has been implemented by increasing the number of IEP members from 6 to 11, and by organising bilateral IEP meetings as platforms for tackling problematic applications, prior to the plenary meeting where all projects are ranked. (9) When the Eurostars IEP overrules the technical experts scores, there should always be a comprehensive and transparent explanation of the reasons for this. This explanation should be fed back to technical experts. This recommendation has been implemented through a process where the IEP members produce, in bilateral meetings, a consensus report for each project s rating, commenting each criterion and score to provide a comprehensive and transparent explanation of their judgement. The consensus reports are provided to the HLG together with the ranking list and the minutes of the IEP meeting. (10) EUREKA Secretariat should ensure that systematic feedback is always given to technical experts, regardless of whether or not application scores are changed by the Eurostars IEP. This recommendation has been implemented, as at the end of each evaluation phase the ESE sends a debriefing of the results to all experts of the individual projects they have evaluated, with the information on the position of those projects in the ranking list and on their performance relative to the acceptance threshold. No evidence was found on whether or not the individual projects consensus reports with the explanations referred to under Recommendation 9 are provided to the technical experts. (11) The voluntary Working Group for good practice dissemination should be made a permanent fixture of Eurostars implementation, and all participating countries should be encouraged to participate in it. The NPC meeting in 2011 granted a formal mandate to the Best Practice Working Group, which has since then held regular meetings to identify and disseminate best practices and promote harmonisation and synchronisation at national level. However, no evidence was found about the implementation of best practices. November 2014 Page 98 of 139

100 (12) To expedite time-to-contract, participating countries should start the national evaluation phase (e.g. financial evaluation) well before the communication of the evaluation rankings. The ESE and NFBs have improved evaluation procedures, mainly through the introduction of a financial eligibility check and communication with applicants. Nevertheless, there is no evidence on national evaluation duration and their relation or impact on the time-to-contract. (13) The EUREKA Secretariat should initiate discussions with those countries that have persistently slow processes (in terms of time-to-contract) with a view of bringing their performance closer to the median performance among all countries. Implementation of the recommendation is on-going, as the ESE, NFBs and NPCs have been able to shorten the average time-to-contract from a median of 369 days (mean: 435 days) in cut-off 3 to a median of 257 days (mean: 282 days) in cut-off 8 (Chapter 6.1.4). However, it took more than 420 days to sign 10% of the contracts in cut-off 8. (14) There should be a global deadline for the signature of Grant Agreements. The failure of any given project to meet this deadline should mean that the project will not be funded, and the unallocated funds will be allocated to the highest ranked unfunded project in the ranking list. This recommendation has not been implemented. (15) The rules for providing feedback to project applicants should be further harmonized across countries, with an emphasis on high-quality feedback and with strict deadlines imposed for providing the feedback. Some NPCs felt that there was still room to improve the quality of the feedback to the applicants, and demanded more information from the central evaluation (Chapter 0). In addition, some SMEs (especially those which did not receive funding) criticised the quality and quantity of feedback from their NPCs. (16) Harmonization of national procedures should be strengthened further by EUREKA Secretariat and European Commission, and specific, measurable milestones should be agreed by HLG to facilitate this process. This recommendation is only starting to be implemented, since harmonisation of funding rules persists as a major challenge, in spite of the progress that has been achieved in some countries. In recent HLG meetings, the need to avoid double auditing, to guarantee simplicity and lean administration, and to reduce the assessment time of financial viability to accelerate the funding synchronisation processes, was expressed 56. The interviewed NPCs requested increased involvement of the HLG in Eurostars governance (Chapter 6.1.5). November 2014 Page 99 of 139

101 (17) Eurostars eligibility criteria should be universally established as the necessary and sufficient condition for any participating project to be funded in any participating country. No participating country should be allowed to impose eligibility conditions that go beyond the Eurostars eligibility criteria. This recommendation has been implemented (Chapter 6.1.4). (18) Dual reporting arising due to the use of different languages should be avoided. Full harmonisation has not been achieved, as many countries still demand national forms to be filled in their national language (Chapter 6.1.5). (19) The EUREKA HLG should assume an active role in monitoring the timeto-contract and in ensuring that countries that lag behind streamline their processes with Eurostars so as to avoid project delays. The recommendation appears not to have been implemented, since the HLG s involvement in Eurostars governance was felt to be insufficient by the NPCs (see Chapter 6.1.5). (20) The remaining bilateral agreements should be signed as a matter of priority. The recommendation has been implemented, as a total of 35 bilateral agreements with the NFBs have been signed. (21) Eurostars Joint Programme should continue to be organized around a Virtual Common Pot principle. The programme continues to be organised around a virtual common pot. (22) However, it is also recommended that Eurostars Joint Programme experiments with the approach of allocating 10% of total earmarked funds to a Real Common Pot, which would be used to finance highly ranked projects that risk not getting funded because of country-level complications. This recommendation has not been implemented. (23) EUREKA Secretariat should initiate an activity intended to explore the integration of EUREKA cluster initiatives in view of learning and implementing integration lessons that could be applied to promote the integration of Eurostars with national R&D support initiatives. No evidence for implementation of this recommendation was found. (24) The post-hoc evaluation of the Eurostars programme, to be carried out in 2013, should emphasize quantitative output metrics to measure firmand programme-level outcomes (the latter both nationally and centrally). The evaluation should use appropriate statistical methods to control self-selection bias and examine both input, output and behavioural additionality effects at the firm level. This report constitutes the implementation of this recommendation. November 2014 Page 100 of 139

102 In summary, both the Interim Evaluation Report s policy recommendations beyond 2013 have been implemented, as well as the recommendation on conducting the Final Evaluation (numbers 1,2 and 24). The uptake of the remaining 21 policy recommendations until 2013 falls into three categories, as follows: Seven recommendations have been fully implemented (numbers 7, 8, 9 10, 17, 20 and 21). Ten recommendations have started to be implemented, have been partly implemented, or implementation is on-going (numbers 3, 4, 5, 6, 11, 12, 13, 15, 16 and 18). Four recommendations have not been implemented (numbers 14, 19, 22 and 23). 6.6 Summary of findings This section of the report summarizes the main findings. The summary is organised along the subthemes target group and scope of the programme, governance, management and operations, funding, and outcomes and impacts of Eurostars. Target group and scope of the programme According to the participating SMEs and the governing and administrative bodies of Eurostars, the programme is relevant to the needs and objectives of the target group. The high level of demand from SMEs is considered as an indication of this strategic fit. Almost all participating SMEs plan to apply to Eurostars again, as the programme corresponds to their needs and objectives. Research performing SMEs are the overwhelming majority of participants, and received about 75% of the money spent by Eurostars. In addition, the programme was successful in addressing young and small and micro firms. Participating SMEs already have experience in national and international funding programmes, and the majority already have pre-existing international links. Most probably, it is because of this experience that the programme documents are found easy to understand by SMEs, and project proposals are prepared without almost any external assistance. However, it is found that applications which involve university and/or PROs have a higher chance of being funded. This may be because the presence of a university or public research organisation signals the high quality of the proposal, or that those organisations are more experienced in preparing application documents, or that those applications provided the best balance between scientific excellence and market knowledge. Based on its bottom-up nature, Eurostars is able to address a wide-variety of innovative technologies, ranging from ICT and its applications, and medical biotechnology, to (renewable) energy and steering and control technologies. Governance In general, ESE is considered a successful Dedicated Implementation Structure. However there is a need to improve managerial and administrative practices, November 2014 Page 101 of 139

103 enhance the qualifications of staff, develop and implement a marketing and promotion strategy, and develop and maintain a well-structured information management system and database for the use of ESE and the other governing and administrative bodies of Eurostars. The governance system of Eurostars functions properly and the interaction between the national and central administrative bodies is effective. However, the international dimension of Eurostars required a complex governance system to be set up. In HLG meetings, various issues critical to the success of Eurostars are regularly discussed without reaching either a consensus for action or rules binding for all countries. Synchronisation and harmonisation of funding, eligibility rules and the adequacy of national budgets were intensely analysed and discussed. However, a consensus was reached on the number of cut-offs per year and improved information flows between the ESE and NFBs. There is a need to increase the ownership and involvement of HLG towards Eurostars. This is seen as critical in order to facilitate synchronisation and harmonisation. Similarly, NFBs involvement remains limited in the governance structure; they should be more intensely involved to achieve a better synchronisation and harmonisation. Eurostars strengthened the SME component within EUREKA and contributed to a much higher visibility of the contribution of SMEs to European competitiveness and innovation. In this way, Eurostars had a significant and long-lasting positive effect on the EUREKA network. Management and Operations The virtual common pot worked well and is the preferred mechanism, since it attracts countries to contribute to Eurostars. Based on the success of the programme, several countries increased their shares significantly beyond the initially agreed level. However, the full benefits of the virtual common pot have not yet been exploited in Eurostars, since some of the countries, in particular Germany (despite an enormous increase of funds while the programme was running) and Spain, do not allocate a sufficient level of funding for the programme. Because of this, some of the good projects selected drop out of the ranking list, which in turn leads to frustration among applicant SMEs. Synchronisation of funding is still the most important challenge facing Eurostars, although some improvements have been achieved. In some cases, participating SMEs face great difficulties because of the longer time-tocontract. There are also parallel application and evaluation requirements as well as dual reporting obligations applied by some of the NFBs. The other major challenge of Eurostars is the harmonisation of funding rules. The representatives of HLG think that this challenge should be addressed through a top-down process with an initiative by national ministries and the EU. The NPCs think that full harmonisation is not possible (for example they cannot change the percentage of funding and the upper limit for finance per SME). Another issue is the need for a better harmonisation of subcontracting rules for universities and PROs. November 2014 Page 102 of 139

104 The process of central evaluation in Eurostars worked well. However, there is a need to improve the transparency and feedback mechanism and to streamline the process in order to decrease the time from application to the availability of IEP evaluation. The time-to-contract, e.g. the time span from application to the signature of the funding contract between participants and NBFs, was rather long at the start of the programme, but went down considerably during the first phase of Eurostars. However, the time-to-contract is fairly long in a significant number of countries. Improvement of the time-to-contract for the programme is urgently needed, as the existing large difference between participating countries calls the whole system into question. Although the number of technical experts as well as those with market experience has increased considerably since 2012, there is still room to improve geographical and gender balance. Also, the NPCs would like to be involved in the process of improving the experts database. SMEs are satisfied with the level of information received about the evaluation of their projects. The reporting workload and the monitoring process in Eurostars are found appropriate by them. They also find the evaluation process clear, transparent and timely. Funding The auditing is considered by NPCs as well as NBFs as harder in Eurostars compared to other EU programmes. NBF auditing within the framework of Eurostars should be abolished and substituted by the use of reports by national independent audit offices as they already audit the NFBs. One of the indicators of the success of Eurostars was the significant increase in the number of applications and hence the increasing demand for public funding by this programme. Hence the share of submitted applications funded declined throughout the lifetime of Eurostars. Similarly, the share of budget demanded by approved projects in relation to the budget demanded by submitted projects went down from 42% in cut-off 1 to 18% in cut-off 10. The relation between the number participants (members of consortia of approved projects) and the number of applicants in submitting consortia moved, in a similar fashion, from 42% to 17%. The increase of the number of applications is especially remarkable when the number of applications per cut-off is mapped onto the number of applications per year. As national budgets is allocated per year, then cut-off 1 correspond to 2008, cut-off 2 to 2009, cut-off 3 and 4 to 2010, cut-off 5 and 6 to 2011, cutoff 7 and 8 to 2012, and cut-off 9 and 10 to This resulted in 90 applications for 2008, 90 for 2009, 149 for 2010, 145 for 2011, 139 for 2012 and 179 for This is an impressive increase in the number of applications per year. As a consequence, the share of projects meeting the quality threshold which were approved for funding declined from 68% for cut-off 1 to 55% for cut-off 10; i.e., an increasing share of high-quality projects were not funded. However, November 2014 Page 103 of 139

105 in terms of budget the slowdown between cut-off 1 and cut-off 10 was less dramatic, going down from 62% to 54%. The size of the national budgets for Eurostars still differs enormously. This refers to both the relative size of the country and the development stage of their national public funding system, and the demand from national SMEs for Eurostars funding. This difference had a significant negative impact on the functioning of the virtual common pot system. The size of funding per participant is judged as adequate by most participants and programme administrators. The financial and economic crisis in Europe had a negative effect on national funding for Eurostars in the majority of countries. The number of projects above threshold decreased considerably due to lack of funds, and for this reason some countries did not participate in the last calls of Eurostars. The countries which did not have shortage of funds have also reported that they were affected negatively by the crisis, since the SMEs they support partnered with their counterparts in countries having funding difficulties. Outcomes and impacts of Eurostars The EU funding has a leverage effect on Eurostars in a way that stimulated national policy-makers to have clear goals and allocate enough funds for the programme. It also created a strong marketing effect towards potential beneficiaries by increasing the prestige and attractiveness of the programme to SMEs. Eurostars stimulated new partnerships of young and small SMEs between participating countries. It is foreseen that nearly all consortia will result in intensive R&D links between firms that will survive in the future without public funding. The results pointed to some long-lasting impacts of the programme, as about 90% of the firms intended to continue the initiated partnerships with other R&D-performing SMEs in Europe whether the consortium was funded or not. Eurostars contributed to employment generation by R&D-performing SMEs in Europe. The causal impact of Eurostars funding on employment growth was calculated using different regression designs analysing the growth-rate differentials between funded and similar but non-funded firms. This differencein-difference estimation is a rigorous way to test for causal programme effects between funded and similar non-funded applicants to the same programme. These results are fairly robust against changes in the underlying assumptions of the evaluation model. The econometric evaluation approach used was tested for heterogeneity of the size of treatment effect with respect to the size of funded SMEs measured in terms of employment, the composition of consortia, the rank of an application in the ex-ante evaluation by technical experts, the size of the Eurostars grant, or whether the SMEs applied and / or received funding for more than one Eurostars project. In addition, the matching approach implies that the employment growth rate of funded firms was nearly twice the employment growth of observably equal non-funded firms. November 2014 Page 104 of 139

106 The identified differentials in growth rates are even more striking as there were hints that the programme still has not reached its full effect, as projects need more time to develop. Eurostars also had a significant positive impact on the technological potential of funded firms. The estimated number of patent applications by funded firms exceeded the number of patent application of non-funded programme applicants. November 2014 Page 105 of 139

107 7 Assessment of findings Eurostars is a unique programme to support bottom-up, near-market-research of SMEs in participating countries. Overall, the results of this evaluation provide evidence for the success of the programme and the worthwhile nature of the public money from the EU and the participating countries. The programme contributed to employment generation, improved the competitive position of participants in the market place and contributed to the development and roll-out of new and improved products and services. There is room for improvement in the implementation of the programme so that the positive overall impacts of the programme will become even larger in the future. This chapter assesses the findings of this review under the same subthemes as in Chapter 6.6, Summary of findings: Target group and scope of the programme, Governance, Management and operation, Funding, Outcomes and impacts of the programme. 7.1 Target group and scope of the programme Eurostar fully reached its target group of R&D performing SMEs. The overwhelming majority of funds was allocated to SMEs. 65% of participants were R&D performing SMEs and another 5% were other SMEs. Even more remarkable is the significant participation of young R&D performing SME as well as R&D performing SMEs which were still micro-firms. Hence, Eurostars was successful in attracting a quite unique target group which was not really present in some countries public innovation funding. This was also evident from the fact that a quite significant share of projects re-applied when the proposal was not funded. On the other hand the programme captures to a large extent R&D performing SMEs experienced in international collaboration, including SMEs which had participated in EU framework programmes for research. So, not surprisingly these SMEs were able to rely on their existing networks and had the experience and capabilities so that they do not need additional support for third parties or parties beyond the applying consortia in drafting and developing the project proposal. However, partners like universities and (applied) public research organisation were instrumental in developing the proposal and in assisting R&D performing SMEs in the R&D phase of the project. Unfortunately, some participating countries do not support university and/or research institutes directly but only via a subcontract with national SMEs. This sometimes put a limit on the direct access of foreign SMEs to those research institutions and hence on reaping the full fruits from a growing ERA. However one should keep in mind that one of the Eurostars strength is the way in which it opens up the choice of international partnerships by SMEs. SMEs used this rather flexibly by selecting their partner primarily to augment their own strength by selecting partners which they already know either from previous business contracts or for the technological reputation of their partners in their community. Hence, Eurostars had a large impact on technology and knowledge transfer between firms of different sizes in Europe as it allows also the possibility to work together with large firms. It also stimulated public-private partnership between the private sector and public sector November 2014 Page 106 of 139

108 research exploiting the knowledge and experience of universities and/or public research institutions. A second important feature of Eurostars was its bottom-up character which did not provide restrictions on the technology space in which SMEs innovate. In line with the traditional technology orientation of EUREKA, ICT was the dominating technology. However, in the course of the projects, other technology gained an increasing share. The expert group welcome this shifting focus and the wider technology space reached by Eurostars projects. In particular, biotechnology projects often demand longer periods to develop before a new product or process is ready for the market. The expert group is convinced that Eurostars delivered a visible boost to SME s innovation potential in Europe and provides significant incentives for SMEs to develop new or advanced products, process, or services. But one should be aware that the funding phase only presents one phase in the life time of an innovation. Hence, there is a need for a programme that stimulates near-market-research to support the commercialization of that product in the market place beyond financial means. The expert group sees also opportunities for supporting successful SME participants by gaining additional support or follow up support from the private sector as the resources demanded by those phases of developing the market for new products and services are beyond the size of R&D which are usual provided by government but also because it is less obvious that severe market failure in these phase is present or can be overcome by government invention. Hence, the expert group believes that Eurostars should be granted the opportunity to fully use follow up support from other national or EU instruments, including financial support from EIF or EIB or EUREKA instruments. Such support might go beyond financial support and include access to platforms or access to contract networks like EEN to spread information of their business model and the outcome of their Eurostars project. 7.2 Governance Overall, the expert group is convinced that the governance structures implemented for Eurostars function appropriately. Eurostars exploited the knowledge, experience and resources of programme administrators and programme coordinators of different levels of government in Europe. The programme is rather demanding with regard to the amount of coordination effort. The Eurostars approach presents a reasonable setup by centralizing the technological selection process whilst allocating the detailed funding decision to the level of participating countries. Hence, the selection of projects reaps the fruits of technological competition as well as competition between SME s business ideas across Europe, at a European level where potentially the best ideas will win and the programme will select the best innovation ideas, which in turn generates opportunities for worldwide competitiveness. However, the exploitation of competition at European level requires coordination of programme implementation and administration across participating countries and hence needs scare resources from a variety of government bodies which makes coordination rather cumbersome. In addition, Eurostars provide a good opportunity for cross-country learning of best practice to stimulate and support R&D in SMEs. Thus, the programme provides a unique platform for transferring and exchanging knowledge and experiences not only November 2014 Page 107 of 139

109 between supported SMEs but also between administrative bodies involved in SME support. First, HLG needs to develop a stronger degree of ownership of the programme as suggested by NFBs. The expert group recognized that progress towards an increased synchronization and harmonization of funding was quite sluggish during the existence of Eurostars. The same topics are discussed repeatedly at HLG meetings without any clear and binding decisions. Hence, the expert group warns that such sluggish improvement might reinforce the inefficiency of the Eurostars systems in the future. Secondly, Eurostars offers a unique opportunity for learning from best practices in implementation of the programme but also for other national R&D programmes for SMEs. These learning opportunities are generally unexploited and the information exchange on national implementation practice or implementation rules is limited. Working groups were initiated but results were not visible to the expert group. Eurostars implementation is strongly rooted in national funding rules and their implementation practice, which is expected to oblige with the experience of SMEs and their management capacity. Hence, Eurostars also might provide a helpful feedback on those implementation practices which might lead to improvements also in national SME support programmes. Thirdly, the information flows from the national level to ESE and vice a versa needs improvement in term of the information provided as well as the timeliness of the information provided. This involves feedback on the details of the central evaluation where the national level needs more information to respond to demands from applicants. More importantly, information on the funding granted at the national level including effective starting dates of projects or the effective flows of money or more detailed information on the grants and dimensions of eligible project costs at the project level should be provided to increase the transparency of the programme. 7.3 Management and Operations The centralized evaluation and ranking of projects has worked well. The evaluation process is fast and the results of technical evaluation were available in time. ESE managed to increase the number of technical experts available for the Eurostars projects. However, the geographic origin of evaluators deviates significantly from the geographical origin of applicants and participants. It also is quite different from the geographical distribution of participants in the EU FPs. More significantly, experts with a proven background in the private sector and hence with detailed knowledge of markets and innovation diffusion are underrepresented. It seems crucial for a nearmarket programme to overcome this lack of market knowledge. The number of experts should be also increased in the light of the expansion of Eurostars 2 and the increase in the number of technical experts per proposal. In addition, to contribute to a reduction in the time-to-contract it will be useful to speed up the evaluation process although the expert group is aware that an even shorter time for evaluation will be hard to reach. In addition, high quality of evaluation is essential for the credibility of the programme and hence there should not be a compromise between speed and quality. The expert group is concerned, however, about an appropriate feedback from the outcome of evaluation results to applicants. In interviews with SMEs, there was quite a mixture of opinions from SMEs about the quality of feedback. Hence, NPCs November 2014 Page 108 of 139

110 should provide more detailed information on the assessment of proposals by technical experts and the IEP. The expert group recognizes that the implementation of Eurostars demanded adjustments of traditional funding rules. Changes of funding rules or even the installation of new national lines of funding were accomplished in a variety of countries. But the expert group is concerned that there is still inefficiency due to overly long procedures for the handling of successful applications at the national level. While the duration of technical evaluation is judged to be comparably short and handled efficiently, the waiting time of finalizing of funding contract at the national level is quite different between countries. The expert group acknowledges that there is considerable progress made in all countries but still the difference between countries is much too large. The analysis shows that a large share of projects suffer from long waiting times for contract signatures (which guarantees funding to SMEs) as well as additional waiting times for effective funding, i.e. when the funding is received by SMEs. Such differences in the handling of national procedures not only affects the national participant but spill over to their international partners in the whole consortia and thus endangers the programme of reaping the full benefits of the international approach to funding of the best innovation idea of R&D performing SMEs in Europe. The attractiveness of Eurostars to R&D performing SMEs in the participating countries is also evident from the increasing number of applications and the high share of resubmissions. This unexpected high demand for the programme challenged ESE as well as the national programme administrations. The expert group acknowledges that despite the high demand, countries managed to bring down the time-to-contract considerably. Despite this success the expert group suggests to increase knowledge on the history of the applications. Hence, information should be demanded from applications whether it is a resubmission or whether the proposal was previously (not successfully) submitted to other funding opportunities at the EU or at the national level. The expert group was surprised by the finding of the Agilis company of the information content of the central Eurostars database managed by ESE. The expert group admits that this database is not developed as an evaluation tool. However, even considering this, the expert group is convinced that there is an urgent need to improve the information content as well as the quality of information included. The Agilis report is a rich source of ideas for how the information content of the ESE database can be improved. The expert group strongly supports the conclusion that the available classification of projects by technology and also the classification of market areas must be improved. The existing classifications are not compatible with existing classifications of industries and technologies. The monitoring of the programme will benefit from a greater degree of comparability with other classifications. In addition, the implementation of the gathering of information stored in this database should be augmented by regularly checking the quality of data entered into the database. Furthermore, this database should be developed into a central information tool on Eurostars not only for ESE but also for participating countries. Hence, more information from the decentralized funding process should be included in the database and also NPC should be granted access to this database fully respecting the privacy November 2014 Page 109 of 139

111 laws and the interests of individuals, firms, universities and research institutes involved in applications and participation. 7.4 Funding Eurostar fulfilled the conditions for EU funding set by Decision No 743/2008/EC of the European Parliament and the Council. EU provided the maximum of EUR 100 million and the participating countries provided an estimated amount of 372 EUR million. At the level of individual projects, the expert group also recognized that participants are of the opinion that the size of the individual grants is appropriate. The implementation of Eurostars funding using the system of a virtual common pot is judged as an appropriate mechanism to raise and distribute funding. Overall participating countries provided more financial resources than initially agreed which demonstrated the incentive effect of the virtual common pot. However, the expert group observed quite an uneven size of national ear-marked budgets including budget extensions of participating countries. Participating countries contribute quite different amounts of public money to the programme. Here the absolute size is not the yardstick, rather the demand from competitive R&D performing SMEs and the development stage of their national systems of innovation. Ear-marked, national budgets for the programme are quite unevenly distributed. This might induce a selection of R&D partners by R&D performing SMEs not by the best strategic fit of a partner into the technological and market plans, but by resources devoted to the programme by participating countries. This potentially leads to inefficiencies in the composition of R&D performing consortia. However, the expert group found that a strategic choice of partners is not widely spread under SMEs. However, there were examples that the insufficient contribution of several participating countries induced frustration for financially unsuccessful but technologically promising projects. In addition, the expert group is also concerned about inefficiencies due to heterogeneity of national funding rules although the group admits that a significant adjustment of national funding rules have been made at the beginning and in early phases of Eurostars. In principle, the rules for funding for SMEs and their partners should be the same across all participating countries. This allows SMEs to compete at the same level using a similar approach with the same consortia. The expert group thinks that the heterogeneity of funding rules is still too large even if some progress has been made while the programme was in place. This heterogeneity concerns not only funding rates or upper ceilings for funding at the national level but also the eligibility of different types of project cost or the ex-ante proofs of project cost demanded. The trade-off between national funding traditions or styles and international harmonization needs to be shifted towards an increased equality across participating countries in order to provide the same incentives and opportunities for all SMEs and their partners. The financial and economic crisis has established the need for fiscal consolidation across all EU Member States. Hence, public R&D budget at the national level are also currently under enormous pressure. The back side of the incentives effect of a virtual common pot which allows and stimulates upward adjustment in favourable times is also the possibility for short-term budget cuts of the annual earmarked budget in bad times. Eurostars had encountered the situation that some countries were not in a November 2014 Page 110 of 139

112 position to contribute the initially agreed amount of their ear-marked budget for Eurostars projects. Furthermore, some countries were not able to take part in the initial call of Eurostars 2. Hence, it is a clear challenge to avoid cuts in Eurostars budgets as this will increase the inefficiency effect (mentioned above) resulting from an even more unequal relative budget. 7.5 Outcomes and impacts of Eurostars R&D performing SMEs with R&D projects funded by Eurostars grew twice as fast as R&D performing SMEs which applied but did not receive funding. This performance differential can be causally attributed to Eurostars funding. By applying an extensive econometric treatment effects analysis on employment growth, it can be shown that the fact of receiving a Eurostars grant leads to a 3% to 3.5% higher growth rate in the observably identical firm, on average. For firms close to the quality threshold, the regression discontinuity design methods even identify a higher growth rate, but this is not an estimate of the average effect of the Eurostars grant. The effect for projects that just got the treatment, i.e. firms being evaluated as just above the threshold, is possibly higher, as otherwise these firms would have had no other chances to implement their R&D projects. From the online survey, it becomes clear that for several firms that did not get the Eurostars grant they applied for; the Eurostars programme had non-negligible, positive effects for attracting other sources of finance including other public funding. Thus qualification in Eurostars may serve as a signal with respect to the positive certification of the firms R&D for other financiers even if the firms did not get the grant because the national Eurostars budgets were not sufficient to fund their projects in a certain cut-off. The estimates based on projects completed in 2012, or earlier, translate into a causal employment generation impact of nearly 8,000 jobs. Given the still relatively small number of completed projects, this estimate will become more precise in the future. The robustness of this inference were carefully checked by using different econometric approaches and tested using a broad spectrum of model variations. Hence, this evidence of a positive impact on employment generation must be judged as quite strong. The expert group also examined heterogeneous treatment effects in the employment growth analysis that could have yielded indications of potential programme implementation inefficiencies, or in other words, projects that perform better than others. Thus, it can be concluded that there is no urgent need to change any of the programme design features with respect to granting process. In addition, the econometric evaluation approach also established a positive and significant impact of the programme on the development of the patent portfolio of funded firms again compared to non-funded programme applicants. Hence, it was concluded that the programme is also successful in stimulating the innovation output of participants. In addition, the online survey makes clear that the magnitude of these findings should be considered as preliminary as several of the projects in the sample are still not completed and about a quarter of the awardees have not yet filed a patent application but intend to do so in the coming three years. November 2014 Page 111 of 139

113 There is also evidence on behavioural additionality of the Eurostars programme, as, in the online survey, many firms stated that, among other effects, the Eurostars application made the collaboration with project partners possible and that the firms improved their project management skills, for instance. According to the online survey, the Eurostars programme has an important effect not only on employment and innovation, but also on the firms eventual market performance. About half of the awardees state that the Eurostars grant has been highly important for reaching new customers with the firms products and services, and about a third of the firms report a high importance of the Eurostars grant for increasing their market shares. The positive programme impact found in the econometric analysis is also confirmed by qualitative methods. The overwhelming majority of funded firms provide detail examples in our interviews with firm owners or project managers of the innovation project funded by Eurostars. So, based on a mixture of rigorous quantitative and qualitative evidence, the expert group concludes that the Eurostars approach to stimulate innovation in R&D performing firms in participating countries was quite successful so far. Hence, Eurostars should be viewed as an impressive example of a successful innovation stimulating programme with a unique funding approach which address the vast heterogeneity of innovation in SMEs by using a bottom-up design where the type of innovation as well as the technology employed is determined by the technological needs of SMEs, and not by public administrators, scientists or technologists. Of course the expert group also detected some weaknesses and room for improvement and hence provides several suggestions which will make the programme even stronger and provide an even larger boost to innovation by European SMEs. November 2014 Page 112 of 139

114 8 Recommendations 8.1 Target group and scope of the programme (1) The ESE and NPCs are urged to ensure that the programme also reaches SMEs without pre-existing international links, by designing and implementing a marketing and promotion strategy. (2) The ESE and HLG should agree on the definition and development time of near market R&D projects for Eurostars, which is neutral to specificities of particular technologies and markets. (3) The ESE should develop an action plan to promote Eurostars participants contacts with R&D and Innovation intermediaries mostly involved in the project preparation phase. (4) The ESE should promote linkages between Eurostars participants and different types of EUREKA projects in order to facilitate Eurostars projects connections with clusters, umbrellas and other thematic or individual projects. (5) The ESE should organize brokerage events, such as pitch competitions, between Eurostars participants and financing institutions (e.g. the European Investment Fund and the European Investment Bank, the European Business Investment Forum (EBAN), VCs, Business Angels, crowd-funding platforms, banks), to increase the possibilities of successful projects to receive follow-up funding from the private sector. A dissemination strategy should be developed to target financiers, other SMEs, and national and international governments bodies. (6) The EC and ESE should check the possibilities for linking Eurostars and the new H2020 SME instrument to capitalize on the potential growth and competitive results of Eurostars SME participants, contributing to further increase their management capabilities and innovative performance as flags of a new generation of global businesses. (7) The ESE should raise awareness amongst the Eurostars beneficiary SMEs of the importance of protecting their innovations IPR. 8.2 Governance (8) It is important that the decisions taken in the HLG meetings are translated without delay into implementation plans with concrete objectives, tasks, as well as allocation of responsibilities (SMART approach 68 ). (9) The HLG should identify countries which impose parallel application and evaluation requirements as well as dual reporting obligations to Eurostars projects, and agree with them to eliminate the national processes. The HLG 68 SMART = Specific, Measurable, Attainable, Relevant and Time-based. November 2014 Page 113 of 139

115 should ensure that this is implemented before the spring cut-off of Eurostars-2 in (10) The ESE should improve the transparency and feedback mechanisms of the central evaluation process to applicants, NBFs and NPCs. The NPCs should have access to the consensus reports of the IEP, as well as to the individual reports of the technical evaluators to be able to give detailed feedback to applicants. (11) The ESE should ensure a more balanced geographical distribution of technical experts in close cooperation with the NPCs. Attention should be paid to achieve gender balance as well. In addition, ESE should expand the pool of technical experts with appropriate knowledge of markets. (12) The IEP should devote special attention to non-standard technical and market (potential breakthrough) solutions as identified in the expertise of technical experts. Possibilities for rapid feedback to and from technical experts should be implemented to ex ante explore the market potential of such projects. 8.3 Management and Operations (13) The frequency of two cut-offs per year should be maintained. (14) The ESE, HLG and NFBs should ensure that the time-to-contract will not exceed 7 months (210 days) as agreed for Eurostars 2 in the Budapest Document. (15) The ESE should decrease the time from application (cut-off date) to evaluation to 4-5 months so that results of the central evaluation will be available to NPCs and applicants within five months (150 days), at the latest. Every NFB needs to sign the funding contracts within 2-3 months. It is recommended that the ESE together with the chair of the HLG and the heads of NFBs reviews the programme management processes of those NFBs which risk not meeting the deadline, and agrees with them the steps to be taken to shorten the process. (16) From the spring cut-off of 2016 onwards, the ESE, HLG and NFBs should agree that for at least half of the projects the all grant agreements between NFB and programme participants will be signed within 210 days of the cut-off date. For 90% of the projects all grant agreements should be signed within one year from the cut-off date. (17) The EC should consider not to grant funding to those Eurostars projects, for which all grant agreements have not been signed within one year of the cut-off s deadline unless the delay in signing a funding contract is caused by members of the consortium funded (18) In order to speed up the contractual demands of project participants, a standard consortium agreement should be provided to applicants. November 2014 Page 114 of 139

116 (19) The ESE should collect the information from all countries about the timeto-money and submit this information to the HLG, and agree on a binding deadline for collection and submission. (20) The ESE should establish and maintain a robust central Eurostars database to collect, store and process reliable data on projects and beneficiaries. The ESE should introduce quality checks of data for the application forms as well as project progress, final and market impact report forms. The time delay for submission of the final reports should be reduced to one year and market impact reports should be available within two years of the end of the project. (21) Information on the exact start- and end-dates of the projects needs to be included in the central Eurostars database. The NPCs should deliver information on signed grant agreements including start- and end-dates of the projects without delay to the ESE. (22) The ESE should modify the application form and the project progress, final and market impact report forms. Detailed revisions of the current specific questions and headings in the forms, as well as reordering, deletion and addition of a number of questions are required to improve the logic of the forms. Correspondingly, the guidelines for these forms should be adjusted accordingly. (23) The ESE should develop a dedicated application form and corresponding guidelines for resubmissions. The resubmissions need to include an explanation on what has been changed, especially in response to comments from previous evaluations received via the NPCs. (24) The ESE and NPCs should jointly define and develop tools to facilitate information exchange between them, improve the quality and quantity of information available from and for both sides, allow monitoring of all project phases, and reduce the administrative cost of bilateral information exchanges. (25) The ESE should design and implement an institutional development plan to strengthen its programme and organizational management practices in order to be able to improve the implementation of Eurostars in line with international best practices, and to hire, retain and motivate qualified staff experienced in the management of R&D and innovation programmes. It is recommended that the ESE receives mentoring from the network s NFBs with relevant competencies. (26) The ESE should introduce a unique identifier for all organizations, not only for SMEs and large firms but also for universities, PROs, and other partners of R&D-performing SMEs. If available, each organization should provide their Participant Identification Code (PIC) for Horizon2020. The application form should include information about whether the project was submitted previously to another national or EU programme. November 2014 Page 115 of 139

117 8.4 Funding (27) The HLG is strongly encouraged to agree on a minimum baseline for funding rules to be harmonized and should ensure that each participating country conforms to this. The maximum funding rates should be the same for each type of partner in all countries. The process should be closely followed up by the HLG and be completed by the first cut-off in (28) The funding of the universities and PROs should be harmonized so that they participate as partners of the foreign R&D-performing SMEs and not only as subcontractors of R&D-performing SMEs from the same country. November 2014 Page 116 of 139

118 9 Annexes 9.1 Supplementary tables Table 9-1: Key figures for Eurostars per budget year Projects Budget year TOTAL / AVERAGE Number of applications submitted Number of eligible applications in % of all applications submitted 88% 77% 85% 88% 83% 86% 85% Number of applications above threshold % applications above the threshold vs eligible % 45% 42% 37% 37% 36% 40% Number of applications approved % application approved vs above threshold 68% 81% 70% 60% 61% 58% 64% Success rate I (approved/submitted) 42% 28% 25% 19% 19% 18% 22% Success rate II (approved/eligible) 48% 37% 30% 22% 23% 21% 26% Applicants / Participants Number of applicants Number of SMEs % of SMEs 75% 74% 71% 72% 72% 72% 72% Number of participants Number of participating SMEs % of SMEs 74% 72% 68% 68% 67% 72% 70% Total project costs Total budget of submitted applications (M ) Total budget of projects above threshold (M ) Total budget of approved projects (M ) in % of total budget of submitted projects Funding % 29% 26% 20% 20% 19% 23% Total funding estimated (M ) EU funding estimated (M ) Source: ESE (2013) Note: As national budgets is allocated per year data for cut-off1 is mapped 2008, cut-off 2 to 2009, cut-off 3 and 4 to 2010, cut-off 5 and 6 to 2011, cut-off 7 and 8 to 2012, and cut-off 9 and 10 to November 2014 Page 117 of 139

119 Table 9-2: EUREKA projects and project cost in 2013 by country Country Projects Individual Clusters Eurostars Individual Projects Clusters Eurostars Individual Projects Clusters Eurostars Individual Projects Source: ESE (2014): EUREKA Annual Report 2013; available in the Internet at fa047d68956e&groupid=10137 Clusters Eurostars AT ,4 7,5 15,8 6% 19% 75% 2% 32% 67% BE ,3 37,7 5,5 13% 54% 33% 9% 79% 12% BG 1 0,3 100% 100% CH ,3 3 31,4 12% 6% 83% 6% 8% 86% CY 3 0,6 100% 100% CZ ,5 5,5 1,9 67% 15% 18% 61% 29% 10% DE ,4 24,8 19% 24% 58% 52% 38% 10% DK ,4 6,5 14% 86% 18% 82% EE ES ,2 48,4 16,9 26% 35% 38% 17% 62% 22% FI ,4 76,2 4,7 6% 65% 29% 0% 94% 6% FR ,4 327,1 25,4 13% 44% 44% 2% 91% 7% GR 1 1 0,5 50% 50% 100% HR % 100% HU ,2 0,3 0,2 87% 7% 7% 86% 8% 5% IE ,3 0,6 0,3 33% 33% 33% 25% 50% 25% IL ,6 4 2,8 56% 22% 22% 35% 38% 27% IS 2 1,2 100% 100% IT ,5 7,7 20% 80% 6% 94% LT 1 0,2 100% 100% LU 2 1 1,2 0,5 67% 33% 71% 29% LV ,8 0,8 0,4 50% 25% 25% 40% 40% 20% MT Approved Projects 2013 Total Project Budget 2013 Approved Projects 2013 Total Project Budget 2013 Number by Country in Share of all EUREKA Projects Share of Total Budget by Country NL ,3 32,5 5% 30% 65% 58% 32% 10% NO % 100% PL ,5 1,5 1,5 59% 18% 24% 60% 20% 20% PT ,1 0,3 89% 11% 0% 98% 2% RO ,1 10,9 0,3 41% 50% 9% 27% 71% 2% SE ,8 12,4 25,2 16% 17% 67% 11% 29% 59% SI ,9 0,5 1,1 80% 5% 15% 81% 6% 13% SK 9 3 1,7 0,6 75% 25% 74% 26% TR ,4 50,4 3,4 33% 49% 18% 15% 80% 5% UK ,9 22,4 24% 76% 4% 96% TOTAL 403,6 799,8 252,9 28% 55% 17% November 2014 Page 118 of 139

120 Table 9-3: Number of applicants by country and cut-off Cut-off County Total AT BE BG CY CZ DE DK EE ES FI FR GR HR HU IE IT LT LU LV MT NL PL PT RO SE SI SK UK CH IL NO TR Others Source: ESE evaluation data (extracted end of December 2013) November 2014 Page 119 of 139

121 Table 9-4: Number of applicants above threshold by country and cut-off Cut-off County Total AT BE BG CY CZ DE DK EE ES FI FR GR HR HU IE IT LT LU LV MT NL PL PT RO SE SI SK UK CH IL NO TR Others Source: ESE evaluation data (extracted end of December 2013) November 2014 Page 120 of 139

122 Table 9-5: Number of applicants with funding contracts signed Cut-off County Total AT BE CY CZ DE DK EE ES FI FR GR HU IE IT LT LV MT NL PL PT RO SE SI SK UK CH IL NO TR Others Source: ESE evaluation data (extracted end of December 2013) November 2014 Page 121 of 139

123 Table 9-6: Share of applicants in projects above the threshold as % of number of applicants Cut-off County Total AT BE BG CY CZ DE DK EE ES FI FR GR HR HU IE IT LT LU LV MT NL PL PT RO SE SI SK UK CH IL NO TR Others Source: ESE evaluation data (extracted end of December 2013) November 2014 Page 122 of 139

124 Table 9-7: Share of applicants with signed funding contract in per cent of the number of applicants above threshold Cut-off County Total AT BE BG CY CZ DE DK EE ES FI FR GR HR HU IE IT LT LU LV MT NL PL PT RO SE SI SK UK CH IL NO TR Others Source: ESE evaluation data (extracted end of December 2013) Annotation: Data for cut-off 9 and 10 are not completed at time of data extraction November 2014 Page 123 of 139

125 Table 9-8: Share of applicants of broad technology categories by country (in per cent at applicant level) Electronics, IT, telecom Mechanical, chemical, measurement technologies Energy & environmental technologies Biotech, pharmaceuticals, medical devices Agriculture, Food, Aquaculture AT BE BG CH CY CZ DE DK EE ES FI FR GR HR HU IE IL IT LT LU LV MT NL NO PL PT RO SE SI SK TR UK Others Total Source: ESE evaluation data (2014) November 2014 Page 124 of 139

126 Table 9-9: Distribution of types of participants by country (in %) Country R&D performing SMEs Other SMEs Universities Research institutes Large companies Others AT BE BG CH CY CZ DE DK EE ES FI FR GR HR HU IE IL IT LT LU LV MT NL NO PL PT RO SE SI SK TR UK Others Total Source: ESE evaluation data (2014) November 2014 Page 125 of 139

127 Table 9-10: Results of Cox-regression of determinants of time-to-contract This table gives the coefficients of a Cox regression model of the time span until the funding contracts are signed talking into account that probably not all contracts are signed yet. The larger are the coefficients the shorter is the time-to-contract. In addition to country effects the model considered potential difference with regard to role of an organisation in the project, the type of organisation, the costs of each project, the number of projects per cut-off for each country, the number of project partners, the rank of the project, technology, the cut-off and the country. The models found a decreasing time-to-contract till cut-off 6 followed by an increase. However, results for cut-off 9 and cut-off 10 suffer from a small number of projects with signed contracts. In addition, the model showed that an increasing number of project per country induced a longer time-to-contract probably as handling of more contracts needs more time for the projects administrators of that country. Similarly, more complex projects (larger number of partners, larger projects costs) demanded more time for signing the contract. Finally, projects involving biotech, pharma, chemistry or agricultural technology took longer duration for concluding the funding contract. November 2014 Page 126 of 139