Introduction. Topic I: description of topics in work programmes. First experiences with Horizon 2020

Transcription

1 First experiences with Horizon 2020 Topic: Feedback from questionnaires and expert workshop Date: 27November 2014 Introduction This note provides a set of recommendations 1 stemming from the IGLO questionnaire on first experiences with Horizon 2020 and refined by a validation workshop with national experts organised by IGLO on 12 November 2014 in Brussels. The questionnaire and the workshop covered four distinct areas of the Horizon 2020 proposal submission and evaluation processes: 1. Topic description in Work Programmes: general experience, the expectations of the applicants and communication with the EC or Executive Agencies services. How applicants approached crosscutting issues (e.g. multidisciplinarity, SSH, etc.); 2. Technical aspects of the Participant Portal (PP) such as the submission of proposals, the completion of online forms and the retrieval of information; 3. Evaluations: the first experiences with the evaluation of proposals, the content of Evaluation Summary Reports (ESR) and their links with the description of topics in work programmes; 4. Clarity of legal, guidelines and Annotated Model Grant Agreement (AMGA). This note summarises the answers received in relation to each area and outlines the main issues raised and the proposed recommendations for further action and possible improvements. A total of 120 questionnaires from 13 different countries were analysed. Respondents were mostly research managers, project administrators and researchers coming from universities, research and technology organisations, public research organisations, SMEs and industry. The recommendations were then discussed and refined by 30 experts from 16 European countries during a workshop on 12 November 2014 in Brussels and presented to representatives of the European Commission and the Research Executive Agency. Topic I: description of topics in work programmes In general, participating experts felt that the broader topic descriptions increased the range of possibilities, making it easier to fit in with a project. This also gives more intellectual freedom to participants, which is very much appreciated and leads to more creativity and to a larger variety of possibilities for creating a project proposal with different partners. However, broader topics lead to more intense (and thus more time consuming) discussions with project partners as the content of the work is more uncertain. It becomes more challenging to find the best fit for the project or to draw the outline. Furthermore, broader topics attract more proposals, thus decreasing the success rate. It is sometimes difficult for the participant to identify which research issues are clearly in or out of the scope and where they fit within European research and innovation strategies. The 1 The contents of this document cannot necessarily be taken as representative of the views of all IGLO Offices. 03 December 2014

2 wording is sometimes too vague, especially for the scope and impact, making it difficult to understand the specific results and scope(s) that have to be achieved by the project. As a result of interdisciplinarity and innovation as being centralconcepts in Horizon 2020, more Small and Medium-sized Enterprises (SME) or industry were involved in consortia. Participants also sought partners outside of their friendly ecosystem which led to interesting discussions and thinking outside the box.interdisciplinarity allows more options/alternatives. Participants needed to find the appropriate partners, to gather a very interdisciplinary (or multidisciplinary) team of experts and involve relevant stakeholders: crucial consortium building. In general the consortia tended to be more multidisciplinary rather than interdisciplinary. Innovation has been deemed a key element for a successful proposal but there were misconceptions and/or confusions on how to quantify the innovation component. In some topics respondents felt that emphasising innovation seemed inadequate as most of the project would actually be research based. Furthermore, interdisciplinarity can be an issue for certain projects and topics that require clear disciplinary approaches. From the feedback received, some universities seemed to have problems with assessing how innovative, i.e. commercial, a project is or should be. They also faced difficulties contacting SMEs or close-to-market activities in order to tackle the innovation aspect. The indication of a budget range remains confusing. This might for example lead a consortium to maximize the budget in order to increase the size/credibility of the project. Most of the experts stated that National Contact Points (NCPs) were well informed and very helpful, as were liaison offices in Brussels. However, helping applicants to refine their thinking in relation to a broad topic remained a challenge. The participants welcomed the initiative from the EC to organise Horizon 2020 infodays. Finally, more information on third country participation and availability of funding would help applicants. This is particularly relevant for those countries, which are not automatically eligible to receive funding from the Horizon 2020 budget. The wording and the expectations described in the topics could be refinedto allow the participants as well as NCP and evaluators to better understand the challenge they are dealing with. For example, a formal quantification and clearer definition of impact, its weighting and clarification of the political thinking behind the topic would be welcomed. This includes a domain-specific definition of Technology Readiness Levels (TRL). Interdisciplinarity and innovation should be used when necessary and adapted to the concrete topic and project type. More training to NCPs and potential applicants could bring better understanding of the topics. With regards to third country participation, a list of available funding or a dedicated website could be set up providing information on cooperation conditions. Topic II: technical aspects of the participant portal The Participant Portal (PP) was found on the whole to bea well-designed tool and as such is welcomed by the participants. Most participants found the navigation through the website to be fairly intuitive. A useful tool is, for example, the already available graphical representation that allows to trace and locate the project along the different stages of the 3 December

3 grant preparation. However, the location of necessary documents is inconsistent between topics or illogical. More generally, information should be systematically given in the same way throughout similar calls. For example, submission services are located on different pages/locations depending on the topic and in some cases templates are not next to corresponding guidelines and instructions. Call and topic search function and sorting are still difficult to use and counterintuitive. Participants actually often used Google. IT issues still exist as the submission of draft proposals sometimes led to their immediate disappearance after submission, while downloading documents worked only on the call page and not the topic page. Participants have conflicting experiences with the RES and IT helpdesk: either the answers were both fast and accurate, or they would come late and simply refer to guidelines or AMGA without further explanation or relevance to the asked question. Some answers didn t come at all. The display of information should be more consistent across all areas of the PP and the documentation should be listed in order of relevance, including an explicit reference to those documents that may not yet be ready. An updated and well-functioning database of funded projects should be available. IT requirements for accessing the different functionalities of the PP should be communicated more clearly. With regards to the IT helpdesk and RES, communication tools such as a direct phone number or a web chat would be welcomed. For example, in ICT there was a web forum to raise questions, which worked well. This service could be broadened to the other parts of Horizon In addition an on-line tracking system or a sort of ticketing for submitted requests is strongly recommended. A more sophisticated search/sorting by keywords/upcoming deadlines for calls would be welcomed. An online tool could help participants to better assess the TRL of their proposals. LEAR information: when the LEAR receives a message about a European project, in addition to the project number, it would be convenient to have also its acronym. Topic III: Evaluations The evaluation of proposals is clearly linked to the broad description of topics. The majority of experts consulted stressed the importance of a fair and balanced evaluation process. Some experts complained that insufficient comments and score explanations were given in the Evaluation Summary Report (ESR), especially when considering the time invested in a proposal. Experts felt that clear links were missing between scores and justifications in the ESR. For example, low scores didn t seem to match with solely positive comments. Moreover, the comments from the experts were sometimes very short and thus not perceived as adequately addressing whether or not the cross-cutting issues were taken into account in the proposal. Participants were disappointed in that regards as they would like to better understand the evaluators view in order to make the most of the feedback and to better understand the funding logic of the framework programme. ESR play an important role as well because they provide information to the applicants on how their project fit the overall aims of 3 December

4 Horizon They can be seen as an opportunity to teach failed applicants on the objectives of EU research funding and high-end policy objectives such as ERA. The success rate for first stage could be kept at minimum in order to have at least a 30% chance of being positively evaluated in the second stage, perhaps by increasing the threshold of eligibility. The impact part of evaluation could provide a clear matrix showing how a proposal contributes to the impacts defined in the topic (also taking into consideration the feasibility aspect). Matrices could also be used when points are deducted in any criterion in order to provide clear justifications of which aspect was not fulfilled and how much it contributed to reduction of the score. The objective should be to present clear link between the weaknesses of proposals and the attributed score. Consensus meetings must be mandatory in order to harmonise the evaluation scores by removing cultural differences in the interpretation of evaluation criteria by single experts and not simply take the average of the scores. A ranking list for all calls for proposals (i.e. calls results) should be published after all ESRs have been send out in order for all applicants (irrespective of the call) to have a reality check. Better use of in-house and FP7 best practices by EC developed in selected areas could be made, with a link to better quality assurance. Ensure a common understanding of the evaluation criteria among the entire evaluation panels and provide training to evaluators and moderators better linked to the call descriptions. Topic IV: Clarity of legal issues, guidelines and AMGA In general, participating experts appreciatedthe simplifications introduced in Horizon 2020 (e.g.the simplified funding scheme). They also felt that most of the necessary information on legal and financial issues was provided; it was, however, sometimes difficult to find. The Annotated Model Grant Agreement (AMGA) is a very comprehensive document; the many practical examples provided are in particular very much appreciated by the participants. However, the navigation around the AMGA is not easy. Some inconsistencies between AMGA for different instruments and schemes remain. In addition, cost management and categories(especially personnel costs) and financial support to third parties still seem not to be very clear. The drop in pre-financing needs to be strongly advertised particularly for SMEs. With regards to timesheets, the 100% declaration of personal costs only applies if it is considered from the start of the reporting period, which goes against the simplification principle. 3 December

5 Finally, on the reporting of personnel costs, the fact that the cost calculation is based on the last closed financial year is a complication, not a simplification, in many EU countries as it does not take into account automatic annual salary raises. This leads to mistakes in institutions as they need to use the manual calculation and may cause financial losses. This provision is also contradictory to the principle of actual costs. The AMGA should include a more detailed table of content than in the version currently published as it is difficult the go through the whole document with only the search by keywords. User friendly media communications (such as webinars, tutorials for beginners, Q&As...) would be very much appreciated by the scientific community as they are not very familiar with administrative procedures. The coordinator s workshops were very well received by the community. Declaration and calculation of personnel costs should be simplified taking better into account the real costs and usual working and accounting practices occurring at the institutions. 3 December