Research Councils UK and UK Shared Business Services Limited

Transcription

1 Research Councils UK and UK Shared Business Services Limited Grant Applicants' Feedback Survey Compiled October 2013 Stephanie Brewer UK SBS Performance Analysis Team

2 Contents Foreword Snapshot of Initial Findings... Page 1 Section 1: Participation... Page 2 Section 2: Pre-submission... Page 3 Section 3: Pre-decision... Page 5 Section 4: Post-decision... Page 10 Section 5: Post-award... Page 13 Section 6: Summary... Page 15 Section 7: Agreed Actions... Page 18

3 Foreword It is with great pleasure that we share this, our third, grants customer satisfaction survey report. The seven UK Research Councils and UK SBS (previously RCUK Shared Services Centre (SSC) Ltd) work in partnership to deliver research funding that has an impact on the growth, prosperity and wellbeing of the UK. In running our grants processes we rely on the involvement, commitment and goodwill of a large number of experts and administrators and we believe that it is absolutely critical to ensure that they have a positive experience whoever they are and whatever their interaction with us. This survey helps us understand their current experience and will inform our plans to ensure each interaction is a positive experience. We have identified four key customer groups: Reviewers, Panel Members, Applicants and Research Organisation Administrators. Our commitment is to run a customer satisfaction survey for each of these groups on an annual basis. The first round of these surveys, to be completed over a 12 month period, will provide us with a benchmark against which all future surveys will be judged. For our third survey we chose to focus on Applicants of Research Council funding. Successful applications are crucial to the work of the Research Councils and are what ensures that the research we fund is of the highest possible quality. It can be a challenge to ensure that each grant application we receive is presented with the best possible opportunity for being funded. Therefore, it is important to provide the best experience possible for our applicants, and in so doing encourage their continuing interest in the application process. We have been pleased to receive a large amount of positive feedback amongst the 1,789 responses, which translates to an overall satisfaction score of 6.2 out of 10. However, we would of course like to see a higher satisfaction score and there are a number of areas where the respondents have identified some opportunities to enhance the applicant experience. This report outlines a number of actions that aim to convert this valuable feedback into improvements over the coming months and we intend to make the outputs of those actions visible to our stakeholders as they are rolled out. Opening up this dialogue with our grants community is extremely valuable to us and we would like to extend our most sincere thanks to those who have contributed to this survey and, of course, for the time and expertise that is given in support of the research grant process. Dr Andrew le Masurier (STFC) Chair of Grants Governance Group Anne McFarlane Head of Grants Service Delivery UK SBS

4 UK SBS Grant Applicants' Feedback Survey Each year the Research Councils invest around 3 billion in research covering the full spectrum of academic disciplines. As part of our commitment to the principle of high quality and cost-effective research the Research Councils and UK SBS, formally known as RCUK SSC Ltd (the Shared Services Centre) have requested feedback on the Applicants experience(s) of the service we provide. A total of 6,804 Grant Applicants were approached to take part in one in a series of surveys, created for specific customer groups within the research community. The survey feedback will help us ensure the overall application process and then later grant maintenance is as effective and smooth as possible, and improve their experience of working with us. Snapshot of Initial Findings Overall Average Score Overall Response Rate % is the average score of how the respondents rated their experience as an Applicant of Research Council funding. This is on a scale of 0 to 10, where 0 equates to extremely dissatisfied and 10 equates to extremely satisfied. Volume of responses Funding Requests AHRC BBSRC EPSRC ESRC MRC NERC STFC of the 6,804 Grant Applicants approached submitted a response, with representation for all of the seven Research Councils. 1,791 is the total number of valid survey responses received, identified by the Research Council each respondent most frequently applies to for funding. Research Council most frequently applied to Available Information Je-S Helptext 8% 24% 48% 17% 72.9 % Funding Guidelines/Handbook 5% 18% 56% 20% of respondents indicated they were either satisfied or very satisfied with the overall information available via specific channels. Scheme or Call specific guidance 5% 17% 57% 20% Very dissatisfied 0% 20% 40% 60% 80% 100% Dissatisfied Neither satisfied nor dissatisfied Satisfied Very satisfied 1

5 Section 1: Participation Overall, a total of 6,804 applicants were approached on behalf of the Research Councils with 1,791 (26.3%) valid responses being received. The applicants were initially asked about their funding requests to the Research Councils. Volume of responses 1. Which Research Council do you apply to most frequently for funding? The volume of responses is spread across the Research Councils which indicates a cross section of experience with each Research Council's grant proposal process that will be reflected in the responses throughout the survey. 0 AHRC BBSRC EPSRC ESRC MRC NERC STFC 2. Please indicate which other Research Councils you have also applied to previously? Volume of responses AHRC BBSRC EPSRC ESRC MRC NERC STFC N/A The applicants were also asked which other Research Councils they had made funding requests to with several multiple responses being submitted where an applicant had applied to more than one Research Council during their research career. This indicates the need to ensure the grant proposal process, where possible, is consistent across all the Research Councils. An N/A option was provided for those applicants who had not applied to any other Research Council apart from the one indicated in their response to Question 1 above. 3. How many proposals have you submitted to the Research Councils during your career? 10+ proposals, 31.0% 1 proposal, 6.3% With over a thousand (57%) responding members having submitted 6 or more proposals this indicates the wealth of experience in the grant proposal process which will be reflected in the responses throughout the survey proposals, 26.0% 2-5 proposals, 36.7% 113 (6.3%) of the overall responses are from applicants who have only submitted one proposal at this stage in their research career which will provide a balance of experience from lessfrequent participants in the grant proposal process. Volume of respondents 2

6 Section 2: Pre- ubmission The applicants were asked about the information available during the initial creation of their grant proposal. 4. As an applicant, how satisfied have you been with information for each of the following: Je-S Helptext Funding Guidelines/ Handbook Scheme or Call specific guidance 0% 20% 40% 60% 80% 100% Scheme or Call specific guidance Funding Guidelines/ Handbook Je-S Helptext Very dissatisfied Dissatisfied Neither satisfied nor dissatisfied Satisfied Very satisfied It is encouraging to see that 72.9% of respondents were satisfied or very satisfied with the overall information made available prior to grant submission during the initial stages of set up with only 7.6% of all respondents indicating they were dissatisfied or very dissatisfied. It is interesting to note that the Je-S Help text option received a higher proportion of neither satisfied or dissatisfied responses and insight into this can be gained from the freetext comments submitted, as indicated below. The participants were given the opportunity to provide free-text comments to support their answer choice and 415 (24%) of the 1,756 respondents who answered this question provided various comments relating to the 3 questions asked above, which have been categorised into 5 key themes. What other information would you like to see? Clearer, specific focused guidelines 237 Provide assessment and decision criteria 103 Link to more specific help text from Je-S form 59 More notice of specific Calls 17 Sufficient existing information 19 No Trend Identified volume of comments The question asked for views on specific areas of available information and the themes identified from the comments received fell into three main areas: guidance, decision making criteria and Je-S help text. 3

7 GUIDANCE: The first and largest area, receiving 237 (53%) of the comments and categorised as 'Clearer, specifically focused guidelines', relates to the overall provision of guidance. This category includes responses such as; "There was a lot of duplication in the materials and at times it was quite difficult to identify which guidance applied to which scheme." and also "It would be easier if all information was in one place. There is always the sense that some information is in the handbook, other info in the call, other information about the application springs up in Je-S and it becomes quite nerve wracking to be sure to follow it all ". 19 (4%) of the comments indicated that there is currently sufficient existing information available. DECISION MAKING CRITERIA: The second theme identified, with 135 (25%) of the free-text comments and categorised as 'Provide assessment and decision criteria', includes comments along the lines of "Details of criteria against which applications will be assessed listed in order of priority or by weighting would be useful." along with "It seems that there is often a mismatch between the text of the call/guidance end the interpretation by the reviewers, panel and research councils." Other topics mentioned in this category refer to the provision of detailed feedback and also providing exact timescales for each stage of the review, both of which are discussed in greater detail later in this report within Section 3, question 6 and Section 4, question 10. JE-S HELP TEXT The third area raised with 59 (13%) comments refers to the need for a 'Link to more specific help text from Je-S form' and included various comments like; "Many of the Je-S help text links go nowhere. Many of the explanations given in those help texts are poor" along with several constructive comments including "The joint system is clearly sensible, but the help sections suffer enormously from being generic, with specific information for each funder hidden away. Since the link to the help pages is from the Je-S form, it would be good if that went to funder-specific information." along with "Including a HTML version of the handbooks would allow links between the help sections of the Je-S interface and directly to the relevant information in the handbook (rather than just linking to a long pdf)." Other comments relating to the Je-S help text included; "Talking with other new applicants, they sometimes do not realise how much help there is in detail on the Je-S Help text and so it could be worth making this clearer in the funding guidelines. Novice applicants seem to be nervous about going on to Je-S before they have done considerable preparation but they aren't able to do this in sufficient detail without consulting the Je-S Help text. It therefore becomes a bit of a barrier." and similarly, "Many of my mentees do not even realise that there is helpful - and, indeed, essential - information available inside the Je-S system." both of which are collaborated by the submitted comment "It was not obvious when I first used Je-S that there was a "Help" section nor that it gave information on individual forms within Je-S." and clearly highlights that there is more work to be done to increase awareness of specific functions available. TIMESCALES: The final factor identified from the submitted responses, with 17 (4%) comments, is 'More notice of specific Calls' and includes "Specific calls need to be advertised with larger submission horizons, and advertised more broadly" along with "Specific calls for funding actions need to be advertised more in advance and probably via at least to people registered on JE-S. Those of us in the University system are often highly pressed for time so if a call is not advertised well it is not likely to be found out about until too late." 4

8 Section 3: Pre-Decision The applicants were asked about the functionality and tracking of their grant proposal, including the initial submission through to responding to reviewer comments. 5. How easy was the process of submitting your proposal via Je-S? 6.4% 0.9% 3.7% Of the 1,791 respondents, a total of 1,752 (98%) indicated how easy they found the process of submitting their grant proposal via Je-S. 34.4% 54.6% Extremely difficult Very difficult Manageable Very easy Extremely easy It is encouraging to see that only 80 (4%) of the respondents indicated that they found the process very or extremely difficult. It is also interesting to see that the highest factor, with 956 (54.6%) responses, indicates that applicants found the grant submission process manageable. It should be noted that the response scale is very wide between the two options of very difficult and manageable with no intermediate level, and although the scale is reflective on both sides (negative and positive) this may have affected the answers given. 462 (26%) of the participants who provided an answer to this question took the opportunity to provide free-text comments to support their answer which includes 298 of those who indicated they found the process 'Manageable' and 47 who found it 'Very difficult'. All the various comments received have been categorised into 15 key themes. Found interface un-intuitive Process gets better with use Areas of repetitive information requested Areas of redundant information to complete Found financial sections difficult to complete Provide clearer guidance Provide checklist of documents to be submitted Difficulty building multi-institute applications Problems with the system Create specific Je-S help text Too much information required up front Frustrating character count and formatting limitations Inability to complete uploadable word documents Inability to link existing 'other research' data Found process reasonably straightforward No Trend Identified Volume of comments Of the 462 (26%) respondents who provided comments 86 (19%) indicated that they found the process reasonably straightforward. 5

9 INTERFACE: It is interesting to see that the top two factors identified firstly, that users 'Found the interface un-intuitive', with 91 (20%) comments and then advised with 65 (14%) comments, that the 'Process gets better with use'. This is highlighted by such comments as "Given the infrequency with which many users engage with Je-S it is not built in a very intuitive or user friendly way." along with "The system for choosing subject categories is extremely cumbersome, especially for an interdisciplinary proposal. One should be able to enter partial information into a page (e.g. name of PI) - it makes things more difficult to have to have all information for each part before it will save it. The Je-S system should implement page, or item level locking rather than only allow one person to edit at a time - this is basic computer science." 21 (5%) comments highlighted issues with the system which included "There were technical issues with Je-s which delayed Co-Investigators registering." along with "Co applicants who are not in academic institutions cannot view the documents easily. This makes collaboration difficult." STREAMLINE: Several categories indicate a clear view from the respondents that they feel various elements of the grant proposal are either repetitive (53, 12%) or of a more generic nature and rather redundant to their specific call or scheme (43, 9%) in addition to those who feel (15, 3%) that too much information is required too early in the process, as highlighted in the following comments; "There is too much redundancy. too little clarity about why box X and box Y are asking for the same information." and "The amount of information required at the first round of assessment is rather excessive given that peer review college members comment mainly on CFS and impact. The supply of additional information could come if the proposal makes it through the first round." The financial section of the grant proposal is identified as an area for potential improvement with 39 (8%) responses raising the issue of how difficult this section of the application can be to complete and includes such comments as "Entering financial information can be tricky. In particular, both institutes that I work for use indexation for approval, while Je-S does not (for certain categories). Then there's directly allocated and directly incurred staff, 100% FEC for certain equipment, but not for others, different categories of support staff, etc, etc." and also "Entering the budget is the most difficult part of the process." along with a suggestion "I think a more streamlined process might make the system more easy such as collation of the documents into one or even supply of a financial form (electronic) to fill in rather than having to fill in on line which could be linked to an institutions financial forms somehow. I realise all institutions are different etc... but it does double or triple the work." GUIDANCE: There are several categories identified from these comments that have appeared earlier in this report, (Section 2, question 4) including 'Create specific Je-S help text' (17, 4%) and 'Provide clearer guidance' which received 34 (7%) comments along the lines of "For specific calls there is sometimes a mismatch between Je-S forms and the call requirements which can be confusing." and "One really has to search for crucial information on e.g. font sizes, page limits for different funding schemes - it is irritating that there are so many differences between schemes and such frequent changes to what is required." In correlation with this are 29 comments identified as 'Provide checklist of documents to be submitted' which includes such comments as "A flow chart which details the "boxes within boxes" to be completed may assist to avoid overlooking some." and "It can be quite difficult to work out what attachments are expected on which pages". The insight provided by these comments clearly highlights the need for further investigation into customer needs and end user training. How easy was the process of submitting your proposal via Je-S? Only 80 (4%) of the respondents indicated that they found the process very or extremely difficult. 6

10 6. Do you use the Je-S functionality to track your proposal between submission and decision? 7. If yes, How useful did you find the information? 100% 90% 66 Very useful 80% No, % 472 Useful Yes, % 50% 40% 217 Neither useful nor useless 30% Not very useful Of the 1,752 respondents, 1040 (59%) indicated they used the Je-S functionality to track their grant proposal submission and it is encouraging to see that over half of these, 538, found the tracking functionality useful or very useful. 20% 10% 0% Volume of respondents who use the Je-S tracking functionality Useless The respondents who had indicated they used the Je-S tracking functionality (1,040, 59%), which included 128 who had indicated that the tracking function was 'not very useful', were given the option to submit free text comments to support their answer. 266 of these respondents (26%) provided various comments which have been categorised into 5 key themes. Limited stage of process detail Clearer indication of timetable Out of date or inaccurate Improve communication of final outcome Useful tool No Trend Identified How useful did you find the tracking functionality? Volume of comments Of the 266 (26%) respondents who submitted comments 163 (61%) refer to the limited detail provided, via the tracking functionality, of the stages of the submission process and includes various comments along the lines of "In the period following application, the description is terribly useless reading merely "With council/in progress". Additionally, a further 70 (26%) comments suggest a clearer indication of the submission timetable would be useful, including meeting dates, number of reviews received and deadline dates with comments such as "It would be useful to see some dates for the return of reviewers' comments, the committee meeting and a decision. As applicants we are held to very strict deadlines so it would be nice to see some commitment in the other direction." This suggests a potential area for development particularly when also considering the responses below from those users who do not use the tracking function. 7

11 8. Please could you tell us why you chose not to use the tracking functionality? Not aware of tracking facility No interest as doesn't change the outcome Contains no useful information Rely on alert s Too busy Already aware of timeline Too difficult to access Out of date or inaccurate No Trend Identified Volume of comments Of the remaining 712 (41%) respondents, who indicated they did not use the Je-S functionality to track their grant proposal submission, 691 (97%) also provided various free text comments to explain their reasoning. With 210 (30%) indicating they were not aware they could track their proposal after submission this again highlights a potential area for development of this specific function. It is interesting to see the similarity in some of the categories to those given above by users of the function, with 113 (16%) indicating that no useful information was provided and 16 (2%) claiming the information was out of date or inaccurate. There is also an element of users (23, 3%) who find the function too difficult to access. However, it is also interesting to see there is a second group of comments (243, 35%) from respondents that are either not interested in tracking their application as they cannot influence the outcome or are too busy and this ability is not seen by them as a priority. 99 (25%) comments also advised that some respondents rely on existing alerts advising them of any update to their grant application, such as reviewer comments received. 9. If you were invited to respond to reviewers comments (PI response) how satisfied were you with this experience? Share of responses 100% 90% 80% 70% 60% 50% 40% 30% Very satisfied Satisfied Neither satisfied nor dissatisfied Dissatisfied Of the 1,752 respondents, 1,517 (87%) indicated how satisfied they were with the experience of responding to reviewers comments. It is encouraging to see 816 (54%) of respondents were satisfied or very satisfied with the experience. 20% 10% 0% Level of satisfaction Very dissatisfied A further 222 (13%) indicated they had never been invited to respond to reviewer comments. An option to submit free text comments was taken up by 685 (45%) of these respondents, including 302 of the 354 who had indicated they were dissatisfied or very dissatisfied with the experience which have been categorised into 10 common themes. 8

12 Unclear if PI responses considered by panel 169 Insufficient timeframe for responses 150 Variable standard of reviewer comments 135 Difficulty accomodating late reviews 109 Fragmented receipt of reviews Provide increased space for reply Publish key dates for PI comments 60 Clearer guidance on which concerns to address 38 System issues 17 Valuable opportunity 129 No Trend Identified Volume of comments TRANSPARENCY: Although it is encouraging to see 129 (19%) respondents felt being given the option to respond to reviewers comments was a valuable opportunity open to them the most frequent response (169, 25%) felt it was unclear if the Principal Investigators' responses were taken into consideration by the panel when discussing the individual proposals. TIMESCALE: The effort required by the applicant to provide the response for the panel is compounded by various issues around the response process with 150 (22%) respondents indicating that insufficient time is given to provide a response, particularly when there has been no pre-published key date of when responses will be required (60, 9%) allowing the applicant to ensure their availability at the appropriate time. FRAGMENTATION: This issue is further compounded by the fragmented receipt of reviews (72, 10%) where an applicant(s) have taken time and used the available page allocation to respond to initial reviews received to only find that additional reviews are sent through at a later date which have to be accommodated into the prepared rebuttal within the initial page allocation. 109 (16%) respondents referred to the difficultly in accommodating late reviews which are received after the agreed deadline, often within days of the panel meeting, where the applicant is again expected to provide a comprehensive response within the initial page allowance, taking into account any contradiction from the late reviewer to those previously received. It is not surprising therefore, to see that 69 (10%) comments suggest an increase in the space given for replies based on the number of reviews received, feeling that demanding the space allocation to respond to 3 reviewers comments is the same as responding to 6 comments gives an unfair advantage to the former. GUIDANCE: In addition, 135 (20%) comments referred to the variable standard of reviewer comments submitted and 38 (6%) suggested that clearer guidance should be provided, either by the reviewers themselves or the Research Council, on which areas should be addressed. 9

13 Section 4: Post - Decision The applicants were asked about the communication of the final funding decision and the functionality of making changes to the grant during its lifecycle. 10. Once the funding decision had been made on your proposal, did you feel that you were provided with informative feedback? Yes, 687 N/A, 114 No, 932 Of the 1,752 respondents, 1,733 (99%) indicated whether they felt they had been provided with informative feedback following the funding decision on their grant application. 932 (54%) of respondents felt they had not received informative feedback on their proposal. All the responding participants were given the opportunity to provide free-text comments to support their answer choice including those who answered Yes or N/A which have been categorised into 11 common themes. Did you feel you were provided with informative feedback? Provide more informative feedback 376 More transparancy of Panel decision 187 Provide feedback quicker No feedback provided Inconsistent policy across schemes Lack of Panel or Reviewer expertise Provide feedback on successful applications Unclear PI responses taken into account Broaden invites to re-apply Feel that feedback has limited use Feedback generally informative 76 No Trend Identified Volume of comments A total of 849 (49%) respondents submitted comments and the overriding factor, with 563 (66%) of the responses, is the need for more informative feedback along with transparency of the final panel decision. This included those respondents who answered 'Yes, they were provided with informative feedback'. 10

14 The responses indicated the feeling that the current feedback, when provided, tends to be brief and generic in nature and does not help in making future applications better with the comments very much following the lines of "Receiving a transcript of a few lines of vague and unsubstantiated text from the minutes of the panel meeting is certainly not what one expects after having undertaken a big effort of putting together a strong and well-developed research funding application." and also "The feedback would have been even more useful if constructive comments and suggestions for improvement were included" along with the viewpoint "your grant proposal was not successful" is a result, not feedback." With relation to the final panel decision the responses also highlighted the applicants' need to understand how the panel reached their funding decision, particularly when high scoring reviewer comments had been received, with various comments along the lines of "You get a good feeling about how 'good' the proposal is from the reviewers comments. However, the feedback letter is the only way to get any information about how the proposal faired at panel. The feedback is often very weak in this regard with often too little specific information to be useful. This is particularly frustrating when trying to improve proposals that were rejected after receiving positive or refutable reviewers comments." Coupled with similar comments such as "No feedback except that it wasn't ranked highly enough. For example, I'd like to know was it close to funding? Was it considered fundable but there just wasn't enough money? Did the panel have any comments? There's a big difference between "sorry it wasn't ranked highly enough" and "we considered it fundable but unfortunately there wasn't enough money to fund it."" along with "It would be useful to know how many proposals went to the board, how many got funded and where your proposal was ranked, in order to know whether to continue working on that area with view to a new submission on a similar topic, or to abandon and refocus." In contrast to this are a few comments from applicants who are also panel members and understand the limitations such feedback can give and include the comment: "For unfunded proposals the committee feedback usually repeats the comments made by the reviewers and indicates that other proposals were rated more highly. Having sat on a committee myself I accept that this is really the only feedback that can sensibly be given. Offering committee feedback seems to set high expectations of fresh insight into the decision that was reached - realistically I'm not sure that that is possible so does it serve a purpose to offer such feedback? along with "As an experienced panel chair, I am well aware that many decisions 'not to fund' are based on lack of resource, not the quality of the proposal. As such the 'reasons/feedback' put together can appear more significant than the nuances of the final ranking. More clarity about ranked position, funding cut offs and the fact that many very good proposals fall short should help here. Some consistency between panels and (re) submissions is also important, as the community can become aggrieved when responding to one set of panel reviews, only to receive a contradictory response from a new panel on a resubmission." Another factor raised was around the different approaches across the Research Councils where 55 (6%) indicated they had received differing quantity and quality of feedback depending on the scheme applied to and 86 (10%) comments advised no feedback was received at all. 76 (9%) comments were submitted advising that the feedback received was generally informative. 11

15 11. If you had a successful grant and had to make changes during the lifecycle, did you use the Grant maintenance functionality in Je-S? No, 268 Yes, 388 N/A, 1077 A total of 656 (38%) respondents identified that they had made changes during the lifecycle of their successful grant. 268 of these (41%) confirmed they did not use the grant maintenance functionality and were asked to provide free-text comments under question 12 to support their answer. Of the 1,733 respondents who answered this question, 1,077 (62%) selected N/A which indicates either the respondent had a successful grant but did not need to make changes during it's lifecycle or alternatively, the respondent wasn't successful with their grant proposal and therefore did not have a need for the grant maintenance function. 12. Please could you tell us why you decided not to use the Grant Maintenance functionality? Not aware of facility Relied on Research Office/Council to make changes (40%) of the 268 respondents who indicated they did not use the grant maintenance functionality within Je-S submitted free-text comments and these have been categorised into 5 key themes. No changes to make Too difficult to access Changes too complex No Trend Identified Volume of comments Similarly to the Je-S tracking functionality discussed earlier in question 8, the highest factor with 136 (53%), indicates that applicants are not aware of this facility and this again suggests there is more work to be done to determine end user awareness of available functions within Je-S. 66 (26%) of the comments advised that applicants relied on their Research Office/Council to make the amendments, either because it was deemed to be their responsibility to do so or because the applicant needed to discuss the changes before any updates could be made. 12

16 Section 5: Post - Award 13. When needing to request changes to your grants, post award, how satisfied have you been with each of the following?: 100% 90% Share of responses 80% 70% 60% 50% 40% 30% 20% 10% 0% Policy/Research funding guidance How to use Je-S The 656 applicants who indicated they had made a change during the lifecycle of their successful grant were asked about the functionality of making changes to the grant once the award had commenced. 650 applicants (99%) answered this question and it is encouraging to see that overall 58% of these respondents were either satisfied or very satisfied with the areas in question. 32% indicated overall that they were neither satisfied nor dissatisfied which may relate to applicants who have not yet had the need to make post award changes. Evidence of this is highlighted to some degree in the subsequent free- text comments submitted which are detailed below. 196 (30%) respondents took up the invitation to submit free-text comments to support their answer, providing various views which give some insight into the above results. These have been categorised into 7 common themes. 267 Timeliness of responses Very satisfied Satisfied Neither satisfied nor dissatisfied Dissatisfied Very dissatisfied Slow response time 62 Unclear Guidance/Policies 42 Direct contact very helpful 31 Process worked fine 19 Changes too complex for Je-S 13 Errors made with requested changes 3 No post award changes made yet 42 No Trend Identified Volume of comments As reported above, some respondents have not yet had to make post award changes to their grant and this is corroborated by the 42 (21%) comments submitted highlighting this. 13

17 POSITIVITY: 19 (10%) of the respondents felt, in their experience, that the process for post award changes had worked fine. There were also 31 (16%) comments praising the helpfulness of staff when applicants made direct contact for assistance with their grant changes, however, this contact was sometimes as a result of a failure within the change process. TIMESCALES: The highest factor with 62 (32%) comments refers to the length of time taken to provide a response or a decision to a grant change with such comments as "Getting a response back about changes can be very slow and it is often very important to have them back quickly". One particular process raised in this category relates to no-cost extensions with several comments along the lines of "...it took 2 1/2 months to get a response -- not a good turnaround time in the context of a 12-month project." Additionally, two comments were submitted with reference to the communication of responses and included; "responses to requests for changes post-award are routed back ONLY to the approver pool and not directly to the PI. I would like a direct response to any requests I make please." GUIDANCE: There were 42 (21%) comments highlighting gaps in the available guidance; "The appropriate guidance on what is and is not possible is not easily available on Je-S. I have to contact the Research councils directly." or lack of understanding with certain policies, where again the subject of extensions was raised with such comments as "There seems to be no link between what programme managers at the Research Council say should qualify for an extension and what response SBS give with respect to declining or agreeing to no-cost extensions: it led to huge confusion and different grants being treated very differently - some extended and some not even though similar reasons given." Two other grant change subjects raised in this category related firstly to maternity leave and included various comments along the lines of "Guidance for how to deal with the grant during periods of maternity leave was not clear. I received different advice on personal contact with the research council (in one case, told to request the grant be put in abeyance; on another case, told to request an extension to the grant). It was not clear to me if there were official guidelines to cover these circumstances." and secondly to the movement of investigators between institutes with several comments like "There was little helpful information on how to transfer a grant from one institution to another. There was helpful Council guidance (i.e. they said I could do this), but it proved difficult in Je-S. The problem was a simple one: I couldn't be listed at two universities at once, so couldn't transfer the grant to myself." "Getting a response back about changes can be very slow and it is often very important to have them back quickly". 14

18 Section 6: Summary 14. How would you rate your experience as an Applicant for Research Council funding (where 0 is extremely dissatisfied and 10 is extremely satisfied)? 6.2 is the average score of how the respondents rated their experience as an Applicant of Research Council funding. This is on a scale of 0 to 10, where 0 equals extremely dissatisfied and 10 equals extremely satisfied Volume Score 0 = extremely dissatisfied 10 = extremely satisfied Of the 1,752 respondents who completed the survey, 1,725 (98.5%) went on to indicate how they would rate their experience as an Applicant of Research Council funding. This is on a scale of 0 to 10, where 0 equals extremely dissatisfied and 10 equals extremely satisfied. There is an even split across the scores with 537 (31%) of respondents giving an overall score of 8, 9 or 10, scores of 6 and 7 being submitted by 657 (38%) respondents and the remaining 531 (31%) providing a score of 5 or below. Through analysing the data available the volume of proposals submitted by an applicant does not appear to effect their overall experience of the submission process and the issues raised by the lower scoring respondents relate to a lack of transparancy of the whole decision making process along with the apparant lack of support for innovative research. Whereas the higher scoring responses acknowledge the support personnel and feel the overall process works well. The success or failure of a particpant's grant proposal, which has not been clarified within the survey, may affect the responses and scores submitted. Further insight into the views of the respondents can be seen in greater detail in question 15 below. 15

19 15. Please provide any other comments relating to your experience as a Research Council applicant not covered elsewhere in the survey. Of the 1,791 respondents who completed the survey, a total of 573 (32%) submitted various comments which have been categorised into 15 common themes. More accountability of reviewers and panel decision 130 Speed up the whole decision process 82 Provide more informative feedback Increase support for innovative projects Streamline the application form More customer care for applicants Staff very helpful Unintuitive Je-S Use more of a 2- stage approach Increase resubmission option Provide higher success rates/ increase available funds Use of impersonal communication Unclear guidance on multi discipline applications System issues with final reporting 7 9 Overall process works well 59 No trend identified Volume of comments POSITIVITY: 59 (10%) respondents consider that the overall grant proposal process works well with such comments as "Je-S usability and reliability has improved enormously since it was introduced. Now we know the system, its use simplifies the application procedures." and also 31 (5%) respondents have found the Research Council and UK SBS staff very helpful when contacted for assistance. TRANSPARENCY: Several of the themes identified from the comments submitted at this stage of the survey relate to issues highlighted earlier in this report including the highest factor requesting 'More accountability of reviewers and panel decisions' with 130 (23%) comments, which refers to the expertise of reviewers and standard of reviews submitted along with the request for more transparency of the final panel decision which can often conflict the reviewer scores. The comments included follow the lines of "Quality of reviewing - still sometimes get illinformed, incorrect or malicious reviews. Difference between Research Councils - sometimes panel rereviews proposal, sometimes they are not supposed to do so. I believe applicants should always have opportunity to respond to reviews and so panels should not re-review proposals." along with "Feedback following grant assessment and decision making is essential and more explanation of how the committee reached its decision as there is often a lack of correlation between referees comments and scores and actual outcome. This gives the impression that the introducing members ignore referee comments." This is further supported by the 60 (11%) comments categorised as 'Provide more informative feedback', which again has arisen previously in this report and includes; "The complete lack of feedback after a decision was made on my application was very disappointing." along with a constructive response of; "It'd be useful to require that reviewers provide practical suggestions/recommendations of the areas in which the proposals they assess could be improved." 16

20 RESUBMISSION: In association with this, is the factor categorised as 'Increase resubmission option' with 21 (4%) comments highlighting the following view: "By far the biggest problem with the Research Council funding process is the policy that prohibits resubmission of a grant application. The rate of funding is simply too low to make this policy workable. There is a great deal of highly reviewed, scientifically sound and strategically worthwhile research that does not get funded on each cycle. It makes no sense to prohibit revision and resubmission of a grant application under these circumstances." TIMESCALE: The second highest category, 'Speed up the whole decision process' with 82 (14%) comments relates both to the overall time taken from the initial submission of a grant application to the final funding decision being announced and also the delay between the funding decision and the actual start of the award payments and includes such comments as: "The grant process took much longer than expected. The announcement of awards was quite delayed and once I was informed that I had received an award, it took a very long time to actually get the grant to my university." along with "The main issue is the length of time the entire application process takes, which is an obstacle to working with non-hei partners who may be making budget / timing decisions." INNOVATION: An interesting factor identified in the responses is the category 'Increase support for innovative projects' with 60 (11%) comments highlighting the opinions and difficulties experienced by applicants with novel ideas and includes various comments along the lines of "It is essential to retain adequate funds for smaller (circa 100K- 250K) grants for new original work that does not necessarily fit into identified grand themes and large teams. Otherwise inventiveness will be stifled." STREAMLINE: Two further categories, also identified earlier in this report, are 'Streamline the application form' (42, 7%) and 'Use more of a 2-stage approach (24, 4%). The first relates to repetitive and redundant areas of the application form and also a rather negative view for the need of the Impact of Pathway type papers with several comments along the lines of "I feel the sections related to impact could be made more streamlined. The need for separate, and rather lengthy, separate statements on "Impact Summary" and "Academic Beneficiaries", in addition to the "Pathways to Impact" statement, means that there is often a significant amount of overlap in the information requested." The second of these categories correlates with the above making the suggestion of adopting a 2-stage approach to grant proposal submissions as explained in the comment - "Writing grants is taking up a great deal of time, often with numerous additional documents. Much of the information is only relevant if the grant is awarded - why not reduce the burden on applicants and just require essential scientific and approximate costings for review and then just obtain all the other details if the grant is awarded?" CUSTOMER CARE: A further theme identified, with 40 (7%) comments refers to 'More customer care for Applicants' and relates to the handling of applicants' needs outside of the standard guidelines and help functions and includes such comments as "My proposal seemed to have been lost in the system for a while. During this time, it was very hard to find out what was going on." and also "I received an saying I was successful, which was very exciting. Ten minutes later I received another saying I was unsuccessful and it had been a mix-up. This was absolutely heart-breaking and really poor performance by the RC." along with a suggestion of "Would be nice to have an NSF-style area leader who is directly responsible for calls in a particular area and involved in the final decision who we could liaise with prior to grant submission." and finally "Surveys like this are a welcome chance to give feedback, so please do continue this exercise in the future." SUMMARY: Overall, the survey results highlight that customers clearly feel there is a need to place a stronger focus on the effectiveness of existing peer review processes and the transparency of the final funding decisions and effective feedback. The results also reveal the need to raise the profile of grant functionalities within Je-S and to also place a stronger focus on the guidance and support provided to enable a better customer experience. 17

21 Section 7: Agreed Actions The survey of Grant Applicants took place in October 2013 and had a 26.3% response rate. Proposed actions are: Agreed Actions in response to the Grant Applicant Survey Report. 1. Revisit how we can improve Je-S Helptext. The Research Councils and UK SBS are to review how guidance is provided between JeS Help, funding guides and scheme/call specific text to ensure that appropriate guidance is available in the most appropriate place(s). 2. Je-S Training. Although the Research Councils and UK SBS are unable to commit to resuming a programme of onsite training, we will endeavor to look at how to provide better online training tools e.g. by the use of video. 3. Improvement to proposal feedback. Research Councils have already signed up to improving transparency of peer review by agreeing to share either reviewer comments or a written feedback note. To help improve the latter, Research Councils will discuss and implement best practice to try and ensure more structured feedback is provided. 4. Research Councils will ensure that the PI response page limit is treated consistently within each call to ensure a level playing field for all applicants. 5. Research Councils and UK SBS to look at response times to maintenance requests to see whether the perception that they always take a long time to be dealt with is supported or whether we need to better manage expectations. 6. Research Councils and UK SBS to make better use of existing communication routes to include items on using status reporting/pre award tracker and post award maintenance expected resolution times. 7. Review JeS forms. Research Councils will review their JeS proposal forms to look at simplifying and streamlining. 8. Feedback to the survey responders (on the RCUK website where the reports are published) 18