Customer Satisfaction Survey

Transcription

1 Customer Satisfaction Survey July 2010 Research and Evaluation Team Page 1

2 Page 2

3 CONTENTS INTRODUCTION...4 Research Questions and Strategy...4 Sampling...5 Analysis...6 FINDINGS How do programmes experience applying for funding from Youth Music? How do programmes experience the Youth Music assessment process? How do programmes experience holding a Youth Music grant?...13 CONCLUSION...15 YOUTH MUSIC RESPONSE...18 Page 3

4 INTRODUCTION The aim of this report is to undertake a basic, internally-focused assessment of the levels of satisfaction that our applicants experience in applying for, and administrating, a Youth Music grant. The main tool used to achieve this was an online customer satisfaction survey, sent to a sample of all Youth Music programmes in April The results presented here will be used to inform delivery of funding and support by Youth Music to their programmes in the coming year. Research Questions and Strategy The primary question addressed in this report asks: To what extent are applicants for Youth Music funding satisfied with the funding procedure and support received over the course of their application to Youth Music? Three secondary research questions were formulated to examine levels of customer satisfaction in further detail. They were designed to ask respondents about their own experiences, specifically as they relate to the various stages of the applications process. The first asks about their experience of applying to Youth Music, the second about their experience of the Youth Music assessment procedure, and the third about the experience of holding a grant in a successful application. In summary, they are as follows: 1. How do grant applicants experience applying for funding from Youth Music? 2. How do grant applicants experience the Youth Music assessment process? 3. How do successful applicants experience holding a Youth Music grant? Customers were considered by the survey to be either those programmes currently holding an active grant with Youth Music, or those having made an application for a grant from Youth Music in the last financial year ( ). An online questionnaire was designed to test customer satisfaction across these three key stages. Customers were asked to rate their levels of satisfaction using responses to a range of focused questions. Provision was also made in the questionnaire to include Page 4

5 subjective responses on particular areas of the application process, and how customers feel they might be improved upon. Subjective responses were deemed to be particularly useful where customers were not satisfied, or had encountered challenges and problems with the existing system. Sampling 435 cases across all Youth Music programmes were identified for participation in the survey (see Figure 1): 1 Figure 1 - Table showing response rates to the Customer Satisfaction Survey 2010 Total cases Number of Responses Response Rate (%) All active grantees MusicLeader NYMO PowerPlay Voltage YMAZ Youth Music Mentors All active grantees, plus unsuccessful grant applicants from 09/10 Open Programmes Overall The survey itself was hosted using the online customer survey tools provided by Constant Contact (see All cases of the target group were ed by the Youth Music programmes team with an invitation to participate in the survey and a link to the specific Constant Contact page intended for completing it. Whilst the response rate by particular programmes was low (Voltage, Youth Music Mentors), the overall response rate is reasonable for a survey this size. The low total number of respondents (n=62) means that statistically reliable results within particular programmes are not obtainable; however, trends and implications across and within 1 Some of these cases will have received the survey twice, as they are involved in more than one programme. This goes some way to explaining why response rates for some programmes were low, as the grantee responded under their main programme of activity 2 8 of these respondents (25%) were unsuccessful applicants Page 5

6 programmes are discussed. Youth Music s flagship programmes - (Open Programmes, Music Leader, Youth Music Action Zones) had the highest number of responses, while all other programmes had a response of less than 5 cases, and in some cases only one or two responses. Much of the survey was general enough to seek responses that would be applicable across the programmes. However, some individual tailoring and adjustment had been undertaken to fit the survey to particular programmes, given their diverse nature. Applicants to the Open Programmes, an unsolicited fund and flagship Youth Music programme, were also asked to provide some further information about what sort of organisations they are, including who they employ and so on. However, because the small sample size for this group, it is not possible to make any meaningful inferences from this data, and therefore it has not been included in this report. Analysis The Constant Contact service was used to collate the raw data from the survey responses, and to make these available as downloadable outputs in.pdf,.csv and.xls formats for further analysis. The outputs used here are summaries sorted by programme, though Constant Contact also has the capability to download raw data for each individual case. Ideally, the analysis of the survey results would have taken a mixedmethods approach, involving the triangulation of statistical and subjective measures of customer satisfaction by each individual case in order to draw out key findings. However, given that slightly different versions of the survey were used for each programme, combining the data using Statistical Package for the Social Sciences (SPSS) proved unworkable. This prevented a reliable quantitative interrogation of the responses, which would have offered both a more general understanding of the satisfaction levels experienced by all YM programmes, and a means to draw comparisons within and between them. Therefore, given that the primary purpose of this report is to discuss the implications of customer satisfaction as it is expressed at an organisational level, queries that are related to a specific aspect of an individual programme should refer to the programme officer for further details. This mixed response rate, combined with individualised versions of the survey, has therefore meant that the results discussed below are taken to be more illustrative and Page 6

7 guiding, rather than representative and statistically significant in a scientific sense. In other words, while it may be desirable to use the survey to say that, for example, 70% of respondents were extremely satisfied with the new application forms, or that the provision of guidance material on an Outcomes Based approach received a rating score of 6.7, the low number of actual responses render such quantifiable statements relatively meaningless. Instead, qualitative measures of customer satisfaction are the main analytical focus in this report, with summary statistics offering supplementary insights. Indeed, these aspects of the survey do offer a useful gauge of customer satisfaction, with some key findings expressed at both organisational and programme-specific levels. The subjective responses from the raw data received were extracted and compiled in a Word document. These have been organised by question, and subjected to systematic coding to gain a sense of recurring themes, pertinent questions, comments and suggestions. Overall, the emphasis of analysis undertaken here is on drawing out key points of critical learning, where Youth Music processes and project delivery might be further improved. Page 7

8 FINDINGS 1. How do programmes experience applying for funding from Youth Music? In most cases, respondents had not contacted a member of Youth Music staff for further clarification about the application guidelines. This suggests that the guidance material already provided by Youth Music was sufficient to draft an application. Those applicants who did contact Youth Music had a number of small and focused queries, and generally felt these to have been resolved in doing so. While these were often very specific points of clarification, they tended to concern budgeting, suitability of proposed projects, documents to be included with the application, feedback on unsuccessful applications and deadlines for submission. While most respondents from the Open Programmes and Music Leader had not experienced any technical difficulties with the online application form, most YMAZ respondents reported experiencing problems. It may be that this group has a particular need for further support in the use of online reporting and submissions. Across all groups, the kinds of problems that were reported had a number of similarities. Specifically: the use of boxes was inflexible saving in a usable format was too complicated designated word limits were too restrictive the forms used had an element of repetition Over half of all respondents had made use of the Youth Music helpsheets, and were generally very positive about their value in supporting the drafting of their proposals. For all groups, the Outcomes Guide was most used, Involving CYP was well used, and Mentoring Guidance was particularly useful to the Open Programmes group, though all forms had been used to some extent. A number of recurring suggestions for new helpsheet topics were raised, broadly in the areas of evaluation, budgeting, Youth Music assessment criteria and training provision/providers. The majority of applications from respondents had drawn on the support offered by their Youth Music Regional Executive Officer (REO). Their role was generally very positively received, and praised for being proactive, encouraging, genuinely Page 8

9 interested and of the highest standard. The nature of the REO support tended to be focused around providing feedback on draft applications and re-submissions, though a broader and more informal sense of support from the REOs was clearly provided and welcomed. However, there were a number of respondents who felt that there was a level of contradiction between Youth Music head office staff and the REOs. As one Music Leader network commented, We received feedback on a draft of our application which proved very useful. However, it slightly contradicted some of the feedback we received from One America Street staff and on discussion with our REO we found out that they had not been aware that we would also receive feedback from another member of the Youth Music team ; while a YMAZ noted that I felt there were minor contradictions within the feedback from REO and YM. Whilst not a problem it did suggest to me that there is a difference between having a regional perspective and a national perspective and this I feel needs to be addressed prior to feedback time. The discrepancy between national and regional processes is clearly an issue that needs to be addressed. While a majority of both Music Leader and YMAZ applicants who responded had received support from Youth Music head office staff in making their application, most Open Programmes respondents had not. Nonetheless, those who had received support were very positive about the experience, and expressed a strong sense of Youth Music staff being informative, supportive and clear in their response to queries. In particular, having head office staff always on hand, and their willingness to respond to both simple and complex problems, was highly valued. As a result, respondents were generally happy with current provision of support and did not feel there were significant aspects of support lacking. However, some further needs were identified by several respondents that would be worthy of further consideration, including unreturned calls and queries being frustrating, and a need for clearer and more explicit assessment criteria. A number of other comments and suggestions for improving the application process were also made by respondents. In particular, the desire for the application forms to be shorter and simpler was often raised. Several respondents also felt that the structured nature of forms might inhibit more intuitive styles of running projects, and they suggested more flexibility and scope for innovation be worked into their design. However, this perspective was contrasted by a number of other respondents, who Page 9

10 pointed out that, the form is long and complex but made me think clearly about what I was requesting, and similarly, that the application process felt more dynamic than in previous phases. It was challenging, but the old version allowed for more wooliness. In this sense, while a detailed application form might require a higher level of concentration and time committed, it also appears to help focus proposed projects at an early stage. There were also several individual points made that may be worth following up. One YMAZ made a point about the timing of decisions on applications coming very late in the year for future planning. Another YMAZ pointed out that applications from existing projects such as theirs found themselves re-explaining how they met the aims of Youth Music. These sentiments were echoed in several other responses, and signal what was described as a need to recognise existing relationships and account for them in the application process, which again suggests the importance of reliable and trusting relationships between Youth Music head office and our funded partners. Page 10

11 2. How do programmes experience the Youth Music assessment process? Most respondents had found the Youth Music assessment process relatively clear, and that the stated timescale for decisions on funding from Youth Music was adhered to as agreed. While timing on decisions was felt to be ok, timing on the release of funding was more problematic. One Open Programme reported that It was indicated that funding would be granted in Feb; in fact decision was March; funding released in April. This affected intended delivery of the project, while another made clear that Decision time was efficient but actual payment was delayed which caused logistical problems for the start of the project. These seem like minor procedural hold-ups that could be easily rectified. It might also be worth including a question in the 2011 survey that asks about the timing of funding as well as the timing of decisions. Most respondents also expressed clear satisfaction with the feedback received from Youth Music after receiving their initial application. Those expressing levels of dissatisfaction in the above considerations, or experiencing problems with their implementation, made most use of the comments section in the survey, and thus, are focused on here. One recurring criticism from those whose funding bid had been rejected, or who had received a request for re-submission, was that the feedback they received from Youth Music was too general, even standardised, and lacking sufficient detail for applicants to understand the reasons behind the decision. One Open Programme pointed out that our feedback simply said that some areas needed developing, but did not explain further. It would be enormously helpful to have more information on exactly what was undeveloped so we could either make changes to the project, or explain more clearly what we do, and another even reported that No feedback was given initially until the regional officer was approached when we first applied. This was disappointing and initially made it difficult for us to re-apply with any confidence. A third Open Programme was more openly critical: We felt the reasons behind failing our project were not entirely honest- in that the letter received was a general letter sent out to all failing groups. When we asked for further information it was concise, lacking in the relevant detail and very contradictory. Again, while the suggestion of dishonesty in the process is quite unfair and certainly uncommon, the frustration experienced by projects when the clarity and consistency of information provided breaks down should be noted. Moreover, the resulting criticism draws Page 11

12 attention to the potential damage to Youth Music s reputation that might occur if programmes feel that the specific details of their application are overlooked. Also, as noted in the responses on helpsheets above, there was some level of need for more clarity on the specific assessment criteria used by Youth Music. A good suggestion made by one of the Open Programmes, which would also go some way towards heading-off the more extreme criticisms from rejected bids, was that It may be helpful to see which sections on the application form hold the most weight and whether there are particular areas that Youth Music would like to develop or that currently do not have Youth Music provision. These related concerns would seem worth considering. Another recurring criticism was a sense of disconnection between the information held by Youth Music head office and our external assessors. One Open Programme noted that, I felt the answers to some of the questions asked were in the application ; while a Music Leader network noted that the assessor did not seem to have the information about the set-up phase that we had sent in. Some of the questions that she had were already covered in what we had submitted but she did not seem to have that information. Similarly, one of the YMAZ schemes pointed out that the YMAZ application process does not seem to make allowance for Youth Music's head office staff knowledge of the project or applicant, leading to frustration that information requested in formal 'Special Conditions' has long been held by head office. Related to this was some specific criticism of the external assessors. One Music Leader network complained that some of the assessment queries assumed a very limited knowledge of project management and the assessor should have researched the organisation better. Another more critical respondent felt that head office staff should get out of your offices and undertake more site visits, which they felt would give a real understanding. While this is perhaps unrealistic given the nature of our role, such sentiments signal the reliance Youth Music places on funded partners in developing effective relationships, and the importance of consistent and clear information circulating amongst them (see point in previous section about relationships). Page 12

13 3. How do programmes experience holding a Youth Music grant? Most respondents who had special conditions attached to the provision of their grant had found these to be understandable, clear, reasonable and helpful. However, there was a specific point made by a number of YMAZ schemes, who were critical of what they felt to be a lack of shared knowledge and the top-down nature of such conditions. For example, one noted that Some did not relate to the application and Youth Music could have gained the information from interim reports, while another felt that our specific conditions seemed to respond to a dearth of knowledge held by head office rather than intended to stretch the applicant to achieve a stronger project. A third YMAZ scheme commented that, I think this is where the weaknesses in the process appear. Special conditions I feel should be defined in consultation with the funded organisation as there are always reasons for decisions made/challenges encountered etc. Furthermore, there was some overlap in the wider responses over assessment and feedback, and those YMAZ projects commenting here specifically about special conditions. For example one YMAZ commented that I steered away from addressing [a specific strategic goal] for reasons that were clearly outlined in the application. I feel that either my reasoning was ignored or not considered acceptable. Either way no feedback has been provided apart from a demand made within the special conditions with regards to addressing this area as part of the work plan. However, given the mixed response rate achieved in this survey, it is hard to tell if these are problems specific to the YMAZ scheme or more generally experienced. In general, Youth Music's interim and final report forms were felt by respondents to allow them to reflect on how their projects had progressed; though many of the Open Programmes reported that, given the recent introduction of such monitoring for their programme, they were as yet to complete these. Satisfaction with the new reporting system was especially clear for the Music Leader group, who were very positive, with one noting I'm impressed and heartened by the MusicLeader reporting templates for Phase 4. They feel clearer and more robust, even though there is less to cover. It will drive some new systems and some changes of practice in our region, for the better. There was also some constructive criticism raised on this issue. One project pointed out that it was challenging for the timing of reporting to occur outside of the financial and Page 13

14 academic years. Another useful suggestion was to allow for feedback from programme participants in the report forms. Indeed, a key way Youth Music head office could better support both marketing and advocacy work would be to centrally manage the circulation of compelling stories and messages received from the programmes. This seems like an especially good point to consider, because the need for exemplary case studies is high at this stage in our strategic operations for all Youth Music departments. Nonetheless, a significant minority of respondents complained that the forms for interim and final reports were both too detailed and restricted by word limits, a somewhat contrary finding, but nonetheless one that signals a need to further clarify the rationale behind the design and intention of the reporting process. Similarly, the idea that Statistics could be collated only at the end and the sometimes repetitive narrative questions could also be covered in one report at the end suggests that methods of evaluation need to be clarified (i.e. the need to establish a baseline in order to measure impacts effectively over time). These recurring criticisms need to be addressed in order to help programmes understand and support best practice in this area. The marketing obligations attached to the holding of Youth Music funding were generally felt by respondents to be easy to implement. However, some respondents expressed problems with the clarity of marketing goals in projects working with multiple partners (and therefore multiple brand identities, logos and so on). For example, one Music Leader network pointed out that they felt marketing was Fine on the ML front but that they feel slightly more removed from YM overall Marketing goals. This seems like a pertinent and timely issue for consideration by External Relations, given that an assessment of brand awareness is currently being undertaken by them. In terms of how programmes actually communicate with young people, a number of other means were reiterated by respondents, particularly telephone calls and face-toface contact. With the significant move towards virtual networks and web 2.0 resources, it is telling that our partner agencies still value and utilise these direct forms of inter-personal communication. It is also telling that these key modes of communication were absent from the survey. Their sustained use by front-line staff echoes the sentiments expressed about head office staff needing to get out of their offices in order to gain a real understanding, and is a reminder of the importance of these kinds of more overtly human aspects in the delivery of effective Youth Music programmes. Page 14

15 CONCLUSION In conclusion, the main finding from the survey has been that all programmes were broadly satisfied, positive and even enthusiastic with the funding procedure and support received over the course of their application to Youth Music. There were no aspects of our operations that were flagged up as being especially problematic or failing our customers. This is an important concluding point to take away and hold close in learning from this exercise, and for pursuing our strategic goals in the future. It is also important to bear in mind that, given the practical problems with the survey as undertaken, it is not possible to draw statistically significant conclusions, and only possible to synthesise a small body of cases and anecdotal commentary. While an effort has been made to include praise and support of this kind, it was considered more useful to unpack insights on how we might further improve our processes. While these critical learning points are the focus of detailed discussion in this report, such an approach is driven by the need to improve our already successful programmes, rather than highlight areas of significant deficiency. Figure 2 below summarises these key learning points, and makes a number of recommendations for addressing them in the future. Page 15

16 Figure 2 - Key Recommendations Critical Learning Point 1 Unsuccessful applicants who received standardised feedback felt that the details of their own particular project were overlooked. Recommendation Implement individualised feedback for unsuccessful grant applicants. Create an assessment criteria helpsheet for each programme. This is related to what applicants felt to be a lack of clarity on Youth Music s assessment criteria. 2 Respondents expressing difficulty in understanding the rationale behind Youth Music s application and reporting requirements. Update the Outcomes Guide helpsheet to explain exactly why this approach is undertaken, and not just how to do it by end September Contradictions between advice Recognise existing relationships. from Youth Music head office staff and the REOs. This also relates to apparent disconnections between head office and external assessors. Address balance between national and regional approaches and ensure consistency of information between Head Office, REOs and external assessors through better internal communication. More clearly define the role of head office staff and those played by regional and front-line staff. 4 Problems with the clarity of marketing goals in projects working with multiple partners and brand identities. Development of brand guidelines to give more detailed guidance about the use of Youth Music s branding to funded programmes. MusicLeader and Youth Music brand guidelines being developed to align the two brands more closely. Marketing section of funding agreement to be expanded to include an overview of YM s marketing strategy, to give funded programmes more of an insight into Youth Music s goals in this area and an understanding of where they fit into the national picture. 5 Delays in the release of funding These seem like minor procedural holdups that could be easily rectified through clearer guidance as to Youth Music s grant giving timescales and conditions. Ensure that grant-giving timescales are clearly communicated. Include a question in the 2011 survey that asks about the timing of funding as well as decisions, in order to confirm this has been resolved. Ensure that funding decisions for solicited programmes are made as early as possible to support the following year s planning. Page 16

17 6 Reliance on web-based communications is not conducive to the development of funder-fundee relationships 7 Running the survey has been a problem a) due to low response, which has meant that it is not possible to draw statistically significant conclusions; and b) because the varied format of the survey to suit each programme has meant that combining responses at a programme-wide level in SPSS is unworkable. Recognise continued value of face-toface and telephone communications between head office staff and funded partners, with site visits where possible to ensure understanding of how projects operate on the ground. Consider the balance between our desire to ask specific, targeted questions to our rather diverse array of projects, against a need for comparable data to achieve meaningful results for the next survey in April Develop a standardised survey for all programmes with opt in/out sections where questions may not apply. Ideally, would also recommend. upgrading SPSS software suit to include the tools necessary to conduct online surveys that feed directly into the analysis software. Page 17

18 YOUTH MUSIC RESPONSE Thank you to everyone who took the time to respond to the survey. The purpose of the survey was to see how we measure up in some key areas of our work, and to identify how we can improve the services we offer. The feedback you have provided has enabled us to draw some useful conclusions and recommendations, which will be taken forward within the organisation. It will help us to refine our grant processes for the coming year for example, it will feed into the review of the Open Programme that is currently taking place. In addition, the information we have gathered will form a useful baseline from which to review our progress when we repeat the survey next year. We have formed a cross-departmental working group to address the recommendations proposed in the survey, and have already identified a number of actions that need to be taken forward. We know, for example, that we need to ensure that we are giving consistent messages and clear information to our grant applicants and those who then receive grants. Over the coming months, you should be able to see progress being made towards the recommendations proposed. Thanks once again for your valuable feedback. We look forward to acting on the recommendations of the report to improve our services to you. Nick Howdle, Director of Programmes Page 18