Final evaluation report

Transcription

1 Final evaluation report MSc Health Policy provided for the Department of Health by Imperial College London Liz Carolan

2 Summary The evaluation gives an overall positive impression of the MSc Health Policy. It is seen by Department of Health (DH) leaders as adding value to the department by meeting most of the learning objectives for this cohort of high-potential individuals. It appears that the high-quality and relevance of the programme may be positively affecting commitment to the health sector among the participants. In a survey at the end of the programme, 94% of the participants said they would recommend the course to a friend, and 87% said it had improved their ability to do their day job. The participants felt that they had acquired the specialist knowledge of health and the tools to evaluate and use evidence that the department hoped they would. There is a very strong sense that the participants initial view that they were being invested in by their employer has not diminished over time, in large part due to the high quality of the programme. There is also an appreciation on the part of both participants and their managers that time away from day jobs to think and build a network is valuable to the teams as well as the individuals. There are some early signs that this is beginning to translate into benefits to the participants performance in their roles. One of the main objectives that the department set for the programme was that it would increase commitment to the wider health field. This appears to have been met, with 94% agreeing with the statement, As a result of doing this course, I am more likely to stay working in the health field. Broader application of the skills and knowledge to day-to-day work was a little bit less obvious to some of those we spoke to. On that point, however, neither the participants nor the line managers we interviewed were overly concerned. Instead, they emphasised the fact that this was a long-term investment in talent, and that signalling this particular aspect of the programme was valuable. As one line manager put it: I saw it more to show we are willing to respond to a high potential star s request for something to meet what he saw as his training needs. While both the participants and their line managers were on the whole positive about the programme, they did give a few caveats. The first is that participation has knock-on effects for the team both during teaching weeks and in the periods in between. While this has on the whole been managed quite well, it is especially a consideration when the benefits will predominantly come in the longer term when the individual may have moved on. The second is around individual participants needing to be mindful of the level of commitment required and the impact this will have on overall work-life balance. However on the whole we have observed that the programme appears to be meeting the objectives of the department. 1

3 Introduction This is the final report of an evaluation of the MSc Health Policy programme delivered between 2011 and 2013 for the Department of Health by Imperial College London. In 2011 the Institute for Government, an independent charity with cross-party and Whitehall governance working to increase government effectiveness, agreed to carry out an evaluation of the first cohort of MSc participants. This was to evaluate the programme as a learning intervention to improve policy-making skills in the department. The contract stated: It is expected that this will include interviews with participants and their line managers to ascertain the benefits that studying on the MSc has brought to participants day-to-day work. IfG s primary goal is to ascertain the effectiveness of a Master s course as a training intervention for civil servants. The evaluation took place in three stages. The first set out the department s expectations for what the programme would ideally deliver in terms of learning; changes in behaviour and organisational result. (See the next section for details on methodology.) The second and third stages tested whether the objectives for the participants learning were met, and looked for signs that this learning was beginning to be applied by participants when they were back in their day jobs. In each of these stages of the evaluation, risks and concerns relating to the programme were examined. This final report builds on interim findings submitted to the department in March of 2013, and summarises the findings of the three rounds of research. 2

4 1. Methodology: setting and testing expectations The evaluation used as its overarching framework the Kirkpatrick model of evaluation of training programmes. 1 The framework The evaluation began by establishing the department s expectations for what the programme would ideally achieve, using three levels of the Kirkpatrick framework. 2 This involved interviews with the head of policy profession, line managers and a focus group with the participants. The objectives set out by the department at this stage began with the organisational objectives for the programme. This corresponds to Level 4 in Kirkpatrick s framework. Leaders told us they wanted to see higher organisational morale; a professional cadre of policy makers and enhanced career options for its staff to move in and out of the health system, benefitting the department. They then outlined the changes to participant behaviour that would need to result from the programme if these organisational objectives were to be achieved. In Kirkpatrick s language, these are Level 3 objectives, and the full set of responses is outlined in Figure 1. Finally, leaders were asked to set out the skills, knowledge and attitudes the participants would need to acquire for this behaviour change to take place. These Level 2 objectives are also in Figure 1. Figure 1: The benefits the department would like to see, mapped onto the Kirkpatrick framework It is an assumption of the evaluation that the link identified by DH leaders between the organisational objectives and the Level 2 and Level 3 objectives exists. It is beyond the scale and timeframe of this evaluation to determine if this is the case. The aim, therefore, is to give the department information on which to determine if it is satisfied with the programme overall. The evaluation process The evaluation took place over a limited timescale it ran roughly in tandem to the delivery of the programme. As a result, the main focus was on establishing whether or not the Level 2 objectives were being met. That is to say, it examined whether or not the participants were acquiring the skills, knowledge and attitudes that the department felt would lead to behaviour change (Level 3) and ultimately to organisational results (Level 4). However, some early indications were gathered with regard to behaviour change. The evaluation process took place in three phases. 1 Kirkpatrick, D. & Kirkpatrick, J., Evaluating Training Programs, 3rd edition, Berrett-Koehler, Level 1: Reaction is not included here as it covers response to the programme, which is recorded by Imperial. 3

5 The first phase included the above-mentioned objective setting with department leaders, and took place during January and February of We also asked the participants to rate whether they felt the learning objectives identified by the DH leadership were: a) what they felt they needed; b) likely to be obtained from the programme. The results of this are summarised in Table 1 in the next section. The second phase, completed in December 2012 and January 2013, tested for early signs the Level 2 objectives were being achieved, and looked at whether or not these were beginning to translate into changes in the behaviour (Level 3) of the participants. An interim report was made available to the department outlining the findings at this stage. The third and final phase took place in January 2014 once the MSc had finished, and aimed to test and validate the initial findings outlined in the interim report. The second and third round of research involved a round each of interviews with line managers, and two focus groups and an online survey with the participants. In addition a member of the Institute s team has attended each of the Imperial College programme management board meetings. 4

6 2. Findings In this section we outline the findings of the evaluation in two parts. We first examine whether or not the participants and their line managers felt that the learning objectives (Level 2 in our framework) for the MSc were met. This primarily draws on research carried out in the second phase of the evaluation, which was tested in the final phase and found to be little changed. The picture that emerges is a primarily positive one, with high levels of confidence among the participants, in particular that they acquired the desired skills and knowledge. We then look at some early indicators as to whether or not learning is beginning to impact behaviours and performance on the job. The findings are again tentative but positive, in particular in relation to participants commitment and feeling of being a health expert. At the end of the programme, participants were able to give some concrete examples of where they had applied their learning. 2.1: Participant s learning The evaluation suggests that by the end of the programme, the learning objectives were broadly met, with particularly positive indications that participants are acquiring specialist health knowledge and feeling that the department is investing in them. During the first phase, the participants were asked if they had felt it likely that they would acquire the expected skills and knowledge; and during the second, they were asked if they felt they were making progress. At that second stage there were concerns that a few learning objectives were not being met in particular knowledge of policy alternatives, rigour and, to a lesser extent, research skills and thinking space. Table 1: Interim findings learning objectives Level 2: Learning The skills, knowledge and attitudes the department would like the participants to acquire Participants thought needed (Phase 1) Participants thought likely to acquire (Phase 1) Managers thought being acquired (Phase 1) Specialist knowledge of health Yes Likely Yes Yes Participants thought being acquired (Phase 2) Skills to find and evaluate Yes Likely Somewhat Somewhat evidence Knowledge of policy Yes Unlikely Somewhat No alternatives and tools Knowledge to facilitate Yes Unlikely Somewhat Yes strategic or big picture thinking Rigour associated with a No Likely - No dissertation Thinking and engaging space Somewhat Likely Yes Somewhat away from day job Feeling invested in Yes Likely Yes Yes However, the picture at the end of the evaluation suggests that by the end of the programme, these concerns were eased and the learning needs had been met. In a survey at the end of the programme, the participants were overwhelmingly confident that they had acquired much of the expected learning from the programme. See Figure 2 for details. 5

7 Figure 2: Summary of responses by participants to the question Would you say you acquired the following skills and knowledge from the programme? 100% Would you say you acquired the following skills and knowledge through the programme? 80% 60% 40% 20% 0% Specialist knowledge of health Skills to find and evaluate evidence Knowledge of policy alternatives and tools Knowledge to facilitate strategic or big picture thinking Rigour associated with a dissertation Thinking and engaging space away from day job Yes Somewhat No The learning objectives are discussed in turn shortly, and are summarised in the table below. Note that comments in blue text were made by participants, and comments in dark red came from interviews with the line managers. 2.1 Specialist knowledge of health In the earlier rounds of research in early 2012, the participants were positive that they would benefit from acquiring more specialist knowledge of health and that this programme would deliver that. At the end of the programme, none of those responding to the survey said they didn t feel they had achieved this. 62% were positive, while the remaining 38% said they had somewhat acquired this. They told us: I now feel like I know more than many others when they are speaking at meetings policy makers are trained to be blaggers and I can see through that more now. I think it is brilliant and the department should carry on it needs to build specialists. Yes, but I knew a lot, although it is stretching. I feel more like I have a set of background knowledge than any specialism. Line managers agreed, giving examples of where their colleagues had drawn on knowledge for projects, such as: We needed a presentation setting the challenges the NHS faces in a global context, and looking at how other countries have been able to face this and [my colleague] was best placed to do this and able to pull out some slides he had on his laptop. 2.2 Skills to find and evaluate evidence The participants were equally positive when we initially spoke to them that they needed, and were likely to acquire, an enhanced set of skills to find and evaluate evidence from the programme. In the second phase of the evaluation, participants gave us some examples of where they had been able to demonstrate these skills. 6

8 I now know about alternative sources of information and data such as the OECD I can find things now I could not before. [On a finance issue] I was able to quickly review the literature, and know that I didn t have to look at everything if I looked in the right place. At the end of the course, all of the participants surveyed said they felt they had achieved this learning objective, either fully (69%) or somewhat (31%). 2.3 Policy alternatives and tools One area where the participants were less confident their needs would be met by the programme was on knowledge of policy alternatives and tools, and there was a less than positive response when we spoke to them both during and after the programme. By the end of the course, this achieved the lowest proportion of positive responses from participants when asked if it had been acquired (44% said yes). However by that stage, a further 50% said that it had been somewhat achieved. They felt this was not really the role of the MSc, initially ranking it joint last in terms of what they are likely to get from the programme. They shouldn t be the ones to teach us [current thinking on policy tools]. Not really there was a few policy cycles that we know are not realistic but may be of use to non-dh. They ignored politics altogether in the neat cycles we did a presentation on this for them. Having said that, there was still some frustration from line managers, who had expected this to be covered. One line manager told us: I don t think I have seen much come through of what he has been exposed to, I have probably never heard him say oh we did this or this is how we sort that so I don t think I am getting a sense of much outside learning coming back. 2.4 Academic rigour When we first spoke to those about to begin the course, they had concerns about whether the rigour associated with the dissertation, that their line managers had said they wanted to see, was something of practical use. Mid-way through the programme, they again ranked this joint last in terms of what they felt they needed and were likely to achieve, saying: They teach us academic rigour, but we live in the real world. We need to know what is good enough, rather than academic rigour. At the end of the programme, the participants were clear that they had acquired this skill when asked three quarters gave it a yes, and no one said no. The concerns over the usefulness of this part of the programme continued. They told us: To be honest our day jobs do not entail work like a dissertation. I've consistently felt throughout the course that the primary research element of the dissertation is unlikely to apply to our civil service roles, given that we work in teams with analytical support. Unless we wish to retrain as analysts or in academia then I believe that conducting our own research is quite a different skill base to what we are required to do in our civil service careers. 7

9 While echoing the sentiment above, some participants did note that the process of undertaking the dissertation had brought benefits in less obvious ways. Less opportunity to apply the rigour associated with developing my dissertation, although scoping and researching my own project has given me additional confidence in my day job. I think the methodology side of the course is useful and necessary but not necessarily something that will be fully replicated in the Whitehall environment. It is however useful when working with analysts and when producing research papers, impact assessments where there is the time and space for such an approach. 2.5 Strategic or big picture thinking The participants were positive about the big picture perspective they were getting of the health system. In the survey at the end of the programme, 81% were certain this objective had been met, while the remainder thought it had been somewhat met. Their responses suggest that this was closer aligned to knowledge than to skills. When asked what their favourite aspect of the programme had been, one participant commented: The most positive aspect thus far has been exposure to a wide range of new health areas in a short period of time you would need to spend 10 years in the department to get a similar coverage of areas. However, another commented that the programme had not covered skills on this a gap which related to the previous comments on policy alternatives and tools. If anything we need skills to do this but this is not the role of the MSc. The line managers were positive about this aspect, with one in particular commenting on how the programme had allowed their colleague to begin to see connections across the health system. She is starting to understand if you get the policies right you can have an impact on lots of people inside the system to understand that how and what we do relates to this thing called health. At the end of the programme, another line manager commented that the broader understanding of the health system had not only been achieved, but was supporting their colleague to take career steps within the department. He s very committed to health and he s now thinking of other moves within the department. I think the course has made him consider a whole range of policy areas he may want to work in. It will make him a good candidate for jobs in the department, both because he s done it and because of the knowledge (of the whole system) he s now got. 2.6 Reflective time and space In the end-of-programme survey, this received the most positive responses, with 94% of respondents saying yes they had acquired this. The participants valued the time away from the department more than they had anticipated, describing the value of having space to think away from the day job. I really value having the block of time to think it suits me better than the drip, drip of and evening course would. I like the two weeks away you need it, it is difficult but necessary. Having a two week block means that it is difficult to drag you back into work. They also valued the ability to build a cohort and have conversations outside the formal aspects of the programme. 8

10 I like the time away from the day job to think about things. That and the group, and getting to think things through. Mid-way through the programme, the participants had not been so positive. They raised the ongoing difficulty of balancing study and work. The line managers we spoke to agreed that this was important and valued the time away as important for their colleagues. What is good about this course is that it does get them away and gives people time to focus you are not in the same kind of sterile environment you are working in, you are going somewhere different, meeting different people, and it does open you up to different ways of thinking. I think it s great that they are off-site and go somewhere else, I don t think it would be right if it was laid on here. 2.7 Feeling invested in In the benchmarking phase, DH leaders told us an organisational objective for the programme was to increase the commitment and retention of high potential individuals at a time when promotion and other opportunities were limited. They told us that the programme should do this by making the individuals feel like the department was investing in them and their careers. The first wave of research indicated that participants had felt invested in through being singled out and selected for participation in the MSc. The second round of interviews found that while you might expect this initial signalling effect to fade, the feeling did not diminish as the course progressed. This was in large part down to the perceived quality of the product delivered by Imperial. When we surveyed participants at the end of the programme, working for the department was still the most popular choice for their future careers (see next section). In the mid-way focus groups, the participants said that they did feel invested in, both as individuals and as a profession within the department. All those present in those sessions said they would recommend the programme to colleagues, and many already had done so. It is refreshing that the department has recognised the need to do a considerable investment. It will get what it wants back. There will be individual benefits, but... when the course is embedded for a few years there will be big collective benefits. The participants gave three reasons why the programme had made them feel invested in: it was being delivered by a prestigious institution: What was good about getting this from Imperial was the credibility, and the quality they were able to really build engagement and pull on what we knew. That it was taught by experts from outside the department: It was great to be taught by people who are real experts in these things who can talk about when they were involved in things. You cannot get this in-house you need the bunch of people sitting around discussing this even discussing it down the pub after is important. And that it was about developing the person and not just about the department I feel like I am getting something out of it personally, outside of my work the fact that it doesn t feel as if it is 100% for the department but also for my personal development is a positive for me. 9

11 2.2: Participants behaviour While it is early days, some findings have begun to emerge about whether the participants are applying their learning on the job as the Level 3 benefits identified in the first wave of research. When asked outright, none of the participants disagreed with the statement that the course improved my ability to do my job, while 88% either agreed or strongly agreed. Figure 3: Summary of responses by participants to the question Do you agree with the following statement? Of the specific behaviour the department said it would like to see, in the first round of research the most consistently mentioned was an increased commitment to the department (and by extension the health field). When we spoke to the participants mid-way through the programme, they were able to give concrete examples of how the programme had positively impacted this. Indeed each of the focus group attendees said they had considered leaving the department and that this investment had prompted them to rethink. I turned down a promotion which would have involved going elsewhere that was in large part due to the course. When I do the list of pros and cons for staying {in DH}, the MSc is very strong on there because of the quality, how much I have been learning, and that it is at Imperial. The line managers were, at that stage, also positive about the effect the programme appeared to be having on commitment, emphasising that those selected already had high levels of commitment, and that the course was acting against diminishing it. He has talked to me about moving on next year, not from the department, he is committed to the department although that is not new. All of them are of the mind-set they will be in health for a while I would think that would be of that mind-set anyway rather than becoming that because they have more expertise on the NHS. The goal appears to be less about increasing commitment, and more about maintaining commitment of high potential staff; I have no way of telling if his commitment has increased but he is definitely interested in the health field I don t see him going off to work for McKinsey. [My staff] on the course were pleased to get on it, are positive about it, and see it as a good thing, which is at odds to how people usually see training. 10

12 By the end of the programme, we found evidence that this has not diminished. All of the participants were still (as far as we were made aware by departmental records) working in the health field. When asked about their future career choices, working for the department was highest-ranking at the beginning of the course, and this remained true at the end. And when asked outright if they felt the programme had made a difference, 94% agreed with the statement, As a result of doing this course, I am more likely to stay working in the health field. As one participant put it at that point: Given the significant changes that have been underway in the DH over the last two years this course was the main incentive for me not to change departments. Figure 4: Summary of responses by participants to the question Do you agree with the following statement? The interim round of interviews also gave an early indication that the participants are beginning to feel more like health experts in their roles, a second Level 3 objective for the programme. They told us: I can see myself transferring into the health system when the course is done. The fact that it is transferable to other parts of the health system is important to me. I see this as an investment in us as part of the wider health service, not just the department. That is how my manager sees it as well. Thirdly, the participants felt they were able to develop networks, and had started to bring these to bear in their day-to-day work. These networks had proven useful, even if there were primarily within the cohort and the faculty. The benefit of a better internal network is that it expedites things I have been able to pick up the phone and talk for 10 minutes rather than organise a formal meeting, to get things resolved. Yes predominantly within the department most of the non-dh participants have dropped like flies. It also helps having a shared language around things so I can call X and know he will have the same basic understanding of, say, economics. The links to academics and the speakers are good I feel more confident and licence to call and ask for informal advice which is consistent with the open policy making agenda. The line managers were slightly more cautious, noting that those selected had good networks anyway and were adept at developing them. He is interested in the networking opportunities both with peers and with senior academics he is canny enough to see that doing this will raise his profile and his network and that that will help him. 11

13 On one occasion he got someone in to explain something, but he is well networked from having been in private office anyway so that wasn t a problem. On the final Level 3 benefit, applying specific analytical skills, there was limited evidence at the interim reporting stage that this is feeding through to the day-to-day work of the participants. At that stage, when asked, line managers struggled to give examples, and the participants told us: It may have disabled me as recently I got too lost in the evidence and took too academic an approach I need to learn to translate what I have learnt. One question I have is how to reconcile what this gives with the civil service reform plans? There are mixed messages in there about becoming on the one hand more specialist, but also gaining experience across departments. However, by the end of the programme, when asked if they had been able to apply skills to find and evaluate evidence, 53% said that yes they had, and a further 40% suggested they had been somewhat able to. Some examples they gave at that stage included: Being able to challenge clinical/analytical colleagues on the robustness of evidence from research studies they argued should drive policy action. Improved research skills... Improved ability to digest and summarise the key points in long documents. Being better equipped to understand and critique research proposals. Using evidence on the effect of competition and mergers to provide advice and evidence on a potential hospital merger. 12

14 3. Caveats identified by those we spoke to As we have stated above, overall the participants and their line managers were positive about the programme and its potential to deliver benefits to the department. However, in their responses to our questions, they outlined a number of caveats. Impact on the wider team While positive about the programme overall, many of the line managers we spoke to during and after the programme highlighted the impact that the programme had on the wider team back in the department. This included planning for and managing absences during teaching weeks. However it also included the periods in between when essays and other work were due. While on the whole it appears that most teams were able to cope, this was helped by the fact that the participants selected were highly diligent and organised and had the support of their colleagues. It was suggested by a few managers that the numbers of participants from each team should be kept as close to one as possible. The dissertation Participants and managers raised concerns about the applicability of the dissertation process to the day job of the participants; that choosing good dissertation topics has been frustrating and inefficient; and that the department is not fully realising the potential of this process. There were also concerns raised about the limited oversight or co-ordination of this process from inside the department. However, it is worth noting that value from undertaking the dissertation process may be one of the longer term outcomes of the MSc process and difficult to assess from the point of view of day-to-day work. It might be worth the department considering the value of the dissertation process again once a few cohorts have passed through the programme. Personal impact of the programme Throughout the evaluation, reference was frequently made by participants and their line managers to the personal investment required in undertaking the programme. While it was acknowledged that this is an inevitable part of undertaking a course at this level, line managers did suggest that at times they were concerned for their colleagues well-being. When asked if they would recommend the programme to a friend none of the participants surveyed said no, however many gave caveats. We have included these below, as they may be usefully passed on as advice to prospective future candidates. I would strongly recommend to others and have done so but with the caveat about work/study/life pressures that come with it! I would recommend this course on the proviso that a friend negotiated a different working pattern to allow for the significant workload. The ability to recommend very much depends on personal circumstance and comes with a big caveat that it is very hard work, and is a big commitment. It is a massive difference to have a full-time job and part-time study, as opposed to part-time work and full-time study. The last six months of the course is very intense and was tricky to undertake at the same time as working full time the impact of this should not be underestimated. 4. Conclusions This evaluation gives an overall positive impression of the MSc Health Policy. The course has proved popular with the participants, with key indicators like would you recommend the course to a friend coming out overwhelmingly positive. The available evidence also shows that the MSc has largely achieved the shorter term objectives of the department, most particularly in terms of acquiring specialist health knowledge and generating the feeling that the department was investing in the participants. In terms of longer-term change, there were positive indications that participants were 13

15 starting to apply their learning on the job and that the course had had a positive impact on retention, though it is beyond the scope of this evaluation to fully assess these aspects. These findings may be helpful in considering the wider objectives of Civil Service s policy profession. Action 9 of the current action plan commits departmental Heads of Policy Profession to: develop plans to transform the development of policy Grades 7 SCS 1, with a focus on increasing: a) deep subject expertise; b) post-graduate qualifications on public policy or business administration; c) wider experience and; d) skills in related disciplines. The MSc Health Policy appears, at least in the area of health, to be an effective way of generating deep subject expertise, helping to creating a cadre of people who consider themselves to be health experts. The MSc was seen as less appropriate for teaching participants how to use policy making tools, with this being seen as something that could be learned on the job. This suggests that any wider use of the expert MSc format would benefit from being embedded in a more structured way of developing policy-making skills giving participants clearer ways of applying their new expertise within the working environment. As the application in the workplace is vital, this may also suggest the more active involvement of line managers and senior departmental management in designing and setting expectations for using such policy making skills across departments. 14