Evaluation of MSc programmes: MSc in Healthcare Management MSc Quality and Safety in Health Care Management MSc Leadership & Management Development

Transcription

1 Evaluation of MSc programmes: MSc in Healthcare Management MSc Quality and Safety in Health Care Management MSc Leadership & Management Development 1

2 An Evaluation of MSc in Healthcare Management, MSc Quality and Safety in Health Care Management, MSc Leadership & Management Development Programmes at the Institute of Leadership, Royal College of Surgeons in Ireland. Prepared by Dr. Jonathan Drennan & Dr. Pauline Joyce Institute of Leadership Royal College of Surgeons in Ireland April

3 1.1 Introduction This report outlines the findings from an educational evaluation that comprehensively measured the impact that masters degrees in healthcare management, leadership & management development and quality and safety in healthcare management had on graduates who completed the programmes at the Institute of Leadership, Royal College of Surgeons (RCSI) in Ireland. The results presented here include graduates evaluation of the quality of their course, the extent to which graduates reported the development of leadership, management and generic capabilities and, the impact the master s programme had on graduates subsequent careers. The participants in this study were graduates who completed a MSc in Healthcare Management, MSc in Leadership and Management Development or MSc in Quality and Safety in Healthcare Management. These programmes have been delivered at RCSI since 2005 with the first graduates in The first section of the evaluation reports on the methods used to evaluate the masters programmes at the Institute of Leadership. This is followed by the presentation of the results which includes: the demographic profile of the sample, the results from the survey that measured graduates experience of their programme of study, the extent to which graduates made gains in the development of leadership capabilities and, finally, the employment destination of graduates following the completion of the programme. 1.2 Methods The evaluation is principally concerned with measuring the quality of the programme and the educational and professional outcomes achieved by graduates as a consequence of the programme. In this study outcomes are classified as short to medium term outcomes and longer-term impacts. Short to medium term outcomes included students satisfaction with their programme of study, academic achievement, the development of generic capabilities (graduates ability to tackle unfamiliar problems, work as a member of a team, 3

4 develop problem-solving, analytical and written communication skills), the development of graduate qualities (motivation toward lifelong learning, the ability to value perspectives other than their own, the ability to investigate new ideas), the impact of the educational programme on professional practice and the development of leadership capabilities. The measurement of longer-term outcomes included the impact of the programme on the graduates employment. This section outlines the methods that were used to measure these outcomes Aims of the Evaluation The aims of this evaluation were: 1) to evaluate the quality of the masters programmes delivered by the Institute of Leadership at RCSI and; 2) to measure the outcomes that occurred in graduates following the completion of their programme of study Sample The participants in this study were graduates who completed a MSc in Healthcare Management, MSc in Leadership and Management Development or MSc in Quality and Safety in Healthcare Management from the Institute of Leadership, Royal College of Surgeons in Ireland (RCSI). All graduates were contacted by informing of them of the study and inviting them to participate. Students who had deferred or did not complete their degree were excluded from the survey. At the time of the study there were 303 graduates from Dublin. 195 of these were surveyed for this study and the response was 102 (52.3%) Measures The following instruments and variables were to evaluate the outcomes associated with the masters degrees: the Course Experience Questionnaire and the Extended Course Experience Questionnaire, the Master s Outcomes Evaluation Questionnaire and sociodemographic variables Course Experience Questionnaire (CEQ) and the Extended Course Experience Questionnaire (ECEQ) 4

5 Graduates experiences of their master s programme were measured using both the Course Experience Questionnaire (CEQ) and a number of scales that comprise the Extended Course Experience Questionnaire (ECEQ). The purpose of the CEQ is to evaluate graduates perceptions of the quality of the courses they completed at college and the extent to which they perceive they have developed generic skills (Ainley and Johnson 2000, McInnis et al. 2001). The value of the CEQ is that it gives a broad perspective on outcomes by focusing on graduates perceptions of their courses rather than on students evaluations of particular lecturers. The CEQ also examines indicators of relative performance in higher education at both system and institutional levels (McInnis et al. 2001). Theories of teaching and learning informed the development of the CEQ (Ramsden 1991, Ainley and Johnson 2000). Its main strength is that the instrument enables the researcher to link students perceptions of the quality of a programme to learning outcomes. In testing it has been shown to be a reliable and valid instrument. It has also been claimed to be valuable in improving the quality of teaching in universities and also for informing student choice, managing institutional performance, and promoting accountability of the higher education sector (McInnis et al. 2001, pg. 3). The CEQ instrument has undergone progressive refinement, development and testing over the last twenty years culminating in the recent development of an Extended Course Experience Questionnaire (Griffin et al. 2003). In its original form the CEQ was found to be a reliable and valid instrument that identified the quality of teaching in different academic departments and institutions (Ramsden 1991). However, the original CEQ instrument was deemed to be lacking in its ability to capture the wider aspects of the student experience in higher education including the impact of a higher education programme on student outcomes. Therefore to comprehensively capture the experience of students at university, in addition to the current scales on the CEQ (Good Teaching Scale; Clear Goals and Standards Scale; Appropriate Assessment Scale; Appropriate workload Scale and Generic skills Scale), five further scales were recommended to extend the instrument. These new scales included the Student Support Scale, Learning Resources Scale, Learning Community Scale, Intellectual 5

6 Motivation Scale and Graduate Qualities Scale. These additional instruments expanded the instrument from a 23-item questionnaire to a 50-item questionnaire with the addition of five scales (see table 1). Table 1 Defining Items of the CEQ and Extended CEQ Scales (Adapted from Ramsden 1991, McInnis et al. 2001, Griffin et al. 2003) Scale Defining Item Good Teaching* Teaching staff here normally give helpful feedback on how you are going Clear Goals and Standards* Appropriate Workload* Appropriate Assessment* I usually had a clear idea of where I was going and what was expected of me in this course. The sheer volume of work to be got through in this course meant it couldn't all be thoroughly comprehended. The staff seemed more interested in testing what I had memorised than what I had understood. Generic Skills* The course developed my problem solving skills Student Support (5 items) Learning Resources (5 items) Learning Community* (5 items) Intellectual Motivation* (4 items) Relevant learning resources were accessible when I needed them It was made clear what resources were available to help me learn I felt I belonged to the university community I found my studies intellectually stimulating Graduate Qualities* The course developed my confidence to investigate new ideas (6 items) Original CEQ scales. Extended CEQ Scales *Scales used in this study The purpose of using the CEQ in this study is to identify the quality of the core aspects of teaching and learning that students experienced during their master s programme (Ainley & Johnson 2000). The CEQ is identified as being particularly valid for the evaluation of coursework master s programmes in healthcare management and quality as students are exposed to a wide variety of subjects, come into contact with a variety of lecturers and experience a variety of educational experiences. The validity and utility of the CEQ has resulted in its widespread use as an indicator of quality and student outcomes. 6

7 Students were asked to rate their level of agreement or disagreement on a fivepoint scale on 39-items that comprised eight scales of the CEQ and the ECEQ. The eight scales used in this study included: the Good Teaching Scale, Clear Goals and Standards Scale, Appropriate Workload Scale, Generic Skills Scale, Learning Community Scale, Intellectual Motivation Scale and the Graduate Quality Scale Master s Outcomes Evaluation Questionnaire Outcomes related to leadership and management were measured using an instrument developed specifically for this evaluation: the Masters Outcomes Evaluation Questionnaire (MOEQ). The items on the subscale were identified and developed from an extensive review of the literature and course documentation pertaining to leadership outcomes in healthcare management. The questionnaire consisted of 31-items measuring four domains seen as relevant to leadership and management practice: 1) ability to change professional practice (9 items); 2) leadership capabilities (6 items); 3) communication and teamwork (6 items) and; 3) problem solving (10 items). The MOEQ was designed to incorporate a retrospective pre-test design. Students were asked firstly to rate their ability following their master s programme (posttest) and then to think back and rate their ability before the commencement of their master s programme (retrospective pre-test/then-test) on a seven-point scale that was anchored by 1 = low understanding/ability to 7 = high understanding/ability. The rationale for using this approach was to account for the confounding factor of response-shift bias and to identify whether the outcomes were being achieved as a consequence of the programme or were due to other influence such as in-service education, further education, maturation or employment (Drennan & Hyde 2008). Table 2 Defining Items of the Masters in Nursing Outcomes Evaluation Questionnaire Scales Scale Defining Items Change Professional Ability to change and influence practice Practice Leadership Communication and Teamwork Ability to use leadership theories to inform professional practice Ability to communicate and work as a member of a team 7

8 Problem Solving Ability to develop solutions to practice problems through inquiry analysis and interpretation Measurement of Sociodemographic, Professional and Educational Variables To test for relationships with the dependent variables (outcome measures) a number of Sociodemographic, professional and educational variables will also be measured (see table 3). Table 3 Sociodemographic, Professional and Educational Variables Measured Sociodemographic Professional Variables Educational Variables Variables Age Current area of work Strand (Healthcare management/quality) Gender Grade (promotion) Academic qualifications Site at which degree completed Employment during masters (full-time/part-time/none) Final award Hours spent on study/writing (workload) Procedure The evaluation survey was delivered online using the web-based online survey system, SurveyMonkey ( As in best practice with surveys, participants were sent up to four reminders to complete the questionnaire Ethical Issues Access and ethical approval was granted from the ethics committees at RCSI Data Analysis Data obtained was analysed by computer using the Statistical Package for the Social Sciences (SPSS version 18.0). Both descriptive and inferential statistics were used in the analysis and description of the data set. To ascertain the specific differences between students ratings of their ability at the end of the course (post-test) compared with their ability at the beginning of the course (then-test) Wilcox on signed rank test was used. 8

9 1.3 Results The demographic profile of the sample that completed the evaluation of the programmes at RCSI in Dublin is outlined in table 4. Table 4 Demographic, Academic and Professional Profile of Master s Graduates Characteristic Age in Years M, (SD) 39.1 (8.0) Gender (%) Female 88.1 Male 11.9 Year Degree Completed (%) Strand Degree Completed (%) MSc in Healthcare Management MSc in Leadership & Management Development MSc in Quality and Safety in Healthcare Management Graduates Evaluation of the Quality of their Educational Programme This section reports on the results of the Course Experience Questionnaire (CEQ) and the Extended Course Experience Questionnaire, which were used to evaluate graduates perceptions of the quality of the courses and the extent to which they developed a number of generic and graduate capabilities. The results in this section are reported in relation to students perceptions of the quality of teaching, the extent to which they were provided with clear goals and standards, the appropriateness of workload, the appropriateness of assessment, graduates overall satisfaction with the course and the development of generic skills. The Extended Course Experience Questionnaire, used in conjunction with the CEQ, measured graduates perceptions of their integration into the learning community of the Institute, the extent to which they were intellectually motivated by the programme of study and the development of graduate qualities. The results are presented in relation to the individual items that comprise each of the scales of the CEQ and Extended CEQ and then in relation to each of the summated scale scores. To aid interpretation and standardise scores across the CEQ and the Extended CEQ the mean item scores in the tables and figures are based on a linear transformation where the 1 to 5 categories (strongly disagree 9

10 to strongly agree) have been recoded from -100 to Positive values indicate degrees of agreement and negative values indicate degrees of disagreement with each of the items. For further ease of interpretation, overall agreement on each item is reported by combining agree and strongly agree categories Graduates Overall Evaluation of their Programme of Study Item Scores on the CEQ Graduates rated the development of generic capabilities as the greatest impact and outcome of their master s degree. Graduates reported that they had attained a number of generic capabilities including the ability to communicate in writing, the development of analytical skills, problem-solving, the ability to tackle unfamiliar problems and their ability to work as a member of a team (table 5). Students also reported that they attained relatively high levels of ability in relation to their ability to plan their own work. Graduates also highly rated the majority of their experiences of teaching, with the vast majority in agreement that lecturers made their subjects interesting, were good at explaining subjects; graduates also perceived that lecturers motivated students to do their best work. Graduates were positive regarding the feedback received from teaching staff on how they were doing on the programme with the majority agreeing that staff made a real effort to understand the difficulties they might be having with their work. Students also reported that they were appropriately assessed and that the programme did not emphasise lower order assessment skills such as factual recall and a reliance on memory. The item the sheer volume of work to be got through in this course meant it couldn t be thoroughly comprehended was the only negatively rated item on the scale. This indicated there was variability in the extent to which graduates perceived they had the time to assimilate the content of the course. In addition the item the workload was too heavy, although positively rated, was rated relatively lower than other aspects of the programme. However, in relation to workload, graduates did perceive that they were given time to understand the content of the course. Graduates also highly rated the organisation of the course 10

11 and all items related to the measurement of clear gals and expectations were positively rated. Percentage agreement was also used to summarise item responses by combining agree and strongly agree categories into one overall agreement category (table 5). The highest level of agreement (86.7%) was related to the item: The course sharpened my analytical skills. High levels of agreement were also associated with other generic skill items including development of problem-solving skills (82.2%), the development of capabilities in written communication (80.0%) and the ability to tackle unfamiliar problems (68.8%). The quality of teaching was also highly rated. Over eighty per cent of graduates reported that lecturers were extremely good at explaining things and that they were motivated to do their best work. Graduates were also in agreement that teaching staff made a real effort to understand the difficulties they may having with their work (68.9%) and that lecturers worked hard to make their subjects interesting (69.0%). Graduates agreed or strongly agreed (63.8%) that teaching staff normally gave helpful feedback on the work submitted. Graduates strongly agreed that they were provided with clear goals and expectations. In particular 82.3% of graduates were of the opinion that the staff made it clear what they expected from students. Two thirds were also in agreement that they had a clear idea where they were going and what was expected from them on the course with approximately half of the respondents reporting that they knew the standard of work expected. Responses to the level of workload expected were variable. Generally, the majority of students were happy with elements that measured workload throughout the programme with over sixty per cent reporting that they were provided with enough time to understand the things they had to learn. However, seventy-five per cent reported that the volume of work meant it could not be thoroughly comprehended. However, the vast majority of graduates did not perceive that the the workload was too heavy. One item was used to measure overall satisfaction with the quality of the programme of study; the vast majority 11

12 of graduates (66.0 %) were in agreement that they were satisfied with the overall quality of their course. 12

13 Table 5 Item Scores* and Percentage Agreement on the CEQ CEQ Scale/Item Item Scores Percentage Agreement Mean* SD Good Teaching The teaching staff normally gave me helpful feedback on how I was doing The staff made a real effort to understand the difficulties I might be having with my work My lecturers were extremely good at explaining things The staff put a lot of time into commenting on my work The teaching staff motivated me to do my best work The teaching staff worked hard to make their subjects interesting Appropriate Workload The workload was too heavy The sheer volume of work to be got through in this course meant it couldn t be thoroughly comprehended I was generally given enough time to understand the things I had to learn There was a lot of pressure on me to do well in this course Clear Goals and Expectations It was always easy to know the standard of work expected I usually had clear idea of where I was going and what was expected of me on this course It was often hard to discover what was expected of me on this course The staff made it clear from the start what they expected from students Generic Skills As a result of my course I feel confident about tackling unfamiliar problems The course helped me develop my ability to work as a team member The course developed my problem solving skills The course sharpened my analytical skills The course improved my skills in written communication My course helped my ability to plan my own work Appropriate Assessment Too many staff asked me questions just about facts To do well in this course all you really needed was a good memory The staff seemed more interested in testing what I had memorised than what I had understood Overall Satisfaction Overall I was satisfied with the quality of this course *Scores range from 100 to Positive scores indicate levels of agreement; negative scores indicate levels of disagreement. Overall percentage agreeing or strongly agreeing on a 5-point scale. 13

14 Student Experience of the Master s Programmes Item Scores on the Extended CEQ Analysis of the items that comprise the three extended CEQ scales was also undertaken to identify the level of intellectual motivation reported by students, the extent to which students reported that they were integrated into the learning community of the Institute of Leadership and the extent to which they perceived they had developed a number of graduate qualities (table 6). The highest level of satisfaction related to the items that comprised the intellectual motivation and the graduate qualities scales. In the intellectual motivation scale the highest item scores related to the items: I found the course motivating, I found my studies intellectually stimulating, overall my university experience was worthwhile. In the graduate qualities scale the items: I consider what I learned valuable for my future, the course provided me with a broad overview of my field of knowledge, the course developed my confidence to investigate new ideas and, in particular, I learned to apply the principles from this course to new situations were also highly rated. Graduates also highly rated items related to the learning community scale. In particular, graduates reported that they felt part of a group of students and staff committed to learning and that they felt they belonged to the university community. In relation to percentage agreement for each of the items (a combination of agree and strongly agree responses), the highest percentage agreement related to the items on the graduate qualities scales and intellectual motivation scales. The item Overall my university experience was worthwhile received the highest level of agreement at approximately ninety-four per cent. Students were also in agreement that the course was motivating, stimulating and worthwhile. Similar levels of agreement were identified in the items that comprised the graduate qualities scale. This scale measured a number of outcomes achieved as a consequence of the programme of study. Over ninety-five per cent of graduates reported that they had learned to apply the principles of the course to new situations with ninety-two per cent reporting that they had developed the confidence to investigate new ideas. These findings are further supported by Joyce (2012) which suggest that education is influenced by the needs of 14

15 postgraduate students to acquire knowledge and skills for application back to practice. The remainder of items that comprised the graduate qualities scale achieved levels of agreement of seventy-eight per cent and greater. Graduates also reported that they felt part of a learning community. Over seventy per cent of respondents were in agreement that they felt they belonged to the university community, that they learned to explore ideas confidently with others and that they felt part of a group of students and staff committed to learning. The majority of students also reported that their ideas and suggestions were used throughout the programme. The only item that received a relatively lower rating at 50% of respondents agreeing was I learned to explore ideas confidently with other people Table 6 Item Scores* and Percentage Agreement on the Extended CEQ CEQ Scale/Item Item Scores Percentage Agreement Mean* SD Intellectual Motivation I found my studies intellectually stimulating I found the course motivating My course has stimulated my interest in my field of study Overall my university experience was worthwhile Learning Community I was able to explore academic interests with staff and students I felt part of a group of students and staff committed to learning I learned to explore ideas confidently with other people I felt I belonged to the university community Students ideas and suggestions were used during the course Graduate Qualities My university experience encouraged me to value perspectives other than my own I consider what I learned valuable for my future The course developed my confidence to investigate new ideas I learned to apply the principles from this course to new situations The course provided me with a broad overview of my field 83.8 of knowledge University stimulated my enthusiasm for further learning *Scores range from 100 to Positive scores indicate levels of agreement; negative scores indicate levels of disagreement. Overall percentage agreeing or strongly agreeing on a 5-point scale Scale Responses to the Course Experience Questionnaire and the Extended Course Experience Questionnaire 15

16 The items that comprise the CEQ and the Extended CEQ were summated into six scales, which measured the overall student experience of teaching, workload, goals and expectations, the development of generic skills, appropriate assessment, intellectual motivation, belonging to a learning community and the development of graduate qualities as part of their course experience. One single item measured overall satisfaction with the programme of study (table 7). Table 7 Mean Scores of the CEQ* and Extended CEQ* Scales Scale Minimum Maximum Mean SD Good Teaching Appropriate Workload Clear Goals and Expectations Generic Skills Appropriate Assessment Intellectual Motivation Learning Community Graduate Qualities Overall Satisfaction *Scores range from 100 to Positive scores indicate levels of agreement; negative scores indicate levels of disagreement. The mean scale scores identified that the development of graduate qualities (M = 56.85, SD = 29.37) was the highest outcome identified by students followed by being intellectually motivated (M = 43.60, SD = 30.17). A high mean score on the intellectual motivation scale indicated that graduates found their studies intellectually stimulating. A high mean score on the graduate qualities scale indicated that graduates developed the confidence to investigate new ideas and the ability to value perspectives different from their own. The development of generic skills (M = 40.88, SD = 29.21) and the extent to which graduates felt part of a learning community (M = 40.23, SD = 29.15) were also positively rated. A positive rating on generic skills scale indicated that graduates were of the opinion that they had developed problem-solving and analytical skills as an outcome of the programme of study; a positive rating on the learning community scale indicated that students felt part of the academic environment. Although the mean scores for the good teaching, clear goals and expectations and 16

17 appropriate assessment scales were rated relatively lower, graduates still positively evaluated these aspects of their programme. No element of the student experience received an overall negative rating, however, there was variability on the scores related to the appropriate workload scale (mean = 7.31, SD = 21.45). Although this aspect of the programme was positively rated, the overall score was relatively low compared to graduates experience of other elements in the course. Overall graduates were highly satisfied with the quality of their programme with a very high overall satisfaction mean score of (SD = 31.05) Graduate Destination The impact of the master s degree on subsequent promotion was deemed by graduates to be relatively influential with over a quarter of those who had changed grade or been promoted to a higher grade since completing their programme of study indicating that this had occurred as a result of holding a master s degree in healthcare management of quality and safety in healthcare management (see figure 2). 17

18 Figure 2 Changed job as a consequence of the Master s programme 1.4 Conclusion The evaluation of the masters programmes at RCSI Dublin identified that the experience of studying at master s level was of a high quality. Graduates also reported that they had developed a number of generic, management and leadership and critical thinking capabilities and as a consequence of the programme. Generic capabilities achieved included problem-solving skills, analytical ability, the ability to plan work, and the ability to communicate at a high level. Graduate capabilities included the confidence to investigate new ideas and the ability to apply the principles of the course to new situations as well as a commitment to lifelong learning. Graduates were positive about the organisation of the programme, the standard of teaching, the feedback from the teachers and the methods of assessment. Graduates also reported that they were involved in the life of the College and that the programmes were intellectually stimulating. In summary the programmes at RCSI, measured using reliable and valid educational outcome instruments, were highly evaluated. Overall students reported high levels of satisfaction with their programme of study and reported that the theoretical and practical components of the programme have a positive impact on their professional working lives. 18

19 References Ainley J., Johnson T. (2000). Course Experience Questionnaire 2000: An Interim Report Prepared for the Graduate Careers Council of Australia. Australian Council for Educational Research, Canberra. Drennan J., Hyde A. (2008) Controlling response shift bias: The use of the retrospective pretest design in the evaluation of a master's programme. Assessment and Evaluation in Higher Education, 33 (6): Griffin P., Coates H., McInnis C., James R. (2003). The development of an extended Course Experience Questionnaire. Quality in Higher Education, 9, (3), Joyce P (2012) Learning as doing -common goals and interests across management and education. Journal of Nursing Management 20(1): McInnis C., Griffin P., James R., Coates H. (2001). Development of the Course Experience Questionnaire. Department of Education Training and Youth Affairs. Canberra. Ramsden P. (1991). A performance indicator of teaching quality in higher education: The Course Experience Questionnaire. Studies in Higher Education. 16, OoO 19