Educational Outcomes of Masters in Healthcare Management Programmes in the Middle East and Ireland: A Comparative Perspective

Educational Outcomes of Masters in Healthcare Management Programmes in the Middle East and Ireland: A Comparative Perspective Dr. Jonathan Drennan Senior Lecturer Institute of Leadership Royal College of Surgeons in Ireland Medical University of Bahrain P.O. Box 15503, Adliya Kingdom of Bahrain Ph: +973 17 351 450 Fax: +973 17 320 939 E-mail: jonathandrennan@rcsi.ie www.rcsileadership.org 1.1 Introduction Increasingly there is a need to demonstrate the impact and quality of educational programmes on student outcomes, especially when similar programmes are hosted in multiple sites. Therefore this paper outlines the findings from an educational evaluation, undertaken through an online survey that comprehensively measured the impact that a master s degree in healthcare management had on graduates completing their degree in the Middle East and Ireland. The results of this study also comprehensively evaluated the quality of the education programmes and the factors associated with graduate outcomes including: graduates evaluation of the quality of their course, the development of generic and leadership capabilities, the development of graduate qualities and relationships between the academic environment in which students are learning their learning styles and engagement and their ability and achievement. The theoretical model guiding this study, Astin s (1993) Input, Environment, Outcome (IEO) model, is based on core concepts that examine the impact or effect of a programme of higher education on student outcomes. The participants in this study were all graduates who have completed a Masters in Healthcare Management or Masters in Quality and Safety in Healthcare Management the Institute of Leadership at the Royal College of Surgeons in Ireland (RCSI). The programme is delivered at three sites: RCSI-Medical University of Bahrain (RCSI- MUB), RCSI Dubai, United Arab Emirates and at RCSI in Dublin, Ireland. Results will be presented in relation to graduates evaluation of their experience of their programme of study, whether students employed a deep or surface approach to their learning, student engagement and sociodemographic, professional and educational variables. Regression models will be presented that identified the relationship between the learning style of students, their experience of the quality of the programme and their level of engagement and the outcomes achieved as a consequence of their master s programme. A comparative perspective of the learning styles and outcomes of students who completed the degree in the United Arab Emirates, Bahrain and Ireland will also be presented. 1

This is important, as due to the growth and diversification of the master s degree a number of pertinent questions remain unanswered regarding the future development and direction of the degree. 1.2 Theoretical Model The theoretical model used in this evaluation is based on core concepts that have previously been used to examine the impact or effect of a programme of higher education on student outcomes. These measures include student inputs (i.e. the characteristics that students possess on commencing the programme), processes (their experience of college) and, outcomes (outcomes that occurred as a consequence of attending a programme of study). Astin s (1993) Input, Environment, Outcome (IEO) model was used to evaluate the outcomes from the masters programmes. Astin (1993) developed a conceptual framework to guide the measurement of the student experience. This framework consists of three components: inputs, environment and outcomes. The model allows for a process of causality to be determined by examining various variables associated with inputs and environment that impact on graduate outcomes as a result of attending college. 1.2.1 Inputs Inputs in this study are defined as the pre-college demographic characteristics of the students. A number of student inputs have been identified as measures to enable both the prediction of student outcomes and as factors of statistical control (Pascarella and Terenzini 1991, Astin 1993). The inputs in this study will include prior education, age, gender, academic qualifications, distance travelled to college, work status during college and nationality. 1.2.2 Environment The second concept in the theoretical model, environment, refers to the actual experiences of the student during their time at college (this includes contact with academic staff, peer interaction, experience of the programme, student engagement and intellectual engagement). Environmental measures equate to the processes that the student experiences during their course of study. In the language of research environment equates to intervention. In this study environmental measures or processes are divided into structural processes and educational processes. Structural processes are defined as those processes that supported the student during their degree and include access to infrastructural support (access to appropriate software and access to computers), fee support, and workload. Educational processes are defined as those associated with the learning process and were identified as students approaches to learning (deep approach/surface approach, active and collaborative learning), the provision of clear goals and expectations, the assessment process, being intellectually motivated, the student s interaction with faculty and the student s experience of teaching. 1.2.3 Outcomes 2

The final and most important conceptual section of the theoretical model, outcomes, relates to the changes that occur in students as a consequence of their programme of study. The outcomes from any educational process are multifactorial and complex; therefore the aim is to identify core outcomes that are most relevant to masters programmes in healthcare management and quality. In this study outcomes are classified as short to medium term outcomes and longer-term impacts. Short to medium term outcomes included students satisfaction with their programme of study, academic achievement, the development of generic capabilities (students ability to tackle unfamiliar problems, work as a member of a team, develop problem-solving, analytical and written communication skills), the development of graduate qualities (motivation toward lifelong learning, the ability to value perspectives other than their own, the ability to investigate new ideas), the impact of the educational programme on professional practice and the development of leadership capabilities. The measurement of longer-term outcomes will include the impact of the programme on the graduates employment. 1.3 Methods 1.3.1 Aim of the Study The aim of this study is to measure the outcomes that occur in graduates who have completed a master s degree in health care management or quality and safety at a higher education institute in the Middle East and Ireland. 1.3.2 Research Questions The following research questions are addressed as a consequence of this study: 1. What impact does master s level education have on graduates development of generic capabilities? 2. What impact does master s level education have on graduates development of leadership capabilities? 3. What impact does master s level education have on graduates development of graduate qualities? 4. What impact does master s level education have on graduates subsequent employment? 5. What are graduates perceptions of the quality of their course? 6. Is there a relationship between students demographic and preprogramme educational characteristics and their educational outcomes? 3

7. Is there a relationship between the academic environment in which students are learning (workload, satisfaction, research supervision, quality of their educational experience), their learning styles and their ability and achievement? 1.3.3 Sample The participants in this study are graduates who have completed a Masters in Healthcare Management or Masters in Quality and Safety in Healthcare Management from the Institute of Leadership, Royal College of Surgeons in Ireland (RCSI). The programme is delivered at three sites RCSI Dubai, United Arab Emirates, RCSI-Medical University of Bahrain, Kingdom of Bahrain and RCSI, Dublin, Ireland. All graduates were contacted by e-mail informing of them of the study and inviting them to participate. Students who had deferred or did not complete their degree were excluded from the survey. To date there are 370 graduates from Dubai, Bahrain and Dublin. 1.3.4 Measures An in-depth review of the higher education literature and course documentation identified a number of outcomes that should occur following a master s programmes in general and healthcare management programmes in particular (Helsinki Conference on Master-level Degrees 2003, Joint Quality Initiative 2004a, 2004b). Core outcomes from master s level programmes include the development of inter- and intra- professional communication, the facilitation of the development of leadership and teaching roles, the development of critical thinking skills, initiating positive change in healthcare settings, and development of advanced management competencies. To measure the inputs, processes and outcomes associated with a master s degree a number of instruments and questions previously validated for use in the higher education sector have been identified. The following variables were measured in the study: Input Factors Age Gender Student background characteristics Pre-programme assessment of the students knowledge and capabilities Professional background Education background Nationality Work status Environment Factors Academic challenge/intellectual challenge of the programme Active and collaborative learning student engagement Student-faculty interaction 4

Supportive environment Students approaches to Learning Students experiences of teaching Outcomes Student satisfaction with their programme of study Development of academic capabilities Development of generic capabilities Development of leadership capabilities Impact of the programme on professional practice Impact of the programme on career progression 1.3.5 Instruments/Variables The following instruments and variables were to measure the inputs, processes and outcomes associated with master s level education: the Course Experience Questionnaire and the Extended Course Experience Questionnaire, the Study Process Questionnaire, Postgraduate Student Engagement Questionnaire and sociodemographic, professional and educational variables. Course Experience Questionnaire (CEQ) and the Extended Course Experience Questionnaire (ECEQ) Graduates experiences of their master s programme were measured using both the Course Experience Questionnaire (CEQ) and a number of scales that comprise the Extended Course Experience Questionnaire (ECEQ). The purpose of the CEQ is to evaluate graduates perceptions of the quality of the courses they completed at college and the extent to which they perceive they have developed generic skills (Ainley and Johnson 2000, McInnis et al. 2001). The value of the CEQ is that it gives a broad perspective on outcomes by focusing on graduates perceptions of their courses rather than on students evaluations of particular lecturers. The CEQ also examines indicators of relative performance in higher education at both system and institutional levels (McInnis et al. 2001). Theories of teaching and learning informed the development of the CEQ (Ramsden 1991, Ainley and Johnson 2000). Its main strength is that the instrument enables the researcher to link students perceptions of the quality of a programme to learning outcomes. In testing it has been shown to be a reliable and valid instrument. It has also been claimed to be valuable in improving the quality of teaching in universities and also for informing student choice, managing institutional performance, and promoting accountability of the higher education sector (McInnis et al. 2001, pg. 3). The CEQ instrument has undergone progressive refinement, development and testing over the last twenty years culminating in the recent development of an Extended Course Experience Questionnaire (Griffin et al. 2003). In its original form the CEQ was found to be a reliable and valid instrument that identified the quality of teaching in different academic departments and institutions (Ramsden 1991). However, the original CEQ instrument was deemed to be lacking in its ability to capture the wider aspects of the student experience 5

in higher education including the impact of a higher education programme on student outcomes. Therefore to comprehensively capture the experience of students at university, in addition to the current scales on the CEQ (Good Teaching Scale; Clear Goals and Standards Scale; Appropriate Assessment Scale; Appropriate workload Scale and Generic skills Scale), five further scales were recommended to extend the instrument. These new scales included the Student Support Scale, Learning Resources Scale, Learning Community Scale, Intellectual Motivation Scale and Graduate Qualities Scale. These additional instruments expanded the instrument from a 23-item questionnaire to a 50-item questionnaire with the addition of five scales (see table 1). Table 1 Defining Items of the CEQ and Extended CEQ Scales (Adapted from Ramsden 1991, McInnis et al. 2001, Griffin et al. 2003) Scale Defining Item Good Teaching* Teaching staff here normally give helpful feedback on how you are going Clear Goals and Standards* Appropriate Workload* Appropriate Assessment* I usually had a clear idea of where I was going and what was expected of me in this course. The sheer volume of work to be got through in this course meant it couldn't all be thoroughly comprehended. The staff seemed more interested in testing what I had memorised than what I had understood. Generic Skills* The course developed my problem solving skills Student Support (5 items) Learning Resources (5 items) Learning Community* (5 items) Intellectual Motivation* (4 items) Relevant learning resources were accessible when I needed them It was made clear what resources were available to help me learn I felt I belonged to the university community I found my studies intellectually stimulating Graduate Qualities* The course developed my confidence to investigate new ideas (6 items) Original CEQ scales. Extended CEQ Scales *Scales used in this study The purpose of using the CEQ in this study is to identify the core aspects of teaching and learning that students experienced during their master s programme (Ainley and Johnson 2000). The CEQ is identified as being particularly valid for the evaluation of coursework master s programmes in healthcare management and quality as students are exposed to a wide variety of subjects, come into contact with a wide variety of lecturers and have a wide variety of educational experiences. The validity and utility of the CEQ has resulted in its widespread use as an indicator of quality and student outcomes. 6

Studies have also used the results of the CEQ to identify the relationship between student satisfaction and student outcomes (Lizzio et al. 2002, Lawless and Richardson 2002, 2004). The educational experience is increasingly been viewed as providing a significant relationship with student outcomes (Pascarella and Terenzini 1991, Astin 1993). In this study the students educational experience will be measured on the good teaching, clear goals and standards, appropriate workload, learning community and intellectual motivation subscales of the CEQ. Outcomes will measured on the generic skills, graduate qualities and overall satisfaction subscales. Students will be asked to rate their level of agreement or disagreement on a fivepoint scale on 39-items that comprised eight scales of the CEQ and the ECEQ. The eight scales used in this study will include the Good Teaching Scale, Clear Goals and Standards Scale, Appropriate Workload Scale, Generic Skills Scale, Learning Community Scale, Intellectual Motivation Scale and the Graduate Quality Scale. Study Process Questionnaire Based on student approaches to learning theory (Marton & Säljö 1976), the Study Process Questionnaire (SPQ) measures three higher order factors associated with learning: surface approach to learning, deep approach to learning and an achieving approach to learning (Biggs1987). Each approach is also associated with a motive and strategy score. The SPQ integrates into Astin s IEO model in that student factors, the teaching context approaches to learning, and the learning outcomes, mutually interact, forming a dynamic system (Biggs et al. 2002: 135). A modified version of the SPQ was developed in 2002 and contains two scales: Deep Approach (DA) and Surface Approach (SA) with four associated subscales, Deep Motive (DM), Deep Strategy (DS), Surface Motive (SM), and Surface Strategy (SS) (Biggs et al. 2002). Entitled The revised two-factor Study Process Questionnaire (R-SPQ-2F), this twenty-item questionnaire was used in this study to measure the graduates preferred approach to learning. This measure was used firstly to identify the preferred learning styles used by master s students in the Middle East and Ireland and, secondly, to identify the relationship between the learning style of students and the outcomes achieved as a consequence of their master s programme. Postgraduate Student Engagement Questionnaire Recently developed in Australia, and based on the US National Survey of Student Engagement (NSSE), the Postgraduate Student Engagement Questionnaire (PSEQ)(Coates 2010) measures a number of processes and outcomes experienced by students during their programme of study. Processes measured include academic challenge, active learning, student and staff interactions, enriching educational experiences, supportive learning environment and work integrated learning. A number of educational outcomes were also measured including the development of higher order thinking, general learning outcomes, interpersonal and communication skills, the ability to undertake and implement change and the development of leadership capabilities. 7

Table 2 Defining Items of the PSEQ Scales Scale Defining Items Overall Academic ability Ability to produce scholarly reports or papers Problem-solving ability Communication and Teamwork ability Ability to Impact on Professional Practice Ability to lead and implement change Research Ability Applied to Practice Ability to evaluate arguments and evidence of competing alternatives to solve a problem Ability to communicate and work as a member of a team Ability to challenge and influence practice Leadership ability Ability to analyse and interpret quantitative data Measurement of Sociodemographic, Professional and Educational Variables To test for relationships with the dependent variables (outcome measures) a number of Sociodemographic, professional and educational variables will also be measured (see table 3). Table 3 Sociodemographic, Professional and Educational Variables Measured Sociodemographic Professional Variables Educational Variables Variables Age Current area of work Strand (Healthcare management/quality) Gender Grade (promotion) Academic qualifications Site at which degree completed Professional qualifications Employment during masters (full-time/part-time/none) Years qualified Final award Hours spent on study/writing (workload) Publication/research/conferences Each of the variables identified has been shown to have an impact on educational outcomes. For example sociodemographic characteristics including gender, age and prior education have been shown to have differing levels of impact on student outcomes following a programme of higher education including ability, achievement and satisfaction (Pascarella and Terenzini 1991, Astin 1993, Drennan and Hyde 2008, Drennan and Clarke 2009). A number of educational variables will also be measured including mode of attendance and workload. Instructional strategies have been shown in the literature to 8

impact at various levels on student outcomes (Pascarella and Terenzini 1991, Astin 1993). 1.3.6 Procedure The survey was delivered online using the web-based online survey system, SurveyMonkey (www.surveymonkey.com). As in best practice with surveys participants were sent up to four reminders to complete the survey. 1.3.7 Ethical Issues Access and ethical approval was granted from the ethics committees at RCSI. 1.3.8 Data Analysis Data obtained was analysed by computer using the Statistical Package for the Social Sciences (SPSS version 18.0). Both descriptive and inferential statistics were used in the analysis and description of the data set through the use of univariate, bivariate and multivariate methodologies. Descriptive statistics (frequencies, frequency per cents, measures of central tendency, and measures of variability) were used to summarise demographic data and results from the instruments used in the study. The types of parametric or nonparametric inferential tests used were be determined by level of measurement and assumptions of normality tests. Data at nominal or ordinal levels of measurement was analysed inferentially using non-parametric tests such as the chisquare test, the Mann-Whitney U test or the Wilcoxon Signed Rank test. Data at scale level was analysed using t-tests, ANOVAs, ANCOVAs and MANOVAS as appropriate. Tests of normality included both visually examining the distribution of data through the use of histograms and boxplots and statistical tests of normality, which include the use of kurtosis and skewness measures, and the Kolmogorov- Smirnov test. Multiple regression analysis was used to find and test complex relationships amongst observed (measured) variables and latent (unobserved) variables, and amongst the latent variables themselves. In particular hierarchical multiple regression analysis will be conducted to determine the linear combination of variables that best predicted the development of generic capabilities, leadership capabilities, student satisfaction, and the development of graduate qualities. To examine self-reported changes in ability of graduates overall a repeated-measures MANOVA was conducted with time as the independent variable, and ability subscales as dependent variables (ability to change practice, leadership ability, communication/teamwork ability, research ability, problem-solving ability, and overall academic ability). 1.4 Conclusion At present the results of the study are being written up and will be presented at the conference. It is intended that they will show the relationship between input factors (demographic and professional factors), Environment (intellectual challenge, student engagement, the student experience) and outcomes (the quality of the educational 9

programme, student achievement, the impact of the programme of study on professional practice and the impact of the programme of career progression). The study will also provide evidence on the learning styles of students who complete the same programme of study in the Middle East and Ireland. This paper will demonstrate the methods that can be used to provide key stakeholders with the evidence that they seek on the quality of educational programmes that are being delivered. The results also provide evidence of the accountability of the education system that deliver the programmes and deliver to the wider community evidence of the standards and outcomes of the programmes. The quality of the student experience in the Middle East and comparing the cohort of students experiences will also be demonstrated. This is important in that due to the growth and diversification of the master s degree a number of pertinent questions remain unanswered regarding the future development and direction of the degree. 10

References Ainley J., Johnson T. (2000). Course Experience Questionnaire 2000: An Interim Report Prepared for the Graduate Careers Council of Australia. Australian Council for Educational Research, Canberra. Astin, A. (1968). Undergraduate achievement and institutional "excellence". Science, 161(842), 661-668. Astin A. (1984). Student involvement: A developmental theory for higher education. Journal of College Student Personnel. 25, 294-308. Astin A. (1993). What Matters in College?: Four Critical Years Revisited. Jossey- Bass, San Francisco. Astin, A. W., and Sax, L. J. (1998). How undergraduates are affected by service participation. Journal of College Student Development, 39(3), 251-263. Biggs J. B (1987). Student approaches to learning and studying. Australian Council for Educational Research, Hawthorn, Victoria. Coates H. (2011) Development of the Australasian survey of student engagement (AUSSE). Higher Education. 60, 1-7. Drennan J, Clarke M. (2009) Coursework Master's Programmes: The Student's Experience of Research and Research Supervision. Studies in Higher Education, 34 (4) Drennan J., Hyde A. (2008) Controlling response shift bias: The use of the retrospective pretest design in the evaluation of a master's programme. Assessment and Evaluation in Higher Education, 33 (6):699-709. Griffin P., Coates H., McInnis C., James R. (2003). The development of an extended Course Experience Questionnaire. Quality in Higher Education, 9, (3), 259-266. Helsinki Conference on Master-level Degrees (2003). Helsinki 14-15 March 2003. Joint Quality Initiative (2004a). Shared Dublin Descriptors for the Bachelor s, Master s and Doctoral Awards accessed 2 nd July 2005 at http://www.jointquality.org/content/ireland/shared/. Joint Quality Initiative (2004b). Shared Dublin Descriptors for Short Cycle, First Cycle, Second Cycle and Third Cycle Awards accessed 2 nd July 2005 at http://www.jointquality.org/content/ireland/shared/. Lawless C. Richardson J. (2002). Approaches to studying and perceptions of academic quality in distance education. Higher Education 44, 257-282. 11

Lawless C. Richardson J. (2004). Monitoring the experience of graduates in distance education. Studies in Higher Education, 29 (3), 353-374. Lizzio A., Wilson K., Simons R. (2002). University students perceptions of the learning environment and academic outcomes: implications for theory and practice. Studies in Higher Education 27, (1), 27-52. Marton, F., & Saljo, R. (1976). On qualitative differences in learning - I: Outcome and process. British Journal of Educational Psychology, 46, 4-11. McInnis C., Griffin P., James R., Coates H. (2001). Development of the Course Experience Questionnaire. Department of Education Training and Youth Affairs. Canberra. Pascarella E., Terenzini P. (1991). How College Affects Students: Findings and Insights From Twenty Years of Research. Jossey-Bass. San Francisco. Ramsden P. (1991). A performance indicator of teaching quality in higher education: The Course Experience Questionnaire. Studies in Higher Education. 16, 129-150. OoO 12