Walden University Master of Science in Nursing Program: Intermountain Healthcare Research Study

Transcription

1 Walden University Master of Science in Nursing Program: Intermountain Healthcare Research Study Final Report October P age Arroyo Research Services

2 CREDITS Arroyo Research Services is an education professional services firm that helps education organizations meet their goals through meaningful research, measurement, evaluation, and consulting services. We help organizations develop and use actionable data to surpass their prior performance. Arroyo Research Services 639 Pennsylvania Rd Arden, NC Contributing Authors Michelle Vruwink Kirk Vandersall 2012 Arroyo Research Services 2 P age

3 CONTENTS Tables Included... 4 Figures Included... 5 Executive Summary... 6 Introduction Background Walden University Intermountain Healthcare The Study Research Questions Methods Research Design Sample Study Findings Student Outcomes Retention and Career Advancement Changes in Beliefs and Practices Changes in Nurse Education, Teamwork, and Mentoring Increased Proficiency Increased Self Efficacy and Career Commitment Health System Outcomes Cohort Effects Program Review Conclusion P age

4 References Appendix 1: Baseline Cohort Focus Group Protocol Appendix 2: Preliminary Student Survey Appendix 3: Preliminary Comparison Group Student Survey TABLES INCLUDED Table 1 Data Collection Timeline Table 2 Study Sample Demographics Table 3 Distribution of Study Sample Job Titles Table 4 Study Sample Graduate Degree Mode of Delivery Table 5 Cohort Program Concentrations Table 6 Cohort Survey Response Rates Table 7 Comparison Group Survey Response Rates Table 8 Likelihood of Remaining with Employer for Two Years (percent) Table 9 Change in Likelihood of Remaining with Employer for Two Years Table 10 Change in Likelihood of Seeking Higher Position within Six Months Table 11 Changes in Position Table 12 Change in Likelihood of Promoting Evidence Based Practice Table 13 Change in Likelihood of Engaging in Research Outside the Classroom Table 14 Change in Percent Reporting Contributing to Publications, Studies, or Presentations in Past Year Table 15 Change in Self Ranking for Ability to Advance Positive Social Change Table 16 Change in Comfort and Ability: Leadership and Mentoring Table 17 Change in Likelihood of Engaging in Leadership and Mentoring Table 18 Changes in Proficiency P age

5 Table 19 Change in Self Ranked Self Efficacy Table 20 Intent to Pursue Additional Certification After Degree Completion Table 21 Intent to Pursue Further Education and Learning After Degree Completion FIGURES INCLUDED Figure 1 Extent Studies Helped Improve in Current Position or as a Nurse... 9 Figure 2 Extent Able to Apply Learning to Career... 9 Figure 3 Learning in Graduate Program, Amount Learned Versus Expected, cohort and comparison groups Figure 4 Extent Studies Helped Improve in Current Position or as a Nurse Figure 5 Extent Able to Apply Learning to Career Figure 6 Importance When Considering Graduate Studies, percent ranked important or very important Figure 7 Learning in Graduate Program, Amount Learned Versus Expected, cohort and comparison groups Figure 8 Cohort Review of Program Quality, actual vs. expected Figure 9 Comparison Group Review of Program Quality, actual vs. expected P age

6 Walden University Master of Science in Nursing: Intermountain Healthcare Research Study EXECUTIVE SUMMARY In June 2008, a cohort of thirty one nurses from the Intermountain Healthcare Urban South Region in Provo, Utah entered the Walden University Master of Science in Nursing program (MS in Nursing). The cohort presented an opportunity to conduct an independent quasiexperimental study, commissioned by Walden University (Walden) and conducted in partnership with Intermountain Healthcare (Intermountain). Walden engaged Arroyo Research Services, an independent education research and evaluation firm, to conduct the study beginning in Fall The aim of the study was to test the following hypotheses of interest to both Walden University and Intermountain: 1. Participating in the Walden MS in Nursing program has a detectible, positive effect on the professional practices, leadership, and retention of nurses in the cohort as compared to nurses participating in other master level nursing programs. 2. Participating in advanced nursing programs as a cohort leads to increased commitment to nursing, completion of the program, and health system effects such as changes in institutional policies and practices. Conducted over four years using the cohort and a comparison group, the study was organized to test these hypotheses through related research questions organized into four main areas: Student Outcomes, Health System Outcomes, Cohort Effects, and Program Review. To answer research questions in these areas, the research team employed a mixed method panel design using the cohort of Intermountain nurses enrolled in the Walden MS in Nursing program and a comparison group of similar Intermountain nurses enrolled in master of science in nursing programs at non Walden institutions. Of the 31 nurses who enrolled in the Walden MS in Nursing program in June 2008, 22 consented to participate in the study. Of the 68 nurses Intermountain identified as meeting comparison group criteria, 34 responded to a screening survey, of which 24 met the criteria and 21 agreed to participate (see Research Design section for additional detail regarding the criteria and approach). Data used to answer the research questions was provided by Intermountain and Walden, and collected directly from the cohort and comparison group by Arroyo Research Services through focus groups and surveys at the beginning, end, and mid point of the study. Although the study included every member of the Walden cohort who agreed to participate and similar numbers of non Walden comparison group members, the limited sample size means that tests of statistical significance should be interpreted with caution. 6 P age

7 Key findings from the study: Retention and Career Advancement A higher percentage of cohort nurses compared to comparison group nurses were very likely or likely to remain with their current employer for at least two years. This was true at both the beginning of the study (96% cohort, 61% comparison) and at the end of the study (100% cohort, 50% comparison); no program effects were detected regarding the likelihood of study participants seeking a higher position or the likelihood of remaining with their current employer. Over 80% of study participants reported being likely or very likely to engage in promoting or initiating evidence based practice within their work, prior to entering the MS in Nursing program. Despite this strong baseline commitment to evidence based practice, cohort members consistently cited it as an area of growth during focus groups. Employers reported that they observed student growth in evidence based practice, providing confirmation of cohort self reported growth. On the final surveys, nearly all (90%) cohort respondents indicated they were very likely to promote or initiate evidence based practice, compared to 67% of comparison group respondents. Similarly, 90% of cohort respondents indicated their studies helped them implement innovation or evidence based practice, compared to 56% of comparison group respondents. While this shows a positive response from a larger percentage of cohort respondents, a comparison of pre and post test survey results showed a statistically significant and positive change among both cohort and comparison group members. Cohort nurses cited specific examples of initiating changes in practice, ranging from very specific examples (discontinued use of the Dubowitz Gestational Age Assessment and implementation of a new hypothermia treatment for cardiac arrest patients) to broader examples such as implementation of simulation training. Several comparison group nurses also provided examples of initiating changes in practice, though with less specificity. Teamwork and Mentoring Walden had a positive impact on student s ability to lead positive social change. In the final survey, 94% of cohort respondents indicated that their studies had helped them advance positive social change. One respondent indicated she did not know what positive social change was until she completed the program. Examples cited included volunteering in community health education activities and changing patient education practices to better reflect the needs of adult learners. From the comparison group, 44% 7 P age

8 of respondents indicated that their studies had helped them advance positive social change. Cohort nurses demonstrated substantially increased understanding of what it takes to be a leader (Effect Size.9), and moderately increased ability in teaching and training others (Effect Size.5). Comparison group nurses had large gains in influencing others in the field (Effect Size 1.5). Survey results did not show cohort member gains in taking leadership roles. This contradicts both cohort nurse and supervisor reports of demonstrated leadership, attributable, at least partially, to the result of response shift bias, where respondents frame of reference shifts as they more deeply understand an issue. While growth in leadership and mentoring may have been noted more often by cohort supervisors, comparison group supervisors also noted growth in the area of leadership and mentoring. Nurse Proficiencies The largest change in proficiency reported by the cohort was in performing action research (Effect Size 1.3), but additional large gains (ES >.6) were found in eight other proficiency areas. Proficiency areas measured included writing and communication, presenting analyses, designing research projects, performing qualitative analyses, performing quantitative analyses, applying research to practice, analyzing issues, retrieving information, performing action research, managing an interdisciplinary team, supervising personnel, and balancing cost containment with patient care. Detailed findings and presentation of data related to nurse proficiencies are reported in the Increased Proficiency section (page 37), within the main body of the report. Moderate effects of master s degree participation for the cohort group were observed in two areas of proficiency: supervising personnel and balancing cost containment with patient care. The comparison group showed large gains (ES >.6) in seven areas of proficiency. Overall, the effect size was larger for the comparison group in five of the areas, while the cohort showed change in more areas. Large cohort effects were observed in developing solutions to organizational problems and in creating new programs. The cohort also showed a moderate positive change in interacting with physicians and other professionals. Self Efficacy and Career Commitment Most (85%) cohort nurses report that their studies helped them improve in their position or as a nurse in general to a very great or great extent, as compared to approximately 60% of comparison group nurses. 8 P age

9 100% 80% 60% 40% 20% 0% 53% 35% 32% 30% 24% 10% 5% 6% 6% 0% Very Great Great Moderate No Extent NA Cohort Comparison Figure 1 Extent Studies Helped Improve in Current Position or as a Nurse Note: Cohort n=19, Comparison n= % 80% 60% 40% 20% 0% 47% 42% 30% 24% 23% 12% 12% 5% 5% 0% 0% 0% Very Great Great Moderate Small No Extent NA Cohort Comparison Figure 2 Extent Able to Apply Learning to Career Note: Cohort n=19, Comparison n=17. 9 P age

10 Walden University Master of Science in Nursing: Intermountain Healthcare Research Study Program Review A higher percentage of cohort respondents reported learning more than expected (50%) from their graduate program than did comparison group respondents (24%). Cohort respondents were also less likely than comparison group respondents to report that they learned the same as expected (14% vs. 24%). No respondents in either group reported learning less than expected. 100% % of respondents 80% 60% 50% Cohort 40% 33% Comparison 20% 18% 24% 24% 14% 0% Much more More Same Figure 3 Learning in Graduate Program, Amount Learned Versus Expected, cohort and comparison groups Note: Cohort n=19, Comparison n=17. Cohort members were less likely to rate the quality of instruction as above or far above expectation than comparison group members (47% compared to 59%), but were much more likely to rate the overall quality of course content to be above or far above expectation (68 % compared to 36%). Conclusions The length and design of the study required synthesis across multiple data sources, presented in the report but necessarily limited to key findings and differences between groups. Assessing 10 P age

11 these data and specific findings and considering the original hypotheses driving the study leads to the following conclusions: Participating in the Walden MS in Nursing program led to detectible positive effects on professional practices and leadership. o The outcomes in the areas of changed practice described in the report for the Walden cohort stood out as having an impact not only on the specific nurses in the cohort, but on the health system in which they worked and the specific units they served. As shown in the graphs above, both the cohort and the comparison groups reported improved knowledge and skill, with the cohort supervisors exhibiting greater enthusiasm for the extent of practical application of knowledge they observed in cohort nurses. o Walden cohort members demonstrated gains in leadership through selfreported examples of demonstrated leadership, changes in understanding of the role of nurses, and supervisor reports of demonstrated leadership. Cohort nurses reported a statistically significant and sizable change in their understanding of what it takes to be a leader, and a moderate change in teaching and training others. These gains were larger than those reported by comparison group members. o Walden cohort nurses reported increasing the extent to which they had published in the past year, while the comparison group reported a decrease., Cohort students were more likely to publish outside of the specific requirements of their degree program. However, this is due, in part, to differences in where students were in sequence of the program relative to the timing of the surveys. Retention among the cohort group was high at the beginning of the study and remains so, making it difficult to detect an effect of the program on retention (82% reported that they were likely to remain in their position for 2 or more years at the beginning of the study; 84% reported the same at the end of the study). We will examine this more fully after December Participation in graduate nurse education as a cohort affected outcomes, but not always as expected. The research team anticipated the primary cohort effects to be program outcomes: increased peer support, program persistence, and program completion. Instead, the study found stronger cohort effects on workplace outcomes than on program outcomes. Specific findings: o Cohort nurses appear to have had a stronger impact on the health system in part because many of them worked in the same units while they were completing their Master of Science degree in Nursing. The cohort effects that resulted, therefore, were related to cohort members ability to effect change in the workplace, working together as colleagues and classmates to implement new approaches to work. 11 P age

12 o Cohort commitment to nursing as a profession started and remained high throughout the study. Therefore, it is difficult to identify the extent of program effects on commitment to nursing. While the cohort and comparison groups had similar outcomes in many areas, the cohort group was more likely to identify and be actively seeking social change aspects to their work. Comparison group participants and their supervisors rarely mentioned examples of social change aspects to the nurses work, whereas the Walden cohort and their supervisors cited specific examples, often without prompting, of changes in practice and policy related to providing social benefits. In the final survey, 90% of cohort respondents indicated that their studies had helped them advance positive social change, while only 47% of the comparison group indicated this specific program outcome. The study identified no formal cohort support at either Walden or Intermountain Healthcare, such as common advisors at Walden, attempts to convene the cohort group, or group communication, but did identify informal collaboration and communication, mostly within departments with multiple cohort members. Finally, Intermountain s strong institutional culture and commitment to evidence based practice made it difficult to identify graduate program effects. Additional study of Walden cohorts in other settings and health systems would assist in determining the impact of cohort participation on nurse and system outcomes, and be useful for distinguishing the unique attributes of the Intermountain cohort from overall MSN program outcomes. 12 P age

13 Walden University Master of Science in Nursing: Intermountain Healthcare Research Study INTRODUCTION Multiple trends are driving increased demand for advanced degrees in nursing including a projected increase in nurse employment of 26% between 2010 and 2020 (driven by the aging population in the United States), increased emphasis on preventive care, and pressure to deliver medical care efficiently and effectively (U.S. Department of Labor, 2012). Increases in advanced specialty care provided by nurses, more team centered approaches to medical care, and increased emphasis on evidence based care raise the level of professionalism expected of nurses. National healthcare organizations, including the National Advisory Council on Nurse Education and Practice, the Institute of Medicine of the National Academies (Institute of Medicine, 2011), and the Magnet Recognition Program, have called for higher educational standards for nurses. In response, hospital systems, including Intermountain, have begun raising education levels in job requirements and hiring preferences. This, in turn, has fueled increased demand from nurses for more bachelor and advanced degree programs in nursing, and a corresponding increase in advanced degree programs in nursing offered by a mix of traditional, blended and online programs. While the changing nature of nursing and healthcare are driving demand for advanced degrees, the quality of degree programs has largely been unexamined by empirical studies of degreerelated efficacy. To address this, Walden and Intermountain have collaborated to conduct the study reported herein. The four year study focuses on a cohort of Intermountain nurses who entered the Walden Master of Science in Nursing (MS in Nursing) program in 2008, comparing them to a group Intermountain nurses who pursued nursing master s degrees at non Walden institutions during the same period. The study examines degree related changes in nursing beliefs and practices, nurse impact on the healthcare system, and changes in retention and promotion using a mixed method, longitudinal design. The study was conducted by Arroyo Research Services, an independent research, measurement and evaluation firm. BACKGROUND Research on outcomes related to advanced degree attainment in nursing is limited. Results from the few available empirical studies suggest that there is a positive relationship between a master in nursing degree and critical thinking ability, career advancement, and personal satisfaction and confidence. Patient care outcomes associated with graduate nursing programs are limited to specialized degrees for nurse practitioners (Considine, Bottie, & Thomas, 2005), and these degrees are not the focus of this study. 13 P age

14 Walden University Master of Science in Nursing: Intermountain Healthcare Research Study Critical Thinking Fowler (1998) argues that critical thinking is one of the key qualities that distinguish a novice nurse from an expert nurse and he provides strategies that promote critical thinking in the profession. The development of critical thinking is a core priority for nursing education in undergraduate and graduate programs (American Association of Colleges of Nursing, 2005). However, there is a lack of empirical evidence demonstrating the benefits of critical thinking in nursing. Among the few available studies, Drennan (2009) found that students who graduated from a master in nursing program demonstrated significantly greater critical thinking abilities than students who were beginning the graduate program. Drennan used the Watson Glaser Critical Thinking Appraisal to measure the abilities of eighty three first year master in nursing program students and 140 graduates from all Master in Nursing programs in Ireland. Results indicated that, after controlling for demographics, graduates had significantly higher critical thinking scores than first year students. Similarly, Pardue (1987) found that among 121 nurses with an associate, baccalaureate, or master in nursing degree, those with baccalaureate and master s degrees had higher critical thinking abilities than those with other degrees. These studies were both observational, and did not examine degree related changes over time. Career Advancement Career advancement is conventionally accepted to be among the reasons nurses traditionally engage in advanced education, (Lava, 1994; Stavropoulou & Biley, 1997), but most studies of career effects on nurses are single sample studies of graduates from specific nursing programs (e.g., Bircumshaw & Chapman, 1988; Kemp, 1988) and don t provide enough data to verify this supposition. Many studies found that nurses with graduate degrees had senior job titles and roles, and higher salaries (Axelsson, Fridlund, Hildingh, Lidell, Martensson, 2009; Hardwick & Jordan, 2002; Hardy, Sinclair, & Hughes, 1984; Pelletier, Donoghue, & Duffield, 2003). All of these studies, however, surveyed nurses a few years after completing a master of science in nursing program within a specific university and without a comparison group, making it problematic to connect the degree to career advancement. Personal Satisfaction and Confidence Master s degrees in nursing can be shown to produce increased personal satisfaction and confidence. In their 10 year follow up study of nurses with a master s degree, Whyte and colleagues 2000) found that participants indicated an increase in confidence and personal satisfaction as a result of the degree. Similarly, Watkins (2011) found that nurses who graduated from a collaborative Master of Science in Nursing program in England and Germany reported increased confidence; participants believed their master s program helped them improve communication, gain respect and credibility, and increased their confidence to make evidence 14 P age

15 based changes in their practice. In a follow up survey of all nursing graduates at one university in the United Kingdom, Hardwick and Jordan (2002) found that 37% of the 43 participants reported increased confidence in questioning decisions relating to care, and the majority reported increased confidence in providing evidence based care. These results are evidenced in other follow up surveys of nurses with graduate degrees (e.g., Boore, 1996; Toren, Kerzman, & Kagan 2011), but typically do not include baseline measurement of career satisfaction or selfefficacy at the beginning of the program. Job Satisfaction Prior studies show job satisfaction is unrelated to graduate education in nursing. Toren and colleagues (2011) compared job satisfaction among nurses with and without a post registration nursing education degree in Israel. Surveys were completed by 419 nurses in two major hospitals and job satisfaction was measured via eleven questions (e.g., My current position meets my idea of an ideal job ). Although nurses with a graduate degree reported slightly greater job satisfaction than those without a graduate degree, this difference was not statistically significant. Leadership Leadership skills are an important component of nursing and can contribute positively to patient and organization outcomes (Cummings, Hayduk, & Estabrooks, 2005; Wong & Cummings, 2007). In a meta analysis of nursing leadership studies, Cummings and colleagues (2008) found that most studies focused on the characteristics and behaviors of nurse leaders, while a few studies examined the effects of participating in leadership development programs. In general, most of the studies reviewed lacked methodological rigor. Quasi experimental studies reviewed in the meta analysis suggested that education programs (e.g., workshops, self directed programs, residency programs) were one of the most important factors affecting leadership skills and competencies among nurses. It was unclear whether these programs were a part of nursing degree requirements, were independent programs sought out by participants, or were part of employer professional development programs. While Claire (1994) found that leadership development was a key component of nursing degree programs, empirical studies of leadership outcomes that result from these programs were unavailable. WALDEN UNIVERSITY Walden provides undergraduate degrees and graduate programs in health sciences, education, management and technology, and social and behavioral sciences. All programs and courses at Walden are conducted online. Within Walden, programs in the School of Nursing in the College of Health Sciences include a Commission on Collegiate Nursing Education (CCNE) accredited Bachelor Degree in Nursing Completion Program, CCNE accredited MS in Nursing and Doctor of Nursing Practice programs, and post master s certificates in nursing (Education, Leadership and 15 P age

16 Management, and Nursing Informatics). In May 2010, Walden was the second largest online graduate nursing program by enrollment (U.S. News and World Report, 2010). INTERMOUNTAIN HEALTHCARE Intermountain is an internationally recognized, nonprofit healthcare system comprised of 22 hospitals, a medical group with more than 185 physician clinics, and an affiliated health insurance company. Its 33,000 employees serve patients and plan members in Utah and southeastern Idaho. Intermountain was awarded MAGNET Status 1, and profiled by the New York Times Magazine for its evidence based model of care (Leonhardt 2009). THE STUDY Walden established its MS in Nursing program in Commensurate with its commitment to objective, third party research on program related outcomes, the University sought the opportunity to conduct a study of the MS in Nursing program as early as In June 2008, a substantial cohort of nurses (31) from the Intermountain Healthcare Urban South Region (Provo, Utah) entered the MS in Nursing program. The cohort presented an opportunity to conduct an independent quasi experimental study, commissioned by Walden and conducted in partnership with Intermountain. Walden contracted with Arroyo Research Services, an independent education research and evaluation firm, to conduct the study beginning in Fall The aim of the study was to test the following hypotheses of interest to both Walden University and Intermountain: 1. Participating in the Walden MS in Nursing program has a detectible, positive effect on the professional practices, leadership, and retention of nurses in the cohort as compared to nurses participating in other master level programs. 2. Participating in advanced nursing programs as a cohort leads to increased commitment to nursing, completion of the program, and health system effects such as changes in institutional policies and practices. 1 MAGNET is an award for quality patient care, nursing excellence, and innovations in professional nursing practice granted by the American Nurses Credentialing Center. 16 P age

17 Conducted over four years using the cohort and a comparison group, the study was organized to test these hypotheses through related research questions organized into four main areas: Student Outcomes, Health System Outcomes, Cohort Effects, and Program Review. RESEARCH QUESTIONS Student Outcomes 1. How do Walden MS in Nursing participants and graduates perform in terms of: Retention in the health system Career advancement Employer satisfaction Student satisfaction with their program 2. What changes in nursing beliefs and practices are most marked among Walden graduates? 3. What changes in approaches to nurse education, teamwork and mentoring are evident in Walden graduates? 4. To what extent do students demonstrate increased proficiency with the skill, proficiency, and knowledge goals of the Walden MS in Nursing program? 5. To what extent do students report increased self efficacy, career commitment or other anticipated outcomes? 6. To what extent do these outcomes differ from participants and graduates of other nursing masters programs? Health System Outcomes 7. Did Walden students have noticeable effects on the health systems in which they worked? 8. What evidence of effects of the Walden education model or specific approaches to nursing can be found within the health systems that employ participants and graduates? Cohort Effects 9. How did group enrollment affect student satisfaction, performance, and persistence within the program? 17 P age

18 10. Did group enrollment result in changes in career advancement, professional success or satisfaction with job and employer? 11. To what extent did cohort participation affect nursing outcomes, nursing practices, nurse retention, or other factors within the health system that employs the cohort? Program Review 12. What did program graduates find most valuable in their Walden experience? 13. What differences in nursing beliefs and practices do program graduates observe between program graduates and other nurses they know? 14. What aspects of the Walden program could be changed to better meet the needs of its graduates and their employers? METHODS RESEARCH DESIGN To answer these questions, the research team employed a mixed method panel design using the cohort of Intermountain nurses enrolled in the Walden MS in Nursing program and comparison group of similar Intermountain nurses enrolled in non Walden master s degree nursing programs. Sample selection and characteristics are described in the next section. The study and associated methods, instruments, consent forms, incentives and reporting were guided by research plans approved through the Walden University and Intermountain Healthcare Institutional Review Boards. Surveys were constructed by the research team based on a review of existing studies using scales designed to capture growth on desired outcomes. Interview and focus group protocols were designed to add depth to the survey responses and to capture detailed input on changes in practice. Measures Cohort and Comparison Group Surveys Cohort and comparison group surveys were used to determine both pre and post growth in outcomes targeted by the MS in Nursing program and student reviews of the program. The survey instruments used for both cohort and comparison groups contained the same items for measuring outcomes, with modification to address differences in consent and program enrollment. Copies of the initial surveys are included in Appendix B and Appendix C; results from 18 P age

19 the survey administration are reported throughout this report. Major areas of survey inquiry included: Nursing practices Beliefs regarding nursing and health care provision Student goals for the program Student self rating on specific knowledge, skills, and proficiencies targeted by the program Student career plans Student satisfaction with various components of the Walden MS in Nursing Cohort and Comparison Group Interviews and Focus Groups Interviews and focus groups were conducted with the cohort and comparison group members as shown in Table 1 Data Collection Timeline. Lines of inquiry for focus groups and interviews: Program effects observed in themselves and others Program related outcomes on the health system or hospital, including changes in policy or approach Specific changes in practices/beliefs/approaches Changes in relationship with peers, supervisors, patients, doctors Changes in career plans, if any, including aspiration and preparation for managing and leading Barriers to successful access to or application of their program Opportunities for more thorough implementation of knowledge and skills obtained in their program Satisfaction with Walden and comparison group programs, student services, etc. Focus groups were conducted separately for cohort and comparison groups. For each round of focus groups, all members of the cohort and comparison groups were invited to participate and were offered $25 gift cards for participating. Focus groups were conducted in person at Intermountain Healthcare s Utah Valley Regional Medical Center (South) and McKay Dee Hospital (North). Focus groups were conducted by one facilitator from the research team with 4 to 6 participants per group, over two day intervals at a range of times for each round to accommodate participant schedules. Six rounds of focus groups were conducted for the cohort group. Cohort group participation per focus group round ranged from 6 to 14 members per round, and averaged 10 participants per round. The initial study design called for surveys of the comparison group, but not focus groups. Due, in part, to limited survey responses and, in part, to the quality of feedback obtained from the cohort focus groups, Intermountain and Walden agreed to add comparison group focus groups to the study for the December 2010 and December 2011 focus groups. Two rounds of focus groups were conducted with the comparison 19 P age

20 group. Comparison group participation was 6 members in the first round and 4 in the second round. Because all comparison group members had graduated as of the December 2011 focus groups, comparison group focus groups were not conducted during the April 2012 round of focus groups. The Baseline focus group protocol is included in Appendix 1. Employer Interviews Cohort and comparison group members were asked to provide permission for the research team to interview their supervisors and employers responsible for the nurses direct management and continuing education. Interviews focused on: Changes in participant practices, work quality, leadership, and other qualities related to program participation Any changes in health system or hospital policy or general practice related to cohort students Satisfaction with, and effects of, group enrollment in the Walden program Employer satisfaction and reviews of the Walden MS in Nursing program Comparisons of the effects of the Walden program to other nursing education programs Hospital/Health System Data Intermountain, with permission from each study participant, provided personnel data to verify demographics, employment, and placement. Specific data points included: Hire date Exit date (if applicable) Placement (job title, department/unit, hospital): initial and current Experience prior to hire Education level at time of hire Basic demographics: date of birth, gender, ethnicity Walden University Data Walden provided data regarding cohort student progress, graduation dates, and completion rates. 20 P age

21 Timeline Table 1 Data Collection Timeline Study Event Timeframe Cohort Start Date June 2008 Study Initiated Fall 2008 Intermountain Healthcare Institutional Review Board (IRB) Approval Jan 2009 Initial Cohort Consent and Survey February April 2009 Initial Focus Groups February 2009 Comparison Group Identification, Consent, and Initial Survey February March 2009 Interim and Post Surveys August 2009 February 2010 April 2012 Interim Focus Groups September 2009 March 2010 December 2010 December 2011 April 2012 Employer/Supervisor Interviews December 2010 April 2012 Follow up Survey Fall 2012 Analysis Analysis was conducted using mixed methods as appropriate to the data. Pre and post survey analysis comparing cohort and comparison group responses used the initial surveys as the starting data point, and the most recently completed interim or post surveys as the ending data point. All analysis links individual respondents across each data point, enabling appropriate use of paired sample tests for scaled items and Somer s d for analysis of directional, non equal interval items. Where appropriate, the study uses calculated effect sizes following Cohen (1988). Effect sizes shown as d for Cohen s d, are reported throughout, and generally show the difference in group means in units of the pooled standard deviation. Researchers use effect sizes to enable comparison across studies using different samples and measures. For the purposes of this study, an effect size of less than 0.30 was considered small, between 0.30 and 0.60 moderate, and over 0.60 large. Statistical procedures were considered to be statistically 21 P age

22 significant where p < Although the sample included every member of the Walden cohort and similar numbers of non Walden comparison group members, the limited sample size means that tests of statistical significance should be interpreted with caution and are not intended to be generalized beyond the specific sample reported in this study. Focus groups and interviews were transcribed and coded by the major lines of inquiry and protocol elements. SAMPLE The cohort studied in this report was comprised of thirty one nurses who enrolled in the Walden MS in Nursing program in summer of 2008 and were employed by Intermountain, a nonprofit health system. All cohort nurses were employed in the Urban South region of Intermountain, which includes Provo, Utah and the southern region of greater Salt Lake City. Walden awards tuition discounts to cohorts from the same institution, and the cohort nurses received a 25% tuition discount from Walden because of their group s relatively large size. Additionally, nurses in the cohort and comparison groups were eligible for tuition reimbursement from Intermountain, which had recently established requirements for bachelor s degrees for nurse manager and nurse educator positions. Nurses also anticipated that many of these positions would also require advanced degrees in future years. Of the 31 nurses who enrolled in the June 2008 cohort, 22 consented to participate in the study. Three members of the cohort entered the Walden RN to MS in Nursing program; the remainder entered the master s program with a BSN or equivalent. The majority (16 out of 22) of nurses selected the Education track. Of the remainder, four selected the Leadership and Management track, and two pursued the Informatics track. Nurses in the cohort represent a diverse range of roles within the health system. They are employed across the system in units from behavioral health to emergency care to maternal health. The comparison group in this study was comprised of 21 nurses from two sub groups. Eligibility criteria for the initial comparison group included enrollment in a master s degree program in nursing at the time of recruitment (April, 2009) and employment in the Urban North region of Intermountain. A preliminary survey was sent to the 33 nurses that Intermountain determined had met these criteria. Of these 33, 24 responded to the initial screening survey, 17 indicated that they were enrolled in a master s degree program in nursing, 1 declined to consent and 14 responded to the full comparison group pre survey. In order to achieve parity between the number of nurses in the cohort and comparison group, the comparison group was expanded in February 2010 to include nurses employed in the Urban South region enrolled in any master s degree in nursing program, excluding nurses eligible for the cohort group. Screening surveys were sent to 35 additional nurses, 10 responded, of which 7 met the criteria for participation. All 22 P age

23 nurses in the comparison group held bachelor s degrees prior to entering their master s programs. The cohort and comparison groups were well matched in terms of prior education and basic demographics. All were seeking master s degrees and identified as white, non Hispanic, with a median age between 45 and 50. Cohort participants had slightly longer average tenure with Intermountain (19 versus 14 years), and were more likely to be female (100% cohort versus 75% comparison). One significant difference was that the comparison group had completed more of their master s programs at the time of their baseline surveys. Additionally, all comparison group members who completed post surveys had completed their programs at the time of the survey. Only 29% of the cohort nurses included in the pre post analysis had completed the program at the time of their final survey, although an additional 41% were expected to complete the program within months of their last survey. In baseline surveys, cohort nurses also reported being more likely to remain with their current employer (86% of cohort nurses were very likely to remain with their employer for two years, 42% of comparison group nurses were very likely to remain with their employer for two years). On nearly all other aspects of work and management skills, cohort and comparison group self rankings were equivalent at baseline. Table 2 provides basic demographic information for study participants in the cohort, initial comparison group, and expanded comparison group. Table 2 Study Sample Demographics n= Median Age Age Range Median Tenure (Years) Tenure Range (Years) % Female % White/ Non Hispanic Cohort Initial comparison group Expanded comparison group Source: Cohort and Comparison Group Pre-Surveys; Intermountain Healthcare. The cohort group of nurses had a higher proportion of nurse educators, both with the formal title of nurse educator, as well as nurse managers and clinical specialists who provided education services to their units. Table 3 shows the distribution of specific job titles among the cohort and comparison groups. 23 P age

24 Table 3 Distribution of Study Sample Job Titles Cohort Comparison n= Nurse Clinical Specialist 13% 6% Nurse Manager 25% 44% Nurse Supervisor 9% 6% RN 34% 38% Nurse Educator 16% 0% Quality Control 3% 6% Source: Intermountain Healthcare, June All members of the cohort and the comparison group were pursuing master s degrees in nursing. While some members of the comparison group were engaged in fully online programs, most participated in programs that were either fully traditional, in person programs or hybrid programs that mixed in person and online course participation. Details regarding mode of delivery are shown in Table 4. Cohort participants were primarily pursuing education concentrations with the Walden MS in Nursing program, reflecting the current positions and responsibilities of the cohort nurses. While all comparison group respondents were identified by Intermountain as pursuing master s degrees in nursing relevant to their positions, data regarding specific universities attended or program concentrations pursued were not available for the comparison group. The percentage of cohort group participants in each Walden program concentration is shown in Source: Comparison Group Post Survey 24 P age

25 Walden University Master of Science in Nursing: Intermountain Healthcare Research Study Table 5 Cohort Program Concentrations. Table 4 Study Sample Graduate Degree Mode of Delivery Cohort Comparison = Fully online 100% 44% Hybrid (in person and online) 28% Traditional 28% Source: Comparison Group Post Survey 25 P age

26 Walden University Master of Science in Nursing: Intermountain Healthcare Research Study Table 5 Cohort Program Concentrations Cohort n= 22 Education 68% Education RN MSN 5% Informatics 5% Informatics RN MSN 5% Leadership & Management 18% Source: Walden Institutional Research, June 2012 STUDY FINDINGS Findings are reported by key area of inquiry for student outcomes, health system outcomes, cohort effects and program review. STUDENT OUTCOMES The core analysis of student outcomes was conducted using data from survey instruments discussed in the research design section of this report. Additional data was provided by structured focus groups. The focus groups incorporated discussions around the same elements as the surveys and were designed to provide the opportunity for deeper exploration and validation of the study questions. Interviews with supervisors of both cohort and comparison group members were structured to provide information about the effect of the MS in Nursing program on the health system and to validate self reported data. Table 6 and Table 7 show the response rate for each survey conducted in the course of the study. As discussed in the data analysis section, where pre post comparisons are used, the initial survey for each respondent serves as the pre survey, while the most recent completed interim survey serves as the post survey. The expanded comparison group was asked to complete only one survey. While a pre and post survey was planned for this group, the majority had completed or substantially completed their studies at the time they were identified for participation in the study. The pre survey was therefore not conducted. 26 P age

27 Walden University Master of Science in Nursing: Intermountain Healthcare Research Study Table 6 Cohort Survey Response Rates Spring 2009 Initial Survey Fall 2009 Interim Survey Spring 2010 Interim Survey Spring 2012 Interim or Post Survey Table 7 Comparison Group Survey Response Rates Spring 2009 Initial Survey Fall 2009 Post Survey Spring 2010 Expanded Group Survey RETENTION AND CAREER ADVANCEMENT The study examined retention and career advancement via questions pertaining to likelihood of remaining with current employer and seeking a change in position. In addition, researchers examined actual position changes. A far higher percentage of cohort nurses were very likely or likely to remain with their current employer for at least two years, as compared to the comparison group. This was true throughout the study; there was no significant effect of the program on either likelihood of seeking a higher position or the likelihood of remaining with their current employer. Table 8 shows the percentage of respondents and their likelihood to remain with their current employer, Table 9 shows the change in likeliness of remaining with employer, and Table 10 shows the change in likeliness of seeking a higher position. 27 P age

28 Walden University Master of Science in Nursing: Intermountain Healthcare Research Study Table 8 Likelihood of Remaining with Employer for Two Years (percent) n= Very likely Likely Somewhat likely A little likely Not likely Cohort Pre % 14.0% 4.0% 0.0% 0.0% Cohort Post % 16.0% 0.0% 0.0% 0.0% Comparison Pre % 15.0% 8.0% 8.0% 23.0% Comparison Post % 17.0% 5.0% 17.0% 28.0% Table 9 Change in Likelihood of Remaining with Employer for Two Years Mean Pre Mean Post Mean Diff. Std Deviation t df Sig (2 tailed) Effect Size Cohort Compa rison Note: Cohort n=17, Comparison n=8. 28 P age

29 Walden University Master of Science in Nursing: Intermountain Healthcare Research Study Table 10 Change in Likelihood of Seeking Higher Position within Six Months Mean Pre Mean Post Mean Diff. Std Deviation t Df Sig (2 tailed) Effect Size Cohort Compa rison Note: Cohort n=17, Comparison n=8. Examination of employment data from Intermountain showed that nearly 60% of nurses made some change in position during the course of their studies, as shown in Table 11. A higher percentage of cohort nurses (45%) remained in the same position than comparison group nurses (35%), a finding consonant with the self reported likelihood to remain in the same position reported above. Table 11 Changes in Position No change Change higher Change lower Terminated Retired Total Cohort Comparison The strong commitment of cohort nurses to their current employer was also evident in focus groups. Many cohort nurses were mid career level, and reported themselves to be employed in positions they enjoyed. Additionally, Intermountain is not only the largest healthcare employer in the region, it has a solid reputation as a health system committed to the development and retention of nurses. Examples from the focus group discussions that illustrate cohort member commitment to their current employer and positions include the following: 29 P age

30 I get calls every six months or so [from recruiters]. I just tell them I m happy. Are you sure, do you really enjoy what you do? Yeah, I do. You re not going to get me away from it... This may not be the answer I should give but in ten years I want to have the same job I have now. I just want to do it better. With respect to nurses who changed to lower level positions, focus group discussions clarified that at least several of these changes were to positions within the selected specialty being studied by the nurse. For example, one change was from an RN to a systems technician position. This particular nurse chose the informatics track and made a deliberate change to a position within her area of interest and new expertise. This nurse explained she had left critical care nursing because of physical reasons. Her new skills gave her the opportunity to find something she loved that was still within healthcare with the ability to impact patient care. CHANGES IN BELIEFS AND PRACTICES The study examined changes in beliefs and practices with respect to several areas, including initiating evidence based practice, engaging in research studies and presentations, and advancing positive social change. These areas were examined through questions that ranked the likelihood of engaging in the particular activity and through open ended questions and focus group discussions in which respondents were asked to provide specific examples of change. Evidence Based Practice Over 80% of survey respondents answered that they were very likely or likely to engage in promoting or initiating evidence based practice within their work, prior to entering the MS in Nursing program. Despite this strong baseline commitment to evidence based practice, respondents consistently cited this as an area of growth in focus groups. Employer interviews provided further confirmation of growth in this area. In the final survey, 90% of cohort respondents (n= 18) indicated they were very likely to promote or initiate evidence based practice. Similarly, 90% indicated their studies helped them implement innovation or evidence based practice. For the comparison group (n=9), 67% reported they were very likely to promote or initiate evidence based practice, while 56% indicated that their studies helped them implement innovation or evidence based practice. While this shows a positive response from a larger percentage of cohort respondents, a comparison of pre and post test survey results showed a statistically significant and positive change among both cohort and comparison group members, as summarized in Table P age