PREDICTIVE ANALYTICS FOR STUDENT SUCCESS: Developing Data-Driven Predictive Models of Student Success

Transcription

1 1 PREDICTIVE ANALYTICS FOR STUDENT SUCCESS: Developing Data-Driven Predictive Models of Student Success Final Report University of Maryland University College January 6, 2015 A Research Project Funded by the Kresge Foundation

2 2 Table of Contents EXECUTIVE SUMMARY... 4 SECTION 1: INTRODUCTION Grant Partnership Objectives and Milestones SECTION 2: LITERATURE REVIEW Theoretical Models of Community College Transfer Student Performance Educational Data Mining Predicting Transfer Students First-Term GPA Predicting Transfer Student Re-Enrollment Predicting Re-Enrollment for Non-Traditional Students Literature Guiding Interventions Community College Transfer Students Transitioning Literature to Support Specific Interventions Checklist Community College Mentor SECTION 3: RESEARCH SCOPE AND DESIGN Research Questions Student Population SECTION 4: DATA SOURCES SECTION 5: SURVIVAL ANALYSIS: REGISTRATION AND WITHDRAWAL IN THE ONLINE CLASSROOM SECTION 6: PROFILES OF STUDENTS USING DATA MINING Profiles of Student Success Further Findings from Data Mining SECTION 7: PREDICTIVE MODELING OF STUDENT SUCCESS Initial Predictive Modeling Predicting Successful GPA Predicting Re-enrollment Updated Predictive Modeling Population Predicting Earning a Successful First-term GPA Predicting Re-Enrollment Predicting Retention Predicting Graduation Summary of Results from Predictive Modeling SECTION 8: GRADUATION RATES SECTION 9: EXAMINING LEARNER BEHAVIOR IN THE ONLINE CLASSROOM Online Classroom Behaviors and Class Performance... 51

3 3 Student Level Online Classroom Behaviors and Course Performance Engagement Profiles and Course Performance Modeling Retention SECTION 10: STUDENT MOTIVATION AND SELF-REGULATION Population Methodology Results Key Findings SECTION 11: INTERVENTION IMPLEMENTATION AND EVALUATION Checklist Participants Results College Success Mentoring Participants Results Jumpstart Summer Results Accounting 220 and Accounting Participants Results Key Findings SECTION 12: DISSEMINATION Presentations at Conferences Publications Learner Analytics Summit Success Calculator SECTION 13: FINANCIAL SUPPORT SECTION 14: CONCLUSIONS Future Directions REFERENCES... 77

4 4 EXECUTIVE SUMMARY The purpose of the Predictive Analytics for Student Success (PASS) project was to: (a) aggregate data across multiple institutions to track the academic progress and completion of community college transfer students, (b) identify factors associated with success, and (c) implement interventions that promote student success. In completing the PASS project research and interventions, University of Maryland University College (UMUC) partnered with two community colleges, Montgomery College (MC) and Prince George s Community College (PGCC). This work was funded by a grant from the Kresge Foundation -- Developing Data- Driven Predictive Models of Student Success. The purpose of the grant was: To build an integrated database tracking students across institutions from community college to UMUC. To use predictive statistical models and data mining techniques to track and model students progress across institutions. To identify factors predictive of students success at UMUC To inform the development of interventions aimed to improve outcomes for undergraduate students transferring from community colleges to UMUC or to other four-year institutions. This report will summarize the data development, research, intervention implementation and evaluation, dissemination, and application creation completed through the PASS project. Phase 1 During the first 24 months (Phase 1) of the grant, UMUC and the partner institutions developed and signed a Memorandum of Understanding (MOU) to ensure data security and establish parameters for data use. The MOU allowed the PASS project team to conduct research using individual student data while protecting student information and confidentiality. Once the MOU was in place, researchers identified the population of interest, conducted an initial literature review to identify variables of interest, and began data collection and exploratory analyses. The research team identified over 250,000 students enrolled at UMUC between 2005 and Of those, over 30,000 students transferred from MC and PGCC. Student demographics, academic performance at the three institutions, behavior in the online classroom at UMUC, and advising data were combined into an integrated, multi-institutional database: the Kresge Data Mart (KDM). The literature review covered student performance in online courses, successful course completion, factors associated with re-enrollment and retention, and the use of data mining techniques in higher education. Existing research showed that factors such as the number of schools attended, the number of credits transferred, and community college GPA influenced student success. Key measures of success included successful course completion and retention.

5 5 In Phase 1, initial data mining was conducted to identify variables that were associated with success. Specific courses were identified as having predictive value in relation to success at UMUC. Regression analyses determined that student online classroom activities prior to the start of a class (i.e., entering the online classroom prior to the first day) and during the early weeks of the course were predictive of successful course completion. Phase 2 Phase 2 of the PASS project was completed in months 25 to 36 of the grant. The initial plan for Phase 2 was to: Secure external evaluators Further develop collaboration with the community colleges Identify the scope of the project Clarify the research plan and conduct associated analyses Begin initial dissemination of research findings UMUC began meeting regularly with the community colleges to develop the Phase 2 research plan and evaluate research findings and grant progress. Two external evaluators were selected to conduct an independent evaluation of the research project. These collaborations proved to be highly beneficial in developing the research program and designing interventions. As a result of the collaborations, new data were identified for collection, and a full scope of the research was outlined in the form of a research plan. A research plan was developed to model students progress and performance from the community college to graduation from a four-year institution. The research plan created a model addressing the relation between students prior academic work and performance at UMUC to include graduation. The full path model of students academic trajectory from community college to UMUC is below. Community College Data UMUC First Term GPA Reenrollment Retention Graduation The plan identified the following research goals for Phase 2: 1. To develop profiles of transfer students at UMUC 2. To identify factors from students community college academic backgrounds that predict success at UMUC 3. To develop predictive models of student success based on demographic information, community college course taking behaviors, and first-term factors. 4. To develop interventions designed to improve the success of students transferring from community colleges to UMUC.

6 6 Phase 2 considered two primary outcomes of interest in predictive models: 1) earning a first-term GPA of 2.0 or above at UMUC, termed successful first-term GPA, and 2) students re-enrollment at UMUC within 12 months following their first academic term, termed retention. Key findings from Phase 2 include: Across studies, age and marital status were associated with success at UMUC. Older, married students were found to be more likely to succeed. Four profiles of student success at UMUC were identified based on students GPAs and retention rates. The profiles differed in terms of community college course taking preferences and course load and in the change in GPA when transferring to UMUC. These results suggest that the degree of student preparedness, particularly in specific target areas (e.g., accounting, economics), is predictive of success at UMUC. Course efficiency, or the ratio of credits earned to credits attempted, in the community college was determined to be a predictor of success at UMUC. The higher the course efficiency, the more likely a student was to succeed. A new factor, delta GPA, was introduced in these analyses, corresponding to the difference between students GPA at the community college and at UMUC. While most students experienced a decreased GPA when transferring to UMUC, the magnitude of this decrease was predictive of students continued enrollment at UMUC beyond the first term. Students who took math or honors courses in community college were more likely to succeed at UMUC, suggesting that rigor of community college courses may prepare students to succeed at a four-year university. Student behaviors in the online classroom indicated high variability in the extent to which they engage in course content and course-related activities. A substantial percentage of students accessed course content and course materials to a limited extent, thus impacting successful course completion. Phase 3 Phase 3, the final year of the Kresge Grant, focused on four goals: 1. Data enhancement 2. Extended research 3. Implementation and evaluation of the interventions 4. Continued dissemination of research findings and intervention results As a result of continued collaboration with the community colleges, the KDM was expanded to include additional variables from the community colleges as well as updated data from UMUC to allow for expanded analyses of retention and graduation. With the inclusion of new data, Phase 3 analyses focused on re-enrollment, retention, and graduation from UMUC.

7 7 Key Findings from Phase 3 include: Demographic Factors. Gender and marital status were associated with both performance (earning a successful first-term GPA) and persistence (re-enrollment). These characteristics may indicate students maturity and commitment to pursuing academic goals. Interestingly, while African American status was negatively associated with earning a successful first-term GPA, it was positively associated with persistence metrics. This suggests that while not always successful in their first semester, African American students are nonetheless committed to their educational goals. Math at the Community College. Across models examining both persistence and performance, variables associated with taking math at the community college were found to be significant predictors. Within our sample, taking math at the community college reflects academic abilities and may also reflect students commitment to meeting the requirements necessary for transfer and graduation. Community College Success and Completion. In models predicting first-term GPA, reenrollment, and graduation, students community college GPA was a significant predictor. This suggests that, overall, performance at the community college matters for success and persistence at a four-year institution. First-Term Performance. As in findings from Phase 2, students performance in their first semester at UMUC remains crucial in predicting re-enrollment, retention, and graduation. In fact, across models, it was the strongest individual predictor of performance. Firstterm GPA may be an indicator of factors contributing to students success, beyond academic abilities. Specifically, students who are better at acclimating to an online university and the demands associated with a four year institution may have a higher first-term GPA and may be more able to persist. Online Classroom Engagement. A particularly rich finding from Phase 3 analyses is the association between student online classroom engagement as measured in the learning management system (LMS) and course performance. The general pattern was that students earning higher grades in a particular course were also significantly more engaged in the online classroom. Further, online course engagement, in combination with students community college GPA, was predictive of overall course performance; such a model linked students community college backgrounds with four-year institutional experience. Phase 3 also included the examination of the efficacy of four interventions aimed at promoting community college transfer student success at UMUC. In addition, two interventions at the community college were used to better prepare students for transfer. Interventions undertaken at UMUC were: Checklist. New student orientation checklist administered to community college transfer students to aid them in navigating online resources at UMUC. Although no significant differences were found, students responding to an evaluation survey found the checklist to be a useful tool.

8 8 Mentoring. Eight week structured mentoring program, where new UMUC community college transfer students were paired with a peer mentor -- a successful student at UMUC who had transferred from the same community college. Each week, mentors ed mentees with study tips and information to support adjustment to UMUC. Although no statistically significant improvements in semester performance were found for mentees, unexpectedly, students serving as mentors had a significantly higher cumulative GPA and a significantly higher rate of successful course completion when compared to the control group of students who were invited to be mentors and elected not to participate. This phenomenon may be due to the bias inherent in the self-selection process. Jumpstart Summer. A program that paired mentoring with Jumpstart, a four-week onboarding course, designed to support students goal setting and academic planning. Four experimental conditions were examined: (a) a control group, (b) students only completing the Jumpstart course, (c) students only participating in the mentoring program, and (d) a Jumpstart Summer group, receiving both mentoring and enrolled in the Jumpstart course. No significant differences in performance were found; however, students successfully completing the Jumpstart course had a higher rate of successful course completion and reenrolled at a higher rate. Accounting 220/221: The online tutoring intervention was developed by faculty for students taking Accounting 220 and Accounting courses with a disproportionally high failure rate both at UMUC and nationally. Students who participated in the online tutoring had a significantly higher GPA at the end of the semester and a significantly higher rate of successful course completion, when compared to students not participating in online tutoring. Interventions developed at the community colleges were: Diverse Male Student Initiative (DMS-I). DMS-I is a two-year program at Prince George s Community College that provides minority male students with role models and academic and career mentoring. DMS-I held a two-day summer institute at PGCC that featured keynote speakers and awarded book and tuition vouchers for early registration to participants with the aim of improving academic planning and persistence. PGCC and UMUC will track and evaluate the success and persistence of students who participated in the program and who transfer to UMUC. Women s Mentoring, Boys to Men, TriO: Women s Mentoring, Boys to Men, and TRiO are comprehensive mentoring programs, developed at Montgomery College, that provide minority students with comprehensive academic and social support throughout their transfer pathways from high school to MC, and ultimately to a four-year institution. MC and UMUC will identify students participating in these programs who transfer to UMUC and will track them to evaluate their performance. UMUC will provide similar mentoring and support if these students transfer to UMUC. Findings from research and interventions were disseminated through ten conference presentations and manuscripts. In addition, a website, was created to share project goals and results with a broad audience of stakeholders.

9 9 This report was produced by the UMUC Office of Institutional Research and Accountability and contains 12 sections: Section 1: Introduction Section 2: Literature Review Section 3: Research Scope and Design Section 4: Data Sources Section 5: Survival Analysis: Registration and Withdrawal for Online Courses Section 6: Mining of Community College Data Section 7: Predicting Student Success Section 8: Graduation Rates Section 9: Data Mining of Online Learner Behavior Section 10: Students Motivation and Self-Regulation Section 11: Implementation and Evaluation of Interventions Section 12: Dissemination Section 13: Financial Statement Section 14: Conclusions

10 10 SECTION 1: INTRODUCTION The purpose of this report is to present the results of research conducted by the University of Maryland University College (UMUC), in partnership with two community colleges, Montgomery College (MC) and Prince George s Community College (PGCC) as part of the PASS project. The scope of work was undertaken as part of a grant from the Kresge foundation and includes: (a) data development, (b) research using data mining and predictive modeling to examine community college transfer student success, and (c) intervention development, implementation, and evaluation to provide academic, social, and institutional support to community college students both prior to and after transfer. The project was broken out into three phases. Each phase was built upon results from the previous phase, resulting in continued research development and comprehensive analyses. A research plan identifying the scope of the project began in Phase 1. The final research design and methods were finalized in Phase 3. The research plan was developed to conceptualize students academic pathways from the community college, to transfer to a four-year institution, to graduation from UMUC. In developing the research plan, specific milestones in students academic pathways were modeled. These included: (a) earning a successful first-term GPA, (b) re-enrolling in the immediate next semester after transfer, (c) retention (re-enrollment within a 12-month window), and (d) graduation within an eight-year period. Each of these milestones was predicted based on data aggregated from the community college and UMUC. Specifically, students demographic information, community college course taking behaviors, indicators of first-semester performance at UMUC, and behaviors in the online classroom were used in predictive modeling and in data mining. A model presenting students academic trajectories from transfer to graduation that guided the research was developed. PASS project goals included: To develop a data sharing partnership and create a memorandum of understanding between partner community colleges and UMUC To build an integrated database tracking students across institutions, from community college to UMUC To integrate data from students community college backgrounds with UMUC performance data for use in research To use data mining to develop profiles of transfer student success at UMUC To identify factors from students community college academic backgrounds that predict success at UMUC Develop predictive models of UMUC first-term GPA, re-enrollment, retention, and graduation based on community college data Examine graduation rates of community college transfer students at UMUC Examine online classroom engagement as associated with course performance Profile students motivational and self-regulatory attributes Develop, implement, and evaluate interventions aimed at promoting community college transfer students success Disseminate research findings

11 11 Grant Partnership UMUC is a four-year public university that offers online degree programs to a diverse population of working adults. With support from the Kresge Foundation, UMUC established partnerships with two Maryland community colleges that also serve large and diverse student populations. Montgomery College (MC), established in 1946, enrolls over 60,000 students annually. Prince George s Community College (PGCC) enrolls more than 40,000 students from approximately 128 different countries. Both institutions serve the metro-d.c. area, but differ in that PGCC serves more low-income students. Both institutions have endorsed the goals of this project and are committed to working with UMUC to find ways to promote student success throughout their academic careers. Objectives and Milestones Specific objectives and milestones were identified for each phase of the research project. These objectives and milestones have been modified throughout the course of the project, but are consistent with grant requirements. Table 1 presents the objectives and milestones for each phase. Table 1. Project objectives and milestones Objectives Milestones Status Phase 1 April 2011 October 2012 Develop a Project Action Plan Data Collection and Preparation Data Analysis Develop a project action and collaboration plan with the partnering agencies. Prepare a data universe (integrated database system) on CC transfer students in the UMUC population (KDM) Understand variables; define student characteristics and retention data; develop data dictionary. Conduct initial predictive analyses and employ data mining techniques to identify factors contributing to students success Conduct ongoing project evaluation. Take action on identified areas for improvement. Complete Complete Complete Complete Project Complete Evaluation Phase 2 November 2012 October 2013 Develop and Analyze data and identify factors that predict Complete Validate success/failure. Analytic Models Validate predictive analyses and models developed Complete of Student through data mining techniques to predict students Success success and retention at UMUC. Disseminate Key Findings Build student profiles based on analyses. Discuss results with Kresge Workgroup and share with advisory board. Discuss results with Project Partners and obtain feedback. Complete Complete Complete

12 12 Objectives Milestones Status Present key findings at national conferences on higher Complete education Develop Work with stakeholders at UMUC and CC partners to Complete Interventions develop a list of potential interventions. Project Conduct ongoing project evaluation. Take action on Complete Evaluation identified areas for improvement. Research Plan 3 Design and develop KDM2 to update and improve data related to student success Complete Plan Phase 3 analyses on expanded integrated data. Complete Phase 3 November 2013 December 2014 Develop Review relevant literature on interventions that Complete Interventions promote student success in online learning. Develop an implementation plan and timeline for Complete piloting of interventions. Implement Pilot Implement and evaluate pilot interventions. Complete Interventions Disseminate Develop and disseminate report on the pilot Complete Results on Interventions interventions Phase 3 Analyses Develop and execute Phase 3 research plan Complete Report Findings Present key findings from Phase 3 analyses at national Complete conferences; publish research in journals Prepare written report of both Phase 3 analyses and Complete Dissemination of Results and Resources Project Evaluation full scope of Kresge grant work. Develop website and repository for educational data mining and student success. Host a national convening on data mining and learner analytics. Deliver final project evaluation. December 2014 Complete December 2014

13 13 SECTION 2: LITERATURE REVIEW The literature review was conducted over the course of the four-year project. This section presents a review of the literature in the following areas: Theoretical models of community college transfer student performance Educational data mining Predicting transfer student first-term GPA Predicting transfer student re-enrollment Literature guiding interventions Community college student transitioning Literature to support specific interventions Theoretical Models of Community College Transfer Student Performance Two theoretical models of community college transfer student performance and persistence have guided work in the field, as well as in the PASS project analyses. The first is Tinto s (1975, 1987) Student Integration Model, which applies a psychological lens to understanding student attrition. The Student Integration Model identifies four aspects of student-institution interactions that have an effect on persistence. Specifically, these are the background characteristics and academic goal commitments that students bring to a university setting, and in turn, their effects on students academic and social integration at the transfer institution. Background characteristics include students demographic attributes, family backgrounds, and experiences prior to college (Tinto, 1975). Goal commitments include learners motivation for degree pursuit and educational expectations as well as institutional commitment to a particular university. Academic and social integration is based on students interactions with a variety of institutional features over time. Tinto (1975) suggests that these interactions may be evaluated based on both structural and normative considerations. Structural considerations refer to objective and explicit social and academic standards that students may have to meet (e.g., a minimum GPA, meeting with an advisor), whereas normative components of integration relate to students identifications with these standards (e.g., earning a high GPA). Tinto emphasized the central importance of students institutional integration, both academic and social, by saying, we learned that involvement matters and that it matters most during the first critical year of college, (Tinto, 2006, p. 3; Upcraft, Gardner, & Barefoot, 2005). At the same time, academic and social integration into a transfer institution are not givens for many students. Building on Tinto s earlier work (1975), Bean and Metzner (1985) developed a model of attrition, reflecting the experiences of non-traditional undergraduate students, termed the Conceptual Model of Non-Traditional Student Attrition. In their definition, non-traditional undergraduate students may be defined as those who are older (i.e., 25 and above, Stewart & Rue, 1983), enrolled part-time, non-residential, commuting to campus, or representing some combination of these characteristics (Bean & Metzner, 1985). Understandably, this population of students is considered to undergo a different socialization process from that of traditional students conceptualized in Tinto s model (1975). Non-traditional students may have different experiences with and potential for institutional commitment and social integration. Bean and Metzner (1985) suggest that this may be because older students exhibit greater characteristics

14 14 associated with maturity and therefore may be less open to the socialization process and because these students have more limited contact with socializing agents (e.g., faculty, peers, Chickering, 1974). More generally, non-traditional students may be less interested in institutions social culture, and rather more concerned with academic offerings and credentials. Juxtaposing the experiences of traditional and non-traditional students, for non-traditional learners there is (a) more limited interaction with faculty and peers as well as with college services (i.e., more limited social integration, as per Tinto, 1975), (b) similarity in academic focus and experience (i.e., parallel classroom experience), and (c) much greater interaction with the external, non-institutional environment (Bean & Metzner, 1985). Based on the differences identified between traditional and non-traditional students, Bean and Metzner (1985) conceptualize students decisions to drop out as predicated on four general types of variables. The first of these are background factors, including students demographics, past academic performance, and educational goals and expectations. The second group of considerations is students academic performance, or factors reflecting learners grades, study habits, and pursuit of major at the transfer institution. The third group of factors are students intent to leave, considered to be more psychological; these include students goal commitment, perceived utility of a given degree, and institutional satisfaction. Finally, unique to this model, the fourth group of factors are external factors that may have a direct effect on students decisions to drop-out; these include finances, out-of-school work, and family commitments (Bean & Metzner, 1985). There are two compensatory relationships between variables identified. First, if students academic outcomes are low, they may nonetheless persist, compensating with high levels of psychological commitment. Further, when academic performance is low, students will persist if environmental factors support their continued enrollment. Conversely, when environmental factors do not support persistence, for non-traditional students, even high academic performance may not be sufficiently compensatory. More generally, Bean and Metzner (1985) suggest that for non-traditional students, environmental factors may have a much more pronounced effect on attrition decisions than do academic factors, as non-traditional students are much more closely affiliated with the non-institutional environments than are traditional students residing on university campuses (Bean & Metzner, 1985; Metzner, 1984). As such, understanding non-traditional student persistence may be particularly challenging at the institutional level as, in large part, it may be attributed to environmental factors that the institution may not be aware of or able to control. Indeed, for non-traditional students, the primary point of institutional interaction has to do with academic factors; as such these areas represent targets for intervention (Bean & Metzner, 1985). As part of the PASS project, we were interested in gaining insight into community college transfer students persistence at a four-year institution by looking longitudinally to consider which background factors, including learner characteristics and community college experiences, may impact student re-enrollment and continued pursuit of educational goals.

15 15 Educational Data Mining Current literature on student success focuses on outcomes such as course success, course withdrawal and retention. For example, variables such as student characteristics, previous course work, grades, and time spent in course discussions and activities may be useful in predicting course success (Aragon & Johnson, 2008; Morris & Finnegan, 2009; Morris, Finnegan & Lee 2009; Park & Choi, 2009). Course-level variables acquired from student login data from the LMS may have predictive value in measuring course withdrawal rates (Willging & Johnson, 2008; Nistor & Neubauer, 2010). Variables such as student characteristics, number of transfer credits, final grade in any given course, experience in online environments, and course load may be useful in predicting re-enrollment and retention (Aragon & Johnson, 2008; Morris & Finnegan, 2009; Boston, Diaz, Gibson, Ice, Richardson & Swan, 2011). Although these studies showcase a variety of findings related to student success, the majority of studies of retention in online learning environments use traditional statistical or qualitative methods. Park and Choi (2009) point out that expansion of methods such as data mining may have utility when student, course, program, and institutional level variables are well defined and institutionally meaningful. Literature related to educational data mining focuses on exploratory research. Data mining is a method of discovering new and potentially useful information from large amounts of data (Baker & Yacef, 2009; Luan, 2001). Educational data mining is a subset of the field of data mining that draws on a wide variety of literatures such as statistics, psychometrics, and computational modeling to examine relationships that may predict student outcomes (Romano & Ventura, 2007; Baker & Yacef, 2009). In educational data mining, data mining algorithms are used to create and improve models of student behavior in order to better understand student learning (Luan, 2002). Data mining methods are most helpful for finding patterns already present in data, not necessarily in testing hypotheses (Luan, 2001). Baker and Yucef (2009) suggest that research in higher education should use a variety of algorithms, such as classification, clustering or association algorithms in determining relationships between variables. Although many definitions of these techniques exist in data mining literature, Han and Kamber (2001) offer the following definitions. Classification is the process of finding a set of models or functions that describe and distinguish data classes or concepts to predict a class of objects whose class label is unknown. Clustering analyzes data objects that are related to similar outcomes without consulting a class label. Association is the discovery of rules showing attribute value conditions that occur frequently together in a given set of data (Han & Kamber, 2001). Recent research suggests that these data mining algorithms can be used to examine variables related to student success. Yu, DiGangi, Jannach-Pennell, Lo, and Kaprolet (2010) used a classification algorithm to explore potential predictors related to student retention in a traditional undergraduate institution. In this study, the authors used a decision tree to explore demographic, academic performance, and enrollment variables as they related to student retention. This study revealed a predictable relationship between earned hours and retention, but also found that at this institution, retention was closely related to state of residence (in-state/out of state) and living

16 16 location (on campus/off campus). The authors speculate that this finding points to the potential utility of online courses in improving retention for out-of-state or off-campus students. Despite these recent developments in exploring variables related to student success in traditional higher education settings, research using data mining techniques to uncover relationships among variables in online courses is limited in scope. The PASS project is designed to fill this gap in the extant literature by utilizing data on online students who attended multiple institutions. Predicting Transfer Students First-Term GPA Generally, the transition from community college to a four-year university has been considered to be a stressful period for students. In early examinations of this transitional period, Hills (1965) determined that when students from junior college transfer to a four-year university they might experience an appreciable loss in their level of grades (p.209), termed transfer shock. Transfer shock has been defined as a decrease in academic performance (i.e., GPA) experienced by students in their first semesters at a four-year university, due to difficulties with adjustment (Keeley & House, 1993). Since Hill s (1965) initial exploration, a wealth of studies have emerged examining transfer shock and students decreases in GPA when transitioning from community college to a four-year university (e.g., Best & Gehring, 1993; Keeley & House, 1993; Preston, 1993; Soltz, 1992). However, recent research has painted a more nuanced picture of transfer shock. Cejda, Kaylor, & Rewey (1998) determined that transfer shock is discipline specific. For instance, while students transferring into mathematics and science majors did experience a drop, those majoring in the fine arts and humanities actually experience an increase in GPA. Further, in a metaanalysis of 62 studies examining transfer shock, Diaz (1992) determined that while the majority of studies did find that community college students experience a transfer shock, it was slight (i.e., one half of a grade point or less); also, the majority of studies reviewed found that students recovered from transfer shock over the course of their university careers. Nickens (1972) skeptical of transfer shock and recovery suggests that transfer students GPAs cannot be distinguished from the GPAs of their native counterparts. Specific decreases in GPA may be explained by difference in institutional practices and any subsequent increases in GPA may be explained by regression to the mean and the attrition of weaker students (Nickens, 1972). Regardless of findings, across studies examining community college students performance when transferring to a four-year university, first-term GPA has been a key outcome of study (Carlan & Byxbe, 2000; Driscoll, 2007; Glass & Harrington, 2010; Hughes & Graham, 1992; Townsend, McNerny, & Arnold, 1993). This may be because first-term GPA has been considered to be a barometer of transfer students success and adjustment to a four-year institution (e.g., Knoell & Medsker, 1965) as well as level of preparedness to meet the academic demands of a four-year university (Carlan & Byxbe, 2000; Roksa & Calcagno, 2008). Further, first-term GPA has been considered to be strongly associated with persistence or students retention and graduation from a four-year university (Gao, Hughes, O Rear, & Fendley, 2002). Indeed, there have been a number of studies examining predictors of first-term GPA for community college transfer students (e.g., Graham & Hughes, 1994; Townsend et al., 1993). Most commonly, demographic factors have been examined as potentially impacting community

17 17 college students transfer success. For example, Durio, Helmick, and Slover (1982) found that demographic factors (i.e., gender and ethnicity) impacted transfer students success. Examining an expanded pallet of variables predicting first-term GPA, Keeley and House (1993) considered students age, gender, ethnicity, college major, residence status, as well as class standing (e.g., sophomore) as predictive of first-term GPA. In particular, age (i.e., being older) and gender (i.e., being female) were found to the associated with higher first-term GPA for transfer students, as was having earned an associate degree prior to transferring. In addition to the focus on students demographic factors, GPA at the community college level has been found to be a key determinant of first-term GPA when students transfer to a four-year institution (Baldwin, 1994; Towsend, McNerny, & Arnold, 1993). However, more research is needed to identify predictors of transfer students success at a four-year university (Johnson, 1987). Course Taking Behavior at the Community College In examinations of community college transfer students performance at four-year institutions, at the forefront have been considerations of students preparedness to handle the challenges associated with university-level course work (e.g., Berger & Malaney, 2003; Keeley & House, 1993; Townsend, 1995; Townsend et al., 1993). Despite concerns over community college transfer students preparedness, limited research has examined the nature of community college students course taking backgrounds to determine predictors of university success. Some studies provide initial insights. For example, Phlegar, Andrew, and McLaughlin (1981) determined that students fundamentally prepared in key subject areas (i.e, math, science, and English) at the community college level performed better upon transferring. Deng (2006) determined that students attending career-focused community college programs outperformed those attending liberal-arts community college programs, when transferring to a four-year university. Rather than considering specific courses of study, Pennington (2006) determined that students enrollment in developmental course work in community college was associated with a decreased GPA upon transfer to a four-year institution. Carlan and Byxbe (2000) found community college major to be significantly associated with first-term GPA; for instance, students majoring in education and psychology had a higher GPA after transfer than did students majoring in business and the sciences. However, it is unclear whether these major-specific differences in GPA drop were associated with different levels of students preparedness or with cross-institutional differences in the academic demands required by these various programs of study. Rather than examining community college majors, Cejda et al. (1998) found students first-term GPA to be related to university major. Parallel to prior findings (i.e., Carlan & Byxbe, 2000) students in the sciences, indeed, experienced a drop in first-term GPA, while students in the fine arts, humanities, and social sciences experienced a GPA increase. This replicated findings that students majoring in the sciences and mathematics (i.e., biology, chemistry, math, physics, accounting, and economics) had a lower GPA than their fellow community college transfer students (James Madison University, 1989). However, the nature of students preparedness for a four-year university and the types of community college academic experiences that may support transfer success have yet to be fully examined.

18 18 Studies examining community college students preparedness have primarily examined students academic backgrounds at the level of the major (e.g., Carlan & Byxbe, 2000). Institutional data sharing as part of the PASS project, allowed the specific courses of community college students to be examined as predictors of performance at the four-year institution. Predicting Transfer Student Re-Enrollment Historically, research on student retention largely focused on the experiences of traditional students, until Tinto (1993) expanded on extant models of retention to consider which factors may impact the retention of non-traditional students. For both traditional and non-traditional students, retention was thought to be a consequence of students academic and social integration (Tinto, 1993). Other research has echoed the central role of social factors in predicting retention for non-traditional students, online, and distance learners (Boston, Diaz, Gibson, Ice, Richardson, & Swan, 2009). At the same time, a number of demographic and community college factors have been considered as predictive of community college transfer students persistence at a fouryear university. Based on a comprehensive review of the persistence literature, Peltier, Laden, and Matranga (1999) determined that gender, race and ethnicity, socioeconomic status, high school GPA, college GPA and interaction variables are all related to persistence. In particular race/ethnicity and prior academic achievement have been robust predictors of persistence (e.g., Astin, 1997; Tross, Harper, Osher, & Knwidinger, 2000; Levitz, Noel, & Richter, 1999), whereas findings with regard to gender have been more mixed, Reason, 2009; St. John et al., 2001). Wetzel, O Toole, and Peterson (1999) used logistic regression, with a dichotomous outcome variable, retained or not. Retention was significantly predicted primarily based on academic factors, including GPA, earning a low GPA which places students at low academic risk, and the ratio of credit hours earned to the credits attempted. Murtaugh, Burns and Schuster (1999) used survival analysis to examine the retention of undergraduate students, enrolling in a university between 1991 and 1996; 25% to 35% of the cohort examined had interrupted enrollment within this period. Specifically, 13.5% stopped out for a single term, 10.8% had stopped out for two terms, and 1.8% had stopped out for three terms, after which they were required to undergo a readmission process. Predictors of stopping out were referred to as hazards. Hazards were examined for one year, two year, and four year retention. Minority status had a higher rate of withdrawal than did white students; also associated with withdrawal was age, high school GPA, first quarter GPA, area of study, and participation in freshman orientation. In particular, Murtaugh et al. (1999) highlight the importance of precollege characteristics in predicting persistence. Looking at a sample of traditional, first time freshman, Cabrera, Nora, and Castaneda (1993) used structural equation modeling to analyze predictors from both Tinto s (1975) and Bean and Metzner s (1985) models to predict student persistence. Cabrera et al. (1993) ranked variables predicting persistence; the most important factor was psychological goal commitment, or intent to persist, followed by GPA, institutional commitment, and encouragement from family and friends. In turn, intent to persist was predicted by institutional commitment, encouragement from family and friends, academic goal commitment, and academic integration these factors having an indirect effect on persistence.

19 19 Whereas the aforementioned studies focused on individual student factors predicting retention, Moore and Fetzner (2009) addressed the institutional characteristics that fostered commitment in non-traditional students. These factors included having a leadership culture that fosters commitment to student success and institutional policies and practices that incorporate student support services and technological support. For online learners, access to services and support that meet their needs was found to be crucial (Moore & Fetzner, 2009). Further, student satisfaction, defined as students happy with their progress and with support received for learning, and with a perception that the knowledge they were learning was valuable, was predictive of retention. Faculty satisfaction, stemming from involvement in curricular design and training in the use of online technologies supporting learning, were found to be key to engagement and contributors to retention (Moore & Fetzner, 2009). Predicting Re-Enrollment for Non-Traditional Students Based on theoretical work (Astin, 1975; Bean & Metzner, 1985; Tinto, 1975), we may expect that community college transfer students persistence may be affected by different factors. First, given that much of the literature examining community college students performance has focused on the degree of student preparedness (Carlan & Byzbe, 2000), learners prior academic experiences may be particularly important to examine, especially as they include not only high school work, as for traditional students, but college-level course work at a two-year institution as well. Further, to the extent that transfer students enter more connected to external factors beyond their experiences at the transfer institution, it may be particularly important to examine learner background characteristics and how these are related to academic factors at the transfer university. Wang (2008), using logistic regression, found that the probability of graduating with a bachelor s degree for students starting at community college was predicted by gender, socio-economic status, high school curricula, educational expectations, community college GPA, college involvement, and math remediation; while persistence, prior to graduation, was predicted by community college GPA and locus of control. Just as in the Wang (2008) study, in the PASS project, researchers looked to students demographics and community college factors, including course taking behaviors as well as overall performance, to predict next-semester re-enrollment. Kreig (2010) examined students at Western Washington University, an institution with a substantial population of community college transfer students comprising each education level, and found that native students were more likely to graduate, even after demographic characteristics and prior academic performance were controlled. Krieg (2010) compares the experience of community college students to that of freshmen at a four-year university. For new community college students, there may be a difficult adjustment to a new learning context, which may result in early attrition if students consider themselves to be incompatible with the new environment. As such, first year retention is a particularly important factor to consider in understanding students persistence and ultimate graduation. While this tension in fit has most commonly been examined by considering the drop in performance (i.e., GPA) that community college students experience upon transfer to a four-year university, alternately termed transfer shock (Cejda, Kaylor, & Rewey, 199; Townsend,

20 20 McNemy, & Arnold, 1993), Krieg (2010) suggests that this may more profoundly manifest in rapid attrition from the four-year institution. More generally, there may be an interaction between transfer student status, first-term GPA, and drop-out rates (Spady, 1970). Specially, those transfer students scoring a low GPA in the first quarter were twice as likely to drop out as were native students (Krieg, 2010). Pascarella and Terenzini (1980) likewise conclude that the majority of attrition occurs in the freshman year, when students are new to the university setting, and further indicate that this marks a misalignment between theory and evidence. For instance, Tinto s model of academic attrition is better suited to modeling student attrition beyond the first year. The difference Krieg (2010) documents, is not specific to low performing students. Even high performing community college transfer students are more likely to drop-out than are their native counterparts. This may be because transfer students have less immediate affiliation and integration into the transfer institution or because these transfer students are required to take prerequisite courses before entering into a major (Krieg, 2010). This points to the importance of looking beyond community college students prior academic performance, to look at specific course taking behaviors at the community college as well as to consider first-term GPA at the transfer institution these factors were examined as part of the PASS analyses. More broadly, these findings are aligned with the interactional relationships identified in Bean and Metzner s model (1985) that suggest that for non-traditional learners, academic success may not be a sufficient factor to promote persistence. Literature Guiding Interventions Intervention with specific populations (e.g., community college transfer students) and in specific contexts (e.g., online universities) are needed, as the majority of interventions have focused on finding solutions that will have a general effect on a broad population of students (Pascarella & Terenzini (1998). Two prominent models of student retention have been proposed, however, both of these models speak primarily to the needs of traditional students. Tinto proposed the Student Integration Model, which identified attrition as resulting from a lack of congruency between students needs and institutional offerings (1987). Specifically, Tinto points to the need for students academic abilities and motivational orientations to match an institution s academic and social characteristics. In determining whether or not students will persist in post-secondary education, Tinto (1987) suggests that two forms of commitments must be in place. The first is students commitment to educational goals and the second is students commitment to remain within a particular institution. From Tinto s model of student retention, conclusions may be drawn regarding the types of factors that interventions to promote retention ought to foster in students; specifically Tinto s model suggest the need for interventions that target students (a) academic abilities, (b) motivational orientations, specifically with regard to the types of educations goals students adopt in pursuing higher education, and (c) institutional connections. Intervention designs should emphasize the correspondence between students abilities or goals and institutional offerings.

21 21 Yet more work is needed to understand how to adapt the Student Integration Model (Tinto, 1987), to reflect the experiences of non-traditional students, transfer students, and students enrolled in online universities, such as UMUC. In particular, factors affecting students retention may deviate from the proposed model based on differences in the type of institution students are a part of as well as students gender and ethnicity (Pascarella & Terezini, 1997). To this end, proposed interventions are geared not only with general UMUC student populations, but also speak to the specific needs of female students (e.g., Girls to Women) and diverse learners (e.g. mentors in the Community College Mentor and College Writing interventions are matched with mentees according to demographic characteristics, including ethnicity. Tinto s (1987) model has further been critiqued for being limited in considering the role that external factors, or considerations independent of students and institutions, may have on retention (Pascarella & Terezini, 1997). These external factors, including financial and familiar considerations, may be particularly important to consider when modeling retention of nontraditional students. Studies have found that often times these students do not persist in postsecondary education because of finances, employment demands, and taxing family responsibilities (Bean & Metzner, 1985). To expand Tinto s model and to consider the needs of non-traditional students those classified as part-time, older, and non-residential (e.g., online learners at UMUC) Bean and Metzner proposed a Student Attrition Model (1985). This model suggests that students persistence and academic outcomes can be understood as a result of four factors, namely: (a) background variables, (b) academic variables, (c) psychological factors, and (d) environmental variables. Background variables refer to students characteristics that may put them at a risk or deficit relative to their peers. These factors include age, high-school performance, gender, and ethnicity. Academic variables include students study habits, the role of advising, and students certainty in their major. Psychological factors reflect students motivation for engaging in postgraduate education these include students goal commitment, the expected utility or value of a degree, and psychological stress. Finally, environmental factors introduced in the model address students responsibilities outside of the university and may represent constraints on students pursuit of educational goals. The role of social interaction is featured in this model, as previous models have identified the importance of social integration in predicting students persistence (e.g., Tinto, 1975; Pascarella & Chapman, 1983), however, the nature of social interactions may differ for traditional versus non-traditional students. In designing interventions, all four of the factors described in Bean and Metzner s model were considered. In particular, mentoring programs targeted students and matched mentees with mentors according to background variables. Further, mentoring programs were intended to help students in mitigating the effects of environmental variables; the intention of these interventions was to provide students with role-models who have successfully persisted, despite limiting external factors. In providing students with an Introductory Check-List and academic tutoring, the interventions were designed to impact academic variables. Finally, psychological outcomes were targeted by encouraging students to take advantage of the advising available through UMUC and providing students with the opportunity to discuss their long term and professional goals with mentors.

22 22 One of the unique challenges for non-traditional students is the identified lack of social integration and social interaction (Bean & Metzner, 1985). A number of interventions were geared toward connecting students with social resources. For instance, the checklist intervention encouraged students to be involved with available student organizations. Further, to the extent that persistence is marked by a match between a student and an institution (Cabrera, Nora, & Castaneda, 1993), a number of the interventions were aimed specifically at helping students recognize others like them as members of the UMUC community. Community College Transfer Students Transitioning Transferring from community college has been identified as a high-stress time for students, presenting academic, psychological, and environmental challenges (e.g., Laanan, 2001). Flaga (2006) identified five dimensions of transitioning. These are, learning resources, connecting, familiarity, negotiating, and integration. The first two dimensions deal with the knowledge and skills that students need in order to be successful, whereas the last three dimensions address how these skills may develop over time. Learning resources refer to the tools students may use to gain information about the university. Three types of learning resources were specified; these were: (a) formal resources provided by the university (e.g., orientation information), (b) informal resources provided by individuals knowledgeable about the university but not officially affiliated (e.g. information from alumni), and (c) initiative-based resources that students gather independently (Flaga, 2006). The second dimension, connecting, refers to the relationships that students are required to form when transferring to a new institution; including (a) academic connections (e.g., with faculty), social connections (e.g., with other students), and physical connects (e.g., with the university environment) (Flaga, 2006). The third dimension, familiarity, emerges when students become more comfortable with their new environment. The fourth dimension, negotiating, occurs when students adjust their behaviors to better fit their new environment. Finally, the fifth dimension, integrating, does not always happen, but involves students shifting their identities to reflect their new institution (Flaga, 2006). Literature to Support Specific Interventions Checklist The Checklist targeted the first two dimensions identified by Flaga (2006) as supporting students transitioning. Specifically, through the checklist, students received support encouraging them to connect with both formal and informal information resources with the intent of forming academic, social, and physical connections. In completing the activities specified in the checklist, students had the opportunity to exercise initiative in connecting with resources and develop familiarity with UMUC as an institution and an academic community. In a qualitative study of community college transfer students, one of the recommendations transfer students proposed as a resource to help their transition was the creation of a transfer

23 23 checklist (Owens, 2007). Indeed, 27% of students expressed a desire for the introduction of a checklist or guide to aid them in the transfer process (Owens, 2007). In describing the features that would make checklists appeal to them, students expressed desires for ease-of-use, online availability, and comprehensiveness (with information ranging from where to park to how to register for classes); as well as checklists that break down complex processes in a step-by-step manner and include necessary contact information (Owens, 2007). This is a particularly important initiative given that surveys of community college students have determined that students have a need for more information (e.g., Harbin, 1997; Andres, 2001) and more assistance (Townsend & Wilson, 2006) as they move to their new institutions. Community College Mentor Peer-mentoring for community college students transferring to four-year schools has been underexamined in the literature. However, mentoring interventions have been broadly used as an avenue to promote students retention (Good, Halpring, & Halprin, 2000; Hoyt, 2000). Flaga (2006) suggests that benefits associated with peer mentoring are not only academic; through mentoring, students gain access to informal learning resources and have the opportunity to socially connect with their peers. Likewise, Good et al. (2000) found mentoring to confer psychological and academic benefits to both mentors and mentees. The mentoring relationship has been identified as supporting three types of outcomes, namely psychosocial, vocational, and role-modeling (Ensher, Heun, & Blanchard, 2003). Psychosocial support refers to mentors providing counseling, friendship, and, encouragement to their mentees (Enscher et al., 2003). Vocational support is considered to be support that enhances the professional lives of mentees (Enscher et al., 2003) and can be extended to include the academic support provided by mentors to new students. Finally, role-modeling refers to mentors demonstrating appropriate behaviors or expectations, either implicitly or explicitly (Enscher et al., 2003). For example, role-models can offer examples of effective study strategies or describe appropriate standards of communication when conferring with professors. Tinto (2001) further suggests that peer mentor relationships can address both specific classwork and general skills associated with successful college completion. Moreover, these benefits can affect mentees as well as mentors (Good et al., 2000; Snowden & Hardy, 2012). Mentoring has been found to be particularly beneficial for minority students (Good et al., 2000; Redmond, 2000). Redmond (2000) suggests that mentoring programs must adopt the following goals to meet the needs of diverse students: (a) promote greater student contact, (b) promote students use of services for support with non-academic problems, (c) intervene quickly when students encounter academic difficulties, and (d) develop culturally-sensitive psychosocial environments. A case study for mentorship in diverse communities is the ALANA (Asian, Latin, Africa, and Native American) mentoring program, targeting minority community college students (Mueller, 1993). The stated goals of the ALANA program were to (a) provide social and academic support for minority students, (b) engage in role-play to help students critically think through

24 24 challenging situations, and (c) assist students in making time-sensitive decisions (e.g., course add/withdrawal). Mentors in the ALANA program seek to maximize social interaction with their mentees as a mechanism for relieving students anxieties (Mueller, 1993). Peer mentoring has been shown to benefit students transitioning from two- to four-year institution and those in distance education programs. For instance, Lenaburg, Auirre, Goodchild, and Kuhn (2012) reported on the impact of a program that oriented community college students to a four-year institution. As part of the program, students were provided with peer mentors. At the conclusion of the program, participants rated their peer mentor experience very highly, commenting that peer mentors were instrumental in explaining the transfer process, providing social support and helping them maintain interest in a four-year institution. Most recent results suggest that peer mentors were instrumental in helping students transition from community colleges to a four-year university. Peer mentoring has also benefitted students new to online learning contexts (Boyle et al., 2012; Brown, 2011). A study of peer mentoring programs in three distance education universities, for example, found evidence of improvement in mentees course passage rates, retention, and sense of belonging (Boyle et al., 2012). Though not evaluated in the empirical literature, the University of California at Berkeley has a mentoring program for transferring community college students: the Starting Point Mentorship Program. Through this program, transferring students are paired with mentors who offer: (a) guidance, (b) motivation, and (c) access to campus and community resources. Specifically, the benefits to mentees are outlined as: advice on study skills, time management and goal-setting, information about the differences in academic and social culture between community college and a four-year institution, encouragement to set and pursue academic goals, and the point-of-view of a current student. Despite the likelihood that peer mentoring can mitigate the shock of student transfer either from community college to a four-year institution or from face-to-face to online environments there have been few experimental studies directly assessing peer mentoring programs impact on key student indicators (Boyle et al., 2010). Further, to date, there have been no such studies of peer mentoring for students experiencing the double shock of transferring from a largely face-toface community college to an online, four-year institution.

25 25 SECTION 3: RESEARCH SCOPE AND DESIGN In Phase 3, research was undertaken to expand on and validate initial findings from Phases 1 and 2. In particular, variables previously identified as potentially predictive of performance and persistence at UMUC, as well as newly introduced factors, were used to predict key outcomes throughout the path model of students academic trajectories. The path model identifies the academic milestones along the path to completion for community college students. (See figure 1.) Figure 1. Path model of students academic trajectory from community college to UMUC. Community College Data UMUC First Term GPA Reenrollment Retention Graduation Research Questions Predictive modeling was used to answer the following research questions related to students performance, persistence (re-enrollment and retention), and ultimate achievement of a credential (graduation) Performance 1. To what extent do demographic characteristics, community college course taking behaviors, and community college performance metrics predict earning a successful first-term GPA (2.0 or above) at UMUC? Persistence 2. To what extent do demographic characteristics, community college course taking behaviors, community college performance metrics, and UMUC first-term GPA predict re-enrolling at UMUC in a semester immediately following the first semester of transfer? 3. To what extent do demographic characteristics, community college course taking behaviors, community college performance metrics, and UMUC first-term GPA predict retention at UMUC, or re-enrollment within a 12-month window following the first semester of transfer? Graduation 4. To what extent do demographic characteristics, community college course taking behaviors, community college performance metrics, and UMUC first-term metrics predict graduation from UMUC? 5. What are the graduation rates of community college transfer students at UMUC?

26 26 Beyond building predictive models of key milestones along students academic trajectories, students experiences while enrolled at UMUC were examined. In particular, two aspects shaping students academic trajectories were examined. First was an examination of students motivational and self-regulatory profiles as they relate to socio-demographic characteristics (e.g., employment status, family structure). Examining motivation and self-regulation as well as probing aspects of students background introduced a deeper examination of students backgrounds that may shape their experiences at both the community college and the transfer institution. Further, data mining analyses were used to examine whether students engagement in the online classroom, as measured by UMUC s LMS, was associated with performance at UMUC. This examined the extent to which students interactions within the online classroom was potentially facilitative of meeting academic milestones (e.g., earning a successful GPA). These in-depth, learner-focused analyses introduced two additional research questions: Learner-Focused 6. What is the association between student motivational and self-regulatory characteristics, socio-demographic factors (e.g., employment status, family structure) and performance at UMUC? 7. What is the nature of students engagement in the online classroom environment and its association with successful course completion? Student Population The population of interest for analyses was defined as first-term undergraduate students, whose first semester of transfer to UMUC, from MC or PGCC, was between Spring 2005 and Spring In this report, a number of outcomes reflecting student success and corresponding to key academic milestones were examined. These are defined as: Successful first-term GPA earning a GPA of 2.0 or above in the first semester at UMUC Re-enrollment enrollment in the immediate next semester after initial enrollment Retention re-enrollment at UMUC within 12 months after initial enrollment Graduation earning a first bachelor s degree from UMUC within a specified time period, specifically within 4, 6, or 8 years of transfer Models predicting each of the target outcome variables were developed, with results presented in Sections 7 and 8. Further, learner-focused analyses were undertaken examining the relation between online course engagement and performance and motivational and self-regulatory profiles, socio-demographic characteristics and performance with results presented in Sections 9 and 10.

27 27 SECTION 4: DATA SOURCES One of the key achievements of Phase 1 of the Kresge research grant was the development of the KDM, an integrated multi-institutional database that aligns the academic work of transfer students across institutions. Data for the KDM came from three student information systems: 1. Banner - Montgomery College s student information system 2. Datatel - Prince George s Community College s student information system 3. PeopleSoft UMUC s student information system. All data were made anonymous to protect students information. Demographic, academic, transfer, and enrollment data were collected on each student from each institution. Demographic data included students gender, age, marital status, and race/ethnicity. Enrollment data included course registration, program of study or major, and student status. Community College academic data included information about students academic history prior to transferring to UMUC, such as course grades, repeated courses, and remedial coursework. Transfer data included the number of courses transferred, transfer GPA, and prior degrees earned. The standardization and alignment of data across institutions was accomplished in Phase 2. Due to the institution-specific design of each student information system, a data dictionary was used to document the name, definition, type, range and default value of each element as it existed in its native system as well as its transformation and standardization in the KDM. As research progressed, categorical and derived fields were developed and added to the data dictionary to enable researchers to try different predictive models. For this research, over 300,000 course records were collected and aligned across institutions. In addition, online classroom behavior data from UMUC s online LMS were added to the database for analysis of student behavior in the online classroom. Classroom behavior is defined by over 30 categories of actions taken in the LMS by a student. Examples of typical actions are: login time, access to various modules within the classroom, and posting of or responding to a conference note. Each action that a student made in the classroom was totaled for each day. Daily actions were aggregated by week, enabling researchers to analyze student activity in a class as it progressed over time. For this research, over 3 million rows of data were available for data mining. Advising data from UMUC s customer relationship management system (CRM), Goldmine, were also added to the database for future analyses. The KDM served as the primary resource for all the analyses and findings for this research grant. A model of the data included in the KDM is presented in Figure 2.

28 28 Figure 2. The Kresge Data Mart After reviewing the initial findings from Phases 1 and 2, the community colleges, in collaboration with UMUC, agreed to provide additional data elements that would enrich the research and analysis. As a result, the data in the database were enhanced, resulting in the second iteration of the Kresge Data Mart (KDM2). These additional data included: students completion of developmental education, whether or not students received financial aid, and students ACCUPlacer scores. A total of 493 source and derived variables were analyzed for inclusion in the dataset. Over 300 variables were tested as part of data mining analyses. Forty key variables were examined in predictive modeling. All data are stored on secured servers and have restricted access for the Institutional Research office, researchers doing analysis on student success, and developers working on the database.

29 29 SECTION 5: SURVIVAL ANALYSIS: REGISTRATION AND WITHDRAWAL IN THE ONLINE CLASSROOM Using social network analysis, Dawson (2010) found that visualizing classroom interaction patterns could provide insights into the nature of interactions for high- versus low-achieving students completing an online course. Dawson (2010) determined that high-performing students primarily interacted with other high-performing students, and likewise, low-performing students were more likely to have interactions with other low-performing students. More importantly, in examining instructor-student interactions, instructors networked with high-performing students (81.7%) at significantly higher rates than they did with low-performing students (34.61%). Social connections in online learning may result in cognitive and learning gains as well. Rovai (2002) found a correlation between levels of engagement in the classroom community and increased levels of content learning and understanding; this was especially true for females. When this type of social and academic engagement is not present, students may withdraw from online learning. Additionally, students academic withdrawal was analyzed using survival analysis. Analyses were run on a dataset for this study containing 19,190 undergraduate UMUC students in OL1 (Online Session 1) in Fall 2011 in 278 distinct courses An exploratory survival analysis was carried out using a Kaplan-Meier estimator. Survival analysis is a statistical technique that can be used to model time-to-event data. In this case, this analysis examines the time it took for a student to withdraw from a particular course (in weeks and days) reflected as a time-to-event. Survival analysis generates a table that indicates a hazard (or withdrawal) rate during the semester. Table 2 presents the withdrawal rate for new and returning students by day. Figure 3 presents the hazard function for withdrawal rates of new and returning students. Table 2. Withdrawal rate for new and returning students by day Week Number of Cumulative Number of Student Proportion of students Withdrawals Withdrawal Withdrawal rate 1 19, , , , , ,

30 30 Figure 3. Hazard function of withdrawal rates for new and returning students. Students withdraw at a higher rate in Week 1 compared to any other week in the course session, with the exception of Week 5, which is the academic withdrawal deadline. Student status, new or returning, may significantly affect student withdrawal rate. New students withdraw at a higher rate than returning students. These findings suggest that interventions targeting new students with interventions in Week 1 may be appropriate. At the conclusion of Phase 1, three goals for the completion of the grant were identified and completed in Phase 2: 1. Validate the predictive models and data mining techniques explored in Phase 1 on an expanded dataset. 2. Build profiles of successful students and their online learning behaviors. 3. Develop interventions to improve the success of students transferring from community colleges to UMUC.

31 31 SECTION 6: PROFILES OF STUDENTS USING DATA MINING Data mining models were used to examine community college transfer students performance at UMUC. Data mining focused on exploratory analyses identifying potential predictors of students success and retention at UMUC. The following questions were considered. 1. Which profiles of students at UMUC can be identified? 2. To what extent does community college course taking differentiate each success profile at UMUC? Data exploration was initially performed by using IBM Modeler, SPSS, SAS JMP 10 Pro, and Excel. Data were transformed and new variables were created as needed. Transformations were performed in Modeler, JMP, and Excel. A variety of black box algorithms were used to develop profiles of students success. The black box algorithms employed were Neural Nets, Boosted Trees, and Random Forests. Profiles of Student Success In addition to independently considering these two outcomes of student success UMUC GPA and retention at UMUC researchers also examined these two predictors jointly. Thus, profiles of student success at UMUC were determined that classified students based on successful GPA and retention. All combinations of the two attributes were examined. Four quadrants were formed with students evidencing a high or low GPA, and being retained or not. These four Success Quadrants were named Stars, Strivers, Slippers, and Splitters. (See Figure 4.) Figure 4. Success Quadrants

32 32 Success Quadrant Percent of population Students in Top Courses Stars 59% 62% Strivers 17% 16% Slippers 15% 13% Splitters 9% 9% Examining the community college course taking behaviors of students belonging to each of these four success profiles yielded a number of key conclusions: Transfer students who took accounting, economics, or higher-level math classes in community college were more likely to earn a first-semester GPA of 2.0 or above at UMUC (i.e., classified as Stars or Splitters). Students who took more classes in history, sociology, psychology, and similar social sciences were more likely to earn a GPA of less than 2.0 at UMUC. The two low-gpa groups, Strivers and Slippers were less likely to take courses in subject areas that Stars took. The Splitters, the smallest group, did not show a distinct pattern of course taking behavior. In likelihood analyses (i.e., comparing the general proportion of course enrollment to actual enrollment for each cluster), Strivers and Slippers showed nearly identical preferences. The average numbers of classes students took and passed in each subject area were compared for Strivers and Slippers. On average, Slippers passed fewer classes in all of the subject areas preferred by Stars than did Strivers. In addition, Slippers are noticeably less likely than Strivers to take courses in several areas: Developmental English Business/management Sociology Psychology The four student success profiles, Stars, Strivers, Slippers, and Splitters, provided a useful framework for understanding students at UMUC and introduced a new outcome measure that combined performance (first-term GPA) and retention (retention within a 12-month window). Factors from the students academic profile at the community college, such as course taking behavior, course load, and change in GPA between the community college and UMUC, were predictive of which student success profile the student would fall in. These results suggest that student preparedness, particularly in specific areas (e.g., accounting, economics) is important in attaining success at UMUC. More exploration of additional outcome variables, such as reenrollment and graduation, are planned for phase 3 of this project.

33 33 Further Findings from Data Mining Using both predictive models and data mining techniques to understand predictors of student success at UMUC, a number of conclusions may be drawn based on analyses undertaken in Phase 2. These findings emerged from looking across studies and across student sub-populations and through the use of varied statistical methods. 1. Student success. Overall, students transferring from MC and PGCC are successful at UMUC. Indeed, 60% of transfer students were classified as Stars, indicating that they were earning a GPA of 2.0 or above in the first term at UMUC and re-enrolling in a subsequent term. Data indicate that earning high grades at the community college was an indicator of successful performance at UMUC. 2. Online Classroom Behavior. Patterns in students behaviors in the online classroom have some value in predicting success. In the analysis of online classroom data, students varied greatly in the extent to which they engaged in course content and course-related activities, with a substantial percentage of students not accessing the course or materials at all. Results from this research have indicated that online classroom activity is tied to course success. Though demographic factors and factors in students community college course-taking backgrounds were predictive of success at UMUC and of students behaviors in the online classroom, more robust data are needed to more fully understand the relationship between academic behaviors and student success. 3. Change in GPA. A new factor, the change GPA between the community college and the first-term GPA at UMUC, was introduced. Many students experienced a decrease in GPA when transferring to UMUC; however, the magnitude of this decrease has predictive value in determine whether or not students are retained at UMUC. More research is needed to better understand the tradeoff between the difficulty of course work and a higher GPA to help determine what strategies community colleges may employ to better prepare students for their academic transition. 4. Transitional Period. Transferring from community college to a four-year institution is a particularly challenging transition for students. For one, students GPAs tend to suffer during the first semester at the four-year institution. The magnitude of the change in GPA seems to have an effect on students retention, differentiating the Strivers and Slippers. For another, indicators of students preparation, such as course efficiency and subject areas, were predictors of success at UMUC. This suggests that students need to prepare for the rigor of UMUC course work. Finally, the number of credits students earned prior to transfer may serve as an indicator of students preparedness to pursue their study at UMUC.

34 34 SECTION 7: PREDICTIVE MODELING OF STUDENT SUCCESS Initial Predictive Modeling Based on exploratory analyses using data mining, predictive modeling, including cluster analyses and logistic regression, were used to model student success using demographic and community college course taking variables. The following questions were considered. 1. What are the demographic profiles of community college students transferring from MC and PGCC to UMUC? 2. Which factors from students demographic profiles and course-taking backgrounds in community college predict success at UMUC overall, and in specific courses? 3. What kinds of online learning behaviors do students transferring to UMUC engage in? In addition to considering variables used in data mining, a number of possible predictors of success not previously considered were included. For example, students course efficiency in community college (the ratio of credits completed to credits attempted) and change in GPA (the difference between students community college and UMUC GPA) were used as predictors. Building upon findings from data mining and particularly exploratory analyses of community college course taking behaviors suggesting that students course taking behaviors at the community college may predict performance at the transfer institution, demographic factors and variables in students community college course-taking backgrounds were examined as predictive of success at UMUC. Predicting Successful GPA Independent Variables. Three types of independent variables were considered. Specifically, these were students demographic characteristics, community college course taking behaviors, and course efficiency. Course efficiency was introduced as a summative measure of community college students course taking that was thought to reflect the real-world cost, both in terms of time and tuition, of students not completing courses as intended. Dependent Variables. Across models run, a dichotomous outcome variable was used. Students first-term GPA at UMUC was the target dependent variable, with a GPA of 2.0 or above being indicative of successful first-term GPA and a GPA below 2.0 being indicative of an unsuccessful first-term GPA. First term referred to students first semester of transfer at UMUC. Logistic regression was used to determine which independent variables might be predictors of success in terms of first-term GPA at UMUC. (See Table 3.) Demographic factors, primarily age, marital status and race, were found to be significantly related to success at UMUC. Specifically, older or married students were found to have higher GPAs at UMUC. When

35 35 compared to white students, students self-identifying as African American, Hispanic, or with an unspecified race/ethnicity tended to have a significantly lower GPA at UMUC. Table 3. Results of Multivariate Logistic Regression Analysis of Success at UMUC (N=7615) Variable B S.E. Sig. Exp(B) Age * Gender Asian Ethnicity African American * Hispanic Ethnicity * Unspecified Race * Married * English Course Taken * Math Course Taken * Speech Course Taken Computer Course Taken Honors Course Taken * Remedial Course Taken Online Course Taken * Course Efficiency * Note: White was used as reference category for race/ethnicity variables thus not used in the logistic regression model. *Statistically significant Table 3 also shows that prior coursework was related to success at UMUC. Math courses and honors courses were related to success at UMUC, while online courses at the community college level were inversely related to success. Finally, course efficiency at the community college was found to be a significant predictor of success at UMUC. Predicting Re-enrollment Independent variables. A number of independent variables were used in these analyses: students community college GPA, race/ethnicity, gender, and age were used as control variables. Then, the predictor of interest, delta GPA, was entered into the model. Dependent Variables. The outcome of interest in these analyses was retention at UMUC, defined as a student s enrollment in a course at UMUC within 1-year of the entering semester. A binary coding (0 or 1) was used depending on whether or not student was retained. Overall, the majority of transfer students (76.35%) were retained at UMUC. After controlling for demographic factors and community college GPA, students change in GPA upon transferring to UMUC was nonetheless a significant predictor of retention. (See Table 4.)

36 36 Table 4. Results of Multivariate Logistic Regression Analysis of Retention at UMUC (N=12637) Variable B S.E. Sig. Exp(B) Age * Gender * Hispanic * African American * Asian * Race/Ethnicity * Unknown Community College * GPA Delta GPA * Note: Excluded from the model were students classified as Non-resident alien, American Indian, Hawaiian/Pacific Islander, or Two or more ethnicities, as these were not significant predictors in the model. *Statistically significant 1. Demographics. In various analyses, students age and marital status were repeatedly found to be predictors of success at UMUC. Older, married students tended to earn higher GPAs and be retained. These findings may be indicative of students greater maturity or dedication to their education goals. At the same time, minority status (i.e., African American or Hispanic) was associated with lower performance at UMUC. More investigation needs to be done to determine how best to reach these underserved populations and improve success. 2. Community College Courses. Course efficiency in community college was determined to be a predictor of success at UMUC. The higher the ratio, the more likely the student will succeed. Similarly, students who took math or honors courses were more likely to succeed. These results point to the importance of considering not only quantitative measures of students course work (e.g., course load) but also qualitative aspects of students work (e.g., honors and math). Updated Predictive Modeling Expanding on initial predictive modeling, models were enhanced to incorporate new data, introduced as part of a second wave of data sharing with the community colleges. Predictive models were further developed and validated in updated predictive modeling. Specifically, models were constructed predicting key milestones in students successful completion of a fouryear institution after transferring from a community college. These were: 1) Earning a successful first-term GPA 2) Re-enrollment in the next semester after transfer 3) Retention within a 12-month window following 4) Graduation with a 4-year, 6-year, and 8-year timeframe.

37 37 Each of the final predictive models is presented in turn. In developing predictive models, over 35 demographic, community college course taking behavior, community college performance, and UMUC first-term variables were examined as potentially offering predictive power. These are listed in Table 5. Table 5. Variables considered in predictive modeling Type of Predictor Listing of Variables Examined Demographic Characteristics Age Gender Race/Ethnicity Marital Status Receiving a PELL Grant at the CC Community College Course Taking Community College Summative Metrics UMUC First-term Metrics Math Enrollment English Enrollment Speech Enrollment Honors Enrollment Dev Education Enrollment Enrollment in an Online Course Successful Course Completion Successful Math Completion Successful English Completion Community College GPA CC Credits Earned CC Credits Attempted Percentage of Courses Withdrawn From Receiving an Associate Degree UMUC First-term GPA UMUC First-term Enrollment Full-Time/Part-Time UMUC First-term Credits Attempted UMUC First-term Credits Earned UMUC First-term Credits Transferred Successful Speech Completion Successful Computer-related Crs Completion Dev Writing Completion Dev Reading Completion Dev Math Completion Exempt from Dev English Exempt from Dev Math Repeating a Course

38 38 Models were constructed to maximize statistical fit while being as parsimonious as possible (i.e., including as few predictors as possible). Model fit was determined by examining the percentage of variance in the outcome variable explained by predictors in the model as well as by considering accuracy of classification (e.g., categorizing students as graduating or not). In predictive models, hierarchical logistic regression was used. Logistic regression predicts the probability of a dichotomous outcome being achieved. As such, all target outcome variables were dichotomized for example, students first-term GPA at UMUC was recoded as being either successful ( 2.0) or not. In hierarchical regression, variables are entered as blocks or in steps, so that variables entered in at a previous step are controlled for when additional predictors are added to the model. Across models, order of entry was: demographic characteristics, community college course taking, community college performance summative measures, and if considered, first-term at UMUC indices. Table 6 presents descriptives of each of the dichotomized outcome variables examined. Table 6. Descriptives associated with each target outcome variable Dependent Variable Population Performance Successful First-term GPA 76.3% earn a GPA 2.0 in their first semester (n=6151) Re-enrollment 66.7% of students re-enroll in a subsequent semester (n=5376) Retention 79.1% of students are retained within a 12-month window (n=5376) Graduation to Date (Spring 2014) 52.7% of students have graduated to date (n=5454) Population Predictive modeling was run on 8,058 transfer students from MC and PGCC. These were students whose first semester of transfer to UMUC occurred between Spring 2005 and Spring Students enrolled in continuing education courses or earning a second bachelor s degree were excluded from these analyses. Demographic characteristics are presented in Table 7. Table 7. Sample Demographic Characteristics (n=8058) Age 28.6 years old (SD=8.4) Gender Female: 57.6% (n=4638) Male: 41.2% (n=3323) Race/Ethnicity White: 24% (n=1956) African American: 43.5% (n=3509) Asian: 10.4% (n=839) Hispanic/Latino: 10.2% (n=821) American Indian: 0.9% (n=75) Unspecified: 14.0% Predictive models for earning a successful first-term GPA, re-enrollment, and retention were run.

39 Race/ Ethnicity: Compared to White Students 39 Predicting Earning a Successful First-term GPA Dependent variable. Successful first-term GPA was used as the dependent variable. Independent variables. Three types of variables were used to predict successful first-term GPA. These were: (a) students demographic characteristics, (b) community college course taking behaviors, and (c) summative measures of community college performance. Among the independent variables of students community college course taking behavior examined, rate of successful course completion at the community college, both overall and in specific subject areas was computed. Successful course completion was defined as the ratio of courses students completed with a grade of C or above to the total number of courses in which students were enrolled. In specific subject areas, successful course completion referred to the ratio of courses in that subject area in which students earned a grade of C or above to the total number of courses in that subject area. While course taking behaviors focused on students specific academic experiences in community college, summative measures (e.g., GPA) looked at students community college careers, overall. The model was overall significant, X 2 (21) = , p<0.001, correctly classifying 76.8% of students as earning a successful first-term GPA or not. Cox and Snell s R 2 suggested that the model explained 9.1% of variance in earning a first-term GPA, while Nagelkerke s R 2 suggested that 13.7% of variance had been explained. (See Table 8.) Table 8. Predicting first-term GPA using demographic characteristics, community college course taking behaviors, and summative measures of CC Background β SE(β) Significance β* Demographic Characteristics Gender* Age** Black*** Hispanic/Latino Asian American Indian Race Not Specified* Marital Status** PELL Grant Recipient*** Community College Course Taking Successful Course Completion Overall*** Successful Math Completion** Successful English Completion**

40 40 Developmental Math Completion** Developmental Writing Completion Developmental Reading Completion Developmental Math Exempt Developmental English Exempt Repeated Courses Summative Measure of CC Background GPA*** Credits Earned Associates Received*** Note: *sig. at 0.05 level, ** sig. at 0.01 level, *** sig, at level In terms of demographic characteristics, gender, age, and marital status were all significant predictors in the model. Specifically, students who were female, older, and married were significantly more likely to earn a successful first-term GPA at UMUC. At the same time, students reporting their race/ethnicity as African American or not designating a race/ethnicity were less likely to earn a successful first-term GPA. Further, receiving a PELL grant at the community college, as an indicator of financial need, decreased the likelihood of students earning a successful first-term GPA. In examining indicators associated with students community college course taking behaviors, students overall rate of successful course completion and rate of successful math completion, and successful English completion were all significant predictors in the model. Further, students completion of developmental math was a significant predictor in the model. Looking to summative measures of community college performance, cumulative GPA, credits earned, and earning an Associate degree were all significant predictors. As can be seen by examining the standardized beta, holding all else constant in the model, students overall rates of successful course completion carry the most impact in increasing students probability of earning a successful GPA. Standardized betas may be interpreted as, for a 1 standard unit increase in successful course completion; students were 5.08 standard deviations more likely to earn a successful first-term GPA. Predicting Re-Enrollment Dependent variable. Re-enrollment was defined as enrolling in a semester immediately following the first term of transfer. Re-enrollment was binary coded as re-enrolled (1) or not (0) Independent variables. Four types of variables were used to predict students re-enrollment and retention. These were: (a) students demographic characteristics, (b) community college course taking behaviors, (c) summative measures of community college performance, and (d) first-semester performance at UMUC. While course taking behaviors focused on students

41 Race/ Ethnicity: Compared to White Students 41 specific academic experiences in community college, summative measures (e.g., GPA) looked at students community college careers, overall. The overall model for re-enrollment was significant, X 2 (19) = , p<.001. The model was able to correctly classify 71.6% of students as re-enrolling or not. Pseudo R 2 measures of effect size ranged from an estimated 12.5% of variance in re-enrollment explained (Cox & Snell s R 2 ) to 17.4% of variance (Nagelkerke s R 2 ) explained. (See Table 9.) Table 9. Predicting re-enrollment using demographic characteristics, community college course taking behaviors, summative measures of CC backgrounds, and UMUC first-term indicators β SE(β) Significance β* Demographic Characteristics Gender*** Age Black* Hispanic/Latino Asian American Indian Race Not Specified* Marital Status** PELL Grant Recipient Community College Course Taking Repeated a Course** Enrolled in a Developmental Course*** Exempt from Developmental Math** Summative Measures of Community College Backgrounds Community College GPA** Cumulative Credits Earned at CC Earned an Associate Degree First Term at UMUC First-term GPA*** First-term Credits Earned*** Enrolled Full Time Cumulative Credits Transferred*** Note: *sig. at 0.05 level, ** sig. at 0.01 level, *** sig, at level Examining demographic characteristics determined that gender and marital status were both significant predictors in the model. Specifically, being female and married increased students probability of re-enrolling in a subsequent term at UMUC. Further, unlike with first-term GPA, race/ethnicity designated as African American or unspecified were significantly positive predictors of re-enrollment.

42 42 In examining students community college course taking behaviors, different predictors than those found to be significant in predicting performance were identified. Specifically, students likelihood of re-enrollment increased if they either enrolled in a developmental course or were exempt from developmental math at their community college. Surprisingly, repeating a course at the community college was found to be a significant, positive predictor of re-enrollment; in other words, re-taking a course in community college increased the likelihood that students would reenroll at UMUC. While this finding may appear to be counter-intuitive, it may reflect the fact that students willing to retake courses may be more committed to achieving an academic credential, despite challenges they may experience. Summative measures of students community college background found only community college GPA to be a significant predictor in the model. Further, despite being a positive predictor of first-term GPA, community college GPA was a negative predictor of persistence or reenrollment. More work is needed to understand why this may be the case. In part, those students earning a high GPA at community college, despite likewise earning a successful GPA at UMUC, may be more averse to transfer shock due to the new four-year context and associated academic demands. Looking at first-term UMUC indicators, as may be expected, first-term GPA and total number of credits earned were significant predictors of re-enrollment. Further, the cumulative number of credits transferred was a significant positive predictor in the model. Number of credits transferred may reflect the pragmatic value of community college course work in helping students meet four-year institutional academic requirements. Examining standardized beta coefficients in the model reveals first-term GPA to be the strongest predictor of re-enrollment at UMUC. Indeed, for every standard unit increase in UMUC GPA, the probability of reenrollment increases by 1.30 standard deviations. Predicting Retention Dependent variable. Retention was defined as re-enrolling within a 12-month window following the first term of transfer. Retention was binary coded as students being retained (1) or not (0) Independent variables. Four types of variables were used to predict students re-enrollment and retention. These were: (a) students demographic characteristics, (b) community college course taking behaviors, (c) summative measures of community college performance, and (d) first-semester performance at UMUC. While course taking behaviors focused on students specific academic experiences in community college, summative measures (e.g., GPA) looked at students community college careers, overall. The model was overall significant, X 2 (17) = % of cases were correctly classified as retained or not. Effect size measures suggest that between 14.8%, according to Cox and Snell s R 2, and 23.1%, according to Nagelkerke s R 2, of variance in retention was explained by the model. (See Table 10.)

43 Race/ Ethnicity: Compared to White Students 43 Table 10. Predicting retention using demographic characteristics, community college course taking behaviors, summative measures of CC backgrounds, and UMUC first-term indicators β SE(β) Significance β* Demographic Characteristics Gender*** Age at Transfer Black** Hispanic/Latino Asian American Indian Race Not Specified* Marital Status** PELL Grant Recipient Community College Course Taking Repeated a Course** Completed Developmental Math* Exempt from Developmental Math Summative Measures of Community College Backgrounds Community College GPA** First Term at UMUC First-term GPA*** First-term Credits Attempted*** Enrolled Full Time*** Credits Transferred Firstterm*** Note: *sig. at 0.05 level, ** sig. at 0.01 level, *** sig, at level As with re-enrollment, a number of demographic characteristics proved to be significant. Again these were gender and marital status, with married females being more likely to persist. African Americans and those having an Unspecified race/ethnicity were both found to be positive predictors of retention. Examining students community college course taking determined that repeating a course, and being exempt from or completing developmental math were all three positive predictors in the model. The presence of math-related variables seems to suggest the important role of the math subject area in determining students academic preparedness and predicting students persistence. In examining summative measure of community college academic backgrounds, cumulative GPA at the community college was again found to be a negative predictor in the model. As a contrast, first-term GPA at UMUC was a significant and positive predictor of retention. The total number of credits attempted in the first term, the dichotomized variables part-time or full

44 44 time status, and the number of credits transferred from the community college were all found to be significant predictors of retention. A number of additional factors as predictive of retention were considered. These included, receiving an Associate degree at the community college, credits earned and attempted at the community college, average community college course load. Further, community college course taking behaviors were not significantly predictive of retention these included enrollments in Math, English, Computer, or Speech courses as well as enrollment in Honors, Developmental, and Online courses. Successful course completion indices were also not found to be significantly associated with retention at the transfer institution. Predicting Graduation Sample. While previous models predicting earning a successful first-term GPA, reenrollment, and retention were run on the full cohort of MC and PGCC transfer students enrolled in their first semester at UMUC between Spring 2005 and Spring 2012 (n=8050), the graduation model was run on a more limited sub-sample. As we were interested in predicting students eight-year graduation rate, only data from cohorts enrolled from Spring 2005 Spring 2006, reflecting six semesters of data, were used. The remaining student cohorts were not examined as they do not yet have eight years since their first term of entry. The graduation model was run on a reduced sample of 2040 students. Dependent variable. Graduation was defined as earning a first credential from UMUC (i.e., Certificate, Associate, Bachelor s) within an 8-year period, based on cohort of entry. The eight-year graduation rate was chosen because the population in this study reflected nontraditional students, who may be part-time or stop-out for various personal, family, and financial reasons; eight years provides a graduation window which gives students sufficient time to earn a credential. Graduation was binary coded as students either graduating within 8 years (1) or not (0). Independent variables. Four types of variables were used to predict students re-enrollment and retention. These were: (a) students demographic characteristics, (b) community college course taking behaviors, (c) summative measures of community college performance, and (d) first-semester performance at UMUC. While course taking behaviors focused on students specific academic experiences in community college, summative measures (e.g., GPA) looked at students community college careers, overall. The model was overall significant, X 2 (17) = % of cases were correctly classified as retained or not. Effect size measures suggest that between 20.0%, according to Cox and Snell s R 2, and 26.7%, according to Nagelkerke s R 2, of variance in graduation was explained by the model. Table 11 includes a model summary.

45 45 Table 11. Predicting graduation using demographic characteristics, community college course-taking behaviors, measures of CC experience, and UMUC first-term indicators β SE(β) Significance β* Demographic Characteristics Gender First_Term_Age*** Minority Status Receiving PELL at CC Community College Course Taking Math Enrollment at CC* Percent Ws at CC Summative Community College Measures Receiving AA at CC CC CUM GPA* CC Credits Earned UMUC First-term Indicators UMUC First-term GPA*** UMUC First-term Credits Earned*** Note: *sig. at 0.05 level, ** sig. at 0.01 level, *** sig, at level Looking at individual predictors in the model, in terms of demographic traits, only first-term age when transferring to UMUC was found to be significant. Further, first-term age was a negative predictor, such that being younger increased students likelihood of graduating. At the community college, enrolling in a course in the Math subject area was a significant predictor of graduation. Although other math-related indices, including completing developmental math and the rate of successful math course completion, were examined as predictors, the dichotomized variable reflecting whether or not students had enrolled in a math course proved to be a sufficient indicator predicting graduation. In terms of summative community college course taking indicators, community college cumulative GPA was a significant positive predictor. First semester at UMUC indices were all significant and positive predictors. Specifically, students GPA in the first semester and the number of credits earned in their first term were significant positive predictors. In examining standardized beta coefficients, first-term GPA was the strongest predictor of eight-year graduation, followed by students having taken a math course at the community college. Again, taking a math course could be interpreted as a variable indicative of students academic preparation or of students willingness to complete requirements necessary for graduation. Summary of Results from Predictive Modeling Looking across predictive models, Table 12 presents information regarding which indicators were significant predictors across models.

46 46 Table 12. Significant Predictors for First-term GPA, Re-enrollment, Retention, and Graduation Predictor First-Term GPA Re-Enrollment Retention Graduation Demographic Factors Gender First-term Age + - Race/Ethnicity Black (-) Unspecified (-) Black (+) Unspecified (+) Black (+) Unspecified (+) Marital Status PELL Grant - Community College Course Taking Overall Successful Course Completion Successful Math Completion Successful English Completion Repeated a Course + + Enrolled in a Developmental Course + Exempt from Developmental Math + Completed Developmental Math + Enrolled in Math at CC + Community College Summative Measures Community College GPA Credits Earned Associates Degree Received + UMUC First Term Factors First-term GPA N/A First-term Credits Earned N/A + + Cum Credits Transferred N/A + + Enrolled Full Time N/A + First-term Credits Attempted N/A + Looking across the models helped identify a number of predictors which seem to be associated with both performance (e.g., first-term GPA) and persistence (i.e., re-enrollment, retention, and graduation). Females and those who were married were more likely to both earn a successful first-term GPA, as well as to persist to graduation. Interestingly, African American status, though negatively associated with performance in the first semester, was positively associated with persistence. This may suggest that although minority status has typically been considered to be an at-risk factor for students success (Greene, Marti, McClenney, 2008), some students may benefit from the flexibility offered by an online institution. Repeating a course was significantly associated with persistence measures re-enrollment and retention. This may indicate that while students struggle, being persistent in completing necessary course work is associated with persistence. Further, a variety of factors related to students math course taking behavior (e.g., being exempt from developmental math, completing

47 47 math at the community college, enrolling in math) were found to be associated with persistence as well as graduation. Examining students overall community college performance, community college GPA was significantly associated with both performance and persistence. At UMUC, first-term GPA was associated with re-enrollment, retention, and graduation. Likewise, the number of credits earned in the first term was associated with re-enrollment and graduation, as they may reflect students commitment to educational goals and credential completion; likewise, being enrolled full-time was associated with retention. Altogether, students volume of course taking in the first semester (i.e., credits attempted, credits earned, full-time enrollment) were associated with persistence. While this section presented results from predictive modeling for first-term GPA, re-enrollment, retention, and graduation, the subsequent section delves into specific aspects of students experience at UMUC. Specifically, Section 8 examines (a) the relation between students online classroom engagement and course performance as well as (b) students motivational and selfregulatory profiles.

48 48 FY Cohorts SECTION 8: GRADUATION RATES In addition to predicting graduation, graduation rates for students in our target cohort (i.e., whose first semester of transfer to UMUC was between Spring 2005 to Spring 2012) were examined. Students were divided into cohorts of entry depending on their first semester of enrollment at UMUC. Cohorts were determined by fiscal year, including the Summer, Fall, and Spring terms of a given year (e.g., Summer 2005, Fall 2005, and Spring 2006). Graduation rates were examined for students earning a first-time bachelor s degree from UMUC. Term of graduation was likewise determined by fiscal year, and graduation rates for 1 to 8 year periods were calculated. Not all starting cohorts at UMUC had been enrolled for a full eight years; graduation rates were computed for as many years as students were at UMUC. Rates of transfer were computed for students, overall, as well as separately for students coming from each of the community colleges. Specifically, graduation rates were computed for students transferring from Prince George s Community College (n=3220) and from Montgomery College (n=4724). Table 13. Graduation rates for MC and PGCC transfer students, FY 06 FY 12 Graduation Rates by Subsequent Fiscal Year Year # Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 FY % % % % % % % % FY % % % % % % % FY % % % % % % FY % % % % % FY % % % % FY % % % FY % % Notes: N=7944; Bachelor Graduates for the entire population to Spring 2014 = 3051 (38%) Examining graduation rates determined that, overall, community college transfer students were successful in earning a credential at UMUC. The eight-year graduation rate was 49%, while the 6-year graduation rates ranged from 44% to 40% of students in each cohort graduating. These are impressive numbers compared to national rates as well as to UMUC overall rate.

49 49 SECTION 9: EXAMINING LEARNER BEHAVIOR IN THE ONLINE CLASSROOM In addition to using predictive modeling to predict key academic outcomes, data mining was used to examine the relation between students online classroom engagement and performance. The online classroom activities come from the LMS. The LMS data came from a proprietary classroom management system call WebTycho. WebTycho was replaced with Desire2Learn (D2L) in However, for the population of students in this study, WebTycho provided the data on student interactions in the classroom between Spring 2011 and Fall Data mining techniques included Neural Nets, Boosted Trees, and Bootstrap Forest. A model s misclassification rate (the proportion of wrong predictions) was used to evaluate the effectiveness of the models. For neural nets, R-squared levels were recorded for both the training subset (on which the model was developed) and the validation subset (on which the model was tested). The analytical focus was on undergraduate students who had transferred to UMUC from Montgomery College or Prince George s Community College. The LMS dataset contained approximately 2.3 million rows, each one representing a unique student/class/term/day combination. The data were examined at two levels: (a) course work and (b) student-level. While at the course work level each student s enrollment was treated as a unique record, such that a student could have been listed in the course work file multiple times, the student level file ensured that there was one record per student. The dataset included only 8-week undergraduate courses only and provided week-by-week totals of each action taken by students in the online classroom. Students online classroom activities were then matched to records in the KDM. There were a total of 30 different online classroom behaviors, 22 of these behaviors were not well represented in the student data. The remaining eight online classroom behaviors were evaluated to determine which had the most variability and seemed to be key indicators of student engagement. Four online classroom behaviors were selected for examination: Open classroom Create a response note Launch a conference Read a conference These actions also served to differentiate students course performance. Although 8-week courses were examined, the scope of analysis was restricted to the first 3 weeks of a class, as there was limited variation throughout the remainder of the course duration and prediction of engagement-related factors early in the course facilitated the possibility of intervention.

50 50 The focus was further narrowed to include only student/class/term enrollments that earned a grade of A, B, C, D or F. (See Figure 5.) Grades of AU, P, S (audit), FN (non-attendance), Incomplete, and Withdraw were excluded. Figure 5. Distribution of course grades 40% 35% 30% 25% 35.1% 32.0% 20% 17.6% 15% 10% 5% 5.8% 9.5% 0% A B C D F All models were run on the final dataset using four classroom activities during weeks 1 3 of an 8-week course where the final grade earned was A, B, C, D or F, and only for students who were found in the KDM. The number of rows in this dataset is 28,021 and the number of unique students is 4,277. Table 14 presents the averages for these unique student/course/session combinations. Table 14. Mean and median values for actions in the online classroom Action Median Mean Open Class Create Response Note Launch Conferencing Read Conference Note The distribution of the aggregate online classroom behaviors (compared to the median of all students) across 8 weeks by grade received are presented in Figure 6. Figure 6. Overall level of online classroom engagement for students by grade.

51 Engagement Index 51 Level of Student Engagement by Grade Week A grades B grades C grades D grades F grades Four sets of findings will be discussed. First, the relation between students online classroom behaviors, at the course level, and course performance was examined. Second, the association between online classroom behaviors and course performance at the student level, across all of their enrollments was analyzed. Third, the relation between an overall online classroom engagement measure and performance was considered. Fourth, modeling the potential relation between online classroom performance and re-enrollment was considered; however, did not prove to be a fruitful avenue of investigation. Online Classroom Behaviors and Class Performance Because these LMS actions have uneven distributions, the median was chosen as a representation of the average value rather than the mean. In each row, med flags were generated indicating whether the values of the key LMS actions were at or above the median or below the median. Figure 7 displays the grades distributions at the course level for students based on whether their online classroom engagement, across the four actions, was above or below the median.

52 52 Figure 7. Distribution of grades above and below the median level of engagement 100% 90% 80% 70% 60% 50% 40% 30% 25% 31% 22% 45% 33% A B C D F 20% 10% 0% 8% 15% Below median 14% 4% 5% Above median As can be seen in the figures above, there are almost twice as many A grades in the above median category as in the below median category, and almost three times as many F grades in the below median category as in the above median category. Higher counts of key LMS actions are associated with the higher grades. Similarly, the lowest grades are typically found along with the lowest LMS counts. In order to meaningfully measure how far above or below the median a LMS action value is, however, the values needed to be indexed to a consistent scale to compensate for the differing ranges. A median difference index (MAD) was created to capture the deviation between a student s behaviors in the online classroom and the median number of such behaviors manifest in the overall sample. Both absolute values and MAD (i.e., median difference) indices were used in predictive models. Model summary information is presented below; however, model fit was modest with a high misclassification rate. (See Table 16.) Table 16. Model fit information for predicting successful course completion. Response Variable Predictors (Weeks 1-3, Grades ABCDF) Model Type Model Performance Validation Set Results Course Grade Open Class (OC) Create Response Note (CRN) Launch Conferencing (LC) Read Conference Note (RCN) Bootstrap Forest R-squared: Misclassification rate: 59.6% Course Grade OC median CRN median LC median RCN median Neural net R-squared: Misclassification rate: 61.9%

53 53 Response Variable Predictors (Weeks 1-3, Grades ABCDF) Model Type Model Performance Validation Set Results Course Grade OC difference-from-median index CRN difference-from-median index Neural net R-squared: Misclassification rate: 59.5% LC difference-from-median index RCN difference-from-median index Course Grade Sum of median difference indexes Neural net R-squared: Misclassification rate: 59.7% Course Grade Course Grade Course Grade Course Grade Course Grade OC difference-from-median index CRN difference-from-median index LC difference-from-median index RCN difference-from-median index Open Class (OC) Create Response Note (CRN) Launch Conferencing (LC) Read Conference Note (RCN) Open Class (OC) Create Response Note (CRN) Launch Conferencing (LC) Read Conference Note (RCN) OC difference-from-median index CRN difference-from-median index LC difference-from-median index RCN difference-from-median index Course level 300 OC difference-from-median index CRN difference-from-median index LC difference-from-median index RCN difference-from-median index Subject area Bootstrap forest R-squared: Misclassification rate: 56.5% Neural net R-squared: Misclassification rate: 60% Bootstrap Forest Bootstrap Forest Bootstrap Forest R-squared: Misclassification rate: 60% R-squared: Misclassification rate: 59.9% R-squared: Misclassification rate: 60.1% Student Level Online Classroom Behaviors and Course Performance In the next set of analyses, the analysis shifted from class-level to student-level. The student/course/term dataset (28,021 rows) was rolled up to yield 4,277 rows, each representing a unique student. In the rolled-up dataset, the four key online classroom actions were represented by the mean of their values across that particular student s entries in the previous dataset. Similarly, each student s grades from each class were averaged. A 0/1 flag was created to indicate if the average grades were 2.0. This value was used as the response variable for most of the predictive models. Because this variable was categorical and not continuous, each model s accuracy calculations could also show misclassification rates. The engagement calculations for this dataset followed the same procedure as the course level calculations, except with student-level figures.

54 54 Different combinations of the variables were tested as predictors of student success. The primary modeling methods used were Bootstrap Forest, Boosted Tree, and Neural Net. (See Table 17.) Table 17. Data mining results for online classroom activities and course performance Predictors (3-wk LMS rolled up + Model Response variable Results - validation set addl ) type Grades_Mean >= 2.0 OPENCLASS_Mean CREATERESPONSENOTE_Mean LAUNCHCONFERENCING_Mean READCONFERENCENOTE_Mean Bootstrap forest R-squared: Misclassification rate: Grades_Mean >= 2.0 OPENCLASS_Mean CREATERESPONSENOTE_Mean LAUNCHCONFERENCING_Mean READCONFERENCENOTE_Mean Boosted tree R-squared: Misclassification rate: Grades_Mean >= 2.0 Grades_Mean >= 2.0 Grades_Mean >= 2.0 Grades_Mean >= 2.0 Grades_Mean >= 2.0 Grades_Mean >= 2.0 Grades_Mean >= 2.0 OPENCLASS_Mean CREATERESPONSENOTE_Mean LAUNCHCONFERENCING_Mean READCONFERENCENOTE_Mean Difference index - OC Difference index - CRN Difference index - LC Difference index - RCN Difference index - OC Difference index - CRN Difference index - LC Difference index - RCN Difference index - OC Difference index - CRN Difference index - LC Difference index - RCN Difference index - OC Difference index - CRN Difference index - LC Difference index - RCN CC_GRADE_POINT_AVERAGE Difference index - OC Difference index - CRN Difference index - LC Difference index - RCN CC_GRADE_POINT_AVERAGE Difference index - OC Difference index - CRN Difference index - LC Difference index - RCN CC_GRADE_POINT_AVERAGE Neural net Bootstrap forest Boosted tree Neural net Bootstrap forest Boosted tree Neural net R-squared: Misclassification rate: R-squared: Misclassification rate: R-squared: Misclassification rate: R-squared: Misclassification rate: R-squared: Misclassification rate: R-squared: Misclassification rate: R-squared: Misclassification rate: Grades_Mean >= 2.0 Sum of difference indexes Bootstrap forest R-squared: Misclassification rate: 0.212

55 55 Response variable Predictors (3-wk LMS rolled up + addl ) Model type Results - validation set Grades_Mean >= 2.0 Sum of difference indexes Boosted tree R-squared: Misclassification rate: Grades_Mean >= 2.0 Sum of difference indexes Neural net R-squared: Misclassification rate: Grades_Mean >= 2.0 Sum of difference indexes CC_GRADE_POINT_AVERAGE Bootstrap forest R-squared: Misclassification rate: Grades_Mean >= 2.0 Sum of difference indexes CC_GRADE_POINT_AVERAGE Boosted tree R-squared: Misclassification rate: Grades_Mean >= 2.0 Sum of difference indexes CC_GRADE_POINT_AVERAGE Neural net R-squared: Misclassification rate: Grades_Mean >= 2.0 OPENCLASS_Mean CREATERESPONSENOTE_Mean Bootstrap forest R-squared: Misclassification rate: Grades_Mean >= 2.0 Difference index - OC Difference index - CRN Bootstrap forest R-squared: Misclassification rate: Grades_Mean >= 2.0 Difference index - OC Difference index - CRN CC_GRADE_POINT_AVERAGE Bootstrap forest R-squared: Misclassification rate: Grades_Mean >= 2.0 Grades_Mean >= 2.0 Grades_Mean >= 2.0 OPENCLASS_Mean CREATERESPONSENOTE_Mean LAUNCHCONFERENCING_Mean READCONFERENCENOTE_Mean CC_GRADE_POINT_AVERAGE OPENCLASS_Mean CREATERESPONSENOTE_Mean LAUNCHCONFERENCING_Mean READCONFERENCENOTE_Mean CC_GRADE_POINT_AVERAGE OPENCLASS_Mean CREATERESPONSENOTE_Mean LAUNCHCONFERENCING_Mean READCONFERENCENOTE_Mean CC_GRADE_POINT_AVERAGE Bootstrap forest Boosted tree Neural net R-squared: Misclassification rate: R-squared: Misclassification rate: R-squared: Misclassification rate: These findings suggest that indexed measures of student engagement based on key online classroom actions, along with a student s community college GPA, can help predict whether or not that student will achieve course success at UMUC.

56 56 Engagement Profiles and Course Performance For this analysis, behaviors were collapsed to identify engagement profiles. High engagement was defined as having the sum of median differences be greater than or equal to zero. Low engagement meant the sum of differences was below zero. The engagement factor was combined with grades and the students were divided into four engagement quadrants as follows: E+G+ E+G E G+ E G High engagement; grade of A, B, or C High engagement; grade of D or F Low engagement; grade of A, B, or C Low engagement; grade of D or F Grades were proportionate to engagement. Nearly three-fourths of A grades were associated with high engagement while nearly three-fourths of F grades were associated with low engagement. However, engagement seemed a less useful discriminator for B and C grades. These quadrants were very suggestive at a qualitative level but were not good predictors in models, most likely because of the large number of low-engagement students who get an A, B, or C grade, or who are retained. Figure 8 displays students membership by grade in each engagement profile. Grade High Engagement Low Engagement A 72% 28% B 57% 43% C 43% 57% D 35% 65% F 27% 73%

57 57 Figure 8. Student engagement by course performance. Examining overall engagement, students earning high grades also demonstrated higher engagement and vice versa. Modeling Retention Models examining LMS behaviors as associated with retention did not prove to be fruitful predictors. This was likely because the majority of students were retained within a 12-month window, limiting variance. (See Figure 9.) Figure 9. Retention of community college transfer students.