Developing Data-Driven Predictive Models of Student Success Kresge Data Mining Project Phase One Report

Transcription

1 1 Developing Data-Driven Predictive Models of Student Success Kresge Data Mining Project Phase One Report University of Maryland, University College November 20, 2012

2 2 Executive Summary This report documents progress on the Kresge Data Mining Grant: Developing Data- Driven Predictive Models of Student Success. The grant was awarded to University of Maryland, University College (UMUC) in collaboration with two community college partners, Prince George's Community College (PGCC) and Montgomery College (MC). The purpose of the grant is to build an integrated database in order to examine student progress across multiple institutions using data mining techniques and statistical models to identify factors that predict online student success. To date, this project has three major accomplishments: 1. UMUC and the two partner community colleges, PGCC and MC, have a data sharing partnership that supports academic research and student success. 2. UMUC designed, developed, and implemented a multi-institutional database of over 250,000 students (called the Kresge Data Mart) that contains the academic history on transfer students from these two colleges integrated with data on UMUC student online behavior and academic performance. 3. Researchers used data mining and predictive modeling to investigate relationships between variables to predict student success. Initial research yielded the following results: 1. Researchers reviewed the extant literature on online education and educational data mining and identified key variables that have been found to predict successful course completion and student re-enrollment and retention. 2. Examination of the student data in the Kresge Data Mart using four classification data mining algorithms and survival analysis indicated that factors in the students classroom behaviors and previous academic background, such as number of schools attended, transfer credits, and transfer grade point average were strong predictors of student re-enrollment in the following semester.

3 3 3. Regression models utilizing predictors revealed that total transfer credit, transfer GPA, activity in course conferencing, and activity prior to the start of a course, contribute strongly to course success. 4. Predictive models verified that course success is a strong predictor of reenrollment. These data mining and statistical modeling identify variables associated with online student success for students transferring from PGCC and MC. These findings guide next research steps, which include validating data mining and predictive models on an expanded dataset, building profiles of successful online students, and developing meaningful interventions to improve success of students transferring from our partner institutions.

4 4 Introduction The purpose of this report is to document progress on the Kresge Data Mining Grant: Developing Data-Driven Predictive Models of Student Success. The research is being conducted by the Evaluation and Assessment research team in the Office of Institutional Research at University of Maryland University College (UMUC). This report is presented in six sections: 1) Section I contains the grant overview, objectives, milestones and a financial update. 2) Section II reviews the research design, general methodologies, and research questions. 3) Section III contains the literature review and research foundations. 4) Section IV reviews the data developed for this project. 5) Section V describes initial results from the analyses of research questions. 6) Section VI describes next steps to stay on track with both project and research goals. Section I: Grant overview The grant was awarded to UMUC, in collaboration with two community college partners, to build a multi-institutional database to examine student progress across multiple institutions. The purpose of this grant is to use data mining techniques to develop data driven predictive models and to identify factors that influence the success of online students. Embedded in the project is the expectation that findings from the research will be integrated into the business processes of each of the participating institutions for the purpose of evaluating the impact on student success. Ongoing evaluation is a key aspect of this agile research project. Over the course of this three-year project, student transcript data from Montgomery College and Prince George s Community College will be integrated with performance and classroom behavior data from UMUC. The research in data mining and predictive modeling will contribute to a process to identify relationships between the student s community college academic history and their performance at the four-year institution.

5 5 Grant partnership UMUC is a four-year public university that offers online degree programs to a diverse population of working adults. As part of this grant, UMUC established partnerships with two Maryland community colleges that also serve large and diverse student populations. Montgomery College (MC), established in 1946, currently enrolls over 35,000 students annually. MC is a diverse institution that serves a large international student population. Prince George's Community College (PGCC) enrolls more than 40,000 students and is also very diverse, serving students from approximately 125 different countries. Both institutions serve the metro-d.c. area, but are different in that PGCC serves more students who are considered low-income. Both institutions have endorsed the goals of this project and are committed to working with UMUC to find ways to better serve transfer students. A memorandum of Understanding (MOU) between UMUC and each partner institution was negotiated and signed in order to clarify the security and use of data used for this research. This MOU allows the research to move forward while protecting the privacy of students (See Appendix A). Objectives and milestones Specific objectives and milestones are presented below for each stage of the research project. These objectives and milestones have been modified slightly from the original grant documents, but are consistent with the grant requirements. Table 1 UMUC Kresge data mining grant objectives and milestones status as of September 1, 2012 Objectives Milestones Status Year One Develop a Identify project requirements. Complete project action Develop a project action and collaboration plan Complete plan with the partnering agencies. Hire a full-time researcher and half time Complete programmer. Build an evaluation framework. Complete Data Work with the different units at UMUC and Complete

6 6 Objectives Milestones Status Preparation partnering institutions to collect data. Clean and transform data. Complete Prepare a data universe (integrated database In process system) on transfer students in the population. Understand variables; define student Complete characteristics and retention data; develop data dictionary. Evaluation of Conduct ongoing project evaluation. In process project Year Two Build the Analyze data and identify factors that predict In process analytic models success/failure. Build the data mining and analytic techniques and In process integrate into the predictive models. Build student profiles based on the models. In process Assess the validity of the models Discuss results with advisory board and obtain feedback. Discuss results with Project Partners and obtain feedback. Evaluate the models in predicting student's chances for success and make recommendations for adjustments to the models Evaluation of Conduct ongoing project evaluation. project Design pilot study Year Three Conduct pilot study Evaluate the pilot Design intervention study. Define requirements for the pilot. Develop implementation plan. For the resulting predictive models, identify points of integration in the UMUC business processes. Partner institutions with transferring students will evaluate their business processes to identify points of integration of modeling techniques for improvement of student retention. Deploy interventions and evaluation systems through a pilot at the partnering institutions and at UMUC. Conduct student retention and analysis of student satisfaction. Evaluate the effectiveness of the pilot and assess transferring student s chances of success and retention with the model interventions.

7 7 Objectives Milestones Status Discuss results with Project Partners and develop recommendations. Plan project revision based on evaluation data. Dissemination of the data mining model and resources Incorporate feedback from the education community to improve the models Comprehensive evaluation of the project Report on the process and results Develop website and repository for resources. Present findings at national conferences on higher education to describe research purpose and results. Organize UMUC conference to engage national discussion around research in data mining and predictive modeling for the purpose of improving student success. Evaluate the attendance, interest and requests from the meetings. Develop and disseminate finding. Use feedback to identify and integrate into enhancement plan. Conduct ongoing project evaluation and develop an evaluation report Solidify data-sharing partnerships with community colleges, documenting roles, processes, and measurable outcomes of student success. Develop an integrated research paper outlining research purpose, research design, processes and products. Financials The Kresge Foundation awarded UMUC $1.2 million to build an integrated database, explore data mining techniques, build predictive models of student success, define intervention strategies to increase student success, deploy those strategies, and continuously monitor and evaluate the strategies and student outcomes. To date, UMUC has expended approximately 41% of the total grant. The expenses encompass a hardware purchase to house the data mining database, expenditures for data collection at the partner institutions, and dedicated salaries for a data mining specialist and a graduate assistant. Additional staff resources are provided in kind by UMUC. The remaining

8 8 grant (approximately $703,333) will be used to support external evaluation and conference development. The financial statement for this grant is in Appendix B. Section II: Research Design The research design for this study was generated from the grant s goal to develop data driven predictive models of online student success using data mining techniques. The design covers exploratory research questions that have been addressed so far. Additional questions are expected to be developed after review of the initial results. This study employs an exploratory multi-method research design. Exploratory research is useful in this context because it allows researchers to investigate relationships between variables and build predictive models that could be replicated and tested. A multi-method research design allows researchers to be flexible in their choice of methodology and to answer research questions using exploratory analyses. With this research design in mind, this study addresses the following exploratory research questions: Question 1: What relationships exist among variables in the extant dataset? Question 2: Which variables contribute to the prediction of online course success? Question 3: Which variables contribute to the prediction of retention in an online environment? Section III: Literature Review This literature review examines online student success and educational data mining literature as it pertains to student success. This section contains three parts: study definitions, online student success literature and educational data mining literature. Definitions Data mining in education research is a broad term that encompasses both data management and statistical techniques. Data management refers to how data are prepared for the

9 9 application of data mining techniques. These techniques are used to explore variables that may predict student progression related to online course success and retention. In order to maintain language consistency, terms used throughout the study are defined below. Students: stateside, undergraduate, first-bachelor s seeking students who transferred from either PGCC or MC. This is also commonly known as the cohort for the study or the population of interest. Course success: obtaining a final grade of A, B, or C in any course. Re-enrollment: enrolling the immediate next semester following the semester in question. Retention: re-enrolling at the institution within 12 months after initial enrollment or in a rolling window of three semesters following the semester of interest. Degree completion: attaining a bachelor s degree after enrolling as a degree seeking student. Student success: a broad term used to indicate course success, re-enrollment, retention, or degree completion. These definitions were determined by reviewing a number of sources including: 1) a review of data mining and online retention literature; 2) institutional publications, such as reports to the Middle States Commission on Higher Education and in-house studies on retention and course success and 3) UMUC, MC and PGCC undergraduate course catalogs. Additional terminology and definitions related to data mining and research methodology are discussed later in this report. Online student success literature Current literature on student success focuses primarily on student outcomes such as course success, course withdrawal and retention. For example, student and course level variables such as student characteristics, previous course work, grades, and time spent in course discussions and activities may be useful in measuring course success (Aragon & Johnson, 2008;

10 10 Morris & Finnegan, 2009; Morris, Finnegan & Lee 2009; Park & Choi, 2009). Courselevel variables acquired from student login data may have predictive value in measuring course withdrawal (Willging & Johnson, 2008; Nistor & Neubauer, 2010). Student, course, program and institution level variables such as student characteristics, number of transfer credits, final grade in any given course, experience in online environments, and course load may have utility measuring re-enrollment and retention (Aragon & Johnson, 2008; Morris & Finnegan, 2009; Boston, Diaz, Gibson, Ice, Richardson & Swan, 2011). Although these studies showcase a variety of findings related to student success, the majority of studies in retention in online learning environments use traditional statistical or qualitative methods. Park and Choi (2009) point out that expansion of methods such as data mining might have utility when student, course, program, and institutional level variables are well defined and institutionally meaningful. Literature related to educational data mining focusses on exploratory research. Educational data mining literature Data mining is a method of discovering new and potentially useful information from large amounts of data (Baker and Yacef, 2009; Luan, 2001). Educational data mining is a subset of the field of data mining that draws on a wide variety of literatures such as statistics, psychometrics, and computational modeling to examine relationships that may predict student outcomes (Romano and Ventura, 2007; Baker and Yacef, 2009). In educational data mining, data mining algorithms are used to create and improve models of student behavior in order to improve student learning (Luan, 2002). Data mining methods are most helpful for finding patterns already present in data, not necessarily in testing hypotheses (Luan, 2001). Baker and Yucef (2009) suggest that higher education research should use a variety of algorithms, such as classification, clustering or association algorithms in determining relationships between variables. Although many definitions of these techniques exist in data mining literature, Han and Kamber (2001) offer the following definitions. Classification is the process of finding a set of models or functions that describe and distinguish data classes or concepts to predict a class of objects whose class label is unknown. Clustering analyzes data objects that are related to similar outcomes without

11 11 consulting a class label. Association is the discovery of rules showing attribute value conditions that occur frequently together in a given set of data (Han & Kamber, pp.23-24). Recent research suggests that these data mining algorithms can be used to examine variables related to student success. Yu, DiGangi, Jannach-Pennell, Lo, and Kaprolet (2010) used a classification algorithm to explore potential predictors related to student retention in a traditional, undergraduate institution. In this study, the authors used a decision tree to explore demographic, academic performance, and enrollment variables as they related to student retention. This study revealed a predictable relationship between earned hours and retention, but also found that at this institution, retention was closely related to state of residence (in-state/out of state) and living location (on campus/off campus). The authors speculate that this finding points to the potential utility of online courses in improving retention for out-of-state or offcampus students. Despite these recent developments in exploring variables related to student success in traditional higher education settings, research using data mining techniques to uncover relationships among variables in online courses is limited in scope. This study is designed to fill this gap in the extant literature by utilizing data on online students who attended multiple institutions. Section IV: Project Data The Kresge project has several goals. One is for UMUC to develop a data sharing partnership with key community colleges. Another is to build an integrated multi-institutional data mart that describes the academic history of transfer students who have attended community colleges prior to enrolling at UMUC. This data mart will be used for data mining and for the development of statistical models that identify factors related to student success in an online environment. To that end, there are several successes in the area of data integration that have been achieved over the past year. First, a meaningful and significant partnership was established between UMUC, MC and PGCC. Second, data were collected from source systems and stored in secure areas for processing. Third, the vision and design for the Kresge Data Mart (KDM) has

12 12 been developed. Fourth, the student population of interest was identified and verified. Fifth, datasets were created for the initial modeling and data mining. The most significant success of this grant so far is the establishment of the data sharing collaboration between the community colleges and UMUC. The partnership agreements have been forged through a series of meetings and an MOU that guides the research and data sharing activities. Research questions have been developed to benefit the students through actionable research that will impact the business processes at each institution. Measurable outcomes have been set and ongoing evaluation is being conducted and shared with partner institutions. The data used in this effort will be collected from all three institutions and stored in a data mart called the Kresge Data Mart (KDM), which is housed on a server at UMUC. This section will describe the overall vision for the development of the infrastructure and the current status of the project. Student Data A core set of data is collected on each student from each institution. This information includes demographics, enrollment, courses, grades, and recent classroom activity in online courses. Each is described below. Demographic data. Demographic data include fields such as address, phone number, age, gender, and race/ethnicity. While some of this information may change over time, for the purpose of this study, the demographic data are considered static. Enrollment data. Enrollment data are typically captured on students who have officially registered. Enrollment data includes course registration, grades, program of study or major, and student status. This information changes each term or semester. Transfer data. Data about a student s academic history at previous institutions are also stored. This information includes number of transfer credits, courses transferred, grades associated with each transfer course, and prior degrees earned. There are two sources of transfer data. One is the community college transcript data that contains all course information on each student. These data are collected and integrated into the KDM. The other source is transcript files which may not include all information on the student s prior courses for only courses that were transferred to UMUC and equated to UMUC courses. The UMUC transcript data are not as

13 13 complete as the community college transcript data, but serve as the basis for courses and credits that are applied to the student s degree. Classroom behavior data. Webtycho, UMUC s proprietary learning management system, stores counts of student activity in various modules within each online class. Data are stored for each session, which is defined as the time between each student login and logoff. Identifying the population of interest The students to be included in the KDM are defined as students who attended UMUC between 2005 and Students who attended at least one community college prior to UMUC enrollment were identified and analyzed as a separate subpopulation. The data will eventually include those who attended a community college of interest, but did not transfer to UMUC. These two subgroups and the larger population of students will serve as the foundation of data stored in the KDM and the backbone for the research. The process of building the populations has multiple steps. The first step was to identify two partner institutions who would share data on students who transferred to UMUC. The second step will be to identify other institutions that are willing to do the same. The third step will be to identify students who transferred from key partner institutions, but who did not transfer to UMUC. This report focuses on the first stage of building an integrated database. The population of interest is transfer students. Two community colleges, Montgomery College and Prince George s Community College, that transfer a large number of students to UMUC were identified. UMUC formed a data sharing partnership with each institution to identify factors related to student success, define academic interventions, implement interventions, and examine their impact on student success. UMUC identified 257,903 students who attended UMUC between 2005 and Of these students, MC used varying matching strategies to confirm enrollment for 21,444 students, and PGCC used a different set of matching strategies to confirm enrollment for 11,046 students. Additionally, the National Student Clearinghouse (NSC) identified 12,776 students who attended MC and 8,697 students who attended PGCC. Furthermore, UMUC analyzed transcript information in its student information system and found even fewer matches. Thus, the Kresge

14 14 Data Mart includes student data on all 257,903 students, as well as information on the matched students matched by the community colleges. The KDM will serve as a critical resource for all predictive analytics project as well as research projects related to student success and integrated data management. One key finding is that the data from the community colleges yielded much more robust and accurate information about students academic histories at previous institution than UMUC s internal data resources. Data integration Institutions store student demographic, enrollment and classroom behavior data in a variety of information systems. For the purpose of this project, data from two UMUC student applications, PeopleSoft and WebTycho, are included in this database. PeopleSoft is a student information system that stores current and historical data about the student and their enrollment history. MC and PGCC have similar systems for storing student data. WebTycho is a proprietary learning management system that records online classroom for every user (student or faculty) during an online session. Figure 1 Description of the Integrated Kresge Data Mart

15 15 Each institution extracted data for each student in the matched population from the source student information systems and learning management systems. The process for merging data across institutions requires standardizing the data and developing a data dictionary. Data from multiple systems are combined in stages. At the first stage, data from each partner institution were collected and cleaned. Definitions of each data element were gathered, standardized, and documented. At the second stage, data from the UMUC s student information system and the online classroom were merged and developed in the UMUC data warehouse. The next stage integrated the community college data with the UMUC data in the Kresge Data Mart (KDM), as shown in the diagram above. The KDM combines information on the students history at prior institutions, their current course performance information, and their online course behavior, as well as common outcome measures identified in the literature. The final stage of the KDM will include information from MC and PGCC on students who transfer elsewhere, as well as students who have transferred from other community colleges to UMUC. This is the data mart that will serve as the base of the final research models and intervention analysis. Current status. The community college data have been collected and a coding structure has been developed. The code continues to evolve, as the data are still being processed. As a result, preliminary analyses presented in this report are based on datasets that are derived from the raw data collected and stored during the data integration process. Data storage. In order to carry out data mining activities, storage was required for the various sources of data and for the various stages of the data integration process. To meet the requirements of building an integrated data mart, UMUC purchased the Oracle Exadata hardware which houses the KDM, as well as the UMUC data warehouse. The data for the partner institutions are stored separately from the data warehouse for the purpose of security and confidentiality. The data sources on the Oracle Exadata hardware, which is in part supported by the Kresge Foundation, have also contributed to other analytical and predictive analytics projects related to predictive analytics. The data integration among various sources adds value to all projects related to student success.

16 16 Data preparation for initial analyses Researchers used sample datasets to develop data mining algorithms and predictive models. The datasets were designed specifically for the analysis of the exploratory research questions. The data mining algorithms and predictive models were created to explore the initial research questions are described in each section below along with initial findings. Data security Secure File Transfer Protocol (SFTP) is the method that is currently being used to transfer data between partner institutions. The storage area, which contains both community college and UMUC data, is secured by the database administrators providing accounts only to staff who are working on the development of the KDM. Section V: Initial findings Research Question 1: Examining variables and relationships This analysis uses data mining techniques in order to examine relationships amongst variables in the extant dataset. The question answered in this analysis is: What relationships exist amongst variables in the extant dataset? Dataset. The dataset used for this analysis contains 2,643 new, stateside, first bachelor s degree seeking UMUC students enrolled in 15 gateway courses that students in the Spring 2011 took to begin their program of study. Since each student can be enrolled in more than one course, this dataset contains 4,331 enrollment records and is comprised of 394 unique sections of these gateway courses. The variables in the dataset include transfer history, enrollment data for the students first semester at UMUC, and online classroom behavior defined as WebTycho actions in the classroom prior to and after the start day of classes (See Table 2).

17 17 Variable Name Emplid Subject Course Number Total Transfer Credits Number of Schools Attended Previously Time Since Last Institution Transferred in at Least One English Course Transferred in at Least One Math Course Earned Associates Degree Course Grouping Length of Course in Weeks Semester Load Re-enrolled Entering Class Opening Course Content Opening Class Roster Table 2 Dataset variables and definitions Variable definitions Unique ID for each new student Four-letter code for each course, i.e.- ACCT, MATH, or PSYCH Subject and course number together create a class A sum of credits/hours transferred from other institutions A count of the number of institutions in the new student s transfer history The number of months since previous institution Flag to indicate if the student had at least one class in the transfer history where the subject was ENGL, EWRT, or WRTG Flag to indicate if the student had at least one class in the transfer history where the subject was MATH, STAT, or ENGR The degree earned by the new student (if any) at a previous institution Developmental (EDCP100), General (MATH 009 and 012), Business and Professional, Communication, Science and Social Sciences, Technology Potential values are 8, 10, 12, and 14 Total credits attempted in Spring 2011 by the student A Yes/No flag indicating whether or not the student enrolled in at least one course the Summer of 2011, excluding the courses enrolled in Spring 2011 Number of times the new student logged into an area of the online course in WebTycho; counts were created for each of the first four weeks of the class as well as the number of times prior to the first day of class Number of times the student opened the Course Content which is an area of the online course; counts were created for each of the first four weeks of the class as well as the number of times prior to the first day of class Number of times the student opened the Class Roster area of the course; counts were created for each of the first four weeks of the class as well as the number of times prior to the first day of class

18 18 Variable Name Using the Chat Feature Opening a Study Group Conference Opening Reserved Reading Opening the Webliography Starting the Conferencing Feature Reading a Conference Note Writing a Conference Note Adding an Attachment to a Conference Note Variable definitions Number of times the student used the Chat Feature in WebTycho; counts were created for each of the first four weeks of the class as well as the number of times prior to the first day of class Number of times the student went to the conference postings in a study group; counts were created for each of the first four weeks of the class as well as the number of times prior to the first day of class Number of times the student opened the Reserved Reading area of the online course; counts were created for each of the first four weeks of the class as well as the number of times prior to the first day of class Number of times the student opened the Webliography area of the online course; counts were created for each of the first four weeks of the class as well as the number of times prior to the first day of class Number of times the student opened the Conferences area of the online course; counts were created for each of the first four weeks of the class as well as the number of times prior to the first day of class Number of times the student opened a note for reading in the Conferences area of the course; counts were created for each of the first four weeks of the class as well as the number of times prior to the first day of class Number of times the student wrote or edited a note in the Conferences area of the course; counts were created for each of the first four weeks of the class as well as the number of times prior to the first day of class Number of times the student attached a file to a note in the Conferences area of the course; counts were created for each of the first four weeks of the class as well as the number of times prior to the first day of class Notes: class roster is a numbered listing of everyone registered for the class; chat feature is a place where classmates can participate in synchronous discussion and view shared dialogue; a conference is an area where students participate in asynchronous group discussion, much like an internet forum; within each conference a student can post a reply, which is also called a note ; reserved readings is a place where faculty may post read-only documents or links to external website for students; webliography is a shared list of web sites relevant to the class. Method. A data mining program, IBM/SPSS Modeler, was used to develop models that explored the dataset for relationships between variables. Four classification algorithms were used: C&R Tree, CHAID, QUEST, and C5.0. Each of the four models has its own advantages and disadvantages; however, two models, C5.0 and CHAID, proved to be the most valuable in the identification of relationships between specific variables. Cross-model data mining techniques were also used to produce a ranking of variable importance.

19 19 Findings. All algorithms combined were used to rank variables across all models. These models generated a ranking for variables related to re-enrollment across data mining models (see Table 3). Table 3 Variable ranking derived from data mining models Ranking Very High Variables 1) Frequency of student activity in the online classroom up to the fourth week of class 2) Entering the online classroom prior to the first day of class 3) Participation in any conference feature 4) Opening and reviewing the Course Content module within the classroom 5) Opening and reviewing Reserved Reading within the classroom High Middle Low Number of Schools Previously Attended Students semester Load Time Since Last Institution Transferred in at Least One English Course Transferred in at Least One Math Course Negligible All Other Variables Note: Very high, high, middle, low, and negligible are statistical categories that describe the strength of the predictive power of the variables included in the re-enrollment model. This ranking indicates that classroom behavior is of very high importance to re-enrollment. In addition, the variables number of schools previously attended and semester load are somewhat important, and other variables are either of low importance or negligible importance. The C5.0 model indicated that the variable Entering Class Week 4, followed by Entering Class Prior to First Day, Number of Schools Previously Attended, Writing a Conference Note Prior to First Day, and Semester Load were most related to re-enrollment (see Figure 2).

20 20 Figure 2 C5.0 model with top five variables related to re-enrollment The CHAID algorithm also indicated that the variable Entering Class Week 4 was most related to re-enrollment (see Figure X). Figure 3 CHAID Model with top 10 variables related to re-enrollment

21 21 Implications. Data mining techniques indicate that student classroom behaviors are of very high importance to re-enrollment. Previous academic background, such as number of schools attended and transfer credits, may also be important. These models suggest variables related to the student behaviors prior to the first day of class and previous academic work merit further examination in statistical predictive models. Research Question 2: Course success The purpose of this analysis is to construct a logistic regression model with a set of covariates and predictor variables that predict course success. This model intends to answer the research question: Which variables, if any, predict student online course success? The intent of this preliminary model is to predict course success from online student behavior in the first four weeks of the course and student s academic background information. 1 The ultimate goal of analysis is to maximize the prediction of online course success. Dataset. The dataset used for this analysis contains 4,558 new, undergraduate, first bachelor-degree seeking students enrolled in 15 UMUC online gateway courses in Spring This dataset also contains transfer data on students from partner institutions, Montgomery College (MC) and Prince George's Community College (PGCC). Table 4 provides descriptive statistics for all students, passing or not passing, in each gateway course for this dataset. 1 Subsequent models may include: growth models, latent class analysis and structural equation models in order to capture a different type of data/information within one system of predictive model. 2 Sessions included in this dataset are Online Session 1 (OL1), OL2, OL3, OL4, and OL5.

22 22. Table 4 Descriptive statistics for key covariates Passed a course Not Passed Passed All Course N % N % N ACCT % 63 41% 154 BMGT % % 229 CCJS % 74 50% 149 CMIS % % 275 EDCP % % 418 EDCP % 38 42% 91 GVPT % 59 56% 105 HIST % 88 61% 145 IFSM % % 620 LIBS % % 1980 MATH % 5 31% 16 MATH % 49 46% 106 PSYC % 77 44% 174 WRTG % 62 65% 96 All % % 4558 Method. For this study, a logistic regression model was developed that utilized available independent variables (covariates and predictors) to predict the desired dependent outcome, course success. A logistic regression model generates the probability of success (in logits) as it is linearly related to the set of independent variables. In this study, a pseudo- R 2 value 3, or value of prediction accuracy, was used for both model diagnosis and selection. Preliminary methods focused on the identification of covariates (e.g. student demographic information and the record of previous academic work) and predictors based on the students coursework behavior. Identification of key covariates. Exploratory factor analysis (EFA) was used to identify key covariates. EFA is an exploratory or data mining method that uncovers underlying constructs, or common structures, among a set of related variables. Previous data mining 3 Pseudo or generalized R squared is a generalization of the coefficient of determination, R 2, to more general regression model such as logistic regression (Nagelkerke,1991).

23 23 analysis 4 indicated that the following covariates in student coursework data were strongly associated with the course success: 1) total number of transfer credits (Xfer_total); 2) amount of time since students attended the last institution (Xfer_time); 3); semester course load (LOAD) and 4) GPA from transferred credits (GPA_C). Table 5 shows the rule to categorize each covariate for use in the logistic regression model. Table 5 Rules for key covariates in student coursework data Attribute Variable Name Rule Total number of transfer credits Xfer_total 0= 0 credit 1= 1-29 credits 2= credits Amount of time since a student attended the last institution Xfer_time 3= more than 60 credits 0= didn t attend 1= 1-24 months 2= over 24 months Semester course load LOAD 1 = 1-3 credits 2 = over 3 credits GPA from transferred credits GPA_C 0 = no information 1 = less than = 2.5 to less than = 3 to less than = over 3.5 Additional covariates related to student behavior prior to the official class starting day (called Week 0) were also found to be associated with course success. Table 6 displays summarized rules for each student behavior covariate in Week 0. 4 See section on cluster analysis for detailed explanation of this process.

24 24 Table 6 Rules for LMS student behaviors in Week 0 Behavior Variable Name Rule Entering a class Opnclss_cat0 0 = Did not enter a class 1= Entered a class at least once Opened a class roster Opnclssrstr_cat0 0 = Did not check the roster 1 = Check the roster at least once Opened a class content Opncrscntnt_cat0 0 = Did not open a class content 1 = Open a class content at least once Created a conference note Createconfnote_cat0 0 = Did not create a conference note 1= Created a conference note at least once Created a response note Createrespnote_cat0 0 = Did not create response note 1 = Created a response note at least once Summary of students week0 behavior prior to the first day TimeZeroTotal Sum of five variables above Using these rules, a logistic regression model was used to evaluate the effect of the identified covariates (in terms of pseudo- R 2 ) on course success for use in the final model (see Table 6). These findings include: 1. Total number of transferred credits is the best covariate predictor of course success (having pseudo R 2 value around.12) It was the best predictor of success amongst the time invariant covariates, or covariates that have consistent value over all models in the analysis using different points in time. 6 The covariate number of schools previously attended (Xfer_NBR) exhibited the linear dependency with a total number of transferred credits thus to remove from the analysis.

25 25 2. GPA from transferred credits is the second best covariate predictor of course success (pseudo R 2 around.11). 3. Semester course load contributes less to course success than other covariates. Analysis of combined covariates (student behaviors and previous coursework behavior) indicates that total number of transfer credits (Xfer_total), amount of time since a student attended the last institution (Xfer_time), GPA from transferred credits (GPA_C), and summary of student s behavior prior to first day (TimeZeroTotal) accounts for approximately 15-20% of the variance based on the pseudo R 2 value (see Table 7). Table 7 Summary of pseudo R 2 estimates Covariates R 2 estimates xfer_total 0.12 xfer_time 0.06 Load 0.01 GPA_C 0.11 Opnclss_cat Opnclssrstr_cat Opncrscntnt_cat Createconfnote_cat Createrespnote_cat TimeZeroTotal 0.07 xfer_total xfer_time LOAD TimeZeroTotal GPA_C 0.19 Building the predictive model. In order to accurately predict course success, predictor variables that measure student online course behavior in Weeks 1-4 were integrated into the logistic regression model. Exploratory analysis based on the data mining techniques and the availability of data indicated the importance of five predictor variables related to online course behavior: 1) entering a class; 2) opening class content; 3) creating a conference note; 4) creating

26 26 a response note and 5) reading a conference note. In order to account for high variability in student online course behaviors predictors were categorized using the following rules (see Table 8): Table 8 Rules for student online course taking behavior beyond Week 0 Behavior Variable Name Rule Entering a class Opnclss_catN 0 = Did not enter a class 1= Entered a class at least once Opened a class content Opncrscntnt_catN2 0 = Did not open a class content more than twice 1 = Open a class content at least twice Created a conference note Createconfnote_catN 0 = Did not create a conference note 1= Created a conference note at least once Created a response note Createrespnote_catN 0 = Did not create response note 1 = Created a response note at least once Read a conference note readconfnote_catn2 0 = Did not open a class content more than twice 1 = Created a response note at least twice In this case, each predictor variable represents a weekly total of distinctive actions per student per class in Weeks 1-4 of an online gateway course (see Table 9):

27 27 Table 9 Weekly total of actions per student beyond Week 0 Variable N Mean SD Percentile OPNCLSS_WEEK OPNCLSS_WEEK OPNCLSS_WEEK OPNCLSS_WEEK OPNCLSSRSTR_WEEK OPNCLSSRSTR_WEEK OPNCLSSRSTR_WEEK OPNCLSSRSTR_WEEK OPNCRSCNTNT_WEEK OPNCRSCNTNT_WEEK OPNCRSCNTNT_WEEK OPNCRSCNTNT_WEEK READCONFNOTE_WEEK READCONFNOTE_WEEK READCONFNOTE_WEEK READCONFNOTE_WEEK UPDTCONFNOTE_WEEK UPDTCONFNOTE_WEEK UPDTCONFNOTE_WEEK UPDTCONFNOTE_WEEK CREATECONFNOTE_WEEK CREATECONFNOTE_WEEK CREATECONFNOTE_WEEK CREATECONFNOTE_WEEK CREATERESPNOTE_WEEK CREATERESPNOTE_WEEK CREATERESPNOTE_WEEK CREATERESPNOTE_WEEK This categorization accomplished three purposes: 1) removed the effect of extreme values of highly variable actions; 2) facilitated a percentile level count of each behavior and 3) examined

28 28 the effect of each cut point 7. Categorization of predictor variables also emphasizes the importance of consistent engagement in a classroom activity, which is indicated by the categorization of 1 as a predictor variable for student action from Week 1 to Week 4 and 0 as an variable indicating no action. 8 Evaluating the predictive model. The covariates and predictors included in the final logistic regression model represent the best model fit (as represented by pseudo R 2 ) to evaluate student course success. In addition, this model allows for determination of the relative contribution of each covariate and predictor at each point in time, which will aid in identifying at-risk students. For example, the power of the model in predicting course success becomes extremely high (close to 90%) and pseudo R 2 exceeding.60 using 8 weeks of data. However, since interventions for increased course success are not useful at the end of the semester, findings focus on the performance of predictors over the first four weeks, particularly the first two weeks of behaviors in the online class. Findings. Initial analysis using the final model indicates the relative contribution of covariates and predictors from Week 1 to Week 4 (see Table 10). Table 10 Relative contribution of Covariates and Predictors Week Covariates xfer_total 15% 18% 16% 17% xfer_time 2% 2% 2% 2% Load 5% 7% 6% 7% TimeZeroTotal 11% 10% 9% 7% GPA_C 7% 9% 8% 9% Predictors createrespnote_total 10% 3% 6% 2% readconfnote_total2 34% 32% 23% 21% createconfnote_total 15% 7% 14% 10% opnclss_total 0% 13% 16% 26% opncrscntnt_total2 0% 0% 0% 1% 7 Cut point refers to numeric values or rules to categorize a continuous value into categories. 8 In other words, the predictor variable represents the sum of student actions across N weeks (0-4) as the final predictor variable in the model (e.g. <action>_total).

29 29 Examination of the relative contribution of covariates indicates that total transfer credit (xfer_total) exhibited the highest level of contribution to the predictive model (15%-18%) followed by Week 0 behavior (TimeZeroTotal), transfer GPA (GPA_C), course load (LOAD) and number of times a student transferred (xfer_time), respectively. Examination of the relative contribution of predictors indicates that the variable reading conference note (readconfnote_total2) has the highest influence in the predictive model (21%-34%). Note that open class (opnclss_total) had no impact on student success in Week 1, but that impact increased Weeks 2-4. The predictive power of the model increases gradually over the first four weeks of the course, from 79% correct prediction of course success at Week One 1, to 82% correct prediction of course success at Week 4. The pseudo R 2 value increased from approximately 19% using the covariate-only model (see Table 4 for value) to 50% in Week 1 predictive model (see Table 11). Table 11 Predictive power of the model by week Outcome/Predicted R2 estimates Count Percent Success Week P/P F/F F/P P/F P/P F/F F/P P/F % 37% 10% 11% 79% % 38% 9% 11% 81% % 39% 9% 10% 81% % 38% 9% 9% 82% Note that the pseudo R 2 value increased from 50% at Week 1 to 59% at Week 4. Group level analysis. Predictive analysis by sub-group based on sessions or courses was conducted using the model. Although findings from this group-level analysis is less informative due to limited sample size, the predictive model maintained a moderate to high level of fit based on pseudo R-squared value (e.g. above mid.30s). 9 The group-level analysis demonstrated that 9 OL5 session had a fair model fit but exhibited low prediction rate. The sample size of OL5 session was too small to draw a conclusion.

30 30 the predictive model maintains high level of model fit and prediction at over 75% in the majority of sub-groups (see Table 12) 10. Table 12 Predictive power of the model by week and session Week Session Stats OL1 R Success OL2 R Success OL3 R Success OL4 R Success OL5 R Success OL123 R Success LIBS150 R Success No LIBS 150 R Success Core LIBS R This sample included all students by session level and course category (general education, and remedial course), but excluded students in LIBS 150. LIBS 150 is a general education course; it is shorter and more orientation/informative course rather than a regular educational course, and it has very high passing rate of 75.3%.

31 31 Success Core No LIBS R Success EDCP100 R Success IFSM201 R Success EDCP100 IFSM201 R Success Implications. Preliminary analysis of variables related to course success identified a set of critical covariates and predictors. Four of five predictors derived from online student behavior show a strong contribution to student success. The final model using the full dataset will examine the effect of these predictors as well as evaluating the impact of other behavioral measures such as accessing course modules. Research Question 3: Student retention The purpose of this preliminary analysis is to construct a logistic regression model with a set of covariates and predictor variables and determine the impact on student retention. This model intends to answer the research question: Which variables, if any, predict retention? Dataset. The analysis utilized the same data set for the prediction of course success with the addition of retention status from Summer 2011, Fall 2011, and Spring The retention rate was 66% (N= 3,015) in this sample. Table 13 Percentage students retained and not retained Retained Not Retained N % N % % %

32 32 Method. As with prediction of course success, this model uses logistic regression to predict retention. Preliminary methods focused on the evaluation of covariates, identified in the previous analysis, predictors based on the students coursework behavior, and course success information (i.e. passed or not passed). Evaluation of covariates. A total number of credit transferred (xfer_total) had the highest predictive power followed by GPA from the transfer credits (GPA_C). The magnitude of covariates pseudo R 2 is lower on the prediction of retention than the prediction of course success. The total covariates yield a pseudo R 2 of Table 14 shows pseudo R 2 estimates from five covariates. Table 14 Retention pseudo R 2 estimates of covariates Covariates R 2 estimates xfer_total 0.08 xfer_time 0.02 GPA_C 0.07 Load 0.01 TimeZeroTotal 0.03 Cov Total 0.14 Evaluation of predictors. For this model, predictors identified for the course success model were used and combined with a new set of predictors: 1) number of courses passed (N_Pass); 2) number of courses not passed (N_NotPass); 3) number of course withdrawals (W) or failure due to non-attendance (FN) (N_WFN); 4) passing LIBS 150 (passed_libs150); 5) passing a General Education course (passed_gened); 6) passing a remedial course (passed_remed) and 7) term GPA (PGA_Term). In this model, student behavior indicators did not predict retention as well as they predicted course success (see Table 15).

33 33 Table 15 Prediction success of combined student behavior indicators by week Retention Course Success Week Low High Low High Findings. When included in the prediction model for retention, covariates predicting course success generated a higher a pseudo R 2 value than student behavior predictors (see Table 16). Table 16 R-squared estimates for retention predictors and covariates Predictors and covariates R 2 estimates passed_libs passed_gened 0.15 passed_remed 0.15 N_Pass 0.19 N_NotPass 0.17 N_WFN 0.16 GPA_Term 0.24 COVS passed_abc passed_gened passed_remed N_Pass 0.35 N_NotPass N_WFN GPA_Term ***COVS= xfer_total xfer_time LOAD GPA_C GPA and number of passing courses had the highest pseudo R 2 estimates among seven indicators. The full model including covariates and passing indicators had a pseudo R 2 of Implications. The effect of covariate and student behavior variables made less of a contribution to this model than the prediction of course success. The final model using the course

34 34 passing indicators 11 indicated a stronger contribution to the prediction of retention. These preliminary results indicate the importance of course success as the primary factor in increasing retention, which means that interventions aimed at improving course success may have a strong indirect effect on increasing retention. Additional exploratory analysis: Student withdrawal patterns The purpose of this additional exploratory analysis was to conduct a survival analysis that examined student course withdrawal patterns. The goal of this study is to increase the understanding of withdrawal patterns so that it can be effectively incorporated in models predicting the retention. Dataset. The dataset for this study contains 19,190 undergraduate UMUC students in OL1 (Online Session 1) in Fall 2011 in 278 distinct courses. This dataset contained student course information data and WebTycho data. Summary statistics of five student demographic and academic behavior variables are listed in Table 17. Table 17 Demographics for OL1 students Variable Frequency Percent Cumulative Percent Gender Unknown Female Male Age Group Above at or below New/Returning Status NEW RET For additional information on course withdrawal related to student behavior variables is explained in the subsequent section. 12 Age 32 is used as a cut point because it is the median age of UMUC students.

35 35 The majority of the students in OL1 of FAl l2011 were female (56.7%). 56.2% of the students were at or below the UMUC median student age of 32 years old. 17.5% of the students were new to UMUC and 82.5% were returning students. Method. An exploratory survival analysis was carried out using a Kaplan-Meier estimator. Survival analysis is a statistical technique that can be used to model time-to-event data. In this case, this analysis examines the time it took for a student to withdraw from a particular course (in weeks and days) reflected as a time-to-event. 13 Survival analysis generates a table which indicates a hazard (or withdrawal) rate during the semester. Additional univariate survival analyses were also conducted to identify retention patterns among age, gender, and student status. Survival analysis. Table 18 shows the number of student withdrawals, cumulative proportion of withdrawals, and withdrawal rate. Week Number of students Table 18 Withdrawal rate in OL1 Number of Student Withdrawals Cumulative Proportion of Withdrawal Withdrawal rate Student rate of withdrawal is highest in Week 1 (N= 404 or.31%). The withdrawal rate gradually drops until Week 5, which is the week before the academic withdrawal deadline. The overall trend and withdrawal pattern is that, in general, student withdrawal rates decrease after Week 1 of the semester. 13 This analysis only examines students who received a grade of W (withdrawal).

36 36 Univariate survival analysis. This univariate survival analysis examined three covariates: gender, age, and student status. Age, which was defined as below or above 32 years old, did not prove to be statistically significant (p =.60) in terms of correlation with withdrawal patterns. Gender, male and female, approached significance (p =.06) in terms of correlation with withdrawal patterns. Student status (new or returning) showed statistical significance in terms of correlation with withdrawal (p <.001): new students withdraw at a higher rate than returning students. Figure 3 shows that the patterns of new and returning student withdrawal are different. Figure 4 Withdrawal rate for new and returning students by day

37 37 Implications. Students withdraw at a higher rate in Week 1 compared to any other week in the course session, with the exception of Week 5, which is the academic withdrawal deadline. Student status, new or returning, may significantly affect student withdrawal rate. New students withdraw at a higher rate than returning students. These findings suggest that interventions targeting new students with interventions in Week 1 may be appropriate. Section VI: Implications and next steps In summary, data mining and statistical modeling can identify variables that relate to student success. Data mining techniques identified the importance of academic background and student course behaviors as variables related to student success. Predictive models of course success and retention using variables gleaned from data mining showed similar outcomes. Specifically, online student course behavior, such as opening and reading conference notes in the first four weeks of class, were correlated with course success. Previous academic background, such as total number of transfer credits and transfer GPA, contributed both to course success and retention. Course success, instead of student behavior, was a key predictor of retention. Student course behaviors showed that students withdraw at a higher rate in Week 1 over any other week in the course session, with the exception of Week 5, and that student status may significantly affect student withdrawal rate. These findings will guide our next research steps, which will finalize and validate predictive models in order to build student profiles. These steps may include, but are not limited to: running predictive models with a full dataset; validating models; sub-group analysis examining community college students in terms of course success, re-enrollment and retention; identifying course sequences that indicate re-enrollment and retention; integrating withdrawal information into predictive models. These additional steps will aid in the building of student profiles that can be used for predicting student success and designing effective interventions.

38 38 References Aragon, S.R. & Johnson, E.S. (2008). Factors influencing completion and non-completion in online community college courses. American Journal of Online Education, 22(3), Boston, W., Diaz, S.R., Gibson, A.M., Ice, P., Richardson J., & Swan, K. (2011). An exploration of the relationship between indicators of the community of inquiry framework and retention in online programs. Journal of Asynchronous Learning Networks, 13(3), Baker, R.S. & Yacef, K. (2009). The state of educational data mining: A review and future visions. Journal of Educational Data Mining, 1(1), Finnegan, C., Morris, L.V., and Lee, K. (2009). Differences by course discipline on student behavior, persistence, and achievement in online courses of undergraduate general education. Journal of College Student Retention, 10(1), Herzog, S. (2006). Estimating student retention and degree completion time: Decision trees and neural networks vis a vis regression. New Directions for Institutional Research, 131, Ho Yu, C., DiGangi, S., Jannasch-Pennell, A., Lo, W., & Kaprolet, C. (2007, February). A datamining approach to differentiate predictors of retention. Paper presented at the Educause Southwest Conference, Phoenix, AZ. Luan, J. & Zhao, C-M. (2006). Data mining: Going beyond traditional statistics. New Directions for Institutional Research, 131, Luan, J. (2001). Data mining as driven by knowledge management and higher education: Persistence clustering and prediction. Paper presented at the SPSS Public Conference, University of California at San Francisco. Luan, J, (2002). Data mining and its applications in higher education. New Directions for Institutional Research, 113, Morris, L.V. & Finnegan, C.L. (2009). Best practices in predicting and encouraging student persistence and achievement online. Journal of College Student Retention, 10(1), Nistor, N., & Neubauer. K (2010). From participation to dropout: Quantitative participation patterns in online university courses. Computers in Education, 55, Park, J-H., & Choi. H-J. (2009). Factors influencing adult learners decision to drop out or persist in online learning. Educational Technology and Society, 12(4), Romero, C., Ventura, S., Espejo, P.G. and Hervas, C. (2008, June). Data mining algorithms to classify students. Paper presented at the1st International Conference on

39 39 Educational Data Mining, Montreal, Canada. Willging, P.A. & Johnson, S.D. (2009). Factors influencing adult student decisions to dropout of online courses. Journal of Asynchronous Learning Networks, 13 (3),

40 40 Appendix A Memoranda of Understanding

41 41

42 42

43 43

44 44

45 45

46 46

47 47

48 48

49 49

50 50

51 51 Appendix B Financial Statement

52 52