Redefining At-Risk Through Predictive Analytics: A Targeted Approach to Enhancing Student Success

Redefining At-Risk Through Predictive Analytics: A Targeted Approach to Enhancing Student Success Amilcah Gomes Assistant Director, Academic Services Center Eastern Connecticut State University

Predictive Analytics Using real-time data to plan for the future Predictive analytics helps your organization predict with confidence what will happen next so that you can make smarter decisions and improve business outcomes. IBM

Institutions That Use Predictive Analytics Predictive Analysis Reporting (PAR) Framework Institutions American Public University System Ashford University Broward College Capella University Colorado Community College System Lone Star College System Penn State World Campus Rio Salado College Sinclair Community College Troy University University of Central Florida University of Hawaii System University of Illinois Springfield University of Maryland University College University of Phoenix Western Governors University Purdue University Rutgers University University of South Florida Eastern Connecticut State University Others

Eastern Connecticut State University Location: Willimantic, Connecticut Institution Type: Public, Liberal Arts University Total Enrollment (2012): 5,237 students, 4,506 undergraduate FTE Nearly 2/3 residential, with 93% of first-time students living on campus 2011 FTFT cohort (N = 931): Female 53.4%; Students of Color 22% Other Characteristics First Gen Students (32.5%), Compass Cohort (50.1%), STEP/CAP students (6.8%), Honors Scholars (2.4%), Athletes (9.7%) 2012 FTFT cohort (N = 979) 2013 FTFT cohort (N = 960)

Institutional Changes Strategic Plan: Student Success Initiative (2008-2013) Dual Advising Model Project Compass (2008-2012) Communities of Practice Early Identification of At-Risk Students Enhancing First-Year Advising Services Targeting tutoring services to high-risk subjects Title III Student Support Services (2009-2014) 2013-2018 Strategic Plan & Targeted Student Success Initiatives

Development of Prediction Models 2008 Multivariate model developed for predicting withdrawal prior to second year Original assumption: students withdraw due to poor academic performance Problem: Original model failed to differentiate between students who left for academic and nonacademic reasons Did not account for financial reasons, motivational variables, engagement factors, etc. Withdrawal risk quintiles were difficult to interpret Professional advisors were unable to develop advising strategies specific to a student s needs 2011 Developed additional multivariate model for academic risk; implemented two-model approach

Withdrawal Model, 2011 Cohort (2008, 2009 Data) Variable B S.E. Wald df Sig. Exp(B) Base male -.087.122.508 1.476.917 female black -.723.278 6.786 1.009.485* white hisp.347.279 1.540 1.215 1.414 oth_race -.674.274 6.069 1.014.510 not_east.408.133 9.401 1.002 1.503 commuter.497.211 5.545 1.019 1.644 Pell_yr1 -.245.161 2.316 1.128.783 first_gen -.076.124.378 1.539.927 Athletics -.647.218 8.816 1.003.524 HsGpa_quint1.734.189 15.070 1.000 2.084 HsGpa_quint2.242.184 1.727 1.189 1.274 HsGpa_quint4.277.186 2.228 1.136 1.319 HsGpa_quint5.080.223.130 1.719 1.084 admit_rating_le_4 -.057.160.125 1.723.945 admit_rating_ge_8 -.573.203 7.985 1.005.564 Vsat_quin1 -.069.184.140 1.708.934 Vsat_quin2 -.320.180 3.157 1.076.726 Vsat_quin4.191.173 1.216 1.270 1.210 Vsat_quin5.172.186.854 1.356 1.188 Stem -.036.177.042 1.838.964 PreEd -.263.165 2.536 1.111.769 Undec.121.132.837 1.360 1.129 ERG_none.484.201 5.834 1.016 1.623 ERG_ABC.164.151 1.181 1.277 1.178 ERG_GHI.269.153 3.089 1.079 1.309 East CT River Campus Not Pell Not FGEN Not Athlete Quintile 3 Rate 5, 6, or 7 VSAT Quint 3 Not STEM Not PreED Declared major ERG DEF got_schol_yr1 -.022.165.018 1.893.978 No schol got_fedloan_yr1 -.227.125 3.326 1.068.797 No Fed Loan Choice -.246.120 4.235 1.040.782 Not #1/No FAFSA Constant -1.282.241 28.336 1.000.277 *Factors that are significant at the 0.10 level have been highlighted.

Academic Risk Model, 2011 Cohort (2008, 2009 Data) Model 2: for classifying 2011 Cohort into Academic-Risk Quintiles, Based on Data from 2008 and 2009 Cohorts. B S.E. Wald df Sig. Exp(B) HsGpa -1.505.180 69.974 1.000.222 ERG_DEF.222.151 2.165 1.141 1.249 ERG_GHI.300.157 3.657 1.056 1.350 ERG_none.091.204.201 1.654 1.096 black.365.222 2.714 1.100 1.441 hisp.429.275 2.445 1.118 1.536 oth_race -.194.245.629 1.428.824 Pell_yr1.140.148.891 1.345 1.150 first_gen.028.123.052 1.820 1.028 male.344.121 8.039 1.005 1.410 Stem.387.168 5.325 1.021 1.472 PreEd -.274.172 2.555 1.110.760 Athletics -.641.216 8.770 1.003.527 admit_rating_le_4.186.144 1.663 1.197 1.204 admit_rating_ge_8 -.414.200 4.271 1.039.661 Constant 2.952.557 28.125 1.000 19.150

Interpreting the Models: What We Saw Factors that had a significant impact (0.10 level): Factors Significant in Both Models Commuter Not East Federal Loans FAFSA Choice Only Significant in Withdrawal Model HS GPA ERG/DRG (CT HS districts) Athlete (negative, both) African American identity (negative withdrawal, positive academic risk) Admissions Rating 8 (negative, both) Males STEM Majors Only Significant in Academic Risk Model

Interpreting the Models: What We Didn t See Factors that did not have a significant impact (0.10 level): Pell eligibility (less significant in GPA model) 1 st Generation Status Undeclared Majors (significant for 2 nd to 3 rd year retention) Multiracial Identities (added to 2012 model) Math SAT (good predictor of GPA > 3.0)

Classifying Withdrawal & Academic Risk Cross-classification of 2011 cohort of entering first-time, full time students by the two types of risk: withdrawal and low academic performance: Academic Risk Quintile 2011 1.00 2.00 3.00 4.00 5.00 Total Q_2011 1.00 99 57 35 35 24 250 2.00 45 47 39 33 23 187 3.00 29 49 50 45 45 218 4.00 10 32 30 49 35 156 5.00 2 15 15 26 62 120 Total 185 200 169 188 189 931 TAC 1 = Intensive 172 TAC 2 = Tutoring 205 TAC 3 = Engaged 232 TAC 4 = Monitor 322

Targeted Advising Cohorts (TAC) TAC 1 = Intensive (high risk withdrawal & high risk GPA < 2.3) TAC 2 = Tutoring (low risk withdrawal & high risk GPA < 2.3) TAC 3 = Engaged (high risk withdrawal & low risk GPA < 2.3) TAC 4 = Monitor (low risk withdrawal & low risk GPA < 2.3)

Targeted Advising Cohorts (TAC) High academic risk TAC 2 TAC 1 Tutoring Intensive Low risk withdrawal High risk withdrawal High risk GPA < 2.3 High risk GPA < 2.3 Low withdrawal risk TAC 4 Monitor Low risk withdrawal TAC 3 Engaged High risk withdrawal High withdrawal risk Low risk GPA < 2.3 Low risk GPA < 2.3 Low academic risk

Fall 2011 Cohort TAC Assignments* By using the two-model approach it was determined that 18.5% of the FTFT students entering were at risk of withdrawal and low academic performance (TAC1), and another 25.5% were at risk of low academic performance (TAC2). 33.2% 18.5% TAC1=Intensive TAC2=Tutoring TAC3=Engaged TAC4=Monitor 25.5% * 35 students originally in TACs 3 and 4 who did not participate in the library assessment and orientation were reassigned to TAC2. 22.8%

Observations Gender Males made up less than half (46.6%) of 2011 cohort, but represented 67% of TAC 1 & 2 students Race/Ethnicity Students of color, particularly African American/Black students, were overrepresented in TAC 2 (46.3% vs. 23.3% overall) Home of Record 80.2% of TAC 1 students were Not East (international, out of state, and students living west of the CT river) students (56.9% overall) Other Characteristics 68.8% of Pell eligible and first generation students were classified in TACs 2 & 4

Summary of Outcomes FTFT 2011 General Characteristics Outcomes (Average) N % Female % 1st Gen % Students of Color # Credits Att. # Credits Earned 1st Yr. GPA Lib.Score Retention 927 53.4 32.5 22.0 29.40 26.18 2.79 1.11 75.5 TAC1 171 32.2 27.5 15.7 28.88 24.18 2.34 0.98 67.3 TAC2 235 37.0 40.0 42.4 28.83 23.13 2.38 0.99 74.9 TAC3 210 66.7 23.8 3.4 29.23 27.85 3.10 1.15 71.9 TAC4 306 69.0 35.0 22.6 30.23 28.49 3.14 1.23 83.7 There is very little variation in credits attempted across the four TACs Students in TAC 1 and TAC 2 on average were slightly less likely to achieve the same level of academic momentum as their classmates in terms of credits earned In addition, the average GPAs for students in TAC 1 and TAC 2 are significantly lower than those in TAC 3 and TAC 4 TAC assignment is clearly related to retention TAC 3 and TAC 4 more likely to utilize tutoring services and devote more hours with math and writing tutors on average

Data-Informed Approach Reassess assumptions about student withdrawal patterns and ALANA student performance on campus Align institutional priorities to significantly enhance student success Better allocate limited staffing and financial resources to support high-impact practices Address persistence and performance issues among high-risk groups

Developing Targeted Interventions Freshman Preference Registration Developmental Advising Focus Leading Indicators Project (Library Orientation Score) Collaboration with Campus Departments Registration Holds and Financial Review Diversity Scholars Program Strategies for High-Performing Students Recognition of ALANA Student Success

Intervention: Major Course Selection TAC FTFT 2011 Cohort Mean CGPA First-Semester GPA for FTFT 2011 & 2012 cohorts by TAC N FTFT 2012 Cohort Major Course CGPA N FTT 2012 Cohort No Major Course CGPA 1 2.35 168 2.54 114 2.48 50 2 2.28 199 2.64 84 2.32 53 2A 2.49 32 2.90 36 3.31 3 3 3.10 205 3.28 121 3.07 55 4 3.15 304 3.25 191 3.22 77 Total 2.78 908* 2.99 546 2.83 238 N * 23 students did not complete the Fall 2011 semester and were not included in the first semester GPA analysis

Intervention: Major Course Selection TAC First-Semester Persistence for FTFT 2011 & 2012 cohorts by TAC FTFT 2011 Cohort Returned Spring 2012 FTFT 2012 Cohort Returned Spring 2013 FTFT 2012 Cohort Major Course Returned Spring 2013 FTFT 2012 Cohort No Major Course Returned Spring 2013 1 89.0% 90.0% 93.0% 84.9% 2 89.7% 93.0% 95.2% 90.7% 2A 82.9% 93.3% 94.4% 75.0% 3 89.4% 94.0% 95.0% 91.2% 4 94.8% 95.0% 95.3% 94.9% Total 90.9% 93.3% 94.7% 90.7%

Intervention: Major Course Selection Year 1 Persistence to Year 2 Year 1 Academic Performance Outcomes (Average) No Maj Cume No Maj ECO 200 Cume GPA grade in 1 st Cume GPA Cohort Overall Crse ECO 200 BUS 201 GPA Crse ECO 200 Semester BUS 201 BUS 201 grade in 1 st Semester FTFT 2011 75.6% 72.5% 77.5% 82.4% 2.65 2.54 2.70 2.52 (C+/B-) 3.19 3.05 (B) FTFT 2012 75.6% 70.0% 77.2% 80.9% 2.86 2.71 2.88 2.46 (C+/B-) 2.92 3.15 (B/B+) Year 2 Persistence to Year 3 Year 2 Academic Performance Outcomes (Average) No Maj Cume No Maj ECO 200 Cume GPA grade in 1 st Cume GPA BUS 201 grade in 1 st Cohort Overall Crse ECO 200 BUS 201 GPA Crse ECO 200 Semester BUS 201 Semester FTFT 2011 61.1% 52.5% 65.0% 76.5% 2.72 2.48 2.91 2.52 (C+/B-) 3.16 3.05 (B) BUAD majors taking BUS 201 and ECO 200 during first semester via FPR Slight improvement in cumulative GPA for students in BUS 201 over ECO 200 Students consistently performed better in BUS 201, regardless of TAC, etc. Some students may be at a disadvantage with ECO 200, particularly African American, Hispanic/Latino, FGEN, PELL, and TAC 1 & TAC 2 students Intervention for both at-risk and high-performing students

Intervention: Registration Holds 36.8% of 2012 cohort had holds just prior to Spring 2013 advising and registration, mostly financial 44% of TAC 1 and 48% of TAC 2 students had holds 45.7% of TAC 2A students had holds 76.2% of STEP/CAP students and 51.1% of nonwhite, non-step/cap students had holds vs. nearly 30% of white students Average midterm GPA with holds: 2.53 (2.81 without holds) Collaboration with Enrollment Management, Fiscal Affairs, and Student Affairs Financial Review Day Intrusive Advising Strategies Individual Day Passes Average cumulative GPA with holds: 2.74 (3.07 without holds) TAC Returned Spring 2013 N Did Not Return Total Not Returned with Holds 1 90.0% 21 11 (52.4%) 211 2 93.0% 12 9 (75.0%) 171 2A 93.3% 3 2 (66.6%) 45 3 94.0% 14 8 (57.1%) 232 4 95.0% 16 8 (50.0%) 320 Total 93.3% 66 38 (57.6%) 979 N

Redefining Perceptions of At-Risk Shifting from deficitbased assumptions Cognitive deficit assumptions mischaracterization marginalization toward a successbased model! Targeted initiatives based on student needs relevant student support

Diversity Scholars Program Spring 2012 Pilot group for FTFT SOC, TAC 1-2, < 2.0 GPA (N = 28) Fall 2012 services for all FTFT ALANA students (N = 133) Developmental advising services had positive impact on GPA for TAC 1, limited impact for TAC 2 when given alone ( high-relational groups) Peer mentoring: participants (2.79) vs. non-participants (2.47) Hispanic/Latino student participants (3.03) vs. non-participants (2.13) Academic & student support interventions (e.g., taking major course in first semester, financial review) FTFT students of color had 3-4 times higher increase in GPA from previous year than white students, except TAC 4 African American students Reduction in GPA gap experienced across all ethnicities Fall 2013 added targeted developmental advising outcomes and major-related opportunities/workshops (N = 138) TAC 1-2 DSP participants (2.94) vs. ALANA non-participants (2.34) TAC 1-2 Hispanic/Latino students outperformed white students during first semester (2.72 vs. 2.59) TAC 3-4 African American students still underperform (2.89 vs. 3.10)

Recommendations for First-Year Success Use multiple approaches to assist students, based on individual needs and types of risk Consider non-academic barriers when advising students (finances, registration holds, campus integration, family issues, etc.) Encourage use of early alert system (APN) and sharing of information across departments Connect students to academic departments and major-related opportunities, particularly high performing students

Where do we go from here? Explore other non-cognitive measures and/or instruments and administer during the fall semester Revisit second-year persistence models developed during Project Compass Develop advising strategies for second-year students Communicate data findings with more faculty Increase access to high-impact practices, particularly for underrepresented students

ANY QUESTIONS?