Multivariate Models of Student Success

Multivariate Models of Student Success RP Group/CISOA Conference April 28, 2009 Granlibakken, Lake Tahoe Dr. Matt Wetstein Dean of Planning, Research and Institutional Effectiveness San Joaquin Delta College Alyssa Nguyen Bri Hays Research Analyst Research Analyst

Introduction Many studies of student success rely on quasi experimental designs Intervention is tested for its effect, usually with a control group Downside lack of multiple control variables More and more, researchers are turning to multivariate logistic regression models

Introduction Logistic Regression s appeal Many of our dependent variables of interest are well suited for dichotomous analysis Techniques have become standard in packages like SAS, STATA, SPSS Allows for multivariate analysis and more holistic understanding of student behavior

Introduction RP Group researchers are leading the way in recent years some examples Wurtz (2008) Logit model for generating placement test recommendations Spurling (2007) Logit model of prior English enrollment on GE course success Younglove (2009) Logit model to recommend concurrent course enrollment for basic skills students CSS (2002) Logit model to validate prerequisites for enrollment in nursing programs

Introduction Some notes on Logit S shaped curve Should be little collinearity among independent variables Goodness of Fit Reliable models have non significant Chi Square values using the Hosmer Lemeshow goodness of fit test Model s ability to correctly classify cases vs. modal guessing strategy My prior use of Logit explaining judicial voting behavior in the U.S. & Canadian Supreme Court

Model of Student Success Current interest: developing multivariate models to examine patterns of student success Background traits (SES, ethnicity, income) Skill levels Norms & attitudes toward college & transfer Engagement in college life & services

Model of Student Success Assessment scores GPA history Student s Background Characteristics Course taking patterns & unit loads George Kuh et. al. 2005. Student Success in College: Creating Conditions that Matter. San Francisco: Jossey Bass. Colleen Moore & Nancy Shulock. 2007. Beyond the Open Door: Increasing Student Success in the California Community Colleges. Sacramento: CSU Sacramento Steve Spurling. 2007. The Impact of an Attained English Competence on Subsequent Course Success. Journal of Applied Research in the Community College, 15 (1): 29 36. Vince Tinto. 2008. Student Success and the Building of Involving Educational Communities, in Promoting Student Success in College, http://soeweb.syr.edu/academics/grad/higher_education Orientation counseling clubs tutoring Keith Wurtz. 2008. A Methodology for Generating Placement Rules that Utilizes Logistic Regression. Journal of Applied Research in the Community College 16 (1): 52 58.

Student Demographics Delta College 30,111 students in 2007 08 58% female 11% African American 28% Hispanic/Latino 20% Asian/Pacific Islander Average age is 24.8 45% qualify for fee waivers (income guideline or child of disabled vet/deceased vet)

Student Demographics Delta College African Americans are underrepresented when examining AA degree attainment, transfer status, and completion of critical 4 courses (1. ENG 1A, 2. COMST 1A, 3. ENG 1B/1D/PHIL 30, and 4. Transfer MATH) Hispanics lag behind other groups on several measures (transfer success, degrees, critical 4 attainment)

Models of Student Success Multivariate Models Cohort Used: Fall 2007 Students Predictor Variables BACKGROUND ENGAGEMENT Age Number of Counseling Services Gender (1 = Female, 0 = male) Tutoring Hours (Math/Science) Ethnic Group (1 = White, 0 = Non White) DSPS Status (1 = DSPS) EOPS Status (1 = EOPS) Low income (1 = BOG fee waiver) NORMS/TRANSFER DIRECTED SKILLS Student Education Plan (1 = yes) Reading Assessment Level (1, 2, 3) Units Attempted Math Assessment Level (1, 2, 3) Prior course work in ENG 79/1A Math GPA (where relevant) (0 = No courses, 1 = success in ENG 79, 2 = success in ENG 1A, 3= success in both)

Models of Student Success Dependent Variables Success in Large Enrollment/Gateway Courses (Success defined as grade of A, B, or C) Psychology 1 History 17A Political Science 1 Math 82 (Transfer Algebra) Persistence to Spring 2008 term Fall 2007 Overall Success Rate (Success defined as Semester GPA >= 2.00) All dependent variables 1 or 0, with positive outcome = 1 Predicted success = a + bx1 + bx2 + bx3 e

Table 1 Predictors of Student Success in Introduction to Psychology (PSYCH 1) Using a Logistic Regression Model (Fall 2007) Prior success in English is the strongest predictor of success in PSYCH 1 Problem coefficients don t have same meaning as in OLS Regression Two tailed Odds When Odds When Change Variable Coefficient p value X is Low X is High in Odds Background Age at term.022.072 68.6 81.5 12.9 White Student.227.174 68.8 73.4 4.6 Female Student.069.645 69.2 70.7 1.5 DSPS Student.731.223 69.8 82.8 13.0 EOPS Student.226.445 69.7 74.3 4.6 BOG Fee Waiver.275.091 72.2 66.4 5.8 Skill Levels Math Level.315.002 ** 64.5 77.3 12.8 Reading Level.016.898 70.5 69.8 0.7 Norms/Seriousness Prior English Success.543.000 *** 58.7 87.9 29.2 Attempted Units.072.001 *** 52.0 74.8 22.8 Educational Plan.477.047 * 71.3 60.7 10.6 Engagement Counseling Services.067.022 * 67.7 85.1 17.4 Orientation Class.360.053 68.5 75.7 7.2 Constant 1.691.000 Hosmer Lemeshow Test 5.08.749 Nagelkerke R Square.171 % Correctly Classified 69.0% Reduced Error Measure 5.2% N 1,038 *** Significant at 99.9% confidence level, ** significant at 99% level, * significant at 95% level

Making it relevant Logit coefficients need to be converted to meaningful data Step 1 Set x to a particular value (example, prior English success = 0) Step 2 Calculate the equation z score using mean * coefficient for other variables Step 3 Compute the antilog of the equation result Step 4 Compute the odds of success by using the formula: antilog/(1+antilog) or EXP/(1+EXP) Holding all other variables constant, the result tells you the odds of success with no prior English success (ranges between 0.0 & 1.0)

Making it relevant Predicted Probability of Success in Psychology 1 and English Course Taking Patterns Predicted Odds of Success 100.0 80.0 60.0 40.0 20.0 0.0 58.7 No English 71.0 Completed Below Transfer English 80.8 Completed Transfer English 87.9 Two English Courses N = 1,038

Table 2 Predictors of Student Success in U.S. History (HIST 17A) Using a Logistic Regression Model (Fall 2007) Age and prior success in English are the strongest predictors of success in HIST 17A. Two tailed Odds When Odds When Change Variable Coefficient p value X is Low X is High in Odds Background Age at term.044.002 ** 38.4 71.9 33.5 White Student.277.071 39.6 46.4 6.8 Female Student.141.360 40.4 43.8 3.4 DSPS Student 1.323.070 42.0 73.1 31.1 EOPS Student.216.509 41.9 47.3 5.4 BOG Fee Waiver.269.129 44.4 37.9 6.5 Skill Levels Math Level.219.026 * 37.7 48.4 10.7 Reading Level.253.040 * 40.7 53.2 12.5 Norms/Seriousness Prior English Success.372.000 *** 31.4 58.3 26.9 Attempted Units.027.200 35.4 44.4 9.0 Educational Plan.268.207 43.4 37.0 6.4 Engagement Counseling Services.034.210 43.5 31.6 11.9 Orientation Class.284.160 41.1 48.1 7.0 Constant 2.871.000 Hosmer Lemeshow Test 3.81.874 Nagelkerke R Square.124 % Correctly Classified 64.4% Reduced Error Measure 24.6% N 841 *** Significant at 99.9% confidence level, ** significant at 99% level, * significant at 95% level

Making it relevant 100 Predicted Probability of Success in U.S. History and English Course Taking Patterns Predicted Odds of Success 80 60 40 20 31.4 39.9 49.1 58.3 0 No English Completed Below Transfer English Completed Transfer English Two English Courses N = 841

Table 3 Predictors of Student Success in U.S. Government (POLSC 1) Using a Logistic Regression Model (Fall 2007) Prior success in English is the strongest predictor of success in POLSC 1. Two tailed Odds When Odds When Change Variable Coefficient p value X is Low X is High in Odds Background Age at term.008.590 62.6 68.4 5.8 White Student.089.603 62.6 64.6 2.0 Female Student.117.451 61.8 64.6 2.8 DSPS Student 1.096.112 62.8 83.5 20.7 EOPS Student.005.987 63.3 63.2 0.1 BOG Fee Waiver.157.375 64.5 60.9 3.6 Skill Levels Math Level.297.006 ** 57.6 71.1 13.5 Reading Level.115.392 60.6 65.9 5.3 Norms/Seriousness Prior English Success.343.000 *** 50.3 73.9 23.6 Attempted Units.041.049 * 51.9 65.7 13.8 Educational Plan.106.619 63.8 61.3 2.5 Engagement Counseling Services.038.189 61.7 74.0 12.3 Orientation Class.639.006 ** 61.0 74.8 13.8 Constant 2.871.000 Hosmer Lemeshow Test 9.862.275 Nagelkerke R Square.115 % Correctly Classified 65.4% Reduced Error Measure 9.8% N 821 *** Significant at 99.9% confidence level, ** significant at 99% level, * significant at 95% level

Making it relevant Predicted Probability of Success in POLSC 1 and English Course Taking Patterns 100 Predicted Odds of Success 80 60 40 20 50.3 58.8 66.8 73.9 0 No English Completed Below Transfer English Completed Transfer English Two English Courses N = 821

Table 4 Predictors of Student Success in Intermediate Algebra (MATH 82) Using a Logistic Regression Model (Fall 2007) Prior success in Math is the strongest predictor of success in Intermed Algebra Note the impact of tutoring Two tailed Odds When Odds When Change Variable Coefficient p value X is Low X is High in Odds Background Age at term.025.039 * 48.0 67.3 19.3 White Student.168.348 54.6 50.4 4.2 Female Student.002.992 51.8 51.9 0.1 EOPS Student.492.273 52.3 40.2 12.1 BOG Fee Waiver.186.302 53.5 48.9 4.6 Skill Levels Math Level.026.809 51.0 52.3 1.3 Prior Math GPA.454.000 *** 27.9 70.4 42.5 Norms/Seriousness Attempted Units.026.247 44.7 53.8 9.1 Educational Plan.082.688 51.4 53.4 2.0 Engagement Tutoring Hours.018.068 51.4 66.1 14.7 Orientation Class.297.168 53.2 45.8 7.4 Constant Hosmer Lemeshow Test 10.04.262 Nagelkerke R Square.127 % Correctly Classified 63.3% Reduced Error Measure 24.3% N 701 *** Significant at 99.9% confidence level, ** significant at 99% level, * significant at 95% level

Making it relevant Predicted Probability of Success in Intermediate Algebra and Prior Success in Math Classes 100 Predicted Odds of Success 80 60 40 20 37.8 48.9 60.1 70.4 0 Prior Math GPA = 1.0 Prior Math GPA = 2.0 Prior Math GPA = 3.0 Prior Math GPA = 4.0 N = 701

Term to Term Persistence N = 11,060 students A number of variables helped explain persistence, including key indicators of engagement (i.e., counseling & orientation services) All other things being equal, the more counseling services received, the greater the likelihood of student persistence

Overall Term GPA (2.0 or higher) N = 11,060 students Holding all other variables constant, greater amounts of counseling produce greater odds of term GPA exceeding 2.0. The same applied for higher reading and math assessment levels, being a woman, being older, and taking more units.

Uses of the Data GE course success presented to Social Science faculty Response Transfer English advisory on all division GE courses Orientation data presented to counselors & matriculation committee Response Student services departments are exploring new modes of orientation to make it more universal

Uses of the Data Learning Center data presented to Title V Steering Committee, Learning Center Directors Data will be presented at HACU Conference in Fall 2009

Concluding Questions Matt Wetstein mwetstein@deltacollege.edu Enjoying a dinner in Bologna, 2006

PSYCH 1 STUDENTS FALL 2007 Z Score Z Score EXP/(1+EXP) EXP/(1+EXP) Change in Variable b Mean b*mean X Values Low High EXP Low EXP High Odds Low Odds High Odds age 0.022 21.40 0.471 18 v. 50 0.780 1.484 2.181 4.410 0.686 0.815 13.0% gender 0.069 0.64 0.044 0 v. 1 fem 0.811 0.880 2.250 2.410 0.692 0.707 1.5% ethnic 0.227 0.29 0.066 0 v. 1 white 0.789 1.016 2.201 2.762 0.688 0.734 4.7% reading 0.016 1.97 0.032 1 v. 3 0.870 0.838 2.387 2.312 0.705 0.698 0.7% math 0.315 1.82 0.573 1 v. 3 0.596 1.226 1.816 3.409 0.645 0.773 12.8% orientation 0.360 0.22 0.079 0 v. 1 yes 0.775 1.135 2.172 3.113 0.685 0.757 7.2% sep 0.477 0.12 0.057 0 v. 1 yes 0.912 0.435 2.489 1.545 0.713 0.607 10.6% eops 0.226 0.09 0.020 0 v. 1 yes 0.834 1.060 2.303 2.887 0.697 0.743 4.5% bog 0.275 0.37 0.102 0 v. 1 yes 0.956 0.681 2.602 1.977 0.722 0.664 5.8% dsps 0.731 0.02 0.015 0 v. 1 yes 0.840 1.571 2.317 4.812 0.698 0.828 12.9% counseling 0.067 1.74 0.117 0 v. 15 0.738 1.743 2.092 5.715 0.677 0.851 17.5% prior english 0.543 0.93 0.504 0 v. 3 0.351 1.980 1.420 7.242 0.587 0.879 29.2% units attmpt 0.072 11.78 0.848 1 v. 15 0.079 1.087 1.082 2.964 0.520 0.748 22.8% constant 1.691 1.00 1.691 zscore 0.855 Predicted Probability 0.894 2.444747 0.7097 No English 58.7 1.4369418 4.207808 0.80798 Completed Below 71.0 Completed Trans 80.8 Two English Cour 87.9 Predicted Probability of Success in Psychology 1 and English Course Taking Patterns Predicted Odds of Success 100.0 80.0 60.0 40.0 20.0 0.0 58.7 No English 71.0 Completed Below Transfer English 80.8 Completed Transfer English 87.9 Two English Courses