Multivariate Models of Student Success

Transcription

1 Multivariate Models of Student Success RP Group/CISOA Conference April 28, 2009 Granlibakken, Lake Tahoe Dr. Matt Wetstein Dean of Planning, Research and Institutional Effectiveness San Joaquin Delta College Alyssa Nguyen Bri Hays Research Analyst Research Analyst

2 Introduction Many studies of student success rely on quasi experimental designs Intervention is tested for its effect, usually with a control group Downside lack of multiple control variables More and more, researchers are turning to multivariate logistic regression models

3 Introduction Logistic Regression s appeal Many of our dependent variables of interest are well suited for dichotomous analysis Techniques have become standard in packages like SAS, STATA, SPSS Allows for multivariate analysis and more holistic understanding of student behavior

4 Introduction RP Group researchers are leading the way in recent years some examples Wurtz (2008) Logit model for generating placement test recommendations Spurling (2007) Logit model of prior English enrollment on GE course success Younglove (2009) Logit model to recommend concurrent course enrollment for basic skills students CSS (2002) Logit model to validate prerequisites for enrollment in nursing programs

5 Introduction Some notes on Logit S shaped curve Should be little collinearity among independent variables Goodness of Fit Reliable models have non significant Chi Square values using the Hosmer Lemeshow goodness of fit test Model s ability to correctly classify cases vs. modal guessing strategy My prior use of Logit explaining judicial voting behavior in the U.S. & Canadian Supreme Court

6 Model of Student Success Current interest: developing multivariate models to examine patterns of student success Background traits (SES, ethnicity, income) Skill levels Norms & attitudes toward college & transfer Engagement in college life & services

7 Model of Student Success Assessment scores GPA history Student s Background Characteristics Course taking patterns & unit loads George Kuh et. al Student Success in College: Creating Conditions that Matter. San Francisco: Jossey Bass. Colleen Moore & Nancy Shulock Beyond the Open Door: Increasing Student Success in the California Community Colleges. Sacramento: CSU Sacramento Steve Spurling The Impact of an Attained English Competence on Subsequent Course Success. Journal of Applied Research in the Community College, 15 (1): Vince Tinto Student Success and the Building of Involving Educational Communities, in Promoting Student Success in College, Orientation counseling clubs tutoring Keith Wurtz A Methodology for Generating Placement Rules that Utilizes Logistic Regression. Journal of Applied Research in the Community College 16 (1):

8 Student Demographics Delta College 30,111 students in % female 11% African American 28% Hispanic/Latino 20% Asian/Pacific Islander Average age is % qualify for fee waivers (income guideline or child of disabled vet/deceased vet)

9 Student Demographics Delta College African Americans are underrepresented when examining AA degree attainment, transfer status, and completion of critical 4 courses (1. ENG 1A, 2. COMST 1A, 3. ENG 1B/1D/PHIL 30, and 4. Transfer MATH) Hispanics lag behind other groups on several measures (transfer success, degrees, critical 4 attainment)

10 Models of Student Success Multivariate Models Cohort Used: Fall 2007 Students Predictor Variables BACKGROUND ENGAGEMENT Age Number of Counseling Services Gender (1 = Female, 0 = male) Tutoring Hours (Math/Science) Ethnic Group (1 = White, 0 = Non White) DSPS Status (1 = DSPS) EOPS Status (1 = EOPS) Low income (1 = BOG fee waiver) NORMS/TRANSFER DIRECTED SKILLS Student Education Plan (1 = yes) Reading Assessment Level (1, 2, 3) Units Attempted Math Assessment Level (1, 2, 3) Prior course work in ENG 79/1A Math GPA (where relevant) (0 = No courses, 1 = success in ENG 79, 2 = success in ENG 1A, 3= success in both)

11 Models of Student Success Dependent Variables Success in Large Enrollment/Gateway Courses (Success defined as grade of A, B, or C) Psychology 1 History 17A Political Science 1 Math 82 (Transfer Algebra) Persistence to Spring 2008 term Fall 2007 Overall Success Rate (Success defined as Semester GPA >= 2.00) All dependent variables 1 or 0, with positive outcome = 1 Predicted success = a + bx1 + bx2 + bx3 e

12 Table 1 Predictors of Student Success in Introduction to Psychology (PSYCH 1) Using a Logistic Regression Model (Fall 2007) Prior success in English is the strongest predictor of success in PSYCH 1 Problem coefficients don t have same meaning as in OLS Regression Two tailed Odds When Odds When Change Variable Coefficient p value X is Low X is High in Odds Background Age at term White Student Female Student DSPS Student EOPS Student BOG Fee Waiver Skill Levels Math Level ** Reading Level Norms/Seriousness Prior English Success *** Attempted Units *** Educational Plan * Engagement Counseling Services * Orientation Class Constant Hosmer Lemeshow Test Nagelkerke R Square.171 % Correctly Classified 69.0% Reduced Error Measure 5.2% N 1,038 *** Significant at 99.9% confidence level, ** significant at 99% level, * significant at 95% level

13 Making it relevant Logit coefficients need to be converted to meaningful data Step 1 Set x to a particular value (example, prior English success = 0) Step 2 Calculate the equation z score using mean * coefficient for other variables Step 3 Compute the antilog of the equation result Step 4 Compute the odds of success by using the formula: antilog/(1+antilog) or EXP/(1+EXP) Holding all other variables constant, the result tells you the odds of success with no prior English success (ranges between 0.0 & 1.0)

14 Making it relevant Predicted Probability of Success in Psychology 1 and English Course Taking Patterns Predicted Odds of Success No English 71.0 Completed Below Transfer English 80.8 Completed Transfer English 87.9 Two English Courses N = 1,038

15 Table 2 Predictors of Student Success in U.S. History (HIST 17A) Using a Logistic Regression Model (Fall 2007) Age and prior success in English are the strongest predictors of success in HIST 17A. Two tailed Odds When Odds When Change Variable Coefficient p value X is Low X is High in Odds Background Age at term ** White Student Female Student DSPS Student EOPS Student BOG Fee Waiver Skill Levels Math Level * Reading Level * Norms/Seriousness Prior English Success *** Attempted Units Educational Plan Engagement Counseling Services Orientation Class Constant Hosmer Lemeshow Test Nagelkerke R Square.124 % Correctly Classified 64.4% Reduced Error Measure 24.6% N 841 *** Significant at 99.9% confidence level, ** significant at 99% level, * significant at 95% level

16 Making it relevant 100 Predicted Probability of Success in U.S. History and English Course Taking Patterns Predicted Odds of Success No English Completed Below Transfer English Completed Transfer English Two English Courses N = 841

17 Table 3 Predictors of Student Success in U.S. Government (POLSC 1) Using a Logistic Regression Model (Fall 2007) Prior success in English is the strongest predictor of success in POLSC 1. Two tailed Odds When Odds When Change Variable Coefficient p value X is Low X is High in Odds Background Age at term White Student Female Student DSPS Student EOPS Student BOG Fee Waiver Skill Levels Math Level ** Reading Level Norms/Seriousness Prior English Success *** Attempted Units * Educational Plan Engagement Counseling Services Orientation Class ** Constant Hosmer Lemeshow Test Nagelkerke R Square.115 % Correctly Classified 65.4% Reduced Error Measure 9.8% N 821 *** Significant at 99.9% confidence level, ** significant at 99% level, * significant at 95% level

18 Making it relevant Predicted Probability of Success in POLSC 1 and English Course Taking Patterns 100 Predicted Odds of Success No English Completed Below Transfer English Completed Transfer English Two English Courses N = 821

19 Table 4 Predictors of Student Success in Intermediate Algebra (MATH 82) Using a Logistic Regression Model (Fall 2007) Prior success in Math is the strongest predictor of success in Intermed Algebra Note the impact of tutoring Two tailed Odds When Odds When Change Variable Coefficient p value X is Low X is High in Odds Background Age at term * White Student Female Student EOPS Student BOG Fee Waiver Skill Levels Math Level Prior Math GPA *** Norms/Seriousness Attempted Units Educational Plan Engagement Tutoring Hours Orientation Class Constant Hosmer Lemeshow Test Nagelkerke R Square.127 % Correctly Classified 63.3% Reduced Error Measure 24.3% N 701 *** Significant at 99.9% confidence level, ** significant at 99% level, * significant at 95% level

20 Making it relevant Predicted Probability of Success in Intermediate Algebra and Prior Success in Math Classes 100 Predicted Odds of Success Prior Math GPA = 1.0 Prior Math GPA = 2.0 Prior Math GPA = 3.0 Prior Math GPA = 4.0 N = 701

21 Term to Term Persistence N = 11,060 students A number of variables helped explain persistence, including key indicators of engagement (i.e., counseling & orientation services) All other things being equal, the more counseling services received, the greater the likelihood of student persistence

22 Overall Term GPA (2.0 or higher) N = 11,060 students Holding all other variables constant, greater amounts of counseling produce greater odds of term GPA exceeding 2.0. The same applied for higher reading and math assessment levels, being a woman, being older, and taking more units.

23 Uses of the Data GE course success presented to Social Science faculty Response Transfer English advisory on all division GE courses Orientation data presented to counselors & matriculation committee Response Student services departments are exploring new modes of orientation to make it more universal

24 Uses of the Data Learning Center data presented to Title V Steering Committee, Learning Center Directors Data will be presented at HACU Conference in Fall 2009

25 Concluding Questions Matt Wetstein Enjoying a dinner in Bologna, 2006

26 PSYCH 1 STUDENTS FALL 2007 Z Score Z Score EXP/(1+EXP) EXP/(1+EXP) Change in Variable b Mean b*mean X Values Low High EXP Low EXP High Odds Low Odds High Odds age v % gender v. 1 fem % ethnic v. 1 white % reading v % math v % orientation v. 1 yes % sep v. 1 yes % eops v. 1 yes % bog v. 1 yes % dsps v. 1 yes % counseling v % prior english v % units attmpt v % constant zscore Predicted Probability No English Completed Below 71.0 Completed Trans 80.8 Two English Cour 87.9 Predicted Probability of Success in Psychology 1 and English Course Taking Patterns Predicted Odds of Success No English 71.0 Completed Below Transfer English 80.8 Completed Transfer English 87.9 Two English Courses