Segmentation Modeling or Classification and Regression Trees (CART)

Transcription

1 Segmentation Modeling or Classification and Regression Trees (CART) Presented by Keith Wurtz Senior Research Analyst Chaffey Community College

2 Examples of Segmentation Modeling to Identify Characteristics of a Specific Population for Target Marketing Identifying Characteristics of Populations to help with Enrollment Management Decisions Comparison between Logistic Regression and Segmentation Modeling

3 Characteristics of a Specific Population for Target Marketing Participation Rates of Chaffey Students by Age Age Years Years Years Years Years Years Years Total # 4,018 6,066 2,395 1,752 1,605 2,304 1,071 19,211 N 21,968 50,091 50,686 57,004 64, ,749 78, ,446 % Note. # refers to the number of students attending Chaffey in the academic year. N refers to the population living in the Chaffey College District taken from the 2000 US Census.

4 Marketing to Year Olds US Census Data allows us to identify the number of year olds living in each block group We can use the mapping software to identify where year olds live Once we know where they live, we can use segmentation modeling (i.e. answer tree or classification tree) to identify enrollment characteristics of these students and then market to them

5 Segmentation Modeling Based on the principle of binary recursive partitioning Where the dependent variable (i.e. success and nonsuccess) are examined for all possible splits of the data at each step of the tree-building process to find the split that most effectively separates the dependent variable into homogeneous groups until it is not possible to continue The model attempts to maximize the number of students who are correctly classified as successes and those who are correctly classified as nonsuccesses. Very similar to logistic regression Benefit of segmentation modeling Allows the identification of groups within groups Example: Hispanic male students years of age who take evening courses prior to earning a grade on record

6 Enrollment Variables used in Segmentation Model Used MIS to identify enrollment characteristics Transfer course enrollment Basic skills course enrollment Occupational course enrollment Credit course enrollment School Location of course Created field for each one that generated number of enrollments aggregated by student

7 Node 8 MORE likely to not enroll in a PE Course, MORE likely to not enroll in a library course, MORE likely to enroll in credit course Age Dichotomous year olds and other Node 0 All other ages year olds Total (100.00) Number of Enrollments in PE Courses Adj. P-value=0.0000, Chi-square= , df=1 <=Did Not Enroll Node 1 All other ages year olds Total (54.93) Number of Enrollments in LIB Courses Adj. P-value=0.0000, Chi-square= , df=1 >Did Not Enroll Node 2 All other ages year olds Total (45.07) Number of Enrollments in HS Courses Adj. P-value=0.0000, Chi-square= , df=1 <=Did Not Enroll >Did Not Enroll <=Did Not Enroll >Did Not Enroll Node 3 All other ages year olds Total (36.18) 9594 Node 4 All other ages year olds Total (18.75) 4973 Node 5 All other ages year olds Total (25.90) 6868 Node 6 All other ages year olds Total (19.18) 5086 Number of Enrollments in Credit Courses Adj. P-value=0.0000, Chi-square= , df=1 Number of Enrollments in SU00 Adj. P-value=0.0000, Chi-square= , df=1 Number of Enrollments in SSS Courses Adj. P-value=0.0000, Chi-square= , df=1 Number of Enrollments at CCFC Adj. P-value=0.0000, Chi-square= , df=1 <=Did Not Enroll >Did Not Enroll <=Did Not Enroll >Did Not Enroll <=Did Not Enroll >Did Not Enroll <=Did Not Enroll >Did Not Enroll Node 7 All other ages year olds Total (17.61) 4671 Node 8 All other ages year olds Total (18.56) 4923 Node 9 All other ages year olds Total (8.15) 2161 Node 10 All other ages year olds Total (10.60) 2812 Node 11 All other ages year olds Total (18.63) 4940 Node 12 All other ages year olds Total (7.27) 1928 Node 13 All other ages year olds Total (0.93) 247 Node 14 All other ages year olds Total (18.25) 4839

8 Segmentation Modeling Results Nodes n % Gain: n Gain (%) Resp: % Index (%) 8 MORE likely to not enroll in a PE Course, MORE likely to not enroll in a library course, MORE likely to enroll in credit course 4, , MORE likely to not enroll in a PE Course, MORE likely to not enroll in a library course, LESS likely to enroll in credit course 4, MORE likely to not enroll in a PE Course, LESS likely to enroll in a library course, MORE likely to enroll in Summer 2, LESS likely to enroll in a PE Course, MORE likely to enroll in a HS course, Less likely to not enroll in SSS course 4, LESS likely to enroll in PE Course, LESS likely to enroll in HS course, MORE likely to not enroll at Fontana MORE likely to not enroll in a PE Course, LESS likely to enroll in a library course, LESS likely to not enroll in Summer 2, LESS likely to enroll in a PE Course, MORE likely to enroll in a HS course, Less likely to enroll in SSS course 1, LESS likely to enroll in PE Course, LESS likely to enroll in HS course, MORE likely to enroll at Fontana 4, Note. N is the number of all cases in the node. % is the percent of all cases in the node. Gain:n is the number of all cases with the target response (i.e year olds). Gain:% is the percent of all cases (e.g.: 1,155/3,044=37.9) with the target response. Resp:% represents the proportion of cases in the node that have the target response (e.g.:1,155/4,923=23.5%). Index(%) gives a measure of how the number of target responses in the node compares to that for the entire sample (e.g.: 37.9%/18.6%=204.4%).

9 Using Information to Develop Marketing Plan Now that we know that year olds prefer the following types of courses MORE likely to not enroll in a PE Course, MORE likely to not enroll in a library course, MORE likely to enroll in credit course MORE likely to not enroll in a PE Course, MORE likely to not enroll in a library course, LESS likely to enroll in credit course We can go back to SPSS and use the Select IF command to identify which courses that meet this criteria

10 Courses Preferred by Year Olds Of the 8,849 enrollments that met the previously stated criteria 13% or 1,127 of these enrollments were in Computer Information Systems courses 310 of these enrolments were in CIS-1 (Introduction to Computer Information) 116 were in CIS-68I (Using the Internet) 91 were in CIS-404 (Fundamentals of Microsoft Windows) 11% or 937 of these enrollments were in Disabilities Programs and Services courses Most of these enrollments were in the independent living courses 8% or 708 of these enrollments were in Business and Office Technologies courses 120 of these were in BUSOT-40A (Beginning Computer Keyboarding) 99 were in BUSOT-46A (Beginning Microsoft Word) 7% or 620 of these enrollments were math courses 190 of these were in MATH-410 (Elementary Algebra) 99 were in MATH-420 (Intermediate Algebra) 83 were in MATH-25 (College Algebra) 72 were in MATH-520 (Arithmetic and Preparation for Algebra) 64 were in MATH-510 (Arithmetic) 4% or 347 of these enrollments were in Child Development Education courses 39 of these were in CDE-4 (Child, Family, and Community) These enrollments were very spread out in mostly transferable courses 3% or 288 of these enrollments were in ESL courses

11 HAVEN AVE 2000 US Census Population Data in the Chaffey College District San Antonio Heights STATE HWY 30 I 15 Upland STATE HWY 83 Rancho Cucamonga HAVEN AVE BASE LINE ST STATE HWY 66 W BASE LINE RD Fontana Montclair I 10 STATE HWY 60 Ontario Chino Hills Chino STATE HWY 71 Number of Year Olds tgr06071grp00.all40t Prepared by Keith Wurtz Date: Miles

12 HAVEN AVE Areas in Chaffey's District with High Concentration of Year Olds San Antonio Heights [` Main Campus STATE HWY 30 I 15 Upland Rancho Cucamonga HAVEN AVE BASE LINE ST STATE HWY 66 W BASE LINE RD [` Fontana Fontana Center STATE HWY 83 Montclair [` Ontario Center I 10 Chino Hills [` Chino STATE HWY 60 Chino Center [_ STATE HWY 71 Chino Campus Ontario Number of Year Olds [` [_ Year Olds Chaffey Locations Chino Campus High Concentration of Year Olds Prepared by Keith Wurtz Date: Miles

13 Identifying Characteristics of Populations to help with Enrollment Management Decisions Institutional Research was asked by the Vice President of Instruction to engage in exploratory research to identify what we know about existing populations that lose the connection with Chaffey College Brainstorming led to the design of six research studies: 1) Enroll in courses but withdraw prior to earning a grade on record 2) Complete a Chaffey College admissions application form but do not enroll in courses 3) Enroll in the Fall semester but do not re-enroll (i.e., persist) the following Spring semester 4) Participate in various combinations of Matriculation services (assessment, orientation, and counseling) 5) Utilize Success Center resources 6) Complete the Chaffey College assessment process but do not enroll in courses As a result, these six studies focused on identifying the unique characteristics of among students who do and do not engage in the aforementioned behaviors.

14 Enrollment Management Study Matrix Study #1 Withdrew Prior to Earning a Grade on Record Study #2 & 2A Applied But Did Not Enroll or earn a GOR Study #3 Did Not Persist (Fall-to-Spring) Study #4 Utilization of Matriculation Services Study #5 Utilized Success Center Study #6 Assessed But Did Not Enroll Data Sources Colleague, SARS Accuplacer Colleague, SARS, Accuplacer MIS, SARS, Accuplacer MIS, SARS, Accuplacer Positive Attendance Colleague, Accuplacer Dependent Variable: Earned GOR or Didn t Applied: Did or Did Not Enroll, & GOR or No GOR Persisted or Didn t Persist None, Some, All Accessed or Didn t Access Assessed: Did or Did Not Enroll Independent Variables: Gender Gender Gender Gender Gender Gender Age Range Age Range Age Range Age Range Age Range Age Range All Studies: Ethnicity Ethnicity Ethnicity Ethnicity Ethnicity Ethnicity Educational Goal Educational Goal Educational Goal Educational Goal Educational Goal Educational Goal Residency Residency Residency Residency Residency Residency ENGL Placement Levels Assessed (Y/N) Assessed (Y/N) Assessed (Y/N) Assessed (Y/N) Math Placement Levels Counseled (Y/N) Counseled (Y/N) Counseled (Y/N) Counseled (Y/N) Reading Placement Levels Basic Skills Course Application Admit Basic Skills Course Basic Skills Course Basic Skills Course Credit Course Application Interest in Student Services Credit Course Credit Course Credit Course Select Study Transfer Course Location Applied Transfer Course Transfer Course Transfer Course Occupational Course In District High School (Y/N) Occupational Course Occupational Course Occupational Course Course Meeting Time Course Meeting Time Course Meeting Time Course Meeting Time Full-Time/Part-Time Full-Time/Part- Time Full-Time/Part-Time Full-Time/Part-Time Academic Level Academic Level

15 Study #3: Students Who Did Not Persist From Fall to Spring No Yes Persisted? Node 0 No Yes Total Enrollment Adj. P-value=0.000, Chi-square= , df=3 First-Time Transfer Student; Returning Student; K-12 Student Continuing Student First-Time Student Unknown Node 1 No Yes Total Node 2 No Yes Total Node 3 No Yes Total Node 4 No Yes Total Counselled Within Semester? Adj. P-value=0.000, Chi-square= , df=1 Counselled Within Semester? Adj. P-value=0.000, Chi-square= , df=1 Age Range Adj. P-value=0.000, Chi-square= , df=1 No Yes No Yes 50 or Older; 20 to 24; 35 to 39; 40 to 49; 30 to 34; 25 to or Younger; Unknown Node 5 No Yes Total Node 6 No Yes Total Node 7 No Yes Total Node 8 No Yes Total Node 9 No Yes Total Node 10 No Yes Total Age Range Adj. P-value=0.000, Chi-square= , df=2 Age Range Adj. P-value=0.000, Chi-square= , df=2 Assessed Adj. P-value=0.000, Chi-square=26.594, df=1 Counselled Within Semester? Adj. P-value=0.000, Chi-square=53.650, df=1 50 or Older; 35 to 39; 40 to 49; 30 to 34; 25 to to or Younger; Unknown 50 or Older; 30 to 34; 25 to 29; Unknown 20 to 24; 35 to 39; 40 to or Younger No Yes No Yes Node 11 No Yes Total Node 12 No Yes Total Node 13 No Yes Total Node 14 No Yes Total Node 15 No Yes Total Node 16 No Yes Total Node 17 No Yes Total Node 18 No Yes Total Node 19 No Yes Total Node 20 No Yes Total

16 The Difference between Logistic Regression and Segmentation Modeling Research was conducted to identify students who are most likely to experience academic probation The purpose was to generate a placement recommendation to students most likely to experience academic probation in order to help lower the number of students experiencing probation Important Notice: Smart Start Program Your test results indicate that you could benefit from participating in the Smart Start program for new first time college students. Please see the testing attendant immediately for additional information on how to complete your COUNSELING and ORIENTATION through the Smart Start program.

17 The Difference between Logistic Regression and Segmentation Modeling Approximately 30 predictor variables were identified to help predict academic probation for students who took the assessment: Placement recommendations in reading, math, and English Test scores in reading, math, and English Educational Background Measures (e.g.: selfreported high school GPA,) In order to check the accuracy of the results both logistic regression and segmentation modeling were used

18 Logistic Regression Results Table 1 Logistic Regression Analysis of Probation as a Function of Educational Background Measures and Test Scores Predictor β Wald Χ² p Odd Ratio 95% CI for Odds Ratio High School GPA is 2.9 or lower *** years or less since last math class *** Constant Test Χ² df p Overall Model Evaluation Wald test 1, *** Goodness-of-fit-test Hosmer & Lemeshow *p <.05, **p <.01, ***p <.001 Note. The names of the predictor variables refer to the coding of 1. For instance, five years since last math class was coded as 1 and every other option was coded as 0.

19 ProbationSTUDYFLAG Segmentation Model Node 0 Not on Probation from FA Total hschgpa Adj. P-value=0.000, Chi-square= , df=3 B to A- ( ) C to B- ( ); C- to C ( ); D to C- ( ) B- to B ( ); Below a D (0-0.9); <missing> A- to A ( ) Of the 18,891 students who assessed from Fall 2003 to Spring 2007, 39% of the students experienced academic probation at least once. If the student had a self-reported high school GPA of 1.0 to 2.4 their probability of experiencing academic probation increased to 50% (n = 7,075) If the student had a self-reported HS GPA from and had taken a math course in the last two years their probability of experiencing academic probation increased to 61% (n = 4,077) If the student had a self-reported HS GPA from 1-2.4, had taken a math course in the last two years, and earned a D in their last English course their probability of experiencing academic probation increases to 72% Node 1 Not on Probation from FA Total or more years Node 9 Not on Probation from FA Total A Node 30 Not on Probation from FA03- Total to 5 years Node 10 Not on Probation from FA Total Node 2 Not on Probation from FA Total lstmthyr Adj. P-value=0.000, Chi-square= , df=3 C; F Node 31 Not on Probation from FA Total to 2 years; Less than 1 year; <missing> Node 11 Not on Probation from FA Total lstengl Adj. P-value=0.000, Chi-square=62.707, df=3 Node 3 Not on Probation from FA Total B Node 32 Not on Probation from FA Total Currently enrolled in a math course Node 12 Not on Probation from FA03- Total <= 84.5 Node 75 Not on Probation from FA03- Total Node 4 Not on Probation from FA Total Node 33 Not on Probation from FA03- Total read Adj. P-value=0.008, Chi-square=14.451, df=1 D > 84.5; <missing> Node 76 Not on Probation from FA03- Total

20 Logistic Regression Segmentation Modeling Student is 3 times more likely to experience probation if self-reported HS GPA is 2.9 or lower Student is 5 times more likely to experience probation if it has been 5 years or less since last math class If the student had a selfreported high school GPA of 1.0 to 2.4 their probability of experiencing academic probation increased from 39% to 50% (n = 7,075) If the student had a selfreported HS GPA from and had taken a math course in the last two years their probability of experiencing academic probation increased to 61% (n = 4,077) If the student had a selfreported HS GPA from 1-2.4, had taken a math course in the last two years, and earned a D in their last English course their probability of experiencing academic probation increases to 72%

21 SPSS Classification Trees 16.0 for Windows Current Cost = $899.00

22 References Borges, Guilherme and Cherpitel, Cheryl. (2001). Selection of screening items for alcohol abuse dependence among Mexican and Mexican Americans in the emergency department. Journal of Studies on Alcohol, 62, 277-.