1 Defining and Measuring College Success Wayne Camara Krista Mattern Andrew Wiley The College Board Research and Development February 24, 2009
2 Remediation and Post Secondary Success 41.4% of post secondary students take at least one remedial class. This percent is even higher among African Americans (61.7%) and Hispanics (63.2%). For all post secondary students taking remedial classes, the graduation (or completion for certificate students) rates were between 30% and 57%, depending on the type and number of remediation courses taken. For those not taking remedial classes, the graduation rate was 69% For Bachelors candidates taking remedial classes, between 17% and 39% of students graduated as compared to 58% for those not taking remedial classes.
3 College Readiness Index Long term goal: To provide states, districts and schools with an aggregate data that estimates the percent of students who are college ready, or on track to be college ready, upon graduation from high school. It is not intended to make high stakes decisions at the student level. It will not be used for admissions or placement decisions. To provide diagnostic information that will identify steps necessary to increase the percentage of students ready for college.
4 The College Readiness Index will have 3 components HS Performance (GPA) Academic rigor index Index of college readiness National Benchmark (SAT Reasoning Test)
5 Creation of the College Readiness Index SAT Validity SAT was revised in March of 2005, so the first class entering college with the new SAT is the entering class of 2006. Tracked data from 100 colleges and approximately 150,000 students. Released data showing the predictive and differential validity results Upcoming research investigates the validity of the SAT for predicting individual course grades and retention to 2 nd year in college.
SAT Validity Study Correlations for Predictors (N=151,316) 6 Predictors R* HSGPA 0.54 SAT-CR + M + W 0.53 SAT-CR 0.48 SAT-M 0.47 SAT-W 0.51 SAT-CR + M 0.51 HSGPA + SAT 0.62 Correlations corrected for restriction of range Pooled within-institution corrections
SAT Validity Study Incremental Validity (N=151,316) Predictors R 1 R 2 ΔR HSGPA (Add SAT-CR + SAT-M) 0.54 0.61 0.07 HSGPA (Add SAT-CR + SAT-M + SAT-W) 0.54 0.62 0.08 SAT-CR + M (Add SAT-W) 0.51 0.53 0.02 HSGPA + SAT-CR + SAT-M (add SAT-W) 0.61 0.62 0.01 * Correlations corrected for restriction of range, pooled within-institution correlations 7
SAT Validity Study Incremental Validity (N=151,316) N SAT HSGPA SAT+HSGPA CONTROL Private 45,786 0.57 0.55 0.65 Public 105,530 0.52 0.53 0.61 SELECTIVE Under 50% 27,272 0.58 0.55 0.65 50-75% 84,433 0.53 0.54 0.62 >75% 39,611 0.51 0.54 0.60 * Correlations corrected for restriction of range, pooled within-institution correlations 8
Correlation of SAT scores and HSGPA with FYGPA by Ethnicity Subgroup 9 American Indian Asian Race/Ethnicity African- American Hispanic White k 16 82 83 86 109 n 384 14,109 10,096 10,486 104,017 SAT-CR 0.41 0.41 0.40 0.43 0.48 SAT-M 0.41 0.43 0.40 0.41 0.46 SAT-W 0.42 0.44 0.43 0.46 0.51 SAT 0.54 0.48 0.47 0.50 0.53 HSGPA 0.49 0.47 0.44 0.46 0.56 SAT, HSGPA 0.63 0.56 0.54 0.57 0.63 k = number of institutions
Subgroup Average Overprediction (-) and Underprediction (+) of FYGPA for SAT Scores and HSGPA by Ethnicity 10 American Indian Asian Race/Ethnicity African- American Hispanic White k 103 109 108 110 110 n 798 14,296 10,304 10,659 104,024 SAT-CR -0.26 0.05-0.30-0.17 0.04 SAT-M -0.25-0.07-0.26-0.16 0.05 SAT-W -0.22 0.04-0.26-0.16 0.04 SAT -0.22 0.01-0.20-0.11 0.03 HSGPA -0.25 0.02-0.32-0.27 0.06 SAT, HSGPA -0.20 0.02-0.17-0.12 0.03 k = number of institutions
11 Creation of the College Readiness Index Data has been collected from a sample of approximately 109 schools for the 2007 entering class. During registration, SAT students provide extensive high school coursework information as well as HSGPA. College data includes grade point average, course taking and retention. Students will be tracked until graduation. College performance data will be used to identify benchmarks linked to college success. Students will be considered college ready if they meet or exceed benchmarks in all three indicators. Future cohorts will also be used to validate the benchmarks.
12 College Readiness Reports Data on the percent of students deemed College Ready disaggregated by key demographic variables. Comparison of college readiness to a peer group (i.e. compare state to nation). Data on the percent of students attaining college readiness according to each indicator. Information on where students behavior could be altered to move more students towards college readiness.
13 Hypothetical Chart Showing Longitudinal data on College Readiness 5 y ear report on colleg e readiness for a School District 60 50 40 30 20 10 50.0 33.3 16.7 45.8 45.0 40.0 41.7 41.7 36.7 37.5 33.3 20.8 21.7 17.5 18.3 0 2009 2010 2011 2012 2013 Colleg e Ready Plus Colleg e Ready Not Colleg e Ready
14 College Readiness Reports Reports will be linked to other sources of information which will provide feedback on steps that can be followed to move more students towards college readiness. SAT Skills Insight Provides descriptive information on the knowledge, skills and abilities for students across SAT score bands. Developed by SMEs based upon a systematic review of items and student performance on each item.
15 SAT Skills Insight Writing Section Teacher Use Examples Writing Section
WRITING Vague Pronouns Although students use pronouns in their speaking and writing, some students may not be able to easily identify a pronoun or what is meant by a vague pronoun. This skill is tested in the SAT writing section. Students who score in the 300-390 band generally have this skill. 16 This sample SAT question, which is an Identifying Sentence Errors question type, illustrates the problem of vague pronouns.
17 Recognizing vague pronouns is also described in the 400-490 score band. The example illustrating this skill at this level is an Improving Sentences type: the student must find the correction rather than just spot the error. Related skills involving pronouns appear in score band 500-590 ( Recognize the antecedent of a pronoun despite multiple distracters that change the number and/or the subject of the referent within the sentence) and score band 200-290 ( Recognize simple pronoun references ). Look to the Suggestions for Improvement for ideas for reinforcement exercises - e.g., ask students to underline the pronouns and antecedents in the next writing assignment).
18 Academic Rigor Index Academic Rigor Index (0-25) Math (0-5) English (0-5) Science (0-5) Social Science (0-5) Foreign Language (0-5)
19 Proposed Academic Intensity Index (AII) Empirically based awards points for either coursework, honors, or AP. The Academic Intensity Index will have 5 sections: English, Math, Science, Social Science/History, and Foreign/Classical Language A preliminary version of an Academic Intensity Index has been created: College level data will be will be used to choose and validate a scale Thus far, preliminary scales have been created and tested using SAT scores as outcome data
20 AII Proposed Math scale Identify courses, by grade, that are associated with college success Each year is reviewed and the student is assigned a value of 1 if they have taken an appropriate class during that year. Class level values are summed to reach a scale of 0 to 5. If a student has taken Calculus, they are assigned a 5 automatically Some examples: Grade Level 8th 9th 10th 11th 12 th Algebra 2 1 1 1 0 0 Geometry 1 1 0 0 0 Trigonometry 1 1 1 1 0
21 Hypothetical Report for a School District: Student Summary for Math Academic Intensity Percent identified as: Asian students Black students Hispanic students White students College Ready 60% 42% 46% 55% Algebra by 10 th grade 85% 67% 69% 78% Trigonometry by 11 th grade 79% 59% 64% 74% Pre-Calculus by 12 th grade 74% 54% 59% 69% 1 Honors or AP class 45% 25% 30% 35% 2 or more Honors or AP class 22% 5% 9% 14%
22 AII Proposed English scale A student is awarded 1 point for having taken 3 years (excluding courses taken concurrently) of English (0/1) A student is awarded between 0 and 4 points depending on honors and AP participation: (0/4) No honors, no AP = 0 1 honors or dual enrollment course, no AP = 1 2 or more honors or dual enrollment course, no AP = 2 1 honors or dual enrollment course, 1 AP = 3 2 or more honors or dual enrollment course, 1 AP = 4 2 or more AP = 4
23 Coursework AII Proposed Science scale A student is awarded 1 point for having taken an 8 th grade science class (0/1) A student is awarded 1 point for having taken biology, chemistry, and physics (0/1) A student is awarded 1 point for having taken 4 years of science (one or more science course in 9 th, 10 th, 11 th, and 12 th grade) (0/1) AP participation 1 point for having taken any science honors or dual enrollment course, Earth Science AP or Biology AP 2 points for having taken a Chemistry or Physics AP (0/2)
24 AII Proposed Social Science scale A student is awarded 1 point for 3 or more years of Social Science (excluding courses taken concurrently) (0/1) A student is awarded 1 point for having 3 or more years of history (excluding courses taken concurrently) (0/1) Students are awarded points for honors, dual enrollment, and AP participation (0/3): 1 point for having 1 or more honors or dual enrollment classes but no AP 2 points for having 1 AP class 3 points for having 2 or more AP classes
AII Proposed Foreign Language Scale Number of Years of Foreign Language (0/3) A student is awarded 1 point for having taken 2 years of language A student is awarded 2 points (0/1) for having taken 3 years of language A student is awarded 3 points for having taken 4 or more years of language) A student is awarded up to 2 points depending on his or her AP, honors, or dual enrollment participation (0/2): 1 point for each class taken within language honors, dual enrollment course, or AP course 25
26 AII - Correlations Among the Sub-Scales 1 2 3 4 5 1. English -- 2. Mathematics 0.38 -- 3. Science 0.39 0.47 -- 4. Social Science 0.56 0.39 0.43 -- 5. Foreign/World Language 0.33 0.32 0.35 0.36 --
27 A preliminary version of AII Mean Scores by ethnicity Ethnicity Total Math English Natural Science Social Science Foreign Language Total group 12.5 2.7 2.3 2.5 2.8 2.2 American Indian 11.5 2.0 2.6 2.3 2.5 1.9 Asian, Asian-American, Pacific Islander 13.4 2.9 2.7 2.8 2.8 2.2 Black or African- American 10.7 1.9 2.4 2.2 2.4 1.9 Hispanic 11.6 2.1 2.6 2.3 2.6 2.1 White 13.0 2.4 2.8 2.6 2.9 2.3 Other 12.5 2.3 2.6 2.6 2.7 2.2
28 A preliminary version of AII Mean scores by family income Income Total Math English Science Social Science Foreign Language less than 30k 11.0 2.0 2.4 2.3 2.4 1.9 30k to 50k 11.8 2.1 2.6 2.4 2.6 2.0 50k to 70k 12.3 2.2 2.7 2.5 2.7 2.1 70k to 100k 12.7 2.4 2.8 2.6 2.8 2.2 more than 100k 13.4 2.5 2.9 2.7 3.0 2.4
29 A preliminary version of AII Mean scores by family income and ethnicity Asian students Black students Hispanic students White students < $30K 12.2 10.0 10.9 11.5 > $100K 14.5 11.9 12.9 13.5
30 Moving Forward for the CRI Data verification with future cohorts of students. Other definitions of college success. Link to PSAT/NMSQT
31 Noncognitive Assessments: Expanding the definition of college readiness
32 Background Choosing students: Higher education admissions tools for the 21 st century (Camara & Kimmel, 2005) Purpose: Identify additional predictors of college success Expand the definition of what constitutes successful performance in college beyond freshman GPA College Board has initiated several projects to address this research area
33 Research collaboration with Michigan State University Identify a broader domain of college student performance: Review university mission statements and department objectives Interview with university staff responsible for student life at Michigan State University Review of the education literature on student outcomes Our systematic search resulted in 12 dimensions of student performance
34 12 Dimensions of Student Performance Broadening the Performance Domain in the Prediction of Academic Success (Schmitt, Oswald, & Gillespie, 2004) 1. Knowledge, learning, mastery of general principles 2. Continuous learning, intellectual interest and curiosity 3. Artistic and cultural appreciation 4. Multicultural appreciation 5. Leadership 6. Interpersonal skills 7. Social responsibility, citizenship and involvement 8. Physical and psychological health 9. Career orientation 10. Adaptability and life skills 11. Perseverance 12. Ethics and integrity
35 Two Noncognitive Measures Situational judgment inventory A situation is presented along with several alternative courses of action. The respondent is asked to indicate what she/he would be most likely and least likely to do. Biodata Short, multiple choice reports of past experience/background and interests/preferences.
36 Study 1: Psychometric adequacy & scale refinement 644 MSU freshmen completed one of the two parallel forms of the biodata and SJI instruments at the beginning of the academic year. Identical empirical-keying procedures were conducted on both instruments at the item level (double-cross validated using randomly split samples). Results indicated significant incremental validity for some of the scales above and beyond the validity of SAT/ACT scores and existing measures of personality in predicting college GPA. The biodata and SJI demonstrated the greatest incremental validity when absenteeism, students self ratings, and peer-ratings of performance were examined (.19,.22, and.14, respectively).
37 Study 1: Standardized Differences Compared with White group Non-cognitive Dimension Black Hispanic Asian Knowledge -0.08-0.20-0.25 Learning 0.01 0.63* -0.19 Artistic -0.19 0.73* 0.15 Multicultural -0.11 0.63* 0.02 Leadership -0.18 0.08-0.30 Interpersonal -0.18 0.33-0.38* Positive values indicate that minorities perform better than White students. The d values for biodata and SJI measures across ethnic and gender subgroups were consistently smaller than those found on cognitive predictors. * p <.05 SJI composite -0.05-0.14-0.21 Citizenship 0.05 0.23-0.14 Health -0.31* 0.06-0.67* Career 0.34* 0.56* 0.14 Adaptability 0.03 0.09-0.41* Perseverance 0.13 0.55* -0.18 Ethics 0.17-0.06-0.13
Study 2: Predicting FYGPA: Total Sample across 10 Institutions (N = 2443) 38
Predicting Self-Rated Performance: Total Sample across 10 Institutions (N = 900) 39
Predicting Class Absenteeism: Total Sample across 10 Institutions (N = 899) 40
Representative Subgroup Differences in Standardized Units 41
42 Percent of Students Selected: Two Composites and Three Selection Strategies Top 85% Top 50% Top 15% Group AB AB+ AB AB+ AB AB+ Hispanic 4.4 4.6 4.1 4.9 3.9 5.5 (+.2) (+.8) (+1.6) Asian 7.6 7.7 9.9 9.5 17.5 12.9 (+.1) (-.4) (-4.6) African-American 17.9 19.8 9.6 13.6 1.3 7.2 (+1.9) (+4.0) (+5.9) White 70.2 67.9 76.4 71.9 77.2 74.4 (-2.3) (-4.5) (-2.8) AB = equally weighted composite of HSGPA and SAT/ACT. AB+ = equally weighted composite of HSGPA, SAT/ACT, Biodata, and SJI.
43 Limitations & Future Research Public relations and acceptance of these measures by consumers (i.e., admissions officers, parents, students). Need to collection reactions to new admissions measures along a variety of dimensions (e.g., fairness, face validity). Fakability in high-stakes situation especially relevant for biodata, less so for SJI. However, note that essays can be coached and edited, and self-reported activities can also be inflated. More research and evaluation efforts need to be conducted when these measures are used operationally in college settings.
44 Study 3: Purpose & Research Questions Purpose: evaluating the utility of the biodata and situational judgment measures in as close to a real admissions situation as is possible Administer new measures to college applicants rather than college freshmen. On an annual basis, collect class absenteeism, self rated performance of the noncognitve dimensions, and commitment to the university from enrolled students; institutions will provide course grades and retention information. Research Questions: The incremental validity of the biodata and the situational judgment measures will be assessed after controlling for high school GPA and SAT/ACT scores. Differential prediction will also be assessed to see whether each measure-outcome relationship differs across various subgroups (e.g., gender and race). The relationship between scores on these noncognitive measures and holistic file review will be examined to test whether these measures could be substituted for the more subjective file review.
45 Thank You Thanks to ATP and Thanks to you
46 Questions, Comments, Suggestions Researchers are encouraged to freely express their professional judgment. Therefore, points of view or opinions stated in College Board presentations do not necessarily represent official College Board position or policy. Please forward any questions, comments, and suggestions to: Andrew Wiley at: Awiley@collegeboard.org