COMPARISON OF SEO SCHOLARS AND

Transcription

1 POLICY STUDIES ASSOCIATES, INC. POLICY STUDIES ASSOCIATES, INC. POLICY STUDIES ASSOCIATES, INC. COMPARISON OF SEO SCHOLARS AND NATIONAL TEST-TAKERS SAT SCORES, Tamar Hellman Stephen Coleman April 2014 Policy Studies Associates, Inc Connecticut Avenue, NW, suite 400 Washington, DC (202) Prepared for: Sponsors for Educational Opportunity (SEO) New York, New York

2

3 Introduction Sponsors for Educational Opportunity (SEO) conducts the SEO Scholars program, which provides selected public high school students with programmatic support that builds academic and social skills needed to get to and through college. Features of the high school portion of the eight-year program include weekly after-school classes, a Saturday Academy that meets three times a month, classes that meet during school breaks, a month-long summer term, and ongoing mentoring. SEO Scholars uses an academic curriculum that supplements a participant s regular high school education and that is in alignment with common core state standards. The program s instruction is delivered by teachers from local colleges and universities and from local public and private high schools to participants referred to as SEO Scholars. As described by the program materials, the majority of the curriculum is aimed at improving SEO Scholars critical reading, writing, vocabulary, grammar, and math skills. The program also aims to enhance SEO Scholars leadership skills, deliver information on colleges and on the college experience, and offer opportunities for participants to explore varied careers. A previous Policy Studies Associates, Inc. (PSA) evaluation of the educational impact of the SEO Scholars program found that participants outperformed their local counterparts on various educational outcomes (Coleman et al., 2012). Given the persistent achievement gap between black and Hispanic students and white students on the SAT (Hedges and Nowell, 1998), SEO was interested in seeing how SEO Scholars, who are predominately black and Hispanic, compared to all college-bound seniors and, in particular, white college-bound seniors on the SAT. SEO commissioned PSA to conduct a study to address this question. This report describes that study and its results. SEO Scholars, 2009 to 2013 The evaluation sample consisted of 299 SEO Scholars from five cohorts of twelfth-grade students, from the school year through the school year. 1 PSA received data from SEO on the students race-ethnicity, numeric GPA, and SAT scores in reading, math, and writing. As shown below in Exhibit 1, the SEO Scholars in these five cohorts were mostly black or Hispanic. SEO Scholars must maintain a GPA of C or higher throughout their participation in the program. A majority (95 percent) of SEO Scholars who were in twelfth grade between 2009 and 2013 had a GPA of 80 or above, equivalent to a B or higher. Of the 15 SEO Scholars with a GPA of 70 to 79, equivalent to a C, 13 had at least a Four SEO Scholars in the 2012 cohort were missing information on GPAs. These students were excluded from the evaluation sample. 1

4 Exhibit 1 Characteristics of SEO Scholars, 2009 to 2013 (N=299) Student characteristics Percent of SEO Scholars Cohort % Race-ethnicity Black 43 Hispanic 39 Asian 18 GPA A+ (97-100) 4 A (93-96) 20 A- (90-92) 27 B (80-89) 44 C (70-79) 5 Exhibit reads: Fifteen percent of SEO Scholars in the evaluation sample graduated in Study Methodology and Limitations We combined the five cohorts of SEO Scholars to form one group and compared this group with the population of SAT from 2009 to The population of - included students with a wide range of GPAs, while the pool of SEO Scholars consisted mostly of students with a GPA of B or A. To address this disparity and to help control for academic performance, the analyses included comparisons of SEO Scholars and of the same GPA. 3 We recoded numeric GPA to match the College Board GPA categories, so that a GPA of 97 or higher is equivalent to an A+, 93 to 96 is an A, 90 to 92 is an A-, 80 to 89 is a B, and 70 to 79 is a C. Each subgroup of race-ethnicity and GPA was to include at least 30 SEO Scholars to draw reliable comparisons. To ensure a large enough subgroup for analysis by race-ethnicity, we combined SEO Scholars and with a GPA of A-, A, or A+ into a broader A 2 College Board provided data on SAT scores. Data were combined for the five years by averaging the means by subgroup and unweighting the average by number of in each year within that subgroup. Effect size calculations used a pooled standard deviation for all within a racial-ethnic subgroup, as this was the only standard deviation available. 3 The comparisons in this study do not control for socioeconomic and geographical characteristics of students that may be different for SEO Scholars and. 2

5 GPA category. SEO Scholars with a GPA equivalent to a C (of whom there were only 15) were excluded from GPA-specific analyses. Each SEO Scholar s composite SAT score was computed as the sum of the student s three component scores in reading, math, and writing. In all instances in this report, SAT scores refer to the SAT composite score. There were slight differences in average SAT scores by year and by race-ethnicity. In combining the five cohorts of SEO Scholars into one sample, only the differences in SAT scores between the three race-ethnicity groups were statistically significant. For each comparison of the average SAT scores of SEO Scholars and, we tested for the statistical significance of the difference between the two scores. Statistical significance indicates the probability of observing a finding by chance alone. All significant differences between scores noted in this report are less than 5 percent likely due to chance. Additionally, we calculated effect sizes (indicated by the letter d ), which gauge the magnitude of the difference in scores between the two groups. Conventions for educational research suggest that effect sizes between 0.10 and 0.20 indicate a small but meaningful difference, between 0.21 and 0.50 an important difference, and 0.51 or higher an impressive difference (Cohen, 1988; Lipsey, 1990). Appendix A includes the results of all significance tests and effect size calculations for the comparisons made in this report. There are a few limitations to this study that are important to consider when interpreting the findings. For example, the students selected for and completing the SEO Scholars program come from low-performing and high-poverty schools, while the characteristics of the schools attended by are unknown but likely, on average, to be higher-performing and have lower rates of poverty. Additionally, though combining the five cohorts maximized the population size, the difference in size between the SEO Scholars population and the population of could affect the reliability of analyses conducted. Lastly, testtakers self-report their GPA, which means the categorization of students by this measure should be interpreted with caution (Kuncel, Credé & Thomas, 2005). While we controlled for academic performance using GPA, there are likely a number of unobserved differences between SEO Scholars and. Without accounting for these unknown differences, we cannot fully attribute differences in SAT achievement results to the SEO Scholars program. SAT Composite Score Comparison In this section, we first compare average SAT composite scores for all SEO Scholars to average composite scores for white, all, overall and by raceethnicity. We follow with comparisons of average SAT scores of SEO Scholars and scores of broken out by GPAs bands, overall and by race-ethnicity. 3

6 Comparison of SEO Scholars to All National Test-Takers, Overall and by Race- Ethnicity SEO Scholars who were in twelfth grade between 2009 and 2013 scored, on average, 1,624 on the SAT, significantly higher than average scores for white and all (1,579 and 1,503, respectively) (Exhibit 2). The difference in scores was particularly large between the SEO Scholars and all (d=0.74). Black SEO Scholars scored, on average, 1,618 on the SAT, which was significantly higher than average scores for black, white, and all on the SAT (Exhibit 2). The largest difference in scores was between the average score for black SEO Scholars and for black (1,275) (d=2.05). Exhibit 2 Black SEO Scholars versus, 2009 to 2013 mean SAT scores Composite SAT score 2,400 2,000 1,600 1,618 1,579 1,503 All SEO Scholars SAT=1,624 (N=299) 1,200 1, Black SEO Scholars (N=127) White (N=4,241,986) All (N=8,049,767) Black (N=1,027,720) Exhibit reads: The average composite SAT score among black SEO Scholars between 2009 and 2013 was 1,618. Hispanic SEO Scholars scored, on average, 1,602 on the SAT, which was significantly higher than the average score for Hispanic and all. While Hispanic SEO Scholars scored, on average, higher than did white, the difference was not statistically significant (Exhibit 3). The difference between the average score 4

7 for Hispanic SEO Scholars and the average score for Hispanic (1,357) was large (d=1.62). Exhibit 3 Hispanic SEO Scholars versus, 2009 to 2013 mean SAT scores Composite SAT score 2,400 2,000 1,600 1,200 1,602 1,579 1,503 1,357 All SEO Scholars SAT=1,624 (N=299) Hispanic SEO Scholars (N=117) White (N=4,241,986) All (N=8,049,767) Hispanic (N=1,237,561) Exhibit reads: The average composite SAT score among Hispanic SEO Scholars between 2009 and 2013 was 1,602. Asian SEO Scholars scored, on average, 1,687 on the SAT, which was significantly higher than average scores for Asian, white, and all. The biggest difference was between the scores for Asian SEO Scholars and all (d= 1.22). Comparison of SEO Scholars with a GPA of A to National Test-Takers with a GPA of A, Overall and by Race-Ethnicity SEO Scholars with a GPA of A scored, on average, 1,697 on the SAT, which was slightly lower than the average score for white with a GPA of A (1,706) and higher than the average score for all with a GPA of A (1,669). In both cases, the differences were not statistically significant. 5

8 Black SEO Scholars with a GPA of A scored, on average, 1,708 on the SAT, which was slightly higher than the average score for white and higher than the average score for all, but in neither case were the differences statistically significant. They did, however, score significantly higher than did black with a GPA of A, who scored, on average, 1,445, and the difference was large (d=1.64) (Exhibit 4). Exhibit 4 Black SEO Scholars with a GPA of A versus with a GPA of A, 2009 to 2013 mean SAT scores Composite SAT score 2,400 2,000 All SEO Scholars SAT=1,697 (N=154) 1,600 1,708 1,706 1,669 1,200 1, Black SEO Scholars (N=52) White (N=2,041,360) All (N=3,343,489) Black (N=238,349) Exhibit reads: The average composite SAT score among black SEO Scholars between 2009 and 2013 who had a GPA of A was 1,708. Hispanic SEO Scholars with a GPA of A scored, on average, 1,661, which was lower than average scores for white and all with a GPA of A, but the differences were not statistically significant. Hispanic SEO Scholars with a GPA of A, however, significantly outscored their Hispanic counterparts with a GPA of A (1,513) and the difference was large (d=1.00) (Exhibit 5). 6

9 Exhibit 5 Hispanic SEO Scholars with a GPA of A versus with a GPA of A, 2009 to 2013 mean SAT scores Composite SAT score 2,400 2,000 All SEO Scholars SAT=1,697 (N=154) 1,600 1,661 1,706 1,669 1,513 1, Hispanic SEO Scholars (N=64) White (N=2,041,360) All (N=3,343,489) Hispanic (N=401,037) Exhibit reads: The average composite SAT score among Hispanic SEO Scholars between 2009 and 2013 who had a GPA of A was 1,661. Asian SEO Scholars with a GPA of A scored, on average, 1,742 on the SAT, which was higher than average scores for white and all with a GPA of A. Only the difference between the average scores for Asian SEO Scholars and all in this category was statistically significant, and large (d=0.53). The Asian SEO Scholars SAT score was, on average, lower than scores for Asian with the same GPA, who scored, on average, 1,763, but the difference was not statistically significant. Comparison of SEO Scholars with a GPA of B to National Test-Takers with a GPA of B, Overall and by Race-Ethnicity SEO Scholars with a GPA of B scored, on average, 1,548 on the SAT, which was significantly higher than the average scores for white (1,475) and all (1,408) with a GPA of B (Exhibit 6). The difference in scores was largest between SEO Scholars and all (d=0.93). 7

10 Black SEO Scholars with a GPA of B scored, on average, 1,563 on the SAT, which was significantly higher than scores for black, white, and all with a GPA of B (Exhibit 6). The largest difference in scores was between black SEO Scholars and black with the same GPA, whose average score was 1,258 (d=1.99). Exhibit 6 Black SEO Scholars with a GPA of B versus with a GPA of B, 2009 to 2013 mean SAT scores Composite SAT score 2,400 2,000 1,600 1,200 1,563 1,475 1,408 1,258 All SEO Scholars SAT=1,548 (N=130) Black SEO Scholars (N=64) White (N=1,710,548) All (N=3,316,866) Black (N=517,285) Exhibit reads: The average composite SAT score among black SEO Scholars between 2009 and 2013 who had a GPA of B was 1,563. Hispanic SEO Scholars with a GPA of B scored, on average, 1,522, which was significantly higher than average scores for Hispanic, white, and all with a GPA of B (Exhibit 7). The largest difference in scores was between Hispanic SEO Scholars and their peers of the same ethnicity and GPA category, whose average SAT score was 1,310 (d=1.58). 8

11 Exhibit 7 Hispanic SEO Scholars with a GPA of B versus with a GPA of B, 2009 to 2013 mean SAT scores Composite SAT score 2,400 2,000 1,600 All SEO Scholars SAT=1,548 (N=130) 1,200 1,522 1,475 1,408 1, Hispanic SEO Scholars (N=49) White (N=1,710,548) All (N=3,316,866) Hispanic (N=597,912) Exhibit reads: The average composite SAT score among Hispanic SEO Scholars between 2009 and 2013 who had a GPA of B was 1,522. Asian SEO Scholars with a GPA of B scored, on average, 1,566, which was higher than average scores for white and all with a GPA of B, as well higher than average scores for Asian with a GPA of B (1,490). There were not enough Asian SEO Scholars with a GPA of B in the sample to conduct a reliable test for statistical significance of the differences in scores. Conclusion This evaluation compared the SAT composite scores of five cohorts of SEO Scholars to the SAT composite scores of five cohorts of, overall, by ethnicity, and by GPA. It found that in a large majority of cases SEO Scholars outscored white testtakers, all, and of the same ethnicity and GPA band on the SAT and in a minority of cases scored similarly to those groups. When broken out by race, this evaluation found that black SEO Scholars scored significantly higher than black, white, and all on the SAT and that the 9

12 difference in average SAT scores between black SEO Scholars and black was particularly large. Hispanic SEO Scholars also scored significantly higher than Hispanic and all on the SAT and the difference in average SAT scores between Hispanic SEO Scholars and Hispanic was particularly large. While Hispanic SEO Scholars scored, on average, higher than did white, the difference was not statistically significant. This evaluation also sought to compare students of similar academic performance by comparing SEO Scholars and with similar GPAs. Black and Hispanic SEO Scholars with a GPA of B scored significantly higher than black, Hispanic, white, and all with a GPA of B. The differences between the two groups were substantial in most cases. Black and Hispanic SEO Scholars with a GPA of A scored significantly higher than black and Hispanic with a GPA of A. They also scored similarly to white and all with a GPA of A, as their average SAT scores did not differ significantly from scores. In light of a previous study conducted by Policy Studies Associates that found that SEO Scholars outperformed a matched comparison group on various educational outcomes, these results further suggest that the additional classroom instruction and support SEO provides participants helps address gaps in academic performance between underserved populations and their college-bound peers. 10

13 References Cohen, J. (1988). Statistical power analysis for the behavioral sciences (second ed.). Hillsdale: Lawrence Erlbaum Associates. Coleman, S., Palmiter, A., Turner, T., Vile, J., Warburton, S., & Reisner, E. (2012). Preparing for college success: evaluation of the educational impact of the SEO Scholars Program. Washington, DC: Policy Studies Associates. Hedges, L.V., & Nowell, A. (1998). Black-white test score convergence since In C. Jencks & M. Phillips (Eds.), The black-white test score gap. (149 81). Washington, DC: The Brookings Institute Kuncel, N.R., Credé, M., & Thomas, L.L. (2005). The validity of self-reported grade point averages, class ranks, and test scores: A meta-analysis and review of literature. Review of Educational Research, 75(1). Lipsey, M. (1990). Design sensitivity: Statistical power for experimental design. Newbury Park: Sage Publications. 11

14 Appendix A Statistical Comparison Results

15 Appendix A Statistical Comparison Results Race-ethnicity SEO Scholars National Effect (SEO Scholars vs. size Mean ) Mean SD N Mean SD N (d) diff. Sig. All GPAs All vs. all 1, , ,049, Yes All vs. white 1, , ,241, Yes Black vs. all 1, , ,049, Yes Black vs. white 1, , ,241, Yes Black vs. black 1, , ,027, Yes Hispanic vs. all 1, , ,049, Yes Hispanic vs. white 1, , ,241, No Hispanic vs. Hispanic 1, , ,237, Yes Asian vs. all 1, , ,049, Yes Asian vs. white 1, , ,241, Yes Asian vs. Asian 1, , , Yes GPA of A All vs. all 1, , ,343, No All vs. white 1, , ,041, No Black vs. all 1, , ,343, No Black vs. white 1, , ,041, No Black vs. black 1, , , Yes Hispanic vs. all 1, , ,343, No Hispanic vs. white 1, , ,041, No Hispanic vs. Hispanic 1, , , Yes Asian vs. all 1, , ,343, Yes Asian vs. white 1, , ,041, No Asian vs. Asian 1, , , No GPA of B All vs. all 1, , ,316, Yes All vs. white 1, , ,710, Yes Black vs. all 1, , ,316, Yes Black vs. white 1, , ,710, Yes Black vs. black 1, , , Yes Hispanic vs. all 1, , ,316, Yes Hispanic vs. white 1, , ,710, Yes Hispanic vs. Hispanic 1, , , Yes Asian vs. all 1, , ,316, N/A Asian vs. white 1, , ,710, N/A Asian vs. Asian 1, , , N/A A-1