Learning from Lotteries: Using Administrative Data to Estimate the Short- and Long-Run Effects of Charter Schools

Transcription

1 Learning from Lotteries: Using Administrative Data to Estimate the Short- and Long-Run Effects of Charter Schools Professor Susan Dynarski University of Michigan & National Bureau of Economic Research

2 Accountability and Flexibility in Public Schools: Evidence from Boston s Charters and Pilots Atila Abdulkadiroğlu, Duke Josh Angrist, MIT Susan Dynarski, University of Michigan Thomas Kane, Harvard GSE Parag A. Pathak, MIT Quarterly Journal of Economics, 2011

3 Background: from Assignment to Evaluation How to improve education and close achievement gaps? Inputs (class size, etc) Incentives (for students and teachers) Choice within the public system Autonomy and decentralization (charters, vouchers) Can schools alone close large achievement gaps? We look at two autonomy / decentralization models in Boston

4 The Charter Model Charter schools are publicly funded, but operate with minimal supervision Nonprofits, universities, teachers, or parents can open charters; Charters are granted by the state DOE Each Charter runs as its own district Charters are funded through tuition paid by sending districts Tuition senders average per-pupil expenditure

5 The Pilot Alternative Pilots were introduced in the wake of charters Free to: allocate staff, set budget priorities, curriculum, and scheduling Boston pilots remain in BPS; typically use BPS student assignment mechanism Pilots are approved by the Boston Teachers Union and school staff Free from: most collectively bargained work rules and district curriculum requirements Covered by: union pay scales, seniority provisions, and employment protection Some accountability

6 Table 1. Teacher Characteristics II. Middle Schools Traditional BPS Schools Charter Pilot Teachers licensed to their assignment 77.8% 53.9% 65.8% Core classes taught by highly qualified teachers 84.8% 70.4% 70.2% Student / Teacher ratio Proportion of teachers 32 and younger 27.1% 74.5% 55.0% Proportion of teachers 49 and older 36.0% 4.8% 13.6% Number of teachers Number of schools

7 Charter and Pilot Assignment Charter admissions Charters cannot use admissions tests Charters use school-specific lotteries when oversubscribed Elementary and middle Pilots use the BPS assignment mechanism The BPS assignment mechanism uses a lottery to break ties at in-demand schools Some Pilots and Charters are under-subscribed or filled with guaranteed applicants and/or siblings

8 The Plan To estimate causal effects of years (grades) spent in a Pilot or Charter school on MCAS test scores To this end, we use two study designs: 1. Quasi-experimental ( lottery ) This gives us a handle on the selection problem Covers only schools with effective lotteries and reasonably good records 2. Observational ( regression ) Relies on statistical controls Covers all public schools in Metro Boston We compare observational results for the lottery subsample to lottery results; this gives us confidence in the full-sample observational findings

9 Data 1. Quasi-experimental samples: Pilot applicants to lottery-using over-subscribed schools exclude guaranteed applicants and siblings with baseline data and MCAS in Charter applicants to over-subscribed Boston charters with usable lottery records exclude guaranteed applicants and siblings with baseline data and MCAS in Observational sample: BPS residents attending BPS schools or a Boston Charter at baseline In state (SIMS) data files; with baseline demographics Have MCAS scores and attending BPS or Boston Charter in outcome years

10 Table 2. Descriptive Statistics Traditional Enrolled in Applicants in Lotto BPS Schools Charter Pilot Charter Pilot II. Middle Schools Female 47.0% 48.9% 49.9% 48.3% 52.6% Black 46.9% 69.4% 50.5% 59.2% 49.8% Hispanic 37.3% 19.0% 28.3% 19.4% 31.2% Special education 24.5% 18.5% 21.4% 19.3% 17.5% Free or reduced price lunch 89.3% 73.1% 85.6% 69.1% 79.3% Limited English proficiency 21.8% 7.1% 21.0% 7.7% 15.0% 4th Grade Math Score th Grade English Score Years in charter Years in pilot Number of Students 12,257 2,382 2,692 1,355 1,917 Number of Schools

11 Quasi-experimental study

12 Quasi-experimental Design: Charters We study charter applicants for spots in 6th (middle school) and 9th grade (high school) Our charter applicant file includes non-sibling first-round applicants who apply to schools in our sample Charters run and document their own lotteries Charters are city-wide with no walk zones The Charter lottery instrument indicates students offered a seat at any Charter to which they applied The Charter risk set is defined by the set of schools to which an applicant applied (e.g., 3 schools generates 7 risk sets)

13 Table 3. Covariate Balance at Charters Middle School High School All Lottos w/ All Lottos w/ Lottos Baselines Lottos Baselines Hispanic (0.024) (0.024) (0.023) (0.023) Black (0.029) (0.030) (0.026) (0.026) White * (0.023) (0.024) (0.012) (0.012) Asian ** 0.019* (0.008) (0.008) (0.011) (0.011) Female (0.031) (0.032) (0.026) (0.026) FRPL (0.029) (0.029) (0.023) (0.023) SPED (0.025) (0.025) (0.020) (0.020) LEP * 0.022* (0.015) (0.015) (0.011) (0.011) Baseline ELA (0.053) - (0.043) Baseline Math 0.095* (0.055) - (0.048) p-value, F-test *

14 Table 3. Covariate Balance at Pilots Middle School High School All Lottos w/ All Lottos w/ Lottos Baselines Lottos Baselines Hispanic (0.025) (0.038) (0.028) (0.028) Black (0.027) (0.040) (0.031) (0.031) White (0.019) (0.022) (0.017) (0.017) Asian (0.014) (0.021) (0.015) (0.016) Female (0.030) (0.043) (0.031) (0.031) FRPL ** 0.065** (0.023) (0.029) (0.026) (0.026) SPED (0.020) (0.034) (0.023) (0.023) LEP * (0.016) (0.030) (0.015) (0.015) Baseline ELA (0.077) - (0.054) Baseline Math (0.078) - (0.057) p-value, F-test

15 2SLS Strategy The second stage controls for lottery risk sets: y igt = α t + β g + j δ j d ij + γ X i + ρs igt + ɛ igt, (1) where d ij indicates i in risk set j, with effect δ j ; α t and β g are year-of-test and grade-of-test effects; s igt is years in charter or pilot The corresponding first stage is: s igt = λ t + κ g + j µ j d ij + Γ X i + Π Z i + η igt (2) The instruments, Z i, indicate lottery offers in student i s risk set

16 Table 4. Lottery Results for Middle School Charters First Stage Reduced Form 2SLS 2SLS w/demos ELA 0.965*** 0.181*** 0.187*** 0.149*** (0.114) (0.061) (0.062) (0.052) N 2,416 Math 0.918*** 0.397*** 0.432*** 0.405*** (0.108) (0.067) (0.075) (0.066) N 2,582

17 Table 4. Lottery Results for Middle School Pilots First Stage Reduced Form 2SLS 2SLS w/demos ELA 1.378*** (0.189) (0.065) (0.047) (0.043) N 3,390 Math 1.291*** (0.181) (0.069) (0.053) (0.048) N 3,851

18 VIV!Estimates!of!the!Middle!School!Math!Effect Score!Difference !0.5!1!1.5!0.75! Difference!in!average!years!in!charter A. Charter Schools VIV!Estimates!of!the!Middle!School!Math!Effect Score!Difference !0.5!1!1.5! Difference!in!average!years!in!pilot B. Pilot Schools Figure 1. This figure plots treatment-control differences in test score means against treatment-control differences in years in charter (Panel A) or pilot (Panel B). The unit of observation is a charter or pilot application risk set (N=34 for charters and N=51 for pilots). The charter slope (unweighted) is.66, and the corresponding 2SLS estimate is.44. The pilot slope (unweighted) is -.006, while the corresponding 2SLS estimate is The pilot graph is produced after dropping one small imbalanced risk set.

19 39/ Figure 2: Results for Charter Middle Schools by Grade: Lottery Winners vs. Lottery Losers Charter MS: Winners vs. Losers (RF) Math ELA Math 0.5 Charter Score Effect ELA Baseline 6th Grade 7th Grade 8th Grade Note: Reduced form regressions, no demographic or score controls.

20 40/ Figure 1: Results for Pilot Middle Schools by Grade: Lottery Winners vs. Lottery Losers Pilot MS: Winners vs. Losers (RF) ELA Math Pilot Score Effect ELA Math -0.4 Baseline 6th Grade 7th Grade 8th Grade Note: Reduced form regressions, no demographic or score controls. Back

21 Observational study

22 Observational Study Methods Full-sample regression estimates offer a handle on external validity Regression model for scores of kid i in grade g, tested in year t: y igt = α t + β g + γ X i + ρ S igt + ɛ igt (3) Includes year and grade effects, demographics, and sometimes a baseline score S igt is a vector of years in Pilot/Charter/Alt/Exam school from baseline to year t s.e.s clustered on student when grades are stacked, and always on school-by-year (2-way) MS models with baseline scores omit students in K-8s Table 9 reports estimates by school level and score type

23 Table 10. Estimates in and Out of the Lotto Sample Charter Elementary and Middle Lottery Observational w/ w/ Baseline In Lottery Not in Lottery Demos Scores Sample Sample I. Elementary School ELA *** (0.018) N 20,058 Math (0.023) N 17,356 II. Middle School ELA 0.149*** 0.144*** 0.158*** 0.082*** (0.052) (0.044) (0.017) (0.014) N 2,416 2,365 31,620 Math 0.405*** 0.386*** 0.312*** 0.129*** (0.066) (0.054) (0.028) (0.020) N 2,582 2,528 35,764

24 Observational vs Lottery Estimates Charters Observational results (with baseline scores) in the lottery sample are remarkably close to lottery estimates This may validate the observational design, though obs results also suggest our lottery-sample charters are better Pilots A match on modest effects for elementary pilots Observational results for middle school pilots are, like lotteries, also negative, in and out of lottery sample Observational results for pilot high schools ELA + Math are positive, while lottery results are insignificant Observational pilot study agrees with lottery in that it shows weaker, mixed effects

25 Long-Run Effects of Charter Schools

26 Stand and Deliver: The Effect of Boston s Charter High Schools on College Preparation, Entry, and Choice Josh Angrist (MIT and NBER) Sarah Cohodes (Harvard) Susan Dynarski (Michigan and NBER) Parag Pathak (MIT and NBER) Chris Walters (Berkeley) Journal of Labor Economics, 2015

27 Motivation and Background Are These Effects Lasting? Charter schools, which face the risk of closure, may just be particuarly good at teaching to the test." We therefore examine the effect of charter schools on outcomes closely linked to earnings and other measures of welfare College preparation (standards, scholarship eligibility, AP, SAT) College entry College choice As applicants age, we will be able to observe yet more outcomes College completion Earnings, home ownership, assets? (we can dream...)

28 Schools and Samples Boston s Charter High Schools Boston had 9 charters enrolling traditional high school students in our study period; we assess 6 (2 are now closed, 1 has poor records) Academy of the Pacific Rim (grades 5-12; 6-12 in our sample period) Boston Collegiate (5-12) Boston Preparatory (6-12) City on a Hill Charter School (9-12) Codman Academy (9-12) MATCH (9-12) These schools were over-subscribed for cohorts applying (expected to graduate ) We exploit these lotteries for causal inference

29 Schools and Samples School characteristics Table 1: Boston School Characteristics Public High Schools Charters Serving Grade 9-12 Charters Serving Grade 9-12 Only Charters in the Study Mean Mean Mean Mean (1) (2) (3) (4) Panel A: Charter School Characteristics Number of years open Days per year Average minutes per day Have Saturday school Avg. math instructions (min) Avg. reading instruction (min) No Excuses

30 Schools and Samples School characteristics Table 1: Boston School Characteristics Public High Schools Charters Serving Grade 9-12 Charters Serving Grade 9-12 Only Charters in the Study Mean Mean Mean Mean (1) (2) (3) (4) Panel B: Comparison with Traditional Boston Public Schools Number of teachers Student/teacher ratio % of teachers licensed in teaching assignment % of teachers 32 and younger % of teachers 49 and older % of core classes taught by highly qualified teachers Avg. per-pupil expenditure $14,614* $14,277 $15,313 $13,990 Title I eligible N (schools)

31 Research Design 2SLS Specification Second Stage: y i = δ j d ij + γ X i + ρc i + ε i j where y i is an outcome for student i; d ij are risk sets; C i is attendance in 9th or 10th grade at our 6 charters; X i is a vector of demographics (10th-grade-year, race, special education, limited English, subsidized lunch, female-race interaction) First Stage: C i = µ j d ij + β X i + π 1 Z i1 + π 2 Z i2 + η i j where π 0 and π 1 are first-stage effects for each instrument

32 Descriptive Stats Table 2: Descriptive Statistics Projected Senior Year (MCAS outcome sample) BPS 9th Graders Mean Mean Lottery Applicants Ever Offer Initial Offer (1) (2) (3) (4) Female (0.021) (0.019) Black (0.021) (0.018) Hispanic (0.018) (0.016) Asian (0.008) (0.006) Subsidized Lunch (0.019) (0.017) Special Education (0.017) (0.015) Limited English Proficiency (0.008) (0.007) Baseline MCAS ELA (0.037) (0.034) Baseline MCAS Math (0.038) (0.034) P-Value Charter Attendance Ever Offer Initial Offer 0.314

33 Results Results: Test Scores Benchmark

34 Results MCAS Table 3: Lottery Estimates of Effects on 10th-Grade MCAS Scores for Samples by Projected Senior Year First Stage Outcome Mean Ever Offer Initial Offer [s.d.] Effect Subject (1) (2) (3) (4) Panel A: (MCAS outcome sample) Standardized ELA 0.228*** 0.134*** *** (0.041) (0.026) [0.833] (0.104) N 3671 Standardized Math 0.229*** 0.134*** *** (0.041) (0.026) [0.911] (0.120) N 3615

35 Results MCAS MCAS Milestones MCAS is NCLB test for 10th-grade students, so has high stakes for schools Scores fall into four performance categories: Failing, Needs Improvement, Proficient, Advanced We plot MCAS score distributions for charter-offer compliers, along with these cutoffs We adapt the methods of Abadie (2002, 2003) to generate these complier distributions The MCAS also has high stakes for students A minimum score is needed to graduate high school

36 Results MCAS Figure 1: Complier Distributions for MCAS Scaled Scores Notes: This figure plots smoothed MCAS scaled score distributions for treated and untreated compliers. The sample is restricted to lottery applicants who are projected to graduate between 2006 and 2013 assuming normal academic progress from baseline. Dotted vertical lines at scaled score 220 mark MCAS needs improvement thresholds, 240 for MCAS proficiency thresholds, and 260 for MCAS advanced thresholds.

37 Results MCAS Table 4: Lottery Estimates of Effects on MCAS Performance Categories First Attempt Ever Mean Effect Mean Effect (1) (2) (3) (4) Panel B: MCAS Math Needs Improvement or Higher ** ** (0.033) (0.015) Proficient or Higher *** ** (0.065) (0.065) Advanced or Higher *** *** (0.061) (0.061) Meets Competency Determination ** * (0.058) (0.056) N 3615

38 Results MCAS Adams Scholarship Eligibility MCAS score determines eligibility for the Adams Scholarship Requires minimum score as well as scoring in district s top quartile Covers tuition (but not fees) at MA public colleges

39 Results MCAS Table 4: Lottery Estimates of Effects on MCAS Performance Categories First Attempt Ever Mean Effect Mean Effect (1) (2) (3) (4) Panel C: ELA and Math Combined Needs Improvement or Higher ** (0.035) (0.016) Proficient or Higher ** ** (0.071) (0.073) Advanced or Higher *** *** (0.035) (0.035) Meets Competency Determination *** ** (0.066) (0.066) Eligible for Adams Scholarship *** Using BPS Cutoffs (0.059) N 3594

40 Results College Preparation Results: College Preparation

41 Results SAT and Advanced Placement SAT Results Nearly two-thirds of our applicants take the SAT Charter enrollment leaves this rate unchanged We plot complier distributions of SAT scores for those who win and lose the lottery K-S tests confirm significant impacts for each score component

42 Results SAT and Advanced Placement Figure 2: Complier Distributions for SAT Scores Notes: This figure plots smoothed SAT score distributions for treated and untreated compliers. The sample is restricted to lottery applicants who are projected to graduate between 2007 and 2012.

43 Results SAT and Advanced Placement Table 5: Lottery Estimates of Effects on SAT Test-taking and Scores Taking Reasoning (1600) Composite (2400) Mean Mean Mean [s.d.] Effect [s.d.] Effect [s.d.] Effect (1) (2) (3) (4) (5) (6) Took SAT [0.482] (0.078) Score Above MA Bottom Quartile ** * [0.435] (0.066) [0.435] (0.067) Score Above MA Median ** ** [0.289] (0.049) [0.275] (0.040) Score In MA Top Quartile [0.160] (0.016) [0.137] (0.017) N 2957 Average Score *** ** (For takers) [166.6] (29.1) [240.1] (42.9) N 1897

44 Results SAT and Advanced Placement Score Above MA Bottom Quartile Table 5: Lottery Estimates of Effects on SAT Test-taking and Scores Math (800) Verbal (800) Writing (800) Mean Mean Mean [s.d.] Effect [s.d.] Effect [s.d.] Effect (1) (2) (3) (4) (5) (6) ** ** [0.458] (0.080) [0.440] (0.061) [0.448] (0.067) Score Above MA Median ** [0.320] (0.057) [0.303] (0.046) [0.294] (0.041) Score In MA Top Quartile * [0.177] (0.028) [0.157] (0.021) [0.147] (0.021) N 2957 Average Score *** * (For takers) [95.5] (16.9) [87.4] (15.7) [86.7] (16.2) N 1897

45 Results SAT and Advanced Placement SAT Results Charters push scores out of the lower half of the MA distribution Charters increase SAT Reasoning (math+verbal) by 74 points SAT Composite (all scores) by 100 points Effects are largest in math These effects are about 0.33σ of national SAT distribution

46 Results SAT and Advanced Placement Table 6: Lottery Estimates of Effects on Advanced Placement Test-taking and Scores All AP Exams Mean Effect (1) (2) Took Exam *** (0.073) Number of Exams *** (0.274) Score 2 or Higher ** (0.068) Score 3 or Higher * (0.052) Score 4 or (0.033) N 2957

47 Results SAT and Advanced Placement Table 6: Lottery Estimates of Effects on Advanced Placement Test-taking and Scores All AP Exams Science Calculus US History Mean Effect Mean Effect Mean Effect Mean Effect (1) (2) (3) (4) (5) (6) (7) (8) Took Exam *** *** *** * (0.073) (0.061) (0.070) (0.093) # Exams *** *** (0.274) (0.070) Score ** * (0.068) (0.032) (0.045) (0.048) Score * * (0.052) (0.015) (0.040) (0.019) Score (0.033) (0.012) (0.019) (0.011)

48 Results Educational Attainment High School Graduation Given strong effects on MCAS, we expected to see increases in HS graduation. We don t. Charter attendance decreases probability of graduating in four years by about 12 points Has no effect on five-year graduation rate Consistent with early findings in an evaluation of a mandatory College For All" curriculum in Michigan Dynarski, Frank, Jacob, Schneider Appears to take more than four years to get weaker students through a college-prep curriculum

49 Results Educational Attainment HS Graduation Table 7: Lottery Estimates of Effects on High School Graduation and Grade Repetition Excl. Transferred and Deceased Mean Effect Mean Effect (1) (2) (3) (4) Panel A: Grade Progression Start 10th Grade On-time (0.027) (0.027) Start 11th On-time (0.049) (0.047) Start 12th On-time (0.057) (0.055) Four-year Graduation ** * (0.063) (0.063) N Five-year Graduation (0.065) (0.060) N

50 Results Educational Attainment HS Graduation & GED Table 7: Lottery Estimates of Effects on High School Graduation and Grade Repetition Excl. Transferred and Deceased Mean Effect Mean Effect (1) (2) (3) (4) Panel B: Graduation and GED Four-year Graduation (0.083) (0.113) N Four-year Graduation or GED (0.081) (0.112) N Five-year Graduation (0.099) (0.097) N Five-year Graduation or GED (0.094) (0.086) N

51 Results Educational Attainment Results: College Attendance and Choice

52 Results Educational Attainment Data We use NSC to track college attendance and related outcomes NSC-participating institutions enroll 94% of Mass. undergraduates Since charter attendance appears to delay HS graduation, we look at enrollment in two time frames Within a year of on-time HS graduation Within two years of on-time high school graduation

53 Results Educational Attainment Table 8: Lottery Estimates of Effects on College Enrollment Enrolled Immediately Enrolled Immediately or Oneyear Later Mean Effect Mean Effect (1) (2) (3) (4) Panel A: Attendance at Any NSC-Covered School Any (0.072) (0.084) Two-year ** (0.051) (0.064) Four-year ** ** (0.070) (0.079) Four-year Public *** *** (0.059) (0.070) Four-year Private (0.076) (0.094) Four-year Public In MA ** ** (0.054) (0.063) N

54 Results Educational Attainment College Enrollment and Choice College enrollment: Point estimates are large ( 6-12 points) but imprecise Two-Year vs. Four-Year College Increased enrollment in four-year colleges: 17 points Large relative to mean of 41% for non-charter students Increase comes from both enrollment margin and a shift from two-year colleges Type of Four-Year College Increased enrollment in four-year public colleges: points Large relative to mean of 14% for non-charter students Increase is mostly at MA public colleges Is this shift anti-selective, as in Cohodes and Goodman s (2013) analysis of Adams Scholarships?

55 Results Educational Attainment Table 8: Lottery Estimates of Effects on College Enrollment Enrolled Immediately Enrolled Immediately or Oneyear Later Mean Effect Mean Effect (1) (2) (3) (4) Panel B: Attendance at Barron's-Ranked Schools Lowest Selectivity Tier Only (0.056) (0.071) Second Lowest Selectivity Tier Only (0.062) (0.068) Top Three Selectivity Tiers (0.056) (0.055) N

56 Figure 3: Lottery Estimates of Effects within Risk Sets SAT Reasoning / Panel A: MCAS Composite-SAT Reasoning 5 3/5 3/ /5 3/ th-grade MCAS Composite Panel B: MCAS Composite-Four-year Within 6 Months Enrollment Four-year Within 6 Months Enrollment /5 3/ /5 3 3/5 3/ th-grade MCAS Composite Notes: This figure plots within-risk-set lottery estimates of the effects of charter school attendance. Panel A plots effects on SAT Reasoning (Verbal + Math) against effects on MCAS Composite scores. Panel B plots effects on the probability of within 6 months enrollment in a four-year college against effects on MCAS Composite scores. The sample in Panel A includes students projected to graduate between 2007 through 2012 while the sample in Panel B includes students projected to graduate between 2006 through Data points are labeled by school number, with schools numbered in decreasing order of their MCAS math effects. Blue circles indicate risk sets in which students applied to one school, while red squares indicate risk sets in which students applied to two. Marker sizes are proportional to the inverse of the standard errors of the estimates on the x-axis. Estimates for a given risk set use the instrument (ever or initial offer) with the larger first-stage t- statistic. The sample excludes risk sets with first-stage t-statistics less than one; this restriction would remove all data for one school, so the largest risk set for that school is retained.

57 Results Heterogeneity in Effects Heterogeneity in Effects In previous work we showed charters boost test scores most for: Nonwhites Special ed English learners Low achievers at baseline Does this pattern hold for college preparation, attendance and choice?

58 Results Heterogeneity in Effects MCAS Scores Special Ed Not Special Ed Below Median Baseline Above Median Baseline Sub Lunch Not Sub Lunch ELA 0.529* 0.379*** 0.423*** 0.358*** 0.394*** 0.558*** (0.272) (0.107) (0.144) (0.116) (0.116) (0.188) Math 0.676*** 0.551*** 0.570*** 0.497*** 0.526*** 0.748*** (0.256) (0.128) (0.148) (0.132) (0.136) (0.223)

59 Results Heterogeneity in Effects SAT and AP Special Ed Not Special Ed Below Median Baseline Above Median Baseline Sub Lunch Not Sub Lunch SAT Score 166.4** 91.8** 138.2*** ** 216.7** (82.6) (45.2) (50.7) (50.4) (41.5) (94.9) Took any AP 0.293*** 0.278*** 0.269*** 0.333*** 0.320*** (0.108) (0.083) (0.086) (0.111) (0.075) (0.130) Score 3+ AP * * (0.059) (0.065) (0.030) (0.103) (0.061) (0.088)

60 Results Heterogeneity in Effects Educational Attainment Special Ed Not Special Ed Below Median Baseline Above Median Baseline Sub Lunch Not Sub Lunch 4-yr HS Grad *** *** (0.159) (0.069) (0.090) (0.086) (0.070) (0.136) 5-yr HS Grad (0.167) (0.070) (0.091) (0.087) (0.074) (0.136) Any College * (0.252) (0.088) (0.118) (0.131) (0.089) (0.190) 4-year College * 0.291*** ** (0.238) (0.086) (0.111) (0.141) (0.085) (0.167)

61 Results Heterogeneity in Effects Can Boys Be Saved? Boy Girl Boy Girl ELA 0.446*** 0.372*** (0.164) (0.118) Math 0.498*** 0.615*** (0.186) (0.145) SAT Score * 4-yr HS Grad ** (64.9) (52.2) (0.101) (0.082) Took any AP 0.227** 0.320*** 5-yr HS Grad (0.106) (0.096) (0.097) (0.077) Score 3+ AP 0.148** Any College (0.068) (0.072) (0.126) (0.103) 4-year College (0.119) (0.116)

62 Interpretation of Effects Switching and Peer Effects Human capital or peer effects? The scope for peer effects is twofold: For starters, charter applicants are positively selected Students love it or leave it; perhaps exit changes the student mix Our lottery strategy retains all winners, so no bias from switching Peer effects might nevertheless explain charter school effectiveness Interactions with peer achievement show larger charter effects among those with low baseline Perhaps charter gains arise from removal of especially low performing or disruptive students Table 11 shows excess switching Is the charter peer advantage growing as students progress through high school? Figure 3 shows that the advantage is shrinking, as the weakest peers exit the traditional schools at a greater rate

63 Interpretation of Effects Table 11: Lottery Estimates of Effects on School Switching and Realized Peer Quality Mean Effect (1) (2) Panel A: School Switching Any switch (0.086) N 3072 Switch excluding * transitional grades (0.082) N 3063 Ever attend an exam school ** (0.042) N 3205

64 Interpretation of Effects Table 11: Estimates of Effects on School Switching and Realized Peer Quality Mean Effect Mean Effect Mean Effect Mean Effect (1) (2) (3) (4) (5) (6) (7) (8) Panel B: Realized Peer Quality First Post-lotto Year Second Post-lotto Year Third Post-lotto Year Fourth Post-lotto Year Peer Baseline ELA * (0.068) (0.073) (0.075) (0.070) Peer Baseline Math * (0.075) (0.079) (0.073) (0.075) Peer Baseline Sum of ELA and Math * (0.140) (0.148) (0.143) (0.140) N

65 Interpretation of Effects Figure 3: Peer Quality For Charter Lottery Compliers Realized Peer Quality in ELA Realized Peer Quality in Math Post-lotto Year Untreated Compliers Post-lotto Year Treated Compliers Notes: This figure plots mean realized peer quality in the first, second, third and fourth years after the lottery for treated and untreated charter lottery compliers. The sample is restricted to lottery applicants who are projected to graduate between 2006 and Realized peer quality is measured as the average baseline score for other students in the same school and year.

66 More Research and Insights

67 More Research and Insights: Heterogeneity in Treatment Effects We extended our analysis to the rest of Massachusetts 3

68

69 More Research and Insights: Heterogeneity in Treatment Effects We extended our analysis to the rest of Massachusetts Found effects outside urban areas are zero to negative Model this heterogeneity (Angrist, Pathak & Walters, AEJ: Applied, 2013) Heterogeneity in effects across populations Heterogeneity in the treatment Heterogeneity in the control 5

70 More Research and Insights: Modeling Policy Changes Massachusetts has a cap on charter schools What would be effect of raising cap? General equilibrium question. Begs for structural model Structural model of market, anchored with variation in causal effects across schools and populations: Walters (2015) Conclusion: marginal effect of expansion would be smaller than our estimates, since those who benefit most are less likely to apply Policy implication: reduce cost of application 6

71 More Research and Insights: Which Practices Drive Effect Variation? What practices explain variation in charter effects? Length of school day/year Tutors Curriculum Teacher recruitment & training Need to measure difference in practices between treatment & control 7

72 More Research and Insights: Which Practices Drive Effect Variation? Michigan Charter School Project Measure the causal effect of Michigan s 242 charter schools Identify the charter practices that are associated with the largest positive effects Survey of practices of both charters & traditional public schools. 8