EDUCATIONAL CHOICE AND STUDENT PARTICIPATION: THE CASE OF THE SUPPLEMENTAL EDUCATIONAL SERVICES PROVISION IN CHICAGO PUBLIC SCHOOLS University of Chicago Prepared for School Choice and School Improvement: Research in State, District and Community Contexts Vanderbilt University, October 25-27, 2009 This paper is supported by the National Center on School Choice, which is funded by a grant from the U.S. Department of Education's Institute of Education Sciences (IES) (R305A040043). All opinions expressed in this paper represent those of the authors and not necessarily the institutions with which they are affiliated or the U.S. Department of Education. All errors in this paper are solely the responsibility of the authors. Do not cite without author s permission. For more information, please visit the Center website at www.vanderbilt.edu/schoolchoice/. I thank Dr. Curtis Jones at the Office for Extended Learning Opportunities at Chicago Public Schools for providing the data in which to complete this analysis. I benefited from helpful comments from Stephen Raudenbush, Carolyn Heinrich, Tyler VanderWeele, an anonymous reviewer from the National Center on School Choice at Vanderbilt University, workshop participants at the Workshop on Education at the University of Chicago, University of Wisconsin-Madison School of Education/Wisconsin Center for Education Research Tuesday Lecture Series and seminar participants at Mathematica Policy Research, Inc., where I began this paper as a Mathematica Summer Fellow. The author can be reached at msteinberg@uchicago.edu.
Abstract The introduction of markets into the existing American public education system implies a reorganization of educational resources. New markets may thrive to the extent that families participate in them and avail themselves and their children of the new educational options created by the marketplace. As such, educational choice must first be understood in the context of how (and whether) families choose to interact with these markets. The Supplemental Educational Services (SES) provision, a key educational choice provision of the federal No Child Left Behind Act of 2002, represents one such rearrangement of public resources coupled with new choice options for families. The SES provision has created a market for tutoring services targeted to low-income students in perennially underachieving public schools. In addition, the SES provision has created a new forum for parental decision-making by endowing parents of children in underperforming Title I schools with a voucher with which to purchase tutoring services. This paper explores how educational choice in the form of SES has unfolded in the context of the Chicago Public Schools (CPS) district by modeling the determinants of selection into SES and examining the characteristics of students who chose to participate in the after-school tutoring intervention between the 2004-05 and 2007-08 school years. Evidence suggests that students in grades 1-8 with lower prior year cognitive achievement and fewer prior year disciplinary infractions were more likely to participate in SES, while students in grades 9-12 with lower prior year cognitive achievement and fewer prior year absences were more likely to participate in SES. This analysis presents among the first empirical evaluations of selection into SES by describing the characteristics of students who chose to engage this new educational choice option.
Introduction Market-based educational choice has a long and much-debated history. Proponents of market-based solutions to education refer to the intellectual origins of educational choice in the work of economist Milton Friedman, who first suggested the use of educational stipends, or vouchers, which families could use to choose among various school options (Friedman, 1962). Advocates of market-based reforms in education describe how allocative and productive efficiencies will be released by opening public education to private vendors (See Chubb and Moe, 1990; Hoxby, 1998; Neal, 2002). In particular, they note that the efficient amount of education will be allocated by the market to parents, subject to parents individual tastes and preferences, by endowing families with the ability to choose among educational options rather than be confined solely to local public suppliers of education, such as community schools. Advocates view a market for education as one which provides both the efficient level of education desired by parents, and the type of education services that best meet the learning needs of their children. On the other hand, critics of market-based choice question whether the market will adequately respond to the needs of families (See Ladd, 2002; Levin, 1998; Burch, 2006). Some appeal to traditional theories of market failures as the source of potential problems with fundamentally reshaping the institutional arrangement of schooling in America (Levin, 1991; Manksi, 1992). In order for a market to operate efficiently, information asymmetries must not distort the decisions made by families with respect to the efficient demand for educational services. However, critics of market-based choice programs worry that information is economically stratified, and that low-income and relatively disadvantaged families are inadequately prepared to navigate choice options, both in terms of the information available to them about education programs and their capacity to make informed decisions in the presence of information asymmetries. As a result, critics of market-based choice question the 1
ability of markets to enhance the educational opportunities available to families, particularly the economically and politically disadvantaged among the population. Efforts have been made to evaluate the extent to which market-based choice policies and programs have impacted student academic achievement (See Witte, 2000; Rouse, 1998; Cullen et al., 2005; Howell and Peterson 2004; Krueger and Zhu, 2004; Howell et al., 2002; Goldhaber, 1996). However, less attention has been paid to the characteristics of students (and families) that choose to participate in educational choice options. Of the studies that assess parental choice in voucher programs (Witte and Thorn, 1996; Levin, 1998; Howell, 2004; Campbell, West and Peterson, 2005; Chakrabati, 2005) and those that assess parental choice between private and public schooling options (Neal, 1997; Belfield, 2004; Bosetti, 2004), little consensus across choice settings exists concerning the factors (student, parent, school and community) and characteristics of students that significantly influence selection into educational choice options. Absent a clear understanding of the factors related to participation in choice options, researchers and policymakers are unable to adequately evaluate whether resources dedicated to market-based educational interventions are reaching those students who would most benefit from them. That is, the efficient allocation of educational resources via voucher-type programs is a function of not only the program s impact on achievement, but also of the characteristics of participants who choose to engage in a particular educational choice option. In particular, researchers and policymakers might deem a policy which re-directs resources to students who need them the most (i.e. students who are reading below grade level) and those with the potential to use the resources the most effectively (i.e. students who are more motivated to participate in additional academic instruction) as an efficient use of educational resources. The introduction of the Supplemental Educational Services (SES) provision presents a unique opportunity to assess the manner in which families engage new market-based educational 2
choice policies and to test the extent to which the neediest and most motivated students are receiving additional academic instruction. Under the federal No Child Left Behind (NCLB) legislation, the SES provision sanctions perennially underperforming Title I schools, requiring the school to dedicate a portion of its Title I budget to offering free tutoring services to students. In doing so, the SES provision has effectively created a tutoring market for low-income students, encouraging private educational vendors to provide tutoring services paid for by publicly-funded Title I dollars. To address parental choice in a new market-based education policy, this paper asks how the demand-side of the education market has responded to the availability of free tutoring services offered through the SES provision. Furthermore, this paper explores, through a series of empirical approaches, the characteristics of Chicago Public School students who participated in SES. As such, this paper presents important descriptive analyses of who chooses to participate in a new educational choice option. One of the primary objectives of NCLB is to facilitate student proficiency in mathematics and reading by the year 2014. Toward this end, NCLB introduced the SES provision to help increase student academic achievement in low-performing schools. Therefore, the extent to which SES has the capacity to impact student academic achievement largely relies on whether students choose to receive tutoring from an SES-approved provider as well as the characteristics of students who participate. However, the perceived under-utilization of SES nationally has been most recently described in a 2008 U.S. Department of Education sponsored report. The report notes that only about 17 percent of SES-eligible students participated in SES in the 2003-04 year (Gill et al., 2008, page XV). In addition, the report finds that participation rates vary by grade level (from 2.4 percent of eligible students participating in SES in grade 12 to 28.3 percent in grade 3) as well as by demographic characteristics. 2 2 From a survey of nine large urban school districts, the U.S. Department of Education report found that 10.1 percent of SES-eligible white students, 11.6 percent of SES-eligible Hispanic students and 16.9 percent of SES-eligible African- 3
This paper aims to shed light on the utilization of SES by identifying the characteristics of students who choose to participate in SES in Chicago. While the extent to which participation levels in SES are deemed adequate is subject to debate, this paper provides evidence to inform the important question of whether students who most need additional academic services (and who can potentially derive the most benefits from them) are receiving SES. In addressing educational choice in the context of the SES provision, this paper contributes to two areas of education policy research. The first is SES as a market-based choice program. Namely, this paper contributes to the growing literature on the SES provision and its role in redefining the institutional relationship between public education and private educational vendors. The second contribution is to contextualize SES choice in a conceptual model and empirically modeling the characteristics of students who participate in SES. In addition, this paper provides a unique perspective on educational choice in Chicago, and in doing so, aims to inform policymakers and education researchers about the demand-side determinants of educational choice. Ultimately, this paper attempts to clarify the role of student characteristics in the decision to participate in SES so that tutoring services may be provided more efficiently and effectively to low-income and minority Chicago school students toward the goal of improving student achievement and other noncognitive outcomes. While this paper does not specifically address the impact of the SES policy provision on student achievement outcomes, this paper provides among the first empirical evidence on the demand-side characteristics associated with participation in SES. 3 In doing so, this work contributes significantly by providing greater transparency around the characteristics of SESeligible students in Chicago participating in SES. This work also aims to inform the pending American students participated in SES in 2004-05. In addition, the report notes that 13.1 percent of SES-eligible students with limited English proficiency (LEP) and 14.6 percent of SES-eligible students with a disability participated in SES in 2004-05 (Source: Gill et al., 2008). 3 The only other study that I am aware of that explicitly model factors related to SES participation is Heinrich et al., 2008, which examine SES in the context of Milwaukee Public Schools. 4
reauthorization of the No Child Left Behind Act by characterizing both the extent to which SESeligible students in a large urban school district are participating in the federally-funded tutoring provision under SES and the factors contributing to the decision to participate in new forms of educational choice options. This paper proceeds as follows. I first provide the policy context by describing the Supplemental Educational Services provision. I then introduce a model of educational choice and relate this model to SES participation. I describe the data and variables to be employed in the empirical estimation, followed by the empirical strategy. I then present the empirical findings and conclude. SES as Policy Context The SES provision represents a new direction in education policy. Traditional market-based educational reforms (e.g. charter schools, voucher programs) rest on three pillars; these include: (a) a re-arrangement of public funding; (b) entrance/participation of private vendors in the market; and (c) parental choice among schooling options. The SES provision marks the next generation in market-based reform, as part of what has been referred to as the new educational privatization (See Burch, 2006). The SES provision represents a new form of educational privatization distinct from traditional choice options such as charter schools and voucher programs (see Burch et al., 2007 and Vergari, 2007 for a detailed description of the institutional design, policy assumptions and political economy of the SES provision). In particular, the SES provision extends the basic design of traditional market-based reforms by offering educational choice around additional academic services rather than providing the option for families to move from a public to a private school. Whereas charter schools and voucher programs require parents to choose between schooling options, SES offers families the option of additional academic instruction without supplanting the status quo schooling experience for students. In addition, SES represents a re-arrangement of 5
institutional relationships between public and private sector where public budgets and private sector decision-making are tied to policy changes. In the context of SES, state and local education agencies take on the additional responsibility of monitoring and evaluating tutoring providers, most of which are private entities outside of the public jurisdiction. This re-arrangement of institutional relationships through policy change leads to what has been identified as a shift from the state as provider to the state as regulator, or the new evaluative state. (Ball, 2006, page 132) The SES provision is a federally-mandated intervention for Title I schools that fail to meet adequate yearly progress (AYP) benchmarks for three consecutive years. The U.S. Department of Education defines SES as additional academic instruction designed to increase the academic achievement of students in schools in need of improvement and stipulates that these services must be provided outside the regular school day. 4 On the demand side of the SES market, any low-income student is eligible for receipt of a voucher for SES if he/she is enrolled in a Title I school that has not made AYP for three years or more. As the text of the SES provision explains, eligibility is not dependent on whether the student is a member of a subgroup that caused the school to not make AYP or whether the student is in a grade that takes the statewide assessments. 5 While the demand-side of the national SES market has become more responsive since 2002-03, the first year of implementation, participation rates remain modest. In 2002-03, approximately 600,000 Title I students were eligible to receive SES. Of these SES-eligible students, approximately 42,000 students (7 percent) received free tutoring services. By 2004-05, the number of eligible students tripled, to approximately 1.8 million students. In addition, participation rates increased to approximately 24 percent of SES-eligible students (approximately 446,000 students) (Gill et al., 2008). 4 Source: U.S. Department of Education, Supplemental Educational Services Non-Regulatory Guidance, June 13, 2005, page 1. 5 U.S. Department of Education, Supplemental Educational Services Non-Regulatory Guidance, June 13, 2005, page 3. 6
On the supply side of the market, SES providers may take the form of public- or privatesector organizations that are approved by the state education agency, such as public schools, charter schools, local education agencies, educational service agencies and faith-based organizations. Private-sector providers may either be nonprofit or for-profit entities. Services include tutoring, remediation, and other academic instruction. Nationwide, the number of approved SES providers increased from 997 in May 2003 (the first full year of implementation) to 2,734 in May 2005 approximately a threefold increase in the supply of SES providers. In addition, during the first year of SES implementation in 2002-03, 60 percent of SES providers were privately operated entities; by May 2005, the share of private providers in the SES market increased to 76 percent (Gill et al., 2008). Per-pupil funding for SES (or, the voucher amount) is drawn directly from district Title I budgets. In particular, a local education agency with at least one school required to offer SES must reserve a minimum of 20 percent of its Title I, Part A allocation, either by reserving 20 percent of total district Title I funds prior to making allocations to schools, or by adjusting Title I allocations to schools in order to set-aside the required funding. In terms of the total Title I funds dedicated to SES, the interaction between the number of students eligible nationwide, the number who participate in an SES program (e.g. the take-up rate), and the variation (across states, and within states across local school districts) in the maximum per-pupil expenditure for SES makes disentangling the exact amount of federal dollars allocated to SES difficult. In 2002-03, Title I funds totaled $10.4 billion; by 2004-05 (the most recent year that SES eligibility and participation data are available), Title I funds rose to $12.3 billion. 6 In 2004-05, approximately 10 percent of 6 Calculations of Title I allocations by the author. (Source: ESEA Grants to Local Education Agencies; U.S. Department of Education) It is interesting to note that 20 percent of total Title I allocations (assuming that 100 percent of Title I students were eligible to receive SES in 2004-05) equals $2.44 billion. However, based on the total number of SES eligible students in the same year (approximately 2.3 million) and the estimated average maximum per-pupil expenditure nationwide ($1610), a shortfall in SES funds available to provide tutoring services would result, amounting to approximately $1.3 billion. This shortfall represents a systematic under-funding of the SES provision. 7
Title I school districts had schools required to offer SES. Therefore, in 2004-05, approximately $246 million were available to fund SES nationwide. 7 However, this calculation likely underestimates the amount of Title I funds reserved for SES, as some districts (such as New York City, Chicago and Los Angeles) have more SES-eligible students than the average-sized school district. As a result, in 2004-05, the amount of Title I funds used to provide SES was approximately $692 million. 8 Furthermore, there is significant heterogeneity in voucher amounts both across states and within states, across districts, which may account for differential rates of SES participation at national and state levels. For example, across school districts in Illinois during the 2008-09 school year, the maximum per-pupil expenditure for SES (e.g. the voucher amount) ranged from approximately $720 to $3,330. 9 Conceptual Model of SES Choice A model of choice necessarily informs the set of factors that describe the underlying mechanism driving the decision to participate in SES. While the mechanism itself is unobservable, a conceptual model of behavior gives shape and dimension to the mechanism by describing the inputs to the choice decision. Toward this end, this analysis refers to the approach taken in prior studies to explicitly model the selection process into alternative schooling options (Manski, 1992; Lankford and Wyckoff, 1992; Goldhaber, 1996; Neal, 1997; Belfield, 2004). 7 To estimate the demand-side of the SES market, I multiplied the total Title I federal allocation ($12.3 billion) for 2004-05 by the approximate number of school districts with SES-eligible students (10 percent; source: Center for Education Policy, 2005); I then multiplied this result by the maximum SES set-aside of 20 percent. 8 I arrived at the total SES funds used in 2004-05 by deriving a weighted average maximum per-pupil expenditure for SES. Using district-level Title I allocation and maximum per-pupil SES expenditure data from the U.S. Department of Education, I first calculated a weighted average maximum per-pupil allocation for each of the 50 states, plus the District of Columbia and Puerto Rico. I did this by weighting each district in each state by its share of the total Title I expenditures provided to the state. I then averaged each state s per-pupil expenditure across the 50 states and the District of Columbia and Puerto Rico by weighting each state s share of the total amount of Title I expenditures nationwide to arrive at an average allocation nationwide. The national average per-pupil expenditure for SES in 2004-05 equals $1610. 9 Source: U.S. Department of Education, ESEA Title I LEA Allocations (www.ed.gov). 8
In the context of SES in Chicago, each family with a child eligible to receive SES is faced with a binary choice decision: C i = {SES, No SES}, where C i is the set of options available to family i. 10 As with the prior literature on school choice, I assume that each family, when faced with the decision to enroll their child in SES, will choose the option which maximizes their individual utility, U i, or the net welfare a family expects to receive by choosing to enroll their child in the SES tutoring provision. While we may consider the decision to participate in SES a family (or student) decision, in general, the schooling decisions that families and students make are influenced by a number of factors. The literature on the ecology of schooling suggests that the characteristics of microdomains such as home, school and community contexts, as well as the influence of peers, shape educational schooling decisions and youth academic and social outcomes (Brooks-Gunn et al., 1993; Epstein and Sanders, 2000). In an analysis of the supply and demand-side conditions influencing participation in SES, we would want to consider the role of not only student and school characteristics, but also community and peer characteristics that likely influence educational choice. However, for the purposes herein, we focus on the student level characteristics that may be related to the decision to participate in additional academic instruction. To add more structure to the model, assume that the utility that family i assigns to participation in SES (k=1 for SES participation, k=0 for no SES participation), U ik, is unobserved by the analyst. Furthermore, the utility to family i of SES participation level k is assumed to be a linear function of observable student factors (captured by the X ik vector) and an idiosyncratic disturbance term (ε ik ). Formally: 10 I use the term family to describe the decision making unit with respect to selecting into participating in SES. However, it is likely that parents of Elementary school students (grades 1-5) are the primary decision makers, while High school students may decide whether or not to participate in SES irrespective of their parents preferences. As such, in the empirical analysis I model the SES participation decision separately for students in the Elementary (grades 1-5), Middle (grades 6-8) and High (grades 9-12) school grades to account for the possibility that the decision-making unit (families, parents, students) differs across school levels. 9
(1) U ik X ik ik Each SES choice option (indexed by participation level k) available to families corresponds to some unobserved family utility. In particular, let U i0 be the utility to family i of choosing to not enroll their child in SES, and U i1 be the utility to family i of choosing to enroll their child in SES. Further, we observe the family s decision (C i ) whether to enroll their child in SES. Therefore, the choice that a family makes with respect to SES participation, C i, will be the realization of a family s idiosyncratic assessment of the relative utilities associated with participation (U i1 ) or nonparticipation (U i0 ) in SES. Let I i be the idiosyncratic assessment of utilities, or latent index, which determines whether or not family i selects their child into SES. Using equation (1), we can characterize the latent index in the following manner: (2) I U U X X ) ( ) i i1 i0 ( i1 i0 i1 i0 Therefore, if I i > 0, family i will choose to enroll their child in SES, and the analyst will observe C i =1. In contrast, if I i < 0, family i will choose to not enroll their child in SES, and the analyst will observe C i =0. This conceptualization of the SES choice decision is given empirical grounding by specifying the arguments that enter into U i. For the purposes herein, I assume the utility function for each family, U i, over the choice set k to be a function of student (S i ) characteristics and a vector of unmeasured (ε i ) characteristics that enter into the family s decision to participate in SES. Formally, (3) U ik = U(S i, ε i ) Consideration (by families) of the utility associated with additional academic instruction in the form of SES is likely related to three particular aspects of the inputs to the utility function. These include: (i) the extent of a student s need for additional academic services in light of his/her demonstrated cognitive ability; (ii) a student s motivation to succeed academically; and (iii) 10
information (likely provided by the school and possibly by the students peers) about SES, such as the type of tutoring services offered and the services that best meet the learning needs of the student. For clarity, the figure below summarizes these inputs to the SES decision. Figure 1. Inputs to the SES Decision Student s Academic Needs Student s Motivation SES Participation Decision Information about SES Each of these three inputs to the SES participation decision has implications for the subsequent empirical analysis. In particular, a student s cognitive achievement likely reflects whether that student is in need of additional academic instruction. A student s motivation is an unobserved character trait, but has been described by authors such as Heckman and Rubinstein (2001) as a measure of a student s ability set with implications for important educational outcomes. For our purposes, we consider that a student s motivation to succeed in school and to seek out additional academic instruction will be reflected in their school attendance and behavior. Finally, the information available to students will likely differ across school contexts; that is, students who attend the same schools will likely have similar information (from teachers and peers) available to them about SES. As such, we will consider the nested nature of SES choice decisions (within schools) to account for school-level differences in information about SES. Finally, this conceptual approach allows us to further refine our research inquiry into discrete (and testable) questions. In particular, are higher (or lower) achieving students more likely 11
to participate in SES? To what extent do students who are more (or less) motivated to receive additional academic instruction participate in SES? To what extent does variation at the schoollevel exist in the probability of students participating in SES? Data & Variables The data employed in this analysis were provided by the Office of Extended Learning Opportunities at Chicago Public Schools (CPS), and include both administrative and survey data. The administrative data includes student-level demographic and achievement data, information on student behavior (including student absences and disciplinary infractions) and SES eligibility and participation for the school years 2004-05 through 2007-08. The survey data are from the Student Connection Survey, which was administered to students in CPS (grades 9-12 in 2005-06 and grades 6-12 in 2006-07 and again in 2007-08) in an effort to develop an understanding of students perceptions of the school environment, and how the environment impacts student achievement and learning. The students responses were evaluated on four domains of the school environment, including school safety (physical and emotional), teacher expectations, teacher support and social and emotional learning. Table 1 summarizes the sample for each SES cohort year. The total number of SES-eligible students corresponds to CPS students who attend a school which has failed to meet adequate yearly progress for three consecutive years. Table 2 summarizes the longitudinal samples. For each column, the total number of SES-eligible students are those students who attended a school which was required to offer SES in each of the years indicated. For example, for the longitudinal sample 2004/05 2007/08, 79,096 students attended schools which offered them the opportunity to participate in SES for four consecutive years. The dependent variable in this analysis is the realization of the latent index described in the section above. In particular, for a given school year, we will observe either C i =1 or C i =0 for an 12
SES-eligible student. Note here that this variable provides information about the extensive margin of SES participation; that is, whether an SES-eligible student participated in even one SES tutoring session. 11 Recalling the SES utility function, I characterize the student-level factors into four dimensions, including: (a) student demographics; (b) cognitive achievement measures; (c) noncognitive measures; and (d) student perception of the school environment. Student demographic data (summarized in Table 3) include the student s grade, gender, race, disability status, lunch status and bilingual status. A student is labeled as disabled if they have an individualized education plan (IEP) in a given school year. A student s lunch status is characterized in one of three ways; either a student pays for lunch, receives free lunch or receives reduced-price lunch. 12 This variable will be used as a proxy for family income, given that a continuous measure of family income is not available. A student s bilingual status is characterized as never bilingual, formerly bilingual or currently in a bilingual program for a given school year. 13 For cognitive achievement data, prior to the 2005-06 school year, students in grades 1-8 in Chicago were tested in mathematics and reading proficiency on the Iowa Test of Basic Skills (ITBS); this exam was mandatory for students in grades 3 through 8. Beginning in 2005-06 and continuing up through and including the 2007-08 school year, students in grades 3 though 8 were tested in mathematics and reading proficiency on the Illinois Standards Achievement Test (ISAT). 14 In contrast, students in the high school grades (e.g. 9-12) do not take a common standardized test as do students in the elementary and middle school grades (e.g. 1-8). As such, I employ a few 11 Data on the intensive margin; e.g. the number of hours of tutoring a student received in a given school year, are unavailable. 12 The SES provision requires that only low-income students (e.g. those receiving free or reduced-price lunch) be SESeligible. 13 Data on a student bilingual status available for the 2005-06 school year; data on student lunch status available for the 2005-06 and 2007-08 school years. 14 Beginning in the 2002-03 school year, the ISAT became the high-stakes test in Illinois (e.g. test scores on the ISAT exam were used to determine whether a school met proficiency benchmarks necessary for making adequate yearly progress). However, only students in grades 3, 5 and 8 took the ISAT for AYP purposes prior to the 2005-06 school year. 13
cognitive achievement measures in this analysis. The first is a high school student s cumulative grade-point-average (GPA), which is the end-of-year GPA for all high school classes taken up through and including the most recently completed academic semester. In addition to the cumulative GPA, I look at a student s fall and spring GPA in the year prior to eligibility for SES; the fall (spring) GPA is the student s GPA from courses taken only in the fall (spring) semester of the school year. I also examine the high-stakes exam for high school students, the Prairie State Achievement Examination (PSAE), which is given to students in 11 th grade. In particular, for test scores (ITBS, ISAT and PSAE) and GPA, prior-year scores are standardized within school year (e.g. Z-scores for the prior-year ITBS or ISAT are created for elementary and middle school students within a school year; Z-scores for the prior-year PSAE, and the cumulative, fall and spring prior-year GPA are created for high school students for a given school year). The use of prior-year cognitive achievement results are grounded in a particular theory of action; that is, a student s performance in the prior year may be such that a student (and the student s family) decides to participate in additional academic instruction such as is provided by SES. The direction of this effect (e.g. are lower achieving students in the prior academic year more likely to take-up SES than higher achieving students) is an empirical question that will be tested herein. The non-cognitive measures include student attendance and behavior. Student attendance is measured as the total number of school absences a student had in a given school year; that is, the sum of fall and spring semester absences. 15 Data on school attendance is only available for high school students (grades 9-12). Total absences (fall plus spring semester) are standardized within school year. Further, the standardized prior-year absences are included in the empirical analysis, guided by a similar logic underlying the inclusion of standardized achievement results. That is, we 15 Data on absences during the 2006-07 school year are available for only the spring semester. 14
explore the extent to which a student s school attendance in the prior year is related to the likelihood of participating in SES in the subsequent school year. Again, the direction of this relationship is testable within the context of this analysis. In addition to school absences, student disciplinary infractions are included as measures of student non-cognitive performance. Data include the total number of disciplinary infractions for each student, in all grades. Data on disciplinary infractions is available for the 2005-06 and 2006-07 school years. Moreover, there is heterogeneity in the severity of disciplinary infractions. In Chicago, each disciplinary infraction corresponds to a level of severity, where the severity of misconduct is categorized on a scale from 1-6. 16 However, given that the data on disciplinary infractions is aggregated by school year for each student, data on severity of disciplinary infraction include only the level of the student s most severe disciplinary act. This variable is included as a control for the heterogeneity across students in the severity of infractions. For example, when comparing two students with the same number of disciplinary infractions in a given year, it is necessary to control for the severity of infractions. The standardized prior-year disciplinary infractions are included in the empirical analysis. Similar to student attendance, consideration is given to whether a student s behavior in the prior school year motivates participation in SES in the subsequent school year. From the Student Connection Survey, data on how students in the middle and high school grades perceive various dimensions of their school environment and schooling experiences are included in the analysis. As mentioned, there are four dimensions on which the survey measured 16 From the Chicago Public Schools Policy Manual, Student Code of Conduct for the Chicago Public Schools for the 2007-08 School Year, the severity of disciplinary infractions include: (a) level 1: Inappropriate student behaviors in the classroom or on school grounds; (b) level 2: Student behaviors that disrupt the orderly educational process in the school or on the school grounds; (c) level 3: Student behaviors that seriously disrupt the orderly educational process of the Chicago Public Schools; (d) level 4: Student behaviors that very seriously disrupt the orderly educational process of the Chicago Public Schools; (e) level 5: Student behaviors that most seriously disrupt the orderly educational process of the Chicago Public Schools; and (f) level 6: Illegal student behaviors that most seriously disrupt the orderly educational process of the Chicago Public Schools. Please see the CPS policy manual for detailed description of the types of infractions/behaviors included in each level of misconduct. 15
student perceptions; they include a student s perception of: (a) school safety; (b) teacher expectations; (c) teacher support; and (d) social and emotional learning. Each dimension is an aggregate of a series of survey questions, which were then scaled to reflect whether the student perceived the school to be excellent, adequate or in need of improvement on each of the four dimensions. In particular, the school safety dimension captures both how physically safe as well as how emotionally safe students feel in the school. The teacher expectations dimension captures the extent to which students perceive that teachers and other adults in their school encourage them to think, work hard, do their best, and connect what they are learning in school to life outside school. The teacher support dimension captures how much students feel listened to, cared about, and helped by teachers and other adults in the school. The fourth dimension, social and emotional learning, is a measure of the level of social capital in the school, and captures students perception of their peers social and problem-solving skills. For the purposes of the empirical analysis, each of the four dimensions are dichotomized, such that a value of one is given to a student who perceives the school to be adequate or excellent on a given dimension, and a value of zero if the student perceives that the school needs improvement on this dimension. The data on student survey responses provide a deeper understanding of student interactions with their school environment, including both their teachers and their peers. Furthermore, none of the previous empirical studies of selection into market-based educational options models the context of schooling in the manner in which the survey data allows. Given that there is likely significant heterogeneity both across and within schools in how students perceive their schooling experiences, this data provide the opportunity to explicitly model how school context is related to both student and school level factors. And, in turn, the extent to which school setting relates to SES participation. 16
Empirical Analysis A series of empirical methods are employed to exploit the unique structure of this dataset. These methods include: (a) descriptive analysis of patterns of enrollment in SES, by student characteristics; (b) cross-sectional (cohort) analysis of the student-level factors related to SES participation; and (c) longitudinal analysis which exploits repeated measures of SES participation among a sub-sample of students eligible for SES between the 2006-07 and 2007-08 school years. Descriptive Evidence of Differential Selection This analysis rests on the premise that characteristics of SES-eligible CPS students are related to participation in the SES provision. As such, it is critical to first establish the extent to which differential selection exists across the four cohort years under consideration. Table 4 presents evidence of differential selection into SES in Chicago. In particular, this table captures the extent to which aggregate differences in cognitive and non-cognitive measures exist among SESeligible students in each cohort year that did and did not participate in SES. For elementary and middle school students (e.g. grades 1-8), it appears that, in each cohort year, SES participants have lower prior-year math and reading achievement than non-participants. The difference between SES participants and non-participants in math is between 0.36 and 0.43 standard deviations, and, for reading, non-participants score on average between 0.36 and 0.54 standard deviations higher than participants. Among high school students who took the high-stakes PSAE test in the prior year (e.g. students who were in 11 th grade in the prior school year but who were eligible for SES in the subsequent school year), the results are quite similar to what we find for middle and elementary school students. In almost all instances (PSAE math and reading differences by cohort year), SES participants performed below non-participants on the high-stakes high school exam. 17 On the math 17 Only the PSAE reading score for the 2005-06 cohort was not statistically different for SES participants and nonparticipants. 17
portion of the PSAE, non-participants scored between 0.09 and 0.44 standard deviations higher than SES participants, and between 0.16 and 0.35 standard deviations greater on the reading portion of the PSAE. However, while the ISAT and ITBS exams provide a snapshot of aggregate mean performance for SES-eligible students in the elementary and middle school grades, the PSAE is limited in that we can only assess mean differences for a single grade within a cohort year. As such, we look at differences across participation status, within cohort year, by student grade-pointaverage (GPA). Three measures of prior-year GPA are used, including a student s: (a) cumulative GPA up through and including the most recent semester prior to the cohort year under consideration; (b) fall semester GPA, which considers only those courses taken in the fall semester of the year prior to the cohort year under consideration; and (c) spring semester GPA, which considers only those courses taken in the spring semester of the year prior to the cohort year under consideration. For the 2004-05 and 2005-06 cohorts, there is no evidence that SES participants and non-participants differed on their prior year cumulative, spring or fall GPAs. However, we find opposite results with respect to the 2006-07 and 2007-08 cohorts. In particular, for the 2006-07 cohort, SES participants had lower prior year cumulative, spring and fall GPAs relative to nonparticipants, on the magnitude of.10.15 standard deviations. These findings indicate that lower achieving students with respect to high school course performance were more likely to participate in SES than higher achieving students, on average. However, for the 2007-08 cohort, we find a different pattern with respect to course performance between SES participants and non-participants. In particular, while there is no statistically significant difference in cumulative GPA between participants and non-participants, SES participants perform, on average, approximately 0.10 standard deviations higher than non-participants in terms of fall and spring semester-specific GPAs. The extent to which these different patterns are related to structural changes in the provision of SES 18
(e.g. school or district-level policies aimed at recruiting students based on characteristics of their course performance) or supply-side changes (e.g. SES providers whose curriculum may be correlated with course performance) is beyond the scope of this analysis; however, a better understanding of what factors drove this change in relative in-school performance among SES participants and non-participants across cohort years is worth exploring. Table 4 also provides aggregate mean differences between SES participants and nonparticipants in terms of non-cognitive performance measures. Evidence on the relationship between school absences and SES participation (among high school students) suggests variation in the composition of students across cohort years. In particular, in only two of four cohorts is the difference in prior year absences significantly different between SES participants and nonparticipants, and the magnitude (and level of statistical significance) varies across these two cohorts. For the 2005-06 cohort, SES participants had fewer prior year absences than non-participants, on the magnitude of 0.033 standard deviations; this difference was marginally statistically significant (p-value = 0.088). For the 2007-08 cohort, SES participants had fewer prior year absences; in this cohort year, the magnitude of the difference was 0.177 standard deviations, and highly statistically significant (p-value < 0.000). In terms of the relationship between prior year disciplinary infractions and SES participation, the evidence appears rather consistent. For the two cohorts in which prior-year absences were available, the magnitude, direction and statistical significance of the relationships are very similar. For the 2006-07 cohort, SES participants had fewer prior-year disciplinary infractions than non-participants, the magnitude of this difference was 0.18 standard deviations (p-value < 0.000). Similarly, for the 2007-08 cohort, SES participants had fewer prior-year disciplinary infractions than non-participants, the magnitude of this difference was 0.19 standard deviations (pvalue < 0.000). 19
From the descriptive evidence on differential selection into SES by cognitive and noncognitive measures, a few trends emerge. First, for all high-stakes tests (ITBS, ISAT, PSAE) in math and reading across almost all cohort years, SES participants score lower on average than nonparticipants in the year prior to SES-eligibility. As noted, the magnitude of these differences is both large and highly statistically meaningful. Second, there appears to be conflicting evidence on the relationship between high school coursework grades and SES participation. Third, there is strong evidence to suggest that student behavior is related to SES participation, with some evidence that school absences are related to SES participation among high school students. Therefore, these descriptive findings justify a more detailed and empirically rigorous analysis of selection into SES. I now proceed to lay out the empirical methods used to model selection to account for the manner in which student and school factors are jointly related to SES participation. Cross Sectional/Cohort Analysis The descriptive analysis explored the extent to which factors considered relevant to the selection process, namely, a student s cognitive and non-cognitive outcomes, were differentially related to SES participation. However, the descriptive analysis is fundamentally unconditional; that is, it summarizes aggregate means of specific dimensions of student characteristics by SES participation status. In order to estimate the differential probability of selection into SES, the proceeding analysis employs observed student characteristics (as detailed in the Data & Variables section) to estimate a conditional probability model. I begin first by specifying the conditional probability of participation as a mean response function. Recalling the model of SES choice from above, we are interested in estimating the probability that student i participates in SES in a given cohort year. For the purposes of modeling the probability of participating in SES, we consider the nested structure of the data, and at this point introduce a j subscript. That is, the nature of the data is such that each student i is nested within a 20
specific school j. The conditional expectation, or probability, of participating in SES may be written as: (4) E(C ij X ij ) = Pr(C ij = 1 X ij ) = μ ij, where C ij is the realization of the latent index variable (I i ), and takes on a value of 1 if student i in school j participates in SES and a value of 0 if student i in school j does not participate in SES in a given cohort year; X ij is a vector of observed student characteristics; Pr(.) is the conditional probability that student i in school j participates in SES in a given school year; and μ ij is the mean response or average value of C ij among all observations (e.g. the student-specific predicted probability of participating in SES). For the purpose of estimating the conditional probability, I employ a logit link function, which links the linear predictor ( X ) to the mean response parameter (μ ij ). The model is specified as: ij (5) logit(μ ij ) = η ij = ij log = X ij 1 ij Having established the basic framework for estimating the conditional probability of participating in SES, we now consider modeling the SES decision. In doing so, this analysis considers the heterogeneity across schools in the types of SES providers offering tutoring services and the extent to which teachers and other school personnel within schools provide information to students about the availability of SES. If we were to consider the effect of schools on the SES participation decision fixed, we could simply estimate the conditional probability of participating in SES as a generalized linear model (GLM) with the inclusion of school fixed effects. Instead, we employ a recent development in hierarchical modeling to estimate the conditional probability of participating in SES. In particular, we use an adaptive centering approach with random effects (Raudenbush, 2009). This approach centers all student-level covariates at the school-level, 21