Alternative Certification in the Long Run: A Decade of Evidence on the Effects of Alternative Certification in New York City

Transcription

1 Alternative Certification in the Long Run: A Decade of Evidence on the Effects of Alternative Certification in New York City Don Boyd,* Erin Dunlop,** Hamp Lankford,* Susanna Loeb,*** Patten Mahler,** Rachel O'Brien*** & Jim Wyckoff** *University at Albany, **University of Virginia, ***Stanford University Draft 1/13/2012 We are grateful to the New York City Department of Education and the New York State Education Department for the data employed in this paper. We appreciate financial support from the Noyce Foundation and the National Center for the Analysis of Longitudinal Data in Education Research (CALDER). CALDER is supported by IES Grant R305A to the Urban Institute. The views expressed in the paper are solely those of the authors and may not reflect those of the funders. Any errors are attributable to the authors.

2 Many school districts struggle to hire qualified teachers, especially in special education, math, English as a second language (ESL), and science. However, over the last ten years the landscape of teacher supply has altered dramatically as a result of the substitution of alternatively certified teachers for unlicensed teachers (Feistritzer, 2008). An increasing body of research describes the characteristics of alternatively certified teachers and compares both their effectiveness on value-added outcomes for students and their attrition to the unlicensed teachers they replaced as well as to teachers from other pathways (Boyd et al., 2006, 2008, 2009, forthcoming; Constantine et al., 2009; Decker et al., 2004; Feistritzer, 2008; Grossman and Loeb, 2008; Kane et al., 2009; Xu et al., 2009). This research finds that alternatively certified teachers disproportionately teach in high needs schools and subjects and are more likely to leave their initial schools and districts than are traditionally prepared teachers. While results vary, these studies also tend to find that the students of teachers who enter teaching through highly-selective alternative routes experience comparable, or in some cases somewhat better, math achievement gains than the students of teachers from traditional preparation pathways teaching in the same schools; and comparable, or in some cases somewhat worse, achievement gains in English language arts (ELA) than the students of teachers from traditional preparation pathways. 1 The widespread hiring of alternatively certified teachers is a relatively recent phenomenon, and, as such, the research on these teachers has focused almost exclusively on teachers with three years of experience or less. In addition, these studies often have considered alternative routes that have been in operation for only a few years. Our understanding of the longer-run effects of the introduction of alternative routes, particularly highly-selective alternative routes is weak. In this paper we estimate longer-run effects of alternative certification in New York City, employing a detailed database of all teachers in New York City from through New York City provides a good opportunity for understanding the long-run effects of alternative routes to certification, as the school district has employed large numbers of alternatively certified teachers for the past ten years. In Fall 2001, the New York City Department of Education (NYCDOE) hired its first cohort of New York City Teaching Fellows (NYCTF). With program growth, in NYCDOE hired nearly 2500 NYCTF teachers, 1 There has been far less systematic research on the effectiveness of teachers from less selective alternative programs, though many teachers enter teaching through these routes. 2

3 constituting about 30 percent of all new hires. Teacher for America (TFA) teachers had taught in New York City prior to 2000, although the number of teachers hired per year had been relatively small, typically less than 100 teachers per year. Hiring of TFA teachers also grew so that for the school year more than 300 new TFA teachers were hired. As a result of this hiring, New York City now employs a large group of teachers who entered through alternative certification pathways and who began their teaching careers between five and 10 years ago. Data on these teachers and all other teachers in the district allow us assess the long-run implications of alternatively certified teachers. We examine four broad research questions: 1. How have the schools and specialties in which alternative route teachers work changed over the 10 year period? 2. How have the characteristics of teachers who obtained certification through alternative routes changed? 3. To what extent has the effectiveness of new teachers changed since the since initial increase in alternatively certified teachers at the beginning of the sample period? 4. How has the attrition behavior of alternative and traditionally certified teachers changed? To answer questions 1, 2 and 4, we examine changes in the background characteristics of alternatively certified teachers, differences in the positions that alternatively certified teachers take (e.g. school level, subject matter, and special populations), and differences in turnover and transfer behavior. To answer the thrid question, we compare new teachers to various groups of experienced teachers, whose effectiveness remains relatively constant over the ten years of our analysis. We find: The New York City Teaching Fellows (NYCTF) program recruits teachers with higher certification scores, higher SAT scores and who attended more competitive colleges than do college recommending teacher preparation programs. This difference has remained relatively constant over the study period NYCTF teachers are considerable less racially diverse than the uncertified teachers they replaced; however, they are more racially diverse than any of the other large pathways supplying teachers to New York City and this feature, as well, has remained constant over the study period. The role of the New York City Teaching Fellows has changed substantially since its inception in 2000 from supplying mostly Childhood Education teachers to difficult-to- 3

4 staff schools to a focus on the difficult-to-staff areas of special education, mathematics, ESL, and science, across a somewhat broader range of schools. What has not changed over the decade, is that the NYCTF program has served as the supplier of last resort for NYC teaching vacancies. Throughout the time period, the students of NYCTF teachers show average achievement gains that are approximately equal in math and a bit lower in English language arts than students of college recommended teachers. They have roughly equal gains in English language arts and somewhat lower gains in math than students of Teach for America teachers. Teachers from all routes show meaningful improvements in effectiveness over the first four years of their careers in both math and ELA. There is also some evidence that NYCTF teachers improve more with experience for English language arts teaching than do teachers entering through college recommending pathways. We find no evident difference in returns to experience for teaching math across teachers from different routes. There is been little systematic change in the effectiveness of recent entering cohorts of teachers relative to a comparison group of experienced teachers compared to earlier cohorts. This pattern is consistent across teachers from all routes in both math and ELA and is robust to alternative definitions of the comparison group of experienced teachers. Recent cohorts of NYCTF teachers are less likely to exit teaching in NYC than earlier cohorts, although they remain more likely to do so than traditional preparation teachers. The exit behavior of Teach for America (TFA) teachers has not changed over time and is substantially higher than either NYCTF or traditional preparation teachers, except after their first year of teaching. High value-added NYCTF teachers are much less likely to exit than their low valueadded NYCTF colleagues, although still somewhat more likely to exit than high valueadded CR teachers. Low value-added NYCTF teachers are much more likely to exit than their low value-added CR peers. BACKGROUND In the years prior to 2000, teacher hiring and retention in New York City was bleak. As evidence of New York City's difficulty in hiring, from at least as early as through roughly half of all new teachers were temporarily licensed (uncertified). Other measures of teacher qualifications were also notably weak. For example, 25 percent of newly hired teachers in had failed the New York State general knowledge certification exam on the first taking, 26 percent had attended undergraduate institutions rated by Barrons as uncompetitive, and, on average, newly hired teachers had average math and verbal SAT scores of 466 and 477, respectively, which is approximately the 30 th percentile of SAT takers. 4

5 New York City also had a weak record of teacher retention, especially in the most challenging schools and among their most qualified teachers. For example, between 1996 and 2002, 20 percent of new teachers in the top quartile on the certification exam left high-achieving schools following their first year, while teachers in low-achieving schools left at an even higher rate of 34 percent. By contrast, 14 percent of teachers in the bottom quartile on the certification exam left high-achieving schools after one year, and 17 percent left low-achieving schools. 2 A number of reforms beginning in 2000 dramatically changed the recruitment and retention of teachers. Teacher compensation increased, especially for entering teachers, As an example, between 2000 and 2008 the salary of starting teachers with a BA jumped by more than 35 percent from $33,186 to $45,530. The district also focused on training and hiring effective school leaders, including developing its own leadership academy, and it enhanced financial incentives and supports for teachers, and made human resource processes more transparent and tied to measures of performance. For example, in 2006, the NYCDOE introduced the Lead Teacher Program, which provides teachers identified as excellent with a $10,000 supplement to mentor and coach other teachers.. Arguably one of the more dramatic changes at the beginning of the 21 st century was the series of reforms in policy and practice that led to the development of the New York City Teaching Fellows (NYCTF). In 1998 the New York State Board of Regents passed regulations ending the use of temporary license teachers by fall of In response, the New York City Department of Education (NYCDOE), working with The New Teacher Project, developed and implemented the NYCTF program in Also in 2000, the New York State Board of Regents created a certification pathway for alternative certification, allowing both NYCTF and TFA teachers to receive certification. The NYCDOE built the NYCTF program into a source for recruiting between 20 and 30 percent of all new teachers. Participants in these alternative-route programs including NYCTF and TFA are expected to complete two hundred hours of pre-service training and pass the Liberal Arts and Science Test (LAST) and the relevant Content Specialty Test (CST) before entering the classroom. These teachers are issued Transitional B certificates, good for three years, following the introductory 2 D. Boyd, H. Lankford, S. Loeb, and J. Wyckoff, Explaining the Short Careers of High-Achieving Teachers in Schools with Low-Performing Students, American Economic Review Proceedings 95 no. 2 (2005):

6 component. As teachers of record, they are expected to enroll in teacher education programs at partner colleges to fulfill certification requirements. Participants enrolled in alternative-route programs must fulfill the same requirements as all other candidates for teaching certificates; thus, by the end of their programs, they have completed a similar set of courses to those taken by graduates of college-recommended programs. However, the costs of entering teaching through an alternative route are substantially less for the individual teacher than the costs of traditional university-based teacher preparation, both because alternative-route teachers earn a salary throughout their training and because, in the Teaching Fellows program and Teach for America, teachers pay a reduced rate for the coursework taken. Conversely, the cost to the City is higher for these teachers because of the subsidized education. Teach for America recruits teachers for high-poverty schools in a number of districts around the country. They recruit teachers nationally, targeting recent graduates of elite colleges and universities; the recruitment process is exceptionally selective. Once corps members are selected, they must attend a summer training institute, run by Teach for America, prior to being placed in a classroom. The pre-service curriculum is designed by TFA but once corps members begin teaching, they continue to take courses with a local partner university. TFA requires a twoyear commitment; at the end of this period, corps members earn certification, and many also earn a master s degree. The NYCTF program is one of the largest alternative-route programs in the country. Prior to entering the classroom as teachers of record, Fellows complete an introductory component, usually offered in the summer, which includes some time in local classrooms. The courses are taught by instructors at the partner universities. Once Fellows begin teaching, they continue to take classes at their partner institution. Most Teaching Fellows complete their programs within two years. Teaching Fellows are generally older than TFA corps members, and approximately 20 percent of Fellows have completed graduate degrees. Teaching Fellows are typically placed in shortage subjects and schools and in the last few years are more likely to teach math, science, and special education than childhood education. While TFA and the Teaching Fellows are the focus of this article, they are not the only alternative routes serving New York City. For example, the Teaching Opportunity Program (TOP) is a collaborative initiative between the City University of New York (CUNY) and the NYC Department of Education to produce middle and high school math, science, and Spanish 6

7 teachers. Participants in TOP also take part in an intensive summer program run by a CUNY campus that includes experiences in local schools. Once they enter the classroom, they continue to take courses at CUNY that count toward both their certification and master s degrees. TOP participants generally complete their requirements for certification and an master s degree in two to three years, after which they are committed to teaching in NYC public schools for an additional two years. The distinction between alternative and traditional can be quite blurry. For example, many participants in traditional graduate programs in teacher education apply for an internship certificate when they have completed sufficient coursework. With this certificate they are able to become the teacher of record in the classroom, earning a salary while they complete the rest of their program and obtain a master s degree. The distinction in the nature of programs and entry into teaching between traditional and alternate routes is muddled in other ways. As a result, in this analysis, we consider only three groups of teachers: those entering teaching through college recommended routes, those entering through the New York City Teaching Fellows program, and those entering through Teach for America. Research on the effects of teacher preparation programs with reduced requirements suggests that such programs increase the supply of teachers and change the composition of the teacher workforce (see McKibbon 1998; Shen 1999). Although there is wide variation in the qualifications of teachers recruited to various alternative certification programs, NYCTF and TFA are both very selective, recruiting teachers who score highly on measures of academic ability such as the SAT, and teacher certification exams (Boyd et al, 2006, forthcoming). A substantial body of research has estimated the effectiveness of early career TFA teachers in comparison to other teachers in their school as well as to traditionally certified teachers (see for example, Henry et al., 2011; Boyd et al., 2006, forthcoming; Hannaway et al., 2009; Decker et al., 2004). Using a variety of methodologies, on average, these studies find that TFA teachers are somewhat more effective at teaching math especially in the upper grades. Effectiveness is measured by the achievement gains of their students. While a large literature speaks to the effectiveness of TFA teachers, far fewer studies have looked at the effectiveness of other alternative routes. Constantine, Player, Silva, Hallgren, Grider and Deke (2009) examine a broad set of alternative route programs finding no discernible difference in the effectiveness of alternative and traditional route teachers, In their study, the 7

8 alternative routes were quite different from the NYCTF and TFA with few differences between the alternative and traditional route teachers in their average scores on college entrance exams, the selectivity of the college that awarded their bachelor s degree, or their level of educational attainment. As described above, the extant literature has assessed the effectiveness of alternative routes, particularly TFA, at one point in time or averaged over a set of years. In addition, they have focused on teachers almost solely in their first two years of teaching. None of the studies has described the development of these programs over time or the effects of alternative route teachers over a longer period of time. These longer-run analyses are the goals of this paper. In considering how alternative certification programs might change over time a number of factors may come into play. First, alternative routes were developed initially, as described above, to fill holes in the district s ability to staff its teaching positions with certified teachers. When the NYCTF program began, approximately half of all new teachers were uncertified and, given the new policy that required the district to hire only certified teachers, the need for teachers that met the qualification was spread across many specialties, particularly in the most difficult to staff schools. In the ensuing years, both the district and local teacher education programs have had time to respond to the new demands and the needs for alternative certified teachers may have changed. The first question that guides this analysis is whether the schools that hire alternatively certified teachers and specialties for which they are hired (e.g. elementary education, math, science, special education) have changed over time. As part of this analysis, we ask whether the specialties for which alternative route teachers appear to be more effective (e.g. middle school math vs. elementary school ELA) have increased in their relative share of alternative route teachers. The background characteristics of alternative route teachers may also have changed over time. These changes could be driven by supply or demand. As the NYCTF and TFA learn which types of candidates for their program are more likely to succeed, they may focus their recruitment more on those individuals. On the other hand, as the economy has changed, the supply of different types of workers interested in teaching may have changed. We ask simply whether the observable characteristics of teachers entering through different routes have changed over the decade. 8

9 Early studies of TFA and the NYCTF, identified high turnover rates of these teachers, particularly of TFA teachers. This higher attrition is expected for TFA given the explicit two year commitment. Yet, there are a number of reasons to think that turnover behavior may have changed over time. First, TFA in particular, has invested in supports for new teachers. Second, the district itself has invested in compensation and teacher supports that could affect the retention decisions of teachers. Third, changes in the economy, especially the recent financial crisis, likely have changed the work opportunities for current teachers in alternative occupation. Finally, other district policies, including those closing the lowest performing schools and opening new small schools, training and recruiting school leaders for those and other schools, and replacing seniority transfer rules with mutual consent hiring, may have affected the career decisions of teachers. For example, with mutual consent it is now easier for highly effective but lowexperience teachers to move between schools. Our third question asks whether the attrition and transfer behavior of teachers from different routes changed over this time period. Finally, the effectiveness of teachers from different routes may have changed over time. Both the NYCTF and TFA spend substantial resources recruiting and selecting teachers. As these programs gain experience, they may be able to target their recruitment and selection towards more effective teachers. The college recommended routes may also have improved over this time period as the alternative routes, the new state and federal policies, and the accountability at that district level all created incentives for improvement. For our final question we ask both whether the relative effectiveness of teacher from different routes changed and whether the overall effectiveness of new teachers improved over the period in question. DATA For this analysis we employ a range of linked data files from NYCDOE and from the New York State Education Department (NYSED): Individual-level administrative data characterizing the backgrounds, qualifications, and career histories of all NYC public school teachers ( ); Student-level achievement test results for grades 3-8 in math and ELA linked to the teaches who taught these students ( ); Administrative and other data characterizing the schools in which teachers teach ( ); 9

10 Data on the first program path of teachers ( ). RESULTS 1. How have the schools and specialties in which alternative route teachers work changed over the 10 year period? As shown in Figure 1, the number of temporary license (uncertified) first-year teachers fell from 5000 teachers in 2000, to virtually zero by This transition was accomplished by the creation of the NYCTF, increases in the number of teachers from traditional teacher preparation programs and a smaller increase in Teach for America (TFA) teachers. In 2004, only four years since its inception, NYCTF supplied more than 2,500 teachers. Figure 1. Number of First-Year Teachers by Pathway and Year 6,000 5,000 4,000 3,000 2,000 1, CR NYCTF TFA Temp Lic Other Path Note: Other Path includes Modified Teaching Licensed, Transitional B Licensed, Temporary Licensed, and Individual Evaluation Teachers. The changes in the pathways through which teachers were recruited affected the attributes of the NYC teaching workforce. We focus on a few of the more striking differences below. The appendix includes more detailed tabulations. Certification Area. Among the most remarkable changes has been the evolution of the recruitment goals of NYCTF program over time. As shown in Figure 2, at its inception the NYCTF program was dominated by teachers whose certification was in Childhood Education (elementary school teachers). Over two-thirds of NYCTF teachers where certified in Childhood 10

11 Education, which comprised about 30 percent of all teachers being hired by NYCDOE with this certification area (Appendix Table A7). However, this quickly changed so that by 2006 fewer than 15 percent of all NYCTF teachers were Childhood Education certified and NYCTF teachers accounted for just over 10 percent of all Childhood Education certified teachers hired that year. Figure 2. Percent of Teachers Certified in Childhood by Pathway and Year Percent CR NYCTF TFA Other Path Note: Other Path includes Modified Teaching Licensed, Transitional B Licensed, Temporary Licensed, and Individual Evaluation Teachers. The NYCTF program shifted to focus its efforts on supplying teachers in key shortage subjects, such as mathematics, science, special education and English as a second language. As shown in Figures 3a-d, NYCTF became the dominant source of supply for teachers in each of these certification areas. By 2006 NYCTF was supplying about 60 percent of all new math certified teachers, 35 percent of teachers certified in science, 50 percent of ESL teachers, and 50 percent of special education certified teachers. In many respects, NYCTF has become the supplier of last resort to difficult-to-staff subjects in NYC. As might be expected, this change in the focus of teacher certification was accompanied by other changes in the nature of teacher placements. We take note of two in particular--the grade assignment of teachers and the poverty status of schools to which they are assigned. The change in certification areas inevitably led to a change in school assignments. In 2002, 68 percent of NYCTF teachers were assigned to elementary schools; by 2010 that figure had fallen to 22 percent. The decline in elementary teacher assignments is roughly matched by equal 11

12 increases in assignments to middle and high schools. Teachers from traditional teacher preparation programs are disproportionately assigned to elementary schools (66 percent in 2010) 12

13 Figures 3a-d. Percent of Teachers Certified by Area, Pathway and Year Percent Math CR NYCTF TFA Other Path Percent Science CR NYCTF TFA Other Path Percent ESL CR NYCTF TFA Other Path Percent Special Education CR NYCTF TFA Other Path 13

14 while TFA teachers are almost exclusively assigned to elementary and middle schools. This has implications for the value-added analysis because the lack of sequential standardized achievement tests in high school preclude value-added estimates for these teachers. Another, less obvious, implication of the change in recruitment focus is the distribution of NYCTF teachers across the distribution of schools by poverty status of the students. As shown in Figure 4, between 2002 and 2010, the percentage of NYCTF teachers working in the poorest quintile of schools declined from 36 percent to 23 percent with a nearly equal increase in the percentage of teachers working in the least poor 40 percent of schools. 3 By 2010, it is still the case that NYCTF teachers disproportionately serve in the poorest 40 percent of schools, but do so less frequently than had been the case in the early years of the program. This coincides with Figure 4. Percentage of Entering NYCTF Teachers by the Poverty Quintile of the School in which they Teach, Percent of Teachers Quintile 1&2 Quintile 5 the shift in focus from placements in poor elementary schools to placements in difficult-to-staff subjects, especially special education, ESL and mathematics. For example, in 2004 only 29 percent of special education teachers and 25 percent of math teachers taught in the poorest quintile of schools. The shift in NYCTF recruitment focus reflects the substantial increase in hiring in these shortage subjects by NYCDOE since Special education is a good example. 3 This change should not be over emphasized as the difference in poverty among NYC schools is not great. As is discussed below, the average first year NYCTF teacher had 96 percent of students eligible for free or reduced price lunch in By 2010, that figure had declined to 89 percent. 14

15 More than twice as many special education certified teachers were hired in each year since 2003 as were hired from 2000 to College Recommended teachers filled some of this difference; however, over 80 percent of the difference was filled by NYCTF teachers (See Appendix Table A11). A similar, but less dramatic, increase occurred for math and ESL. 2. How have the characteristics of teachers who obtained certification through alternative routes changed? Academic Ability. The NYCTF program has consistently recruited teachers with strong credentials as measured by tests of academic ability (Appendix Table A6) and the competitiveness of their undergraduate colleges (Appendix Table A1). NYCTF recruits teachers who on average score consistently better than teachers entering NYC public schools through the traditional preparation programs (College Recommended (CR)) or through the "Other" pathways group and consistently somewhat worse than those entering through TFA. This pattern is replicated in the Barron's rankings of the undergraduate colleges of teachers. About a third of NYCTF teachers graduated from the most competitive colleges, while about 12 percent of CR teachers and more than 60 percent of TFA did so (Appendix Table A1). School Placements. The first teaching assignment of teachers from different pathways varies substantially, as might be expected from the differing goals associated with the pathways. NYCTF and TFA teachers are much more likely to teach students who are poor, Black or Hispanic, have been suspended from school, and who have lower math and ELA achievement test scores than CR teachers (Appendix Table A17). For example, in 2010, the students of an average first-year NYCTF teachers in grades 3-8 scored 28 percent of a standard deviation below those of the average traditional teacher preparation teachers and about 0.04 standard deviations above the students entering the average first-year TFA teacher's classroom. While there is some variation, similar patterns exist across other measures and over time. In short, NYCTF and TFA teachers have consistently been assigned to what appear to be more challenging classrooms. Racial/Ethnic Composition of Teachers. In 2000, 77 percent of first-year teachers who were black were entering teaching with temporary licenses. As seen in Figure 5, the percentage of black teachers fell significantly when temporary licenses were phased out (see right vertical axis and solid black line). Uncertified teachers were much more likely to be black than teachers entering through other pathways, including the NYCTF teachers who replaced the uncertified teachers. However, NYCTF teachers are more racially diverse than teachers from other certified 15

16 pathways. In 2008, 44 percent of NYCTF teachers were nonwhite, while 33 percent of CR teachers and 27 percent of TFA teachers were nonwhite (Appendix Table 18). Number of Teachers Figure 5. First-Year Black Teachers by Pathway, and Percent of First-Year Teachers who are Black CR NYCTF TFA Temp Lic % Black Total Teachers 3. To what extent has the effectiveness of new teachers changed since the since initial increase in alternatively certified teachers at the beginning of the sample period? Average Differences Across Pathways. The next part of this study examines the effectiveness of teachers from different pathways and how this effectiveness have changed over time. There are several components to the analysis. First, the base model examines the average effects of teachers from different pathways over the full time-period of our data. In this base model, the standardized achievement level (test score) of a student is modeled as in equation (1). A ijst = 0 + 1A ijs(t-1) + 2A 2 ijs(t-1) + X it 3 + C ijst 4 + T jst 5 + j + s + ijst (1) Here, the achievement (A) in math (ELA) of student i in year t with teacher j in school s is a function of his or her prior achievement in both ELA and math and of prior achievement in both subjects squared, time-varying and fixed student characteristics (X), characteristics of the classroom (C), characteristics of the teacher (T), indicator variables (fixed effect) for the pathway by which the teacher entered teaching in NYC (Π), a fixed-effect for the school (υ), and a random error term (ε). The standard errors are clustered at the teacher level to account for the fact that teacher pathway is a teacher-level variable. All of the estimates described below include all of the variables in this base model (see 16

17 Appendix Table A19 for math and A20 for ELA) and various additional variables, typically interactions, to address specific research questions. Student characteristics include gender, race/ethnicity, poverty status, days absent during the prior year, and suspensions in the prior year. The aggregate classroom (teacher by grade by school by year) student characteristics include race/ethnicity, poverty status, average attendance in the prior year, average suspensions in the prior year, average student test scores in the prior year, and the standard deviation of student test scores in the prior year. Teaching experience is measured by dummy variables for each year of teaching from the first year through the twentieth and then an additional dummy variable for experience greater than twenty years. In addition, the model includes fixed effects for years, grades, and schools. All teachers in this analysis are initially certified through one of the following pathways: college recommended, Teach for America, New York City Teaching Fellows, independent evaluation, other alternative certification paths, temporary license (uncertified), or pathway unknown. All analyses include teachers from every pathway, but our description of the findings focuses on the first three pathways: college recommended (CR), Teach for America (TFA), and the New York City Teaching Fellows (NYCTF). For ease of interpretation, we estimate two versions of this model. In the first set of analyses, we are concerned with understanding how the returns to experience differ across pathways. Here all comparisons are relative to NYCTF teachers. The second set of analyses focus on how the effectiveness of teachers within pathways has changed over time. Here the reference group for our analyses are a group of College Recommended teachers who had 5 to 15 years of experience in 2001 and who remained teaching in NYC through We chose this reference group to facilitate comparisons of how the effectiveness of teachers entering through various pathways changed over time. Since we have normalized achievement scores by year and subject, referencing entering teachers from one pathway against those entering from another suggests how the relative effectiveness across pathways may have changed over time but does not show how effectiveness across cohorts within a pathway may have changed. By referencing against a group of teachers whose valueadded is expected to be constant over time, we significantly enhance our ability to make across cohort comparisons. 4 We also created two other experienced teacher reference groups, one that contained all CR teachers with 5 to 15 years of experience in 2001, regardless of whether they 4 We thank Ila Deshmukh Towery and Vicki Bernstein, who each suggested this idea. 17

18 remained teaching in NYC in subsequent years, and one with CR teachers with 10 to 20 years of experience at the time each of our cohort groupings entered teaching. Results from each of these formulations of the reference group are substantively the same. We present results from the 5 to 15 years of experience in 2001 and who remained teaching in NYC through 2001 below as the teachers in this group are unchanging over time. As set out in Equation 1, our estimates control for all the attributes of students that typically remain constant from one year to the next, such as parental support and home environment, through the inclusion of the prior achievement scores. It also controls for all the characteristics of schools that do not change over the time period by including a school fixed effect. We estimate models separately for math and for English languages arts by grade groupings, elementary grades (grades four and five) and middle school grades (grades sixth through eight) as well as a model that pools all grades. We explore several other models that make alternative assumptions regarding the determinants of student achievement, including the use of student fixed effects rather than school fixed effects, and additional lags in prior achievement. Unless noted, the results presented do not differ in any meaningful way from these alternative specifications. In general, we find that mean value-added differences among pathways mirror prior work--nyctf and traditional preparation (CR) produce similar results and TFA produces somewhat larger achievement gains in math. As shown in Appendix Table A19 (math) and A20 (ELA), CR teachers have students whose achievement gains exceed those of NYCTF teachers by standard deviations in math and standard deviations in ELA. The students of TFA teachers exceed those of NYCTF teachers by in math and are equivalent in ELA. NYCTF teachers have students whose achievement exceeds that of the uncertified teachers by in math and in ELA. However, in this paper we are less interested in understanding the relative performance of preparation pathways at a point in time than we are in understanding how the effectiveness of pathways have evolved over the last ten years. Changes in Pathways Differences Across Cohorts. We assess how the relative effect of pathways may have changed over the period by examining the effectiveness of successive cohorts of new teachers by pathways. We group teachers into cohorts based on the year in which they began teaching in NYC: Early , Middle , Later , and Recent These groupings correspond to the changing nature of the NYCTF 18

19 program and changes in the NYC teacher labor market. We then interact pathways and cohorts to get a sense of whether, the teachers recruited by each pathway in the Early period differ in effectiveness from those recruited in the Later period. We also employ a continuous measure of cohorts using the year in which teachers enter. Those results, not shown here, do not substantively differ from the results for cohort groups shown below and are available from the authors. The effectiveness of cohorts of entering teachers from each pathway is estimated relative to a unchanging group of experienced college recommended teachers are shown in Table 1. 5 The results provide some interesting insights. First, in math NYCTF teachers show no significant differences in any cohort from the group of experienced college recommended teachers employed as controls. Since these models control for the average effect of experience across all pathways, they suggest that NYCTF teachers are as effectiveness as the control group and that successive cohorts are as effective as the Early NYCTF cohort. These results hold across both grade groupings for math. For ELA, NYCTF teachers perform less well in the Early cohort than the reference group in grades 4 and 5, and while there is some evidence of improvement in the Late cohort, there is no consistent pattern of improvement. In grades 6 through 8 NYCTF teachers perform as well as the reference group and are equally effective across all cohorts. Teachers in the 4 th through 8 th grade math cohorts of college recommended teachers appear to evidence improvements in their effectiveness of math instruction. To explore this issue in greater detail we estimated these relationships for just first and second year teachers within each cohort and found no systematic relationship across cohorts. This suggests the patterns in Table 1 may result from a combination of changes in cohort effects and differential pathway gains to experience. As a result, we are unable to conclude there is a pattern of systematic improvement of CR math teachers across cohorts. The story is similar in ELA. Although, the first two cohorts of 4 th and 5 th grade CR teachers are less effective than the reference group, which is not true of the latter two cohorts, there is no evidence of improvement across cohorts, especially when we examine these patterns only on the sample of first and second year teachers. 5 As described above, the reference group for the results presented in Table 1 are all College Recommended teachers with 5 to 15 years of experience in 2001 who remained teaching in NYC through We created two other experienced teacher reference groups, one that contained all teachers with 5 to 15 years of experience in 2001, regardless of whether they remained teaching in NYC in subsequent years, and one with teachers with 10 to 20 years of experience at the time each of our cohort groupings entered teaching. Results from each of these formulations of the reference group are substantively the same, both in the magnitude of effects and in statistical significance. Results from these specifications are available from the authors. 19

20 Likewise for TFA teachers, there is no consistent pattern of improvement across cohorts, despite indications for individual cohorts of improvement. Here again, the estimates for first and second year teachers failed to demonstrate that a pattern of improvement. Table 1: Value Added by Pathway, Cohort, Subject and Grade* Pathways and Math ELA Cohorts Grades 4-8 Grades 4-5 Grades 6-8 Grades 4-8 Grades 4-5 Grades 6-8 Teaching Fellows Early ** *** (0.011) (0.012) (0.022) (0.009) (0.011) (0.013) Middle *** *** (0.009) (0.012) (0.015) (0.007) (0.010) (0.009) Late * b (0.012) (0.017) (0.017) (0.008) (0.014) (0.010) Recent (0.018) (0.045) (0.022) (0.016) (0.030) (0.019) College Recommended Early * * ** *** (0.007) (0.008) (0.013) (0.005) (0.007) (0.007) Middle c *,c ** *** c (0.008) (0.010) (0.015) (0.006) (0.009) (0.009) Late **,c * * (0.010) (0.011) (0.017) (0.007) (0.010) (0.010) Recent a (0.017) (0.018) (0.030) (0.011) (0.016) (0.015) Teach for America Early ** * (0.016) (0.020) (0.024) (0.012) (0.016) (0.016) Middle (0.017) (0.027) (0.024) (0.010) (0.021) (0.012) Late ***, b ***, a ** (0.015) (0.023) (0.021) (0.010) (0.017) (0.012) Recent ** c (0.022) (0.039) (0.027) (0.017) (0.034) (0.019) *All estimates relative to College Recommended teachers who had 5-15 years of experience in 2001 and who taught each year through *, **, *** indicate statistical significance of difference with experienced college recommended teachers at.001,.01 and.05 levels respectively. a, b, and c indicate statistical significance of difference with Early cohort of same pathway at.01,.05 and.10 respectively. 20

21 Selection into Tested Subjects. A concern in the value-added analyses is the potential role of selection bias; that teachers from certain pathways are systematically assigned to students in ways that are not accounted for by our model and which predispose these teachers to higher or lower value added. For example, NYCTF and TFA teachers are more likely to teach high poverty and lower achieving students. We believe the controls of our models provide a reasonable approach to address selection bias arising from the assignment of teachers to students with particular characteristics. 6 However, there is another type of selection into the sample that could also influence our results. As noted earlier, NYCTF teachers from later cohorts are more likely to be placed into high schools than teachers from the other pathways. It could be the case that these high school teachers, for whom we do not have value-added estimates, were systematically more qualified than their counterparts who were placed in elementary and middle school and if NYCTF teachers were more likely to be subject to this sorting, it may bias our NYCTF estimates downward as our estimates are based solely on elementary and middle school student scores. Table 2 below shows the mean LAST score (New York's general knowledge certification exam) for first-year teachers by pathway and cohort for all teachers certified in math. These scores are separated by teachers whose students took a math 3-8 grade exam (tested) and those teachers certified to teach math Table 2. Mean LAST Scores of First-Year Teachers by Pathway, Cohort, and Tested Students for Teachers Certified in Math CR NYCTF TFA Cohort tested not tested diff tested not tested diff tested not tested diff Early Middle Late Recent whose students did not. All high school math teachers would be in the latter group. If more qualified teachers were opting to teach high school, then we would expect the difference (diff) between these scores to be positive. We find some evidence of that, especially for NYCTF teachers in the early cohort, but this was not the case in later cohorts, especially relative to other 6 Since teachers have not been randomly assigned to preparation pathways, schools or classrooms, there remains the potential for these sources of selection to bias our findings, despite the numerous controls included in our estimates (Rothstein, 2009 ). 21

22 pathways. Therefore, it does not appear that this type of selection is driving our value-added results. Experience Effects By Pathway. To investigate whether there are differential effects of experience over time and pathway, we augment the base model described above with pathway indicator variables interacted with experience indicator variables. Because the use of these interactions incorporate both returns to experience and the changing composition of the workforce as some teachers exit, we also estimate models that examine the returns to experience for teachers with 3 or more years of experience, 5 or more years of experience, and 7 or more years of experience, respectively. This has the effect of holding such attrition constant. In all of these analyses, coefficients are compared to NYCTF teachers with comparable experience. We find that all teachers experience meaningful gains in effectiveness over the first four years of their careers in both math and ELA. For math achievement, we do not find significant differential returns to experience by pathway (Appendix Table A21). However, for ELA, there is evidence that NYCTF are less effective in the first year but gain in effectiveness relative to CR teachers over the following three years. These ELA results are statistically significant and typically about half the size of the difference between the average first and second year teacher (an effect size of 0.02 to 0.03). They are also about the size of the difference in effectiveness between NYCTF teachers and college recommended teachers in their first year. Thus, although college recommended teachers are more effective in their first year than NYCTF teachers, that advantage is eliminated during the first few years of teaching. 4. How has the attrition behavior of alternative and traditionally certified teachers changed? Average Differences Across Pathways. The retention analyses are based on the career paths of New York City public school teachers who were in their first year of teaching between and , a population of 84,560 teachers. 7,8 Each year, we examine whether teachers remain in the same school, transfer schools within NYCDOE, or exit teaching in NYCDOE. We limit the analysis to each of a teacher's first five years of teaching. Examining retention of more experienced teachers would constrain us to focus the analysis on the relatively few cohorts and observations that began teaching early in the last decade. 7 We limit our analysis to teachers we observe in their first year of teaching in New York City to reduce selection bias associated with teachers who began teaching prior to We have retention data for 81,758 of these teachers. 22

23 We examine retention both by examining simple descriptive statistics of retention and by estimating multinomial logit models with school fixed effects, controlling for pathway. 9 In our logit analyses, we control for all pathways into teaching, but in the text and tables we focus on college recommended (CR) teachers, New York City Teaching Fellows (NYCTF), and Teach for America (TFA) teachers. We include school fixed effects, estimating differences in attrition across teachers in the same school, to control for the fact that teachers are not randomly distributed across schools, and teachers decisions to transfer or exit are likely a function of working conditions and other school attributes. As we discussed previously, NYCTF and TFA teachers often work in different types of schools than CR teachers, and the placement of NYCTF teachers has changed over time. Similar to our approach above, we are interested in whether more recent cohorts of teachers have different retention patterns than earlier cohorts. Before turning to the results of our multinomial logit regressions, Table 3 presents descriptive statistics of the cumulative retention decisions of all teachers by year of experience, unconditioned by school fixed effects. Each column shows the cumulative retention behavior at Table 3: Cumulative Retention NYC Math Teachers by Years of Experience Year Initial school All CR NYCTF TFA Cumulative transfers (but not exits) All CR NYCTF TFA Cumulative exits All CR NYCTF TFA Notes. Results are approximate because the included cohorts change by year of experience. Therefore, proportions may not sum to 1. the end of the appropriate year. By the end of the fifth year of teaching, teachers from traditional teacher preparation programs persist at higher rates, both in their original school assignments and 9 For retention decisions following years 2-5, we also control for whether the teacher had transferred schools in the past. We do not present these results, but they consistently indicate that teachers who transferred in the past are significantly more likely to transfer again or to exit teaching in NYC. 23