#6: Do Vouchers Improve Education?

Transcription

1 THE FACTS ABOUT VOUCHERS #6: Do Vouchers Improve Education? Ten years after the start of the nation s first publicly funded voucher program in Milwaukee, there are no clear cut findings in the research literature that vouchers improve education. In fact, there is very limited data from the existing publicly funded programs. Of the three publicly funded voucher programs in the United States, only Cleveland s has maintained an on-going, state-commissioned evaluation. The Wisconsin legislature ended any state evaluation of the Milwaukee program in 1995, and the Florida legislature has provided for no evaluation of its voucher program. None of the private schools in Milwaukee or Florida that receive public money through vouchers is required to test students or report test scores. In fact, many of the purported findings that do exist on the educational value of the voucher programs have been obscured by rhetoric. As one pro-voucher educational consultant in Milwaukee noted in the Wall Street Journal, Findings are irrelevant unless they make headlines Headlines and spin are everything. 1 Insofar as research on voucher programs does exist, attention has largely focused on the impact on student test scores. 2 What follows is a synopsis of the research findings from publicly funded voucher programs in Milwaukee, Cleveland, and Florida, some privately funded voucher programs, and recent research examining the impact, if any, of vouchers on public school test scores in Florida. Research on publicly funded vouchers There have been state-commissioned evaluations of both the Milwaukee and Cleveland voucher programs. In Milwaukee, there were no appreciable academic gains at all. In Cleveland gains were found in one subject, science, for voucher students attending wellestablished private schools, with declines in test scores for students attending newly-opened private schools. Both sets of findings have been challenged by Harvard s Paul Peterson, whose methodology and objectivity has been criticized by many in the education research community. In the case of Milwaukee, a third researcher, Princeton economist Cecilia Rouse, has evaluated both the Peterson and Wisconsin studies, finding mixed results. Overall, there is no clear evidence that publicly funded vouchers have improved educational performance of voucher students. Milwaukee Parental Choice Program! In Milwaukee, the state commissioned the most thorough research yet conducted on vouchers. This research, conducted by a University of Wisconsin Madison team led by John Witte over a five year period, found no appreciable academic gains in reading and math from vouchers. 3 Attrition rates were high: 44% and 32% in the first two years. 4 Reasons for leaving the program included application and fee problems, transportation, Revised July 31, M Street Suite 400 Washington, DC Telephone Fax pfaw@pfaw.org Web site

2 2 and the limitation on religious instruction (before the state Supreme Court allowed the expansion of the program in 1998). 5 Among parents whose children remained in the program, satisfaction was high. 6 Parents of children who remained in the voucher program had higher educational levels a key determinant of a student s academic achievement than those who did not. 7! Using Witte s data, Paul Peterson s team employed different assumptions and statistical techniques and claimed that there was a statistically significant gain for voucher students in the third and fourth years of the program. 8 This finding was disputed by many in the research community, who argue that by the third year the control and experimental groups were not comparable. The 30% annual attrition rate primarily students doing poorly in the voucher program ensured that those who remained were an academically superior subset, not a random sample. 9 Peterson s methodology has also been criticized on other points, including his reliance in some cases on tiny samples in some cases as low as 26 students and for dismissing group differences between voucher school students and the control group (non-choice public school students). 10 Witte described Peterson s reanalysis of the Milwaukee data as a confusing, tortured effort to try to find any evidence that students enrolled in private schools do better than any students in the Milwaukee Public Schools. 11 Peterson s unconventional reporting of statistical significance tests has drawn fire not only from Professor Alex Molnar, then at the University of Wisconsin Milwaukee, who described Peterson s term substantially significant as an important-sounding characterization with no precise research meaning, but also from the pro-voucher Wall Street Journal, which wrote that he had been loose with his claims. 12 Some have pointed to contradictions between Peterson s claims of academic achievement and his own statistical data, 13 as well as a lack of adequate controls to take prior academic achievement of students into account. Other criticisms include a failure to adequately address variables that affect a child s success in school, including parental involvement and expectations, parental employment, marital status, family size and receipt of Aid to Families with Dependent Children. 14! A third analysis of the same data was conducted by Cecilia Rouse of Princeton University. Rouse found that the voucher students had made some small gains in math, and that "the [voucher] effects on the reading scores are as often negative as positive and are nearly always statistically indistinguishable from zero. 15 The findings of a positive voucher effect for math were only for the subgroup of students who were in the voucher program over a four-year period, which again does not account for the significant number of dropouts from the voucher program. Moreover, Rouse showed that Milwaukee public schools serving low-income populations that have small class sizes and additional state funding keep pace with voucher schools in math gains and substantially outpace them in reading. 16 Rouse does not assert that this proves that small classes are the causal factor but calls for more investigation; however, Alex Molnar and Charles Achilles of Eastern Michigan University do find that reducing class size is more effective than a voucher policy in helping at-risk students, as does Princeton economists Alan Krueger and Diane Whitmore. 17 Referring to a different study in which Peterson found large improvements in test scores for African American students and claimed these gains outpace those observed in the Tennessee class size reduction experiment called STAR, Molnar and

3 3 Achilles fault Peterson for making an apples-to-oranges comparison. They found that when the apples-to-apples comparison is made between similar students, For those [minority] students, the STAR effects were approximately double the total effects [of vouchers]. 18! In 1995, Wisconsin lawmakers who support vouchers responded, not by changing the voucher program, but by eliminating any further state-sponsored research into the educational results of vouchers. The most recent Wisconsin state audit of the voucher program found that some hopes for the program most notably, that it would increase participating pupils academic achievement cannot be documented, largely because uniform testing is not required in participating schools. 19 Wisconsin taxpayers thus have absolutely no current information on whether vouchers are having any positive effects on education. Cleveland Scholarship and Tutoring Grant Program! The state-sponsored study of the Cleveland voucher program, conducted by an Indiana University team led by Kim Metcalf, found that after two years there was no improvement in the overall test scores of those students using vouchers in established private schools. On a subject-by-subject basis, there were gains for voucher students only in science. 20 Overall test scores and the other four subject scores revealed no differences between voucher and public school students. There were also students who used vouchers at the new HOPE Academy schools, created by David Brennan, an entrepreneur who helped push the legislature into adopting vouchers to receive voucher money and students. These students scored below their peers in both public schools and the established private schools in all five subjects tested. 21! Responding to Metcalf s first year report, Paul Peterson and his research team claimed that findings from his 1997 evaluation of the two HOPE Academy voucher schools showed significant academic gains. 22 He was hired to evaluate these schools by their founder David Brennan, a prominent voucher advocate. 23 Peterson criticizes the Indiana University study primarily for failing to include the HOPE Academy scores and for using second grade test scores taken prior to entry in voucher schools as a basis for comparison with third grade voucher scores. 24! Metcalf responded with a strong article entitled Advocacy in the Guise of Science. In it, he suggested that the Peterson researchers are strong supporters of vouchers and have done much to promote the implementation of voucher programs throughout the country. So it is possible that they are engaged in a deliberate effort to misrepresent the Cleveland data in order to influence educational policy. 25 Specifically, he responded that he did include the HOPE scores but put them into a different section because those students took a different test. With regard to the use of second grade tests, he points out that assessing first year results of an experiment without a baseline is a little like trying to determine who won a basketball game by looking only at the points scored in the second half of the game. 26 The Ohio Legislative Committee on Education Oversight (LCEO), responsible for monitoring Cleveland s voucher program, further discredited Peterson s criticisms.

4 4 The LCEO had appointed Peterson to its technical review committee of the Indiana study, and charged that Peterson released his critique to the Wall Street Journal, the Washington Post, and the World Wide Web because he didn t like the results, even though the study s methods are viewed as appropriate and credible by disinterested scholars. 27 Florida A+ Plan! In Florida, vouchers are available to students living in the attendance zone of any schools designated as failing by the state any two years out of four. To date the Florida program includes just two elementary schools in Escambia County. It is just now entering its third year, and there has been no research on its impact on achievement among voucher students. There has been an effort by researchers to assess the impact of the voucher threat in Florida on public school student test scores (see below). Research on privately funded vouchers During the 1990s, voucher advocates formed a number of organizations to provide privately funded vouchers to mostly urban, low-income students. The foundations supporting these groups also funded research in some of the cities, conducted by Paul Peterson and colleagues. The most prominent report to come out of this research to date has been the two-year study of private vouchers in New York City, Washington, D.C., and Dayton, Ohio released in August, Peterson wrote that the African American students using vouchers to attend private schools scored 6.3 percentile points better after two years on combined reading and math tests than the control group students who had applied for vouchers but remained in public schools. They found no statistically significant positive or negative effects for Hispanic or white students using vouchers, although 51% in the New York City study were Hispanic, for example, and 24% in Dayton were white. 29 Since students applying for these private vouchers were entered into a lottery (separate in each city) and researchers were able to test the students and gather background data, Peterson argued that these studies are akin to randomized field trials or experimental studies in medical science, and since they do not suffer from the shortcomings of the Milwaukee and Cleveland studies the validity of the findings is beyond reproach. Peterson claimed that vouchers could lead to a "a nontrivial closing of the gap" in achievement between African American and majority students. 30 The study was immediately criticized by researchers who pointed out a number of flaws. These included the presence of bias in the study potentially resulting from disappointment effects among some parents who lost in the voucher lottery, as well as selection bias due to the fact that significant numbers of students dropped out of the study in subsequent years. Another problem undermining the study s findings are the unexplained dramatic inconsistencies in achievement gains, where some groups do very well while others do poorly. 31 The presence of these issues forced Peterson to use statistical corrections precisely like those he claimed to be avoiding thus undermining his claim that this research adhered to a gold standard of social science research. 32

5 5 Unexplained, erratic test results! One of the study s partner agencies, Mathematica Policy Research, Inc. took the extraordinary step of releasing its own press statement entitled Voucher claims of success are premature in New York City. Mathematica revealed that, in New York City, all of the test score gains for African American students were concentrated in one of the four grades, with no explanation for this phenomenon. 33 Yet the improvement shown by this one subgroup was so marked that it resulted in statistically significant improvement for all grade levels of African American students when the four grades were averaged together. The truth is that most African American voucher students in New York did not do better than their public school peers.! Similarly erratic results were evident in Dayton, where students were in grades 3 through 9 in the second year of the program: African American students in three of the grades (3 rd, 5 th & 7 th ) showed large gains of between 14.5 and 19.7 national percentile points in combined math and reading scores while those in the other four grades (4 th, 6 th, 8 th & 9 th ) actually declined for the latter cohort by nearly 15 percentile points. 34 No research basis for expanding vouchers! Despite Peterson s claim that this three-city study indicated that vouchers might lead to the eradication of the achievement gap in this country, it in fact offers no evidence even if the study s findings were valid that would support larger scale, publicly funded voucher programs. The authors themselves caution that generalizing beyond this small, three city, privately-funded voucher program to larger-scale programs is problematic: A much larger [voucher] program could conceivably have quite different program outcomes. 35! One reason for this is the presence of what are known as peer effects. Much of whatever academic advantage private schools appear to have may likely be due to the fact that private schools can unlike public schools select and choose their students, thus having some control over the peer environment in the schools. As a few poor, urban students with vouchers enter these private schools that are made up predominantly of children of higher socioeconomic status, there may be a positive peer effect that helps to boost their academic achievement. To the extent this is true, this would indicate at best the possibility of potential gains for those few voucher students able to gain admission to established private schools, but not for larger numbers entering new private schools, were there to be a growth in the number of private schools as a market response to a large expansion of voucher programs. 36 Metcalf s findings in Cleveland point to exactly this problem: Voucher students in already established private schools did as well as their public school peers better in one subject, science. But those who attended new schools the ones opened precisely to accept voucher students did worse in all subjects than their peers in both the older private schools and public schools.

6 6 Research on the impact of vouchers on public schools In a recent study widely touted by advocates of private school vouchers, researcher Jay Greene claims that the threat of vouchers caused significant improvement in Florida public schools rated F in 1999 on the state s Florida Comprehensive Assessment Test (FCAT) exams. 37 Greene established that F-rated schools showed greater gains than schools with higher grades, and went on to argue that it was the threat of vouchers that motivated these schools to do better. In fact, as several researchers have shown subsequently, the study has serious flaws and does not demonstrate that the improved test scores of students at these F - rated schools was caused by or in any way related to the threat of vouchers. Greene s study flawed! Significant methodological flaws in Greene's analysis have been identified by Rutgers University researchers Gregory Camilli and Katrina Bulkley, including the fact that he uses school-level statistics rather than the more conventional student-level ones and yet asks in effect that his resultant "effect size" data be interpreted in the usual way. This leads to greatly inflated estimates of the gains made by the 'F'-rated schools. They find Greene's conclusions to be "implausible." 38! A second study, by Haggai Kupermintz of the University of Colorado at Boulder, demonstrates that the "gains" made by Florida's 'F'-rated schools may have been more apparent than real. Kupermintz' analysis supports what had become the conventional wisdom in Florida at the time, shortly after the FCAT test results were announced in June This explanation was that the schools knew what was required to avoid an 'F' grade: they needed only to pass any one of the three sections on the FCAT, reading, math or writing. Of the three, writing was a far easier target for specific test preparation. Schools did not want to be branded again as "failing schools," they knew the easiest route to escape that fate, and they succeeded but without necessarily improving actual classroom learning. As Kupermintz concludes, Greene s report provides a a false sense of a dramatic success with respect to vouchers. 40! Greene failed to pursue and investigate other plausible explanations and causes for the improved (FCAT) scores in F -rated schools. For example, Greene overlooks the significant amount of extra resources, both state and local, that were directed towards the F -rated schools. This enabled schools to reduce class size and extend the school day and year in order to help their students improve their performance, and may well have been one of the real causes of improvements in these schools. 41 Resources and accountability can improve public schools, not vouchers! Greene also never considered whether the very fact that schools were rated F the socalled Scarlet Letter effect regardless of the possibility of vouchers, stimulated the observed performance improvements. An Economic Policy Institute (EPI) report by Stanford researcher Martin Carnoy did examine this question empirically, testing Greene s voucher theory against a scarlet letter theory. 42 Like Florida, Texas and

7 7 North Carolina have accountability systems that draw public attention to school failure, and Florida had a similar system prior to the introduction of vouchers in Carnoy and colleagues compared the post-voucher test score gains in Florida against test score gains in these three non-voucher contexts:! Specifically, EPI utilized the fact that Florida s FCAT had been the basis of rating schools even before vouchers were added in They compared gains in Florida in before vouchers were introduced by students in schools given a critically low rating in 1996 (comparable to the F rating of the current system) with Greene s data on the gains made by students in F-rated schools (post-voucher). They found that improvements in math scores were larger in when there was no voucher threat, while gains in reading and writing were larger after vouchers. They concluded that vouchers do not have a significant positive impact on public school performance. 43! Comparing Texas with Florida, the EPI study found overall test score gains (not just for failing schools) in Texas and Florida to be similar. However, for the failing schools, gains in Texas were as large or larger than the gains calculated by Greene for Florida s schools, casting doubt on Greene s claim that a voucher threat was the impetus for such growth in Florida. 44 Indeed, Greene's own research in Texas points to an alternative explanation for Florida s rising test scores: accountability, testing, and increased resources can promote school improvement, independent of vouchers. Greene himself, in an earlier article, found that the state's Texas Assessment of Academic Skills (TAAS) the set of tests that is the centerpiece of the Texas state accountability system was critical to what he has called the "Texas school miracle" (the significant gains in some Texas achievement results during the 1990s). 45! In North Carolina, the EPI study found that the low-performing schools had larger test score gains than higher-rated categories of schools, as in Florida again, with no threat of vouchers. The authors, therefore, find that Greene inappropriately attributed the differential gains to the voucher program rather than to the other effects of being labeled a failing school, such as shame, increased scrutiny, and possibly additional resources, 46 and that the results that Jay Greene found for Florida probably have little or nothing to do with vouchers Bob Davis, Class Warfare: Dueling Professors Have Milwaukee Dazed Over School Vouchers, Wall Street Journal, October 11, 1996, p. A1. 2 Just as the public provision of K 12 education has many goals, so should any evaluation of schooling investigate a variety of outcomes. In addition to student achievement in certain academic subjects, voucher research should also look at a variety of other factors including, but not limited to: rates of student persistence, repetition and dropout, indicators of discipline and behavioral problems, differences in curriculum and extracurricular offerings, experiences of students with special needs, impact on voucher students of attending sectarian schools, impact on community and state support for public schools, cost-effectiveness of various models for schooling, long term effects on civic participation, and financial impact on parents due to additional costs incurred for tuition, fees, transportation, extracurricular activities, etc. Voucher studies to date, however, have not evaluated this range of subjects.

8 8 3 John Witte, et al., Fourth Year Report: The Milwaukee Parental Choice Program, Madison, WI: Wisconsin Department of Public Instruction, 1994, p Witte, Fifth Year Report: The Milwaukee Parental Choice Program, Madison, WI: Robert M. La Follette Institute of Public Affairs, University of Wisconsin Madison, 1995, p ibid. 6 Witte, Fifth Year Report, p Witte, Fourth Year Report, p. 2; John Witte, et. al., Fifth Year Report, p. 4; Correspondence to the office of Senator Ted Kennedy on Greene, Peterson and Du: The Effectiveness of School Choice in Milwaukee, from Peter Cookson, Jr., Ph.D., Director, Center for Educational Outreach and Innovation, Teachers College, Columbia University, August 23, Jay Greene, Paul Peterson, and Jiangtao Du, The Effectiveness of School Choice in Milwaukee: A Secondary Analysis of Data from the Program s Evaluation, Cambridge, Mass: Harvard University, John F. Kennedy School of Government, 1996, p Henry M. Levin, Educational Vouchers: Effectiveness, Choice and Costs, Journal of Policy Analysis and Management, Vol. 17, no. 3, , p. 377; Cookson, correspondence. 10 Cookson correspondence. 11 John Witte, Reply to Greene, Peterson and Du: The effectiveness of school choice in Milwaukee: A secondary analysis of data from the program s evaluation, August 23, The real lesson of Milwaukee s voucher experiment, Education Week, August 6, Bob Davis, Class Warfare: Dueling professors have Milwaukee dazed over school vouchers, Wall Street Journal, October 11, 1996, pp. 1ff. 13 Alex Molnar and Walter C. Farrell, Jr., The Harvard Study and the Politics of Private School Choice, p John F. Witte, Reply to Greene, Peterson and Du: The Effectiveness of School Choice in Milwaukee: A Secondary Analysis of Data from the Program s Evaluation,, August 23, 1996, p. ii. 15 Cecilia Elena Rouse, Private school vouchers and student achievement: An evaluation of the Milwaukee Parental Choice Program, Quarterly Journal of Economics, v. 113, no. 2, pp. 553ff., May Cecilia Elena Rouse, Schools and Student Achievement: More Evidence from the Milwaukee Parental Choice Program, Economic Policy Review, v. 4, no. 1, March 1998; Private school vouchers and student achievement: An evaluation of the Milwaukee Parental Choice Program, Quarterly Journal of Economics, v. 113, no. 2, pp. 553ff., May Alex Molnar, Smaller Classes and Educational Vouchers: A Research Update, Keystone Research Center, 1999, pp Alan B. Krueger & Diane M. Whitmore, Would smaller classes help close the black-white achievement gap? Paper presented at Brookings Institute Edison Schools, Inc. conference Closing the gap: Promising approaches to reducing the achievement gap, January, Alex Molnar & Charles Achilles, Voucher and class-size research, Education Week, Oct 25, 2000; William G. Howell, Patrick J. Wolf, Paul E. Peterson, and David E. Campbell, In defense of our voucher research, Education Week, February 7, An Evaluation: Milwaukee Parental Choice Program, State of Wisconsin Legislative Audit Bureau, Report 00-2, February 2000, cover letter. 20 Kim K. Metcalf, Evaluation of the Cleveland Scholarship and Tutoring Grant Program , Bloomington, IN: The Indiana Center for Evaluation, Indiana University, 1999, pp Two types of analysis were used. In one, in which students background characteristics and academic achievement prior to entering the voucher program were taken into account, there was no improvement for voucher students in established private schools in reading, math or social studies, while there were measurable gains in language arts and science. In the second analysis, when classroom characteristics (class size, teacher experience and education level) were also taken into account, there were no statistically significant differences in reading, math, social studies, language or the overall scores. Voucher students did score better in science, by 2.2 percentile points (38.3 versus 36.1). 21 ibid. Voucher students attending newly established private schools had lower scores in all five subjects and overall in both analyses (with and without classroom characteristics). 22 Jay Greene, William Howell and Paul Peterson, New Findings from the Cleveland Scholarship Program: A Reanalysis of Data from the Indiana University School of Education Evaluation, Cambridge, Mass: Harvard University, Taubman Center on State and Local Government and John F. Kennedy School of Government, 1998, p. 1.

9 9 23 Legislative Office of Education Oversight, Memo to the members of the Legislative Committee on Education Oversight, RE: Evaluation of Cleveland Scholarship and Tutoring Program, July 23, 1998; Voucher Study Shows no Big Academic Gain, Cleveland Plain Dealer, March 28, Jay Greene, William Howell and Paul Peterson, New Findings from the Cleveland Scholarship Program: A Reanalysis of Data from the Indiana University School of Education Evaluation, Cambridge, Mass: Harvard University, Taubman Center on State and Local Government and John F. Kennedy School of Government, 1998, pp Kim Metcalf, Advocacy in the Guise of Science, Education Week, September 23, ibid. 27 Legislative Office of Education Oversight, Memo to the members of the Legislative Committee on Education Oversight, RE: Evaluation of Cleveland Scholarship and Tutoring Program, July 23, William G. Howell, Patrick J. Wolf, Paul E. Peterson & David E. Campbell, Test-Score Effects of School Vouchers in Dayton, Ohio, New York City, and Washington, D. C.: Evidence from Randomized Field Trials, Paper prepared for the annual meetings of the American Political Science Association, Washington, D. C., September Cambridge, Mass: Harvard University, Taubman Center on State and Local Government and John F. Kennedy School of Government, 2000, available at accessed August 28, ibid., p Edward Wyatt, Study Finds Higher Test Scores Among Blacks With Vouchers, New York Times, August 29, Martin Carnoy, Do school vouchers improve student performance? Washington, D.C: Economic Policy Institute, William G. Howell, Patrick J. Wolf, Paul E. Peterson, and David E. Campbell, In defense of our voucher research, Education Week, February 7, Voucher Claims of Success Are Premature in New York City Second-Year Results Show No Overall Differences in Test Scores Between Those Who Were Offered Vouchers and Those Who Were, Press release, Mathematica Policy Research, Inc. September 15, Available at: accessed September 15, Carnoy, As with the New York data, these grade-by-grade results for Dayton were not released at the time of the Peterson report, but were shared only later by one of the authors, William Howell, with Carnoy. 35 William G. Howell, Patrick J. Wolf, Paul E. Peterson & David E. Campbell, Test-Score Effects of School Vouchers in Dayton, Ohio, New York City, and Washington, D. C.: Evidence from Randomized Field Trials, Paper prepared for the annual meetings of the American Political Science Association, Washington, D. C., September Cambridge, Mass: Harvard University, Taubman Center on State and Local Government and John F. Kennedy School of Government, 2000, available at accessed August 28, 2000, p.24) 36 See Carnoy 2001, p.33, for further discussion. 37 Jay P. Greene, An Evaluation of the Florida A-Plus Accountability and School Choice Program, New York: Manhattan Institute, February, Under Florida s A+ Plan, passed in 1999, the Florida Comprehensive Assessment Test (FCAT), which had been in use for several years, became the basis of a system whereby schools are graded A through F, and students at schools receiving an F grade any two years out of four become eligible for transfer to better-performing public schools or for vouchers to use at participating private schools. 38 Gregory Camilli and Katrina Bulkley, Critique of An Evaluation of the Florida A-Plus Accountability and School Choice Program, Education Policy Analysis Archives, v. 9, no. 7, March 4, 2001, available at accessed March 9, See, for example, Stephen Hegarty, "Why Are Florida Children Writing So Much Better?" St. Petersburg Times, June 21, Haggai Kupermintz, The Effects of Vouchers on School Improvement: Another Look at the Florida Data, Education Policy Analysis Archives, v. 9, no. 8, March 19, 2001, available at accessed March 22, Superintendent: Tests show children at voucher schools improving, Associated Press State and Local Wire, (Dateline: Tallahassee) April 25, 2000; Jodi Wilgoren, Florida s vouchers a spur to 2 schools left behind, New York Times, March 14, 2000; Jenny LaCoste, Schools face new year of F-list stress, Pensacola News Journal, June 21, 2000.

10 10 42 Carnoy, 2001, pp. 25ff. 43 Doug Harris, What caused the effects of the Florida A+ program: Ratings or vouchers? in Carnoy, 2001, p Carnoy, 2001, p.27; Amanda Brownson, A replication of Jay Greene s voucher effect study using Texas performance data, in Carnoy, 2001, p Jay P. Greene, "The Texas School Miracle Is for Real," City Journal, v. 10, no. 3, Summer Helen F. Ladd & Elizabeth J. Glennie, A replication of Jay Greene s voucher effect study using North Carolina data, in Carnoy, 2001, p Ladd & Glennie, 2001, p. 52. Carnoy points out (p. 29) that none of these studies were able to isolate the effect of state accountability systems. We cannot, he writes, necessarily attribute the increase in test scores in these states to the fact that they have rigorous accountability systems, just as there is no basis for claiming that the rising scores in Florida were due to the threat of vouchers.