School Resource Analysis - Student Performance in Maryland City

Transcription

1 School Resources and Achievement in Maryland Report by Eric A. Hanushek University of Rochester September 1996 L Ai, t eo

2 A central assertion in the complaint of Bradford et al. v. Maryland State Board of Education et al. is that the State provides insufficient resources to ensure an adequate education for the students of Baltimore City. Behind this assertion is the notion that resources are directly related to the quality of education provided to students. This latter notion proves to be untrue across the nation, as substantial research indicates that there is no consistent relationship between resources and student performance. It also proves untrue in Maryland, where spending and resources do not have any strong influence on student performance. Moreover, Baltimore City schools appear to achieve less than would be expected after considering both the size of its at-risk population and the resources that it has available. Baltimore's schools and fiscal situation are best considered within the context of U.S. schools. This report establishes the following about U.S. schools: adequate schooling of the population is very important to society; the patterns of costs and performance in schools over the past quarter century have indicated problems with the central regulation of schools; and research shows large and important inefficiencies in the operations of schools. This report also incorporates an intensive analysis of student performance in the State of Maryland. Through analysis of the s_ix content areas of fifth grade testing covered by the Maryland School Performance Assessment Program (MSPAP), it becomes clear that variations in resources across schools and districts are not a significant determinant of variations in student performance. Consistent with the national research, it is clear that the effectiveness of school management is much more important than the resources commonly identified as being key to school performance. On this score, ~ltimore City schools appear to do worse than would be expected._given their student population and resources. The Importance of Quality Schooling A substantial amount of research confirms the importance of high quality schooling. This research is evaluated in Hanushek with others [1994]. Individuals with more and better schooling systematically earn more than those with less. This well-documented relationship has gotten even stronger in recent years as demands in the economy for skilled workers have risen. Beyond higher earnings, more educated workers have better employment opportunities with less chance of unemployment; are more able to adapt to new technologies and work requirements; are more successful at raising families; and have better overall health. Moreover, a better educated labor force supports more rapid productivity growth in the economy as a whole. These relationships have been confirmed in research that considers both the amount of schooling that individuals get and the quality of that schooling. Most economists believe that the demand for skilled workers will continue to grow into the future. Technology in the United States has developed in ways that use more educated workers and recent developments are expected to continue. Cost and Performance of U.S. Schools The overall story about what has been happening in U.S. schools is clear: the rapid increases in expenditures on schools of the past three decades have simply not been matched by measurable increases in student performance. Moreover, detailed studies of schools have shown a variety of inefficiencies, inefficiencies which, if corrected, could provide funds for a variety of improvement programs.

3 There has been a dramatic rise in real expenditure per pupil over the entire century. After allowing for inflation, expenditures per pupil have increased at almost 3½ percent per year for 100 years (see Hanushek, Rivkin, and Jamison[1992], Hanushek and Rivkin[1997]). This remarkable growth is not explained away by such things as increases in special education or changes in the number of immigrant students in the school population, though those have had a noticeable impact on school expenditures. Matched against this growth in spending, student performance has at best stayed constant, and may have fallen. While aggregate performance measures are somewhat imprecise, all point to no gains in student performance over time. The path of achievement on reading, math, and science exams, shown in figures l-3, is representative of the pattern of performance for the population and for racial subgroups (U.S. Department of Education [1994]). These figures display the performance over time of a representative sample of 17-year-olds on the various components of the National Assessment of Educational Progress (NAEP). Figure 4 shows the changing pattern for SAT tests. While there has been some recovery from their low point, SAT scores have remained below where they were in the mid1960s. The SAT scores are subject to selective test taking. Nonetheless, when this is accounted for, a noticeable decline in performance remains (see Congressional Budget Office [1986]). Finally, U.S. students have not compared well with those in other countries. The comparisons of United States and Japanese students in the early 1980s showed, for example, that only five percent of American students surpassed the average Japanese student in mathematics proficiency (see McKnight et al.[ 1987], National Research Council[ 1989]). The poor performance on international tests remains even when all students are given a form of the U.S. NAEP tests (e.g., LaPointe et al. [1989]). These comparisons suggest future problems that may face the U.S. economy. The problems of performance are particularly acute when considered by race or socio-economic status. Even though there has been some narrowing of the differences in performance, the remaining disparities are large, and incompatible with society's goal of overall equity. The changes in aggregate spending on schools have not been sufficient to eliminate, even to reduce much, the longstanding performance gaps between advantaged and disadvantaged students. Moreover, the narro~ting of racial and ethnic differentials that have occurred are not easily explained by simple school spending changes (Cook and Evans 1996). Some people have argued that changes in the student population have put extra pressures on schools, thus absorbing some of the extra funds that have been available over time. For example, expansion of the special education population or increased numbers of immigrants may put added burdens on school budgets. Specific analyses of these various factors, however, have not supported the arguments either that other costs are responsible for a majority of expenditure increases or that the student population on net has gotten worse. Various studies of special education (e.g., Hanushek and Rivkin [1997], Chaikind et ai. [1993], Consortium on Productivity in the Schools [1995]) indicate that mandatory expenditure for special education has been greater than that for regular education. Yet, the growth in special education cannot explain more than a fifth of the growth in expenditure since passage of the federal statutes on special education. Moreover, two recent analyses of the importance of changing family background suggest that, if anything, students have gotten better over time (Grissmer et al. [1994]], Cook and Evans [1996]). While single parent families and children in poverty have increased as a proportion of the population, parents' education has improved and families have gotten smaller---offsetting unfavorable movements.

4 The summary, of the aggregate movements in costs and performance is simple: rapidly growing expenditure and inputs has not yielded tangible results in terms of student test performance. No evidence suggests that on the whole the nation has been shortchanging schools. More importantly, past performance does not suggest that simple increases in spending will improve schools. Specific Studies of Schools The aggregate results are confirmed in more detailed studies of schools and classrooms. Previous summaries in Hanushek[1986, 1989] have been expanded and carried forth through 1994 (Hanushek, Rivkin, and Taylor [1996], Hanushek [1996]). These studies document a variety of common policies that increase costs but offer no assurances of commensurate improvements in student performance. The wide range of careful econometric studies reviewed indicate that key resources--ones that are the subject of much policy attention--are not consistently or systematically related to improved student performance. The expanded set of studies include 377 separate estimates of the effects of various school resources on student performance. These studies, found in 90 separate publication, offer very strong evidence that standard spending or resource policies do not consistently affect student outcomes. The key elements are summarized in Table 1. The most dramatic finding is that smaller class sizes usually have no general impact on student performance, even though they have obvious implications for school costs. While some specific instruction may be enhanced by smaller classes, student performance in most classes is unaffected by variations in class size in standard operations of, say, class sizes between 15 and 40 students.: This lack of relationship is evident in Table I, which includes a tabulation of the 277 available estimates of the effects of teacher-pupil ratios. Few of the estimates give any confidence that there is a real relationship (i.e., few are statistically significant). Moreover, positive estimates are almost equally matched by negative estimates. For example, only 15 percent of the studies find positive and statistically significant influences of teacher-pupil ratios (i.e., suggest with normal confidence that smaller classes are better), while 13 percent find statistically significant negative effects (i.e., suggest with normal confidence that smaller classes are worse). The basic econometric evidence is supported by experimental evidence, making it one of the clearest results from an extensively researched topic. 2 Nevertheless, even in the face of high costs which yield no apparent performance benefits, the overall policy of states and local districts has been to reduce class sizes in order to try to increase quality (see below). LThere may be special programs, say ones falling outside of the range of normal operations, where smaller classes are effective. For example, the Success for All program and the reading tutorial program of the University of Texas at Dallas appear to show that early one-on-one instruction may be beneficial. But these are different from general reductions in overall class sizes or pupilteacher ratios (see Hanushek with others[1994], Hanushek [1996], and Hanushek, Rivkin, and Taylor[ 1996]). 2An early review of experimental evidence is found in Glass and Smith [1979]. More recently, the State of Tennessee conducted an extensive statewide random assignment experiment of reduced class size in grades K-3. Except perhaps for kindergarten, no gains in student performance were associated with being in the smaller classes (Word et al. [19901).

5 A second, almost equally dramatic, example is that obtaining an advanced degree does little to insure that teachers do a better job in the classroom. It is essentially as likely that a teacher with a bachelor's degree would elicit high performance from students as it is for a teacher with a master's degree. Only 16 of 171 separate estimates of the effects of teacher education find positive and statistically significant effects, while 9 find negative and statistically significant effects (see Table 1). Again, since a teacher's salary invariably increases with the completion of a master's degree, this is another example of increased expenditure yielding no gains in performance. The final major resource category with a direct impact on school spending (through salary determination) is teacher experience. The evidence on the effectiveness of experienced teachers is more mixed than for the previous categories, but it does not provide convincing support of a strong relationship with performance. These resource effects are important for two reasons. First, variations in instructional expenditure across classrooms are largely determined by the pupil-teacher ratio and the salary of the teacher (which in turn is largely determined by teacher's degree and experience). If these factors have no systematic influence on student performance--which the evidence shows they do not--expansion of resources in the ways of the past are unlikely to improve performance. Second, either explicitly or implicitly schools have pursued a program of adding these specific resources. 3 Table 2 traces these resources over the past several decades. These are the changes that have led to the dramatic cost increases identified earlier. The schools currently have record low pupil-teacher ratios, record high completion of master's degrees, and more experienced teachers than at any time at least since These factors put in resource terms what has been the result of many specific programs that have contributed to the rapid growth in per pupil spending. But, they have not led to improvements in student performance. Schools do not regularly ensure that increased student performance flows from increased expenditure. 4 3Expansion of these resources is often implicit, resulting, for example, from the introduction of new programs which in turn expand specialized staff. It is at the same time clear that these resource increases are not the simple result of governmental mandates to expand school activities such as those for special education (Hanushek and Rivkin[1997]). Instead, they are reflective of a consistent policy to increase the intensity of instruction. ~Recent discussions of the existing literature reinforce these conclusions. The discussions of existing work in Hedges et a/.[1994] provides evidence that some school systems appear to use money effectively in the sense of getting positive effects from increased real resources or expenditure, while many others provide no basis for believing that money will be spent effectively. Their analysis is entirely consistent with the findings and conclusions here, They concentrate most of their energy on the much more limited question of whether there is evidence anyplace that expenditure ever matters. In essence, they devote all of their attention to the fact that 15 percent of the studies in Table 1 find positive and significant effects of larger teacher-pupil ratios--as contrasted with the 5 percent that would be expected by statistical tests if no school ever used small classes effectively. Unfortunately, this approach completely ignores the fact 85 percent of the estimated effects of teacher-pupil ratios, 91 percent of the estimated effects of greater teacher education, and 71 percent of the estimated effects of teacher experience give no confidence of a positive effect--including some proportion in each case even giving confidence of a perverse negative effect. These latter comparisons are the key to policy considerations centered on overall changes in resources allocated to schools. See also 4

6 There is no reason to presume that adding extra resources to a district employing the existing decision making structure will improve student achievement. In some instances, it may. In others it may not. In still others, achievement may actually fall. The overwhelming majority of results suggest, however, that we should have no confidence that adding resources will consistently and systematically improve student performance. None of this accumulated evidence suggests that any minimum level of resources is needed for promoting student achievement. Nor is there any suggestion in the underlying data that there are significant nonlinearities, say through diminishing marginal returns to resources, in the relationship between expenditure and student achievement. This latter possibility might imply that there are significantly stronger effects of added resources when the overall level of resources is low, but little evidence supports that. s These conclusions follow directly from the underlying studies. And they underscore why simple resource policies--by themselves or related to policies toward school financing--are likely to be ineffective. A final aspect of this analysis deserves attention. Even though there has been a concerted effort to identify specific aspects of schools and teachers that systematically improve student performance, no such factors have been reliably identified. Individual studies tend to find one or more specific factor that is correlated with performance, but factors thus identified seldom hold up to further scrutiny in other studies. Furthermore, no systematic evidence is available to indicate why some districts tend to make better choices--ones leading to heightened student performance--than other districts make. While there is no consensus about what specific factors affect student performance, there is overwhelming evidence that some teachers and schools are significantly better than others. For example, within inner city schools, progress of students with a good teacher can exceed that of students with a poor teacher by more than a year of achievement during a single school year. 6 The dramatic differences in performance are simply not determined by the training of teachers, the number of students in the classroom, or the overall level of spending. Resource differences do not describe the differences in performance across teachers and schools. This holds even in places where there Hanushek [1994, 1996]. This fact is also why they return to emphasize the importance of how money is spent in their conclusions. 5A somewhat different approach relates school experiences to labor market success of workers, although this analysis is largely concentrated on results across different states. Recent attention has been given to estimates indicating a relationship between school resources and individual earnings (e.g., Card and Krueger[1992]). One interpretation of this is that such a relationship held in the 1920s and 1930s when sampled individuals received their education and when the level of resources was very different than that observed today (see Burtless[1996]). Recent analyses have, however, raised significant questions about the reliability and validity of the original findings (Betts[1996], Heckman, Layne-Farrar, and Todd[1996], Hanushek, Rivkin, and Taylor[1996]), Speakman and Welch[1995]. 6These comparisons provide indications of improvements in standard test performance across classrooms after considering both starting achievement levels and family influences on performance (Hanushek[ 1992]).

7 have been dramatic increases in resources (e.g., Kansas City). Therefore, measures of resource differences cannot be ~d to indicate potential problems (such as underfunding or resource shortages). Nor can, be used to describe policies that wi!! yield better student results. A primary task of school reform is increasing the likelihood that a student ends up in a high learning environment. Expenditures and Resources do not Measure Quality The most direct implication of these prior analyses is that district expenditure and resources do not provide a measure of school quality. Variations in expenditure or variations in resources such as pupil-teacher ratios do not indicate which schools are adding to student performance. Even though the popular press and others tend to concentrate on resource variations, the previous evidence shows that extensive research and analysis does not confirm the importance of such variations for school quality. Clearly, such variations have important meaning for state and local taxpayers, because they represent real expenditure from tax receipts. They just are not meaningful in an academic performance sense. The evidence also provides insights for the other side of thinking about resources. Providing more resources is unlikely by itself to lead to improved student performance. As schools are currently organized and run, there is little reason to expect added resources to be converted systematically into better student performance. It is very poor public policy to throw taxpayer resources at schools with so little hope that they will be employed effectively. The Importance of Families This discussion has concentrated entirely on the role and effectiveness of schools. Families also have very important effects on student performance. In fact, some believe that the role of families entirely do-ainates that of schools. Existing analysis of student performance makes it clear that student outcc. 3 are highly influenced by families: students from more advantaged families will on average perform better than students from less advantaged families. Policy generally is not directed at changing the relevant aspects of families. In our society, we generally are reluctant to have government intervene in families unless there is extreme and injurious behavior involved. The previous results about schools indicate that simple resource policies are unlikely to be successful in achieving any goals about improved student performance for disadvantaged students. The foregoing results apply equally to performance of students from good and bad backgrounds. Even though programs might have to be adjusted to deal with family disadvantages, no evidence suggests that just providing added funding will be successful. Failure to recognize the importance of families can provide very misleading analysis. Specifically, if advantaged families tend also to value schooling more highly and tend to provide extra resources for their children's schools, it is absolutely essential to distinguish the influences of these two factors. It is entirely incorrect to infer that just changing the spending of the low expenditure school will produce achievement comparable to that of the high spending school. If the difference in observed test performance is largely due to differences in family backgrounds, changing spending while keeping the same family involvement in education would be expected to leave student

8 achievement unaffected. In other words, spending would rise, but the factors truly having a causal influence on achievement would not change, and student achievement would remain the same as before the policy change. In order to change student performance--the rightful objective of education policy making--actions must be taken to alter the factors that causally determine student achievement. Just finding some other factor that is related to a causal influence is insufficient. This situation is precisely the case in the simple comparisons of district spending and student performance. When family backgrounds and other nonschool influences on student performance are not appropriately taken into account, the results of statistical analyses will provide a misleading indication of what would be expected from a change in school resources. The econometric evidence that was reviewed previously (Table 1) is based on statistical approaches to separate family and other influences from the influences of schools. When that is done appropriately, variations in school resources are seen to have little to no systematic influence on student performance. Overall Interpretation of Available Resource Findings The general conclusion that variations in resources across schools do not systematically affect student performance surprises some. Economists believe that there is an obvious explanation for this. The most startling feature of schools--a feature distinguishing schools from more successful parts of our economy--is that rewards are only vaguely associated with performance, if at all. A teacher who produces exceptionally large gains in her students' performance generally sees little difference in compensation, career advancement, job status, or general recognition when compared with a teacher who produces exceptionally small gains. A superintendent who provides similar student achievement to that in the past while spending less is unlikely to get rewarded above what would be the case for spending the same or more.- If there are few incentives to obtain improved performance, it should not be surprising to find that resources are not systematically used in a fashion that improves performance. In part, this implies that effective accountability systems must be put into place. The performance of schools must be measured and must be relevant to individual school personnel. Without a focus on student performance and the role of schools in determining performance, it is difficult to imagine that significant school improvement is likely. Performance in Maryland Schools There is no reason to believe that the situation in Maryland schools is different from that in other states. 7 The findings described previously come from research covering the entire nation, and none of the previous analysis indicates that specific states or regions do particularly better than others in terms of the effectiveness of their resource usage. The extensive testing program of the MSPAP 7The studies previously reviewed were drawn from across the United States. While these undoubtedly included students and schools in Maryland, none explicitly focus on the situation in Maryland.

9 does permit direct analysis of spending and performance in Maryland. This analysis provides a basis for considering whether or not there are special performance results in Maryland that differ from the rest of the country. The analysis confirms that the national evidence is relevant for Maryland; i.e., that spending patterns and results in Maryland follow the findings elsewhere. This analysis considers the 1995 performance of students in reading, writing, language usage, mathematics, science, and social studies. The MSPAP tests were designed to measure school performance and to compare school performance to a set of educational goals (see Maryland State Department of Education [1995]). The extensive testing program has been an important component of ensuring accountability in Maryland schools through the broad dissemination of information about student performance. The statistical analysis summarized here follows the structure developed over the past thirty years by researchers interested in understanding the pattern of student and school performance (Hanushek [1979]). In this case, it is aided, however, by the unusually rich data about student performance over time and across grades in Maryland. The analysis seeks to understand how schools influence the fifth grade performance in the different content areas measured by the testing program. The statistical modeling focuses on the set of students whose performance can be observed in both the fifth grade in 1995 and the third grade in By taking into account the earlier performance of students, the statistical analysis can provide much more precise answers about what influences student performance. One explanatory factor for fifth grade scores is the student's third grade scores, implying that the statistical analysis can concentrate on what factors determine the change in performance over time between the third and fifth grades. The third grade scores will capture the main effects of individual ability differences, of prior school and family effects, and of other factors affecting early achievement. Thus, attention is directed at the school and family effects relevant for the time between third and fifth grade test measurement. Models of this form are commonly called 'value-added' models and are generally acknowledged to be the best way to conduct such statistical analyses. The analysis considers school-level performance. The analytical design here explains school performance in each content area by performance of the same set of students in the third grade, by characteristics of the individual students, and by school and district resources applied to their education. Even though aggregate school-level performance is analyzed, little is lost when compared to an analysis of individual student-level performance. First, prior student performance is available for precisely the same students as those who take the fifth grade tests, so the school level analysis involves a straightforward aggregation which has minimal effects on any statistical analysis. Second, the only resource analysis possible employs data at the school and not individual student or classroom level, again implying that aggregation of performance across students has minimal effects on the precision of any statistical estimates. 8 8The estimates of the effects of school resources on student performance will be unbiased with this aggregation. Moreover, any losses in the precision of these estimates will be minimal, because resources are measured only at the level of the schools--implying that even with individual data, the estimated school effects could not capitalize on any within-school variations in resources that might exist.

10 While some statistical and analytical arguments might still be made for generally preferring to analyze individual student performance, the design of the Maryland testing program precludes this. The MSPAP employs a matrix testing scheme, where individual students take randomly assigned subtests within each content area. This approach is an efficient way of developing aggregate school scores, since individual students are not burdened with taking the full set of tests that would be required to measure performance across all subareas of a content area. But, as stated in the score interpretation guide, "Outcomes Scale Scores for individual students are not adequately reliable or valid for use under any circumstances" [emphasis in original] (Maryland State Department of Education [ 1995]). The third and fifth grade student test results first are converted to scale scores according to commonly accepted psychometric techniques. These scale scores then are then identified with specific proficiency levels. The proficiency levels are designed to be comparable across tests and years (whereas the scale scores are not)fl They are also used in setting state goals for performance, where satisfactory performance in a grade/content area for a school means that 70 percent of the students must achieve "satisfactory" performance (i.e., level 3 or better). This analysis concentrates on the proportion of fifth grade students in each school achieving satisfactory or better in the six separate content areas. Technical Details of Analysis The specific models estimated employ logistic regressions of the proportion of students in each school achieving at the satisfactory level or above for each of the fifth grade content areas tested. The logistic functional form recognizes that the proportion satisfactory must fall between 0 and 1, whereas a linear regression would not constrain values to fall in that range. 1 The outcome variable of interest is the log of the odds ratio, or log(s/l-s) where s is the proportion of students scoring at the satisfactory level or above. Each of the estimated models regresses the logistic outcome measure: on the proportion of students in each of the third grade proficiency levels;" on the proportion of each race/ethnic group 9The creation of proficiency scales, a nonlinear transformation of the underlying test scores, is designed to measure specific skills of students. The variations of measured student achievement within the proficiency levels are not reliable indicators of meaningful individual differences in performance. ~ A preliminary statistical analysis indicated that predicted values for a number of Baltimore City schools fell below zero in the linear models of proportion achieving satisfactory scores. This preliminary analysis confirmed that linear regressions were inappropriate. While other functional forms could be employed, the logistic form is the most common for problems such as this one and alternative forms are unlikely to have a substantive influence on the results (see Hanushek and Jackson [1977]). ~Because the proportion of students in each of the proficiency categories must sum to one, the estimated models exclude the bottom category. This specification implies that the estimated effect of each of the third grade proficiency levels is taken in comparison to students at the lowest proficiency level. Similarly, each of the other categorical variables (such as race and ethnicity) are estimated

11 other than white (i.e., American Indian, Asian, African American, and Hispanic); on the proportion of students who are male; on the proportion of students identified as in special education; and on the proportion of students receiving free or reduced price lunch. By including the proportion of students in each separate level of proficiency on the third grade scores, the most general relationship possible with the available tests is estimated. ~2 Two basic samples of students are used in the analysis, and the same set of basic performance relationships is estimated for each. The first contains all students for whom both fifth and third grade MSPAP tests are available and can be matched across the testing years. The second restricts attention to students who remain in the same school for the third and fifth grade. For the larger first sample, an additional variable is added to the analysis to indicate whether the student had changed schools between third and fifth grade. Each sample has strengths and weaknesses for this analysis. Students who moved will have had the combined experiences of different schools, and the measured attributes of their fifth grade school may not accurately reflect their total schooling experiences. On the other hand, nonmovers are a smaller and selected sample of all students being educated in Maryland. By repeating the analysis across the different samples and by obtaining the same findings about school resource effects, there is confirmation that the nature of the sampling of students did not lead to the statistical conclusions. School Resource Measures The focus of the analysis is how resource differences across schools and districts influence student achievement. Three basic formulations of school resources are employed in the analysis. First, aggregate spending per student across districts is used as a summary of the resources available, consistent in large part with the general notion that overall resource levels are key to performance. The analysis considers both current expenditure and instructional expenditure, each of which is compared separately with the average number of pupils attending and the pupil enrollment for the district. Second, the pupil-teacher ratio and average teacher salary in each school are investigated--reflecting the general emphasis on these factors in both policy deliberations and in the complaints. Finally, a series of real resource differences are separately considered, following the basic structure of the analysis of Ronald Ferguson. Specifically, for each school, the-models include the pupil-teacher ratio, the proportion of teachers with five or more years of experience, the proportion of teachers with exactly a master's degree, the proportion of teachers with greater than a master's degree, and the average age of teachers. Each of the teacher variables is defined for the full time teachers in each school, while the pupil-teacher ratio reflects total enrollment compared to FTEs in each building. Baltimore City schools Two separate analyses compare the performance of schools in Baltimore City to those in the rest of the state. First, a common performance model is estimated but an indicator variable is included relative to one omitted category, as identified in the definition. ~2This approach to estimating the value-added models differs significantly from that of Ronald Ferguson, "Statistical Estimates for Testimony in Baltimore City Schools Litigation," mimeo, August 8, 1996 (see below). 10

12 to identify schools in Baltimore City. In this formulation, the estimated parameter for the Baltimore City variable indicates whether average performance in Baltimore City schools is above or below that in other districts of the state, after allowance is made for differences in students and in resources available across schools. Second, performance models are estimated for schools outside of Baltimore City, and performance for Baltimore schools is predicted on the basis of student and school characteristics of Baltimore schools. The predicted performance, based on the operation of other districts, is then compared to the actual performance in Baltimore City schools. This alternative approach provides similar information about average performance in Baltimore City schools after adjusting for student and resource differences except that the adjustments are made entirely on the basis of expected performance found in districts outside of Baltimore City. Results of Maryland Analysis The various estimates of student performance relationships quite uniformly show that higher achievement in the third grade raises fifth grade achievement. They also show that higher proportions of African American students and higher proportions of students on free or reduced lunch reduce fifth grade achievement. The proportion of students in special education has little systematic effect on school differences. The main focus, however, is the effects of school resources and the performance of Baltimore schools. Table 3 provides a summary of the estimates of basic resource effects. As described, three alternative formulations of the resources available to districts and schools were employed. Each provides a qualitatively similar picture of the relationship between resources and student performance. For the total financial resources measured by either current expenditures per ADA t3 or instructional expenditure per ADA, the estimates are overwhelming negative--i.e., more spending is related to worse student performance. For reading, the estimated negative effects are even statistically signifi :at at the conventional 5 percent level. For the six outcome measures and the two different summaries of tc ources, only one estimate (for the effect of current expenditure on writing achievenzent) is ~.. to be positive. The clear conclusion is that the level of spending on schools across districts is not the primary determinant of student performance. In fact, even disregarding the confidence in any estimated results (i.e., the statistical significance), there is not even a positive association between a district's spending and the performance of its students after allowing for other, nonschool differences among students. Table 3 also provides estimates of school resources using the alternative formulations. The other two categorizations of resources (with salaries or with real teacher characteristics) again show little reason to believe that resources are directly related to student performance. For school financial resources, the estimated effect of teacher salary on writing performance is the only statistically significant estimate across the six content areas. Beyond the very few statistically significant results for real resources, the remaining estimates tend to be evenly distributed between positive and negative estimates. In other words, the pattern of availability of common teacher and school attributes does not appear to influence systematically the achievement of students. ~3Alternative estimates of the effect of spending per enrolled student provided qualitatively similar results. Therefore, this analysis concentrates on the conceptually superior measure related to average number of students in attendance. 11

13 The estimated effects of any resource changes are also very small. For example, the positive and statistically significant effect of teacher salary for the writing performance suggests that a $1,000 increase in the average salary of a school in Baltimore would increase the percentage of students scoring satisfactory or better by 0.3 percent, i.e., for the average Baltimore school, the percentage satisfactory would go from percent to percent. Afifty percent increase in the average salary for the typical Baltimore school would move the percentage scoring satisfactory or better just to percent. And, again, the estimates for the other content areas give little confidence that increasing salaries of teachers would have any impact on student performance, at least as the schools are currently run. These estimated performance relationships reinforce the previously presented national results. There is no reason to believe from these results that simply providing more resources to existing schools will lead to improved student performance. Moreover, these results are not driven by simple measurement issues. Instructional expenditure is no more related to school performance than current expenditure. And, the use of specific real resources does not show systematic effects. Table 4 presents similar estimates for all districts except Baltimore City. This approach was taken to ensure that the results were not driven by differences between Baltimore and the rest of the state. The results from this restricted sample of schools are very similar, leading again to little support of pure resource policies. Tables 5 presents information about how well Baltimore City schools perform after adjusting for its student population and its resources. Table 5 presents a summary of the estimated coefficients for an indicator variable for Baltimore City that is included for the estimated models for the entire state (i.e., those previously summarized in Table 3). With a single exception, Baltimore is estimated w.~rse than would be expected on the basis of its student population and the resources devoted ~c~re ~,.n not, this estimated negative effect is statistically significant. " is it--.lant ~ :-.:. magnitude of the estimated low performance in Baltimore. For ~ t::... iaodels, where salary had a positive and significant effect on student rates of satisfactory performance, it would take an average salary increase of $27,800 within a typical Baltimore school to overcome the adverse effect of being in Baltimore. Or, simply on the basis of overall comparisons, if the typical Baltimore City school performed at the level of schools elsewhere in the state given its student population and school resources, the rate of satisfactory performance on the fifth grade writing test--with no increase in resources--would be 35.6 percent instead of 26.8 percent. The overall rate of satisfactory performance in Baltimore City is clearly low, but Baltimore schools are still not performing at a level comparable to what would be expected in the rest of the state. The detriment for writing in Baltimore City is larger than for other content areas, but outside of Baltimore the same students and resources would generally be expected to yield higher satisfactory performance. While social studies rates of satisfactory performance are predicted to remain essentially the same, districts outside of Baltimore are predicted to increase the rates in language usage by 32 percent; in science by 22 percent; in math by 11 percent; and in reading 8 percent. 14 ~4These comparisons use the estimated effect of being within Baltimore City in the models of school financial resources. There are minor differences across the models with different measures of 12

14 Table 6 uses the models estimated for schools outside of Baltimore to predict what achievement should be in Baltimore (given the student population and resources devoted to schools). Again, with a single exception, the average Baltimore school performance is below what would be predicted if Baltimore schools performed at the level of other districts in the state. Tables 7-10 replicate the previous estimates except only students who have remained in the same school from third to fifth grade are included in the estimation. (This sample includes approximately half of all tested fifth graders). The results of this estimation are qualitatively quite similar. There are a few more positive (but statistically insignificant) estimated effects of resources, and predicted Baltimore City performance is not as consistently above actual performance. Nonetheless, even in this more restricted sample, the overall impression is that resource differences are not a significant factor driving school performance differences. Moreover, Baltimore City schools still appear to underperform after making adjustments for student characteristics and available resources. Comparison with the Estimation of Ronald Ferguson Dr. Ronald Ferguson has conducted an analysis of student achievement that on the surface appears quite similar to that presented here but in reality is quite different, t5 His analysis concentrates entirely on performance in mathematics, a content area in which Baltimore City does relatively well in overall student performance and in relation to other districts in the state. More importantly, he conducts his entire analysis in terms of individual student performance on the tests, instead of in terms of school level performance for which the tests were designed and implemented. He estimates a very specific relationship between third and fifth grade scores. He first transforms individual scores into standard deviation units about the mean. This transformation of individual scores is not appropriate because it does not recognize the varying ranges of proficiency on the test. In his statistical models, he then estimates different relationships between third and fifth grade scores when third grade scores are above and are below the mean score on the test. This transformation has several disadvantages. First, varying effects are estimated within the proficiency category that includes the mean individual scale score. Second, it places a rigid relationship on variations within proficiency level, when these individual variations are not meaningful. Third, it makes strong assumptions about the relationship among performance at different proficiency levels. (The approach of providing separate estimates for each of the proficiency levels pursued in the analysis previously reported avoids all of these problems). He also misinterprets his regressions with regard to the performance of Baltimore City schools. In his analysis he shows that average mathematics performance in Baltimore is below that in school resources. Also, as shown in Table 5, only the coefficients in the models of writing, language, and science are statistically significant. The predicted increases are the relative change in performance, so that the language increase from a satisfactory rate of 18.3 percent to 24.2 percent is a performance gain of 32 percent. ~SRonald Ferguson, "Statistical Estimates for Testimony in Baltimore City Schools Litigation," mimeo, August 8,

15 the rest of the state. When he accounts for differences in the racial and ethnic backgrounds of the students and for differences in free and reduced lunch participation, however, Baltimore students still perform worse than other students, but the differences are no longer statistically significant. He interprets this as indicating that the poorer performance of the Baltimore students is not caused by the poor quality of the school system. The proper way to show this, however, would be to consider the effect of Baltimore in the context of his more complete models that include school resources. He apparently never did that. The previously reported results show that Baltimore does significantly poorer than would be expected given its students and resources. His analysis provides no reason to modify the previously reported results. There are fatal analytical problems caused by his improper use of the student performance measures. Even ignoring those, his results are mute about the performance on Baltimore schools (given students and resources) and give little reason to believe that more resources by themselves will improve student performance. Conclusions Students in Baltimore City are performing at significantly lower levels than those in the rest of the state. Across the six content areas of testing, the percentage of students achieving a satisfactory score in a typical Baltimore school is one-third to one-half the rate in Maryland schools outside of Baltimore. While the idea that providing more resources to schools would improve the performance of their students has considerable popular appeal, it is not supported by the evidence. Extensive national evidence plus direct investigation of Maryland schools shows little consistent or reliable relationship between resources and student performance. The national results suggest that improved management that can utilize available resources better is more likely to improve schools than just providing extra resources to existing schools. This conclusion is underscored in Baltimore City schools where current performance is below what would be expected based on the student population and the resources that are available. 14

16 References Betts, Julian R. "Is there a link between school inputs and earnings? Fresh scrutiny of an old literature." In Does money matter? The effect of school resources on student achievement and adult success, edited by Gary Burtless, Washington, DC: Brookings, Burtless, Gary. "Introduction and summary." In Does money matter? The effect of school resources on student achievement and adult success, edited by Gary Burtless, Washington, DC: Brookings, Card. David, and Alan B. Krueger. "Does school quality matter? Returns to education and the characteristics of public schools in the United States." Journal of Political Economy 100, no. 1 (February 1992): Chaikind, Stephen, Louis C. Danielson, and Marsha L. Brauen. "What do we know about the costs of special education? A selected review." Journal of Special Education 26, no. 4 (1993): Congressional Budget Office. Trends in educational achievement. Washington, D.C.: Congressional Budget Office, Consortium on Productivity in the Schools. Using what we have to get the schools we need. unknown: Consortium on Productivity in the Schools, Cook, Michael D., and William N. Evans. "Families or schools?" Explaining the convergence in white and black academic performance", mimeo, Glass, Gene V., and Mary Lee Smith. "Meta-analysis of research on class size and achievement." Educational Evaluation and Policy Analysis 1, no. 1 (1979): Grissmer, David W., Sheila Nataraj Kirby, Mark Berends, and Stephanie Williamson. Student achievement and the changing American family. Santa Monica, CA: Rand Corporation, Hanushek, Eric A. "Conceptual and empirical issues in the estimation of educational production functions." Journal of Human Resources 14, no. 3 (Summer 1979): "The economics of schooling: Production and efficiency in public schools." Journal of Economic Literature 24, no. 3 (September 1986): "The impact of differential expenditures on school performance." Educational Researcher 18, no. 4 (May 1989): "The trade-off between child quantity and quality." Journal of Political Economy 100, no. 1 (February 1992): "Money might matter somewhere: A response to Hedges, Laine, and Greenwald." Educational Researcher 23, no. 4 (May 1994):

17 --. "Assessing the effects of school resources on student performance: An update". Working paper No. 424, Rochester Center for Economics Research, University of Rochester, Hanushek, Eric A., and John E. Jackson. Statistical methods for social scientists. NY: Academic Press, Hanushek, Eric A., and Steven G. Rivkin. "Understanding the 20th century growth in U.S. school spending." Journal of Human Resources 32, no. 1 (Winter 1997). Hanushek, Eric A., Steven G. Rivkin, and Dean T. Jamison. "Improving educational outcomes while controlling costs." Carnegie-Rochester Conference Series on Public Policy 37 (December 1992): Hanushek, Eric A., Steven G. Rivkin, and Lori L. Taylor. "Aggregation and the estimated effects of school resources." Review of Economics and Statistics 78, no. 4 (November 1996). Hanushek, Eric A., with others. Making schools work: Improving performance and controlling costs. Washington, DC: Brookings Institution, Heckman, James S., Anne Layne-Farrar, and Petra Todd. "Does measured school quality really matter? An examination of the earnings-quality relationship." In Does money matter? The effect of school resources on student achievement and adult success, edited by Gary Burtless, Washington, DC: Brookings, Hedges, Larry V., Richard D. Laine, and Rob Greenwald. "Does money matter? A meta-analysis of studies of the effects of differential school inputs on student outcomes." Educational Researcher 23, no. 3 (April 1994): Lapointe, Archie E., Nancy A. Mead, and Gary W. Phillips. A worm of differences: An international assessment of mathematics and science. Princeton, N J: Educational Testing Service, Maryland State Department of Education MSPAP and Beyond, Maryland School Performance Assessment Program: Score Interpretation Guide. Baltimore, MD: Maryland State Department of Education, McKnight, Curtis C., F. J. Crosswhite, J. A. Dossey, E. Kifer, J. O. Swafford, Ken J. Travers, and T. J. Cooney. The underachieving curriculum: Assessing U.S. school mathematics from an international perspective. Champaign IL: Stipes Publishing Co., Mullis, Ina V. S., John A. Dossey, Eugene H. Owen, and Gary W. Phillips. NAEP 1992 mathematics report card for the nation and the states. Washington, DC: National Center for Education Statistics, National Research Council. Everybody counts: a report to the nation on the future of mathematics education. Washington, D.C.: National Academy Press, Peltzman, Sam. "The political economy of the decline of American public education." Journal of Law 16

18 and Economics 36 (April 1993): Silva, Fabio, and Jon Sonstelie. "Did Serrano cause a decline in school spending?" National Tax Journal 48, no. 2 (June 1995): Speakman, Robert, and Finis Welch. "Does school quality matter? --A Reassessment". Texas A&M University (mimeo), January U.S. Department of Education. The condition of education, Washington, DC: National Center for Education Statistics, Williams, Paul L., Clyde M. Reese, Jay R. Campbell, John Mazzeo, and Gary W. Phillips. NAEP 1994 reading. A first look. revised ed. Washington, DC: National Center for Education Statistics, Word, Elizabeth, John Johnston, Helen Pate Bain, B. DeWayne Fulton, Jayne Boyd Zaharies, Martha Nannette Lintz, Charles M. Achilles, John Folger, and Carolyn Breda. Student~teacher achievement ratio (STAR), Tennessee's K-3 class size study: Final summary report, Nashville, TN: Tennessee State Department of Education,

19 Table 1. Percentage Distribution of Estimated Effect of Key Resources on Student Performance (377 studies) Resources number of estimates Statistically significant + I Statistically insignificant + Insignificant, unknown sign Teacher-pupil ratio Teacher education Teacher experience % 13% 27% 25% 20% Source: Hanushek, Rivkin, and Taylor [1996] 18

20 Table 2. Public School Resources, Resource Pupil-teacher ratio % teachers with master's degree median years teacher experience current $1,903 $2,402 $3,269 $3,864 $4,116 $4,919 $5,582 expenditure/ada (I $'s) Source: U.S. Department of Education [1994] 19