Recommendations to Improve the Louisiana System of Accountability for Teachers, Leaders, Schools, and Districts: Second Report

Transcription

1 Recommendations to Improve the Louisiana System of Accountability for Teachers, Leaders, Schools, and Districts: Second Report Douglas N. Harris Associate Professor of Economics University Endowed Chair in Public Education Director, Education Research Alliance for New Orleans Tulane University For presentation at the January, 2015 BESE and Accountability Commission Meetings 1

2 Summary Accountability is an important part of Louisiana s system of public education. While the current accountability structure has some strengths, it also has weaknesses that undermine quality instruction and student outcomes in particular, teacher and school accountability are misaligned and the School Performance Score (SPS) does a poor job of measuring what schools contribute to student learning. The school accountability system also creates uneven incentives to improve performance. The following recommendations are intended to avoid these problems, while maintaining the system s current advantages. Following these recommendations would give Louisiana arguably the best accountability system in the country. Recommendation #1: Focus more on student growth in order to better measure the performance of schools. The School Performance Score (SPS) is based on test score levels, which mainly reflect the achievement levels of students when they started school, not how well their recent teachers taught them. Yet, student growth is now almost absent from the SPS. The state should aim for a split between growth and achievement levels. This would do a better job of measuring true school performance by accounting for where students are when they enter the school. This would also improve alignment with teacher accountability, which includes student growth as 50% of the evaluation. Recommendation #2: Reduce uneven incentives and avoid incentive cliffs by increasing SPS points more gradually as students move to higher performance levels. Currently, schools receive 36 times as many points for getting a student from a score of 17 to 18 on the ACT than they do for getting a student from 18 to 19. While the apparent goal of this extreme approach is to provide incentives for schools to meet certain levels of proficiency, the policy creates steep cliffs that may have the unintended effect of reducing achievement for the lowest- and highest- performing students. Giving more weight to achievement growth also helps create more even incentives (see Recommendation #1). Recommendation #3: Create a larger number of school letter grades in create incentives for all schools to improve. The SPS is on a 150- point scale and schools are grouped into only five categories (A, B, C, D, and F). Schools in the middle of the categories have little incentive to take additional steps because moving the SPS the equivalent of half a letter grade is extremely difficult. If there were more categories, the next level would always be within reach. Recommendation #4: Add students college entry and first- year college persistence into the high school and district SPS calculations. This would address one of the main problems with accountability systems generally that the measures do not align with the outcomes of greatest long- term interest. 2

3 Recommendation #5: Eliminate the first over- ride provision in the teacher accountability system, which automatically places teachers who are Ineffective on either measure in the Ineffective performance category. The purpose of this provision is ostensibly to ensure that low- performing teachers are held accountable, and to avoid having the classroom observations serve as a loophole from tenure removal and/or dismissal. This is a legitimate concern, but the over- ride also makes the stakes disproportionate to the validity of the measures. Recommendation #6: Commission a full- scale third party evaluation of the entire accountability system focused on educator responses and student outcomes. Accountability systems are intended to send message and create incentives for educators to improve practice. There is no way to know whether this is happening without collecting data about educator perceptions, understanding, and reactions. The remainder of this document provides additional details and adds other, more modest recommendations. States across the country have struggled with the design of their accountability systems for more than two decades. With these changes, Louisiana would have one of the best accountability systems in the country. Rather than weakening accountability, these recommendations make accountability smarter and make it more likely to state will improve students academic performance. 3

4 Accountability Limitations from the First Report The first report from this evaluation listed specific criteria that apply to any accountability system. Based on these criteria, the overarching strength of Louisiana s accountability system is that all the levels focus on the same student outcome measures. There are also strong incentives to include as many students in the accountability system as is reasonably possible. The information provided to teachers through the Compass system is designed to provide teachers feedback on practices that are likely to increase student outcomes, so, in this sense, the information provided aligns with accountability. There are also several solvable limitations: School accountability is poorly aligned with teacher accountability the student outcome portion of the teacher accountability system focuses on student learning or growth, but school accountability focuses almost entirely on student achievement levels. School accountability violates the cardinal rule by not holding schools accountable for what schools contribute to student learning; rather, it focuses on snapshots of student achievement as particular points in time. The system, at all levels, creates uneven incentives for educators to improve. The system places students, teachers, and schools into broad categories, creating incentive cliffs where only particular teachers and schools have significant incentives to improve and, even then, the incentive is to focus on specific, and sometimes, arbitrary groups of students. The teacher accountability system, and specifically one of the over- ride provisions, attaches stakes to the measures that are disproportionate to their validity and reliability. While the state does conduct regular diagnostic tests of the accountability system with the student testing (and now teacher evaluation) data that LDOE collects, there has been no systematic and rigorous evaluation of the most important elements the responses of educators and effects on student outcomes. The recommendations below start with school- level accountability because this is the level with the most current problems. Then, I proceed to school leaders and teachers. In some cases, the recommendations are for explicit changes in policy and in other cases, where we have less evidence, the recommendations are more tentative, e.g., developing pilot programs or exploring promising ideas. 4

5 School Accountability Recommendations Recommendation #1: Focus more on student growth in order to better isolate the performance of teachers and schools. The SPS is based mainly on the percentages of students who reach various performance standards, but this only partly captures school performance and mainly rewards schools that are able to attract students who are high- performing before they entered the school. The growth bonuses (up to 10 points on top of the 150) are a step in the right direction, but as this is only 7.5% of the base potential points, additional steps should be taken to make student growth play a larger role. To address this, I recommend adding average teacher value- added to the SPS and assigning 50% of the 150 SPS points to this measure. This would not only do a better job of measuring the performance of the schools, but improve alignment between teacher, school, and district accountability. It would also send a clear message to school and district leaders that they should focus on hiring, retaining, and developing the best teachers. The above approach also has the advantage of building value- added into the base of the SPS rather than having to add progress points. The very idea of progress points suggests that it is less important than the main 150- point scale, but, as I have shown, growth should play a more fundamental role. This approach also rewards growth for all students, not, as with the current progress points, just those who are not proficient. 1 The system could be set up to reward schools somewhat more for generating growth among low- performing students, but the present goes much too far, not rewarding growth for proficient students at all. The new school value- added measure would enter into the SPS formula on a smooth sliding scale where small increases in average teacher value- added yields small increases in SPS points. This would avoid creating any new incentive cliffs. Note that the average teacher classroom observation rating would not be included in the SPS because the classroom observations are determined within the school. This would tend to inflate the classroom observations and therefore the SPS. Given the argument that the system focuses too much on levels and too little on student growth and learning, it is worth considering why we should not shift toward 100% growth. First, in the elementary grades, it would create little incentive to 1 Note that any rules for the calculation of teacher value- added that restrict which students are included in the calculation would need to be revisited when teacher vale- added is averaged for the SPS. If it is necessary to exclude students in the teacher value- added calculation, then some additional steps will be necessary to add these students into the school value- added estimate. 5

6 improve achievement in the early grades. If students did poorly on the first test they took in grade 2, then this would make the subsequent growth bonus and teacher value- added components look higher. Second, standardized tests are not generally very informative at the high end of the scale there is no room for growth. Schools should no longer be rewarded so highly for serving students who had high scores to start with, nor should they be punished for not generating growth on a measure where growth is hard to identify. Recommendation #2: Reduce uneven incentives and avoid incentive cliffs by increasing SPS points more gradually as students move to higher performance levels. As noted in the first report, placing student performance into a small number of categories inevitably means that schools will focus on those who are near the performance bars. The current method gives schools no points for moving students from Unsatisfactory to Below Basic, but then a large jump in points for moving from the Below Basic to Basic. This creates an incentive for schools to focus on Below Basic students because they offer the greatest bang for the buck in the SPS. But this also means devoting less time and attention to students in the lowest and highest performance categories (e.g., Unsatisfactory) and Advanced). The current system also gives too much weight to the performance categories given the cut- offs are chosen somewhat arbitrarily. Instead, I propose increasing the points more gradually for getting students to higher levels of achievement. There are currently five performance categories (and therefore four cut- offs) on a 150- point scale. For example, you might give schools 30 points for Approaching Basic, 80 points for Basic, 110 points for Mastery, and 150 for Advanced. This still involves giving more points for getting students to Basic (80-30=50 points), but still gives a smaller amount (30 points) for getting to each other level. This is just one example. There are other ways that the SPS point distribution could be more gradual. Recommendation #3: Create a larger number of school letter grades to create incentives for all schools to improve. Specifically, add +/- extensions to the SPS and DPS letter grades. Placing whole schools into broad performance categories with letter grades creates problems similar to placing individual students into broad categories like below basic. Allowing for D+, C-, and so on allows for greater nuance while maintaining the basic intuition that letter grades provide. This way all schools will have an incentive to improve, whereas currently only those who are at the high end of the B- D categories have much incentive. Recommendation #4: Add students college entry and first- year college persistence into the high school and district SPS calculations. There is little question about the need for the vast majority of students to go on to some type of post- secondary education a two- year college or a four- year college. It is now relatively easy to measure these outcomes in the National Student Clearinghouse, which tracks students if they attend essentially any public or private college nationally. 6

7 I recommend a focus only on college entry and first- year (fall- to- spring) persistence because these are most under the control of high schools. It is the responsibility of counselors and teachers in high schools to guide students toward colleges that are a good fit for their skills and preferences. Even on- time college completion can take many years, which means that any steps taken by high schools in the short term will only show in completion many years down the road. Also, the longer students are in college, the more their completion depends, not on the high schools, but the colleges themselves. In developing the point system, it is important to recognize some basic facts from economic research: (a) Every year of education is associated with increased labor market success, even when this does not result in a degree or other credential; (b) Four- year degrees have a greater return than two- year degrees, but both are associated with a host of long- term economic, health, and social outcomes; and (c) Students are more likely to graduate if they do not transfer. 2 As a general rule, more points should be given to credentials and other intermediate outcomes that are most closely associated with the long- term outcomes. The evidence therefore implies four- year institutions should get somewhat more points than two- year institutions and persistence in the same institution should get more points than transfers of any sort. But, to avoid incentive cliffs, some points should be awarded for all types of entry and persistence. In addition to determining how the points would vary for different types of institutions and entry versus persistence, decisions would have to be made about how much weight the college measures should receive compared with the current measures, high school test scores and high school graduation. Above, I recommended focusing more on student learning or growth specifically, calculating the predicted test scores and rewarding schools based on how well students do compared with those predictions. I recommend the same here with college outcomes. For high schools, college outcomes can be predicted based on 8 th grade test scores and student demographics. 3 In fact, the same value- added model already being used could be applied to college outcomes. 2 There are several reasons for this. For example, when students transfer, they often lose many of their credits, requiring them to stay in college longer in order to receive a credential. Also, when students transfer, they become detached from friends and mentors who might be able to help them, academically and socially. This is not to say students should never transfer, but the goal should be to come up with a good match from the beginning. 3 Some high schools begin at earlier grades. The point here is that the model should account for information from the year just before students enter the school whose accountability measure is being calculated. 7

8 Notice that this recommendation excludes reference to career outcomes. This is not to say that career preparation is not one of the objectives of Louisiana schools. On the contrary, even the college outcomes are really stepping stones to career and other long- term outcomes. But there are several problems with using career outcomes as performance measures for schools. First, career success has to be judged over a much longer period of time than college outcomes especially with college outcomes limited to the first year, as is recommended above. Even college graduates often start their careers either unemployed or working in low- paying jobs that are ill- suited to their backgrounds. This is mainly because they have little job experience, not because of anything wrong with their schooling. Also, some job outcomes that may seem like failures in the short term are actually successes in the long- term, e.g., when students take unpaid internships. For these reasons, it is difficult to judge career success in the short term, and school accountability cannot afford to wait for the long term. Second, short- term job success is based to a significant degree on local economic conditions. Should schools be punished because the unemployment rate happens to be high in their community? No, this goes against the principle that we hold people accountable for what they can control. In contrast, college opportunities are fairly evenly spread across the state. Every county has at least a community college. Third, most of what we now expect from schools, as defined in state standards, are more directly aimed at getting students ready for college. This makes sense for the same reason that college outcomes are included as school performance measures. Therefore, if we want to align the accountability system with state standards, focusing on college outcomes makes more sense. The fact that it would be unwise to build career outcomes into school accountability does not mean nothing can be done to encourage schools to help students prepare for careers. For example, funding for vocational programs encourages schools to provide direct career preparation. In summary, college outcomes are arguably the most important educational outcomes, yet the accountability system ignores this. Now that the data are available, this problem can and should be fixed and high schools should bear more of the responsibility of preparing students not just academically, as well as financially and socially for the rigors of college life. Additional School Accountability Recommendations Below are some additional, but more minor recommendations to improve school accountability: (a) Align the performance thresholds for ACT between TOPS and SPS. The key cut- off for school accountability with the ACT is set at 18, but this does 8

9 not align with TOPS eligibility thresholds. It would help simplify matters for schools if these targets were aligned. (b) Align the EOC and LEAP student performance categories. The LEAP categories are Advanced, Mastery, Basic, Approaching Basic, and Unsatisfactory while the EOC categories are Excellent, Good, Fair, and Needs Improvement. This adds unnecessary complexity. The state should use the same number of categories and assign the same labels. Also, these cut- offs should be aligned so that a student who is fair on LEAP is likely to stay in that category when shifting to the EOCs. (c) Rename graduation index to reflect the fact that it is also based partly on test scores. This will help clarify the relative weight assigned to the different measures. The District Performance Score (DPS) is a roll up of the SPS therefore all the recommendations above would indirectly apply to the DPS. School Leader Accountability Recommendations I recommend no major changes in school leader accountability for reasons explained below, but do recommend that the state take some exploratory steps that might eventually lead to accountability changes. As indicated in the first report, the focus of principal accountability on student learning outcomes is more indirect and less standardized. Principals are held accountable for student learning directly through their SLTs, negotiated with district officials. They are also accountable indirectly because few principals have tenure and those who do only have it as teachers. Districts are held accountable for the SPS/DPS and have the ability to dismiss principals without due process, e.g., if they perceive student performance to be inadequate. I am advising against using the SPS for state- level principal accountability, but recognize that the issue is not as clear- cut as with other recommendations. Many of the goals we have for students are difficult to measure. Holding principals accountable for student outcomes in more indirect and less standardized ways allows principals more latitude to target goals that the SPS might miss. Also, the SPS is determined largely by teachers and, especially in the short run, principals do not have complete control over who teaches in their schools. Recent changes in law do give principals much more authority over teachers than in the past (see later discussion of teacher tenure and dismissal), but, even for an aggressive principal, it could take 3-4 years to revamp an ineffective teacher workforce. Even if all teachers were dismissed, it takes time to attract quality teachers, build a school culture, and take other steps necessary for school improvement. Also, principals have limited 9

10 control over who applies to their schools. Rural schools, for example, have difficulty attracting high- performing teachers. This is an ambiguous case with the cardinal rule. Principals have some control over their teachers, but it is far from complete. Given the difficulties of evaluating principals based on student outcomes, a more promising approach, building off the teacher accountability system, is to begin building a practice- based evaluation system for principals. A group of principals, and perhaps outside experts, could visit the school, survey staff and students, and use a rubric for evaluating principal practices. This is analogous to what we are doing with teacher evaluations and Compass, based on classroom observations. I recommend that LDOE look into this option on a preliminary basis through discussions with state administrator organizations and national experts and perhaps move forward with a pilot. Such a system has been in place for many years in England and there have been recent advances in the U.S. 4 In the meantime, I view the more indirect and practice- based approach to evaluating school leaders to be the preferable form of school leader accountability. Principals are already held accountable for student outcomes because those who hire them are held accountable for the SPS/DPS. Keeping the SPS and value- added out of the principal measure should also be able to take a broad perspective on all educational goals, not just those that are easily measured. LDOE should consider calculating principal value- added measures and issuing reports that describe patterns of variation (e.g., variation in performance overall an din certain kinds of schools) both for the state as a whole and specific districts. However, I recommend stopping short of providing to districts individual principal value- added measures. This recommendation helps avoid the problems with principal value- added measures noted above and focus attention on principal practice. In addition, this would help introduce the idea to districts and provide more analysis on which to reconsider the use of principal value- added in the future. Teacher Accountability Recommendations Recommendation #5: Replace the first over- ride provision in the teacher accountability system, which automatically places teachers who are Ineffective on either measure in the Ineffective performance category. The purpose of this provision is ostensibly to ensure that low- performing teachers are held accountable, and to avoid having the classroom observations serve as a loophole from tenure removal 4 Andrew C. Porter, Joseph Murphy, Ellen Goldring, Stephen N. Elliott, Morgan S. Polikoff & Henry May (2008). Vanderbilt Assessment of Leadership in Education: Technical Manual 1.0. Downloaded September 14, 2014 from: center/school- leadership/principal- evaluation/pages/vanderbilt- Assessment- of- Leadership- in- Education- Technical- Manual- 1.aspx. 10

11 and/or dismissal. This is a legitimate concern, but the over- ride also makes the stakes disproportionate to the validity of the measures. I recommend replacing the current over- ride rule with one that applies only when the other measure is less than or equal to 3 (Effective: Emerging). In other words, a teacher in the lowest performance category on either student outcomes or classroom practice would only be automatically in the Ineffective category for overall performance if there are other measure was 3 or below. Unlike the current over- ride, this would mean that teachers who appear highly effective on one measure could not end up in the Ineffective category. This would also guarantee that both measures are still contributing to the final performance determination and therefore reduces the stakes attached to any individual measure, and reduces the chance that a truly excellent teacher is wrongly dismissed. This would also continue to avoid the above loophole: it would be difficult for a principal to save truly ineffective teachers by placing them in the Highly Effective category. In the appendix, I discuss some other ways in which a screening approach would also help avoid the above truly Ineffective teachers receiving other ratings. This approach would not require any changes in state policy, but is instead intended as guidance to school districts. Additional Teacher Accountability Recommendations While not listed among the main recommendations, the following changes would also improve teacher accountability. (I use letters to distinguish these subsidiary recommendations from the main recommendations.) Also, note that one of the main recommendations under school accountability is to place average teacher value- added into the SPS. The value- added measure therefore takes on increased importance and deserves more attention. (a) Keep the basic structure of the current teacher value- added model, but make some small improvements in the statistical methodology. As noted in the first report, the current model is generally consistent with the state of the art in the research community, but there are some ways in which it could be made better. First, the current model considers only the outcomes of the most recent cohort of students and therefore ignores teacher performance from prior years and increases statistical uncertainty. Therefore the first recommended change is to use multiple cohorts of students, i.e., create a rolling average. 5 5 This approach cannot be applied to all teachers because they have too few years of experience or breaks in their careers or positions that make it impossible to calculate value- added two years in a row. In those cases, only one year of value- added would be used. Some might object that this treats 11

12 Without the rolling average the new SPS, with more even balance of levels and growth, will be less stable over time than the current measure. I also recommend assigning the highest possible prior score to students when this information is missing in the data system (e.g., because a student recently moved from a private to a public school). 6 This will encourage schools to report data on all students and not indirectly try to exclude them by creating missing data. As noted earlier, value- added measures can be subject to ceiling effects and therefore yield inaccurate performance measures for schools serving high- performing students. This problem has also been recognized with teacher effects and the LDOE has taken the reasonable approach of capping the predicted achievement of students, so that students with very high prior achievement can still show improvement. Another way to accomplish the same thing would be to base value- added on the scores of students who are below the Advanced level, and assign the highest possible value- added to students already in the Advanced category if they remain in that category. I recommend this latter approach because it would serve the same purpose and be easier for educators to understand. (b) Keep the second over- ride, but align the percentile ranges with the percentiles used to assign value- added scores to performance categories. With the second over- ride, a school leader can use discretion to change a teacher s student outcome performance category from 2 to 3 or vice versa if teacher value- added is in the 20 th to 80 th percentiles. However, the cut- off between performance categories 1 and 2 on value- added is at the 10 th percentile. This adds confusion without improving the system. Aligning the value- added percentiles with the second over- ride would improve simplicity. (c) Carry out additional diagnostic testing for bias in teacher value- added measures. Studying bias is more difficult than it seems. The usual approach is to identify groups of teachers for whom there is some reason to think are at a disadvantage, and then compare the average value- added for these teachers with others. This is not a perfect test, but it is a useful start. These diagnostics should be carried out, and publicly reported, for the following groups of teachers: teachers unequally but, taken to its logical end, such logic would imply that we should not improve the measures for some teachers because we can t improve them for all teachers. Such differences are also inevitable with any performance measure. We would never use the same classroom observation for a kindergarten teacher as for a high school teacher. Also, all school and teacher performance measures reflect the different numbers of students that educators have been responsible for. The best course is to use all the information available for every teacher and that is what the current proposal does. 6 It would also be wise to add in a missing data indicator for these students. 12

13 Teachers assigned more than 10 percent special education or ELL students. Teachers in courses where the content may not be well aligned with the test, such as remedial and advanced courses in middle school. (The use of EOCs in Louisiana makes this much less prevalent in high school.) Teachers with more than 50 percent of their students below the 10 th percentile or above the 90 th percentile. Teachers with more than two- thirds of their students eligible for free or reduced price lunches. Also, LDOE should check for implicit tracking in middle schools; test whether the mean test score level is similar across classrooms in each subject. If there is such tracking going on, this suggests that the middle school value- added measures may be biased. To their credit, some of these tests have already been conducted by the LDOE staff. If any of these diagnostic tests suggest the possibility of some type of bias, the model should be adjusted and/or a flag should be added to scores that are susceptible to bias. Information about the results from these tests should be made publicly available. (e) Keep the Ineffective cut- offs at the same or lower percentile thresholds, or add confidence intervals. Researchers almost uniformly recommend reporting and using confidence intervals for measures used as part of high- stakes decisions. The same is true with value- added because we are more certain about the value- added of some teachers than others and because it is unfair, unwise, and possibly illegal to dismiss teachers whose true performance has a reasonable probability of being near or above average. However, the current system accounts for statistical uncertainty in other ways, by placing teachers in low- performing categories only if they are in the bottom 10th percentile in value- added. Being labeled Ineffective based on the classroom observations is even more rare. This means that even when confidence intervals are relatively wide, it is highly likely that teachers in the bottom category are below- average. Statistical confidence is also partly addressed in the current model through the shrinkage method, which reduces the chances that teachers with few students are placed in the Ineffective category by chance alone. I recommend continuing to current approach to addressing statistical uncertainty because it is reasonable and maintains some stability in the accountability system. 13

14 (f) Keep the current basic structure rather than switching to an alternative such as student growth percentiles (SGPs). The logic of value- added measures is similar to SGPs. In both cases, we use prior information about student performance and background to predict where students will end up in some later year. The difference between this prediction and the actual student outcomes is the measure of teacher performance. The first substantive difference is that SGP measures the outcome and the prediction on a different scale. As the name implies, the SGP focuses on the students percentiles rather than their scaled scores. This has the partial advantage of being more intuitive to parents and educators who are accustomed to hearing about percentiles. But for accountability purposes, we are interested in the performance of the teachers not the students, and value- added measures can be easily converted into percentiles afterwards, as LDOE now does (see above). The second substantive difference is that SGP measures do not account for any student background measures or school characteristics such as class size. This places some teachers at a disadvantage compared with other teachers. Since Compass rewards high value- added teachers, this will tend to unfairly punish teachers serving disadvantaged students exactly the opposite of what we want if the goal is to get all students to proficiency. There is a common misconception that the calculation of SGPs is simpler than value- added. In reality, the reverse is arguably true. SGPs impose fewer assumptions on the test scales, which requires more complex statistical analysis. In any event, the differences in results between VA and SGP are no greater than the differences between different types of VA models. This is not surprising since the underlying logic of both approaches is the same. Another perceived advantage of SGP is that is allows for creating intuitive pictures for parents that communicate how each child is doing and how much improvement is necessary to reach different performance standards. But this has nothing to do with the VA versus SGP. It is just that the people who developed the SGP spent more time thinking about how to communicate results to parents. Recognizing this, LDOE is already working on providing more information to educators about what each student s predicted achievement is for the coming year. (g) To improve communication of the state s intended high standards for all students, discussion of value- added should refer to predicted scores not expected scores or typical scores. One of the main misunderstandings about value- added is that it sets lower standards for some groups of students. This is not the case, however, that misunderstanding is reinforced by the current language. Students do have different predicted scores, but they should not have different expected scores. Changing the way we talk 14

15 about value- added will go a long way toward building the support for it that is deserved. 15

16 A Full- Scale Evaluation The above recommendations apply to specific levels of accountability schools, leaders, or teachers. The final recommendation cuts across all of them. Recommendation #6: Commission a full- scale third party evaluation of the entire accountability system focused on educator responses and student outcomes. No accountability system, no matter how carefully designed, can anticipate every conceivable problem. The way policymakers often think educators can or should respond is often at odds with reality and with the complexity of teachers jobs and the diversity of their thinking about the goals of schooling and means of achieving these goals. Therefore, policymakers cannot simply rely on good design. There has to be rigorous evaluation, preferably done by external evaluators, that involves deep analysis of random samples of anonymous schools and teachers and goes beyond the data sent to the state as part of the accountability system. Anonymity is essential to ensure that educators are honest in their responses and selecting sites at random is necessary to ensure that those selected are representative of the state as a whole. Such an evaluation also needs to be large enough to account for the variety of school and classroom circumstances. For example, we would expect schools with A letter grades to respond differently to accountability than schools with F grades. Likewise, some schools serve students with lower family incomes and levels of parent education. Prior research shows not only that student background influences achievement, but that schools instruct students differently across backgrounds, even when they have the same level of academic achievement. Within each selected school, the evaluators should seek out teachers whose measured performance covers the full spectrum, on value- added, SLTs, and classroom observations. From prior research, we know that even high- performing schools have low- performing individuals. This evaluation should collect and analyze data from educators on at least the following key topics: General instructional responses to accountability Re- allocation of attention and resources to students near performance cliffs (see earlier discussion) Gaming behavior in which steps are taken that are not instructionally sound, but that nevertheless increase measured performance Perceived validity of performance measures at all levels (teachers, leaders, and schools) Perceived credibility of sanctions and incentives Overall impressions of whether and how accountability improves and weakens the quality of instruction and leadership 16

17 It is ultimately the reactions of educators to accountability that can drive school improvement and we can only gauge these reactions with such an evaluation. The above evaluation is designed to better understand how well the system improves practice and how it might be improved, even beyond the other recommendations made here. Conclusion Louisiana s system of accountability for districts, schools, leaders, and teachers Is similar to other states and above- average in its focus on student growth and multiple measures of teacher performance. The recommendations outlined above are meant to address the weaknesses outlined in the first report and make the state a national leader. 17

18 Appendix: Screening Method To see how classroom observations can be improved, it is worth looking to the medical profession. It is common for doctors to screen for major diseases, using procedures that can identify the vast majority of people who could possibly have the disease. Some patients who test positive will have the disease and some will not that is, some will be misclassified as false positives. Those who test positive on the screening test are given another, gold standard test that is more expensive than the initial test but much more accurate. They do not average the screening test together with the gold standard test to create a weighted average, as we do with some of our educator performance measures. Instead, the two pieces are considered in sequence. Ineffective teachers could be identified the same way. Value- added measures, like medical screening tests, are relatively inexpensive, but some would argue not very accurate. So, a value- added score should lead us to collect additional information (e.g., more classroom observations, student surveys, portfolios) to identify truly low- performing teachers and to provide feedback to help those teachers improve. The most obvious problem with this approach is that value- added measures are not designed to capture all potential low- performers. They are statistically noisy, for example, so many low- performers will get high scores by random chance; no additional data would be collected and the low performance would go undetected. Some teachers would slip through the cracks. The false positive rate could also be very large. With value- added as the only screener, too many teachers would be screened as tentatively ineffective, so that additional information would be collected at some point. Using multiple years of prior data in the screening process would help, but if teacher performance varies over time, then prior years might not be as relevant to assessing current performance. A real trade- off exists here in how to use multiple years of data. For this reason, it would be inadvisable to make value- added the sole screener. Instead, additional measures, such as past performance on other measures, could be used as a screener in conjunction with value- added. If teachers failed on either measure, it would trigger collection of additional information. There is a second way in which value- added could be used as a screener not of teachers, but of the classroom observers who rate teacher practice. As with value- added, observations also suffer from validity and reliability issues. Two observers can look at the same classroom and see different things, meaning that inter- rater reliability is low. That problem is more likely when the observers vary in how they are prepared for observing teachers or in how they define teacher effectiveness. The example given earlier is a case in point. The classroom observer might be aware of the teacher s prior performance, and this may color her observations. In general, under traditional evaluation systems, principals give high scores to the vast majority of teachers. Consciously or not, they might think, I know and like this teacher so I 18

19 will give her a high observation score. Or, as the leaders of the schools, principals may worry that low scores reflect poorly on their own performance. While there is no way to eliminate these types of problems, value- added measures could be used to reduce them. To see how, note that researchers have found consistent, positive correlations between value- added and classroom observations scores. They are far from perfect correlations (mainly because of statistical noise), but they provide a benchmark against which we can compare or validate the scores across individual observers. Inaccurate classroom observation scores would likely show up as being weakly correlated with value- added measures of the same teachers. In particular, if observers fell into the common problem of giving high ratings to almost all teachers, then the comparison with value- added might make this problem evident. Conversely, if observers based their scores on what they already know about teachers value- added, the observer ratings would be distorted in ways that make the correlations might be very high, which might also be a red flag. This approach will work less well when part- time observers are used because they will have fewer observations. A smaller sample size means less confidence in the correlation estimates. When flags are raised, an additional observer might be used to make sure the information is accurate. In other words, value- added, along with other measures, can help screen the performance of not only teachers, but observers as well. Used in these ways, value- added would be a key part of the process possibly a significant part of the decision according to RTTT without being the determining factor in personnel decisions. As noted in the main text, using the screening approach could improve both the efficiency of teacher accountability by reducing the amount of information that has to be collected, and by improving the quality of classroom observations. 19