Ohio Department of Education Teacher Evaluation System (OTES) Pilot CSP

Transcription

1 Ohio Department of Education Teacher Evaluation System (OTES) Pilot CSP Final Report June 29, 2012

2 Final Report for CSP PROJECT: ODE Teacher Evaluation System (OTES) Pilot CLIENT REFERENCE NUMBER: CSP MGT PROJECT NUMBER: MGT PROJECT MANAGER: Susan Zoller CONTRACT START DATE: November 2011 PERIOD OF PERFORMANCE: November 2011 June 2012 DATE OF REPORT: June 30, Executive Summary Background MGT of America, Inc. (MGT) conducted a comprehensive, mixed-methods evaluation study of the four teacher evaluation options being implemented in the Ohio Teacher Evaluation System (OTES) Pilot Program during the school year. All districts in the state were invited to participate in the pilot; 139 were selected. As part of its application for participation, each district identified which of the four approved teacher evaluation models they planned to implement. The four options included: 1. OTES model 2. OTES model with locally-developed student growth measures 3. Local evaluation system aligned to OTES model (e.g. Danielson, Marzano, other) 4. Local evaluation system aligned to OTES model (e.g. Danielson, Marzano, other) with locallydeveloped student growth measures The evaluation examined the implementation and impacts of the four OTES options to inform further refinement of the OTES model. The research questions included the following areas: Implementation of the selected model Impact on teacher effectiveness and behavior Impact on administrative behavior and school/local Educational Agencies practices, policies, and procedures Impact on student achievement Sustainability Best practices in teacher evaluation 1 P age

3 Methodology MGT gathered data from the 139 Local Educational Agencies (LEA) participating in the OTES pilot using online surveys in January 2012 (mid-year through the pilot school year) and April 2012 (summative). The online surveys were distributed by the Ohio Department of Education to all 659 participants teachers, teacher evaluators, and administrators. The initial survey had a 49% response rate; the summative survey had a 44% response rate. Responses were received from at least one person at each LEA. In addition to the surveys, MGT conducted site visits to a stratified, random sample of 12 LEAs selected based on the evaluation model that they had chosen, their geographic location (NE, NW, SE, SW, urban, and central), and the type of LEA (Exempted Village, City, Local, Community School, and ESC). MGT staff visited the selected sites during April and May 2012, and conducted a total of 75 interviews with participating and non-participating teachers and principals and reviewed pre-identified documentation of implementation. MGT staff also interviewed the superintendent in each LEA. MGT reported findings from the initial survey in the February Mid-Term Report and the Case Study data in the April Quarterly Report. Findings and Recommendations IMPLEMENTATION: All LEAs were implementing at least some of the components of the OTES model, including nearly 100% implementation of the pre- and post-observation conferences. However, some aspects of the OTES model had limited implementation. For example, only about 32% reported using student growth information and only 35% reported holding a year-end summative conference. Nearly 20% of the LEAs initially indicated that they planned to develop their own local model, but several had instead chosen to implement the OTES model. 91% of those who answered the survey indicated that their LEA did not make any significant modification while implementing the OTES model. Implementation of this new model had an unanticipated positive impact on the frequency and level of interactions between teachers and administrators. Nearly all participants interviewed mentioned this as a positive effect. RECOMMENDATIONS: Adopt all components, at least provisionally, and leave them in place for a second pilot year. Make the tools, forms, and structure available through a statewide online system of support that helps both teachers and evaluators manage all the parts and pieces. Create a clear and simple flow chart showing activities on a sample timeline a Year-at-aglance. Provide clear documentation to identify what is required and what is recommended as best practice. Samples of each should be included both strong and weak examples to support the goal of transparency and improved teacher and evaluator performance. 2 P age

4 Provide ongoing, in-depth, and accessible professional development. Improve face validity of the system by ensuring that the system is fair, equitable, and reliable for all teachers. Present the summative teacher evaluation ratings in actionable terms that provide guidance for decision-making about classroom practices and professional development needs. Conduct another pilot during IMPACT ON TEACHER BEHAVIOR AND EFFECTIVENESS: There was mixed impact on teacher behavior identified through this study. Only half of those responding indicated that analysis of student data, multiple assessments, and the focus on professional growth had an impact on instructional practice. The most frequently identified impact came from teacher understanding of SMART goals (goals that are Specific, Measurable, Attainable, Relevant, and Time-Bound), collaboration and communication between teachers and their building administrator, and alignment between SMART goals and classroom activities. RECOMMENDATIONS: Maintain the use of SMART goals as an expected/required component of the system. Provide professional development aimed at using data, developing multiple assessments, and differentiating instruction to impact instructional practice. IMPACT ON ADMINISTRATIVE BEHAVIOR AND SCHOOL/LEA PRACTICES: None of the LEAs reported making changes to their policies or procedures based on the OTES pilot. However, many reported that they were waiting until the model was finalized before they would make any adjustments. LEAs also reported not making changes in their union contracts, except for some Memoranda of Understandings (MOUs) put in place for the pilot year. Few of the LEAs reported having a comprehensive communications plan to ensure that nonparticipants were informed about pilot activities and plans. Participating teachers reported that their colleagues ranged from curious to nervous. Some teachers expressed concern about having other teachers become trained evaluators, indicating that they did not think peer evaluation was appropriate. At least one LEA had specific contract language preventing teachers from serving in this capacity. Several principals thought this might be a way to reduce the burden for administrators. RECOMMENDATIONS: Invite Ohio teacher, administrative, and school board organizations to assist in the development of model policies and procedures or contractual language. 3 P age

5 Invite input or feedback from administrators and teachers on ways to differentiate evaluations based on teacher performance to reduce the perceived burden of implementing this model with all teachers. IMPACT ON STUDENT ACHIEVEMENT: Although one goal of this study was to review the impact on student achievement, there were significant barriers to gathering data in support of the research questions. During the pilot year, only teachers working in grades 4-8 in reading or mathematics had access to value-added data and only a few of the participating teachers fell into that group. Less than 35% of participants reported analyzing student growth data as a component of the evaluation. RECOMMENDATION: Conduct an analysis of the impact of the OTES model on student achievement once the model has been implemented for long enough to show trends and draw conclusions. Ensure that student growth scores are based on more than one year and have, to the degree possible, factored out the effects of external conditions impacting student learning. SUSTAINABILITY: Many participants expressed concern about the time commitment to conduct evaluations using the new model. Principals who were using OTES with only 1-2 teachers doubted their capacity to conduct that level of discussion with all members of their staff, given all their other obligations. Participating teachers and principals both arrived for site visit interviews with large file folders or stacks of documents. There is a need for electronic forms and support structures. RECOMMENDATIONS: Develop an electronic system of support for the OTES model that supports all phases of the process and allows the uploading of artifacts or evidence in a variety of formats by both the teacher and principal. Provide ongoing, accessible professional development in how to use the electronic system of support. REVIEW OF BEST PRACTICES IN TEACHER EVALUATION: MGT conducted a literature review of current teacher evaluation activities and practices. Recent rapid changes in teacher evaluation practices have been prompted by federal and state policy initiatives. Before 2010, only four states were using student achievement as a predominant influence in how teacher performance was assessed. By 2011, there were 13 and an additional 10 states that had student performance as a small percent of teacher evaluations. Many states, including Ohio, have developed or adopted value-added measures (VAM), a term applied to a range of approaches that vary in their data requirements and statistical complexity. Many have also tied compensation to the value-added scores. However, researchers disagree on the appropriate use of VAM. 4 Page

6 There is significant debate on how to evaluate teachers, but significant agreement exists in that teacher effectiveness can and should be evaluated. However, teacher evaluation is best grounded in criteria for which teachers have control: setting SMART goals, regularly measuring and monitoring the learning progress of students, adjusting classroom practice to address gaps in learning or to accelerate learning through differentiated instruction, engaging in professional development aligned with needs assessment results, and collaborating with support staff, other teachers, and administrative leaders to address the needs of all students. RECOMMENDATIONS: Avoid using VAM data for high-stakes decisions about an individual teacher s effectiveness, and include VAM data as only a small percent of the evaluation. Conduct rigorous pilot studies of how the results of various value-added statistical models correlate with other measures for student growth and teacher effectiveness. Focus at least as much attention on the alignment of standards and professional development for both teachers and administrators. 5 P age

7 2.0 Overview of Research Project This project used a comprehensive, mixed-methods approach to study the Ohio Teacher Evaluation System (OTES) Pilot Program during the school year. The evaluation examined the implementation and impacts of the four OTES options to inform further refinement of the OTES model and strategies for scaling up valid and reliable teacher evaluation approaches that are relevant and useful to teachers and principals at Local Educational Agencies (LEA), as well as to inform the overall Ohio Human Capital Management System for educating today s youth. The evaluation explored the relevance and usefulness of the OTES model for guiding LEAs and the Ohio Department of Education (ODE) in the implementation of high-quality teacher evaluation approaches aligned with the Ohio Standards for the Teaching Profession and best practices in measuring teacher quality. The evaluation used the Ohio Standards for the Teaching Profession as the basis for defining effective teaching, and examined the impacts of using OTES options on teachers professional growth and development, administrative behavior and school/lea processes. This report includes data from a summative survey of participants conducted in April 2012 and summary findings for the study as a whole. The formative and summative feedback from the evaluation will identify and describe accomplishments, impacts, and challenges of the OTES Pilot Program options that LEAs piloted during the school year. 6 P age

8 3.0 Methodology This research project included data gathered through two online surveys, site visits to selected LEAs and charter schools, and a literature review of best practices in teacher evaluation. Each approach is described in detail below. Online Survey Tools MGT conducted a formative survey of all participants in the OTES Pilot Program in January All of the participants from 139 LEAs were surveyed. The initial survey emphasized answering the research questions for implementation with some attention to impacts on teacher effectiveness and administrative behaviors. The summative survey was conducted in April 2012 to fully address the research questions and identify any changes over time. MGT analyzed the data from both surveys using descriptive statistics in SPSS and Excel. Open-ended questions were analyzed using qualitative coding, which identified common trends and concerns among pilot LEAs. The spring follow-up survey used indicators from the initial survey coding to gather quantitative data to further examine the trends and concerns that emerged in the formative survey data. The data from the initial survey included the role of the respondents and further questions were differentiated based on that role. Teacher questions included their perceptions about the level of knowledge among principals about the new evaluation methods, the amount and source of training about the new methods, and the range of teacher evaluation tasks that had been completed to date. Administrator questions included the changes made or planned in policies, procedures, or union contracts, the amount and source of training, and the range of teacher evaluation tasks to date. The survey was intended to provide baseline data for evaluation of the OTES pilot project as well as background information needed for the selection of case study districts. The survey was designed to gather information from each of the 139 LEAs in the pilot to ensure that each participating principal, teacher, and district and union representative at each site had an opportunity for input and feedback. The initial survey had a 49% return rate (325/659 pilot participants). The data from the summative survey included questions regarding the implementation of OTES, the effect of OTES on teacher, evaluator, and student behaviors, and the impact on LEA administration. The survey was intended to provide summative information regarding all aspects of the OTES pilot. The summative survey had a 44% return rate (288/659 participants). Of the 288 LEA participants who answered the survey, 268 (93%) said they were directly involved with implementing the OTES in their building or district. The remaining 20 (7%) participants were not directly involved in implementing the OTES. For those not involved with OTES implementation, the survey skipped to the last survey question about alignment between the OTES model and organizational goals. The summative survey results included in this report are only from those directly involved in implementing the OTES model during the pilot year. Survey participants identified their roles as follows: 7 P age

9 30% identified their role as building principals, not involved in evaluations 18% were evaluators, including principals 17% were district administrators 30% were classroom teachers 2% were union leaders 3% were various program coordinators The design of both sets of survey questions was informed by analysis of the new Ohio teacher evaluation criteria and methods, the ODE s fall 2011 training survey results, and a literature review of best practices in teacher evaluation. The questions from both the initial and summative surveys are located in Appendix A. Case Study Site Visits The case studies examined the relevance and usefulness of the OTES model for guiding LEA s implementation of high-quality teacher evaluation methods aligned to the Ohio Standards for the Teaching Profession and best practices in measuring teacher quality. Twelve case study sites were selected from a stratified random sample of OTES pilot sites. The types of districts and geographic regions in the statewide distribution of case study sites are shown Exhibit 3-1. Exhibit 3-1 Type and Geographic Region of Case Study Sites CASE STUDY SITE 1 CASE STUDY SITE 2* OTES - Student Growth Measures CASE STUDY SITE 3 CASE STUDY SITE 4 CASE STUDY SITE 5 CASE STUDY SITE 6 Exempt Village City Local Local Local Exempt Village Northeast Urban Northwest Central Northeast Northwest OTES - No Student Growth Measures CASE STUDY SITE 7* CASE STUDY SITE 8 CASE STUDY SITE 9 CASE STUDY SITE 10 CASE STUDY SITE 11 CASE STUDY SITE 12 City City Education Service Center City Local Community School Southwest Urban Northeast Northwest Southeast Southeast Source: MGT of America, Inc., *These two sites included locally-developed evaluation system components in their OTES pilot. 8 P age

10 Two evaluators from MGT conducted site visits at the 12 LEAs between March 26 and April 5, The evaluators conducted a total of 75 interviews, reviewed and gathered documents that had been previously requested, and reviewed teacher evaluations during the site visits. No teacher evaluation documents were copied or taken by MGT. Data were organized for analysis based on protocols developed by MGT to address the research questions. Responses from interviewees were coded to identify issues and patterns across locations. Best Practice Review MGT conducted a review of the literature available to describe best practices in teacher evaluation. The review included three components: trends in teacher evaluation methodology, perspectives on valueadded measures (VAM) from measurement and research experts, and information regarding merit pay and bonus incentives. The complete best practice review, including bibliography, is located in Appendix B. Research Questions This research was based on a set of questions identified by ODE. The questions dealt with the implementation of OTES (how well or to what degree OTES was implemented during the pilot year), the impact of OTES (what effect did OTES have on the teachers, administrators, and other evaluators who implemented the model), the impact on student achievement (what effect did OTES have on student learning), and the impact on LEA policies or practices. Additionally, research included questions dealing with the sustainability of the OTES model and a review of the literature regarding best practices in teacher evaluation. The research questions are defined below: 1. Implementation: Review the ongoing implementation of the pilot in the selected schools to identify successes and areas in need of improvement. This includes sub-questions such as: To what extent were teachers, administrators, and union leaders involved in the design and implementation? What is the fidelity in relation to the project plan? To what extent were comprehensive communication plans developed and successfully utilized? What were the best practices of the most effective implementers? 2. Impact on Teacher Effectiveness and Behavior: Report the pilot program s impact on effectiveness and behavior as measured by student achievement and value-added measures. This includes changes in individual instructional practices and levels of embedded change within LEAs. This includes sub-questions such as: What student achievement and growth measures were used? 9 P age

11 What were the intended and unintended consequences on instructional practices? 3. Impact on Student Achievement: Examine impact on student achievement. 4. Impact on Administrative Behavior and School/LEA Processes: LEA level. Sub-questions may include: Have LEA policies and procedures changed? Examine impact at the school and To what extent has the pilot evaluation model impacted professional development? What is the nature and degree of alignment of organizations process and performance outcomes across school and LEA? 5. Sustainability: Examine the sustainability of the evaluation system. 6. Best Practices: Monitor and review research and practices in other states and districts to make a summary and recommendations for future refinement of the project. Each of these research questions has been explored during the course of this study. Several questions were explored using multiple methodologies to understand the issue from various perspectives, to validate responses, or to review implementation. The matrix below (Exhibit 3-2) shows the source of data used to answer each research question. Exhibit 3-2 Research Question Data Sources RESEARCH QUESTION 1. Implementation Involvement of teachers and unions Fidelity Communications plan Best practices of implementers 2. Impact on Teacher Effectiveness and Behavior 3. Impact on Student Achievement 4. Impact on Administrative Behavior and School/LEA Processes 5. Sustainability INITIAL SURVEY DATA SOURCES CASE STUDY LITERATURE REVIEW FINAL SURVEY X X X X X X X X X X X X X 6. Best Practices X X Source: MGT of America, Inc., P age

12 4.0 Findings and Recommendations This section of the report provides a summary of the findings gathered from the initial survey and case study site visits that were previously reported in the February 2012 and April 2012 reports. New to this report are the data from the summative survey. Initial Survey Findings Summary The initial survey was conducted in January 2012 via an electronic survey tool. ODE sent the survey tool to all OTES participants. In total, 325 of approximately 659 pilot participants completed the survey (49%). Of the 139 districts participating in the OTES pilot, at least one person from 123 of those districts completed the survey (89%). Approximately 59% of survey respondents work in rural school districts, 25% in suburban districts, and 17% in urban settings. Implementation Survey respondents reported a high level of engagement and participation in the design and implementation of the new teacher evaluation methods. Ninety-nine percent of building principals are actively involved and 95% of them serve as the primary teacher evaluator. They reported focusing mostly on implementing the teacher performance component, which is only 50% of the new evaluation methods. Most had not yet addressed the other 50% of the evaluation criteria pertaining to value-added student data. They expressed concern about the standardized use of student data for all teachers, content areas, and student levels. Most of the participating LEAs reported conducting formal classroom observation in accordance with the OTES implementation timeline; however, only about two-thirds of them had conducted the postobservation conference meeting or provided a written observation report. They also reported that the new teacher evaluation methods were overwhelmingly time consuming and therefore not feasible to fully implement without the creation of full-time evaluation staff at the building level, especially for those working in rural or small districts. These results suggest the need to make modifications to OTES so that it is more adaptable to each school s culture. Impact on Teacher Behavior and Effectiveness Approximately 65% of those who answered the survey reported successes as a result of implementing the new teacher evaluation method. The biggest successes included gaining an appreciation for the how the new teacher evaluation methods foster more reflective practice among teachers and enhanced communication between teachers and evaluators, who in most cases are building principals. These successes help focus instructional practices on SMART Goals and more accountability for meeting standards and new teacher evaluation criteria. Pilot sites also reported some success with improving evaluation measurement tools and uses of data in the instructional decision-making process. These types of successes can have a positive impact on instructional practices and student learning. 11 P age

13 Impact on Administrative Behavior and School/LEA Processes Survey respondents saw value in the theory behind OTES, but were concerned about its current design which was perceived to be too complex and time consuming, two factors that make it difficult to roll-out in its current format. They generally thought the teacher performance component, which constitutes 50% of the criteria, was valuable; however, some components of the rubric are difficult to interpret. Many questioned the validity of using 50% student growth and have unanswered questions about how to measure student growth in a fair and reliable manner for all teachers in a building or district. Some principals reported disagreeing with the teacher evaluation results when they applied the criteria. Sustainability Those responding to the survey expressed widespread concern for the complexity of the evaluation process and the length of time needed to complete forms, which indicated a need to simplify the method and make it more user-friendly. Suggestions for improving sustainability include: Adjusting the process to better match the daily work flow of principals and teachers. Simplifying steps to reduce the time between steps to provide more timely feedback. Simplifying the forms to reduce paperwork. Considering different levels of evaluation for different teachers, i.e. suggestions of using the piloted method for only teachers whose performance is marginal or problematic. Aligning the criteria more closely with teachers areas of influence, e.g. content areas aligned with valid student assessments for all teachers. Delineating student-control factors from teacher-control factors. Case Study Findings Summary The case studies examined the relevance and usefulness of the OTES model for guiding LEAs implementation of high-quality teacher evaluation methods aligned to the Ohio Standards for the Teaching Profession and best practices in measuring teacher quality. Between March 26 and April 5, 2012, two evaluators from MGT conducted site visits at 12 LEAs that were selected from a stratified random sample of OTES pilot sites. The types of districts and geographic regions had been determined through a review of all districts participating in the pilot and based on the percent of such districts in the state. Implementation LEAs were giving most attention to implementing the teacher evaluation components, identifying how well teachers were meeting standards 1-5 through formal observations, post-observation progress conferences, written observation reports and teachers use of self-assessment tools, and setting SMART goals. The least attention had been given to evaluating teachers collaboration, communication and professionalism for meeting standards 6-7, and documenting informal walkthroughs. 12 P age

14 During case study visits, MGT staff reviewed documentation as evidence of implementation of OTES. Written teacher evaluations were the most prevalent type of documentation with 75% of LEAs able to show documents as evidence of their OTES pilot efforts. At 67% of the case study sites, LEAs had implemented procedures related to the new teacher evaluation process. At 75% of the LEA sites where documentation was not available, administrators explained that a comprehensive communication plan would be created only after the OTES guidelines were fully developed at the state level and adopted by their local school boards. During site-based interviews, teachers and principals reported spending more time this pilot year communicating during one-on-one meeting conferences about teachers instructional practices and strengths and areas for improvement than they had in previous years. Teachers find high value in doing the self-assessment and including it as part of the discussion with their evaluator. Teachers and principals reported a clearer, shared understanding of the teacher evaluation criteria based on in-depth examination of the OTES rubric and individual teachers rubric scores following formal observations. In this regard, OTES is fostering more meaningful communication about teacher effectiveness between teachers and principals. Impact on Teacher Behavior and Effectiveness During interviews, teachers participating in the pilot and non-participants alike verbalized a general anxiety stemming from confusion about the definition of and appropriate interpretation of value-added measures (VAM) and student growth scores. Teachers and some administrators expressed concern about how VAM scores relate to teachers performance scores and how much weight VAM scores will carry in decisions about teachers employment and pay scales under the OTES model. Evaluators found a general lack of understanding among those at LEAs about the state-of-the art of VAM metrics, which are undergoing intense research and development. They expressed concern that the OTES model does not provide clear guidelines for the VAM formula or sources of value-added scores. There is a lack of understanding that student growth scores can include metrics for school and learner characteristics and measures for groups of teachers who collaborate in teaching student cohorts. There also is a lack of understanding that multiple years of data are compiled into VAM scores. The confusion about VAM and student growth at LEAs is not surprising, given the current widespread technical debates among policymakers and measurement experts. There was also concern and relief that the state plans to allow individual LEAs to define the VAM used in their setting. Those who expressed concern mentioned that it might not be fair and equitable across the state, and teachers in one area might be being held to a higher or different standard than those in a neighboring district. Those who expressed relief indicated that they would rather be held to locallydefined standards than statewide standards that might not be appropriate for their populations. Teachers and administrators in both groups were unsure how the student growth measures were going to be developed in their LEA and were very concerned about having salaries tied to the new evaluations. 13 P age

15 Impact on Student Achievement The case studies investigated the extent to which LEAs had been able to identify impacts of the new teacher evaluation on student achievement, and found that it is too early in the OTES pilot to draw conclusions about impacts on student achievement or value-added impacts of teachers on student growth. Impact on Administrative Behavior and School/LEA Processes The evaluators reviewed documentation of any new incentives or policies and procedures at case study sites where available. Most districts have not yet started to create policies and procedures in support of the new model of teacher evaluation. Teachers and principals reported finding high value in the SMART goal process which fosters more reflective teaching practices among pilot participants and informs professional development choices more aligned to professional growth needs. Both teachers and administrators reported expecting to revisit the SMART goals in the final evaluation conference, but found that the document did not include any reference to the SMART goals. Most people believed that goal setting should remain as a required element in the OTES model. Those interviewed also reported that OTES creates a need for teachers in-depth training on SMART goals, assessment literacy and data-driven practices, differentiated instructional methods, and methods for fostering self-regulated learners. There were concerns about the perceived move away from unannounced observations, especially in districts whose union contracts preclude the use of walk-through visits for evaluation. Sustainability The amount of time principals need to spend on conducting a full OTES evaluation for a single teacher was not viewed as feasible to scale up for all staff. Administrators viewed the current OTES model as a valuable and comprehensive way to evaluate new teachers and those with marginal performance. However, there was a lack of understanding about either making modifications to OTES or using some of the other OTES tools that might be more appropriate for a veteran or distinguished teacher. One of the major concerns expressed during the site visits was the need for electronic forms and support structures. The evaluators observed both principals and teachers paging through large folders of documents to identify and describe a single observation. Most participants identified this as an obstacle to sustainability. Summative Survey Findings The summative survey was conducted in April 2012 and was sent to all OTES participants. The survey had a 44% return rate (288/659). Of the 288 LEA participants who answered the survey, 268 (93%) said they were directly involved with implementing the OTES in their building or district. The remaining P age

16 participants (7%) were not directly involved in implementing the OTES. For those not involved with OTES implementation, the survey skipped to the last survey question about alignment between the OTES model and organizational goals. The survey results outlined in this report are from only those who reported being directly involved in implementing the OTES during the pilot year. Participants identified themselves as follows: 30% were building principals not involved in evaluations 18% were evaluators, including principals 17% were district administrators 30% were classroom teachers 2% were union leaders 3% were various program coordinators Implementation Participants were asked, To what extent did your district modify the OTES model to fit your school culture during this year s pilot? 91% of those who answered the survey indicated that their LEA did not make any significant modification while implementing the OTES model. The other 9% who indicated their LEA did make significant modifications to the OTES model were asked to briefly describe the modifications and reason(s) for modifications. These participants indicated that modifications to the OTES model focused primarily on aligning OTES language and forms with their LEA s existing teacher evaluation process for goal-setting, walkthroughs, examples of evidence, use of the Danielson model, number of observations, and the use of peer evaluators. The percentage of respondents who described modifying OTES is shown in Exhibit P age

17 Exhibit 4-1 Types of OTES Modifications by LEAs Source: MGT of America, Inc., Exhibit 4-2 shows pilot participants answers to the question, Which of the following OTES components did you complete or participate in during the pilot year? There was a high degree of fidelity implementing the formal classroom observations and associated pre- and postobservation conferences. In addition, 84% of pilot participants said they used SMART goals and the teacher self-assessment tool. Only 70% used the OTES written evaluation report component. Exhibit 4-2 also indicates that the majority of LEAs did not use the OTES professional growth plan, lesson reflection, or professional development resources. Approximately 32% of pilot participants said they analyzed evidence of student growth data as part of the OTES pilot. 16 P age

18 Exhibit 4-2 Fidelity of Implementing the OTES Model (N=247) Source: MGT of America, Inc., These data make clear that participants did not implement all components of OTES, but did not feel like that omission constituted a significant modification of OTES. Information gained during the onsite visits reported earlier indicated that many participants were unaware or unsure of some of the components. It is unclear whether the lack of implementation is due to lack of awareness or competence. Exhibit 4-3 includes data about the ease of use of the OTES process and documents. Participants were asked, Rate how easy it was to use each of the following components of OTES. The most difficult component for 75% of pilot participants was information about student growth measures, and 50% said the year-end summative form was difficult. The evaluation rubric for the standards and the SMART goalsetting form were also difficult for about 40% of pilot participants. Although participants found the SMART goal form difficult to use, the large majority of pilot participants indicated that they did conduct a SMART goal-setting conference as part of their OTES implementation activities. Exhibit 4-3 OTES - Ease of Use OTES COMPONENTS N DIFFICULT OR SOMEWHAT DIFFICULT VERY OR SOMEWHAT EASY Student Growth Measure Information from ODE % 25% Year-end Summative Form % 50% Rubric for teacher evaluation Standards 6 and % 58% SMART goal setting form % 59% Rubric for teacher evaluation Standards % 60% Self-assessment form % 83% Source: MGT of America, Inc., P age

19 Impact on Teacher Behavior and Effectiveness One of the goals of the new teacher evaluation system is to identify and define effective teaching practices. Ideally, the evaluation system should positively impact instructional practices. To understand the impact, the survey asked participants, What, if any, impacts did the new teacher evaluation system have on instructional practices among pilot teachers during the school year? At least two-thirds of those who answered the survey agreed about the top three areas of impact that the new teacher evaluation system had on instructional practices: Understanding SMART goals Improving collaboration and communication between teachers and their building administrator Alignment between SMART goals and classroom activities Exhibit 4-4 shows that half of those who answered the survey said analysis of student data for progress monitoring, use of multiple assessments, and the focus on professional growth had an impact on instructional practice. Exhibit 4-4 also shows that the majority of OTES pilot participants did not see OTES impacts on other important classroom practices, e.g. re-teaching areas of weakness in student learning or differentiated instruction. Exhibit 4-4 Impact of OTES on Instructional Practice (N = 243) Source: MGT of America, Inc., Participants were asked, To what extent does the new teacher evaluation system impact teachers professional development needs? Pilot participants indicated that OTES has wide-spread impact. Three-fourths of participants indicated six common professional development needs, as shown in Exhibit 18 Page

20 4-5. Five of the six top needs pertain to use of student learning data to inform instructional practice. Designing lessons aligned to the new Common Core learning standards is also an area in which OTES has a strong impact on professional development needs. Exhibit 4-5 OTES Impacts on Professional Development Needs (N= 239) Selecting assessments to monitor student learning for specific lessons and benchmarks No or Slight Impact Moderate Impact Extensive Impact 21% 48% 31% Analyzing student learning data from multiple measures 22% 40% 38% Designing and implementing SMART goals 23% 54% 23% Understanding value-added measures (VAM) 24% 42% 34% Designing lessons aligned to Common Core standards 24% 45% 31% Using student learning data to select differentiated instructional strategies 25% 49% 26% Designing classroom assessments 29% 43% 28% Implementing differentiated instruction 29% 45% 26% Using VAM scores to inform instructional and administrative practices for school improvement Involving students in self-assessment, progress monitoring and goal setting 30% 40% 30% 30% 44% 26% Best practices for specific content areas 31% 51% 18% Communicating assessment results with students, parents and colleagues 35% 46% 19% Fostering self-directed strategy use among students 37% 45% 18% Integrating technology into lesson plans 41% 43% 16% Team teaching and teacher collaboration strategies 42% 42% 16% Source: MGT of America, Inc., Impact on Student Achievement The Summative Survey did not collect data about impact of the OTES pilot on student achievement. Impact on Administrative Behavior and School/LEA Processes Participants were asked, To what extent does the OTES model support alignment between your organizational processes and performance goals? 56% of those who took the survey responded to this question (161 respondents). About 84% of those who responded indicated that although they have not all reached full implementation, they agreed that OTES supports moving in the direction of their organizational and performance goals. Another 16% of those who answered this question indicated that 19 Page

21 the OTES model does not support alignment between organizational processes and performance goals at the LEA level because it is too time-consuming or complex to feasibly implement with all teachers during a school year and because it does not align with current contracts in place. Sustainability As shown in Exhibit 4-6, 57% of pilot participants indicated that their LEAs did not have a plan for new contract language related to the use of teacher evaluation results when making compensation, placement, or retention decisions. The large majority of pilot sites also do not have a comprehensive communication plan that explains OTES nor have they made changes to policies or procedures that reinforce the new teacher evaluation process and use of student achievement data as part of teacher evaluation. These survey results indicate that there is much to be accomplished before OTES is institutionalized at a sustainable level. Exhibit 4-6 Status of Plans to Align LEA Policies or Procedures with OTES (N = 240) LEA has new contract language related to the use of teacher evaluation results in compensation, placement and retention decisions. LEA has a new comprehensive communication plan that explains the new teacher evaluation system. LEA has new policies and/or procedures that support using student achievement data as a part of the teacher evaluation process. LEA has new policies and/or procedures that reinforce the new teacher evaluation process. Source: MGT of America, Inc., NO PLAN DRAFT PLAN PARTIAL PLAN FULL PLAN 57% 26% 13% 4% 40% 33% 22% 5% 33% 36% 26% 5% 25% 37% 28% 10% Exhibit 4-7 shows how those who took the survey rank a number of supports for sustainability of OTES in response to the question: What supports need to be in place to ensure sustainability of the new teacher evaluation system within your district? There was a high degree of consensus about the top five priorities for sustainability. Nearly two-thirds said the highest priority is step-by-step guidelines for implementing OTES. Other top priorities include: Ongoing OTES training for administrative and peer evaluators Professional development and support for use of VAM scores to inform instructional and administrative practices for school improvement Online OTES data management system 20 P age

22 SUPPORTS FOR SUSTAINABILITY Exhibit 4-7 Supports for OTES Sustainability (N = 238) NOT A PRIORITY LOW PRIORITY MEDIUM PRIORITY HIGH PRIORITY Step-by-Step guidelines for implementing OTES 1% 9% 26% 63% Ongoing OTES training for administrative and peer evaluators 0% 8% 33% 59% Support for how to use VAM scores to inform instructional and administrative practices for school 1% 8% 41% 50% improvement Professional development and system supports for understanding value-added measures 1% 9% 41% 49% Online OTES data management system with evaluator and teacher access 2% 12% 38% 49% Interactive evaluation rubric for use with hand-held devices for capturing data in the classroom 7% 20% 28% 45% Tool for integrating Collaboration, Communication and Professionalism (Standards 6 and 7) with SMART goals Evaluation option to conduct progress monitoring of teacher s SMART goals Tool or guideline for integrating CEUs for licensure with teacher evaluation results Source: MGT of America, Inc., % 19% 47% 29% 2% 21% 50% 26% 6% 36% 45% 12% The survey included questions regarding the Teacher Rating Matrix that had been developed to describe how the teacher performance and student growth components would create the final teacher summative evaluation score. The survey asked, Ohio's new system for evaluating teachers combines a rating of teacher performance (based on classroom observations and other factors) with a rating of student academic growth. These two ratings are each weighted at 50 percent. How clearly does the matrix (shown in the survey) explain the method for computing a summative evaluation rating? Although 84% of those who answered the survey said that the Teacher Rating Matrix was Somewhat or Very Clear, many OTES pilot participants raised concerns about how summative ratings are assigned to teachers based on the matrix. When asked for suggestions to help clarify or change the Teacher Rating Matrix that combines student growth and the teacher performance ratings, 37% of those who participated in the survey (107 respondents) responded with comments related to four issues, as shown in Exhibit P age

23 Exhibit 4-8 Teacher Rating Matrix Feedback 10% 16% Validity Concerns 21% 53% Clarification Suggestions Source: MGT of America, Inc., Their most pressing concern pertains to the validity of the summative ratings. About 53% of those who offered suggestions or comments about the matrix questioned the validity of the proposed formula for determining a teacher s summative evaluation score. They raised concerns about its fairness, equity and inter-rater reliability for all teachers across grade levels and content areas, and variance in student sub-groups. They questioned the practice of only assigning an accomplished rating to teachers whose students student growth measure (SGM) outcomes exceed one year s growth, since one year s growth has always been considered the gold standard. They also said the summative rating needs to reflect if a teacher is ineffective for ethical or communication problems even if student growth is good. Finally, they would like to see the research base behind the matrix design. In addition, 10% said they have concerns about using scores from SGM that have not yet been identified. They consider it premature to implement a high-stakes teacher evaluation system using SGM before the measures have been thoroughly piloted and researched for technical soundness with measures at all grade levels aligned to the new Common Core learning standards. Another 21% offered suggestions for adding more explanation of the range in student growth scores that correlate to each possible summative teacher rating. They asked for more explicit examples of how the student growth scores impact teacher evaluation ratings so that teachers better understand the expectations for student growth associated with the different levels in the matrix. About 16% offered suggestions related to the matrix format, including making a decision-tree type flow chart for assigning a summative rating, aligning the display of matrix categories with the OTES rubric, and replacing the numbers with descriptive rating categories. The numbering system was said to be confusing for two reasons: 1) The numbers are perceived to be a carryover from the previous, less holistic evaluation system; and 2) The numbers are confusing to some because they are not certain what the numbers represent. 22 P age

24 5.0 Summary MGT conducted a comprehensive review of the pilot OTES in order to answer research questions dealing with implementation and impact. Survey data were gathered from participants both in the middle of the pilot year (January 2012 initial electronic survey) and near the end of the pilot year (April 2012 summative electronic survey). Data were also gathered through documents collected and face-to-face interviews at sites selected as case study locations. Conclusions Several conclusions can be drawn from the data collected in this study. The sequence of the following list of conclusions is not meant to imply greater or lesser value or importance. The value of SMART goals. LEA pilot participants perceive SMART goals to have a strong positive impact on instructional practices, although nearly half reported that the SMART goals form was difficult to use. Since SMART goals are based on a review of evidence related to student learning needs, the process of establishing SMART goals can inform plans for differentiated instruction and selection of multiple measures to monitor student learning. There is a need for more professional development support for setting SMART goals and improving teachers assessment literacy, both of which have the potential to positively impact student learning growth. The student growth measures (SGM). There is a significant need for a more thorough explanation about how to identify student growth measures. This is a problem not only for teachers working in non-tested areas, but also for those teaching in tested areas since the new Common Core standards and assessments are yet to be developed and validated. The impact of SGM on individual teachers. There is a need for more explanation about how the results of student growth measures are computed into value-added ratings for individual teachers. The Teacher Rating Matrix. There is a need to address concerns about the validity of the Teacher Rating Matrix, including an explanation of the research-based rationale for why one year of academic growth for students is not an acceptable goal for accomplished teachers. The research behind the model. There is a need for a more thorough explanation of the technical measurement design behind value-added scores so teachers better understand the meaning of SGM and how it relates to their classroom practice. A comprehensive understanding of the model. LEAs in the pilot were not using the full complexity of the OTES model and will likely need more supports to do so. Many participants did not understand if there could be some differentiation between new or struggling teachers and veteran/distinguished teachers. Their questions did not imply that being new equated with struggling or that being a veteran equated with being distinguished. However, they expressed a need to address teachers differently and were unsure if they could under this new model. 23 P age