QUALITY EVALUATION ROLLOUT DATA ANALYTICS CONVENING OCTOBER 11, 2013

Transcription

1 QUALITY EVALUATION ROLLOUT DATA ANALYTICS CONVENING OCTOBER 11,

2 WELCOME AND INTRODUCTIONS 2

3 INTRODUCTIONS Please share with us: Your name Your State or district Your position What you hope to contribute to or take away from today s convening 3

4 TODAY S OBJECTIVES Create visual models to best display key evaluation data in a dashboard and/or scorecard format for either States or districts. Develop a State-specific action plan to successfully implement a data dashboard and/or scorecard in your State. Outline post-convening follow-up activities led by Reform Support Network (RSN) staff to collect examples of State and/or local work that is influenced by the products developed by the subcommittee. 4

5 AGENDA Activity Introductions and Review of Agenda Review State Context and Purpose/Examples of Dashboards and Scorecards Introduce Critical Questions and Menu of Metrics for States State Team Pilot Development State Debrief Break State Development of Dashboards and Scorecards Status Report Lunch State Development of Dashboards and Scorecards State Collaborative Team Time Break Action Planning Review of Next Steps; Close Time 8:00 a.m. 8:20 a.m. 8:20 a.m. 8:50 a.m. 8:50 a.m. 9:10 a.m. 9:10 a.m. 9:35 a.m. 9:35 a.m. 10:00 a.m. 10:00 a.m. 10:15 a.m. 10:15 a.m. 11:20 a.m. 11:20 a.m. 12:00 p.m. 12:00 p.m. 12:45 p.m. 12:45 p.m. 1:30 p.m. 1:30 p.m. 2:30 p.m. 2:30 p.m. 2:45 p.m. 2:45 p.m. 3:45 p.m. 3:45 p.m. 4:00 p.m. 5

6 REVIEW STATE CONTEXT AND PURPOSE/EXAMPLES OF DASHBOARDS AND SCORECARDS Corey Chatis and Larry Stanton 6

7 SESSION OBJECTIVES Experts will review findings from interviews with States on data centralization and evaluation system components. Experts will use examples to provide an overview of dashboards and scorecards. Participants will review and discuss promising practices for the development of dashboards and scorecards. 7

8 EVALUATION SYSTEM CRITICAL COMPONENTS States named the following components of their evaluation systems as the most critical: Establishing Student Learning Objectives (SLOs) or student achievement goals that are high-quality, fair and consistent with the evaluation model Training evaluators on student growth measures, SLOs and observations Improving evaluator and educator understanding of the evaluation model Providing professional development for teachers Maintaining fidelity of implementation 8

9 STATE EDUCATION AGENCY (SEA) LEVERAGE POINTS Public reporting: highlight disparities between student achievement and teacher performance among schools/districts. Ensure high quality assessments. SLOs: quality assurance; set cut scores for performance levels. Teacher supports: use data to provide tailored resources. Ensure inter-rater reliability. Review and approve district evaluation systems to ensure quality and fidelity to State requirements. Establish corrective actions for districts with high degree of misalignment among components or non-differentiation. Train evaluators. Communicate with superintendents regarding student learning. Implement tools for principals, including communications. 9

10 SEA COMPARATIVE ANALYSIS Degree of Centralization Decentralized Centralized Ohio Maryland New York Rhode Island North Carolina Delaware SEA Access to Evaluation Data Very Limited Data Access Extensive Data Access Ohio Maryland Rhode Island New York North Carolina Delaware 10

11 COMMON QUESTIONS FOR MORE CENTRALIZED STATES 1. What is the relationship among the components of the evaluation system? Are there any non-differentiators? 2. How are districts, schools and individual principals implementing the components of the evaluation system? Are they differentiating among teachers? 3. Which districts and schools have especially weak relationships between student growth and evaluation ratings by principals? 4. What is the range and overall quality of the SLOs being used? 5. Are districts and schools using data from the evaluation system to make informed human capital decisions? 11

12 COMMON QUESTIONS FOR MORE DECENTRALIZED STATES 1. What is the degree of consistency and validity of rating results across local educational agencies (LEAs) and schools within LEAs? 2. Does the evaluation system differentiate between teachers of different effectiveness? 3. What is the quality of SLOs, and what is their relationship with the statewide growth data? Note: There was less commonality among the questions posed by the more decentralized States. 12

13 ANNUAL SCORECARDS A scorecard is an annual progress report on the key elements of the educator evaluation strategy; metrics should be aligned with dashboard measures but not identical. Scorecards combine reports on implementation with some analysis, and they may include 15 to 18 metrics organized into three or four categories (for example, implementation, alignment, distribution, impact, perceptions). Over time, adding trend and benchmark data as they become available will help give meaning to the scorecard. The primary audiences for SEA scorecards are SEA management and board, policy-makers and the public. 13

14 DASHBOARDS Dashboards provide actionable information to support implementation of key elements of the evaluation strategy and answer the question, Are we doing what we said we would do? Dashboard data should change regularly at least quarterly to reflect the progress of implementation. Every dashboard metric should be a prompt for action there needs to be something that someone can do as a result of the data. Dashboards should be limited to no more than 10 to 12 key metrics if everything is important, nothing is important. Dashboard metrics should change over time to reflect changes in implementation priorities. Determining the audience for the dashboard is the first step in its development. 14

15 DASHBOARD OR SCORECARD? 15

16 POTENTIAL USES OF SEA DASHBOARD Problem solving at internal SEA evaluation project team meetings Discussion of evaluation implementation at SEA leadership team meetings Regular updates to SEA board Reporting on evaluation implementation to stakeholders (for example, unions, business groups, press, education interest group associations) 16

17 POTENTIAL USES OF LEA DASHBOARD Problem solving at internal LEA project team Discussing evaluation implementation at district leadership team meetings Regular updates to LEA board Reporting on evaluation implementation to stakeholders (for example, unions, business groups, press, etcetera) 17

18 SEA DASHBOARD EXAMPLE 18

19 POTENTIAL USES OF SEA SCORECARDS Annual report to the governor, State board, stakeholders and the public Annual feedback to districts on implementation Highlight LEAs that are doing a good (poor) job implementing the new system Identify needs for additional attention and resources Comparisons with other States (on common metrics) 19

20 POTENTIAL USES OF LEA SCORECARD Annual report to the board, stakeholders and the public Annual feedback to schools on implementation Highlight schools that are doing a good (poor) job implementing the new system Identify needs for additional attention and resources Comparisons with other districts (on common metrics) 20

21 SEA SCORECARD EXAMPLE 21

22 GROUP DISCUSSION Who are the most important audiences for evaluation data in your State? Which of the tools would be most useful in your State? SEA Dashboard LEA Dashboard SEA Scorecard LEA Scorecard 22

23 INTRODUCE MENU OF METRICS FOR STATES Corey Chatis and Larry Stanton 23

24 SESSION OBJECTIVES Experts will share a sampling of critical questions and associated metrics that can be used in dashboards and scorecards. Experts will guide States through the process of moving from critical questions to metrics to tool development. 24

25 METRICS AND AREAS FOR DISCUSSION Elements to consider: % versus # Interim versus final data Unit of analysis teacher, evaluator, school, district, SEA, etcetera Integration with more robust analytics for example, drill down capability Descriptive information definitions, caveats, etcetera Best practices: Clarify audience and use the decision or behavior the data are intended to inform Use cascading metrics from SEA to district to school Align dashboard and scorecard leading versus lagging indicators Provide feedback mechanisms from users to suggest enhancements and revisions 25

26 KEY QUESTIONS TO TRACK Fidelity of implementation: Are schools and districts implementing the evaluation process with fidelity? Alignment of ratings: Are ratings consistent with student growth and achievement? Distribution of ratings: How are ratings distributed across schools and districts? Impact: What is the impact of evaluation ratings on human capital decision making? Educator perceptions: Do teachers and evaluators believe that the evaluation system is making a difference? What follows is a menu of metrics that SEAs and LEAs can choose from for LEA or SEA scorecards and dashboards. 26

27 FIDELITY OF IMPLEMENTATION Question Dashboard metrics Scorecard metrics Data requirement Do district evaluation systems meet SEA requirements? % of districts with approved evaluation systems [SEA] % of districts with approved evaluation systems [SEA] For States that require approval of district evaluation plans/systems, data on approval status of each district What evaluation systems are districts using? NA # of districts using each type of evaluation system [SEA] For States that allow districts to select their evaluation system model, data on which model is being used For components of the system allowing district choice, which options are districts using? NA # of districts using each type of observation rubric [SEA] # of districts using each assessment type for SLOs [SEA] For States that allow districts choices within the evaluation components, data on which districts are using which tools 27

28 FIDELITY OF IMPLEMENTATION Question Dashboard metrics Scorecard metrics Data requirements Have evaluators completed the required evaluator training and/or certification? % of evaluators completing training [SEA/LEA] #/% of schools with all evaluators trained/certified [SEA/LEA] % of evaluators completing training [SEA/LEA] #/% of districts with 90% of evaluators trained/certified [SEA] % of all evaluations conducted by trained/certified evaluator [SEA/LEA] % of districts with all evaluations conducted by trained/certified evaluator [SEA] Timely access to a data base that identifies all potential evaluators and their training/certification status Are there evaluators who do not differentiate teacher ratings? NA % of evaluators for whom > 90% of their scoring of teachers fall within the same scoring category [SEA/LEA] Final ratings by evaluator Note: Several States have expressed interest in examining inter-rater reliability. While important, this type of analysis is more appropriate for research rather than a dashboard or scorecard format. 28

29 FIDELITY OF IMPLEMENTATION Question Dashboard metrics Scorecard metrics Data requirement How many teachers have approved SLOs? % of teachers with approved SLOs [LEA] % of schools with 90% of teachers with approved SLOs [SEA/LEA] % of districts with 90% of teachers with approved SLOs [SEA] % of teachers with approved SLOs [LEA] % of schools with 90% of teachers with approved SLOs [SEA/LEA] Information on SLO approval in timely manner for dashboard report How many teachers have achieved their SLO targets? NA % of teachers by districts who have achieved all their SLOs [SEA] % of teachers by schools who have achieved all their SLOs [LEA] Teacher-level data on which SLOs have been met and not met Note: Several States have expressed interest in examining the quality and consistency of SLOs across districts. While important, this type of analysis is more appropriate for research rather than a dashboard or scorecard format. 29

30 FIDELITY OF IMPLEMENTATION Question Dashboard metrics Scorecard metrics Data requirement Are evaluators observing teachers consistent with the schedule? % of eligible teachers on track with required observation schedule [LEA] % of pre-tenure teachers on track with required observation schedule [LEA] % of schools on track with observations for all eligible teachers [SEA/LEA] % of teachers who were observed consistent with the schedule [LEA] % of districts with 90% of teachers observed consistent with schedule [SEA] Timely data on individual teacher observations for dashboard reports; summary data on individual teacher observations for scorecard Clear expectations for scheduling observations of each type of teacher Are teachers being observed by more than one evaluator? NA % of schools using multiple observers in individual teacher evaluations [SEA/LEA] Individual teacher evaluation records, including observer for each observation 30

31 FIDELITY OF IMPLEMENTATION Question SEA Dashboard metrics SEA Scorecard metrics Data requirement Have districts evaluated all teachers for whom an evaluation is required? % of eligible teachers with completed evaluations [LEA] % of districts with 90% of eligible teachers evaluated [SEA] % of teachers with completed evaluations [LEA] % of districts with 90% of teachers evaluated [SEA] % of schools with 90% of teachers evaluated [SEA/LEA] Individual personnel records that include scheduled evaluations and final evaluation 31

32 ALIGNMENT OF RATINGS Question Dashboard metrics Scorecard metrics Data requirement What is the relationship between teacher summative ratings and student achievement? NA # of districts in the top quartile for teacher ratings and the lowest quartile for student achievement [SEA] # of schools in the top quartile for teacher ratings and the lowest quartile for student achievement [LEA] Summative evaluation ratings divided into quartiles and student achievement measures divided into quartiles What is the relationship between professional practice and student growth component ratings? NA % of teachers with aligned growth and practice ratings (within one level) [LEA] % of districts with average professional practice rating more than one category different from average student growth component rating [SEA] Analysis of final growth and practice ratings; probably not available until all ratings are entered into a State system and data can be analyzed 32

33 DISTRIBUTION OF RATINGS Question Dashboard metrics Scorecard Metrics Data requirement What is the distribution of summative evaluation ratings? NA % of teachers by school receiving each summative rating [LEA] % of teachers by grade level receiving each summative rating [LEA] % of teachers by subject area receiving each summative rating [LEA] % of teachers by district receiving each summative rating [SEA] Final summative ratings; available as soon as ratings are entered What is the distribution of summative evaluation ratings for high and low Supplemental Education Service (SES) districts and schools? NA % of teachers by schools in highest and lowest SES quartiles receiving each summative rating [LEA] % of teachers by districts in highest and lowest SES quartiles receiving each summative rating [SEA] % of students on free and reduced-price lunch at the LEA and school level; districts and schools divided into quartiles What is the distribution of summative evaluation ratings for priority schools (schools in the lowest level on the SEA accountability system)? NA % of teachers in priority schools receiving each summative rating [LEA] % of teachers by district in priority schools receiving each summative rating [SEA] Final summative ratings for teachers in priority schools 33

34 IMPACT Question Dashboard Metrics Scorecard Metrics Data requirement How are professional development resources being informed by evaluation data? How are promotions being informed by evaluation data? How is compensation being informed by evaluation data? % of teachers receiving the lowest rating that have an individualized professional development plan [LEA] % of teachers receiving the lowest rating with an assigned mentor [LEA] % of professional development budget allocated to content aligned with area of greatest need based on evaluation data [LEA] % of teachers receiving the lowest rating that have an individualized professional development plan [LEA] % of teachers receiving the lowest rating with an assigned mentor [LEA] NA % of teachers by summary rating category that have received a promotion since their last evaluation [LEA] NA % of teachers by summary rating category that have received a merit-based raise since their last evaluation [LEA] Professional development budget coded by content area and topics aligned with evaluation system; teacher-level professional development plan data; mentor data Teacher-level promotion data Teacher-level compensation data, including reason for raise 34

35 IMPACT Question Dashboard Metrics Scorecard Metrics Data requirement What are the relative retention rates of teachers with the highest and lowest evaluation ratings? NA % of pre-tenure teachers by summary rating category that have been retained in the school [district] [SEA and LEA] % of tenured teachers by summary rating category that have been retained in the school [district] [SEA and LEA] Individual ratings, tenure status and employment status Are tenure-eligible teachers with low ratings being granted tenure? NA % of tenure-eligible teachers receiving one of the lowest two ratings who are granted tenure that year [LEA] 35

36 EDUCATOR PERCEPTIONS Question Dashboard metrics Scorecard metrics Data requirement What is the response rate of teachers and evaluators on the SEA s post-evaluation survey? % of teachers completing survey [SEA/LEA] % of evaluators completing survey [SEA/LEA] NA Survey records Do teachers think the evaluation process contributes to their development? NA % of teachers indicating that the evaluation system contributes to their development [SEA/LEA] Survey data from evaluated teachers Do evaluators think the evaluation system contributes to the development of the teachers they evaluate? NA % of evaluators indicating that the evaluation system contributes to the development of the teachers they evaluated [SEA/LEA] Survey data from evaluators 36

37 QUESTIONS FOR THE SEA TEAM Is the question important to your State? What action can the SEA or LEA take in response to the data? Does the data needed to answer the question exist? If yes, where are the data SEA, LEA or another location? Are the data collected, updated and available during the school year (dashboard) or at the end of the year (scorecard)? Can we identify performance targets for the metric? What s the basis for setting targets? 37

38 EXAMPLE: SEA DASHBOARD METRICS Fidelity of Implementation % of districts with approved evaluation systems % of schools with all evaluators trained/certified % of districts with 90% of evaluators trained/certified % of teachers with approved SLOs % of districts with 90% of teachers with approved SLOs % of teachers who were observed consistent with the schedule % of pre-tenured teachers who were observed consistent with the schedule % of schools on track with observations % of eligible teachers with completed evaluations Educator Perceptions % of teachers completing survey % of evaluators completing survey 38

39 EXAMPLE: LEA DASHBOARD METRICS Fidelity of Implementation % of evaluators completing training % of schools with all evaluators trained/certified % of evaluators completing training % of teachers with approved SLOs % of schools with 90% of teachers with approved SLOs % of teachers who were observed consistent with the schedule % of pre-tenured teachers who were observed consistent with the schedule % of schools on track with observations % of teachers receiving the lowest rating that have a individualized professional development plan % of teachers receiving the lowest rating with an assigned mentor Educator Perceptions % of teachers completing survey % of evaluators completing survey 39

40 EXAMPLE: SEA SCORECARD METRICS Fidelity of Implementation # of districts using each type of evaluation system % of districts with all evaluations conducted by trained/certified evaluators % of districts with 90% of teachers observed consistent with schedule % of districts with 90% of teachers evaluated Alignment of Ratings # of districts in the top quartile for teacher ratings and the lowest quartile for student achievement # of schools in the top quartile for teacher ratings and the lowest quartile for student achievement % of teachers with aligned growth and practice ratings (within one level) 40

41 EXAMPLE: SEA SCORECARD METRICS Distribution of Ratings % of teachers receiving each summative rating % of teachers in priority schools receiving each summative rating Impact % of pre-tenure teachers by summary rating category that have been retained in the school % of tenured teachers by summary rating category that have been retained in the school % of tenure-eligible teachers receiving lowest two ratings who are granted tenure that year Educator Perceptions % of teachers indicating that the evaluation system contributes to their development % of evaluators indicating that the evaluation system contributes to the development of the teachers they evaluated 41

42 EXAMPLE: LEA SCORECARD METRICS Fidelity of Implementation % of all evaluations conducted by trained/certified evaluators % of teachers who were observed consistent with the schedule % of schools using multiple observers in individual teacher evaluations Alignment of Ratings % of teachers with aligned growth and practice ratings (within one level) Distribution of Ratings % of teachers by grade level receiving each summative rating % of teachers by subject area receiving each summative rating % of teachers in priority schools receiving each summative rating 42

43 EXAMPLE: LEA SCORECARD METRICS Impact % of professional development budget allocated to content aligned with area of greatest need based on evaluation data % of teachers receiving the lowest rating that have a individualized professional development plan % of teachers receiving the lowest rating with an assigned mentor % of teachers by summary rating category who have received a promotion since their last evaluation [LEA] % of pre-tenure teachers by summary rating category retained in the school % of tenure-eligible teachers receiving lowest two ratings granted tenure Educator Perceptions % of teachers indicating that the evaluation system contributes to their development [LEA/SEA] % of evaluators indicating that the evaluation system contributes to the development of the teachers they evaluated [LEA/SEA] 43

44 STATE TEAM PILOT DEVELOPMENT 44

45 SESSION OBJECTIVES States work within teams to begin to: Determine which tool they would like to develop for this session Identify critical questions in one category Identify the metrics to address those critical questions 45

46 PILOT DEVELOPMENT With your State team, review the menu of questions and metrics. Then, identify: The tool you want to focus on for this session One area where you want to focus your initial data efforts The key questions within that area that are most important to your State Which metrics you want to use to answer that question Experts and RSN staff will be available for questions and support. 46

47 STATE DEBRIEF 47

48 SESSION OBJECTIVES States report out the successes and challenges from their pilot development. 48

49 GROUP REPORTING Take two to three minutes to explain your State s experience during the previous session. Specifically, answer the following questions: Is the process working for your State? If not, where is your State having the most trouble? What input or feedback do you have based on your State s experiences? 49

50 BREAK Please Return at 10:15 a.m. 50

51 STATE DEVELOPMENT OF DASHBOARDS AND SCORECARDS 51

52 SESSION OBJECTIVES Participants will work in State teams to begin creating prototypes of a dashboard and/or scorecard. 52

53 DASHBOARD AND SCORECARD DEVELOPMENT With your State team, work to answer the questions outlined on the template and identify the set of metrics you will want to use. Experts and RSN staff will be available for questions and support. Afterward, teams will present their work in pairs and receive feedback. Please be ready to report out on: The tool(s) and critical questions you have selected One to two challenges you ve encountered thus far 53

54 STATUS REPORT 54

55 SESSION OBJECTIVES States report out on their progress and discuss their biggest successes and challenges. 55

56 STATE TEAM PAIRS Please meet with your State team pair and facilitator: New York and New Jersey (Larry Stanton) North Carolina and Delaware (Jon Fullerton) Ohio and Maryland (Corey Chatis) 56

57 STATUS REPORT Each State should take turns describing their progress. Focus on the following questions: (10 min) Which tools have you decided to focus on? Why? What questions have you addressed? What are one to two challenges that remain? Afterward, the other State will respond addressing the following questions: (10 min) Will the selected metrics answer the team s priority questions? If the team is building multiple tools, are the metrics aligned across tools? Do you have ideas for addressing their biggest challenges? 57

58 LUNCH Please Return by 12:45 p.m. 58

59 STATE DEVELOPMENT OF DASHBOARDS AND SCORECARDS 59

60 SESSION OBJECTIVES Participants will work in State teams to continue creating prototypes for a dashboard and/or scorecard. 60

61 DASHBOARD AND SCORECARD DEVELOPMENT With your State team, continue to develop your dashboard and/or scorecard, incorporating feedback from the previous session. Using the worksheet template, identify the final metrics for your dashboard or scorecard. Experts and RSN staff will be available for questions and support. In the next session, be prepared to present your work to your partner State, including: What was our focus for the day, including tools and priority questions? Where did we make progress? Where did we have difficulty? Why? What are our next steps? 61

62 STATE COLLABORATIVE TEAM TIME 62

63 SESSION OBJECTIVES Participants will come together in pairs to present their prototypes and provide feedback. 63

64 PAIR FEEDBACK PROTOCOL 1. Presenting team describes their work, addressing the following questions: What was our focus for the day, including tools and priority questions? Where did we make progress? Where did we have difficulty? Why? What are our next steps? 2. Responding team asks questions for clarification. 3. Responding team provides feedback, noting positive aspects of the work, areas in need of additional attention and specific recommendations for next steps. 4. Teams switch roles. 5. Afterward, each team considers the feedback to revise their plan and develop next steps. 64

65 BREAK Please Return at 2:45 p.m. 65

66 ACTION PLANNING 66

67 SESSION OBJECTIVES Participants will work within their State teams to develop a 90 day plan for starting to implement the prototype data tool(s). 67

68 ACTION PLANNING Each State team will work with a facilitator to begin planning how to implement their tool(s). Use the template provided to identify four to six key implementation steps over the next 90 days. Consider the guiding questions in the template. Also begin to think through the additional technical assistance the RSN can provide to support your design and implementation of this prototype. We will discuss this before we close. 68

69 STATE FACILITATORS Delaware Bill Horwath Maryland Chad Rubalcaba New Jersey Corey Chatis New York Larry Stanton North Carolina Jon Fullerton Ohio John Luczak 69

70 NEXT STEPS 70

71 FOLLOWING UP AND NEXT STEPS Experts and the RSN will stay in contact in the coming months to identify success stories and lessons learned. What would be most useful for your State needs in terms of RSN follow up? Regular subcommittee calls One-on-one or small group check-ins Expert feedback on prototypes Assist in creating stakeholder engagement plans Outline data and IT considerations Other strategies 71

72 QER PORTAL 72

73 THANK YOU 73