Sample Career Ladder Program Evaluation Process

Transcription

1 Sample Career Ladder Program Evaluation Process SAMPLE The Purpose of the Career Ladder Program Evaluation Process The goal of the program evaluation process is to determine the impact and effectiveness of newly designed and implemented career ladder systems. The program evaluation process will measure the extent to which the following have been achieved: Operational/Process Success: The implemented system functions as designed. Results/Outcomes: The intended outcomes are achieved. Stakeholder Perceptions & Investment: Key stakeholder groups understand and are invested in the system. These program evaluation data collection and analysis efforts are designed to provide information that will enable you to: Continue to refine the system and improve supports for implementation. Understand the effect of the implemented system on teacher and student outcomes. Monitor stakeholder perceptions of the system and implement the necessary communications, change management and training strategies. Monitor system cost and sustainability. Process Overview The following is a high-level overview of the scope and sequence of a typical program evaluation process. 1. Phase I Establish Program Evaluation Scope and Framework 1.1. Define key metrics for the program evaluation process aligned to the system s guiding principles and goals, which drive the system s design; determine what it would mean to be successful according to each metric Operational/process measures of success (i.e., Does the system work as designed?) System outcomes/results measures of success (i.e., Does the system achieve the intended results?) Stakeholder perceptions measures of success (i.e., Do key stakeholders perceive the system to be successful?) 1.2. Determine data needs for baseline and ongoing key metric measurement 1.3. Identify and establish necessary systems for data collection and organization 1.4. Refine and finalize key metrics based on feasibility of measurement and reporting 2. Phase II Data Collection and Analysis 2.1. Establish a baseline for key metrics pre-implementation 2.2. Set targets and/or benchmarks for each key metric 2.3. Gather, track and analyze data to establish key metrics during and post implementation 2.4. Gauge progress toward targets or benchmarks 3. Phase III System Assessment 3.1. Assess operation of the system, including: implementation, functionality, and sustainability/costs 3.2. Assess results of the system, including: teacher attraction, retention, performance and satisfaction 3.3. Assess stakeholder perceptions of the system(s) for teachers, administrators, and district staff 1

2 4. Phase IV Recommendations 4.1. Establish recommended next-steps to leverage results from the program evaluation process to refine and improve system operations and outcomes Timeline The following section provides high level guidance around a possible timeline for executing the program evaluation process. It is important to note that some phases of the program evaluation process should ideally occur in advance of, or concurrently with, system implementation to ensure that the appropriate structures are in place to execute a successful evaluation process. Pre-Implementation 1-3 months Phase I Establish Program Evaluation Scope and Framework Prior year data: rosters, evaluation, student achievement, etc. Initial Implementation 6-9 months Phase II Data Collection and Analysis Data Collection Teacher rosters Recruitment surveys Budget data from schools and/or district Teacher, Administrator, and HR surveys (Mid- Year) Teacher evaluation data (Mid-Year) Student achievement data (Mid-Year) Phase III System Assessment End of Year 1-3 months Phase IV Recommendations Teacher evaluation data (EOY) Student achievement data (EOY) Teacher, Administrator, and HR surveys (EOY) Guiding Research Questions The research questions below are an example of the types of questions that should be asked when conducting the program evaluation process for a performance-based career ladder system. While not exhaustive, this general list should serve as a solid starting point to be adapted and elaborated upon. This series of guiding questions includes both quantitative and qualitative research questions. As a result, different approaches to data collection and analysis will be needed to appropriately evaluate different questions (see the Appendix for further details). Operational/Process Success o Implementation Were there persistent challenges that impacted the ability of the career ladder system to be implemented? Types of challenges include: Technological/Systems-related Cultural Administrative o Functionality Does the career ladder system function as designed (e.g., are teachers entering and progressing through the system as intended)? Does the selection process result in appropriate candidate selection for career ladder roles? Were the inputs or guidelines used to drive selection into, and progression through, the career ladder system appropriate (e.g., evaluation ratings, etc.)? Are the selected candidates successful in their roles? What are the outcomes as a result of having each leadership role in a school? 2

3 o Do existing evaluation systems provide adequate, accurate data to appropriately inform career ladder eligibility/progression? Are teachers in career ladder roles provided with the guidance, development and support necessary to be successful in their role? Do teachers serving in career ladder roles have clarity around the scope and responsibilities of the role? Do teachers being supported by career ladder roles have clarity around the support they should expect to receive from this role? Is the workload affiliated with the role sustainable, given other teacher duties and provided release time (if any)? Are there areas of challenge/concern resulting from the career ladder system as designed? Are there areas of challenge/concern not currently addressed by the career ladder system, but could or should be (e.g., retention of teachers in specific schools, grades, or subjects)? Sustainability & Costs Do projected career ladder program costs align to actual costs? Is the implemented career ladder system sustainable and affordable (and, if relevant, scalable), given results to date? Results/Outcomes o Retention Do high performing teachers intend to stay longer due, at least in part, to the implemented career ladder system? Do high performing teachers actually stay longer due, at least in part, to the implemented career ladder system? Do teachers in the school(s) with implemented career ladder roles intend to stay longer, at least in part, as a result of the support provided by these roles? Do teachers in the school(s) with implemented career ladder roles intend to stay longer due, at least in part, to future access to career ladder roles? o Attraction of Top Talent Did new hires surpass the average quality of performance of new hires from previous years, as measured by their first-year evaluation scores? Do high performing new hires cite the career ladder system as a reason for being attracted to the school/district/network? Was the average new hire date earlier than in previous years? What are possible reasons for this? o Teacher Quality & Performance Did teacher quality, as measured by the performance evaluation system and student outcomes, increase for teachers who were directly supported by implemented career ladder roles? Did teachers holding career ladder roles see a change in their performance? o Student Performance Did student performance change (positively or negatively) in the classrooms of teachers receiving support from the implemented career ladder roles? Did student performance change (positively or negatively) in the classrooms of teachers serving in a teacher leadership role? o Teacher Satisfaction Did teacher satisfaction increase as a result of there being teacher leadership opportunities available within the school/district/network? 3

4 o Did teacher satisfaction increase for teachers who were directly supported by the implemented career ladder roles? Did teacher satisfaction increase for teachers holding career ladder roles? Student Satisfaction (Optional) Did student satisfaction increase? Did student satisfaction with instruction in the areas supported by career ladder roles (e.g., reading intervention) increase in the classrooms of teachers receiving support from the implemented roles? Did student satisfaction of teachers participating in career ladder roles see a change? Stakeholder Perceptions & Investment o Do teachers and administrators believe the existing career ladder system: Is transparent, accessible, easy to understand and easy to navigate? Is positively impacting overall school performance? Provides the right capacity, in terms of time and/or skills, in the right ways? Generates attractive opportunities (especially for top performers) that lead to increased attraction, retention and recognition of top talent? Is implemented well? o Do district/hr staff and leadership believe the implemented career ladder system: Functions appropriately in that it provides the right capacity (e.g., time and/or skills) in the right ways and serves its intended purposes? Is affordable and sustainable (and, if relevant, scalable)? Requires further refinement or implementation support? 4

5 Career Ladder Program Evaluation Data Sources SAMPLE Operational/Process Success Were there persistent challenges that impacted the ability of the career ladder Implementation system to be implemented? Technological/Systems-related - IT/Data Team staff survey, interviews, or focus groups Cultural Administrative Functionality Sustainability & Costs Does the career ladder system function as designed (e.g., are teachers entering and progressing through the system as intended)? Does the selection process result in appropriate candidate selection for career ladder roles? Were the inputs or guidelines used to drive selection into, and progression through, the career ladder system (evaluation ratings, etc.) appropriate? Are the selected candidates successful in their roles? What are the outcomes as a result of having each role in a school? Do existing evaluation systems provide adequate, accurate data to appropriately inform career ladder role eligibility/progression? Are teachers in career ladders roles provided with the guidance, development and support necessary to be successful in their roles? Do teachers serving in career ladder roles have clarity around the scope and responsibilities of the role? Do teachers being supported by career ladder roles have clarity around the support they should expect to receive from this role? Is the workload affiliated with the role sustainable, given other teacher duties and provided release time (if any)? Do projected career ladder program costs align to actual costs? - Teacher roster (multiple years) (multiple years) - Teacher survey or interview - Role-specific rubric evaluation data - Student performance data - Role-specific rubric evaluation data - Teacher evaluation component data - Student achievement data by teacher - Evaluation system rubrics + supporting materials - Teacher leader survey, interviews, or focus groups - Teacher leader survey, interviews, or focus groups - Supported teacher survey, interviews, or focus groups - Teacher leader survey, interviews, or focus groups - Teacher leader time use inventory - Program budget data from school and/or district - Initial system cost projections 5

6 s - Salary data Is the implemented career ladder system sustainable and affordable (and, if relevant, scalable), given results to date? - Current and forecasted budget data from school and/or district - Initial system cost projections - Current and forecasted career ladder headcount / scale information Results/Outcomes Do high performing teachers intend to stay longer due, at least in part, to the implemented career ladder system? - Teacher rosters (multiple years) Do high performing teachers actually to stay longer due, at least in part, to the implemented career ladder system? Retention Do teachers in the school(s) with implemented career ladders intend to stay longer, at least in part, as a result of the support provided by these roles? Do teachers in the school(s) with implemented career ladders intend to stay longer due, at least in part, to current or future access to career ladder roles? Attraction Teacher Quality & Performance Student Performance Teacher Satisfaction Did new hires surpass the average quality of performance of new hires from previous years, as measured by their first-year evaluation scores? Do new hires cite the career ladder system as a reason for being attracted to the school/district/network? Was the average new hire date earlier than in previous years? What are reasons for this? Did teacher quality, as measured by performance, increase for teachers who were directly supported by implemented career ladder roles? Did teachers holding career ladder roles see a change in performance? Did student performance change (positively or negatively) in the classrooms of teachers receiving support from implemented career ladder roles? Did student performance change (positively or negatively) in the classrooms of teachers serving in a teacher leadership role? Did teacher satisfaction increase as a result of there being teacher leadership roles/opportunities available within the school/district/network? Did teacher satisfaction increase for teachers who were directly supported by implemented career ladder roles? Did teacher satisfaction increase for teachers holding career ladder roles? - Teacher new hire roster (multiple years) or teacher roster with hire date (multiple years) - Recruitment survey, interviews, or focus groups - Teacher new hire dates (multiple years) - Recruitment survey, interviews, or focus groups - Supported teacher roster (multiple years) - Leadership role roster - Supported Teacher roster - Student achievement data by teacher (multiple years) - Student achievement data by teacher (multiple years) - Supported teacher roster 6

7 Did student satisfaction increase? - Student survey, interviews, or focus groups Did student satisfaction with instruction in the areas supported by career - Student survey, interviews, or focus groups ladders (e.g., reading intervention) increase in the classrooms of teachers Student Satisfaction - Supported teacher roster receiving support from implemented career ladders? Did student satisfaction of teachers participating in career ladder roles see a change? - Student survey, interviews, or focus groups Stakeholder Perceptions & Investment Do teachers believe the existing career ladders framework is transparent, accessible, easy to understand and easy to navigate? Do teachers believe the existing career ladders framework is positively impacting overall school performance? Teacher Perceptions Do teachers believe the existing career ladders framework provides the right capacity, in terms of time and/or skills, in the right ways? Do teachers believe the existing career ladders framework generates attractive opportunities (especially for top performers) that lead to increased attraction, retention and recognition of top talent? Do teachers believe the existing career ladders framework is implemented well? Do administrators believe the existing career ladders framework is transparent, accessible, easy to understand and easy to navigate? Do administrators believe the existing career ladders framework is positively impacting overall school performance? Administrator Perceptions Do administrators believe the existing career ladders framework provides the right capacity, in terms of time and/or skills, in the right ways? Do administrators believe the existing career ladders framework generates attractive opportunities (especially for top performers) that lead to increased attraction, retention and recognition of top talent? Do administrators believe the existing career ladders framework is implemented well? Do district/hr staff believe the existing career ladders framework functions appropriately in that it provides the right capacity, in terms of time and/or skills, in the right ways and serves its intended purposes? HR Staff Perceptions Do district/hr staff believe the existing career ladders framework is affordable and sustainable (and, if relevant, scalable)? Do district/hr staff believe the existing career ladders framework requires further refinement or implementation support? 7