INSTRUCTIONAL COACHING: AN EVALUATION OF THE MODEL AT NORTHWEST AEA

INSTRUCTIONAL COACHING: AN EVALUATION OF THE MODEL AT NORTHWEST AEA By Judy Sweetman EdTech 505 Boise State University Prepared for the CEO and Director of the Educational Services Division of Northwest AEA, Sioux City, IA

SUMMARY The instructional coach model, researched and designed by Jim Knight and the KU Center for Research on Learning, was implemented in all of the public and private schools in Northwest AEA over a period of three years, beginning with the 2009 2010 school year. The instructional coach model consists of a partnership program of professional development for the purpose of increasing teacher use of research-based strategies and interventions. An instructional coach helped teachers to improve their content knowledge, instruction, classroom management, and student achievement. Formative assessment was addressed with teachers to drive instruction and to help them provide constructive feedback to students. Coaches were instructed to allot 60% of their time to coaching and 40% to other assigned responsibilities. This percentage was explained to superintendents and published on Northwest AEA s website. Instructional coaches were asked to record on a Google spreadsheet the amount of hours they had spent on instructional coaching activities during the months of September 2011 through March 2012. They were also asked to note the amount of time they had spent on other designated responsibilities, such as SINA/DINA (Schools/Districts in Need of Assistance), Iowa Core Curriculum, or AIW (Authentic Intellectual Work). The amount of time spent on each responsibility was averaged to determine if coaches were spending 60% of their time coaching, or if they were spending less time than 60%. After averaging the percentage of time each instructional coach recorded on the individual spreadsheets, it was determined that coaches were spending 46% of their time on instructional coaching and 54% on other assigned responsibilities. This percentage of 46% was based on the broad definition of coaching as defined on page 7 of this report and also in the letter to coaches in Appendix A. Because instructional coaches were not, for the most part, keeping a record of their coaching time, nor were they completing a reflection report (one of the eight tasks of the program defined on page 6), the definition of instructional coaching had to be defined much more broadly for this evaluation report than in the research studies of Jim Knight and the KU Center for Research on Learning the program in which the instructional coaches were trained. This could imply that the program has not been implemented with fidelity, and therefore may not achieve the same results as the research. Additionally, this could imply that instructional coaches may have spent even less time coaching than the average time reported of 46%. 2

INTRODUCTION Jim Knight, from Kansas University, helped to develop the instructional coaching model based on research conducted at The KU Center for Research on Learning and the Pathways to Success program. Instructional coaches are considered professional developers yet equal partners, confidants, and colleagues as they work one-on-one or with small groups of teachers in assigned schools. Based on the needs of the teacher and/or the needs of the class, instructional coaches share information on researchbased practices and interventions, leading the district one teacher at a time towards positive change and school improvement. Instructional coaches, according to Knight, can help teachers impact student learning in four different areas. These are: planning content, formative assessment, delivering instruction, and community building/classroom management. During the 2009 2010 school year, trainers from The KU Center for Research on Learning instructed the first group of instructional coaches at Northwest AEA on these four areas. The second group of coaches was trained the following year, and the final group was trained during the 2011 2012 school year. The goal was to increase student achievement in the school districts the agency served through this new model, and provide a partnership program of professional development so teachers would be more likely to implement strategies. Northwest AEA employed 19 instructional coaches to serve all of the public and accredited private schools in 10 counties of northwest Iowa. A coach served approximately 2,000 students. Because of student population, some coaches had just one school, while others had up to five different schools. Each of the instructional coaches employed by Northwest AEA also had many other duties; therefore, it was determined by the CEO that instructional coaches should spend 60% of their time coaching, and 40% of their time completing their other assigned duties. At the March Educational Services meeting at Northwest AEA, the CEO met with the instructional coaches to inquire how coaching was going. The CEO was informed that because of many additional duties required by the State Department of Education, it was not possible to devote 60% of their time to instructional coaching. The CEO wanted to know exactly how much time was being spent on instructional coaching. The purpose of this report, therefore, was to determine the discrepancy between the 60% of time allotted for instructional coaching and the 40% of time allotted for other responsibilities. Each of the 19 instructional coaches was asked to determine how much time they had spent on coaching during the months of September 2011 through March 2012. They were also asked to determine what percentage of their time was spent on specific responsibilities other than coaching. The percentage of time spent coaching compared to other responsibilities would help the CEO and Director of Educational Services make a decision about time allotment for instructional coaching for the 2012 2013 school year, as well as a decision about the implementation of the program. 3

DESCRIPTION OF THE PROGRAM EVALUATED Northwest AEA, an intermediate agency, serves 36 public school districts and 33 accredited, private schools in a 10-county area of northwest Iowa. Student population is around 43,000 students, with 5,000 having special education needs. There are 3,239 teachers. The mission of Northwest AEA is to provide visionary leadership and quality, student-centered services through relationships with families, schools and communities. Instructional coaches at Northwest AEA were trained over the course of a school year by trainers from the KU Center for Research on Learning. Instructional coaches learned how to roll out the model in a school, and how to work with teachers through six principles: equality, choice, voice, reflection, dialogue, and praxis (applying new ideas to real-life practices). Coaches learned about a variety of learning structures and effective questioning strategies, and were provided with books, field guides, and workbooks. They were required to attend a weeklong APL (Instructional Skills and Strategies) training and encouraged to attend a one-day seminar on classroom management with Randy Sprick. Coaches were also encouraged to attend the annual Instructional Coaching Conference held each fall at Kansas University. The first instructional coaches at Northwest AEA were trained during the 2009 2010 school year, the second group was trained the following year, and the last coaches were trained during the 2011 2012 school year. The 2011 2012 school year was the third year of implementing the instructional coach model at Northwest AEA. The first two years were voluntary for the schools, and approximately two-thirds of the districts were using the model by the 2010 2011 school year. All 69 districts were involved by the third year. Instructional coaches were instructed to spend 60% of their time in their assigned schools, implementing this model. Coaches who had just one assigned school were to spend three days each week in their school. Coaches who were assigned more than one school district might spend as little as one day in their public schools and half a day each week in their private schools. If coaches did not have a specific meeting or task within their assigned school on a particular day, they were encouraged to be visible during the designated time each week. In the capacity of an instructional coach, the coaches worked with individual teachers, small groups of teachers, administrators, entire staffs, and/or with groups of students, providing resources, analyzing data, facilitating building leadership teams, serving on child find teams, modeling, and sharing interventions, research-based strategies, and respectful conversations. Instructional coaches were to consider themselves equal partners with teachers, not as evaluators or as spies. Research has shown that when implemented with fidelity, the instructional coach model has increased student achievement, and through a partnership program of professional development, has increased teacher use of research-based strategies and interventions. 4

Program Objectives Jim Knight developed the program objectives for the instructional coach model based on the work done previously with the Pathways to Success program and the KU Center for Research on Learning. The studies were conducted in nine middle and high schools over a period of seven years. The objectives of the program are: Positively impact teachers instructional practices Enable teachers to implement scientific, research-based practices Assist the schools with and through the school improvement process Assist school-based teams with analysis of student achievement data and teacher implementation data Assist with instructional decisions that will impact students identified as at risk of academic or behavioral failure Provide demonstration, feedback, and coaching of instructional and classroom management strategies. Collaborate and build networks with special education strategists, administrators, and teachers Change the focus of traditional professional development to sustainable efforts to learn and implement effective strategies, expertise, and support in curriculum, instruction, and assessment. Program Components There are four major components of the instructional coach model, known as The Big Four. They provide a framework and a focus for the instructional coach, and were considered by Knight to have the biggest impact on helping students. The Big Four are: 1. Behavior - Coaches helped teachers to develop and teach expectations; kept track of the ratio of interactions; assisted with effective, corrective comments; kept track of time students were on task; helped teachers plan out instructional content; and helped teachers to see where they could give students more time to interact with the content. 2. Content knowledge - Coaches helped teachers to use essential questions to align their lessons with state standards; helped teachers to map the most essential content to be taught; helped teachers identify key content structures; assisted teachers in identifying, defining, and teaching concepts; and assisted teachers in enhancing their instruction so more students mastered the content. 3. Direct instruction Coaches provided teachers with advance and post organizers, modeled in their classrooms, assisted teachers in asking higher-order questions, assisted teachers in developing quality assignments, and gave direction to teachers in providing effective feedback to their students. 4. Formative assessment Coaches assisted teachers in developing course and unit questions; helped teachers to create appropriate, high-quality assessments; assisted teachers in how to provide constructive feedback; and showed teachers ways in which they could involve students in the entire assessment process. 5

There are eight tasks that instructional coaches used to assist teachers in implementing The Big Four. These are: 1. Enroll - The instructional coach sought to find teachers interested in becoming involved with the model. 2. Identify Coaches identified with which of the Big Four to begin. 3. Explain The instructional coach explained interventions, suggested what teachers should watch for during the model lessons, and/or asked about and addressed teachers concerns. 4. Model The instructional coach modeled to show a teacher how to implement a particular intervention, a specific research-based strategy, or a classroom management strategy. 5. Observe The instructional coach observed a teacher to watch for data related to critical teaching behaviors, fidelity to scientifically proven practices, student behavior and performance, and/or additional specific teacher concerns. 6. Explore data collaboratively Both the teacher and the instructional coach identified what data would be gathered and how to use it. 7. Support The instructional coach differentiated the relationship with each teacher in order to fit the particular needs of each teacher. 8. Reflect Coaches completed an After-Action Report to reflect about what was supposed to happen, what actually happened, what accounted for the difference, and what would be done differently next time. Materials that were provided to instructional coaches in order to implement the program with fidelity were the following: Killion, J. & C. Harrison. (2006). Taking the lead: New roles for teachers and school-based coaches. Oxford, Ohio: National Staff Development Council. Knight, J. (2007). Instructional coaching: A partnership approach to improving instruction. Thousand Oaks, California: Corwin Press. Knight, J. (2007). Instructional coaching institute: A partnership approach to improving instruction, 3-day workbook. Lawrence, Kansas: Instructional Coaching Group. Knight, J. (2002). Partnership learning fieldbook. Lawrence, Kansas: The University of Kansas Center for Research on Learning. Sharer, T.J., J.T. Anastasio, & D.J. Perry. (2007). Teaching: The book. Instructional skills and strategies for the experienced and the novice teacher. Camillus, New York: APL Associates. Sprick, R., Ph.D. (2006). Coaching classroom management: Strategies and tools for administrators and coaches. Eugene, Oregon: Pacific Northwest Publishing. EVALUATION METHOD Participants The participants involved in the evaluation were the 19 instructional coaches employed solely by Northwest AEA. Eighteen of the instructional coaches were full-time 6

employees and one was part-time. There were three additional part time coaches employed half time by Northwest AEA and half time by their own school district. The Director of Educational Services Division, who oversaw all of the instructional coaches, suggested not including in the evaluation report these three additional coaches, as their responsibilities were very different than the responsibilities of the other 19 coaches. Seven of the 19 coaches had been an instructional coach for three years, eleven had been coaching for two years, and one was an instructional coach for one year. Most were assigned to the same school districts for the two or three years they were coaching in order to establish and sustain relationships with the teachers. Other participants who were key personnel in the program were the CEO of Northwest AEA and the Director of the Educational Services Division of Northwest AEA. Both of these two people were instrumental in designing the rollout of the program, obtaining the training for the coaches, and overseeing the implementation of the program. Procedures At the March Educational Services Meeting, at which the CEO was present to discuss the progress of the instructional coach model, he inquired if coaches were allotting 60% of their time to coaching and 40% to other responsibilities. He was informed that this was not possible due to additional responsibilities required by the State Department of Education, especially the responsibility of helping districts implement the Iowa Core Curriculum. Coaches estimated that they were spending approximately 40% of their time coaching and 60% of their time completing the other responsibilities. The CEO wanted to know how much time coaches were actually spending on coaching. A letter was sent to each of the instructional coaches asking them to complete their own Google spreadsheet on the percentage of time they spent coaching each month from September 2011 through March 2012, and the percentage of time they spent on other responsibilities (The letter can be found in Appendix A.). Because it would be impossible for each coach to remember exactly what happened several months ago, instructional coaching had to be defined more broadly than was defined by Jim Knight and the KU Center for Research on Learning. Therefore, instructional coaching was defined as: Time spent coaching in an assigned coaching school Planning for an assigned coaching school even when in the office Modeling for a teacher Co-teaching with a teacher Teaching for a teacher while teacher was working with other students/observing other teachers Observing students, serving on Care Teams, IEP meetings, etc. Sitting in an assigned school waiting for someone to find them Providing professional development for an assigned coaching school Facilitating Leadership teams AIW (Authentic Intellectual Work) in an assigned coaching school Analyzing data for an assigned school 7

The other responsibilities for which instructional coaches were to provide a percentage were: SINA/DINA (Schools/Districts in Need of Assistance) Iowa Core Curriculum Educational Services/Regional/PLC meetings Meetings in Des Moines other than Iowa Core Providing content area professional development in unassigned schools AIW (Authentic Intellectual Work) in unassigned schools Professional conferences Student events Assessment in unassigned schools Other in-house meetings Exchange days/sick days/snow days/comp time Other In addition, instructional coaches were asked to keep track of the time it took them to complete their individual spreadsheet in order to calculate how much it cost the agency to generate this data. Data Sources All 19 instructional coaches completed a Google spreadsheet. Some computed the averages for each area, and some just provided the number of hours or days. To determine the percentage of time each instructional coach spent on coaching as compared to other responsibilities, the number of days the instructional coach worked each month was multiplied by eight to get a total number of hours worked between September 2011 and March 2012. Then, for each responsibility, the number of hours was added up and divided by the total number of hours worked to determine the percentage of time allotted for each responsibility. The business manager provided the hourly rate for each instructional coach. He provided two figures salary only and salary, FICA and IPERS. These figures were multiplied by the number of hours it took each instructional coach to compile the data for their individual time sheets to determine the cost to the agency to generate this data. RESULTS The results of the data indicated that the 19 instructional coaches were spending an average of 46% of their time on instructional coaching, and 54% on other assigned duties. This is 14% less time allotted to instructional coaching than the CEO had asked the instructional coaches to spend. Table 1 (page 9) depicts the amount of time spent on each major, assigned responsibility. 8

Table 1: Percentage of Coaches Time Responsibilities Percentage of Time Spent Coaching 46% SINA (Schools in Need of Assistance)/DINA (Districts in Need of Assistance) 5% Iowa Core 6% Ed Services meetings/regional meetings/plc (Professional Learning Communities) meetings 5% Meetings in Des Moines other than Iowa Core 3% Providing content area Professional Development in unassigned schools 5% AIW (Authentic Intellectual Work) in unassigned schools 6% Professional conferences 2% Student events 1% Assessment in unassigned schools 1% Other in-house meetings (KU Strategic Instruction Model, Data team, Leadership Council, etc.) 3% Exchange days/sick days/snow days/comp time 6% Library/Media 3% ELL/Title III 1% CTE/Counselor support 1% License renewal 1% Other 5% DISCUSSION The purpose of this study was to determine if the 19 instructional coaches employed by Northwest AEA were spending 60% of their time on instructional coaching and 40% of their time on other responsibilities, as instructed by the CEO and published to the administrators of the 69 school districts. Coaches did not believe that they were meeting the 60% requirement because of mandates from the State Department of Education, on which they were spending more and more time. Based on the average time from the time sheets completed by each of the 19 instructional coaches, time spent on instructional coaching was 46%. This was 14% less than the amount of time the CEO required of the coaches. The studies completed by Jim Knight and the KU Center for Research on Learning indicated that instructional coaching was a research-based model from which districts could expect impressive results. One of the reasons for the high results was the partnership philosophy established between the coaches and the teachers. Another reason was the resources, training materials, and research-based strategies and 9

interventions that the coach could share with teachers in order for the teacher to positively change instruction and to raise student achievement. Because instructional coaches employed by Northwest AEA had not been previously instructed to keep a record of their coaching activities, it was difficult for coaches to go back and try to remember what they had done several months ago. Therefore a very broad definition of coaching was used in order to generate a percentage of time. Using this definition, as opposed to The Big Four and the eight tasks developed by Jim Knight and the KU Center for Research on Learning, provided an estimate of the time spent in assigned schools, rather than an accurate percentage of time spent on instructional coaching activities. An overall evaluation of the program, based on reported use of coaches time, is that instructional coaches at Northwest AEA have not been able to devote their time to the model as intended by Jim Knight and the KU Center for Research on Learning. Recommendations for the program would be the following: The books and workbooks purchased for the coaches contain many useful strategies. Unless a coach reviews these periodically, these resources will be forgotten or not implemented with fidelity. Reviewing a chapter or two during each PLC meeting or Educational Service meeting would bring the information back to the forefront of coaches minds. For the 2012 2013 school year, keep track of coaching time four different times throughout the year during two-week time slots. The two-week periods might take place in September, November, February, and April. This will help coaches and administrators determine more accurately what is taking up coaches time. Determine prior to the 2012 2013 school year the definition and components that will constitute instructional coaching. The definition used for the purpose of this study was quite broad, and did not necessarily meet the criteria described by Jim Knight and the KU Center for Research on Learning. If implemented with fidelity, the program at Northwest AEA should get the same results as the original research studies. Send a survey to all school districts in Northwest AEA listing the services, and having them rate the ones they use the most and which ones they think they could do without in order to free up time for instructional coaching. Table 2: Cost to Generate Coaches Time Sheets Item Analysis Time Salaries Salaries with FICA & IPERS Instructional coaches completing time logs 70.5 hours $2,977.06 $3,444.00 Table 3: Invoice to Northwest AEA for Evaluator Services Item Analysis Cost Daily Rate 3 days x $500.00 $1,500.00 Expenses Meals, mileage, motel $500.00 Total $2,000.00 10

APPENDIX A Letter to Instructional Coaches at Northwest AEA Dear Coaches: As many of you know, I have been working on my Masters of Educational Technology Degree at Boise State University for the past several years. I am in my last required course, Evaluation of Educational Technologies. Our final project is to write a 10- page Evaluation Report, using real data not hypothetical. (An evaluation is different from research in that it looks to improve programs or projects, while research draws causal links and leads towards the development of knowledge. Evaluation is the systematic process of collecting and analyzing data in order to make a decision, or to determine whether and to what degree program or project objectives have been or are being achieved.) After our Ed. Services meeting last Friday, I, as I m sure many of you, were still thinking about our meeting. The question about how much time was actually spent on coaching kept rolling around in my head. Finally, late Sunday night, it clicked. Why not do my 10- page report on something for which we all need more information: the amount of time spent coaching. I emailed Tim and Pam, and both were in favor of my doing this. So, here comes the work part. I will need two sets of data: time spent on coaching and amount of time it took you to complete this data gathering. Please go back through this past school year from September 1 through March 30. For each month, please figure out the percentage of time that was spent on coaching and the amount of time that was spent on other activities such as SINA/DINA, Iowa Core, meetings in Des Moines other than Iowa Core/SESA 7, AIW in a school to which you are not assigned, professional development in a school to which you are not assigned, content area professional development in a school to which you are not assigned, professional conferences, student events, etc. Please look at each week in either five or 10 segments morning and afternoon. That way it will be easier (I hope) to figure out the percentages for each week/month. Also, to stay consistent, our definition of coaching is going to be: Time spent coaching in an assigned coaching school Planning for an assigned coaching school even when in the office (but not after hours) Modeling for a teacher Teaching for a teacher while teacher is working with other students/observing other teachers Co- teaching with a teacher Observing students, serving on Care Teams, IEP meetings, etc. Sitting in an assigned school waiting for someone to find you Providing professional development for an assigned coaching school Facilitating Leadership teams AIW in your assigned coaching school Analyzing data for your assigned school 11

I am going to share a Google Spreadsheet with you. Look at the tabs at the bottom of the spreadsheet and find your name. Please enter your data ONLY on the spreadsheet that has your name on it. No one else will have the link or viewing privileges to this spreadsheet. Neither Tim nor Pam will know individual times that you report, as the final report will only have our total data. None of this information will be used to evaluate anyone s performance. I promise not to share any individual data with anyone! Individual data will be kept confidential. All I ask is that you try to be as accurate as possible so that the data will be useful. Tim and Pam are planning to use the information in the report to make decisions about next year, and to share with administrators. If you have time (for example, while you are in your school waiting for someone to find you), please begin working on this. Pam is also going to give us time to finish up at our April 13 Ed. Services meeting. I will need to have all of the data entered by April 13, as that only gives me two weeks to write the entire report, as it is due May 1, plus Tim and Pam would like the information as soon as possible. Thank you for your help! I really appreciate it. Please let me know if you have questions. Judy 12