Impact of the Payne School Model on Student Achievement Cannon County Schools Woodbury, Tennessee First-Year Report: 2004 05 Data William W. Swan, Ed.D., The University of Georgia Athens, GA.
Impact of the Payne School Model on Student Achievement Cannon County Schools Woodbury, Tennessee First-Year Report: 2004 05 Data William W. Swan, Ed.D. The University of Georgia Athens, GA Abstract Analyses of Covariance (ANCOVA) using the Tennessee Comprehensive Assessment Program (TCAP) in Mathematics and Reading for Grades 2, 4, 5, 6, 7, and 8 in 2004 05 as dependent variables and the TCAP in Mathematics and in Reading in 2003 04 for the same students in the prior year in each area respectively as the covariates were conducted for two groups of students. One group was students of teachers scoring in the High Model Fidelity implementation and another was students of teachers scoring in either Low Model Fidelity or no implementation. Statistically significant results were identified for Grade 2 in both Mathematics and Reading, favoring the High Model Fidelity group, and for Grade 8 in Reading, favoring the Other group. Of the total of 12 ANCOVAs six for Mathematics and six for Reading eight sets of adjusted means favored the High Model Fidelity group. No analyses of disaggregation variables were possible because these student data were not provided. The results support the positive impact High Model Fidelity Implementation of the Payne School Model on increasing student achievement in elementary and middle schools in Mathematics and Reading. For more information please contact: Center for Study of Economic Diversity, PO Box 665, Highlands, TX 77562, 800-468 468-9950 (phone), 281-426 426-6930 6930 (fax). 2
Introduction and Purpose The federal No Child Left Behind Act (2001) and corresponding state legislation throughout the United States requires that schools use research based programs to increase student achievement in all academic areas, with particular emphasis on reading/english/language arts and mathematics. Consistent with these mandated foci on student achievement, Dr. Ruby K. Payne initiated research to determine the impact of the implementation of the Payne School Model A Framework for Understanding Poverty, Learning Structures, and Meeting Standards and Raising Test Scores materials and training on student achievement in the areas of reading/english/language arts and mathematics. Context for Cannon County Schools Cannon County Public Schools, located in Woodbury, Tennessee, serves 2,104 students from kindergarten to 12 th grade. There are six elementary schools in the district that serve pre-kindergarten to eighth-grade students and one high school that serves ninth- to 12 th - grade students. All seven schools in the district were involved with aha! Process, Inc. throughout the 2004 05 school year. All staff members received A Framework for Understanding Poverty, Learning Structures, and Meeting Standards and Raising Test Scores training from aha! Process consultants. During the course of the 2004 05 school year, technical assistance was provided quarterly by aha! Process consultants. Elementary (K 5) An aha! Process consultant provided four technical-assistance sessions to selected elementary teachers. Teachers met with the consultant by grade level. During the first semester, technical-assistance sessions focused on: Discussing how to effectively work with parents Completing student data grids Developing time and content grids Direct-teaching input strategies Identifying interventions for those students who are struggling, based on student data grids During the second semester, the aha! Process consultant conducted classroom observations and continued to meet with the teachers for technical-assistance sessions. These sessions focused on: Reviewing and revising time and content grids Completing a resource analysis on the four lowest students in the classroom, based on the student data grid Incorporating mental models into daily lesson plans Sharing strategies among teachers 3
Secondary Districtwide, all sixth- to eighth-grade students are served on one of the six elementary campuses, and all ninth- to 12 th -grade students are served at Cannon County High School. All sixth- to 12 th -grade teachers received content-specific technical assistance provided by aha! Process consultants. English/Language Arts Four technical-assistance sessions were provided to English/language arts teachers by aha! Process consultants, with each teacher receiving on average two days of technical assistance. During the first semester, technical-assistance sessions focused on: Reviewing concepts from Learning Structures Developing time and content grids Completing student data grids Developing common bell work to address targeted standards Identifying target students using the student data grid Introducing the Payne Lesson Design During the second semester, technical assistance focused on the following and also included classroom observations by an aha! Process consultant: Reviewing the Payne Lesson Design Developing common bell work Creating step sheets Aligning curriculum to standards Math An aha! Process consultant provided four technical-assistance sessions to math teachers, with each teacher receiving on average 1.5 days of technical assistance. During the first semester, the technical-assistance sessions focused on: Reviewing concepts from Learning Structures Developing time and content grids Completing student data grids Creating 10-question tests Identifying target students based on data grids During the second semester, the aha! Process consultant conducted classroom observations and continued to meet with the teachers for technical-assistance sessions. During the second semester, technical-assistance sessions focused on: Reviewing input strategies Creating 10-question tests Reviewing and revising time and content grids Designing a consistent plan and label strategy for math problem solving Completing a resource analysis of target students 4
Science An aha! Process consultant provided four technical-assistance sessions to science teachers, with each teacher receiving on average two days of technical assistance. During the first semester, the technical-assistance sessions focused on: Identifying specific strategies for planning, sorting, and scientific inquiry Aligning curriculum with standards Developing time and content grids During the second semester, the aha! Process consultant conducted classroom observations and continued to meet with the teachers for technical-assistance sessions. During the second semester, technical-assistance sessions focused on: Teachers success with strategies, specifically with mental models, reading strategies, and plan and label strategies Incorporating step sheets and question making into daily lessons Unpacking the standards Reviewing and revising time and content grids Developing strategies to improve students performance on benchmark assessments Social Studies An aha! Process consultant provided four technical-assistance sessions to social studies teachers, with each teacher receiving on average two days of technical assistance. During the first semester, the technical-assistance sessions focused on: Embedding instructional strategies from Learning Structures, particularly mental models and plan and label strategies Developing time and content grids During the second semester, the aha! Process consultant conducted classroom observations and continued to meet with the teachers for technical-assistance sessions. During the second semester, technical-assistance sessions focused on: Reviewing instructional strategies Reviewing and revising time and content grids Introducing question-making strategies Unpacking the standards Using the Payne Lesson Design Methodology The research design to determine impact had two dimensions. The first dimension was establishing Model Fidelity at each school. If teachers were not implementing the Payne School Model effectively, differences in student achievement could not be attributed to the use of the model. The second dimension was determining the statistical significance of the impact of the Payne School Model on student achievement in reading and mathematics. The design for this analysis was a post-test-only comparison design for two groups (High Model 5
Fidelity vs. Other) using the analyses of covariance to adjust for initial differences between the groups of students. Model Fidelity The Instructional Framework Scale Observation (2003) was used to assess the fidelity of the implementation of the Payne School Model. This instrument consisted of 47 indicators criterion-referenced to key model components/activities and was used by the aha! Process consultants to determine the fidelity of the implementation of the Payne School Model. The median inter-rater reliability for the instrument is.83, with a range from.72 to.95. Analysis of Student Achievement Data A post-test-only comparison design for two groups High Model Fidelity (50% or higher score on the Instructional Framework Scale Observation) vs. Other (49% or less on the Instructional Framework Scale Observation or not observed) using the analyses of covariance to adjust for initial differences between two groups was employed to determine the statistical impact of the implementation of the Payne School Model on student achievement. No comparison school district was available. The independent variable was the implementation or non-implementation of the Payne School Model (High Model Fidelity vs. Other). The expected level of first-year implementation is 50%, which was the criterion used to differentiate the two groups. The dependent variables were standardized test scores in the TCAP in Reading and Mathematics. (See the following website for more information on the Tennessee Comprehensive Assessment Program, Accountability, Assessment/Evaluation/ Research, and the 2005 Report Card: www.state.tn.us/education/.) The covariates were 2003 04 prior test scores for students in 2004 05 on the TCAP in Mathematics and Reading, respectively. Analyses were conducted for the total groups at each grade level. No disaggregation analyses could be conducted because demographic descriptors were not available. The level of statistical significance was set at p <.05. Results The results are presented in two sections Model Fidelity and Student Achievement. Model Fidelity aha! Process consultants used the Instructional Framework Scale Observation (2003) to observe teachers who were identified either by the building principal or the aha! Process consultant as potentially implementing the Payne School Model effectively. This resulted in from one to five teachers being observed in each of the five schools in Grades 2 8. The number of teachers per grade who met the criteria for High Model Fidelity group ranged from one to three; the remaining teachers who were observed and those teachers who were not observed ranged from six to nine, and they were placed in the Other group. Of the nine teachers who were in the High Model Fidelity group, six had a score of 100%, and one had a score of 98%. 6
Student Achievement Tables 1 and 2 (see final two pages) contain an overview summary of the statistical analyses for Mathematics and Reading by grade for Grades 2, 4, 5, 6, 7, and 8. The data for Grade 3 had only one student, and thus no analyses were conducted for this grade. No analyses on disaggregations were conducted because no data on gender, race/ethnicity, LEP (Limited English Proficiency), poverty, or IDEA (Individuals with Disabilities Education Act of 2004) disability were provided. A narrative explanation of the results is provided below for each analysis conducted. Mathematics The ANCOVA-adjusted means for Mathematics for the High Model Fidelity group were higher than for the Other group for four of the six grades (Grades 2, 5, 6, and 7). There was one statistically significant difference in the means for Grade 2 (F = 14.030, p <.000) in favor of the High Model Fidelity group. While the ANCOVA-adjusted means were not statistically significantly different for other grades, the directional difference was maintained in favor of the High Model Fidelity group. Reading The ANCOVA-adjusted means for Reading for the High Model Fidelity group were higher than for the Other Group for four of the grades (Grades 2, 5, 6, and 7). There were two statistically significant differences in the adjusted means one for Grade 2 (F = 24.554, p <.000) in favor of the High Fidelity Group and one for Grade 8 (F = 4.064, p <.046) in favor of the Other group. High School S The only data available for the high school were descriptive statistics for Math and Reading in 2005, with the following results: 2005 Mathematics: Number of Students = 62; Range = 19 51; Mean = 36.47; Standard Deviation = 7.764 2005 Reading: Number of Students = 183; Range = 0 55; Mean = 39.19; Standard Deviation = 9.021 The numbers of students are different because of incomplete data on all students. No analyses could be conducted because the high school test is a pass/fail test, and no other assessment was provided. While eighth-grade data could theoretically be used as a covariate, the time period between eighth grade and 11 th grade precludes a meaningful adjustment. Discussion Discussion of these results is constrained by several factors. First, this was the first year of implementation of the Payne School Model. While the Model Fidelity scores for some teachers significantly exceeded the 50% criterion of the first year, this score reflected performance during the middle/end of the Spring Semester and is not an index for performance throughout the year. Even higher performance would be anticipated in the second-year implementation. Second, no comparison schools were available to provide 7
direct comparisons to schools not even associated with the Payne School Model. Third, there were no data on demographic variables to assist in interpretation of impact of the Payne School Model on gender, race/ethnicity, LEP, poverty, or IDEA disability. The pattern of the ANCOVA-adjusted means of the High Model Fidelity group exceeding those of the Other group in eight of the 12 analyses is positive, especially for the first year of implementation. Three patterns deserve additional investigation, perhaps by focusing on the disaggregation variables the overall pattern across both Mathematics and Reading; Grade 2 and Grade 4; and Grade 4 and 8. The overall pattern of results in Mathematics and Reading in terms of directions of difference is the same; this is relatively rare and should be studied to see if it continues and why. The results for Grade 2 (in favor of the High Model Fidelity group) and for Grade 4 (in favor of the Other group) for both Mathematics and Reading merits further examination in subsequent years to determine if the pattern persists over time for these groups of students. And the results for Grade 4 and Grade 8 that favor the Other group should be examined in the coming year to see if this pattern recurs for these two years and why. While conducting ANCOVA analyses on the disaggregation variables and making comparisons to other schools would have increased the value and depth of these findings, these results strongly support the positive impact High Model Fidelity implementation of the Payne School Model on increasing student achievement in all six grades examined for both Mathematics and Reading except Grade 8 Reading. Recommendations Based on these results, two recommendations for continuing improvement should be considered regarding the examination of the impact of the Payne School Model on student academic achievement. 1. Demographic data must be provided, along with achievement data for each student, to determine differential impact on gender, race/ethnicity, LEP, poverty, and IDEA disability. Identifying the differential impact of the Payne School Model on these disaggregate groups is crucial to being able to generalize results. 2. Demographically similar schools that are not using the Payne School Model must be identified in order to more thoroughly examine the positive nature of the results from this study. 8
References Payne, Ruby K. (2005). A Framework for Understanding Poverty (Fourth Revised Edition). Highlands, TX: aha! Process. Payne, Ruby K. (2003). Instructional Framework Scale Observation. Highlands, TX: aha! Process. Retrieved from http://www.ahaprocess.com/school_programs/research_&_development/. Research design package analyses, reliability information for instruments. (2005). Highlands, TX: aha! Process. Retrieved from http://www.ahaprocess.com/school_programs/research_&_development/. Tennessee Department of Education. (2005). Tennessee Comprehensive Assessment Program, Accountability, Assessment/Evaluation/Research, 2005 Report Card. Retrieved from www.sate.tn.us/education/. 9
Table 1 Summary of Results for ANCOVA for Mathematics 2004 05 Cannon County Schools (Tennessee): High Model Fidelity (HMF) vs. Other Grade Group Sample Size Covariate ANCOVA Adjusted Means Direction of Difference F Probability 2 nd HMF 17 1 st Grade 612.541 HMF > Other 14.030 p <.000* Other 112 577.391 4 th HMF 16 3 rd Grade 470.959 HMF < Other 3.008 p <.085 Other 126 486.815 5 th HMF 13 4 th Grade 514.274 HMF > Other 1.940 p <.167 Other 90 503.827 6 th HMF 17 5 th Grade 532.909 HMF > Other 2.268 p <.134 Other 122 524.505 7 th HMF 32 6 th Grade 536.678 HMF > Other.080 p <.778 Other 101 535.320 8 th HMF 50 7 th Grade 550.204 HMF < Other.217 p <.642 Other 90 551.998 * p <.001 10
Table 2 Summary of Results for ANCOVA for Reading 2004 05 Cannon County Schools (Tennessee): High Model Fidelity (HMF) vs. Other Grade Group Sample Size Covariate ANCOVA Adjusted Means Direction of Difference F Probability 2 nd HMF 16 1 st Grade 655.790 HMF > Other 24.554 p <.000* Other 111 620.949 4 th HMF 16 3 rd Grade 476.642 HMF < Other 3.080 p <.081 Other 126 491.784 5 th HMF 13 4 th Grade 512.110 HMF > Other.432 p <.512 Other 90 508.540 6 th HMF 17 5 th Grade 525.933 HMF > Other 1.762 p <.187 Other 121 518.076 7 th HMF 32 6 th Grade 527.584 HMF > Other.043 p <.836 Other 100 526.813 8 th HMF 90 7 th Grade 535.236 HMF < Other 4.064 p <.046** Other 49 541.671 * p <.001 ** p <.05 11