Evaluation of the SIG1003(a) After-School Club Programs 2010-2011 Courtney C. Zmach, Ph.D. Grant Evaluator Federal and State Grants School District of Collier County September 27, 2011 Please contact Dr. Zmach at zmachc@collierschools for further information about this report.
Evaluation of the SIG1003(a) After-School Club Programs Background During the 2010-2011 school year, four Collier County Title I elementary schools received SIG 1003(a) funding to offer an after-school Club to provide extended learning opportunities to at-risk students. The objective of the after-school Club program was to provide intensive intervention to accelerate the learning of students in grades 3 5. Third grade students scoring in the 40 th percentile on the SAT-10 and fourth and fifth grade students scoring a Level 1 or 2 on the FCAT were eligible for the after-school Club. Schools were in various stages of development in utilizing after-school Clubs at their sites. For math support, all schools used America s Choice Math Navigator. For reading, two schools used American Reading Company s Research Labs, one school used Voyager Passport and the fourth school used CARS and STARs. Teachers received professional development during the school year. After meeting with principals at each of the four sites during fall 2010, it was clear that they are committed to provide the best possible program at their respective schools. Principals were interested to determine whether students attending the after-school club will experience greater success on the FCAT and whether scores will increase enough to help their school meet Adequately Yearly Progress (AYP). The focus of this report is on the overall results of programs during the 2010-2011 school year. Primary Evaluation Questions The purpose of this evaluation is to provide information about how conditions, such as program dosage measured via student attendance, affect student outcomes. The evaluation will address the following overarching questions: 1. Do students participating in the after-school Club demonstrate learning gains in reading and math during the study period? a. How does dosage, in terms of program dosage /attendance, affect achievement? 2. How was the after-school Club program implemented? a. What worked and what could be done differently next time? b. What are teachers perceptions of the quality and utility of the after-school Club? 3. Did the schools meet adequate yearly progress? Selected Assumptions about After-School Clubs In evaluating the after-school Clubs, there are certain considerations that should be appropriately recognized in the data collection and interpretation. Among these are the following: 1. After-school Clubs have existed for varying lengths of time and are at different stages of becoming fully implemented. 2. Students will receive two 45 minute sessions each day (Homework, plus reading or math); however, one site also provide up to two additional hours of intervention per week to provide science and writing support. 1 P a g e
3. Where possible, students receive after-school support from their school-day teacher or from a teacher at their grade level. 4. Some teachers teach during the after-school program two days a week, while others teach four days a week. All after-school teachers receive one hour of planning for every two days they teach. 5. All students receive a snack each day. 6. After-school Clubs received funds to pay salaries, purchase supplies and provide student transportation, if needed. 7. Students at the school sites were also eligible to receive supplemental educational services provided through Title I Basic. Methods Participants After-School Club Students At the start of the 2010-2011 school year, the prior achievement of students in grades 3, 4, and 5 was examined to identify students in need of extended learning opportunities. Third grade students were identified using one of two criteria: below the 40 th percentile on the grade 2 SAT-10 or third grade retention; fourth and fifth graders were identified as having a Level 1 or 2 on FCAT. A letter was sent home to parents extending an invitation for the identified students to participate in Club. Three programs operated Monday through Thursday from 2:50 to 4:30 and one operated from 2:50 to 5 pm. Students were expected to participate on all days the program was in session. Participants received a daily snack and transportation was available. A total of 388 students participated during the school year. After-School Club Staff The focus of the Club was providing additional academic support in reading, math, science, along with 45 minutes of daily homework help. Thirty-nine school staff served as teachers in the after-school club programs. Each program was managed by after-school coordinators with support as needed from the school s administration. Data Collection and Analysis Data collection took place during the 2010-2011 school year. The primary achievement outcome measures used in the analyses are FCAT reading and mathematics scores. Secondary measures include the assessment scores from Math Navigator and Florida Assessments for Instruction in Reading (FAIR). Schools also provided pre-post scores from the Math Navigator program used during the after-school club. Math Navigator growth from the pre-post test scores are analyzed and discussed in the results section of this report. During the 2010-2011, students in Florida complete FAIR testing between 2 to 3 times. As recommended by the Florida Department of Education and Florida Center for Reading Research, student growth using the FAIR reading comprehension abilities scores (RCAS) are also examined (Foorman & Greenberg, 2011). To learn more about the context under which the after-school Club was implemented, a brief teacher survey was given to all after-school teachers and coordinators and informal classroom walkthrough (CWT) visits were completed, both during spring 2011. Teachers were visited two times during the spring semester to gather contextual information. The visits were brief and non-evaluative in nature and the 2 P a g e
focus was on the following areas: classroom environment, focus on instruction, and student learning behaviors. The after-school coordinators also maintained and submitted after-school Club student attendance logs. The attendance log is the primary measure of program dosage and utilized in the FCAT outcome measure analyses. Adequately Yearly Progress (AYP) results were released in summer 2011. Results To understand the context of these results, it important to examine the attendance, or dosage, that students received this extended learning opportunity. According to the provided records, the length programs were offered ranged from 55 days to 111 days, with an average attendance length of 86 days. Students attended, on average, 57 days or 63% of the time. All programs had students who attended 4 days or less and also had students with near perfect attendance. Another important contextual feature is that the four sites did not make AYP at the end of 2010-2011. These results should be interpreted with caution for two primary reasons: 1) the small sample sizes and 2) the lack of standard setting between the FCAT 2.0 and the earlier FCAT assessment at the time of these analyses. Nevertheless, these FCAT assessment data can be used as a valuable point of reference when examining students' achievement. The FCAT math and reading achievement are reported in two ways at each grade level: Club Only students and All Students. FCAT Mathematics For FCAT Math achievement, nearly all grades (grades 3, 4, and 5) and years (FY10 and FY11) of Club students scored at a lower level than the school-wide average (see Table 1). Grade 4 and 5 students who participated in the Club appear to have made greater gains compared to the grade level results. This is illustrated by a review of the difference in developmental scale scores (DSS). This is important since students in the Club Only groups at grade 4 and 5 started the FY10 school year at lower DSS. For example, on average, at grade 4, the difference in DSS scores for Club students was +127 points between years, while the All Students group was +72 points. Club students in grade 3 achieved a slightly lower DSS score in FY11 as compared to the Cohort of grade 3 students; however, because of the potential narrowing of the gap in future years, the results suggest there may be an added value in students attending the After-School Club. FCAT Reading Next, FCAT reading is examined (see Table 1, bottom panel). All grades and years of Club students scored at a lower level than the school-wide average. Similar to the FCAT Math results, students attending the Club achieved a greater gain from one year to the next. In fact, according to analyses, the Club students in grade 4 achieved a gain of 247 points, while grade 4 students in the All School group gained 209 points. 3 P a g e
Table 1. FCAT Achievement: Club Students and All Students Mathematics Club Only Difference between FY10 All Students Difference between FY10 School FY10 FY11 and FY11 FY10 FY11 and FY11 Third Grade A -- 1191 -- -- 1429 -- B -- 1330 -- -- 1380 -- C -- 1138 -- -- 1227 -- D -- 1242 -- -- 1329 -- Average 1225 1341 Fourth Grade A 1220 1297 77 1357 1382 25 B 1201 1409 208 1383 1492 109 C 1206 1332 126 1324 1436 112 D 1455 1552 97 1476 1518 42 Average 1271 1398 127 971 1393 72 Fifth Grade A 1317 1354 37 1463 1494 31 B 1392 1390-2 1521 1511-10 C 1327 1411 84 1411 1483 72 D 1450 1552 102 1549 1624 75 Average 1372 1427 55 1486 1528 42 Difference Difference Reading Club Only between FY10 All Students between FY10 School FY10 FY11 and FY11 FY10 FY11 and FY11 Third Grade A -- 1061 -- -- 1299 -- B -- 1137 -- -- 1266 -- C -- 943 -- -- 1109 -- D -- 1179 -- -- 1299 -- Average 1080 1243 Fourth Grade A 1108 1380 272 1256 1458 202 B 1148 1426 278 1316 1530 214 C 911 1167 256 1129 1371 242 D 1269 1451 182 1364 1540 176 Average 1109 1356 247 1266 1475 209 Fifth Grade A 1254 1297 43 1450 1528 78 B 1414 1454 40 1536 1524-12 C 1249 1377 128 1371 1438 67 D 1479 1503 24 1559 1573 14 Average 1349 1408 59 1479 1516 37 Source: CCPS Data Warehouse 4 P a g e
FCAT Achievement by Attendance Group To better understand the impact of the attending the after-school club, the next analysis examined the achievement of students based on their attendance in the Club program. The after-school club was offered between 55 and 111 days. On average, students attended the Club for 58 days with an average attendance of 63% across the school year. There was variation in the time students attended the program, with some students entering late and others withdrawing from the program early. The duration that students attended ranged from 1 day to 111 days. There were 333 students with attendance above the mean and 191 with attendance below the mean. Table 2 displays the average DSS scores by attendance group using available FCAT data. In all cases, the group with the attendance above the mean had higher DSS scores. Table 2. FCAT DSS Average, Club Students by Attendance Group FCAT Math FCAT Reading Attendance Group FY10-DSS FY11-DSS FY10-DSS FY11-DSS Above Mean 1323 1377 1232 1282 Below Mean 1262 1282 1125 1244 Source: CCPS Data Warehouse and author s analysis Math Navigator Achievement Three sites provided pre- and post-test data from the Math Navigator program. The after-school clubs utilized a placement test to determine the unit that best meets the needs of the students. The grade levels and sites selected different units for their students, thus making it challenging to analyze data across grade levels or units. Two third-grade sites completed the Beginning Place Value unit and showed growth between 14 to 21%, while the other third-grade site completed the Understanding Addition and Subtraction showing 14% growth from pre-to-post. Fourth graders at the different sites completed different units. The site that completed Multiplication and Division showed the largest growth at 27%. Another site completed the unit on Place Value and showed 19% growth. Fourth-graders at the third site declined between 1 and 2% on the Addition and Subtraction Units. Two fifth-grade sites completed Place Value: From Decimals to Billions and showed between 23-24% growth from pre-to-post. Two fifth-grade sites also completed the Understanding Multiplication unit. Students demonstrated between 9 to 11% growth on this unit. A comparison with other groups of students (non-club students) was not available. FAIR Achievement During the 2010-2011, students in Florida complete FAIR testing between 2 to 3 times per school year. Two schools in the evaluation completed FAIR testing 3 times (or assessment periods [AP] during the year and the other two completed testing 2 times during the school year. Reading comprehension abilities scores (RCAS) were analyzed for students participating in the Club. Students at three sites started from a similar point. As Figure 1 shows, the average RCAS score for Club students increased over the three assessment periods (AP1 to AP2 to AP3). The graph also displays standard error bars, which show the variability of the scores by taking into account the sample size and the standard deviation for each site. 5 P a g e
Reading Comp Ability Score 350 340 330 320 310 300 290 A B C D 280 270 AP1 AP2 AP3 Figure 1. Reading Comprehension Ability Scores, Average by Site Classroom Walkthroughs In spring 2011, two rounds of classroom walkthroughs (CWT) were conducted at three sites using an evaluator-modified CWT form. 1 During each round, visits were made to all content area teachers classrooms. In all, 39 classrooms were visited (12 Reading, 17 Math, 6 Homework Help, 4 Other (technology, writing, science, or vocabulary). Visits were approximately 3 to 5 minutes in duration. The physical classroom arrangements were conducive for 36 out of 39 visited spaces, while in three cases, the evaluator was unable to make such a determination. Students in most classes were found to be either working as a whole group or students were working on their own. The distribution of classroom grouping formats is displayed in figure 2. 1 Classroom Walkthroughs did not take place at one site because CWTs began the same week that the after-school program ended. The letters (A, B, and C) do not necessarily correspond with the letters (A, B, C, D) used in previous analyses. 6 P a g e
8 7 6 5 4 3 2 1 0 Whole Group Small Group w/teacher Learning Centers Paired Individual Other A B C Figure 2. Club Grouping Formats In 34 out of 39 classrooms, it appeared that teachers had the needed instructional materials available. In 5 classes, it was not possible to make such a determination. Math Navigator materials and Overhead/ board/chart paper were the most frequently used materials. Figure 3 displays the materials used during the CWTs. 8 7 6 5 4 3 2 1 0 A B C Figure 3. Club Identified Instructional Materials 7 P a g e
A variety of instructional practices were observed during the CWTs. In 80% of classrooms (31 out of 39), teachers provided students with opportunities for practice. In 49% of classrooms, teachers were observed coaching or modeling for students. Teachers were also observed providing students with adequate amounts of think (or wait) time before answering in 74% of classes, while in the balance of classes, it was not possible to make a determination. Teachers in 23 out of 39 classes used questioning to extend student responses to given questions/tasks. Figure 4 displays the range of available instructional practices. 14 12 10 8 6 4 2 0 A B C Figure 4. Club Identified Instructional Practices CWTs were brief in duration; therefore, it was challenging to determine the level of cognitive complexity in 10 classes. Seven classes were deemed to have a High level of complexity (analyze, judge, justify, create), 11 had a Moderate level of complexity (demonstrate, summarize, compare) and 11 classes had a Low level of complexity (define, recall, identify, locate, paraphrase). Students were exceptionally well behaved during the CWTs. It was evident during all interactions with staff and from CWTs that all Club staff (100%) hold high expectations for student behavior. In terms of classroom/student engagement, 16 classes were judged to have active levels of engagement with authentic and student-led activities, 22 were observed to have compliant levels of classroom/student engagement and for one class, it was not possible to determine with certainty. Students in nearly all classrooms engaged in appropriate interactions with the teachers (38 out of 39) and with their peers (35 out of 39). As a credit to the Club staff and students, all students displayed a clear knowledge of the After-School Club routines. 8 P a g e
After-School Club Survey After-School Club staff completed an evaluator-created survey. The purpose of the survey was for CCPS and schools to understand more about implementing After-School Clubs. Thirty-eight Club staff (34 teachers; 4 coordinators) responded to the survey and their responses are summarized. The After-School Club teachers performed a variety of roles during the school day, including classroom teacher, general secretary, reading coach, pool teacher K-5, technology teacher and ELL resource. Of these staff, 23 reported that they volunteered to work for their Club program, while 12 were asked if they wanted to work for the Club. Of the Club staff who perform an instructional role during the regular school day, there was variation in the length of time they taught. Six staff reported they are in their first year of teaching, 15 have taught 2 and 8 years, and 11 have taught between 9 and 15+ years. Thirty-two teachers reported having experience teaching in an after-school program. Eight teachers reported teaching 2 to 4 in an after-school club, seven teachers reported teaching 5 to 8 years in an after-school club and six teachers reported teaching in an afterschool club for 9 to 14 years. No teachers have worked in an after-school club for 15+ years. The After-School Club staff were asked to rate their satisfaction with various aspects of implementation, ranging from support from school staff to the daily number of minutes provided by the program. The overall trend is clear--of the teachers who responded to the survey, the majority are satisfied in all areas (see table 3). Such areas of strengths include: the support that teachers perceive from the school leaders and teachers, how students are identified for the After-School Club, progress students made during the program the daily number of minutes and the duration of the program across the year. The responses were slightly mixed in the following area: support from parents. Seventy percent of teachers reported being satisfied or very satisfied with parents, while nearly one-third were dissatisfied. Table 3. Satisfaction with the Implementation of the After-School Club Support from your principal and other school leaders Support from other teachers Support from parents Very Satisfied Satisfied Dissatisfied Very Dissatisfied No response 22 11 2 0 0 18 15 2 0 0 5 18 10 0 0 How students are identified for the After-School Club 12 20 3 0 0 The progress that students made during the After-School Club 12 16 5 1 1 Daily number of minutes of the program 16 17 2 0 0 Duration of program across the school year 16 14 4 1 0 Source: After-School Club teacher survey Teachers also reported on the challenges they face in their After-School Club classrooms (see table 4). Responses were mixed. Student behavior problems were perceived to be the greatest challenge. Students lacking the necessary knowledge or skills to learn what teachers were trying to teach and 9 P a g e
students being bored were also perceived challenges. Most teachers found the math and reading materials to be an appropriate difficulty level for students. Table 4. Reported Challenges in the After-School Club Classrooms Not a A minor A moderate A serious No challenge challenge challenge challenge response Student absenteeism 11 15 7 1 1 Student behavior problems 7 9 9 10 0 Students not motivated or engaged (bored) Student lacked the knowledge/skills to learn what you were trying to teach Reading materials are too difficult for students Reading materials are too easy for students Math Navigator materials are too difficult for students Math Navigator materials are too easy for students Source: After-School Club teacher survey 11 8 8 7 1 9 11 10 4 1 17 8 5 2 3 28 3 1 0 3 17 8 3 2 5 20 8 2 0 5 Teachers at the different sites received professional development. Such trainings were provided at the school sites by math or reading program consultants and by the after-school coordinators. The survey asked Club teachers to think about their own professional development needs. Twelve teachers responded. Top suggestions included additional professional development or training in math, more time to collaborate and plan with colleagues, and time management/lesson planning strategies. The final survey question asked respondents to offer the Top Three suggestions you would you offer to your school or other schools planning an After-School Club. The following list include aspects of the after-school club that teachers noted they would continue: continue to tie our after-school curriculum to the regular curriculum have great communication with all stakeholders have motivated teachers on board build in planning time for teachers have a behavior system in place in the after-school program Teachers also offered suggestions for future improvements: train the teachers before the after-school program begins identify students at the end of the previous school year hands-on activities and field trips to build background knowledge more extracurricular options that focus outside the core content areas as well as social and communication skills 10 P a g e
increase the duration of the program (five days a week and throughout the school year) more time and emphasis on homework better quality of homework that is consistent across all grade levels Recommendations Students who participated in the After-School Clubs are fragile learners scoring at or below the 40 th percentile on the SAT-10, retained third graders, or scored at Level 1 or 2 on the FCAT. The students who participated in the Clubs demonstrated gains from FY10 to FY11 that might not have occurred had it not been for the after-school clubs. While these data cannot confirm whether the after-school program was the cause, the intensity of the intervention would be a likely contender. After completing this review of data, it is clear that Club staff and students are invested the after-school clubs. The student attendance was consistently high throughout the school year. If future funding is available, data support the continuation of an after-school program. Suggestions for continued programs include: Continuous professional development. While Club staff may be experienced after-school club instructors, continuous improvement is necessary for growth. Teachers suggested additional math training and motivation and behavior management training. Review Club policies for admitting late entry students. While it is recommended to include late entering students who demonstrate need, it may detract from the intent to include students who are higher achieving. Review school (and/or grade level) homework policies. Teachers recommended that homework be of an increased quality and consistent across grade levels. While a change may not be necessary, a review would be worthwhile. Support from parents. The school may want to examine ways to increase the perceived lack of support from parents. Utilize a computer-based program to maintain attendance records. The CCPS created program Student Pass could be utilized for such a purpose. Reference Foorman, B. & Greenberg, S. (2011, February 2). Florida Assessments for Instruction in Reading (FAIR) Using Scores for Growth and Instruction. Presented at the 2011 Florida Educational Technology Conference (FETC) Annual Meeting, Orlando, FL. 11 P a g e