Department of Accountability and Academic Outcomes. Evaluation Report Third Grade Academy,

Transcription

1 Department of Accountability and Academic Outcomes Evaluation Report Third Grade Academy, Report No September 2010

2 EVALUATION REPORT OF THIRD GRADE ACADEMY, Charleston County School District Dr. Nancy J. McGinley Superintendent Janet S. Rose, Ph.D. Executive Director Department of Accountability and Academic Outcomes Laura F. Donnelly, Ph.D. Program Evaluator Department of Accountability and Academic Outcomes September 2010 Report No

3 Executive Summary During , the CCSD Third Grade Academy (TGA) program provided enhanced academic assistance and support for students who needed additional assistance, especially in reading. Program Features Eleven sites, with 19 classes, serving 220 students Students got a head-start through the TGA Quick Start 2009 summer program. Small class size Extended day (4 hours/week) Voyager intervention daily Reading and Running with the RiverDogs incentive program Achievement Results during Results for MAP reading were positive. o Percent of TGA students making substantial growth increased by 18.5 percentage points compared to 7.7 percentage points for a comparison group of non-tga students. o TGA was most effective for low-performing students (e.g., for students in the lowest quartile in fall 2009, TGA students gained 16.9 RIT points vs points for non-tga). Results for MAP math were positive. o Percent of TGA students making substantial growth increased by 17.8 percentage points compared to a decrease of 0.1 percentage points for non-tga. o TGA was effective at all fall MAP levels, at least to the 75 th percentile (e.g., both TGA and non-tga students made gains but TGA gains tended to be 2-3 RIT points greater). Results of a tentative PASS analysis for lowest-quartile students were mixed. o ELA: 38% of TGA students scored at Met/Exemplary vs. 44% of non-tga. o Writing: 32% of TGA students scored at Met/Exemplary vs. non-tga; 9% of TGA scored at Not Met 1 vs. 19% of non-tga. o Math: 28% of TGA students scored at Met/Exemplary vs. 16% non-tga. Attendance and Behavior Attendance: TGA students were more likely to have no absences (15% TGA vs. 9% non-tga); non-tga students were more likely to have 1-5 absences (45% non-tga vs. 37% TGA). Suspensions: suspension rates for TGA and non-tga students were not notably different (out-of-school suspension: 12% TGA vs. 9% non-tga; in-school suspension: 1% each). Future of the TGA Program: Twelve sites will serve 360 students. Board policy will strongly encourage student participation. Student selection process will ensure that Tier II students are selected. Addition of Associate Teacher at each TGA site will support small group and one-on-one instruction. Rich professional development offerings for TGA teachers will increase instructional effectiveness and deepen content knowledge in all areas. The reading and exercise incentive program, Reading and Running with the RiverDogs, will be a vital component of the program during i

4 During , about 220 third-grade students were selected to participate in the Third Grade Academy (TGA) that operated within 11 CCSD schools: Burns, Dunston, Frierson, Hursey, James Simons, Mary Ford, Midland Park, Mitchell, North Charleston ES, Sanders-Clyde. (Note: an 11 th site, Goodwin, operated as an independent TGA site). Students were selected for the program if they needed additional assistance in reading and math. These students got a headstart in the program by attending the TGA Quick Start summer program that operated for 12 days during the summer of During the summer program, students were placed with the teachers whose classes they would be in for the school year in order to build relationships with their teachers and classmates prior to the beginning of the school year. Elements of the yearlong TGA program included small class size, extended day learning (twice per week for a total of four additional hours per week), daily supplemental instruction in ELA and math through the use of the Voyager intervention program, and a reading and exercise incentive program called Reading and Running with the RiverDogs. This report evaluates the progress of TGA students by comparing them to a group of similar students. The report concludes with information about how the TGA program is being modified during Non-TGA Comparison Group As a means of gauging TGA effectiveness, progress made by TGA students during is compared to that made by a group of third-graders attending Title 1 schools in CCSD (all Title 1 schools, including those with TGA programs, but excluding the TGA participants). These schools are used for comparison purposes because their students provide the closest match for participating students. The students are fairly similar in terms of demographics as well as participation in the free/reduced meals program (although the TGA students are somewhat more likely to be Black and participating in the free/reduced meal program). Table 1 provides student information about the 214 TGA and 1,494 comparison-group students. This report assesses the change in performance from (as 2 nd graders; the year prior to TGA implementation) to (as 3 rd graders) on MAP. Because PASS begins in third grade, no prior year PASS data are available, but PASS results for both groups, as well as their attendance and behavior during , are also compared. The non-tga comparison group comprises 3 rd grade students at 31 Title 1 schools. Table 1: Demographic Data on Third Grade Academy (TGA) Students Compared to Other Third Graders in Title 1 Schools (Non-TGA) during TGA Students Non-TGA Students Number of students 220 1,494 Male % Race/Ethnicity Black 86.4% 71.8% White 1.9% 14.5% Other 11.8% 13.7% Special education 11.7% 12.2% Free/reduced meals 97.7% 88.3% Overage (1 or more years) 20.6% 14.2% 1

5 Achievement Results During MAP Results Table 2 provides data on student growth in MAP reading for both groups of students, comparing their growth in to their growth during , when the TGA students were third graders in the TGA program and the non-tga students were other third graders at Title 1 schools. Table 2 indicates that TGA students were much more likely to experience Substantially High Growth 1 in than they had in (increased from 24% in to 43% in ). The difference between the two years for the non-tga students was notably less (increased from 27% to 34%) Table 2: Substantially High Growth on MAP Reading, Compared to for TGA and Non-TGA Students Difference TGA 24.3% 42.8% points Non-TGA 26.6% 34.3% points The following charts provide detailed results for TGA and non-tga students in MAP reading for (as second-graders) compared to (as third graders). TGA: Reading MAP Growth (VCG based results for 173 students, TGA vs. as 2nd graders) Non TGA: Reading MAP Growth (VCG based results for 982 students, TGA vs. as 2nd graders) % 42.8% % 34.3% % 39.3% 41.4% 39.9% % % (2nd) (3rd) (2nd) (3rd) Substantially High Typical Substantially Low Substantially High Typical Substantially Low The above charts for MAP reading illustrate positive results for TGA. For TGA students, there was a large reduction in the Substantially Low Growth category, with a large increase in the 1 The categories of Substantially High Growth, Typical Growth, and Substantially Low Growth reflect the amount of growth made by each student on MAP from fall to spring in comparison to the amount of growth made by a Virtual Comparison Group (VCG) of 51 students from other districts who are specifically matched on factors such as beginning RIT level, testing dates, and demographics of the school. The category of Substantially Low Growth corresponds to z-scores below Typical Growth reflects z-scores ranging between Substantially High Growth is used for z-scores greater than

6 Substantially High Growth category. While the non-tga students also improved across the two years, the differences were much less dramatic. Table 3 (and Table 5 in the following section) provides three pieces of information about TGA and non-tga student performance on MAP during the year, both overall and by their fall 2009 reading (or math) performance (the quartile corresponding to their national percentile rank): (1) number of students tested; (2) average change in reading (or math) RIT from fall 2009 to spring 2010; (3) percent of students who IMPROVED their national percentile in reading (or math) from fall 2009 to spring 2010 (by at least one percentile point). Table 3: Change in MAP Reading Scores for TGA and Non-TGA Students from Fall 2009 to Spring 2010 (1) Number of (2) Average Students with Change in Reading Fall & Spring RIT, Fall to Spring Reading Scores (RIT Points) (3) Percent Who Increased Reading Percentile from Fall to Spring TGA Non-TGA TGA Non-TGA TGA Non-TGA Fall 2009 Reading Quartile Quartile 1 (1-25 th percentile) % 67.9% Quartile 2 (26-50 th percentile) % 54.7% Quartile 3 (51-75 th percentile) % 53. Quartile 4 (76-99 th percentile) N/A N/A 29.9% All Students 208 1, % 56.3% Table 3 shows that, overall, TGA students made greater gains in reading. It also shows that for both TGA and non-tga students, students who score at lower levels in the fall tend to make greater reading gains. Most importantly, this trend is especially strong for the TGA students. TGA students in the bottom 25 th percentile gained an average of 17 RIT points in MAP reading (compared to a 13-point gain for non-tga students) and over four-fifths increased their percentile rank (compared to about two-thirds of the non-tga students). TGA students scoring within the 26 th - 50 th percentile gained 12 points (compared to about 10 points for non-tga students), with over two-thirds increasing their percentile rank (compared to just over half of the non-tga students). At the 51 st - 75 th percentile level, TGA and non-tga students made comparable RIT gains, although non-tga students were somewhat more likely to increase their percentile rank. In sum, it appears that the TGA program is more beneficial for lowerperforming readers. 3

7 The following chart shows the variation in student growth on MAP reading for TGA students depending on their reading teacher(s). Each bar represents students having the same reading teacher (bars are labeled with a teacher number, from T1-T19, and the number of students in the reading class). TGA Reading MAP Growth during by Teacher 9 7 Tchr (# ss) 54% 31% 17% 15% 17% T1 (13) 67% T2 9% 64% 27% T3 (11) T4 (10) 75% 25% T5 23% 38% 38% T6 (13) 38% 31% 31% T7 (16) 27% 13% T8 (10) T9 (15) 36% 36% 29% T10 (14) T11 (5) 13% 38% T12 (8) 55% 45% T13 (11) 67% 33% T14 (15) 45% 36% 18% T15 (11) 33% 25% 42% T16 38% 67% 13% T17 (8) 33% T18 42% 33% 25% T19 Substantially Low Growth Typical Growth Substantially High Growth In math, Table 4 indicates that TGA students were much more likely to experience Substantially High Growth in than they had in (increased from 28% in to 46% in ). In contrast, there was no change in the likelihood of making Substantially High Growth for non-tga students (39% in both years). Table 4: Substantially High Growth on MAP Math, Compared to for TGA and Non-TGA Students Difference TGA 28.2% points Non-TGA 38.8% 38.7% points 4

8 The following charts provide detailed results for TGA and non-tga students in MAP math for (as second-graders) compared to (as third graders). TGA: Math MAP Growth (VCG based results for 174 students, TGA vs. as 2nd graders) Non TGA: Math MAP Growth (VCG based results for 983 students, TGA vs. as 2nd graders) % % 38.7% 47.7% % 37.5% 24.1% % 23.8% (2nd) (3rd) (2nd) (3rd) Substantially High Typical Substantially Low Substantially High Typical Substantially Low The above charts for MAP math illustrate positive results for TGA. For TGA students, there was a large increase in the Substantially High Growth category, along with a similar-sized reduction in the Typical Growth category. In contrast, the non-tga distribution experienced very little change across the two years. Table 5: Change in MAP Math Scores for TGA and Non-TGA Students from Fall 2009 to Spring 2010 (1) Number of (2) Average Students with Change in Math Fall & Spring RIT, Fall to Spring Math Scores (RIT Points) (3) Percent Who Increased Math Percentile from Fall to Spring TGA Non-TGA TGA Non-TGA TGA Non-TGA Fall 2009 Math Quartile Quartile 1 (1-25 th percentile) % 68.1% Quartile 2 (26-50 th percentile) % Quartile 3 (51-75 th percentile) % 56.6% Quartile 4 (76-99 th percentile) N/A N/A 43.2% All Students 214 1, % 60.2% The math results shown in Table 5 point to a fairly uniform benefit for TGA students across fall performance levels, at least up to the 75 th percentile. Non-TGA students made gains at all performance levels as well, although their gains tended to be 2-3 RIT points less. In addition, the likelihood of students increasing their math percentile rank was somewhat greater for TGA students at Quartile 1, 2, and 3. Thus, the TGA program is associated with substantial levels of MAP math growth. 5

9 The following chart shows the variation in student growth on MAP math for TGA students depending on their math teacher(s). Each bar represents students having the same math teacher (bars are labeled with a teacher number, from T1-T19, and the number of students in the math class). TGA Math MAP Growth during by Teacher % 42% 45% 31% 44% 29% 53% 33% 63% 42% 25% Tchr (# ss) 31% 38% T1 (13) 8% T2 36% 18% T3 (11) T4 (10) 75% 8% 17% T5 46% 25% 23% T6 (13) 43% 31% 29% T7 (16) T8 (10) T9 (15) T10 (14) T11 (5) T12 (8) T13 (11) 27% T14 (15) 73% 42% 27% 25% 38% T15 (11) T16 T17 (8) 8% T18 33% 42% T19 Substantially Low Growth Typical Growth Substantially High Growth PASS Results PASS testing begins in third grade, so changes in PASS cannot be evaluated for the TGA program participants. However, because the TGA program is targeted for students who struggle in reading (e.g., score below the 25 th percentile), a tentative PASS comparison can be made using a subset of TGA and non-tga students comprised of students who scored at or below the 25 th percentile on the relevant MAP test. That is, the 2010 PASS reading performance of TGA students versus non-tga students can be compared for the subset of students who scored in the bottom quartile of the spring 2010 MAP reading assessment. The spring 2010 MAP reading assessment can be used to determine the bottom quartile for the writing PASS analysis as well. The same approach can be used for the 2010 PASS math performance, restricting the analysis to those TGA and non-tga students who scored in the bottom quartile of the spring 2010 MAP math assessment. 6

10 TGA vs. non TGA Students: 2010 ELA PASS Results for Students Scoring in Lowest Quartile on Fall 2009 MAP Reading TGA vs. non TGA Students: 2010 Writing PASS Results for Students Scoring in Lowest Quartile on Fall 2009 MAP Reading % 5.6% % % 4.2% 30.2% 25.8% 50.9% 42.2% 59.7% 51.4% 11.3% 13.6% TGA (n=159) Non TGA (n=469) Exemplary Met Not Met 2 Not Met % 8.8% TGA (n=159) Non TGA (n=473) Exemplary Met Not Met 2 Not Met 1 9 TGA vs. non TGA Students: 2010 Math PASS Results for Students Scoring in Lowest Quartile on Fall 2009 MAP Math 4.6% 1.8% 13.9% 22.9% % 43.4% 34.4% 40.9% TGA (n=131) Non TGA (n=396) Exemplary Met Not Met 2 Not Met 1 The above charts, which provide results only for the lowest-scoring students as of fall 2009 MAP testing, show only small differences between the two groups in terms of their PASS scores. In ELA, TGA students were somewhat more likely to score at Not Met, especially Not Met 2. In writing, TGA students were about equally likely as non-tga students to score Not Met, although within this category, TGA students were more likely than non-tga students to score Not Met 2 rather than the lower Not Met 1 level. The data do suggest a possible TGA advantage in the area of math, where about 28% of first-quartile TGA students scored Met or Exemplary compared to 16% of comparable non-tga students. 7

11 Attendance and Behavior Table 6: Attendance and Behavior in for TGA and Non-TGA Students Attendance TGA Students Non-TGA Students No absences 15.4% 9.1% 1-5 absences 36.9% 44.7% 6-9 absences 20.6% 22.6% 10+ absences 27.1% 23.6% Out-of-school suspensions TGA Students Non-TGA Students No out-of-school suspensions 88.3% 90.6% 1 out-of-school suspension 6.1% 6.8% 2 out-of-school suspensions 3.3% 1.3% 3+ out-of-school suspensions 2.3% 1.4% In-school suspensions TGA Students Non-TGA Students No in-school suspensions 99.1% in-school suspension 0.5% 0.7% 2 in-school suspensions 0.5% 0.1% 3+ in-school suspensions % Review of data relating to attendance and behavior during as displayed in Table 6 reveals the following: Attendance: TGA students were somewhat more likely to have had no absences (15% for TGA vs. 9% for non-tga). Out-of-school suspensions: TGA and non-tga students received out-of-school suspensions at about the same rate (12% for TGA vs. 9% for non-tga). In-school suspensions: TGA students and non-tga students received in-school suspensions at about the same rate (about 1% for both TGA and non-tga). Summary and Going Forward Conclusions Reading/ELA. Considering change in performance from to for MAP, results for TGA students were more favorable than those for non-tga students, especially for students who scored below the 50 th percentile in the fall. A tentative analysis of PASS ELA and writing results for the lowest-performing readers revealed little difference between the TGA and non-tga students. Overall, the results point to favorable results for TGA students in the area of reading. Math. In math, TGA students were much more likely to make Substantially High Growth in MAP in compared to , while there was very little change across the two years for the non-tga students. A tentative analysis of PASS math results for the lowest performers found that TGA students were more likely than non-tga students to score Met or Exemplary. Together, the MAP and PASS results suggest that the TGA program was beneficial to students in the area of math. 8

12 Future of the TGA Program The TGA program was developed to meet the needs of struggling learners during their third grade year through interventions in reading and math in addition to the regular third grade curriculum. The progress made by TGA students during is promising. After one year of TGA implementation, CCSD is expanding the academy model from 11 sites to 12 sites (Frierson will not have a TGA program in ; two new program sites Chicora and Memminger will be added). A total of 24 classes will operate at 12 sites during : Burns, Chicora, Dunston, Goodwin, Hursey, Mary Ford, North Charleston Elementary, James Simons, Memminger, Mitchell, Pinehurst, and Sanders-Clyde. While some invited students failed to participate in the TGA program during , a new Board policy will ensure that students who need the TGA intervention actually get it (in January 2010, the CCSD Board of Trustees adopted a policy specifying that students who are eligible for but choose not to participate in an intervention pathway cannot be promoted). Several of the primary features of the TGA program are being continued during , including that students are taught by highly effective teachers who were specially chosen for the program. The student-teacher ratio is remaining at 15 to 1, and the Voyager intervention program in reading and math will be used. It was determined that the following areas should be carefully examined to guide the expansion efforts: student selection, additional program support, professional development, and strong implementation of the incentive component. Student Selection. MAP data from clearly point to the strongest results in TGA for those students who enter third grade reading substantially below average, highlighting the importance of targeted student selection. To ensure that TGA serves Tier II students who are the best fit for the program, those students whose winter and spring second-grade MAP scores fell between the 10 th - 25 th percentiles were initially selected for the program. Review teams consisting of the principal, second and third grade teachers, ESOL and special education teachers confirmed student eligibility, using DIBELS data if necessary to validate students placement in or exclusion from the program. To the extent that spaces remained in the program after the Tier II students were selected, Tier III students (those falling between the 1 st 9 th percentiles) were also included. Additional Program Support. The major difference between the TGA program and the program is the introduction of an Associate Teacher at each site to support the TGA teachers and students. This position was created specifically for TGA to provide for the extension of the school day without actually adding time to the regular schedule by providing another adult to support small group and one-on-one instruction. Associate Teachers were screened first by teams of teachers and administrators and then interviewed by the TGA principals and teachers. Each team ultimately selected the candidate who provided the best fit for their school. These individuals are primarily responsible for providing direct explicit instruction using Voyager Passport and Vmath, intervention programs specifically designed to be administered to small groups of students. Additionally, Associate Teachers are responsible for maintaining the data management system and meeting with the two TGA teachers to plan daily for data-driven instruction. 9

13 Professional Development. Because TGA teachers will not be teaching for extended days twice a week during , they will be given the opportunity to participate in some rich professional development offerings. These offerings have been designed to increase instructional effectiveness and deepen the content knowledge of TGA teachers in all areas, including science and social studies. TGA teacher days are demanding because they are responsible for accelerating student achievement by effectively engaging students in the interventions while also teaching the regular third grade curriculum. Providing TGA teachers with opportunities to collaborate professionally and deepen their knowledge is essential for the success of the program. Strong Implementation of Incentive Program. Finally, the program would not be complete without the incentive component, Reading and Running with the RiverDogs, which began during Students earn points for reading and exercising beyond the school day. They are able to earn extra points by performing acts of kindness for others that require them to be physically active (e.g., raking leaves for an elderly neighbor or roller skating with younger siblings while dinner is being prepared). The children are frequently visited by Charlie the RiverDog, who dances and exercises with them, as well as RiverDog staff members who read to them. The year will culminate with an invitation for all students and family members to attend an evening RiverDogs game and participate in some of the pre-game activities. It is anticipated that the improvements outlined above will produce even greater benefits from the TGA program during

14 Accountability & Academic Outcomes: Report User Feedback Title of Report: Evaluation Report: Third Grade Academy, Person(s) Preparing Report: Laura Donnelly Date of Report: September 2010 After you have had an opportunity to use or review this report, please take a moment to provide your feedback. Your comments will help ensure future reports are relevant and useful to you, the user. Please return to Janet Rose by fax ( ) or courier (Accountability & Academic Outcomes, 75 Calhoun St.) Please select the one category which best represents your role in CCSD: Principal/Asst. Principal Senior District Leadership Parent/Community Member Teacher/Teacher Coach Other District Staff School Board Member Other School Based Staff Other: Please rate each of the following aspects of the report using a 5 point scale, with 1 as Unsatisfactory and 5 as Excellent. (If not applicable, please put N/A) Please rate the following aspects of the report: (or N/A) Content Format / Ease of use Amount of Information Value to You How often do you think you will refer to the report during the year? More than 5 times 2 5 times Only once Never How will/do you use this report? What was missing/not adequately covered? How can we improve this report? Any other comments?