Project Excel Interim Evaluation Report Year 3 (2001-2002)

Project Excel Interim Evaluation Report Year 3 (2001-2002) January 2003 Fairfax County Public Schools Office of Program Evaluation

FAIRFAX COUNTY PUBLIC SCHOOLS Daniel A. Domenech, Ph.D. Superintendent of Schools Department of Educational Accountability Michael E. Glascoe Assistant Superintendent Office of Program Evaluation Recardo V. Sockwell, Ph.D., Coordinator Michelle Perlstein Serafin, M.Ed., Principal Evaluator

PROJECT EXCEL INTERIM EVALUATION REPORT: YEAR 3 (2001-2002) EXECUTIVE SUMMARY Introduction This interim evaluation report describes the academic progress made by students at twenty schools participating in the Project Excel initiative in Fairfax County Public Schools (FCPS). A four-year evaluation began in the middle of the project s first year, January 2000, in response to a request from the Superintendent. Focusing on student achievement, this report uses test data collected in the third year of Project Excel, the 2001-2002 school year. A final report is planned in fall 2003. Across the four years, the evaluation reports show the grade-by-grade academic performance of students, as well as track the performance of the first cohort of full-day kindergarten students through the grade 3 Standards of Learning (SOL) tests. Project Excel was created in spring 1999 as one of the school division s major initiatives promoting high student achievement for all children. More specifically, Project Excel was to address a major concern that the selected schools would not likely reach state accreditation standards, unless additional resources and incentives were instituted by the division. In 1999 Excel schools pass rates on most of the SOL tests were below the 70 percent level required by the state for accreditation beginning in 2006. The twenty participating elementary schools enrolled the highest percentages of students receiving free or reduced-price meals (FRM) and students receiving English for speakers of other languages (ESOL) services students who had performed as a group lower than students at other FCPS schools on past measures of academic achievement. The resources, opportunities, and expectations provided to schools through Project Excel are consistent with current divisionwide Targets 1 and 2 which state All students will be reading at grade level by the end of second grade and All schools will meet or exceed the Virginia Standards of Accreditation, as measured by the SOL tests in mathematics, science, English, and history and social science. The initiative has three main program components accountability, increased learning time, and an enhanced academic program. Accountability The primary accountability for Excel schools is the expectation that they meet state accreditation standards, which are based on the SOL tests. In addition, Excel schools are expected to show annual progress on the Schoolwide Achievement Index (SAI), an index developed by FCPS that combines SOL and Stanford Achievement Test results for grades 3-6. Based on year-to-year change on the SAI, Excel schools have an opportunity to earn financial awards. For FY 2002, the cost of SAI bonuses was $2.1 million. Schools that do not make progress by the end of the four-year evaluation period may face a variety of consequences to promote greater school success. Such consequences ultimately will be determined by the Superintendent, but reflective of the views of school, cluster and division staff, and researched best practices. ES-1

Increased Learning Time Excel schools were able to increase the amount of time available for student learning in two ways. First, in all Excel schools, kindergarten students are in school for a full day instead of half a day. Second, Excel schools provide students at all grade levels with more time for learning than is typical in other Fairfax County elementary schools. In most Excel schools, the increase in learning time was accomplished by eliminating Monday early closing, adding 2.5 hours to the school day on Mondays. For FY 2002, the costs of full-day kindergarten and extending the school day were $5.1 million and $4.3 million, respectively (School Board s Approved Budget, FY 2003). Enhanced Academic Programs All Excel schools were able to strengthen their academic programs by implementing technologybased phonics in kindergarten and first grade and an instructional model in one or more grades, K-6. To implement technology-based phonics, Excel schools could choose one of two commercially available integrated learning systems aimed at improving early reading skills: the Computer Curriculum Corporation (CCC) SuccessMaker program or the Waterford Early Reading Program (Waterford). Prior to implementation, both learning systems were investigated by the Instructional Services Department and deemed appropriate for use at Project Excel schools. For FY 2002, the cost of instructional models was $1.6 million. This cost covers materials, training, and additional staffing needed to implement the models. Purpose of the Evaluation, Evaluation Questions, and Evaluation Design The purpose of the evaluation is to determine the success of Project Excel as a whole, and to increase the likelihood of its success, by reporting progress data at regular intervals to program staff, the Superintendent, and the School Board throughout the evaluation s duration. At the request of the Superintendent, the evaluation focuses exclusively on student performance on local and state tests. The evaluation questions are of three types: descriptive questions about the extent to which Excel students or schools met performance standards, year-to-year comparison questions about the extent to which Excel schools or students showed annual improvement and, Excel-to-non-Excel school comparisons (N20). The next 20 (N20) most educationally at-risk schools in Fairfax County Public Schools (FCPS) as measured by the SAI were used in the comparison questions. The N20 schools were similar to Excel schools in terms of the overall percentage of ESOL students but different from Excel schools in terms of the overall percentage of Free or Reduced Price Meals (FRM) students and the initial higher academic performance of its students. Though the N20 schools are used for comparison purposes, they differ in demographics from the Excel schools. The same N20 ES-2

schools will be used throughout the four-year evaluation. The table below contains the demographics for both sets of school during the 2001-2002 school year. Demographic Data for Excel and N20 Schools 2001-2002 Average Percent of Students on FRM Average Percentage of ESOL Students Excel Schools N20 Schools 51.96 32.23 30.32 28.35 Findings The findings to date are organized by grade span and evaluation question. While a continuous decline in academic performance is often associated with schools educating high populations of at-risk students, such has not been the case with Excel schools. In general, the interim findings show that Excel schools are either making progress or holding the same level of performance as observed during the baseline year. The greatest challenge for Excel schools as a group seems to be with sustaining the academic progress made from one year to the next. Student Achievement in Grades K-2 To what extent do Excel students meet performance standards for kindergarten, first grade, and second grade? Meeting established achievement standards historically has been a significant challenge for students at Excel schools. With only two exceptions across 13 subtests, close to half of the K-2 students scored at or above the relevant performance standard. Thus, students in grades K-2 performed about at the average level in relation to division and national standards for the Early Childhood Assessment Package (ECAP), Stanford Diagnostic Reading Test (SDRT), Developmental Reading Assessment (DRA), and Stanford Achievement Test, 9SA, Abbreviated (Stanford 9SA). To what extent do Excel students show yearly improvement in average scores for kindergarten, first grade, and second grade? During all three years of implementation, most students met the performance expectations for performance on the ECAP (i.e., reached Emergent level). Kindergarten students fourth quarter ECAP scores essentially equaled the 2000 initial baseline performance in spring 2002. That is, while kindergartners continue to learn, the learning appears about at the same rate between 2000 and 2002. For first grade, the DRA performance in spring 2002 was significantly higher than initial levels in 2000, which may be attributed to having full-day kindergarten and Waterford. On the SDRT, performance in spring 2002 was similar to initial levels in spring 2000. For second grade, the DRA was first administered to all grade 2 students in the spring of 2001. On the DRA text levels there was a statistically reliable increase between spring 2001 and spring 2002. This cohort was the first group of second graders who received full-day kindergarten in ES-3

1999-2000 and subsequently showed an increase on the first grade DRA in spring 2001 from spring 2000. In addition, DRA gains were observed for students with zero, one or both risk factors (i.e. ESOL or FRM) with the largest gains observed for ESOL students. On the Stanford 9SA at grade 2, overall, performance in spring 2002 was similar to performance in spring 2000. However, there were significant gains between 2000 and 2002 for FRM students on the word study skills subtest (6 scaled score points) and ESOL students on the reading vocabulary subtest (9 scaled score points). Student Achievement in Grades 3-6 To what extent have Excel schools reached their SAI progress goals? The SAI is an annual expectation for academic improvement for all schools based on grade 3-6 test scores. As such, schools may never reach their goals, reached them once, or more than once. Six Excel schools met or exceeded all of their 2001-2002 SAI annual goals and were eligible for a gold award. Other Excel schools made progress, but not sufficient to qualify for awards during 2001-2002. Since 1999-2000, 17 of the 20 schools have performed well enough on the SAI to qualify for a bonus (bronze, silver or gold) award. Three schools have never made sufficient progress towards SAI goals to qualify for bonus awards. Where do Excel schools stand with respect to the state accreditation standards? A major impetus for creating the Project Excel initiative was the belief that the identified schools were not on course for reaching accreditation standards by 2006, the point at which the state intends to initiate consequences for schools and the division. Based on 2002 SOL results, ten schools have already reached the Fully Accredited status, and two have reached the status of Provisionally Accredited/Meets Standard. This finding, in the context of several years remaining to meet accreditation status, suggests optimism for the seven Excel schools rated as Provisionally Accredited/Needs Improvement, and for the one Excel school now rated as Accredited with Warning. Compared to N20 schools, do Excel schools show greater year-to-year improvement on the SOL and Stanford tests? While the N20 schools are those with the next lowest performance on the SAI, they have lower percents of at-risk students than Excel schools. Thus, the use of the N20 (non-equivalent) schools as a comparison for Excel schools not only sets a high standard, but also implies the belief that additional resources can help Excel schools to overcome their higher at-risk factors. In this context, progress is indicated when Excel schools equal the gains made by N20 schools or close the gap between their performance and that of the N20 schools. For this evaluation question, N20 and Excel schools were compared for year-to-year change in percent passing for the SOL tests or percentage of students at or above the 50th percentile for the Stanford 9TA tests. Project Excel schools showed increases from spring 1999 to spring 2002 on all grade 3 and grade 5 SOL tests. Moreover, increases were greater for Excel schools than N20 schools at grade 3 in all content areas and grade 5 in all content areas but mathematics, where Excel schools matched the amount of progress made by N20 schools. On the fourth grade Stanford 9TA tests, between fall 1999 and fall 2001 Excel schools showed gains on each subtest, with the exception of grade 6 reading. Excel schools made greater progress in all content areas at grade 4. ES-4

To what extent do Excel students who remain in the same school for two years make greater gains on the SOL tests than non-excel students who remain in N20 schools for two years? A consistent concern expressed by staff at Excel schools was that mobility of students severely impacted their school s performance. As a proximal correction for mobility, the evaluation studied the performance of students who were not mobile (i.e., students who remained in the same school for two years and took both the grade 3 SOL tests in spring 2000 and the grade 5 SOL tests in spring 2002). Results showed that Excel students equaled the achievement gains made by N20 students from grade 3 to grade 5 on the SOL tests. Moreover, this pattern of performance held true for both FRM and ESOL students in Excel and N20 schools. Implications of the Findings To Date Concern over meeting school accreditation standards was a compelling reason for initiating Project Excel in 1999. After three years, 12 of the 20 Excel schools have met or exceeded the state s provisional accreditation benchmarks to be reached by 2006. Others are making progress towards this standard. It appears that the initiative is associated with schools moving in the direction needed for accreditation success. A second reason for initiating Project Excel was concern over the achievement gap between at-risk students (Excel) and less at risk students (N20). This interim report indicates, particularly for K-2 program components such as full-day kindergarten and Waterford, positive effects on reading skills for the highest at risk groups, FRM and ESOL students. It must be noted, however, that the number of ESOL students included in testing may be masking improvements on the SDRT and Stanford 9SA measures. Although ESOL are those students likely to benefit greatly from the Project Excel experience, application of state rules to locally administered standardized tests have resulted in a greater proportion of ESOL students being excluded from these standardized tests in comparison to the DRA. By excluding a substantial proportion of ESOL students on the SDRT and Stanford 9SA, Excel schools showed less overall increases in performance at first and second grade than they might have otherwise experienced. Therefore, SDRT and Stanford 9SA data should be interpreted in the context with other data (ECAP and DRA) that have been more inclusive of ESOL students. The current evaluation data paint a picture that is both encouraging and challenging. While the evidence suggests that much has been accomplished, it also appears that important work remains to be done. That work should focus on ensuring that achievement gains for FRM and ESOL students in the primary grades (K-2) are maintained and that they are increased in the upper elementary grades (3-6). Moreover, final judgment of the overall Project Excel initiative should be withheld until the four year evaluation is complete for several reasons: (1) the first full dose cohort (i.e., the group of students who entered kindergarten when Excel began) has not yet taken the grade 3 SOL tests and will not do so until spring 2003; (2) greater improvement may be observed as instructional models are implemented for a longer period of time; (3) the SAI is likely to continue to enhance teacher motivation over time; and (4) additional information and training scheduled for Excel school staffs may enable greater success. ES-5

For the School Board and the Superintendent Recommendations Continue to fund all of the Project Excel components, since they are interrelated, their independent effects cannot be separated, and many participating schools are making progress. Endorse the development and use of appropriate interventions for individual Project Excel schools that have not made continuous progress over the past three years, such that the end result is improvement of educational conditions and outcomes for students. For Instructional Services, Clusters, and Excel Schools Identify successful implementation practices at individual Excel schools that can be shared with other Excel and non-excel schools enrolling high percentages of high-risk students. ES-6