Impacts of Early Literacy Intervention

Transcription

1 1101 Pennsylvania Ave. NW, Suite 600 Washington, DC P F Impacts of Early Literacy Intervention Prepared for Bremerton School District In the following report, The Hanover Research Council examines the effects of Bremerton School District s early literacy intervention, comprised of high quality preschool and full-day kindergarten, on student DIBELS scores. MARKET EVALUATION SURVEYING DATA ANALYSIS BENCHMARKING INNOVATIVE PRACTICES LITERATURE REVIEW

2 Overview As discussed in a July 2009 report published by The Hanover Research Council, much of the current body of early childhood education research has indicated the importance of early interventions targeted towards improving children s preparation for school. For example, a number of studies have provided strong evidence of the beneficial effects of a high-quality preschool education on children s future academic success, social development, and school readiness. In particular, many of these studies have indicated that children who attend a high-quality early childhood education program are more ready for kindergarten and have better language and cognitive skills in early elementary grades than children who do not attend such programs. Studies have also found that disadvantaged students may be among the strongest beneficiaries of preschool education. 1 Another topic of strong interest in the literature surrounding early childhood learning is the impact of full-day versus half-day kindergarten. Current research on kindergarten programs suggests that students who attend full-day kindergarten are better prepared academically than their peers who attend half-day programs. Similar to studies of preschool, many researchers are in agreement that these effects are particularly pronounced among disadvantaged students. 2 Both sets of studies deal with the potential existence of a fade-out of benefits of early childhood interventions. As children progress beyond kindergarten and into the primary grades, some of the effects of preschool or full-day kindergarten programs appear to wear off. In response to this issue, many educators, administrators, and school districts have sought to develop coordinated, multi-year interventions, often referred to as PK-3 programs. These programs typically incorporate a variety of early childhood education components such as high-quality preschool, full-day kindergarten, small class sizes, and well-qualified teachers. Further, these programs 1 Notable studies of the effects of preschool include: Barnett et al. Effects of Five State Prekindergarten Programs on Early Learning. National Institute for Early Education Research. October 2007; Magnuson, Ruhm, and Waldfogel. Does Prekindergarten Improve School Preparation and Performance? Economics of Education Review. Vol. 26. No. 1. February 2007; Schweinhart. Benefits, Costs, and Explanation of the High/Scope Perry Preschool Program. Paper Presented at the 2003 Biennial Meeting of the Society for Research in Child Development. April 24-27, 2003; Gormley, Phillips, and Gayer. Preschool Programs Can Boost School Readiness. Science. Vol June 27, 2008; and Henry and Rickman. The Georgia Early Childhood Study: Andrew Young School of Policy Studies, Georgia State University Studies of the effects of FDK and HDK include: Plucker and Zapf. Short-Lived Gains or Enduring Benefits? The Long-Term Impact of Full-Day Kindergarten. Education Policy Brief, Center for Evaluation & Education Policy. Vol. 3. No. 4. Spring 2005; Plucker et al. The Effects of Full Day Versus Half Day Kindergarten: Review and Analysis of National and Indiana Data. Indiana Association of Public School Superintendents, Information and Research Commission. January 9, 2004; Lee et al. Full-Day vs. Half-Day Kindergarten: In Which Program Do Children Learn More? American Journal of Education. Vol. 112, February 2006; Walston and West. Full-Day and Half-Day Kindergarten in the United States: Findings from the Early Childhood Longitudinal Study, Kindergarten Class of US Department of Education Institute of Education Sciences. June 2004; Elicker and Mathur. What Do They Do All Day? Comprehensive Evaluation of Full-Day Kindergarten. Early Childhood Research Quarterly. Vol ; Del Gaudio Weiss and Offenberg. Enhancing Urban Children s Early Success in School: The Power of Full- Day Kindergarten. Paper Presented at the Annual Meeting of the American Educational Research Association

3 ensure the alignment of curriculum from prekindergarten through the third grade. 3 While research on the effectiveness of PK-3 programs is more limited than the body of work concerning preschool and full-day kindergarten separately, a number of studies indicate that programs that feature comprehensive, multi-year interventions, and incorporate aspects of the PK-3 approach, do have a positive and lasting effect on student outcomes. 4 As we discuss below, Bremerton School District has already implemented a set of well-aligned, comprehensive early childhood interventions that include both highquality preschool and full-day kindergarten. The following report demonstrates how these programs are well in keeping with the current body of literature surrounding preschool, full-day kindergarten, and PK-3, by improving student learning outcomes throughout the district. Scope of the Report In this report, we examine two early childhood intervention programs used throughout the Bremerton School District. We begin by addressing the issue of the differential impact of full-day kindergarten (FDK) versus half-day kindergarten (HDK), tracking the progress of Bremerton students enrolled in either program as they move from kindergarten through the second grade. As Bremerton implemented the program throughout the district beginning in the fall of , the second graders were the first FDK participants. As we describe in further detail below, we selected a comparison group composed of third graders that were the last cohort of students to participate in HDK in the district. In our second section, we examine the effects of Bremerton s Early Childhood Care and Education (ECCE) initiative. The initiative represents a collaboration between the district, Head Start programs, preschools, and child care providers within the area. The district provides its early learning partners with access to a literacy coach, reading curriculum, and staff development, among other forms of support. 5 Through this partnership, the district seeks to increase the number of students entering kindergarten with strong early literacy skills, decrease the number of students with 3 Arthur Reynolds, Katherine Magnuson, and Suh-Ruu Ou. PK-3 Education: Programs and Practices that Work in Children s First Decade. The Foundation of Child Development Working Paper: Advancing PK-3. No. 6. January Studies that deal with issues related to PK-3 include: Reynolds. Success in Early Intervention: The Chicago Child-Parent Centers. Lincoln: University of Nebraska Press. 2000; Reynolds, et al. Effects of a School-Based, Early Childhood Intervention on Adult Health and Well-Being: A 19-Year Follow-Up of Low-Income Families. Archives of Pediatric Adolescent Medicine. Vol No ; Ramey and Ramey. Early Learning and School Readiness: Can Early Intervention Make a Difference? Merrill-Palmer Quarterly. October 1, 2004; Horacek et al. Predicting School Failure and Assessing Early Interventions with High-Risk Children. Journal of the American Academy of Child & Adolescent Psychiatry. Vol. 26. No ; and Ramey et al. Head Start Children s Entry into Public School: A Report on National Head Start/Public School Early Childhood Transition Demonstration Study. Civitan International Research Center, The University of Alabama at Birmingham. November 28, Early Learning (Ages 3-5). Early Childhood Care & Education Group (ECCE), Bremerton School District. 3

4 learning differences linked to literacy, and better align preschool curriculum with Bremerton s K-3 instruction. 6 The third and final section investigates the impact of these programs in combination. We first examine the effect of ECCE preschool followed by full-day kindergarten compared to full-day kindergarten without the ECCE component. We then compare the impact of ECCE preschool and full-day kindergarten with ECCE preschool and half-day kindergarten. Our selected measure of student performance is the Dynamic Indicators of Basic Early Literacy Skills (DIBELS). The DIBELS measures are based on five early literacy concepts that have been identified by the National Reading Panel: phonemic awareness, alphabetic principle, accuracy and fluency, vocabulary, and comprehension. 7 The assessments are composed of a variety of timed tasks which result in an assigned fluency score. Depending on the student s grade level and the time of year in which the assessment is administered, more than one task may be assessed. A summary of multiple fluency scores for each administration of the assessment is provided as an instructional recommendation, where students are identified as being at benchmark, needing strategic intervention, or needing intensive intervention. The assessment is administered three times each year, allowing teachers, administrators, and researchers to longitudinal trends. Key Findings Below we present a brief overview of the major findings of our analysis. Each point is discussed in greater detail throughout the rest of the report. FDK vs. HDK: Students in the second grade cohort that participated in Bremerton s full-day kindergarten performed better on DIBELS assessments than students from the same cohort that entered the district after kindergarten, having received half-day kindergarten instruction elsewhere. This difference persists through the fall of second grade. FDK vs. HDK: Bremerton students with low socioeconomic status (as measured by free or reduced price lunch) that attended full-day kindergarten score higher on nearly all DIBELS assessments than similar students that attended a half-day program, from the middle of kindergarten through the beginning of second grade. ECCE vs. No ECCE: Students that attended a preschool linked to Bremerton s ECCE initiative score higher than their non-ecce peers on nearly every 6 Claus von Zastrow. What Works in Early Childhood Education: A View from the Field. Public School Insights. June 23, DIBELS Data System. Center on Teaching and Learning, University of Oregon. 4

5 DIBELS assessment through the end of kindergarten. The difference between these two groups is no longer significant as the students progress into first grade. ECCE-FDK vs. FDK-Only: Students that attended a combination of ECCE preschool and full-day kindergarten exhibit an initial advantage during the kindergarten year over students that did not participate in ECCE but did attend full-day kindergarten in the district. This advantage is evident in DIBELS scores through the middle of kindergarten but then disappears after that point, as students that did not participate in ECCE preschool begin to gain ground. ECCE-FDK vs. ECCE-HDK: While starting off at roughly the same level academically, by the middle of kindergarten, students with low socioeconomic status that participated in both ECCE preschool and full-day kindergarten outperformed students participating in a combination of ECCE preschool and half-day kindergarten. A significant gap persists through the middle of first grade. Methodology Our main approach to assessing the impact of these programs is largely based on an earlier study conducted on Bremerton s FDK program. 8 Following the earlier study, we conduct bivariate analyses to measure the effects of Bremerton s ECCE preschool and full-day kindergarten programs on student performance on the DIBELS assessments from kindergarten through the end of second grade. For each program comparison, we primarily examine differences in mean scores between two groups. For example, in our investigation of full-day and half-day kindergarten, we calculate the mean DIBELS scores for all full-day students and compare this figure to the mean scores of half-day students. In addition to reporting the mean score for each assessment for the two groups, we ran t-tests on the results. The t-test is a statistical analysis tool which is used to determine if the means of two groups are statistically different. Once a t-value is obtained, the level of significance is calculated (p-value) to determine the probability that the difference in means was a result of chance. The smaller the p-value, the more significant the difference between means, that is, the more likely it is that the result was not obtained by chance. Any p-value less than 0.05 is considered statistically significant, while values less than 0.01 are considered very significant, and those less than are considered extremely significant. We report p-values in the final column of each table comparing mean scores in the sections below. The learning outcomes data included in our analysis was drawn from the DIBELS data system, using login information provided by Bremerton. 9 Through the system s 8 Bremerton School District Full Day Kindergarten Evaluation. Provided by Bremerton School District. 9 DIBELS Data System. 5

6 data farming feature, we pulled DIBELS assessment information for kindergarten through the end of second grade for every student in the district within the following cohorts: Third Graders (K, 1 st, 2 nd ) Second Graders (K, 1 st, 2 nd ) First Graders (K, 1 st ) Kindergartners (K) As discussed in the sections below, some of our analysis includes comparisons of two separate cohorts. For example, in our examination of the effects of full-day and halfday kindergarten, we compare students from the second grade cohort that received full-day instruction with students from the third grade cohort that received half-day instruction. In other cases, we combined cohorts to form a larger sample for analysis. For example, when looking at the effects of ECCE preschool in combination with full-day kindergarten compared to full-day kindergarten alone, we found that analysis of single cohorts was difficult due to small sample sizes. We therefore combined all kindergarten through second grade cohorts in order to better enable us to discern statistically significant differences between comparison groups. We restrict the majority of our analysis to continuously enrolled students. This means that the students included in our discussion below were marked as being in the district for all relevant grades from kindergarten through third grade (i.e., third graders were enrolled in Bremerton for kindergarten, first, second, and third grade; second graders were enrolled in Bremerton for kindergarten, first, and second grade; etc.). We did this primarily for two reasons. First, the preschool data provided by Bremerton indicates which students, among all currently active K-3 students, attended a previous preschool (PP). In our dataset, we marked all students that had not been tagged as PP, as not having attended preschool. By including only continuously enrolled students in our analysis, we are therefore avoiding the potential mistake of mislabeling students that had actually attended preschool but were not included in Bremerton s preschool file due to their inactivity during the past year. The same goes for students marked as having participated in the ECCE initiative. Second, including only continuously enrolled students further allows us to control for any potential effects of students moving in and out of the district. If students are continuously enrolled, we can be confident that the effects of instruction on DIBELS scores can only be attributed to the district. This is not to say that source of instruction is the only factor we need to account for in our analysis. As we will see throughout the report, it is also important to keep in mind other characteristics such as race, gender, and socioeconomic status (as represented by free or reduced price lunch). We are careful to identify potential limitations imposed by the comparability of groups with regard to these background characteristics. 6

7 The Effects of Full-Day Kindergarten vs. Half-Day Kindergarten In this section, we examine the effects of full-day kindergarten (FDK) and half-day kindergarten (HDK) on students DIBELS scores. As mentioned in the introduction, our approach was initially modeled on an earlier study of Bremerton s FDK program. The earlier study compared DIBELS scores of students enrolled in FDK within Bremerton School District with the scores of students that transferred into the district after kindergarten, receiving HDK in another district. The analysis was first conducted within single cohorts by comparing the scores of second graders that received FDK in Bremerton with second graders that entered the district after kindergarten. This analysis was repeated for the first grade and kindergarten cohorts. The table below provides an example of the approach taken in the earlier Bremerton study. Here we are examining the differences in DIBELS scores between second grade cohort students that were enrolled in Bremerton s FDK compared to students from the same cohort that moved into the district after kindergarten, having received HDK elsewhere. Using demographic and socioeconomic data provided by Bremerton, we found that these two groups are fairly well-matched with regard to race, (54.26 percent of the FDK group is white, compared to percent of the HDK group), gender (42.60 percent of the FDK group is female compared to percent of the HDK group), and socioeconomic status (60.54 percent of the FDK group receives free or reduced price lunch compared to percent of the HDK group). FDK vs. HDK Second Grade Cohort 10 FDK HDK (Bremerton) (Other District) t-value Statistically Significant p-value Fall 1 st grade Letter Naming Fluency (n=222) (n=29) 2.28 Yes Phonemic Segmentation Fluency (n=221) (n=28) 3.48 Yes Nonsense Word Fluency (n=221) (n=28) 1.95 No Winter 1 st Grade Phonemic Segmentation Fluency (n=219) (n=41) 3.73 Yes Nonsense Word Fluency (n=220) (n=41) 2.10 Yes Oral Reading Fluency (n=218) (n=41) 1.68 No Spring 1 st Grade Phonemic Segmentation Fluency (n=218) (n=46) 2.97 Yes Nonsense Word Fluency (n=219) (n=46) 2.66 Yes Oral Reading Fluency (n=219) (n=46) 1.61 No Note that the samples (and therefore resulting scores and statistics) are somewhat different than those reported in the earlier Bremerton study. This is due to the fact that we restricted our samples to individuals for which Bremerton provided demographic and socioeconomic data. We did this to ensure that our samples are evenly matched in terms of background characteristics. We further added recently available assessment scores from the spring of second grade. Despite these differences, the trends displayed in this table generally follow those reported in the earlier study. 7

8 FDK vs. HDK Second Grade Cohort 10 FDK HDK (Bremerton) (Other District) t-value Statistically Significant p-value Fall 2 nd Grade Nonsense Word Fluency (n=220) (n=74) 2.47 Yes Oral Reading Fluency (n=220) (n=74) 2.00 Yes Winter 2 nd Grade Oral Reading Fluency (n=220) (n=80) 1.51 No Spring 2 nd Grade Oral Reading Fluency (n=222) (n=78) 0.91 No Similar to the earlier study, we see that there are statistically significant differences between FDK and HDK student DIBELS scores from the fall of first grade through the fall of second grade. From an initial look at this breakdown of data, it appears that students that received FDK in Bremerton outperform their HDK peers. As the author of the earlier study notes, however, this approach exhibits a number of limitations that make it difficult to determine the relationship between the FDK treatment and the higher performance on DIBELS assessments. First, as we have already mentioned, the HDK comparison groups used in the earlier study were composed of students that entered Bremerton after kindergarten, having received HDK somewhere else. Due to their movement into the district from elsewhere, these students likely exhibit different background characteristics than the students that enrolled in kindergarten within the district. Not only did they receive a different type of instruction, they lived in another geographical area, went to school in a different district, etc. For the type of analysis we are conducting, it is important to ensure that the comparison groups are as similar as possible, differing only in their having participated in FDK or HDK. Second, kindergarten DIBELS scores were unavailable for HDK students that received instruction outside of Bremerton. Therefore, there is no baseline data to determine whether these HDK students started out at the same level academically as Bremerton s FDK students. Since the DIBELS results are only available for the start of first grade for this group, we do not know how they performed upon their entrance to kindergarten. If students in the HDK group started at a significantly lower level than their FDK peers, gaps in scores in later assessments could merely be a result of stronger students attending FDK, while weaker students attended HDK outside of the district. An alternative, and potentially stronger, approach the earlier study also employed was to compare the scores of the first cohort to receive FDK instruction in the district ( second graders) and the last cohort to receive HDK instruction in the district ( third graders). This method has an advantage over the previous approach in that it allows the researcher to control for out-of-district differences in 8

9 students. Further, as DIBELS scores are available for both groups starting with entrance to kindergarten, we now have baseline data for comparison. Therefore, we elected to take this approach as the starting point for our analysis. First, we provide some demographic and socioeconomic information concerning the students included in the two groups. As displayed below, we immediately notice a limitation imposed by the data. We see that while the groups are fairly evenly matched in terms of race/ethnicity, there are stark differences with regard to their gender and socioeconomic status (as measured by lunch code). 2nd Grade FDK Cohort and 3 rd Grade HDK Cohort Demographic and Socioeconomic Information FDK HDK Race/Ethnicity American Indian or Alaska Native 3.14% 3.70% Asian 6.28% 6.48% Black or African American 6.28% 11.57% Caucasian or White 54.26% 54.17% Hispanic or Latino 9.87% 8.33% Multiracial 14.80% 10.19% Not Provided 1.79% 1.39% Pacific Islander or Nat Hawaiian 3.59% 4.17% Sex Female 42.60% 55.09% Male 57.40% 44.91% Lunch Code Free/Reduced Price 60.54% 73.61% Paid 39.46% 26.39% In order to determine whether these differences in demographic and socioeconomic makeup of the comparison groups would present a barrier to our analysis of FDK and HDK, we examined DIBELS scores by gender and socioeconomic status (SES). While we did not find statistically significant differences in DIBELS scores when broken down by gender, we did find a significant difference by lunch code. On every DIBELS assessment administered from the beginning of kindergarten through the end of second grade (with only one exception), we found that students receiving free or reduced price lunch scored significantly lower than paid students. Using this variable as a proxy for SES, we see that students with lower SES consistently perform worse on DIBELS tests on average. Since the third grade HDK cohort has a higher percentage of these students, we would expect that they would start out lower on DIBELS assessments at the entrance to kindergarten. As displayed in the table below, this is the case for both fall 9

10 kindergarten assessments, though the difference is only statistically significant for the Letter Naming Fluency assessment. FDK vs. HDK All Continuously Enrolled 2 nd and 3 rd Graders FDK HDK t-value Statistically Significant p-value Fall K Initial Sound Fluency (n=218) 8.99 (n=187) 1.75 No Letter Naming Fluency (n=218) (n=186) 2.68 Yes Winter K Initial Sound Fluency (n=220) (n=202) 4.16 Yes Letter Naming Fluency (n=221) (n=202) 4.52 Yes Phonemic Segmentation Fluency (n=221) (n=202) 4.67 Yes Nonsense Word Fluency (n=220) (n=201) 5.24 Yes Spring K Letter Naming Fluency (n=222) (n=209) 6.10 Yes Phonemic Segmentation Fluency (n=222) (n=209) 5.31 Yes Nonsense Word Fluency (n=222) (n=209) 6.95 Yes Fall 1 st Letter Naming Fluency (n=222) (n=210) 3.05 Yes Phonemic Segmentation Fluency (n=221) (n=208) 4.48 Yes Nonsense Word Fluency (n=221) (n=208) 4.01 Yes Winter 1 st Phonemic Segmentation Fluency (n=219) (n=211) 3.52 Yes Nonsense Word Fluency (n=220) (n=210) 4.09 Yes Oral Reading Fluency (n=218) (n=210) 3.23 Yes Spring 1 st Phonemic Segmentation Fluency (n=218) (n=211) 3.60 Yes Nonsense Word Fluency (n=219) (n=211) 4.32 Yes Oral Reading Fluency (n=219) (n=212) 3.47 Yes Fall 2 nd Nonsense Word Fluency (n=220) (n=207) 2.31 Yes Oral Reading Fluency (n=220) (n=206) 2.13 Yes Winter 2 nd Oral Reading Fluency (n=220) (n=206) 1.77 No Spring 2nd Oral Reading Fluency (n=222) (n=209) 2.01 Yes Beyond the fall of kindergarten, we see a significant gap between FDK and HDK scores on nearly every DIBELS test through the end of second grade. However, since these students started at different academic levels, it is difficult to decipher whether future variations in performance (beyond the fall of kindergarten) are tied to type of kindergarten (FDK or HDK) or to the initial difference in academic starting points. In other words, we cannot say with confidence that the gap in scores is due 10

11 to the FDK treatment. Therefore, we sought to control for differences in socioeconomic status that could potentially cloud our interpretation of the results. We accomplished this by isolating students identified as receiving free or reduced price lunch (low SES students). As illustrated in the table below, there was no statistically significant difference between the two groups at the beginning of kindergarten. This provides evidence that our two groups are roughly comparable in terms of their academic starting points (pre-treatment) and that we may proceed with our analysis. FDK vs. HDK Continuously Enrolled 2 nd and 3 rd Graders Receiving Free or Reduced Price Lunch FDK HDK t-value Statistically Significant p-value Fall K Initial Sound Fluency 9.57 (n=131) 8.27 (n=132) 1.42 No Letter Naming Fluency (n=131) (n=132) 1.09 No Winter K Initial Sound Fluency (n=133) (n=148) 3.62 Yes Letter Naming Fluency (n=134) (n=148) 2.92 Yes Phonemic Segmentation Fluency (n=134) (n=148) 3.40 Yes Nonsense Word Fluency (n=133) (n=147) 3.89 Yes Spring K Letter Naming Fluency (n=135) (n=156) 4.62 Yes Phonemic Segmentation Fluency (n=135) (n=156) 3.69 Yes Nonsense Word Fluency (n=135) (n=156) 6.37 Yes Fall 1 st Letter Naming Fluency (n=134) (n=155) 1.85 No Phonemic Segmentation Fluency (n=133) (n=154) 3.51 Yes Nonsense Word Fluency (n=133) (n=154) 3.63 Yes Winter 1 st Phonemic Segmentation Fluency (n=131) (n=154) 2.49 Yes Nonsense Word Fluency (n=132) (n=154) 4.10 Yes Oral Reading Fluency (n=130) (n=154) 2.38 Yes Spring 1 st Phonemic Segmentation Fluency (n=130) (n=155) 2.52 Yes Nonsense Word Fluency (n=131) (n=155) 3.92 Yes Oral Reading Fluency (n=131) (n=155) 2.83 Yes Fall 2 nd Nonsense Word Fluency (n=134) (n=151) 2.14 Yes Oral Reading Fluency (n=134) (n=150) 1.53 No Winter 2 nd Oral Reading Fluency (n=133) (n=152) 1.37 No Spring 2 nd Oral Reading Fluency (n=134) (n=154) 1.93 No

12 Mean DIBELS Score DISTRICT ADMINISTRATION PRACTICE As the table above displays, low SES students that received FDK scored higher on nearly all DIBELS assessments from the middle of kindergarten through the beginning of second grade. Similar to the earlier Bremerton study, the differences in scores are statistically significant for all assessments of pre-reading skills. Unlike the earlier study, we also found statistically significant differences on Oral Reading Fluency scores in the winter and spring of first grade. We further confirmed these findings by conducting a similar analysis of the effects of FDK and HDK on low SES student DIBELS scores using all available continuously enrolled FDK students from the kindergarten through second grade cohorts and HDK students from the third grade cohort. While there were differences in scores between the two groups on the fall kindergarten assessments, they were not statistically significant. Thereafter, every test up to the Nonsense Word Fluency assessment in the fall of second grade featured a statistically significant difference between low SES FDK students and low SES HDK students. As the graph below clearly displays, students in the FDK group consistently outperformed their HDK peers through the beginning of second grade. The group continued to receive higher scores through the end of second grade, though the differences between groups were no longer significant after the fall of second grade. FDK vs. HDK Continuously Enrolled K-3 Students Receiving Free or Reduced Price Lunch FDK HDK Turning back to the second and third grade cohort comparisons, another way of investigating the differences between the two groups is to look at the average growth in fluency scores. Average growth in fluency scores measures the difference between 12

13 a later skill assessment and an earlier assessment of that same skill. 11 For example, we measure the growth in scores between the administration of the Initial Sounds Fluency assessment in the winter of kindergarten and the administration of the Initial Sounds Fluency assessment in the fall of kindergarten. As the earlier Bremerton study noted, One would expect the rate of improvement on the DIBELS to be substantially different during the kindergarten year when comparing full and half day students. Do the results reflect that prediction? And did that accelerated growth continue through the first and second grade or did half-day students make up ground? 12 The table below presents the average growth in fluency scores for the second grade cohort FDK group and the third grade cohort HDK group. 2 nd Grade Cohort FDK vs. 3 rd Grade Cohort HDK, Continuously Enrolled Students Receiving Free or Reduced Price Lunch Average Growth in Fluency Scores FDK HDK t-value Statistically Significant p-value Initial Sounds Fluency Fall K to Winter K (n=130) (n=132) 3.08 Yes Letter Naming Fluency Fall K to Winter K (n=131) (n=132) 2.85 Yes Fall K to Spring K (n=131) (n=131) 4.24 Yes Fall K to Fall 1 st (n=130) (n=131) 1.35 No Phonemic Segmentation Fluency Winter K to Spring K (n=134) (n=147) No Winter K to Fall 1 st (n=132) (n=145) No Winter K to Winter 1 st (n=130) (n=145) No Nonsense Word Fluency Winter K to Spring K (n=133) (n=146) 3.42 Yes Winter K to Fall 1 st (n=132) 9.77 (n=145) 0.19 No Winter K to Winter 1 st (n=130) (n=145) 1.84 No Winter K to Spring 1 st (n=129) (n=145) 1.93 No Oral Reading Fluency Winter 1 st to Spring 1 st (n=129) (n=153) 1.29 No Winter 1 st to Fall 2 nd (n=129) (n=149) No Winter 1 st to Winter 2 nd (n=128) (n=150) No Note that nearly all measures of changing scores within kindergarten (fall k to winter k, fall k to spring k, or winter k to spring k) display a significantly higher average growth for FDK students. This suggests that FDK provides a substantial boost to low SES student DIBELS scores during the kindergarten year, as compared to HDK. 11 Note that this approach was also used in the earlier Bremerton study. 12 Bremerton School District Full Day Kindergarten Evaluation. 13

14 Percent in Category DISTRICT ADMINISTRATION PRACTICE Similar to the earlier Bremerton study, however, we see that there are no significant improvements in the level of growth by FDK students beyond the kindergarten year. In addition to studying differences in scores and average growth in scores between low SES FDK and HDK students, we can examine longitudinal changes in student performance within the entire second grade cohort FDK group on its own. While this does not provide a comparison with students from HDK programs, it enables us to gain some insight into how all students benefit from FDK. First, we can use the summary measures of student scores or instructional recommendations, assigned to students in the fall, winter, and spring of each year. The figure below presents a breakdown of students categorized as: at benchmark, needs strategic intervention, or needs intensive intervention from the fall of kindergarten through the spring of second grade. As the figure illustrates, the percentage of FDK students at benchmark dramatically increases during the kindergarten year, while the proportion of students needing either type of intervention decreases. Though the percentage of students fluctuates after the kindergarten year, it rests at a substantially higher point by the end of second grade (approximately 60 percent of students at benchmark) than it did at the beginning of kindergarten (approximately 40 percent of students at benchmark). Percent of 2 nd Grade Cohort FDK at Benchmark, Needing Strategic Intervention, or Needing Intensive Intervention All Continuously Enrolled Students 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Intensive Strategic Benchmark The earlier Bremerton study used the instructional recommendations to examine the longitudinal DIBELS data in another manner. The study placed students into three 14

15 Percentage of Students at Benchmark DISTRICT ADMINISTRATION PRACTICE categories as indicated by their initial kindergarten achievement on the DIBELS: students at benchmark in the fall of kindergarten; students needing strategic intervention in the fall of kindergarten; and students needing intensive intervention in the fall of kindergarten. The study proceeded to track the percentage of students within each group that tested at benchmark on subsequent assessments, comparing the results of continuously enrolled students in the second grade cohort (FDK) and the third grade cohort (HDK) groups. While we have seen that comparisons between all FDK and all HDK students do not provide a fair measure of potential gaps in DIBELS scores, due to the differences in the socioeconomic composition and the academic starting points of the two groups, it is helpful to track how FDK students within each initial instructional recommendation category performed from kindergarten through second grade. The figure below presents the changing percentages of students within each initial instructional recommendation category that are marked as at benchmark over time. For example, by the winter of kindergarten, percent of students that were initially listed as needing strategic intervention (upon entrance to kindergarten) were at benchmark. This percentage increased again to 83.7 percent by the spring of kindergarten. Percent of 2 nd Grade Cohort FDK at Benchmark by Initial Instructional Recommendation Category All Continuously Enrolled Students 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Fall K Winter K Spring K Fall 1st Winter 1st Spring 1st Fall 2nd Winter 2nd Spring 2nd Started Benchmark Started Strategic Started Intensive As the figure above displays, there is clearly a dramatic boost in the percentage of FDK students achieving benchmark in the winter and spring of kindergarten among those who were initially identified as requiring strategic or intensive intervention. However, as we move beyond the spring of kindergarten, we see the percentages of students at benchmark in these two groups begin to dip back down. Nevertheless, while the steep jump in percentages of students at benchmark is not fully maintained, 15

16 high percentages of students initially identified as needing strategic (52.17 percent) or intensive (39.39 percent) intervention are at benchmark through the spring of second grade. Summary of Section Findings When comparing students from the second grade cohort that received FDK in Bremerton with students from the same cohort that transferred into the district after kindergarten, having received HDK elsewhere, the FDK group outperforms the HDK group through the beginning of second grade. However, since the two groups may have different background characteristics that we cannot account for, most importantly due to the fact that the HDK group moved to Bremerton from other districts, it is difficult to determine the relationship between FDK and higher DIBELS scores. This is further confounded by the fact that baseline kindergarten DIBELS data for the HDK group is unavailable, precluding us from establishing whether the two groups started out at the same levels academically. While we are unable to conclusively state that students enrolled in FDK perform better than their HDK counterparts overall as a result of the full-day program, the analysis in this section clearly suggests that FDK has a greater beneficial effect on low-income students than HDK does. When we only look at students receiving free or reduced price lunch, we see that if these students are enrolled in FDK, they perform significantly higher on DIBELS assessments than their HDK peers. This difference appears by the middle of kindergarten and persists through the beginning of second grade. Key Points Low SES students that attended FDK outperform their HDK counterparts on DIBELS assessments starting at the middle of kindergarten and continuing through the beginning of second grade. While the results are less conclusive, the same trend appears for all FDK students (regardless of socioeconomic status). 16

17 Effects of the Early Childhood Care and Education (ECCE) Initiative In this section, we seek to analyze the effectiveness of Bremerton s Early Childhood Care and Education (ECCE) initiative on its own. Before we begin our analysis, it is important to note a few obstacles we encountered in assessing the impact of this program. First, we do not have baseline data for students upon entrance to preschool, as the DIBELS data system only provides data for the district beginning with kindergarten. Therefore, we do not know the pre-treatment academic starting point of these students. Since a large proportion of the comparison group included in our analysis did not attend any preschool program, this entrance to preschool data would not be available anyways. In the case of students that did not attend preschool, we would need assessment data from when they were preschool-age to provide a baseline for comparison. Nevertheless, without knowing the baseline of these students, it is more difficult to determine the impact of the program on students DIBELS scores. Next, the vast majority of students in our dataset that were identified by Bremerton as participating in preschool also participated in the ECCE initiative. Due to the extremely small number of students that attended non-ecce preschools, we are unable to make meaningful comparisons between ECCE preschool and non-ecce preschool participants. As displayed below, we therefore make comparisons between ECCE preschool participants and children participating in any alternative options (students that attended other preschools or did not attend preschool at all). Regardless of this limitation, the approach enables us to study the performance of students that participated in ECCE preschool compared to other options that students have available to them. ECCE vs. No ECCE In the following comparison, we examine students that participated in the ECCE initiative versus those that did not participate. As mentioned above, the vast majority of students that did not participate in ECCE were not marked as attending any other type of preschool. We therefore compare ECCE participants with a combined group of students attending other preschools or not attending any preschool. We conducted this analysis using a combined dataset of all continuously enrolled kindergarten through third graders, tracking their DIBELS scores through the end of second grade. We first note that when all cohorts are included, the breakdown of students between ECCE and No ECCE, results in two well-matched groups in terms of race/ethnicity, gender, and socioeconomic status. 17

18 ECCE vs. No ECCE All Continuously Enrolled K-3 Students Demographic and Socioeconomic Information ECCE No ECCE Race/Ethnicity American Indian or Alaska Native 3.08% 3.32% Asian 3.96% 4.65% Black or African American 7.49% 8.23% Caucasian or White 56.39% 53.65% Hispanic or Latino 7.71% 9.03% Multiracial 14.32% 15.01% Not Provided 2.86% 2.92% Pacific Islander or Native Hawaiian 4.19% 3.19% Sex Female 46.92% 49.93% Male 53.08% 50.07% Lunch Code Free/Reduced Price 67.18% 67.99% Paid 32.82% 32.01% As displayed in the table below, students that participated in ECCE score higher than students in the No ECCE group on nearly every DIBELS assessment through the end of kindergarten. ECCE vs. No ECCE All Continuously Enrolled K-3 rd Graders ECCE No ECCE t-value Statistically Significant p-value Fall K Initial Sound Fluency (n=437) 9.74 (n=649) 3.19 Yes Letter Naming Fluency (n=437) (n=648) 4.06 Yes Winter K Initial Sound Fluency (n=436) (n=703) 2.25 Yes Letter Naming Fluency (n=438) (n=703) 2.97 Yes Phonemic Segmentation Fluency (n=438) (n=703) 4.04 Yes Nonsense Word Fluency (n=435) (n=703) 2.99 Yes Spring K Letter Naming Fluency (n=440) (n=731) 2.00 Yes Phonemic Segmentation Fluency (n=440) (n=731) 1.84 No Nonsense Word Fluency (n=440) (n=731) 2.03 Yes Fall 1 st Letter Naming Fluency (n=235) (n=538) No Phonemic Segmentation Fluency No Nonsense Word Fluency (n=231) (n=536) No While a statistically significant gap in scores disappears after the spring of kindergarten, significant differences (in an unexpected direction) appear on three assessments administered in the first and second grade. We found that on the 18

19 Phonemic Segmentation Fluency and Nonsense Word Fluency assessments in the winter of first grade, and the Nonsense Word Fluency assessment in the fall of second grade, students in the No ECCE group scored significantly higher than the ECCE group on average. These differences are displayed below. Other Assessments Displaying Significant Differences All Continuously Enrolled K-3 rd Graders ECCE No ECCE t-value Statistically Significant p-value Winter 1 st Phonemic Segmentation Fluency (n=231) (n=536) Yes Nonsense Word Fluency (n=232) (n=537) Yes Fall 2 nd Nonsense Word Fluency (n=157) (n=293) Yes There appears to be a fairly clear explanation for these discrepancies in later assessment scores. We noticed that as we progress from kindergarten to first and second grade assessments, the composition of the sample becomes unbalanced in terms of paid versus free and reduced price lunch students. For example, when we include all of the students in our dataset (K-3), the two groups are evenly matched in terms of socioeconomic status percent of ECCE students have free or reduced price lunch compared to percent of the No ECCE group. As we move to first grade DIBELS assessments, however, the kindergarten class drops out of the analysis, resulting in a higher percentage of free and reduced price students in the ECCE group (74.89 percent) compared to the No ECCE group (64.74 percent). This change in sample composition likely explains the counterintuitive results on some of the later assessments, since as noted above, low SES students tend to perform lower than the rest of the student population on DIBELS. We can address this problem by once again only focusing on lower income students. When we restrict our sample to free and reduced price lunch students, we find a significant difference between ECCE and No ECCE groups through the middle of kindergarten, and then no statistically significant differences between the groups from that point forward through the end of second grade. ECCE vs. No ECCE Continuously Enrolled K-3 rd Graders Receiving Free or Reduced Price Lunch ECCE No ECCE t-value Statistically Significant p-value Fall K Initial Sound Fluency (n=292) 8.52 (n=429) 3.21 Yes Letter Naming Fluency (n=292) (n=429) 2.99 Yes Winter K Initial Sound Fluency (n=291) (n=474) 2.61 Yes Letter Naming Fluency (n=293) (n=474) 2.14 Yes Phonemic Segmentation Fluency (n=292) (n=474) 3.41 Yes Nonsense Word Fluency (n=290) (n=474) 2.34 Yes

20 ECCE vs. No ECCE Continuously Enrolled K-3 rd Graders Receiving Free or Reduced Price Lunch ECCE No ECCE t-value Statistically Significant p-value Spring K Letter Naming Fluency (n=293) (n=496) 1.02 No Phonemic Segmentation Fluency (n=293) (n=496) 0.78 No Nonsense Word Fluency (n=293) (n=496) No The above results indicate that ECCE students perform better during the kindergarten year than students that did not participate in the initiative. At the same time, we must keep in mind the caveat that we do not know the academic starting point of these two groups upon entrance to preschool or at preschool-age (in the case of students that did not attend preschool) and therefore cannot say for certain that participation in ECCE is responsible for this gap in performance. Nevertheless, as both groups appear well-matched in terms of race/ethnicity, gender, and socioeconomic status, we would generally expect the students to perform at similar levels. Based on available data, the results suggest that ECCE students receive an initial academic boost that lasts at least through the middle of kindergarten. 13 In order to further examine the performance of students that attended ECCE, we can next look at the average change in DIBELS scores. Since we determined that the two comparison groups are similar in terms of race/ethnicity, gender, and SES when all K-3 students are included and the significant differences in mean scores are present only during the kindergarten year, we illustrate these changes during the kindergarten year in the table below. ECCE vs. No ECCE All Continuously Enrolled K-3 Students Average Growth in Fluency Scores During Kindergarten Year ECCE No ECCE t-value Statistically Significant p-value Initial Sounds Fluency Fall K to Winter K (n=431) (n=642) No Letter Naming Fluency Fall K to Winter K (n=433) (n=642) No Fall K to Spring K (n=434) (n=638) Yes Phonemic Segmentation Fluency Winter K to Spring K (n=435) (n=691) Yes Nonsense Word Fluency Winter K to Spring K (n=433) (n=691) No It is interesting to note that while we only found two instances in which there are statistically significant differences in average growth in fluency scores during the 13 As the above tables display, in our analysis of all continuously enrolled K-3 rd graders we see a statistically significant difference in scores through the end of kindergarten. By contrast, when we only look at students receiving free or reduced price lunch, the difference is statistically significant through the middle of kindergarten. 20

21 Percentage at Benchmark DISTRICT ADMINISTRATION PRACTICE kindergarten year (change in Letter Naming Fluency from fall k to spring k and change in Phonemic Segmentation Fluency from winter k to spring k), both indicate that the No ECCE group was improving at a faster rate than the ECCE group. Coupled with the earlier analysis of differences in means, it appears that while ECCE students have an initial advantage over those that did not participate in ECCE preschool, the No ECCE group begins to close the gap during the kindergarten year. This closing of the gap during the kindergarten year is also apparent when we review the percent of students in each group listed as at benchmark, according to the instructional recommendations provided in the fall, spring, and winter of kindergarten. As the figure below depicts, a lower percentage of students that did not participate in ECCE are at benchmark at entrance to kindergarten than students that did participate in ECCE. However, by the winter of kindergarten the gap begins to narrow and has nearly disappeared by the spring of kindergarten. 90% ECCE vs. No ECCE Percentage of Students at Benchmark All Continuously Enrolled K-3 Students 80% 70% 60% 50% ECCE No ECCE 40% 30% Fall K Winter K Spring K Summary of Section Findings When examining the effects of high quality preschool on its own, we found that students that participated in Bremerton s ECCE initiative scored higher than students that did not participate in ECCE on nearly every DIBELS assessment through the end of kindergarten. This suggests that ECCE students have an initial advantage over their non-ecce counterparts during the kindergarten year. However, as mentioned above, the students that did not participate in Bremerton s partnership 21