Washington 21st Century Community Learning Centers Program Evaluation: Year 2. Neil Naftzger Matthew Vinson Feng Liu Bo Zhu Kimberly Foley

Transcription

1 Washington 21st Century Community Learning Centers Program Evaluation: Year 2 Neil Naftzger Matthew Vinson Feng Liu Bo Zhu Kimberly Foley JANUARY MONTH YEAR 2014

2

3 Washington 21st Century Community Learning Centers Program Evaluation: Year 2 January 2014 Neil Naftzger Matthew Vinson Feng Liu Bo Zhu Kimberly Foley 20 North Wacker Drive, Suite 1231 Chicago, IL Fax: Copyright All rights reserved _01/14

4

5 Contents Page Executive Summary... i Program Quality... i Evaluation Questions and Methods...v Methods...v Summary of Key Findings... vi Chapter 1: Introduction...1 Evaluation Questions...1 Reasoning for Chosen Evaluation Questions: Importance of Program Quality...1 Organization of Report...2 Chapter 2: Methods...3 Data Sources and Analysis...3 Chapter 3. Primary Characteristics of Washington 21st CCLC Programs and Participants...9 Grantee Characteristics...9 Center Characteristics...12 Summary of Grantee and Center Characteristics...24 Chapter 4: Leading Indicators...25 Overview of Leading Indicators...25 Selected Leading Indicators...26 Organization of Leading Indicators Chapter...28 Organizational Context...28 Summary of Organizational Context Findings and Recommendations...36 Instructional Context...36 Summary of Instructional Context Findings and Recommendations...53 Mutually Reinforcing Context...54 Summary of Findings and Recommendations in Relation to the Community Context Domain...64 Youth Outcomes Leading Indicators...65 Determining Program Improvement Priorities From the Leading Indicator System...68 Chapter 5: Assessing 21st CCLC Program Outcomes...73

6 Within-Program and Impact Analyses...73 Within-Program Analyses...73 Summary of Within-Program Analyses Findings...87 Impact of 21st CCLC Participation on Student Achievement...88 Summary of Impact Analyses Results...95 Conclusions...97 References...100

7 Executive Summary Information summarized in this report is based on data collected and analyzed by American Institutes for Research (AIR) and the David P. Weikart Center for Youth Program Quality (Weikart Center) as part of a statewide evaluation of Washington 21st Century Community Learning Centers (21st CCLC) programs. Results represent findings from Year 2 of a three-year statewide evaluation. The purpose of this executive summary is to (1) set the context for the evaluation design with regard to a primary focus on program quality, (2) outline the evaluation questions and methods, and (3) summarize key findings within each of the identified evaluation questions. To set the context for the evaluation design, a brief discussion on program quality, AIR s framework for understanding afterschool program quality, and the leading indicators of afterschool program quality developed in collaboration with the Washington Office of Superintendent of Public Instruction (OSPI) are provided. Following the discussion on program quality, the evaluation questions and methods are outlined, and a summary of key findings within each of the identified evaluation questions is presented. Program Quality Research on Program Quality Program quality and the implementation of best practices supported by research are increasingly recognized as pressing issues for the afterschool field (Granger, Durlak, Yohalem, & Reisner, 2007). Research on the impact of participation in afterschool programming on students academic and behavioral outcomes often produces mixed and inconclusive results (Granger, 2008). For example, three noteworthy meta-analyses of the impact of afterschool programming found that a majority of the reviewed studies did not find better outcomes for afterschool participants relative to nonparticipants (Durlak & Weissberg, 2007; Granger, 2008; Lauer et al., 2006; Zief, Lauver, & Maynard, 2006). However, others found average positive effects of afterschool program participation on students academic and behavioral outcomes, which they attributed to subsets of higher quality programs driving the overall average positive student outcomes (Durlak & Weissberg, 2007; Lauer et al., 2006). In short, average positive outcomes across several afterschool programs were likely due to the effectiveness of a small number of high-quality individual programs. These findings highlight a key relationship between the quality of afterschool programming and the attainment of desired program outcomes. Meaningful progress has been made relative to understanding how elements of program quality support quality afterschool programs and the attainment of desired youth outcomes. For example, a growing body of research suggests that program outcomes in the form of enhanced student academic achievement outcomes are realized by delivering developmentally appropriate programming that is grounded in core principles of youth development (Birmingham, Pechman, Russell, & Mielke, 2005; Durlak & Weissberg, 2007). The delivery of developmentally appropriate activities that align with principles of youth development varies as a function of staff competencies, interpersonal skills, and knowledge (Vandell et al., 2005). Leading experts agree that staffs ability to form meaningful personal staff-student relationships facilitates the delivery of interactive and engaging program activities (Eccles & Gootman, 2002). Likewise, staff ability to design and deliver developmentally appropriate and interactive program activities is likely to Washington 21st CCLC Program Year 2 Evaluation: Executive Summary i

8 differ as a function of the level of support provided by the program overall. As noted by Smith (2007), Glisson (2007), and Birmingham et al. (2005), a program climate that supports ongoing staff reflection on and involvement in efforts to improve program quality is a key aspect of effective youth-development programs. Programs characterized by a supportive and collaborative climate encourage staff to engage in self-reflective practices that improve overall program quality. AIR Framework for Program Quality The evaluation team at AIR has engaged in extensive work evaluating afterschool programs and providing technical assistance to support high-quality programming. The framework outlined in Figure I represents the most recent research related to the path to quality in afterschool programs as well as the evaluation team s collective expertise with afterschool programming. As shown in Figure I, the achievement of desired youth outcomes is a function of complex interactions between several program elements: Youth Characteristics. These are the characteristics and contributions youth bring to the afterschool setting that influence how they engage with and benefit from afterschool programs. Community Context. The resources and characteristics of the local and school community context serve to support meaningful partnerships to develop program goals, program design, and provide program guidance. Program Participation. Youth are more likely to benefit from afterschool program participation if they attend consistently, over a period of time, and participate in a variety of activity types. Program Quality. Program quality comprises a series of practices and approaches that support the provision of developmentally appropriate, high-quality settings and activities at the point of service. This includes practices and approaches adopted by (a) activity leaders working directly with youth and (b) the organization as a whole, which provides an infrastructure to support implementation of effective practice in the design, delivery, and evaluation of afterschool programming. Washington 21st CCLC Program Year 2 Evaluation: Executive Summary ii

9 Figure I. AIR s Quality Framework for Afterschool Programs Leading Indicators of Afterschool Program Quality To assess the extent to which centers implement research-supported best practices and approaches, a set of newly defined leading indicators of afterschool program quality was developed in collaboration with OSPI and the Weikart Center and discussed in the Year 1 report. The leading indicators are meant to further complement programs participation in the Youth Program Quality Improvement process, to provide additional information regarding how well programs are progressing in implementing research-supported practices, and more importantly, to identify areas in need of improvement. A primary goal of the statewide evaluation was to provide 21st CCLC grantees with data to inform program improvement efforts regarding their implementation of research-supported best practices. AIR, the Weikart Center, and OSPI worked collaboratively to define a series of leading indicators using data collected as part of the statewide evaluation. Specifically, the leading indicator system was designed to do the following: Summarize data collected as part of the statewide evaluation in terms of how well the grantee and its respective centers are adopting research-supported best practices. Allow grantees to compare their level of performance on leading indicators with similar programs and statewide averages. Facilitate internal discussions about areas of program design and delivery that may warrant additional attention from a program improvement perspective. Washington 21st CCLC Program Year 2 Evaluation: Executive Summary iii

10 The leading indicator system is focused on quality program implementation as opposed to youth or program outcomes. It is hypothesized and supported by research indicating that more consistent implementation of research-supported best practices supports the attainment of desired youth outcomes. During Year 2 of the evaluation, the leading indicators continued to be developed in order to meet the goals in providing grantees with opportunities for program improvement. The adopted leading indicators are organized into four overarching contexts: (1) Organizational Context, which is focused on practices that occur among staff and management; (2) Instructional Context, which is focused on practices that occur at the point of service, where staff and youth directly interact; (3) Mutually Reinforcing Context, which is focused on practices related to coordinating and aligning afterschool programming and activities with the regular school day, family, and community contexts; and (4) Youth Outcomes Leading Indicators, which are focused on the change in youths proficiency in reading/english language arts (ELA) and mathematics. Organizational Context Leading indicators within the Organizational Context examine both staff development and internal communication and collaboration among program staff. Programs characterized by a supportive and collaborative climate permit staff to engage in self-reflective practice to improve overall program quality. Self-reflective practice is more likely to lead to high-quality program sessions that provide youth with positive and meaningful experiences. Three leading indicators fall under the Organizational Context: (1) Staff Capacity; (2) Continuous Improvement, which is assessed by scales measuring program climate and internal communication and collaboration; and (3) Leadership and Management. Instructional Context Leading indicators in the Instructional Context focus on the practices and approaches adopted by frontline staff to design and deliver activity sessions that intentionally support youth skill building and mastery that align with centers objectives and principles of youth development. There are two leading indicators in the Instructional Context: (1) Quality of Instructional Content and (2) Quality of Instructional Processes/Strategies. Mutually Reinforcing Context The Mutually Reinforcing Context focuses on relationships between the 21st CCLC program and context external to the program that significantly impact the success of the program. Community partners, families, and schools play an important role in the 21st CCLC programs by expanding program activities, facilitating program sustainability, and providing important information about student needs. Three leading indicators are associated with the Mutually Reinforcing Context: (1) Family Engagement, (2) School Context, and (3) Community Context. Youth Outcomes Leading Indicators The Youth Outcomes Leading Indicators focus on whether students who regularly attend 21st CCLC programming (defined as more than 30 days of attendance during the programming Washington 21st CCLC Program Year 2 Evaluation: Executive Summary iv

11 period) shifted between state proficiency categories in reading/ela and mathematics between the and school years. Evaluation Questions and Methods Evaluation Questions A key objective of the evaluation was to understand how well centers were implementing research-supported best practices and approaches and to assess the impact of 21st CCLC participation on students academic and behavioral outcomes. Specifically, the evaluation was designed to answer the following evaluation questions: 1. What were the primary characteristics associated with the grants and centers funded by 21st CCLC and the student population served by the program? 2. To what extent is there evidence that centers funded by 21st CCLC implement researchsupported practices related to quality afterschool programming? 3. To what extent is there evidence of a relationship between center and student characteristics and the likelihood that students demonstrated better performance on program attendance and youth outcomes, with a particular emphasis on exploring the relationship between leading indicator status and these outcomes? 4. To what extent is there evidence that students participating in services and activities funded by 21st CCLC demonstrated better performance on youth outcomes as compared with similar students not participating in the program? Methods Data Sources To address the evaluation questions, the evaluation team collected data from the following sources: 21st CCLC Profile and Performance Information Collection System (PPICS). Data collected through the Annual Performance Report (APR) module of PPICS on grantee, center, and student characteristics were extracted from PPICS. Youth Outcome and Related Data From Comprehensive Education Data and Research System (CEDARS). Academic and demographic information for 21st CCLC participants and nonparticipants attending the same schools as 21st CCLC participants were pulled from the CEDARS database. Site Coordinator Survey. Site coordinators were surveyed about a variety of program operations related to implementation of best practices in quality afterschool programming. Staff Survey. Program staff were surveyed about a variety of program operations related to implementation of best practices in quality afterschool programming. Youth Program Quality Assessment (YQPA) Data. Program staff completed surveys, self-assessments, and observations as part of a quality assessment improvement program to support grantees completing the Youth Program Quality Improvement (YPQI) process. Washington 21st CCLC Program Year 2 Evaluation: Executive Summary v

12 Analysis Descriptive analysis of PPICS data on grantee, center, and student characteristics along with cluster analysis techniques were used to provide an overall description of Washington 21st CCLC operating in the school year. Both descriptive analysis and Rasch analysis of site coordinator and staff survey responses were used to assess the extent to which centers implement research-supported best practices aligned with the previously described leading indicator system. In order to group centers into clusters on the basis of their scores on leading indicator data, hierarchical cluster analysis was used. Correlational multilevel modeling techniques were employed to explore the relationship between student- and center-level characteristics associated with 21st CCLC sites and youth outcomes. Finally, a propensity score matching approach was used to assess the impact of 21st CCLC programming on youth outcomes by comparing participants with similar nonparticipants from the same schools. Summary of Key Findings A summary of key findings within each of the identified evaluation questions is provided. 1. What were the primary characteristics associated with the grants and centers funded by 21st CCLC and the student population served by the program? Grantee Characteristics A total of 55 Washington 21st CCLC grantees were active during the school year. A majority of grantees (80 percent) were considered mature grants not in the first or last year of the five-year funding cycle. Grantees were roughly split between the categories of school-based (53 percent) and nonschool-based (47 percent) grantee. Center Characteristics A total of 183 centers were in operation across the 55 active grantees for the school year. A majority of centers (96 percent) were school based. Centers mainly served children in elementary school (37 percent) and middle school (34 percent); 14 percent of centers served high school students. Centers provided an average of 4.4 days of programming a week over eight months. Roughly half of centers targeted students for enrollment due to students low performance on local or state assessments. A total of 3,029 staff members worked in centers for the school year. Centers most commonly employed a mix of mostly school day teachers, other school staff, and college students (42 percent), and mostly school day teachers (28 percent). Washington 21st CCLC Program Year 2 Evaluation: Executive Summary vi

13 A majority of centers offered mostly enrichment activities (45 percent) or a variety of activities (27 percent). A total of 24,379 students attended 21st CCLC programming for at least one day. Of the total 21st CCLC participants, a majority (61 percent) were regular attendees (attended for 30 days or more). On average, 21st CCLC regular participants attended 60 days of programming. Overall, centers had approximately 82 regular attendees and 133 total attendees. A majority of 21st CCLC participants were Hispanic (45 percent) or white (36 percent). Most attendees (71 percent) qualified for free or reduced-price lunch, 19 percent were classified as limited English proficient, and 11 percent were classified as special needs. 2. To what extent is there evidence that centers funded by 21st CCLC implement researchsupported practices related to quality afterschool programming? As previously noted, leading indicators of afterschool program quality were developed to examine how well centers implemented research-supported best practices. Findings related to Evaluation Question 2 are summarized according to the overarching contexts for the leading indicators and specific leading indicators within each context. Organizational Context Staff Capacity. Active participation in professional development and training is essential in supporting staff capacity. Center staff were asked to report on the frequency and type of professional development/training attended during the school year. A majority of staff (63 percent) reported participating in some form of training. The most common topic of the trainings attended by staff included strategies for delivering high-quality academic enrichment activities and activities to support youth development. Continuous Improvement. This leading indicator includes the following aspects of continuous program improvement: (1) program climate, (2) internal communication as reported by site coordinators, and (3) internal communication as reported by center staff. Key findings within these aspects of continuous program improvement are summarized below. Program Climate. The average scale score for program climate reported by center staff fell within the agree response category (scale response categories included strongly disagree, disagree, agree, and strongly agree), suggesting that most staff reported supportive, collaborative program climates. A majority of centers (76 percent) fell in the agree response category, and 22 percent fell in the disagree or strongly disagree response category. Center staff were most likely to disagree with the following statements: (1) there is adequate time to plan individual activity sessions and (2) staff participated fully in program decision making. Internal Communication Site Coordinator. For site coordinators, the average scale score for internal communication fell within the a couple of times per year response category (response categories included never, a couple of times per year, about once a month, and nearly every week), suggesting that practices related to internal communication were implemented on an Washington 21st CCLC Program Year 2 Evaluation: Executive Summary vii

14 infrequent basis. A majority of site coordinator scale scores (65 percent) also fell in the a couple of times per year response category. Internal Communication Center Staff. For center staff, average center scale scores fell within the a couple of times per year response category. However, 27 percent of centers fell within the a couple of times per year response category, and 57 percent of centers fell in the about once a month response category. These results suggest that staff are slightly more likely to engage in strategies for internal communication with other program staff as opposed to engaging in internal communication strategies with their site coordinators. Center staff were least likely to report using data to set program improvement goals with other staff, a shift from when the least frequently implemented activity was observing other afterschool staff delivering programming. Leadership and Management. The average scale score reported by staff fell in the three category (response categories included one, three, and five from the YPQA), suggesting that most staff reported that the center s leadership and management support youth-staff relationships and a positive development focus, promote staff development, and are committed to ongoing improvement. A majority of centers (77 percent) fell in the three category, and 10 percent fell in the five response category. Instructional Context Quality of Instructional Content. Three separate scales were used to assess aspects of programming related to the quality of instructional content: (1) alignment of program activities with program objectives, (2) intentionality in program design as reported by site coordinators, and (3) intentionality in program design as reported by center staff. Key findings within each of these aspects of quality of instructional content are summarized below. Alignment of Program Activities With Program Objectives. To assess the extent of alignment between program objectives and program activities, site coordinators were asked to provide the top three program objectives, and steps were taken to assess how frequently sites delivered activities aligned with their top three program objectives. For example, it is expected that programs identifying improving students grade-level proficiency as a top objective would spend a significant amount of time on academic enrichment activities. From this analysis, results indicated that a majority of centers (90 percent) delivered activities aligned with their identified top three program objectives. Intentionality in Program Design Site Coordinator. Site coordinators were asked to report how frequently staff leading program activities engaged in strategies reflective of intentional program design. Average site coordinator scale scores fell in the frequently response category (response categories included never, sometimes, frequently, and always), suggesting that site coordinators felt practices related to intentional service delivery are commonly adopted by activity leaders. Forty-five percent of site coordinator responses fell in the frequently category. Intentionality in Program Design Center Staff. Center staff members were asked to report how frequently they engage in strategies indicative of intentional program design. The average center scale score also fell in the frequently response category. A majority of centers (74 percent) Washington 21st CCLC Program Year 2 Evaluation: Executive Summary viii

15 fell in the frequently response category. This indicates that staff are slightly more likely to report engaging in practices related to intentional program design relative to site coordinator reports of how frequently staff engage in the same practices. Quality of Instructional Processes/Strategies Four separate scales were used to assess aspects of programming related to the quality of instructional processes/strategies: (1) point of service quality, (2) youth-centered policies and practices, (3) youth ownership according to the site coordinator survey, and (4) youth ownership as reported on the staff survey. Key findings within each of these aspects of quality instructional processes/strategies are summarized below. Point of Service Quality. The average scale scores for the overall point of service quality fell within the functioning near optimal category. Percentages of staff respondents stating that point of service quality, safe environment, and supportive environment were functioning near optimal were 81 percent, 100 percent, and 89 percent, respectively, although a majority of responses fell in the still room for improvement category for interaction and engagement. Youth-Centered Policies and Practices. The average scale scores for this leading indicator fell within the three category (response options included one, three, and five), with a majority of centers (76 percent) falling in this category and 10 percent falling in the one category. These responses suggest that most staff report that programs tap youth interests, build youths skills, and involve youth in the structure and policy of the program. Youth Ownership Site Coordinator. The average site coordinator scale score fell in the disagree response category (response options were strongly disagree, disagree, agree, and strongly agree). However, in looking at the distribution of site coordinator scale scores, a majority (51 percent) of site coordinators fell in the agree category. There may be room for growth in defining more organizational- or state-level strategies for cultivating youth ownership. Opportunities for Youth Ownership Center Staff. The average staff survey scale scores fell in the agree response category. A majority of centers fell in the agree category (60 percent), and 37 percent fell within the disagree category. Staff Capacity to Create Interactive and Engaging Settings. Staff were asked to rate the collective capacity of frontline staff to create and provide interactive and engaging program settings. Average staff scale scores fell in the agree category, suggesting that staff generally agree that frontline staff adopt strategies likely to produce interactive and engaging settings. With regard to the distribution of centers across response categories, a majority of centers (70 percent) fell in the agree response category, and 12 percent fell in the disagree category. Service Delivery Practices. Questions on the service delivery practices scale asked staff to report on practices they adopt in their own work with youth. The average staff scale score fell in the available occasionally response category. Likewise, 56 percent of centers fell in the available occasionally response category. Washington 21st CCLC Program Year 2 Evaluation: Executive Summary ix

16 Mutually Reinforcing Context The Mutually Reinforcing Context focuses on relationships between the 21st CCLC program and context external to the program that significantly impacts the success of the program. Three leading indicators are associated with the Mutually Reinforcing Context: (1) family engagement, (2) school context, and (3) community context. Family Engagement. Survey questions on the site coordinator survey assessed center approaches to communicating with families. The average family communication scale score fell within the sometimes response category (response options were never, sometimes, and frequently), which is indicative of programs typically communicating with families once or twice a semester. A majority (74 percent) of site coordinator responses fell in the sometimes response category. School Context. This leading indicator is meant to capture the degree to which 21st CCLC staff members align the design and delivery of programming to the school day and individual student needs. Survey questions related to linkages to the school day and data use were asked on the site coordinator and staff surveys. The average site coordinator scale score fell within the minor strategy response category for linkages to the school day, indicating that most sites employed only a portion of strategies for establishing linkages with the school day. Fifty-three percent of site coordinator respondents fell within the minor strategy response category, and 32 percent fell within the major strategy category. For staff responses, the average scale score fell within the disagree response category, suggesting that, on average, most staff have an incomplete sense of both student academic needs and school day curriculum and/or instructions. Sixty-five percent of centers fell in the agree response category, and 31 percent fell in the disagree category. The average scale score for data use for both site coordinators and staff fell in the occasionally use category, suggesting that the degree to which they use data is limited. Seventy-three percent of site coordinators and 55 percent of staff responses fell into this category. Community Context. The leading indicator for community context is meant to capture the degree to which partners associated with the center are actively involved in planning, decision making, evaluating, and supporting program operations, as well as the extent to which the program adopts practices supportive of family and community engagement. The average site coordinator scale score on the partner involvement scale fell within the do informally response category (response options included did not do, do informally, and do formally). Generally, although centers work with partners in many ways, they have a tendency to do so on an informal basis as opposed to following formal policies and procedures. Of activities that site coordinators engage in with partners, 19 percent do so formally, while 60 percent do so informally. It is also important to note that 112 centers had partners that are actively involved in the provision of programming directly to youth. The average scale score for the family and community engagement scale, which is meant to capture barriers to family and community involvement in the program, fell into the three category (response options included one, three, and five). Seventy-seven percent of staff respondents fell into this category, suggesting that a majority of staff reported policies at their centers that promote family and community engagement in the program. Washington 21st CCLC Program Year 2 Evaluation: Executive Summary x

17 Youth Outcomes The leading indicator is meant to capture descriptively the extent to which students participating in the program moved from one state proficiency category to another between the and school years. For example, 14 percent of regular attendees moved from the well below standard category to the below standard category in the school year in mathematics. 3. To what extent is there evidence of a relationship between center and student characteristics and the likelihood that students demonstrated better performance on program attendance and youth outcomes, with a particular emphasis on exploring the relationship between leading indicator status and these outcomes? Correlational relationships between center- and student-level characteristics and youth outcomes were explored in order to examine whether centers leading indicator status was related to program attendance, academic performance, and unexcused absences. It was hypothesized that there here would be a negative correlation between center membership in clusters where scores on all leading indicators were below average and youth outcomes. Key findings include: Membership in the Instructional Context Content Below Average cluster was negatively associated with reading scores, as measured through students reading state assessments. Membership in this cluster was also positively associated with unexcused absences. Each of the findings was consistent with what was hypothesized. Center membership in the Organizational Context Below Average cluster was negatively associated with program attendance (this was consistent with what was hypothesized), while displaying a positive relationship with credits earned and cumulative GPA (which was not expected). Membership in the Instructional Context Process Below Average cluster was found to be significantly and positively related to program attendance. This is surprising but may be related to the high number of elementary only programs in this cluster, as elementary programs often have higher attendance than middle or high school programs. The number of days of attendance in 21st CCLC programs was found to have a positive relationship with academic performance-related outcomes. 4. To what extent is there evidence that students participating in services and activities funded by 21st CCLC demonstrated better performance on youth outcomes as compared with similar students not participating in the program? Propensity score matching was employed to examine the impact of 21st CCLC programming on participants as compared to nonparticipants with similar characteristics from the same schools. Outcomes explored included academic performance and unexcused absences. Key findings are summarized as follows: Small but significant positive effects were found for reading and mathematics achievement at both 30-day and 60-day participation levels when pooled across grades. Students in the treatment group with 30-day participation achieved standard deviation units higher on reading and standard deviation units higher on mathematics than nonparticipants. For 60-day participation group, 21st CCLC Washington 21st CCLC Program Year 2 Evaluation: Executive Summary xi

18 participants scored and standard deviation units higher than nonparticipants on reading and math, respectively. There was a significant positive impact of the program on the cumulative GPA of students with 60+ days of treatment; the cumulative GPA of this group was standard deviation units higher than the comparison group. A significant positive effect for this group was also seen for percentage of credits earned. The participant group showed a statistically significant, negative impact of the 21st CCLC programming on unexcused absences. Students in both the 30-day and 60-day treatment groups had unexcused absences of 66 percent and 39 percent of the level in the nonparticipant group, respectively. Impacts varied across grade levels. For example, a significant negative impact on cumulative GPA was found for students in the 60-day treatment group in Grade 9, and there was a significant positive impact on students in Grades 10 and 11. Recommendations This report s findings on leading indicators, correlational relationships, and impact analyses provide guidance for grantees on areas for continued growth in the upcoming years, including (1) using data to inform services for individual students, (2) allowing staff more time for planning and preparation, and (3) identifying ways to incorporate more youth ownership into the program at grade-appropriate levels. These results are very similar to those identified in the Year 1 report. In addition, there appears to be some evidence that (a) there are opportunities for growth in terms of how centers go about designing and delivering activities from a content perspective and (b) that enhanced levels of practice in this area are related to better school-related outcomes. Although OSPI has an infrastructure for supporting instructional quality from a process perspective, it may want to give consideration to the types of supports it could provide to enhance the manner in which 21st CCLC programs support the cultivation of skills and knowledge from a content perspective, particularly in relation to the needs of participating youth. Although a variety of positive program effects were demonstrated in this year, OSPI is interested in further exploring the types of impacts 21st CCLC is having on social-emotional learning, 21st century skills and competencies, and noncognitive outcomes. Toward this end, in Year 3 of the evaluation, AIR will be working to collect information from grantee project directors on what they believe their programs are impacting in these areas and what their priorities should be in terms testing measurement strategies to assess program impact on such outcomes. Steps will be taken to select a sample of instruments designed to measure high-priority outcomes and pilot those in a small number of centers during spring semester of the school year. Finally, the leading indicators represent a substantial investment of time and effort to provide Washington 21st CCLC grantees with actionable data to guide and support program improvement efforts. A key goal of the Year 3 evaluation will be to better understand the efficacy of these tools as a vehicle for supporting quality improvement efforts and to highlight portions of the system that are proven to have especially high value to grantees and OSPI. Washington 21st CCLC Program Year 2 Evaluation: Executive Summary xii

19 Chapter 1: Introduction Throughout the past nine years, 21st Century Community Learning Centers (21st CCLC) in the state of Washington have provided afterschool and expanded learning programming to enhance the academic well-being of students in high-poverty communities. This report highlights how well afterschool programs (funded by 21st CCLC, subsequently referred to as centers) throughout Washington have fared relative to meeting the goals and objectives for supporting student growth and development as specified by the Washington Office of Superintendent of Public Instruction (OSPI). Information discussed in the following sections is based on data collected and analyzed by (AIR) and the David P. Weikart Center for Youth Program Quality (Weikart Center) as part of a statewide evaluation of Washington 21st CCLC programs. The results represent findings from Year 2 of a three-year statewide evaluation, which will conclude in January Evaluation Questions A key objective of the statewide evaluation of Washington 21st CCLC-funded programming was to understand both how well centers were implementing programming in terms of research-supported practices and approaches and what impact participation in 21st CCLC-funded activities had on student academic outcomes. More specifically, the evaluation was designed to answer the following set of evaluation questions: 1. What were the primary characteristics associated with both centers funded by 21st CCLC and the student population served by the program? 2. To what extent was there evidence to suggest that centers funded by 21st CCLC had adopted research-supported practices related to the provision of quality afterschool programming? 3. To what extent is there evidence of a relationship between center and student characteristics and the likelihood that students demonstrated better performance on program attendance and youth outcomes, with a particular emphasis on exploring the relationship between leading indicator status and these outcomes? 4. To what extent is there evidence that students participating in services and activities funded by 21st CCLC demonstrated better performance on youth outcomes as compared with similar students not participating in the program? Reasoning for Chosen Evaluation Questions: Importance of Program Quality Collectively, the domain of evaluation questions represents both the goals and objectives OSPI has specified for the 21st CCLC program and emerging issues across the national landscape of afterschool programming. For example, Granger (2008) notes that afterschool research often demonstrates mixed and inconclusive results regarding the impact of participation in afterschool programming on students academic and behavioral outcomes. Granger (2008) cites three noteworthy meta-analyses of the impact of afterschool programming that found a majority of the studies included in each meta-analysis did not find better outcomes for the afterschool participant group relative to the comparison group (Durlak & Weissberg, 2007; Lauer et al., 2006; Zief, Washington 21st CCLC Year 2 Evaluation 1

20 Lauver, & Maynard, 2006). However, both Durlak & Weissberg (2007) and Lauer et al. (2006) found average positive effects of participation in afterschool programming on students academic and nonacademic outcomes, which they attributed to higher quality programs driving the overall positive outcomes. In short, average positive outcomes across several afterschool programs were likely due to the effectiveness of a small number of high-quality individual programs. These findings highlight a key relationship between the quality of afterschool programming and the attainment of desired program outcomes. Although meaningful progress has been made in understanding elements of quality in afterschool programming (e.g., Granger, Durlak, Yohalem, & Reisner, 2007; Little, 2007; Vandell et al., 2005; Wilson-Ahlstrom & Yohalem, 2007; Yohalem, Wilson-Ahlstrom, Fischer, & Shinn, 2009), these understandings have largely been used to support the development of quality assessment tools and improvement systems to help afterschool programs better understand (1) criteria for afterschool program quality, (2) how well they measure up to identified criteria, and (3) steps that can be taken to modify programming and enhance program quality. This reflects the stance of leading researchers that the most pressing issue before the afterschool community is developing effective quality-improvement systems (Granger et al., 2007). OSPI, in collaboration with the Weikart Center, has taken steps to craft a quality assessment improvement system and support grantees in completing the Youth Program Quality Improvement (YPQI) process, which utilizes a youth development framework to combine selfassessment, action planning, skill development, and targeted technical assistance to enhance program quality. To address Evaluation Question 2 as noted previously, this year s report summarizes and expands the leading indicators introduced in the Year 1 report. The leading indicators developed as part of the statewide evaluation are meant to further complement programs participation in the YPQI process, to provide additional information regarding how well they are progressing in implementing research-supported practices, and more importantly, to identify areas of program operations in need of improvement. OSPI s use of the YPQI process and leading indicators provides 21st CCLC programs with an infrastructure to make data-driven decisions about program improvement in a timely, meaningful, and systematic way. As indicated, one of the goals of the statewide evaluation is to explore the relationship between measures of program quality, as measured by the leading indicators, and student academic and behavioral outcomes. Exploring this relationship is especially helpful in refining the leading indicator system according to measures of program quality that relate to student outcomes. Organization of Report The following sections provide a summary of the methods, including data sources and analytic techniques, to address the primary evaluation questions. Following an overview of the evaluation methods, key grantee and center characteristics are summarized, with a particular emphasis on characteristics that are considered key to improving student academic achievement and attaining desired program outcomes. The leading indicator system is then summarized and explained with regard to how information relates to future evaluation and technical assistance efforts. Finally, analyses for assessing relationships between center- and student-level characteristics and student outcomes, as well as for evaluating the impact of 21st CCLC participation on student-level outcomes are summarized, along with conclusions and recommendations to guide future evaluation and program improvement efforts. Washington 21st CCLC Year 2 Evaluation 2

21 Chapter 2: Methods Data Sources and Analysis Data collected and analyzed in this report come from four primary sources, including administrative data systems and surveys. Each data source and associated methods of data analysis are described. 21st CCLC Profile and Performance Information Collection System (PPICS) PPICS is a Web-based data collection system developed and maintained by AIR on behalf of the U.S. Department of Education. Data on the full domain of 21st CCLC programs funded nationally, including those in Washington, are collected through this system. Data collected through the Annual Performance Report (APR) module of PPICS on center characteristics in relation to the programming period were extracted from PPICS and utilized in several analyses contained in this report, including information on program operations, staffing, activities provision, and student attendance rates. A total of 184 programs associated with 55 active 21st CCLC grantees, during the programming period, were represented in the data set extracted from PPICS. (Note: A single 21st CCLC grant typically has more than one program associated with it.) Youth Outcome and Related Data From CEDARS AIR constructed a unique data collection module for Washington integrated within PPICS that allowed for the collection of student-identifiable information that was extracted from the system and provided to OSPI. OSPI used this information to perform a series of merges against state data warehouses to obtain Measurements of Student Progress (MSP) reading and mathematics scores, High School Proficiency Exam (HSPE) reading scores, cumulative GPA, credits earned, and the number of unexcused absences, as well as additional demographic information about the students in question from the Comprehensive Education Data and Research System (CEDARS), a longitudinal data warehouse of educational data maintained by OSPI. OSPI also identified students not participating in 21st CCLC programming who attended the same schools as 21st CCLC participants and provided the same testing and related CEDARS information for these students. These data were used both to conduct the impact analyses predicated on comparing 21st CCLC participant with nonparticipant outcomes and to construct the set of models needed to explore the relationship between center and student characteristics and student achievement and related outcomes. Site Coordinator Survey An online survey of site coordinators working in 21st CCLC programs active during the school year was administered in spring The site coordinator was defined as the individual at a given center who was responsible for the day-to-day operations of the program and was the initial point of contact for parents and staff when questions or issues arose on-site. Generally, site coordinators are seen as important middle managers in the delivery of 21st CCLC programming at a given site. Washington 21st CCLC Year 2 Evaluation 3

22 A total of 184 site coordinator surveys were administered. Completed surveys were received from 173 site coordinators, for a response rate of 94 percent. The survey addressed the extent to which centers engaged in practices that the research indicates are supportive of effective afterschool programming. Sets of survey questions were organized to create scales measuring the following dimensions of program operations: Program objectives Activity enrollment policies and recruitment approaches Access to and use of student data Linkages to the school day Staffing approach and challenges Other operational challenges Intentionality in activity and session design Creation of interactive and engaging settings for youth Opportunities for youth ownership Internal communication designed to support program development and improvement Practices supportive of cultivating effective partnerships Practices supportive of parent involvement and engagement Professional development and training Data obtained from the site coordinator surveys were used both to support the leading indicator process and to construct variables included in analyses to assess the relationship between center characteristics and 21st CCLC participant outcomes. Staff Survey The purpose of the online staff survey was to obtain information from frontline staff who worked directly with youth during the school year. A particular focus of the survey was on practices that support both positive academic outcomes and youth development outcomes. As with the site coordinator survey, the staff survey included sets of questions associated with a given scale, as well as open-ended questions to assess dimensions of program operations. Dimensions of program operations assessed on the staff survey included the following: Program objectives Intentionality in activity and session design Practices supportive of academic skill building, including linkages to the school day and using data on student academic achievement to inform programming Practices supportive of positive youth development Opportunities for youth ownership Internal communication designed to support program development and improvement Washington 21st CCLC Year 2 Evaluation 4

23 Program climate in terms of how staff view the organizational supports and structures as supporting their work with youth Training participation Completed surveys were received from 1,090 center staff from 181 centers. The number of completed staff surveys received per center ranged from one to 19, with an average of six completed surveys per center. As with the site coordinator survey, data obtained from the staff surveys were used to support the leading indicator process. Youth Program Quality Assessment (YPQA) Data As noted previously, OSPI, in collaboration with the Weikart Center, has taken steps to craft a quality assessment improvement system and support grantees in completing the YPQI process. As part of this process, observations were conducted by program staff as a self-assessment or by trained external observers of activities provided by 21st CCLC grantees, and the YPQA Form A was scored to provide an estimate of how safe, supportive, interactive, and engaging the observed session was for participating youth. In addition, although the YPQA Form A is meant to measure program quality at the point of service, the YPQA Form B is a rubric completed by program staff on how well the program has adopted organizational processes that are likely to engender and facilitate point of service quality. Both YPQA Form A and B data were uploaded directly to the Weikart Center via the Center s online score reporter. It is important to note that participation in the YPQI process was voluntary for Washington 21st CCLC grantees during the school year. As a result, PQA Form A data were available for only 72 centers associated with 33 grantees. Form B was provided in relation to 62 centers associated with 32 grantees. Analytic Approach and Methods It is important to note that the findings outlined in this report are primarily quantitative in nature. This approach was largely driven by both the evaluation questions being answered and the resources available to carry out the project during Year 2 of the project. Analyses highlighted in this report fall within five general categories: 1. Descriptive Analyses. Information related to grantee, center, and student characteristics obtained from PPICS, the surveys, and the PQA were analyzed descriptively to explore the range of variation on a given characteristic. Some of the leading indicators also were calculated employing descriptive analysis techniques. 2. Analyses to Create Scale Scores. Many questions appearing on the site coordinator and staff surveys underpinning the leading indicators were part of a series of questions designed to assess an underlying construct/concept, resulting in a single scale score summarizing performance on a given area of practice or facet of 21st CCLC afterschool implementation (e.g., practices that support linkages to the school day). An example is shown Figure 1, which outlines the questions making up the Intentionality Program Design scale that appeared on the site coordinator survey. Washington 21st CCLC Year 2 Evaluation 5

24 Figure 1. An Example of a Survey Scale Calibrated Using Rasch Techniques For scales such as this, Rasch scale scores were created using staff and site coordinator responses to a series of questions to create one overall score. These scale scores ranged from 0 to 100, where higher scores were indicative of a higher level or more frequent adoption of a specific quality practice or set of practices. Scale scores resulting from the application of Rasch approaches can also be used to classify what portion of the rating scale the average scale score fell within. For example, the statewide mean value for the Intentionality in Program Design scale highlighted in Figure 1 was 59.82, which put the statewide average in the frequently range of the scale indicating the typical staff member responding to the survey reported engaging in these practices on a frequent basis. This approach also allowed the evaluation team to explore the distribution of centers in light of what response option their average scale score put them in. The primary benefit of this approach is the capacity to distill responses from several questions down into one overall score for the center, simplifying the process of interpreting how a center did on a given element of quality, particularly in relation to other programs in the state. 3. Hierarchical Cluster Analysis. Hierarchical cluster analysis was employed to combine centers into groups based on how well they scored on the leading indicator data collected Washington 21st CCLC Year 2 Evaluation 6