ExCEL After School Programs: Evaluation

Transcription

1 Critical Investments in Support of Continuous Quality Improvement

2 ACKNOWLEDGEMENTS We would like to acknowledge the following people for their active contribution to the evaluation and report: We are grateful to the ExCEL office and additional employees of San Francisco Unified School District for their support of this evaluation. Mele Lau-Smith, Executive Director of Family Engagement and Community Partnerships Karina Henriquez, ExCEL Program Administrator Lucy Hong, ExCEL Fiscal and Program Lead Tanya Avila, Quality Improvement Consultant Amy Adkins, Health and Wellness Specialist, ExCEL Jovanna Badger, ExCEL District Coordinator Joyce Cheung, ExCEL Budget Administrator Jeanette Coleman, ExCEL Support Staff Kitty Ou, Senior Statistician, Research, Planning and Accountability Department Karen Polk, ExCEL District Coordinator Jennifer Quevedo-Serrano, ExCEL District Coordinator Alice Tam, ExCEL District Coordinator Wendy Tran, ExCEL Support Staff The Principals, Academic Liaisons and school day staff at the ExCEL sites throughout the District We also extend our thanks to the Agency Directors, Site Coordinators, and Line Staff of the community agencies that provide services to the children and youth of San Francisco every day. Their active participation in the evaluation is key to the success of this report. Public Profit Evaluation Team Jessica Manta-Meyer, Senior Research Associate and Project Lead Linda Lu, Research Associate Emily Salvaterra, Research Assistant Saili Willis, Project Assistant Lindsay Cattell, Graduate Intern Justine Wolitzer, Graduate Intern Corey Newhouse, Project Director, Founder and Principal Page 1

3 Table of Contents ACKNOWLEDGEMENTS 1 PUBLIC PROFIT EVALUATION TEAM 1 KEY FINDINGS 4 ABOUT SAN FRANCISCO EXCEL 6 THEORY OF ACTION 7 WHAT MAKES A QUALITY PROGRAM? 8 PROGRAM PARTICIPATION 10 EXCEL PROGRAMS SERVED A MORE VULNERABLE POPULATION COMPARED TO THE DISTRICT AS A WHOLE. 10 EXCEL PROGRAMS SERVE YOUTH WITH BOTH INTERNAL AND EXTERNAL MOTIVATION TO PARTICIPATE 12 YOUTH ATTENDED REGULARLY IN THE EARLY GRADES, BUT REGULAR PARTICIPATION DROPPED OFF IN THE OLDER GRADES 13 EXCEL PROGRAMS MET GRANT REQUIREMENTS 15 PROGRAM QUALITY 16 EXCEL PROGRAMS PROVIDE A SAFE PLACE FOR STUDENTS THROUGH THE USE OF RESTORATIVE PRACTICES 16 EXCEL PROGRAMS FOSTER POSITIVE RELATIONSHIPS SUPPORTIVE ADULTS AND POSITIVE PEER RELATIONSHIPS AND RESPECT FOR DIVERSITY 21 EXCEL PROGRAMS PROVIDE CARING ADULTS IN PARTICIPANTS LIVES 22 EXCEL PROGRAMS CAN SUPPORT POSITIVE PEER RELATIONSHIPS 23 EXCEL PROGRAMS RESPECT DIVERSITY, BUT NEED TO CREATE MORE OPPORTUNITIES TO INCORPORATE DIVERSITY PRACTICES. 25 EXCEL PROGRAMS PROVIDE OPPORTUNITIES FOR YOUTH TO LEARN AND DEVELOP MASTERY, BUT COULD PROVIDE STRONGER OPPORTUNITIES FOR YOUTH VOICE AND LEADERSHIP 26 EXCEL PROGRAMS PROMOTE SKILL-BUILDING 26 MANY, BUT NOT ALL, YOUTH HAVE OPPORTUNITIES TO LEAD AND MAKE CHOICES IN EXCEL PROGRAMS 27 ALIGNMENT WITH THE SCHOOL DAY 29 MANY EXCEL PROGRAMS SUPPORT SCHOOL DAY GOALS, BUT SCHOOL DAY AND AFTER SCHOOL STAFF SHOULD CONTINUE TO STRENGTHEN THESE PARTNERSHIPS THROUGH REGULAR MEETINGS AND SUPPORT FROM THE ACADEMIC LIAISON 29 FAMILY ENGAGEMENT 32 PARENTS OF EXCEL PROGRAMS FEEL RESPECTED BY STAFF, BUT ARE UNCERTAIN ABOUT THEIR OPPORTUNITIES TO PARTICIPATE. 32 Page 2

4 PARENTS FEEL MOST POSITIVELY TOWARDS PROGRAMS WHERE THEIR CHILDREN FEEL SAFE, SUPPORTED, AND PREPARED ACADEMICALLY. 33 DIRECT BENEFITS TO YOUTH 34 YOUTH GET A CHANCE TO TRY NEW THINGS IN THEIR EXCEL PROGRAM 34 PROGRAM PARTICIPANTS FEEL MORE ENGAGED WITH THEIR SCHOOL, A CRITICAL STEP TOWARD IMPROVING ACADEMIC ACHIEVEMENT 36 EXCEL PROGRAMS SHOULD STRENGTHEN LEARNING STRATEGIES AND ACADEMIC BEHAVIORS 38 MODEST CONTRIBUTORY BENEFITS: EXCEL PROGRAMS MAY INCREASE SCHOOL DAY ATTENDANCE, SUPPORT ENGLISH LEARNER REDESIGNATION, AND REDUCE CHRONIC ABSENTEEISM 40 EXCEL PARTICIPANTS ATTEND SCHOOL MORE OFTEN THAN THEIR NON-PARTICIPANT PEERS 40 INCREASED PARTICIPATION IN EXCEL CAN HAVE A MODEST IMPACT ON ENGLISH LEARNER REDESIGNATION RATES 41 EXCEL PARTICIPATION MAY SLIGHTLY IMPACT CAHSEE RESULTS 42 APPENDIX A: DETAILED YOUTH SURVEY RESULTS 43 APPENDIX B: DETAILED PARENT SURVEY RESULTS 45 APPENDIX C: DETAILED KEY STAFF SURVEY RESULTS 47 APPENDIX D: EXCEL S IMPACT ON ACADEMIC OUTCOMES 49 SCHOOL DAYS ATTENDED 49 REDESIGNATION RATES 50 CHRONIC ABSENCE RATES 50 CAHSEE PASS RATES 51 SUSPENSIONS 51 SUMMARY 51 LIMITATIONS 51 Page 3

5 KEY FINDINGS In , ExCEL continued its critical investments to improve the quality of ExCEL after school programs with the goal of achieving improved student outcomes for participants. These investments included implementing Restorative Practices, development of Quality Action Plans based on program evaluation data, and redefining the role of the Academic Liaison, or school day staff assigned to support the after school program and facilitate the programs communication with the rest of the school community. The evaluation found evidence that these investments may have had an impact on the youth and the programs based on youth reports of bullying and on school day alignment, school engagement, and academic behaviors. Moreover, there is evidence that participation in ExCEL programs has a modest relationship to school day attendance and chronic absenteeism, as well as on English Learner redesignation. Specific findings include: ExCEL participants attended more days, on average, in than in or Elementary participants attended 136 days in compared to 123 days in High school participants attended 29 program days compared to 24 in ExCEL programs may increase school day attendance, support English Learner redesignation, and reduce chronic absenteeism. Students who attend more days of ExCEL also attend more regular school days. Specifically, the average ExCEL participant attended 0.54 more school days per year as compared non-participants, a finding driven by middle school participants who attended 0.72 more school days per year compared to non-participants. Among ExCEL participants, each extra day attending ExCEL was associated with a higher likelihood of being redesignated in the spring semester. For each extra day of ExCEL attendance, English Learner students increased their likelihood of being redesignated by about times. This result was consistent across grade levels and when controlling for gender and race/ethnicity. ExCEL programs have successfully implemented strategies to support restorative practices. Over two-thirds of youth (73%) report participating in restorative circles and 76% report that staff used the restorative questions with them (when applicable). ExCEL participants continue to report positive relationships with adults in the program and their peers (nearly 90%). However, some youth (about 20%) still do not report opportunities for leadership in their program. Page 4

6 The rates of reported bullying dropped, possibly because of the introduction of Restorative Practices in most ExCEL programs. Fewer than 2 in 10 youth (17%) report they have had mean rumors or lies spread about them, while only 14% report they have been hit or pushed in their programs more than once. Although this is a marked improvement over the previous year where nearly 30% of youth reported verbal or physical bullying, it is still a point of concern. High school youth generally rate their programs of higher quality and report better outcomes compared to their younger peers. However, only 1,088 high school youth completed a survey although nearly 9,000 youth participated in high school programs at some point for an effective response rate of only 12%. This suggests that ExCEL may wish to increase input from high school youth to get a more representative picture. Across several dimensions, English Learners are more likely to rate their programs of higher quality and report better outcomes compared to their peers. Similarly, participants who join the program because they want to join are more likely to rate the programs and their outcomes higher compared to their peers who joined the program because their parents or teachers told them to join. Youth report both increased school engagement and improved academic skills and strategies, both of which are key contributors to academic success. However, some ExCEL programs need to continue to improve in these critical areas. Staff survey results point to a pattern of modest after school - school day alignment. ExCEL may wish to continue to prioritize supports for this alignment as it will likely lead to additional positive outcomes for the youth and the programs Parents of ExCEL programs feel respected by staff, but continue to be uncertain about their opportunities to participate. Nearly 2 in 10 (17%) parents report that they do not know whether there are opportunities for parent participation in ExCEL.. Page 5

7 ABOUT SAN FRANCISCO EXCEL San Francisco ExCEL is the after school programs office of the San Francisco Unified School District (SFUSD), administering and monitoring federal and state funding for school-based after school programs and aligning programming with district goals for student success. The ExCEL after school program office is part of the Student, Family and Community Support Department (SFCSD) which supports students and families, by building school capacity to address student needs, and working to achieve District-wide academic goals. San Francisco ExCEL seeks to increase student achievement and pro-social student behaviors so all students can become high achieving and joyful learners. ExCEL seeks to reach these goals by providing comprehensive after school programs, which consist of a mandated mix of academic, recreational/physical, and enrichment components that are open to all students at the school site for low or no cost. Principals select local community-based organizations to serve as Lead Agencies through a contract with the District. Each agency has demonstrated experience in developing and implementing after school programs and established relationships in their respective communities. In the school year, 22 community-based organizations operated ExCEL programs in 89 schools throughout San Francisco. This included 61 programs that serve elementary aged youth, 19 that serve grades 6-8, and 16 that serve high school youth. 1 1 Some schools serve more than one grade level, such as K-8 schools. Page 6

8 Theory of Action Existing research suggests that young people who come to high quality out-of-school time programs often are most likely to demonstrate positive outcomes in a variety of dimensions, including a wide range of socio-emotional and youth development skills, engagement with school, and improved academic skills and performance. Research also shows that youth who regularly attend high quality programs are more likely to demonstrate strong relationships with peers and with a caring adult, develop new skills, experience new aspects of the world around them, be safer, and be more engaged in school. These positive changes in turn support other positive outcomes for youth, such as enhanced school performance. Young people are most likely to benefit from San Francisco ExCEL programs, therefore, when they regularly attend a high quality program. In this way, San Francisco Unified School District s ExCEL programs can serve as a launching pad for student success, providing a unique program where young people learn and practice important skills and gain new experiences. The figure below provides a visual model of the ways in which San Francisco ExCEL programs contribute to positive outcomes for young people. This Theory of Action informs the evaluation design and the organization of this report. FIGURE 1. THEORY OF ACTION FOR SAN FRANCISCO EXCEL PROGRAMS Regular Participation Program Attendance In High Quality Programs Safe Environment Supportive Adults Positive Peer interactions Promoting Diversity Opportunities to Lead Support for Skill Building School-day aligned Has Direct Benefits for Youth New skills and experiences Improved academic behaviors School engagement Contributes to Other Positive Outcomes School Day Attendance English fluency Academic performance Page 7

9 San Francisco ExCEL programs share common goals. Each program must meet grantstipulated performance requirements, including enrollment and attendance. Each program also strives to provide youth with an environment that is safe, supportive, engaging, and interactive, and to have a direct impact on student s academic behaviors and school engagement that supports the District s goal of graduating students who are career and college ready. In addition to sharing these goals, the programs share a common framework through which progress toward their goals can be measured. The San Francisco ExCEL evaluation combines multiple data sources to explore the extent to which sites are meeting program performance goals, providing high quality services for children and youth, and demonstrating benefits for participants. What makes a quality program? San Francisco ExCEL defines quality programs in alignment with the Expanded Learning Collaborative, a city-wide effort to increase access to and the quality of out-ofschool time opportunities for young people throughout San Francisco. These quality elements include: Safe Environment: Program spaces are physically safe and free of hazards. Moreover, programs are emotionally safe. Youth do not experience either verbal or physical bullying. Programs develop and follow restorative practices through circles, restorative questions and other program elements as a way to respond to conflict and inappropriate behavior and as a way to foster a positive emotional and social climate throughout the program. Positive Relationships: Programs develop strong, positive youth-adult relationships between youth participants and adult program leaders. Programs support positive peer interaction and the development of pro-social skills. Finally, programs build strong communities by promoting respect for diversity through both program norms and behaviors and programming that acknowledges and explores issues of difference and social justice. Opportunities to Learn and Lead: Programs provide strong skill-building and learning opportunities through enrichment and academic activities that move beyond homework help. Participants have opportunities to make meaningful choices within activities and about the program as a whole. The program provides leadership opportunities for all participants. Page 8

10 The table below summarizes the key data sources by report section. TABLE 1. KEY DATA SOURCES BY REPORT SECTION Report Section Measures Data Source Program Participation Counts and characteristics of youth served Average days attended, attendance rates Unique attendances (participant-days) EMS (ExCEL s Cityspan attendance system) California Department of Education s Dataquest SFUSD demographic data on participants Program Quality Safe environment, restorative practices Supportive adults, positive peer relationships, support for diversity Engaging learning and leadership opportunities Experience of program quality by key participant populations Aligning with the school day Engaging families Youth, parent and key staff surveys SFUSD demographic data on participants Direct Outcomes New skills and experiences Increased school engagement Improved academic behaviors Experience of program quality by key participant populations Youth, parent and key staff surveys SFUSD demographic data on participants Contributory Outcomes Increased school day attendance, decreased chronic absenteeism, English learner redesignation rates, CAHSEE pass rates SFUSD academic data on participants and non-participants at school sites with an ExCEL program Page 9

11 PROGRAM PARTICIPATION ExCEL programs served a more vulnerable population compared to the District as a whole. ExCEL programs served 21,967 youth during the school year, or 38% of all students enrolled in SFUSD that year. These youth were generally similar to the SFUSD population, although there were some notable differences. ExCEL programs served a greater proportion of youth who were English Leaners and who were eligible for free or reduced price lunch, compared to the District. More than one-third (34%) of ExCEL participants were English Learners, compared to 28% of SFUSD. Similarly, 65%% of ExCEL participants were eligible for free or reduced price lunch, compared to 61% of SFUSD students. ExCEL programs served a similar proportion of students with Special Needs (11%) compared to the District (12%). Moreover, ExCEL programs served a greater proportion of students from high-need zip codes as determined by the Department of Children, Youth and Their Families in the most recent Children s Services Allocation Plan. 2 In , 52% of ExCEL participants were from the Mission/Bernal Heights (94110), Outer Mission/Excelsior/Ingleside (94112), Bayview/Hunters Point (94124), and Visitacion Valley (94134) (Figure 2), whereas only 47% of all District students were from these neighborhoods. FIGURE 2. PROPORTION OF EXCEL PARTICIPANTS BY HOME ZIP CODE Source: San Francisco ExCEL s Cityspan (EMS) attendance records for 22,498 youth; 568 participants with zip codes outside of San Francisco were not included in this map. Darker green indicates zip codes where a larger numbers of ExCEL participants live. 2 San Francisco Department of Children, Youth and Their Families, (May 2013), Children s Services Allocation Plan , prepared by LFA Group. Page 10

12 ExCEL programs also served a greater proportion of youth who scored below proficient on the California Standards Tests (CSTs) in the prior year. 3 Half of all ExCEL participants scored below proficient on the English Language Arts (ELA) CST and over half (53%) on the Math CST. By comparison, only 39% of all District students scored below proficient on the ELA CST, and only 42% on the Math CST. ExCEL programs served an even greater proportion, compared to the District, of students who are scored particularly low on the CSTs (below basic or far below basic). ExCEL programs served a greater proportion of African American and Latino students compared to the District, and a lower proportion of Asian and White students. In other words, Latino and African American students were somewhat over-represented in ExCEL programs, while White and Asian students were under-represented, as compared to the District. In particular, nearly one-third (31%) of ExCEL participants are Latino, compared to 27% of District students. Taken together, this suggests that ExCEL programs may be a critical venue where the District can reach those students who are, or are likely to be, on the wrong side of the achievement gap. 3 Standardized tests were not administered in Page 11

13 ExCEL programs serve youth with both internal and external motivation to participate Youth who attend ExCEL programs do so for a variety of reasons. Some may join because an adult (e.g., a parent, teacher or principal) in their life wanted them to. Others may join because their friends are in the program. Still others may join out of an internal motivation to do so. For younger youth in ExCEL programs, a majority (57%) joined because their parents wanted them to. One-quarter (25%) reported that it was their idea to join the ExCEL program. The proportion of youth who are internally motivated to join the ExCEL program increases among high school youth. More than half (56%) of youth in high school ExCEL programs joined because they themselves wanted to join. Still, 19% of high school youth report joining because an adult wanted them to join. TABLE 2. AS THEY GET OLDER, YOUTH ARE INCREASINGLY INTERNALLY MOTIVATED TO ATTEND EXCEL PROGRAMS All grades n=3,534 n=1,622 n=1,088 n=6,244 I joined this program because my parents wanted me to % 9% 38% I joined this program because my friends wanted me to. -- 8% 9% 8% I joined this program because my teacher or principal wanted me to. -- 2% 10% 5% I joined this program for another reason. 9% 17% 12% Overall: External motivation to join program (I joined because of parents, friends, teacher/principal, or other) Overall: Internal motivation to join program (I joined because I wanted to) Source: Youth Surveys, Spring % 44% 63% -- 25% 56% 37% Page 12

14 Youth attended regularly in the early grades, but regular participation dropped off in the older grades Research shows that when youth attend structured, high quality programs regularly, they reap the intended benefits. Youth do not need to attend a specific number of days there is no magic attendance number. However, research does suggest that at about 100 days, young people are more likely to benefit from participating in the program. For younger youth in ExCEL programs those in grades K-5 about 78% attend 100 days or more. On average, elementary-aged youth attend 136 days. For middle school youth those in grades 6-8 the average number of program days attended is 90 (Figure 3). Among high school youth, only 5% attended 100 days or more; the average days attended among high school youth was 29 days. ExCEL participants attended more days in , on average, than in prior years. In particular, youth in the lower grades (K-5) attend about 13 more program days on average than in FIGURE 3: AVERAGE DAYS ATTENDED BY GRADE LEVEL K High Schools Overall* Average days attended Average days attended Average days attended Source: San Francisco ExCEL s Cityspan (EMS) attendance records for 22,498 youth ( ), 19,917 ( ), 21,875 ( ); * High overall average in occurs due to a temporary decrease in the number of high school programs in The jump in the number of high school students, who tend to attend far fewer days than other grade levels, pulled down the overall average in Asian youth attended more ExCEL program days than any other racial or ethnic group, attending 93 program days on average. Multiracial youth (91), Pacific Islander youth (90) and Filipino youth (89) attended similarly high numbers of program days on average. White youth tended to attend the fewest program days, attending 77 days on average. Youth classified as English Learners attended more ExCEL program days on average than their peers. English Learners attended 105 program days on average, whereas youth Page 13

15 who are native speakers of English attended 91 days on average and youth redesignated as English Fluent attended 73 days on average. Middle school and high school programs tend to experience high degrees of turnover; that is, the program serves many youth but retains them in the program for only a small number of days. However, middle school students attended, on average, 68% of the program days between their first and last days attended. In other words, while middle school youth are participating in the program, they tend to attend regularly. For example, a student may attend for only a few months, but attend two-thirds of all program days for those few months. This suggests that middle school programs may be able to keep youth coming back to the program regularly for a period they are getting youth to attend at least 3 days a week but that they may want to look for strategies to keep youth engaged in the program over several months or to follow up with youth when they first drop off in attendance. By comparison, high school youth attend, on average, only 38% of available program days between their first and last day of attendance. Page 14

16 ExCEL programs met grant requirements ExCEL after school programs are expected to meet specific attendance targets based on their grant funding amounts. SF ExCEL programs must meet an 85% attendance target (at least 85% of progress toward contracted attendance) established by the California Department of Education (CDE). Of the 74 elementary, K-8, and middle school ExCEL programs,44 elementary schools, 6 K-8 schools, and 6 middle schools met at least 85% of their total program attendance targets for This is similar to the school year, when 41 elementary schools, 5 K-8 schools, and 7 middle schools met their total program attendance targets. Of those that did not meet the attendance target in , the average progress toward contracted program attendance was 75%. Of the 16 high school ExCEL programs, 13 met at least 85% of their total program attendance target for the school year. By comparison, in the school year, 4 of the 8 participating high school ExCEL programs met or exceeded their total program attendance target. However, of those programs that did not meet 85% of contracted attendance in , the average progress toward program attendance was only 56%, similar to last year s 57%. FIGURE 4. PROPORTION OF SITES MEETING ATTENDANCE GOALS DEFINED BY CDE, BY PROGRAM TYPE, AND ES K-8 80% 76% 83% 100% MS 46% 58% HS 50% 81% 0% 20% 40% 60% 80% 100% Met Total Program Attendances Goal Met Total Program Attendances Goal Source: San Francisco ExCEL s Cityspan attendance records for 22,498 youth. In Elementary site n=55; K-8 n=6; Middle School n=13; High School n=16; in Elementary site n=54; K-8 n=6; Middle School n=12; High School n=8. Page 15

17 PROGRAM QUALITY [The program at our school] is an incredible program! It is such a valuable experience for our students between the activities, support, and relationships with the staff and kids. The use of restorative practices and collaborating with day time staff to support behavior and academic needs is awesome and much appreciated. Key Staff (no role selected) ExCEL programs provide a safe place for students through the use of restorative practices The foundation of a successful after school program is an environment in which all students are physically and emotionally safe. Research has demonstrated that students perceptions of danger are associated with their school performance and achievement test scores. 4 That is, students perform better in schools where they feel safe. This suggests that students lower order needs (e.g., safety and well-being) are essential to address before students can attend to higher order tasks such as learning. Programs that provide a safe environment also allow students to feel a sense of belonging and express themselves in authentic ways. Further, environments that promote physical and emotional safety are conducive to positive peer and staff-student relationships. SFUSD Restorative Questions When responding to conflict, a restorative approach consists in asking the people involved these key questions: 1. What happened, and what were you thinking at the time of the incident? 2. What have you thought about since? 3. Who has been affected by what happened, and how? 4. What about this has been the hardest for you? 5. What do you think needs to be done to make things as right as possible? In prior years, about 9 in 10 youth reported that they felt safe in their ExCEL program. However, large numbers (as high as 34% in the younger grades, report the range) reported that they experienced bullying in the programs. These findings were comparable to the school day, suggesting that they reveal a pattern of bullying in the larger school community, not one specific to ExCEL programs. 4 Ratner, H., Chiodo, L., Covington, C., Sokol, R., Ager, J., & Delaney-Black, V. (2006). Violence Exposure, IQ, Academic Performance, and Children's Perception of Safety: Evidence of Protective Effects in Merrill- Palmer Quarterly: Journal of Developmental Psychology, 52(2), Page 16

18 In , however, the rates of reported bullying dropped, possibly because of the introduction of Restorative Practices in most ExCEL programs (see the next section). Less than 2 in 10 youth (17%) report they have had mean rumors or lies spread about them, while only 14% report they have been hit or pushed in their programs more than once. Although this is a marked improvement over the previous year where nearly 30% of youth reported verbal or physical bullying, it is still a point of concern. Most youth (nine in ten) report being able to tell an adult in their program if they see bullying occur (Table 2). Reported incidences of bullying are higher for younger youth compared to older youth. Elementary students report the highest rates of bullying compared to other grade groups. While less than 10% of high school students report being verbally and physically bullied, more than 15% of elementary and middle school youth report having been harassed. Elementary youth continue to need the most support in reducing bullying as more than 20% report being verbally harassed in their programs. Reported incidences of bullying are generally higher for males than females. Females are more likely to respond that they feel safe while 12% of females report having been physically harassed, 16% of males say the same. Moreover, 92% of females report that if they see bullying they can tell an adult, compared to 89% of males. TABLE 3. SAFETY BY GRADE LEVEL All grades n=3,534 n=1,622 n=1,088 n=6,244 If I see bullying, I can tell an adult in this program. v 90% 90% 92% 90% Other kids never/rarely hit or push me in this program when they are not just playing around.* v l Other kids at this after school program never/rarely spread mean rumors or lies about me.* v l 84% 85% 94% 86% 79% 85% 92% 83% Gender difference is v Motivation to join is Ethnicity difference is l English Learner difference is statistically significant Source: Youth Surveys, Spring *For the actual text of these survey questions, which are presented reversed from how they appear on the survey, see the California Healthy Kids Survey examples at CHKS.WestEd.org. Males and English Learners were more likely to report bullying compared to English fluent participants (including both native speakers and those who have become English fluent). African American youth were more likely to report bullying compared to their peers while Asian youth are less likely to report bullying compared to their peers. Page 17

19 ExCEL participants in grades 6 and up indicated whether they joined the program because they wanted to (self-motivated to join the program) or because someone told them to join such as a teacher or parent. Those that were self-motivated were less likely to report being harassed in the program. Parents overwhelmingly perceived their children s programs as being safe. Nearly all parents (98%) agreed that their child s after school program is a safe place. Similarly, 9 in 10 staff agreed that their program is a safe place for their students, and most staff reported that they are equipped to help students when they are bullied. Results from the San Francisco Unified School District s California Healthy Kids Surveys (CHKS) indicate that ExCEL elementary-aged youth in report moderately higher levels of verbal harassment compared to reports from SFUSD. While 52% of elementary-aged youth report never being verbally harassed during their school day, 43% of ExCEL elementary-aged youth report the same. ExCEL elementary programs in report slightly lower levels of physical harassment than elementary-aged youth at the District level. Fifty-one percent of ExCEL elementary youth report never being physically harassed in their program compared to 48% for SFUSD. Results from the SFUSD s CHKS report for middle and high school youth indicate that ExCEL consistently reports moderately lower levels of verbal harassment than SFUSD across all grade groups. As an example, 14% of youth in ExCEL middle schools report being verbally bullied compared to 23% at the District level. Similarly, 8% of high school youth in ExCEL programs report verbal harassment compared to 14% at the District level. FIGURE 5. VERBAL BULLYING IN MIDDLE AND HIGH SCHOOLS - EXCEL AND DISTRICT COMPARISONS How many times have you had mean rumors or lies spread about you? 100% 80% 14% 23% 8% 14% 60% 40% 86% 77% 92% 86% 20% 0% ExCEL SFUSD ExCEL SFUSD Middle School 6-8 High School 9-12 Never, Rarely Sometimes, Often Source: Youth Surveys, Spring 2014; SFUSD California Healthy Kids Survey, HS, Page 18

20 Generally, rates of physical bullying are lower in high schools than middle schools. ExCEL programs report approximately the same rates (6%) of physical harassment as the District (7%). However, ExCEL middle school youth report lower rates of physical harassment (15%) than the District (19%). FIGURE 6. PHYSICAL BULLYING IN MIDDLE AND HIGH SCHOOLS - EXCEL AND DISTRICT COMPARISONS How many times have you been pushed, shoved, slapped, hit or kicked by someone who wasn't just kidding around? 100% 80% 15% 19% 6% 7% 60% 40% 85% 81% 94% 93% 20% 0% ExCEL SFUSD ExCEL SFUSD Middle School 6-8 High School 9-12 Never, Rarely Sometimes, Often Source: Youth Surveys, Spring 2014; SFUSD California Healthy Kids Survey, HS, Since 2009, the San Francisco Unified School District has been implementing Restorative Practices (RP) at school sites to respond to bullying and teasing and as an alternative to suspensions and other punitive responses to negative behaviors. In 2011, the ExCEL program, with support from the RP Office, began to bring those program elements into the after school hours. In part as a response to the bullying rates reported in prior years, ExCEL launched a full Restorative Practices initiative in , which charged all ExCEL programs with participating in professional learning cohorts, implementing restorative circles in their programs, and using restorative practices such as the restorative questions in response to conflicts or other problem behaviors in the program. Page 19

21 Survey results indicate that ExCEL programs have implemented these practices in their programs, especially among elementary programs. Elementary youth reported higher participation in community circles than other grade groups. Participation levels begin to drop as youth get older. While 88% of 3 rd to 5 th graders report having participated in community circles, 60% of 6 th -8 th graders and 43% of 9 th -12 th graders report the same. Among the 6 th through 12 th graders, approximately 75% report the adults having used the restorative questions in a conflict (Table 3). TABLE 4. RESTORATIVE PRACTICES BY GRADE LEVEL All grades n=3,534 n=1,622 n=1,088 n=6,244 In this after school program I have participated in community circles one or more times. l If I have been in a conflict, the adults in this program have used the restorative questions. 88% 60% 43% 73% -- 74% 80% 76% Gender difference is v Motivation to join is Ethnicity difference is l English Learner difference is statistically significant Source: Youth Surveys, Spring English Learner students are more likely to report participating in community circles; 80% report doing so compared to only 73% of English fluent participants. There are several contributing factors to reported lower participation for older youth. When program staff were asked what additional technical assistance and professional development they needed, many requested more support for restorative practices. As one staff wrote, I don t think the after school staff has been given enough PD on restorative practices. In addition, ExCEL staff noted that some programs are using other terms for community circles so students may not have known exactly what the question meant. The Restorative Practices Cohorts were GREAT. Please continue this practice and support. ExCEL Site Coordinator Page 20

22 ExCEL programs foster positive relationships supportive adults and positive peer relationships and respect for diversity Research on youth development suggests that after school programs provide a valuable context for youth to develop positive relationships with their peers and adults. 5 These relationships can provide a foundation that can encourage youth to pursue positive social connections with others outside of the program. Additionally, opportunities to develop relationships with adults and peers can buffer youth from the negative impact of risk factors (e.g., poverty, living in violent neighborhoods). 6 Thus, the quality of student-staff and peer relationships is a key element in providing a supportive environment to students. After school programs are in a unique position to further the goals of diversity, equity, access and inclusion, such as through structured activities that allow participants to learn about each other and their different cultures as well as through programming that explore critical concepts of diversity and equity such as prejudice and discrimination, justice and respect. 7 We love the programs our after school program designed and implements with our students. They are so open to student ideas and bend over backwards to make them successful. The after school staff is professional and show high expectations for the students they serve, while also building close relationships with them. School day staff who works in the ExCEL program 5 Dworkin, J. B., Larson, R., & Hansen, D Adolescents accounts of growth experiences in youth activities in Journal of Youth and Adolescence 32(1), Masten, A. S., & Coatsworth, J. D The development of competence in favorable and unfavorable environments: Lessons from research on successful children. American Psychologist, 53, California Tomorrow. (2003). Pursuing the Promise: Addressing Equity, Access, and Diversity in After School and Youth Programs. Page 21

23 ExCEL programs provide caring adults in participants lives A great number of youth reported the presence of supportive adults in their ExCEL program. Overall, more than 9 in 10 youth (91%) reported that they feel the adults in the program are happy to see them. Youth reported the presence of an adult who cares about them and that adults in the program tell them when they are doing a good job in similarly positive numbers: 90% and 88%, respectively. Recognition of the presence of supportive adults seems to vary by age, and is at its highest in high school. For example, 97% of high schoolers reported that the adults in their program are happy to see them, as compared to 91% of elementary age youth and 89% of middle school youth. High schoolers responded more positively than other age groups to the other two related items as well (Table 4). TABLE 5. PRESENCE OF SUPPORTIVE ADULTS BY GRADE LEVEL All grades n=3,534 n=1,622 n=1,088 n=6,244 I feel like the adults are happy to see me here. v l 91% 89% 97% 91% There is an adult in this program who cares about me. v 90% 94% 92% 90% The adults here tell me when I do a good job. v l 87% 87% 94% 88% Gender difference is v Motivation to join is Ethnicity difference is l English Learner difference is statistically significant Source: Youth Surveys, Spring Participants who are self-motivated to join the program are much more likely to report supportive adults in the program. 8 English Learners were more likely to report supportive adults compared to their English fluent peers. Latino youth were more likely to report supportive adults than their Asian or white peers, and were more likely than their African American peers to report specifically that adults in the program are happy to see them. Overall, parents are overwhelmingly positive regarding the presence of supportive adults in their child s ExCEL program. In fact, at all but one site, at least 85% of parents reported that their child feels comfortable with the after school staff, and that there is an adult their child can talk to about his or her problems. Said one high school parent, The staff has been really great in making my child feel comfortable there and feel like adults 8 The question about why a participant joined the program was asked only of those in grades 6 and up (selfmotivated n = 962, externally motivated n=1,646). Page 22

24 care about her. She's also gotten a lot of great opportunities for jobs and community service through this program. Staff agreed. Overall, at least 9 in 10 staff agreed that the adults in their ExCEL program really care about every student (92%), want every student to do their best (93%), and listen to what youth have to say (93%). Across roles, Academic Liaisons, Site Coordinators, and enrichment providers had the highest overall percentage of positive responses. [The Site Coordinator] and the ASP staff have amazing and thoughtful relationships with the students and provide an integral part to students' emotional development and sense of belonging. Academic Liaison ExCEL programs can support positive peer relationships Many but not all youth report that ExCEL programs support the development of positive peer relationships and of community in the program. Over 8 in 10 report that they feel like they belong in the program. Nearly 8 in 10 report that the program helps them make friends and a similar proportion report that most youth respect each other in the program. There is significant variation among grade level, with high school participants much more likely to report a sense of belonging, the opportunity to make friends and that youth respect each other (Table 5). Meanwhile, middle school participants are more likely than elementary participants to report a sense of belonging and that most youth in respect each other. Page 23

25 Moreover, the evidence reveals wide variation among programs themselves. While at some programs, nearly 100% of youth report positive peer climate on all of these measures, at a handful of programs, fewer that two-thirds of youth report positive peer climate. Therefore, some ExCEL programs may want to take additional steps to create community and strengthen opportunities for interaction and relationship building. TABLE 6. POSITIVE PEER CLIMATE BY GRADE LEVEL All grades n=3,534 n=1,622 n=1,088 n=6,244 At this program, I feel like I belong. v 81% 86% 92% 84% Most youth respect each other in this program. v l 78% 81% 90% 81% This program helps me to make friends. v l 77% 76% 84% 78% Gender difference is v Motivation to join is Ethnicity difference is l English Learner difference is statistically significant Source: Youth Surveys, Spring English Learners were more likely to report positive peer interaction and communitybuilding compared to their English fluent peers. Participants who are self-motivated to join the program are much more likely to report positive peer interaction and community-building. [This program] gives my daughter more confidence and a greater sense of being connected to her school and community. Parent of high school student Page 24

26 ExCEL programs respect diversity, but need to create more opportunities to incorporate diversity practices. Nearly all youth (93%) reported that their programs respect their language and cultures. Staff were also likely to report that their programs respect the language and cultures of their young people. Ninety-one percent (91%) of staff reported that their programs foster an appreciation of student diversity and respect for each other, and 93% feel their programs emphasize showing respect for all cultural beliefs and practices. Although most youth agreed that their programs respect their language and cultures, slightly fewer (85%) agreed that they get to do activities with people who are not a part of their normal group, and nearly one-fourth (24%) disagreed that they get to talk to other people about their race, ethnicity, gender or culture (Table 6). Taken together, these results indicate that while programs demonstrate appreciation of diversity, there could be efforts to make different diversity practices more prevalent. TABLE 7. RESPECT FOR DIVERSITY BY GRADE LEVEL All grades n=3,534 n=1,622 n=1,088 n=6,244 The people in my after school program respect my language and culture. v I get to do activities with people who are not part of my normal group. v I get to talk to other people about my race, ethnicity, gender or culture. v l -- 92% 94% 93% -- 85% 86% 85% -- 69% 87% 76% Gender difference is v Motivation to join is Ethnicity difference is l English Learner difference is statistically significant Source: Youth Surveys, Spring 2014; these questions only appeared on surveys for 6 th grade and up. Participants who were self-motivated to join the program were much more likely to report practices the build diversity. English Learners were more likely to report that they get to talk to others about their race, culture, ethnicity or gender compared to their English fluent peers (83% and 75% respectively). African-American and Latino youth were more likely than their peers of other races and ethnicities to report opportunities to talk about elements of their identity with others. Page 25

27 ExCEL programs provide opportunities for youth to learn and develop mastery, but could provide stronger opportunities for youth voice and leadership Programs that provide engaging activities for youth can increase youth s sense of mastery and accomplishment. Through project-based learning, increased challenges over time and collaborative learning, ExCEL programs can build life and work skills, selfconfidence and self-efficacy. Input and choice in programs can increase a sense of ownership, belonging and community in a program. Coupled with opportunities to reflect on the outcome, making meaningful choices in a program can also develop critical thinking skills and increased self-knowledge. Age-appropriate leadership opportunities can also provide appropriate challenge, and increase self-confidence and self-efficacy, while also building critical leadership skills such as public speaking, planning, and coordinating a group. ExCEL programs promote skill-building Most youth agreed that their after school program provides opportunities to build skills. Ninety-four percent (94%) of youth reported that adults in the program expect them to try hard to do their best, while 81% reported that the program challenges them in a good way. However, only 76% reported that the program has made them better at something they used to think was hard, a relatively low percentage (Table 7). Page 26

28 Across the board, high school youth had higher percentages of positive responses to these items than elementary or middle school youth, most notably in response to whether the program helps them get better at something they used to think was hard: 84% of high schoolers agreed, as compared to only 74% of younger youth. TABLE 8. SKILL-BUILDING BY GRADE LEVEL All grades n=3,534 n=1,622 n=1,088 n=6,244 The adults in this program expect me to try hard to do my best. v l 95% 92% 94% 94% I am challenged in a good way. v l 80% 79% 87% 81% I am better at something I used to think was hard. v l 74% 74% 84% 76% Gender difference is v Motivation to join is Ethnicity difference is l English Learner difference is statistically significant Source: Youth Surveys, Spring Participants who were self-motivated to join the program, African-American and Latino participants, and English Learners were more likely to report that they receive support for skill-building. Many, but not all, youth have opportunities to lead and make choices in ExCEL programs Overall, many youth, but not all, reported opportunities for leadership in their ExCEL program. Over 8 in 10 youth reported that they get to help other students in their program and just under 8 in 10 reported that their ideas count in the program. Only seven in ten youth (72%) reported that they help to make rules or choose things to do in their program (Table 8). Page 27

29 TABLE 9. LEARNING AND LEADING BY GRADE LEVEL All grades n=3,534 n=1,622 n=1,088 n=6,244 I get to help other students in this program. 85% 73% 76% 81% I feel like my ideas count here. 80% 76% 88% 80% I help make rules or choose things to do in this program. 74% 69% 80% 73% Gender difference is v Motivation to join is Ethnicity difference is l English Learner difference is statistically significant Source: Youth Surveys, Spring Females, participants who were self-motivated to join the program and English Learners were more likely to agree that they have leadership opportunities than their peers. Latino youth were more likely than Filipino, Asian, and white youth to report helping make rules or choose things to do in the program, and to report feeling like their ideas count. Parents and staff generally agree with the youth. Among parents, more than 9 in 10 reported that their child s program provides opportunities to try new things (95%) as well as activities that meet their child s interests (92%). Just over 8 in 10 parents (84%) and a similar proportion of key staff (86%) report that the ExCEL program provides participants with opportunities to make a difference. Over 9 in 10 key staff reported that the ExCEL program encourages opportunities for students to decide things like class activities or rules. Staff responses varied by site. In particular, at 19 sites, fewer than 80% of staff reported that This program gives students opportunities to make a difference. This suggests that at least some programs can improve leadership and civic opportunities for youth. The ASP has provided guided homework practice for students as well as enrichment. Recently, the fun/clubs activities seem to have outweighed the academic and leadership opportunities. Academic Liaison Page 28

30 Alignment with the School Day Many ExCEL programs support school day goals, but school day and after school staff should continue to strengthen these partnerships through regular meetings and support from the Academic Liaison After school partnerships between schools and community partners can benefit youth, the CBOs and schools. Specifically, after school programs that intentionally support academic behaviors and learning can improve academic outcomes for participants. 9 After school can benefit schools by providing additional caring adults in the lives of its students, supporting a positive school climate through events and performances, provide enrichment, science, art, leadership and recreation not available during the school day and reinforce academic concepts Principles that Support School After School Partnerships 1. A shared vision for learning and success, with explicit focus on supporting academics 2. Blended staffing models that enable crossover between school and after school staff 3. School-after school partnerships at multiple levels within the school 4. Regular and reciprocal collection and sharing of information about student progress 5. Intentional and explicit contrast between school and after school environments Finally, CBOs can benefit by having the opportunity to meet their mission to support -Harvard Family Research Project youth in most need of their services, by accessing school day resources and expertise to improve program quality and staff engagement, and by having access to school day information to individually support students. 11 During , ExCEL transitioned the role of certificated teachers in ExCEL programs from Lead Teachers to Academic Liaisons. This shift moves the teacher role beyond direct instruction to students to intentional collaboration with the ExCEL Site Coordinator and School Principal to develop academic strategies after school that best support students. The Academic Liaison role includes: Using data to determine program offerings for students, Developing after school strategies to support student achievement, Communicating between school day and after school, At the high school level, coordinating credit recovery efforts. 9 Little, Priscilla. Supporting Student Outcomes Through Expanded Learning Opportunities. Harvard Family Research Project. January Ibid. 11 Ibid. Page 29