Evaluation of the San Francisco Beacon Initiative

Transcription

1 S O CIAL PO L ICY R ESEAR C H A S S O C I A T E S Evaluation of the San Francisco Beacon Initiative Final Report November 13, 2008 Prepared by: Prepared for: Asha Mehta San Francisco Beacon Initiative Director 1390 Market St., Suite 900 San Francisco, CA (415) Project No Hanh Cao Yu, Ph.D. Charles Lea Jill Leufgen Anna Rubin Social Policy Research Associates With contributions from: Wally Abrazaldo 1330 Broadway, Suite 1426 Oakland, CA Tel: (510) Fax: (510)

2 This page intentionally left blank. Insert blank page here when making double-sided copies

3 Acknowledgement Without the tremendous support of many individuals, this report could not have happened. We would like to acknowledge the following Beacon Directors and their staff for hosting our evaluation team s site visits and conducting the youth surveys: Michael Funk, Sean Yeung, Mayasha Jackson, Takija Gardner, Eli Horn, Chana Kennedy, Chris Rivera, Dave Macgillis, Naomie Wright, Starr Miles, Andrew Tunai, Michelle Cusano, Shawn Brown, Carol Hill, Ben Wong, Valerie Tulier, and Elena Royale. Additionally, we would like to thank Laura Moye at the Department of Children, Youth and Families and Mark Min at Cityspan for their assistance in facilitating access to the administrative data. We would also like to thank the survey translation teams of Nancy Yu, Cynthia Tang, Aleixo Goncalvez Flores, and Sandra Harvey. Lastly, we are infinitely grateful to Beacon Initiative Director Asha Mehta and to Sylvia Hom for their guidance, patience, and wealth of knowledge and keen insights into San Francisco Beacon Initiative. 1

4 This page intentionally left blank. Insert blank page here when making double-sided copies

5 Executive Summary In 2007, the SF Beacon Initiative (SFBI) underwent a major strategic planning process to reexamine SFBI s alignment with the Beacon vision and mission. The vision was to set a future course for the Initiative that ensures that the Beacon Centers promote youth and family centers in public schools that become beacons of activity for the surrounding neighborhood. In conjunction with this process, the goal of the Evaluation of the SF Beacon Initiative was to take stock of the usefulness of the evaluation tools, data, and analysis while assuring alignment of the evaluation with the new SFBI strategic plan. 1 Towards this end, Social Policy Research Associates (SPR) helped SFBI review the existing San Francisco Beacon Theory of Change (ToC), Quality Standards, evaluation methods, and processes for continuous improvement. Specific evaluation objectives included: Conducting in-depth interviews with Beacon Center directors to review and recommend changes to the ToC, Quality Standards, and evaluation. Providing SFBI with information on the eight Beacon Centers progress towards meeting quality standards and outcomes for youth. Making recommendations for future Beacon evaluations, specifically focusing on: (1) role of lead agencies, (2) role of SFBI, (3) family and adult engagement. In addition, we were interested in examining the ways in which a number of the Beacon Centers have evolved from the original concept of the Beacon Centers as community hubs, and the impact of this on the Centers structure, operation, and practices. Methodology In order to collect the data necessary to produce this report, we used a variety of data collection methods. The qualitative components included: Review of key documents, such as previous SFBI evaluation reports, the SFBI 2007 strategic plan, Beacon Center specific documents and websites, and the Minimum Compliance Quality Standards completed by the Beacon Directors in Site visits to all eight Beacon Centers to conduct (1) interviews with Beacon and Program Directors, line staff and school staff, youth (to produce mini-case studies); (2) focus groups with parents; and (3) observations. Focus groups with 95 middle school youth to understand motivations for their participation or non-participation in the eight Beacon Centers. The quantitative component consisted of the following: 1 Since the 2005 San Francisco Beacon Initiative (SFBI) Evaluation by Resource Development Associates, there has been a hiatus of evaluation activities. In December 2007, SFBI commissioned SPR to resume evaluation activities and provide recommendations for new directions in the evaluation to reflect the latest strategic plan. ES-1

6 Review and analysis of the Contract Management System (CMS) data. Administration of a combined Youth Individual Assessment (YIA) and Youth Satisfaction Survey (YSS), which was administered in three languages: English, Spanish, and Chinese in May of Review of district data on Beacon and non-beacon participants school attendance, discipline, and test scores from the school year. The Beacon Centers In Chapter 2 of the report, we provide an overview of the key characteristics of the eight Beacon Centers and the complexity of the efforts that have gone into operating the Centers. The Beacon Centers have grown tremendously over the years, operating in at least eight neighborhoods and across 27 school and other sites with budgets of approximately $7 million. In assessing the extent to which the Centers are providing environments that are visible, accessible, and safe, a number of centers are reporting that they are in progress or have not met a few of the quality standards. For example, based on site visits and CMS data from Spring 2008, only four of eight of the Centers offered programming until 8:00 p.m. Monday through Friday. However, accessibility was a major area of focus for many of the Beacon Centers and all sites increased programming until 8 p.m. where it was possible (leaving only two sites without this cause of school issues). By the end of the year, Beacon Centers had programming at least two nights a week in 6 of 8 sites; whereas at the beginning of the school year, this was true in only in one or two sites. The total number staffing positions funded by Beacon money varied considerably, ranging from 4.6 FTE to The average is closer to 5.0 FTE, typically with a Beacon Director, a Program or Site coordinator, and Safety and Support staff forming the core staff. The San Francisco Department of Children, Youth and Families provides a key funding source for Beacon centers; however, the Centers also reported receiving $2 million in ExCEL funding. Providing a diverse programming (e.g., academic support, enrichment, and recreational) is a strength of many of the Centers. However, the Centers were growing their family and community programming, and therefore, were not consistently offering adult- as youth programming. Participation and Youth Developmental Experiences Participant Characteristics The analysis of demographic characteristics of participants during the Summer of 2007 and the school year shows that the Beacon Centers had a higher percent (44%) 2 of youth in middle school than in any other grade level. Most participants (47%) were between 10 and 13 years old, and there was a slightly higher percentage (53%) of male youth. The racial/ethnic 2 Hub and satellite sites combined. ES-2

7 groups with the highest percentage of youth were Asian (46%) and African American (22%), and the majority (76%) of youth were fluent English speakers. The adult participants were distributed evenly across age brackets, although 40% of them were at least 55 years old. A higher percent (61%) of adult participants were female, and the majority of adult participants were Asian (62%). While most (67%) adults were fluent English speakers, nearly 25% were not. Participation Patterns Attendance across all eight Beacon Centers was an impressive total of 7,650. Following are some specifics on the participation patterns: Total participation: Six of the eight Beacon Centers met participation quality standards by serving at least 600 participants during the combined Summer of 2007 and school year. Seven of the Centers served at least 70% youth participants as compared to adults. Adult participation: Overall, a substantive number (100 or more) of adult participants were concentrated at half of the sites. At three sites, adults comprised 5% or less of the total participants. Daily participation: During the fall of the school year, five out of eight Centers met the Beacon Quality Standard for daily attendance; however, only three sites met it during the Spring. 3 Hub site participation: Four of the eight Beacon Centers served a greater percentage of participants at their hub site. Overall, 58% of the total number of youth and adults participate in activities at the hub sites, although the quality standards specify that 80% of the programming should occur at the hub sites. Summer vs. school year participation. At Beacon hubs, more youth attended during the school year than during the summer of 2007, and averaged more overall days of attendance during the school year. However, youth averaged more days of attendance per week during the Summer session. Adult participation. As for adult attendance at Beacon Center hubs, a greater number of adults attended during the school year. In addition, adult participants attended more days on average during the school year. This is largely due to the fact that SFUSD does not allow adequate school access to continue adult programming in summer. Range of program/activity participation by school levels: During the school year, elementary school participants at Beacon hub sites attended more frequently and spent more total hours in group activities than middle and high school youth. 3 The average daily attendance of some sites may be higher than is shown below because of activity days with missing attendance records. This may be especially true during the spring session, for which some sites are missing a great deal of missing attendance information. ES-3

8 Middle vs. high school youth participation patterns. During the school year, middle school participants attended Beacon Centers more frequently than high school participants did, but on the days that high school youth did attend, they spent more time at the Beacon Center than those youth in middle school. Academic Support & Enrichment participation. The average number of hours elementary participants spent in Academic Support & Enrichment was significantly higher than both other groups of participants. In addition, middle school youth spent a significantly higher average number of hours in Academic Support & Enrichment than hub participants in high school. Middle school participants also spent, on average, significantly fewer hours on Sports & Recreation than both elementary and high school participants. Youth Survey Results The survey data provided important youth feedback on the quality of their youth developmental experience at the Beacon hubs. Our analysis of the data show that the majority of youth who responded enjoyed coming to Beacon and felt safe at their Beacon Center. The majority also felt that they learned new things and did interesting activities at Beacon. In addition, most older respondents felt that Beacon encouraged them to achieve by having goals for the future. The majority of youth surveyed felt that people at Beacon respected their culture and heritage, but fewer replied that Beacon helped them know about their culture and heritage, or stated that they learned about people from different cultures. This suggests that Beacon Centers need to pay more attention to this aspect of programming, particularly given the diversity of their participants. While higher percentages of middle/high school respondents than elementary respondents reported that they have developed supportive relationships with Beacon adults, over one-quarter of all youth felt that there was no Beacon adult who really cared about them. As for relationships with their peers, both elementary and middle/high school respondents reported having positive peer influences, and most of the older youth indicated they have friends with whom they have supportive relationships. In the area of skill development, youth were asked about the leadership activities they participated in at Beacon. Most older respondents responded affirmatively on questions about opportunities to work with others, while lower percents of elementary respondents did so on these questions. However, fewer than half of older respondents had participated in the majority of the leadership activities. Challenges to Achieving Beacon Mission and Quality Standards Beacon Centers throughout the San Francisco Beacon Initiative continue to experience a number of challenges achieving Beacon mission and Quality Standards that also prevent them from ES-4

9 aligning with the community hub model. The most common sets of challenges that were cited include: Parent and community engagement: Engaging parents and the community in Beacon programs is a significant challenge for seven of the eight Beacon Centers. Many Centers continue to struggle in engaging parents in developmental, enrichment, and community activities (e.g. parenting classes, job training, violence prevention initiatives, etc.). Factors that prevent parents and the community from engaging in Beacon programming include the location of Beacon Centers and the large number of working class and immigrant families within these communities. Availability of school space: Five Beacon Centers reported the lack of available of space as a major issue. Key factors that affect a Center s ability to obtain adequate space include a lack of cooperation from school teachers and construction at school sites. Relationship with host schools: Five of the eight Centers reported a number of challenges in operating and implementing programs at their school sites. Beyond the availability of school space, these challenges, in most cases, were related to changes in school administrators and staff, and conflicting and/or a lack of understanding of Beacon goals. Participant attendance requirement: As described earlier, achieving the Quality Standard attendance requirement 4 is a major challenge experienced by a number of the Beacon Centers. Community violence, transportation, and types of programming were among the common factors that respondents expressed as factors associated with this challenge. Integration of ExCEL funds: Many Beacon directors blend the ExCEL funds with their other sources of funding to support the Beacon Center. As a result, many directors expressed some difficulty in offering programs outside of the ExCEL academic requirements. In addition, Beacon directors reported that the ExCEL funds only allow them to serve/count students who are attendees of the host school. Thus, this has limited many Centers ability to provide programs and services to adults and the community, which further prevent them in aligning with the community hub model. These challenges have greatly affected some Centers ability to meet specific Beacon participation standards and achieve broader program outcomes. However, a number of Centers have utilized some promising practices that allow them overcome such challenges. 4 Beacon Centers are to serve at least 150 participants a day, and 600 or more children and youth, and adult family members unduplicated per year. ES-5

10 Recommendations A major focus of the 2008 Beacon evaluation was to understand the evolution of the Initiative over the past decade and the extent of mission drift as well as the development of creative practices, and provide recommendations from the Beacon directors on ways to revise both the San Francisco Beacon Theory of Change (ToC) and the Quality Standards. When asked, a number of Beacon directors and staff mentioned they had not seen the ToC model or Quality Standards in a while, or never knew they existed. However, upon closer examination most Beacon directors and staff stated that the ToC model is an accurate reflection of outcomes that they are trying to achieve at the Beacon site-level; yet, there were a few elements that they reported were missing from the ToC model. More specifically, the roles and responsibilities of Initiative-level leadership, while well laid out, do not accurately reflect who is or who needs to be responsible for ensuring that some Initiative-level or cross-site activities happen. The following are suggestions of how the ToC model can better assist Beacon Centers in achieving their outcomes. Develop more clarity around the roles of key leaders within the Beacon governance structure. When the ToC was developed, it did not specify that there would be an initiative-wide governance body, a role that the steering committee assumed. The role of key leadership groups, such the Steering Committee, needs to be more explicitly delineated within the ToC, particular because these groups can play a critical role in working to bring about systems accommodation and support, and also garner financial and other support from city departments and school district leaders. Expand and standardize the role of the lead agencies. Standard role of lead agencies are fiscal sponsorship, fiscal management, administrative oversight, and human resources. However, SFBI and Beacon directors and staff suggested that the role of the lead agencies be explicitly defined, expanded (e.g., around staff development) and standardized, so that Beacon Centers and SFBI can hold them accountable and build on their youth development and community building expertise. Specify and who should play the role of the TA Intermediary and how to fully support and resource the TA Intermediary functions. The ToC included the roles of the TA intermediary, which was formerly the role of the Community Network for Youth Development (CNYD). However, since CNYD is no longer a part of the Initiative, the SFBI Director has assumed some of the intermediary roles. However, it is not possible for SFBI s small staff to take on all of these responsibilities. Beacon directors and staff feel that the ToC needs to be updated to include which stakeholder is responsible for fulfilling these intermediary roles. Strengthen and support the role of SFBI Director and staff. Beacon Center leaders suggest that the SFBI staff continue to play the multiple roles of establishing a clear vision, coordinating relationship with SFUSD as a systems advocate, fundraising, and facilitating cross-site fertilization and development. Nevertheless, Beacon Center leaders acknowledge that the SFBI staff cannot assume all of the initiative level roles and responsibilities that were left unfulfilled after CNYD s departure. ES-6

11 Additions to the Quality Standards Beacon directors and staff mentioned a number of elements that were missing from the Quality Standards and provided suggestions as to what standards and information should be included: Provide more clarity on expectations for particular outcomes standards. For instance, most Beacon directors and staff acknowledged that the career development standard was underdeveloped at their center. In developing programs and activities around this standard, Beacon staff articulated that they would like to be aware of what is required of them as a Center when offering career development services. Create more specific standards. Beacon staff stated that some standards are in need of sub-standards. For instance, while health is included as a standard, it could be expanded to include more specific standards of mental, physical, and/or sexual health. Increase host schools accountability. A number of Beacon staff stated that they would like to see some explicit requirements for the host school to help ensure a positive relationship. For instance, one Beacon staff suggested that this standard be modified to require all school staff to be educated on the mission and purpose of a Beacon Center. Focus on adult and community development. A number of Beacon staff expressed that the Quality Standards heavily emphasize youth, and not adult, family, and community outcomes. Therefore, they suggest that the Quality Standards have specific outcomes for adults, the families, and communities. Recommendations for Improving and Aligning the Evaluation and Data Collection Systems The data collection system for the Beacon sites need considerable improvement to make any conclusion about the impacts of the Beacon Centers on youth, family, school, and community outcomes. Our overarching recommendation is: SFBI and Beacon Centers need to invest more time and energy to improve the (1) accessibility to district data for student outcomes; (2) accuracy and usability of the CMS data; and (3) redesign of the youth surveys to improve their validity. Specifically, Ensure that Beacons administer properly worded permission slips that allow access to students district data files including test scores, grade data, attendance, disciplinary action, and obtain timely comparison group data from the SFUSD early in the program year. Train Beacon staff to do correct and thorough data entry of information on youth and adult participation. For example, ensure student s grade level and the school where a student is enrolled are required fields. ES-7

12 Develop a field that identify participants activity location (hub vs. satellite) and ensure unique Client IDs (e.g., district HO numbers) for each Beacon participant to avoid duplicate entries when a participant moves from one center to another and from summer to school year sessions. Re-vamp pre- post survey so that it is more concise and better tracks to the expected outcomes of Beacon participation, and prioritize administration of surveys at hub sites because these sites are considered the core of Beacon services. Consider further focusing evaluation indicators of success to assure alignment with the Theory of Change, the Quality Standards, and SFUSD s outcome goals. See Appendix C, for example of indicators of a Beacon Scorecard. Conclusion The SFBI and Beacon Centers have evolved and grown considerably since their establishment in The variety of programming, number of youth and adults served, involvement of numerous stakeholders, partners, and supporters, and the presence of innovative projects such as the Beacon Young Adolescent Initiative with New York Beacons and Gateway to Fitness attest to the scale and reach of a mature Initiative that is highly complex and sophisticated. At the same time, both external and internal factors such as the complexity of the neighborhoods, populations, blended-funding streams, changing initiative-level support structures, and crosssystem relationships (e.g. SFUSD space issues), that each of the Beacon Centers serve suggests that more work needs to be done to reach out to community residents in order to become a true community hub. Although the ToC needs updating to clearly reflect stakeholders responsibilities for specific roles, it generally provides a robust framework to guide the Centers work. The Quality Standards provide high expectations for Centers performance that Beacon directors need to continue to meet; however, the conversations that will occur about revisions needed to this guiding document will likely determine whether there needs to be some customization to the local neighborhood context. ES-8

13 CONTENTS ACKNOWLEDGEMENT EXECUTIVE SUMMARY... ES-1 I. ABOUT THE EVALUATION... I-1 Methodology...I-2 Overview of the Report...I-3 II. III. THE BEACON CENTERS... II-1 SFBI and Lead Agency Roles...II-1 Center Structure and Setup...II-3 Locations...II-3 Hours of Operation...II-6 Staffing...II-7 Budget...II-10 Funding Sources...II-10 Programming...II-14 Summary...II-16 BEACON CENTER PARTICIPATION AND YOUTH DEVELOPMENTAL EXPERIENCES... III-1 Data Sources...III-1 Contract Management System Data...III-1 Survey Data...III-2 District Data...III-3 Demographics, Participation and Attendance...III-3 Demographics...III-4 Participation...III-8 Average Daily Attendance (ADA)...III-12 Hub Youth Attendance...III-14 i

14 Adult Attendance at Hub Sites...III-20 Hub Survey Results...III-22 Safety at Beacon...III-24 Welcoming at Beacon...III-26 Engagement & Collaboration...III-27 Comprehensive Programming...III-28 Supportive Relationships...III-29 Meaningful Participation & Sense of Belonging...III-32 Involvement in the Community...III-35 Challenging & Engaging Skill Building...III-36 Youth Habits...III-38 Youth Assessment of Beacon...III-40 District Data...III-41 California Standards Test (CST) Proficiency...III-42 Average Daily Attendance (ADA)...III-43 Suspensions...III-44 Summary...III-45 IV. KEY CHALLENGES TO ACHIEVING BEACON MISSION AND QUALITY STANDARDS...IV-1 1. Parent and Community Engagement... IV-1 Location of Beacon Centers... IV-2 Working Class Families... IV-3 Immigrant Families... IV-3 2. Availability of School Space... IV-3 Cooperation of Teachers... IV-4 Construction at School Sites... IV-4 3. Relationship with Host School... IV-5 Changes in School Administration... IV-5 Conflicting Goals... IV-6 4. Participant Attendance Requirement... IV-7 Community Violence... IV-7 Transportation... IV-8 Program Offerings... IV-8 5. Integrating ExCEL Funds... IV-9 ii

15 Limits Program Offerings and Target Population... IV-9 Conclusion... IV-10 V. ASSESSMENT OF THE THEORY OF CHANGE, QUALITY STANDARDS AND RECOMMENDATIONS...V-1 How accurately does the Beacon ToC and Quality Standards reflect the current goals, activities, and outcomes?... V-3 Assessment of the Theory of Change & Quality Standards... V-4 Recommendations for the Theory of Change... V-5 Assessment and Recommendations for the Quality Standards... V-9 Recommendations for Changes in the Quality Standards... V-10 Additions to the Quality Standards... V-11 Recommendations for Improved Data Collection... V-13 Conclusion... V-14 APPENDIX A: PROMISING PRACTICES AND INNOVATIVE PROGRAMMING... A-1 APPENDIX B: COMPLIANCE STANDARDS FOR BEACON CENTERS: COMPLIANCE STANDARDS RESULTS FOR B-1 APPENDIX C: SAMPLE BEACON SCORECARD TO ALIGN WITH SFUSD STRATEGIC GOALS AND SCORECARD... C-1 APPENDIX D: ANALYSES OF SELECTED ATTENDANCE BY GRADE LEVEL... D-1 iii

16 This page intentionally left blank. Insert blank page here when making double-sided copies

17 I. ABOUT THE EVALUATION In 2007, the SF Beacon Initiative (SFBI) underwent a major strategic planning process to reexamine SFBI s alignment with the Beacon vision and mission. The intent of this process was to set a future course for the initiative, which ensures that the Centers promote youth and family centers in public schools that become beacons of activity for the surrounding neighborhood. In conjunction with this process, the goal of the Evaluation of the SF Beacon Initiative was to take stock of the focus and usefulness of the evaluation tools, data, and analysis while assuring alignment of the evaluation with the new SFBI strategic plan.1 Towards this end, Social Policy Research Associates (SPR) was engaged in 2008 to help SFBI review the existing Beacon theory of change, quality standards, evaluation methods, and processes for continuous improvement. SPR s goal was to surface strengths and challenges of the current evaluation system and assist in the revision of the standards, measurement tools, and data collection processes to better match with the current strategic focus. Our specific evaluation objectives included the following: Conducting in-depth interviews with Beacon Center directors to review and recommend changes to the ToC, Quality Standards, and evaluation. Providing SFBI with information on the eight Beacon Centers progress towards meeting quality standards and outcomes for youth. Making recommendations for future Beacon evaluations, specifically focusing on: (1) role of lead agencies, (2) role of SFBI, (3) family and adult engagement. In addition, we were interested in examining the ways in which a number of the Beacon Centers have evolved from the original concept of the Beacon Centers and the impact of the growth of the Initiative from 8 to 25 schools through the exponential increase in state-funding for afterschool leveraged by Beacons. We examined opportunities of this expansion and costs of a potential mission drift on the Centers structure, operation, and practices. Exhibit I-1 below 1 Since the 2005 San Francisco Beacon Initiative (SFBI) Evaluation by Resource Development Associates, there has been a hiatus of evaluation activities. In January of 2008, SFBI commissioned with Social Policy Research Associates to resume evaluation activities and provide recommendations for new directions in this evaluation. I-1

18 summarizes the original concepts, costs, and opportunities of the current state of SFBI, as analyzed in the Reality Check meeting in January This table provided an important frame of reference for our data collection and analysis. Exhibit I-1. Assessment of Beacon Centers Relative to the Original Concept 2 Original Concept Costs Opportunities 1. Beacon Centers transform public schools into one of the major gathering places in the community, open year-round, after school, in the evenings, on weekends, in the summer. 2. Beacon Centers are an intergenerational space for the whole family and neighborhood. 3. A Beacon Center makes powerful contributions to people s lives because of the concentration, intensity, and range of programs in the Center. 4. It is connected to other programs and ongoing efforts in the community. 5. The youth and adults in the neighborhood play a large part in deciding what a Beacon Center offers. Most Beacon Centers have few/no programs after 6 p.m. although current MOU with the school district allows access until 8 p.m. M-F, and one weekday until 9 p.m. Most Beacon Centers primarily serve the age group represented in the school population and only youth who attend that school site. There is little neighborhood and family use of most Beacon Centers. (Some exceptions during summer.) Programs decentralized from the school site have led to decreased concentration and intensity at most Beacon Centers. Coupled with shorter hours of operation, this leads to decreased neighborhood impact. Low attendance is also an issue at several Beacon Centers. There are different degrees of connection and understanding of role in relation to community efforts. Most Beacon Centers do not currently have a regular means of assessing community need and building neighborhood ownership of Beacons (e.g. inactive Community Councils or Parent Councils). 17 schools (in addition to the 8 original sites) have afterschool programming from 3-6pm that is connected, centralized, and coordinated through the Beacons Most of the 8 Beacon Hub sites offer daytime programming (e.g. wellness services, technology education, in-class support) in addition to afterschool programs and are much more integrated into the fabric of schools than was originally conceived The Sunset Beacon created a vibrant community space by renting a storefront property when access to school building was limited by SFUSD due to budget concerns. Visitacion Valley Beacon became a central organizer in neighborhood violence prevention efforts when community violence led to low attendance in afterschool programs Chinatown Beacon developed a District 3 Youth Development network that has greatly increased collaboration of agencies in the region By working in multiple school sites in a neighborhood, some Beacons have been able to facilitate principal partnerships with each other and larger community development efforts 2 This table is derived from a handout prepared by the SF Beacon Coordinator for a January 29, 2008 meeting to discuss strategic alignment of the Department of Children Youth and Their Families Resources with SFBI vision and mission. I-2

19 Methodology In order to collect the data necessary to produce this report, we used a variety of data collection methods. The qualitative components included the following: Review of key documents, such as previous SFBI evaluation reports, the SFBI 2007 strategic plan, and Beacon Center specific documents and websites. Review of Minimum Compliance Quality Standards completed by the Beacon Directors in Site visits to all eight Beacon Centers to conduct day long interviews and observations. Key activities included: In-depth interviews with Beacon and Program Directors, Interviews with line staff and school staff, Interviews with youth to produce mini-youth case studies, Focus groups with parents. Observations of program activities (some at satellite sites). Incorporation of findings from a study of hard-to-reach middle school students. In May of 2007, SPR conducted a total of 16 youth focus groups across the eight Beacons sites (two focus groups per site with each focus group lasting from minutes each over 2.5 weeks in May 2007). We interviewed a total of 95 youth who met the low-participating and non-participating criteria. The quantitative component consisted of the following: Review and analysis of the CMS data. Administration of a combined Youth Individual Assessment (YIA) and Youth Satisfaction Survey (YSS) in May This survey was administered in three languages: English, Spanish, and Chinese. Review of district data from the school year. Where possible, we conducted a separate analysis of youth participants at the main hub Beacon site. This is to assess the primary effect of DCYF funding on youth outcomes. Overview of the Report This report consists of six chapters. Chapter 1 provides an overview of the evaluation goals and methods. The remaining chapters aim to achieve the following: Chapter 2 provides an overview of the Beacon Centers. In this chapter, we review the initiative structure with specific information on the SFBI role and lead agency role. We also focus on a description of the Centers structure and setup, staffing, funding sources, and programming. I-3

20 Chapter 3 summarizes youth and adult participant characteristics, demographics and participation rates (at hub vs. satellite sites). In addition, we report key youth outcomes based on our quantitative analysis. Key areas of reporting include satisfaction with Beacon Center experience, activities/participation, safety, and youth development outcomes. Chapter 4 discusses key challenges faced by Centers, particularly as they affect their ability to achieve the original mission and vision of the Beacon initiative. Chapter 5 discusses the implications of the challenges on the Beacon Theory of Change and Quality Standards. We provide key recommendations for further development of specific standards that Beacon staff perceive as missing or under developed. In addition, we provide our overall recommendations for the leadership roles of SFBI, lead agencies in building the infrastructure of the Beacons initiative, and make recommendations to better prepare for future evaluations. I-4

21 II. THE BEACON CENTERS In order to assess how the Beacon Centers have evolved from the original concept of the Beacons, we begin with a review of the initiative structure with specific information on the SFBI role, and the lead agencies role. In addition, we also summarize several aspects of the Beacon Centers in order to address assess their alignment with the overall Beacon mission: (1) the Centers structure and setup (how decentralized are the Centers operations), (2) staffing (are the staffing positions strategic), (3) funding sources (what other funds do Centers received in addition to the DCYF funds), and (4) programming (is there a diverse range, and what is provided to youth vs. adults vs. family/community). SFBI and Lead Agency Roles The San Francisco Beacon Initiative (SFBI) is a public-private partnership that promotes youth and family Centers in San Francisco public schools. The following delineates the roles of SFBI and the lead agencies. Role of SFBI. Based on the Beacon s Theory of Change, SFBI s role encompasses the following responsibilities: Relationship Building Build relationships, teamwork, and responsibility among Beacon site stakeholders. Leveraging existing support systems Support and pressure service systems to accommodate the needs of Beacon Centers, facilitate the negotiation with service systems for the commitment of resources for the Beacon Centers, and facilitate establishment of long-term partnerships between service systems and Beacon Centers Funding Initiate the development of a core funding strategy and cultivate donors. Communications Manage the communications and public support campaigns, ensuring commitment of funding. Implementing Theory of Change - Institutionalize the Theory of Change and strengthen the sites commitment and meaningful use of the Theory of Change. II-1

22 Disseminating best practices Ensure greater diversity and prevalence of agency use of best practices. Capacity building Provide needed implementation resources and effective supports that build site capacity Disseminating youth development model Build an understanding and commitment to youth development and strengthening the implementation of youth development through staff development, cross-site exchange, and individual site consultation Evaluation Instigate and manage the development of evaluation strategies, manage the evaluation resulting in information that is clear, meaningful, useful and compelling to stakeholders, support ongoing use of evaluation findings by Beacon stakeholders, and broadly disseminate evaluation findings. Our interviews with Beacon Directors confirmed that these are appropriate roles that SBFI has and continue to play. A number of additional roles and responsibilities were recommended, which we will discuss in Chapter 5. Role of Lead Agency. The lead agency does not have responsibilities outlined in the Theory of Change. However, respondents at site visits described what they believed the lead agency s role should be. Although each lead agency functions in a slightly different way, there was consensus that the lead agency should play the following two key roles: Fiscal management - Lead agencies are responsible for managing the grant, accounting, and payroll. Human resources Lead agencies are charged with risk management, hiring and evaluating staff including doing background checks and medical clearance, and providing professional development and training opportunities. Exhibit II-1: Lead Agencies Beacon Centers Lead Agency Type of Lead Agency Bayview Hunter s Point Urban Services YMCA Violence Prevention, Youth Development, Social Services Chinatown Community Educational Services Youth Development Mission Mission Neighborhood Center Human Services OMI-Excelsior Urban Services YMCA Violence Prevention, Youth Development, Social Services Richmond Richmond District Neighborhood Center Youth, adult and family programming Sunset Moss Beach Homes Foster care II-2

23 Beacon Centers Lead Agency Type of Lead Agency Visitacion Valley Visitacion Valley Community Center Multi-service, family resource Western Addition Buchanan YMCA Youth Development, Healthy and Fitness In addition to these two key roles, some sites also suggested that lead agencies help connect Beacon to other youth service agencies and community groups, provide additional social services to Beacon participants, and be the expert in the community, ensuring that the Beacon serves participants in a culturally and linguistically appropriate manner. Several sites also said that the lead agency should help them secure additional financial resources by helping them find grants or providing a grant writer. Center Structure and Setup Locations Beacon Centers are located in eight neighborhoods throughout San Francisco: Bayview, Chinatown, Mission, OMI/Excelsior, Richmond, Western Addition, Sunset, and Visitacion Valley. In , 13 elementary schools, 8 middle schools, 4 high schools, and 3 other types of locations held Beacon programming. In each neighborhood, one school serves as the hub and the other schools or locations are satellite sites. According to the Quality Standards on Accessibility, 80% of the programming should occur at the hub site. Bayview Gloria Davis Elementary School is currently the hub for this center. The school will open to 9 th and 10 th graders in the beginning of the school year. This site was chosen because it is in the heart of the Bayview community and the principal will allow youth from the community to attend the center. Dr. George Washington Carver Elementary School and Thurgood Marshall High School are satellite sites. Chinatown This Beacon Center runs afterschool programs at 7 elementary schools in and around Chinatown and also offers programming at CES s offices. Gordon J. Lau is designated as the hub, but they currently do not offer any programming beyond running the afterschool program there due to resistance from teachers to sharing classroom space. They also offer satellite programming at Francisco Middle School and Galileo High School Mission Everett Middle School is the only site of operation.. OMI/Excelsior James Denman Middle School is the only site of operation. II-3

24 Richmond George Washington Elementary School serves as the hub for the Richmond Beacon Center with satellite sites at Presidio Middle School and Roosevelt Middle School. Sunset The hub for the Sunset Beacon Center is A.P. Giannini Middle School with satellite sites at 4 elementary schools. They also run programming out of West Sunset Recreation Center and have a stand alone administrative office/community center. Visitacion Valley Visitacion Valley Middle School serves as the hub II-4

25 Exhibit II-2: Beacon Sites E.S. M.S. H.S. Other Bayview Beacon Center Gloria R. Davis Middle School (hub) Dr. George Washington Carver Elementary School Thurgood Marshall High School Chinatown Beacon Center Gordon J. Lau Elementary School (hub) Jean Parker Elementary School John Yehall Chin Spring Valley Elementary School Sherman Elementary School Redding Elementary School Garfield Elementary School Francisco Middle School Galileo High School Community Educational Services Mission Beacon Center Everett Middle School (hub) OMI/Excelsior Beacon Center James Denman Middle School (hub) Richmond Village Beacon Center George Washington High School (hub) Presidio Middle School Theodore Roosevelt Middle School Sunset Neighborhood Beacon Center (SNBC) A.P. Giannini Middle School (hub) Francis Scott Key Elementary Sunset Elementary Ulloa Elementary Robert Louis Stevenson Elementary West Sunset Recreation Center SNBC Community Center Visitacion Valley Beacon Center Visitacion Valley Middle School (hub) Western Addition Beacon Center X X X X X X X X X X X X X X X X X X X X X X X X X X John Muir Elementary School (hub) X Wallenberg High School X Total This school is not currently undergoing reconstruction and is not open to students. II-5

26 Hours of Operation According to the Quality Standards on Accessibility, Beacon Centers should be open until 8:00 PM weekdays, offer Saturday programs at least one Saturday per month or for a total of 48 hours on Saturdays during the school year, offer programming during major breaks in the school calendar, and offer programming during the summer months. The current MOU with the school district allows Beacon Centers to be open until 8:00 PM on weekdays, with one night until 9:00 PM, one Saturday per semester, and during the summers. 1 Currently, the majority of Centers are not open all of these times. All Beacon Centers are open Monday through Friday during afterschool hours (approx. 3-6pm). Additionally, many Centers are open during the school day for administrative tasks and offer lunchtime programming All sites except for Chinatown and Bayview due to school facilities access issues offer evening programming. Four out of 8 Beacon Centers offer programming until 8:00 PM all week from Monday through Friday. Mission is open until 8:00 PM Tuesday through Thursday, and Western Addition is open until 8:00 PM on Fridays. All sites except for Chinatown and Bayview offer some kind of Saturday programming, although not on a regular basis. Several Beacon Centers offer programming such as college field trips during winter and spring breaks, but not all of the Centers remain open due to school facilities access issues. All of the Centers are also open during the summer Exhibit II-3: Hours of Operation School Year Hours Summer Hours Weekends Bayview 2 M-F, 9-6 Some programming Chinatown M-F, hours vary according to site Mission M-F, 3:15-6:30, T, Th, 3:15-8 W, 12:15-8 F, 3:15-6 M-F, 9-4 M-F One Saturday per semester 1 Note that there is no school holiday access. 2 Bayview plans on being open year round, evenings, and weekends starting August 08 II-6

27 School Year Hours Summer Hours Weekends OMI-Excelsior M-F, 8-8 M-F, 9-3:30 Occasional Saturday programming Richmond M-F, 11-1, 3-8 M-F, 8-6 Held outreach activities on 4 Saturdays Sunset M-Sa, 9-8 M-F, 1-6 Occasional Saturday programming Vis Valley M-F, 9-8:30 M-F, 10:30-3 Saturdays, 9:30-12:30 Western Addition M-Th, 9-7, F, 9-8 Full summer programming One Saturday per semester Staffing According to the 2008 Reality Check document, core DCYF funds should go to the following key positions with the optional additions of a case manager, site manager, and administrative support: Exhibit II-4: Recommended Staffing Structure Beacon Center Director (1.0 FTE: on-site) Youth Programs Coordinator (1.0FTE) Neighborhood Family Organizer (1.0FTE) Safety & Support Because Beacon funding and funding from other sources, such as ExCEL, blend together at many of the sites, it was difficult to tease out which staff positions were funded by which source. For example, one director reported that his staff is paid by a combination of the two funding streams, depending on which hours of the day they work. The staffing as described below is based on the positions reported in the most recent budget of the 2007/2008 work plan. It is important to note that the staff positions listed are not inclusive of all the staff at any given Beacon center, but rather the positions funded directly by Beacon money. In addition to these staff and staff paid through other funds, Beacon Centers have also recruited workers from a variety of programs such as MYEEP (Mayor s Youth Employment and Education Program), Americorps, and Experience Corps (senior volunteers) to supplement their staff at a lower cost. II-7

28 Exhibit II-5: Staffing funded by DCYF Beacon core dollars Bayview FTE Beacon Director 1.0 Program Director 0.3 Site Manager 0.7 Case Manager 1.0 MIS Manager 0.5 Safety and Support (2) 0.3 Lead Safety and Support 0.3 Admin Assistant 0.1 Tutor 0.2 Program Leader (2) 0.3 Total FTE 5.3 Chinatown FTE Director 0.6 Youth Empowerment Manager 0.3 Youth Program Coordinator 1.0 Program Leader 0.5 Summer Program Leader (4) 0.2 Summer Program Coordinator 0.2 Student Support Coordinator 1.0 Program Manager 0.1 Summer Youth Personnel (3) 0.2 Total FTE 5.1 Mission FTE Beacon Director 1.0 Youth Services Director 0.1 After School/Summer Program 0.5 Director After-school Academic Coordinator 0.5 Case Manager 1.0 Lead Tutor 0.4 Lead Safety and Support 0.4 Safety and Support 0.3 Program Leader Support 0.4 Total FTE 4.6 OMI-Excelsior FTE Dir. of School Based Services 1.0 Dir. of Operations 0.5 Case Manager 1.0 Site Manager 0.2 Manager of Information Services 1.0 Safety & Support (2) 0.5 Admin Assistant 0.7 Gang Case Manager 0.2 Parent Liaison 0.3 Total FTE 5.9 II-8

29 Exhibit II-5: Staffing (continued) Visitacion Valley FTE Richmond FTE Beacon Director 1.0 After School & Community 0.5 Programs Director AND Queer Youth Services Coordinator Youth Services Programs Director 1.0 Office Manager & Youth Employment Coordinator Case Manager & Latino Services Coordinator Leadership and Black Student Services Coordinator Arts & Technology Coordinator 1.0 Teen Center Coordinator 1.0 Deputy Director 0.1 Data Entry Staff 0.2 High School Activity Leader 0.3 Middle School Site Director (2) 0.2 Summer Middle School Activity Leaders 0.7 High School Tutor 0.2 Evening Program Coordinator 0.2 Youth Program Assistants (8) 0.1 Total FTE 10.4 Beacon Director 1 Beacon Site Manager 1 Community and Youth Program 1 Coordinator Youth Program Director 1 Case Manager 0.5 Safety and Support 0.4 Afterschool Coordinator 0.4 Total FTE 5.3 Sunset FTE Beacon Director 0.2 Program Assistant 0.6 Safety and Support 1.0 Director of Afterschool Programs 0.2 Experience Corp Director 0 Director of Community Programs 0.7 Case Manager 1.0 Director of Operations 1.0 Ulloa Afterschool Staff 0.6 Bamboozled.org Team Leader 0.2 SCREAM film staff 0.1 Total FTE 5.6 Western Addition FTE Program Director 0.9 Program Coordinator 0.5 Site Coordinator 0.8 Safety & Support 0.4 Enrichment Leader (2) 0.8 Technology Coordinator 0.7 Total FTE 4.9 II-9

30 Budget According to the 2007/2008 work plan, Beacon Centers reported budgets ranging from $384,375 to $584,375. Each Beacon lead agency receives a total of $384,375 of DCYF Beacon core funds per year per Beacon site. The amounts above these core dollars are the result of fundraising efforts by the lead agency associated with other DCYF programs (e.g. Afterschool for All expansion). For all Beacons, staffing accounted for almost half or more of the budget. Bayview Chinatown Mission Exhibit II-6: Beacon Center Budgets OMI- Excelsior Richmond Sunset Western Addition Visitacion Valley Staff 60% 48% 52% 61% 62% 49% 49% 66% Fringe Benefits 16% 11% 9% 18% 13% 6% 13% 18% Professional 0% 6% 0% 0% 0% 0% 1% 5% Subcontractors 0% 11% 14% 0% 0% 0% 10% 0% Other Program Expenses 5% 7% 8% 5% 4% 26% 11% 0% Program Materials & Supplies 4% 3% 2% 1% 6% 8% 3% 0% Administrative 15% 14% 15% 15% 15% 12% 13% 10% Total Budget $384,375 $384,375 $384,375 $410,375 $584,375 $434,375 $384,375 $384,375 In addition to Beacon core funds, Beacon Centers were able to leverage additional grant money from DCYF as well as public and private foundations. Seven out of eight Beacon Centers also received ExCEL funding from San Francisco Unified School District (SFUSD). The ExCEL (Expanded Collaborative for Excellence in Learning) After School Program is an umbrella program for a variety of state funded grants including California's After School Education and Safety, 21st Century Community Learning Centers, and 21st Century ASSETS grants. All of the Beacon Centers except Mission receive ExCEL funding for their schools for a total of almost $2 million. Mission no longer receives ExCEL funding directly, but closely coordinates with the ExCEL director at Everett Middle School. 3 Several sites use ExCEL money to fund their programs until 6:00 PM, and then use Beacon money to fund programming after that time. ExCEL funding comes with specific requirements including that the funding can only be used for students at the school for which it was designated and, that funds must be divided evenly 3 Mission listed ExCEL funding on its work plan, but reported during the site visit that the school decided not to turn over the ExCEL money to them. II-10