ExCEL After School Programs Evaluation Plan

Transcription

1 ExCEL After School Programs Evaluation Plan Section One: Vision for the Evaluation In , San Francisco ExCEL is trying to accomplish two goals simultaneously: First, ExCEL is striving for increased compliance and uniformity in the wake of several years of audit findings and compliance issues. At the same time, ExCEL seeks to preserve the unique aspects of its model, including the unique approach of its various CBO partners who oversee the day-to-day operations of individual ExCEL sites. Toward that end, ExCEL seeks to gather information to establish the nature and success of its current programs in order to tell this year s story of San Francisco ExCEL. This story needs to be told both to the district, state and federal parties invested in program compliance, and to the CBO partners, parents of enrolled students, district staff and other community stakeholders invested in program quality and a diversity of opportunities for San Francisco young people. In future years, this story will also serve as a starting point for the professional development and support the ExCEL office offers sites and will suggest evaluation questions ExCEL will pursue. As an additional process goal, ExCEL seeks to build upon its relationship with its many CBO partners by ensuring that information collected by the evaluation process is returned to sites quickly. This transparency is key both in light of recent compliance concerns and as sound program practice; sites who receive their information back quickly will trust that ExCEL evaluation dollars are being put to good use and will be able to use these data to inform ongoing San Francisco ExCEL has contracted with Public Profit, an Oakland-based firm with extensive experience in out-of-school time, youth development, program administration and working with large districts with multiple CBO partners, to develop and execute an evaluation plan in line with these goals. Prepared by Public Profit 1

2 Section Two: A Theory of Action for ExCEL Programs We are mindful that the Public Profit team is entering a complex system with many established program quality expectations and reporting requirements. Specifically: ExCEL programs must answer to a variety of program implementation and outcome requirements. ExCEL programs must reach certain program performance targets such as number of kids served, average daily attendance, and total attendance (total kiddays) to comply with federal and state grant guidelines. They must also support San Francisco Unified School District s (SFUSD) current strategic goals around access and equity among students, student achievement, and district accountability to students and families. ExCEL programs are also accountable to the community as a whole. For several years, ExCEL has been a leader in the citywide Afterschool for All Group (AFA), which seeks to create afterschool opportunities for all children and families who want them and has established standards for program quality. Finally, ExCEL uses the California After School Program Quality Self-Assessment Tool (QSA) to guide program reflection and Both AFA s standards and the QSA tool therefore establish standards for program quality for ExCEL programs. Since this is a year to establish the nature and success of ExCEL s current programs, Public Profit proposes that ExCEL use a research-based Theory of Action developed in Public Profit s work with a variety of out-of-school time programs. We hypothesize that students who participate regularly in high quality programs demonstrate the greatest improvements in key outcomes, including safety, school engagement, social skills development, and academic skills and academic behaviors. Theory of Action for San Francisco ExCEL Programs Regular Participation Program Attendance In High Quality Programs Safe Supportive Interactive Engaging Diverse, accessible, inclusive School-day aligned Skill-Building Physically active Has Direct Benefits for Youth Improved safety New skills and experiences Strong relationships with peers and adults Improved academic behaviors School engagement Contributes to Other Positive Outcomes English fluency Academic performance Higher test scores Prepared by Public Profit 2

3 The above Theory of Action addresses many of these program implementation and outcomes requirements. It also will direct the writing of the story about the uniqueness and effectiveness of ExCEL and its many programs. The writing of this story, through data collection and reporting, is guided by a set of four evaluation questions: 1. What progress have ExCEL programs made toward target enrollment and daily attendance rates? 2. In what ways are ExCEL programs providing high quality services? 3. What changes in behavior, attitude, experiences and skills are observed among out of school time participants (i.e., direct outcomes)? 4. What changes in academic performance are observed among out of school time participants (i.e., contributory outcomes)? This story, or the results of this year s evaluation, will then be used to modify the above Theory of Action for future years. Prepared by Public Profit 3

4 Section Three: Evaluation Activities and Data Sources Public Profit proposes a mixed methods study design that combines observations from site visits, surveys, and student-level data to explore the evaluation questions. Key evaluation activities for the ExCEL evaluation include: Create data collection tools to reflect ExCEL priorities, including program quality and student outcomes. Coordinate with attendance management system (Cityspan) to capture attendance information. Observe program activities at 40 sites, a sample of ExCEL s programs, using an observation tool aligned with AFA and ExCEL-specific program quality goals (site visits). Create and maintain evaluation dashboard addressing attendance and program quality. Field online participant and staff/principal surveys in Spring Available in English, Spanish and Chinese. Prepare a final evaluation findings report that includes an analysis of participation, program quality and participant outcomes. Provide site-level data reports that summarize key program metrics. Prepare drafts of mandated federal and state reporting requirements for review and completion by ExCEL staff. Evaluation Data Sources EVALUATION QUESTION INDICATORS DATA SOURCES What progress have ExCEL programs made toward target enrollment and daily attendance rates? Attendance Average daily attendance Safe environment Supportive environment Interaction Program Grant Amounts (ExCEL) Program Attendance (Cityspan) In what ways are ExCEL programs providing high quality services? Engagement School-day alignment Site Visits Stakeholder surveys Physical activity Diverse, accessible and inclusive environment Prepared by Public Profit 4

5 EVALUATION QUESTION INDICATORS DATA SOURCES Skill-building activities Increased safety New skills What changes in behavior, attitude, and skills are observed among out of school time participants (i.e., direct outcomes)? New experiences Relationships with adults Relationships with peers Stakeholder surveys Academic behaviors School engagement What changes in academic performance are observed among out of school time participants (i.e., contributory outcomes)? English Fluency Academic Improvement: CST ELA Academic Improvement: CST Math CAHSEE Pass Rates Credits Earned High School Graduation A to G Completion Data from RPA: English Learner Status State Assessments CAHSEE A to G Status Credits Earned High School Graduation Note: Our ability to successfully complete the evaluation is directly related to the quality and timeliness of information received from individual sites and from the SF ExCEL program office. Substantial errors or inconsistencies with the data provided to Public Profit will affect the deliverable dates noted in the following section. Prepared by Public Profit 5

6 Section Four: Evaluation Timeline Month Activities Purpose August 2011 Evaluation Overview Presentation to Site Coordinators and CBO Partners at August Institute Learn about how programs will be assessed as part of the annual citywide evaluation; understand the timeline for key evaluation activities. Convene Working Group Determine guiding priorities for Evaluation, especially defining program quality. September 2011 October 2011 Develop Site Visit Protocol Present Site Visit Protocol Presentation to Site Coordinators. Mandated Reporting Federal APR and CDE Elementary and Middle School annual reports Site Visit Pilot Test run of Site Visit Protocol at 5 ExCEL sites. Presentation on Evaluation Update to CBOs on Evaluation activities and progress. Determine definition of program quality to be used in Ensure transparency of the evaluation. Determine reasonable (observable) definition of program quality to be used in Ensure transparency of the evaluation. November 2011 December 2011 Draft Stakeholder Surveys Create draft student and other surveys based on Evaluation Plan and prior year surveys Program Profiles Summaries of prior year based on data from the district. Site Visit Protocol Finalize items and structure. Monthly Dashboard Internal dashboard highlighting Cityspan and site visit information goes live. Mandated Reporting CDE ASSETS annual report Determine additional measures of both program quality and student direct outcomes. Demonstrate commitment to transparency. Provide information for on-going Determine reasonable (observable) definition of program quality to be used in Tell the story of ExCEL as it unfolds. Prepared by Public Profit 6

7 Month Activities Purpose January 2012 Present Final Site Visit Protocol Update CBO Partners and Site Coordinators on Evaluation progress and present site visit protocol. Quality Assurance Check (Cityspan) Check data entry and activity categorization in light of APR and CDE reporting requirements. Ensure transparency of the evaluation. Prepare selected sites for a site visit. Familiarize all sites with the definition of program quality as determined for January April 2012 February 2012 March 2011 April 2012 June 2012 Conduct Site Visits Finalize Stakeholder Surveys Translate into Spanish and Chinese. Load into online platform. Present Stakeholder Surveys Update CBO Partners and Site Coordinators on Evaluation progress and present stakeholder surveys. Field Online Stakeholder Surveys Quality Assurance Check (Cityspan) Check data entry and activity categorization in light of APR and CDE reporting requirements. Draft Annual Report Determine structure and priorities. Quality Assurance Check (Cityspan) Check data entry and activity categorization in light of APR and CDE reporting requirements. Conduct 40 site visits to observe program quality. Support on-going program improvement at those selected sites. Ensure transparency by returning site visit report within 2 weeks of site visit. Support grant compliance through site visits of FPM-selected sites. Gather measures of program quality and student direct outcomes. Ensure transparency of the evaluation. Familiarize all sites with the definition of program quality and direct outcomes to be measured by surveys. Gather measures of program quality and student direct outcomes. Begin to outline the story about ExCEL programs. Ensure ExCEL s Summer Programs compliance with federal and state grant requirements. Prepared by Public Profit 7

8 Month Activities Purpose July 2012 August 2012 September 2012 Obtain Demographic & School Day Attendance Data from district All but CST scores. Obtain Test Score Data from district Deliver Program Profiles Mandated Reporting Federal APR annual report draft submitted to ExCEL CDE Elementary/Middle School and ASSETS annual reports prepared Ensure ExCEL s Summer Programs compliance with federal and state grant requirements. Provide information for on-going Ensure ExCEL s Summer Programs compliance with federal and state grant requirements. Provide information for on-going Demonstrate commitment to transparency. Provide information for on-going Annual Report Report to ExCEL and sites. Tell the ExCEL story. Support on-going program improvement. Prepared by Public Profit 8