Foreword Houston/Harris County After School Research Compendium, Boys & Girls Clubs of Greater Houston CASE Partnership Project

Transcription

1

2 Foreword It is my great pleasure to introduce the First Annual Houston/Harris County After School Research Compendium, summarizing local research and evaluations from a wide variety of area after school providers. Measuring the wide range of after school outcomes in a community as large as Houston is challenging, but this compendium attempts to summarize key findings and accomplishments. The following Houston area after school providers generously provided their reports on outcomes to contribute to this compilation: Boys & Girls Clubs of Greater Houston The Boys & Girls Clubs of Greater Houston (BGCGH) serves over 10,000 Houston and Galveston area youth at their ten area centers each year. A recent internal evaluation of their summer learning program demonstrated positive outcomes related to preventing summer learning loss, even showing increases in reading and math performance among some participants. CASE Partnership Project The CASE Partnership Project is a collaboration between the Harris County Department of Education, the Houston Galveston Area Council (HGAC), and the Texas Workforce Commission (TWC), with additional support from the Houston Endowment. The project served an average of almost 2,100 students per day across its 32 sties, enabling parents to maintain full time employment or participate in training. Over 90% of parents reported the program helped them worry less about employment related issues, and school day teachers reported improvements in homework quality and completion, classroom participation and motivation, and student behavior. YMCA of Greater Houston The YMCA serves over 8,000 Houston area youth during out of school time each day. Their internal evaluations demonstrate increased commitment to learning, enhanced character development, and increased problem solving skills among participants.

3 21 st Century Community Learning Centers The 21st Century Community Learning Centers (21st CCLC) program supports academic enrichment programs during out of school time, particularly for high need schools. The federal grant is awarded to the Texas Education Agency (TEA), which runs a competitive grants process to eligible applicants. Greater Houston area grantees with Cycle 5, 6, or 7 21 st CCLC grants in include Aldine ISD (Cycle 6), Be a Champion (Cycle 6), Boys and Girls Club of Brazoria County (Cycles 6 & 7), Communities in Schools of Southeast Harris County (Cycles 5 & 7), Communities in Schools Baytown (Cycle 6), Communities in Schools of Brazoria County (Cycle 6), Harris County Department of Education (Cycles 5, 6, & 7), Houston Gateway Academy (Cycle 5), Houston ISD (Cycles 5, 6, & 7), Galveston ISD (Cycles 5 & 7), Pasadena ISD (Cycle 7), S.H.A.P.E. Community Center, Inc. (Cycle 7), Spring Branch ISD (Cycle 7), and Waller ISD (Cycle 7). Key outcomes from the state level of evaluation on 21 st CCLC programs includes higher reading and math TAKS scores for students in grades 9 12, fewer disciplinary referrals for students in grades 6 12, and increased likelihood of grade promotion for frequently attending students in grades Also included in this compendium are two independent, local research reports conducted through the Houston community collaboration, ENRICH After School. What does the Houston Community Believe about After School? This independent research study, conducted by Dr. Peter Wit, with the Department of Recreation, Park, and Tourism Science at Texas A&M University, found considerable agreement in the Houston community regarding the need for after school programs, desired program outcomes, and key program components and elements that should make up a quality after school program. Responses also characterized quality programs as affordable, with opportunities for children to receive help with homework, participate in physical activity, and participate in a variety of different enrichment activities, including those that children don t get to participate in during the school day. After school Programming and Delinquency Referrals in Harris County This independent retrospective study, conducted by Dr. Clete Snell, Associate Professor and Chair of Criminal Justice at the University of Houston Downtown, found the highest number of referrals to the Harris County Juvenile Probation Department for delinquent offenses over the last 10 years came

4 from a small number of communities in Harris County. Most of the communities with a high number of delinquent referrals had fewer than three after school programs. There was also a general trend that areas with six or more after school programs were located in low delinquency neighborhoods. This was especially true among elementary and middle school aged children. Together, these studies and evaluations demonstrate that after school opportunities deliver real results, and also that there is a continuing need for community wide investment in after school for Houston s youth. It is my sincere hope that through this compendium, the Houston community can continue to work together to communicate about local research and best practices, while ensuring that all Houston area children have access to a high quality after school program. Lisa M. Thompson Caruthers, MSW, Ph.D. Director, Cooperative for After school Enrichment Harris County Department of Education Houston, Texas

5 Table of Contents Boys and Girls Club of Greater Houston... 1 CASE Partnership Project YMCA of Greater Houston Texas 21 st CCLC: Year 2 Evaluation Report What Does the Houston Community Believe about After School After school Programming and Delinquency Referrals in Harris County

6 Boys and Girls Clubs of Greater Houston Report provided by: Prentiss L. Collins Jr. Vice President of Operations Boys & Girls Clubs of Greater Houston 1520 A Airline Drive Houston, Texas

7 Academic Success Summary: The Boys & Girls Clubs of Greater Houston, BGCGH is committed to providing a world-class Club experience that assures success is within reach of every young person who walks through our doors, with all members on track to graduate from high with a plan for the future, demonstrating good character and citizenship, and living a healthy lifestyle. Academic Support and Enrichment: Primary focus School Year (OST) & Summer Provide student with individual support or tutoring in test-taking, study skills, or specific subject areas o Power Hour program: daily homework assistance program. Each club member has dedicated space and staff support to ensure that they leave the club each day with completed homework. Provide students with small group support or tutoring in test-taking, study skills, or specific subject areas. o Daily High Yield Learning Activities (HYLAs). HYLAs are fun Club activities that help young people to apply what they learned in the classroom in a practical way. o HYLAs are done in small or large groups, and incorporate games, activities, competitions/challenges that encourage problem solving, critical thinking to reinforce subject/academic content mastery Science projects; spelling bees; math basketball, book club, robotics competition, scavenger hunts, etc o Texas AIM- Significant numbers of our club members are in need of an increased academic support that we are not best positioned to address. As a Texas AIM program site, we have been able to partner with Sylvan Learning Centers to provide more than 125 club members intensive remediation and academic tutoring five days a week, for 14 weeks with specialized tutors at no cost to club members; this is a value of approximately $ 640 per student. Through this partnership, during the summer we are able to offer 2-week academic camps to help combat summer learning loss. Both the summer and the school year program consist of a specific content focus for the entire duration of the program to ensure content and skill mastery.

8 Provide additional opportunities during the summer break to accelerate learning and/or target enrichment opportunities fun with a purpose o Summer Learning Loss Program (SLL) was designed to infuse a variety of High Yield Learning Activities that reinforced state mandated learning objective and TEKS in the areas of reading, math and science. In addition to purposeful learning based activities, members had the opportunity to participate in partnerdriven programs that offered application-based experiences to help members connect learning in a meaningful way. Recommendations: Offer additional reading or math courses for students who need additional support I believe that you can weave in the following concepts that we will be working to implement over the coming school year. These items have not yet been run in our clubs, but this is the direction that we are headed: Academic Year Customized Power Hour sessions each week that focus on areas of academic deficiencies for club members. o Intentional activities that will be geared at helping to strengthen student weaknesses based on school progress reports and regular report cards o To be held in small groups once/twice a week to help students that are struggling in similar content areas. Will utilize HYLAs, learning games, and web-based activities Instituting 2 nd 8 th grade math and reading/writing groups weekly o Intentional math and reading projects that are a part of the Club experience and regularly reinforce state-wide learning objectives Academic Success Outcomes Desired Outcome Indicator of Success What Data to Collect Club members achieve ontime # and % of club members that Report Cards grade progression progress to next grade in (1) Transcripts Club members don t regress during the summer month (SLLP) school year BGC Pre & Post test results Club members graduate high # and % of club members that Transcripts school graduate high school Copy of Diploma BGCGH school communities are more supportive of BGCGH programs and activities # and % of school/club collaborations # and % of school administrators indicating a willingness to collaborate # and % of community school staff volunteering in clubs Survey Data Volunteer Data (extracted from Vision) Memorandums of Understanding (BGCA Outcome) Youth demonstrate increased engagement in school and learning, and show improved academic success # and % of club members that show academic improvement in content areas # and % of club members that indicate increased selfefficacy Report Cards Survey Data

9

10

11

12

13

14

15

16 CASE Partnership Project Report provided by: Demori Driver, LMSW, MPH Research Coordinator I Division of Research and Evaluation Harris County Department of Education 6300 Irvington Boulevard Houston, Texas

17 H C D E D I V I S I O N O F R E S E A R C H A N D E V A L U AT I O N - A U G U S T ANNUAL EVALUATION REPORT CASE Partnership Project

18 CASE PARNERSHIP PROJECT EVALUATION PROGRAM BACKGROUND Established by the Texas legislature in 1889, the Harris County Department of Education (HCDE) was originally established to govern the schools of the Harris County region. As independent school districts emerged, HCDE evolved into a supportive role, becoming a local education agency that serves the school districts of Houston and its surrounding areas, and seeks to equalize educational opportunity for all students. HCDE currently operates schools for special needs students, and provides a host of supportive services designed to improve the quality and cost effectiveness of the 26 area school districts, and numerous public charter schools, in and around the greater Houston area. HCDE formed the Cooperative for After-School Enrichment (CASE) in 1999 in order to provide a new supportive service for Harris County students: high-quality after-school programs. In its first year, CASE managed seven after-school sites, serving a total of 165 students. As the after-school field grew, CASE became the facilitator of a multi-district after-school consortium, serving over ten thousand students annually. During its thirteen year tenure, CASE has received over one-hundred million dollars in funding from national, state, and local governmental entities, as well as from charitable foundations and corporate sponsors. CASE is committed to building partnerships, and to providing the resources, leadership, and training needed to support high-quality after-school programs. Its goal is to increase the quality, capacity, and number of after-school programs in Harris County. Three strategic directions guide CASE s work in the Houston community: 1) Enhancing Program Quality, 2) Building Relationships and Expanding Resources, and 3) Promoting After-School. In , CASE managed grant funds for 71 comprehensive after-school programs: 32 through the Partnership Project, and 39 through 21 st Century Community Learning Centers (21 st CCLC) Texas After-school Centers on Education (ACE) funding. CASE also awarded grants to 15 non-profit service providers working in both comprehensive and smaller scale programs through After-School Initiative (ASI) funding provided by the Houston Endowment. In addition, CASE managed an AmeriCorps program, and implemented Kids Day projects sponsored by local and national businesses and grants. In the past nine years, CASE-supported programs have served between 10,000 and 16,000 Houston-area students annually (see Table 1). The CASE Partnership Project is the result of collaboration between HCDE, the Houston- Galveston Area Council (HGAC), and the Texas Workforce Commission (TWC), with additional support from the Houston Endowment. HCDE commits local funds to serve as matching dollars that activate a commitment of Federal childcare funds through TWC. The Partnership Project concept extends beyond

19 traditional child care subsidy programs; it involves schools and families as well. Through this collaboration, the CASE Partnership Project is able to access two streams of funding, Direct Care and Quality Improvement, with the common goal to support out-of-school time programming that promotes school readiness, early learning and literacy, and target economically disadvantaged students between the ages of Funded programs use the hours before and after school to address students academic and social development needs while engaging families in each student s education. PROGRAM NEED Harris County encompasses the Houston-Sugar Land-Baytown metropolitan area, which is the fifth largest metropolitan area in the Unites States 1, with over 6 million residents. Houston is the fourth largest city in the United States, and among the fastest growing regions in the country. 2 There is evidence that the children and youth of this fast-growing area largely do not participate in after-school programs. Research by the After-school Alliance suggests that only 15% of Texas K-12 students participate in after-school programs, but that 51% of Texas children not in after-school programs would likely participate if they were available. 3 CASE estimates that more than 350,000 children in Harris County do not have access to after-school opportunities. CASE seeks to address these needs by facilitating access to high-quality out-of-school time programming; an approach that s supported by area school officials. A recent HCDE Division of Research and Evaluation community needs assessment demonstrated that 75% of responding superintendents and assistant superintendents, and 88% of responding principals believed there was a need for additional after-school opportunities in their district. Because HCDE serves the entire county, CASE is uniquely positioned to collaborate across school districts and charters in order to ascertain the areas of greatest need for after-school opportunities. Sites are selected for programs based on rigorous selection process detailed in the next section. 1 Houston is fifth largest metro in U.S. (2012, April 5). ABC Local News. Available: 2 United States census Bureau. (2012). Census estimates show new patterns of growth nationwide. Available: 3 After-school Alliance. (2009). Texas After 3pm. Available 2

20 Table 1. CASE Service Summary Program Year Partnership Project Texas ACE Total Campuses Students Campuses Students Campuses Students , , , , , , , , , , , , , , , , , , , , , , , , , , ,567 Figure 1. Distribution of HCDE/CASE Comprehensive After-School Programs

21 GRANT MANAGEMENT Program Eligibility Public, private, and charter schools with at least 40% of their student body identified as economically disadvantaged are eligible to apply for funding. In order to participate in the Partnership Project and access grant funding, schools must have a current childcare license, subcontract with a licensed childcare provider, or obtain a childcare license from the Texas Department of Family and Protective Services. Schools may only use funds to provide programming for economically disadvantaged students between the ages of 4 and 12, or up to the age of 19 with a documented disability. Grantee Award Process Schools are selected by CASE for Partnership Project funding based on two components: 1) a Request for Proposals (RFP) and 2) an internal grant management review. The RFP process begins with all interested campuses completing an RFP application. This application is reviewed and scored by three external community reviewers. Reviewers are volunteers from the Houston out-of-school time community who have a vested interested in the viability of the field. The second component applies only to returning applicants. In order to assess if the applicant has a history of delivering a high-quality program in compliance with grant requirements, CASE staff conducts a review of past program performance with a continuation rubric. The continuation rubric is intended to determine a retuning sites capacity to meet grant requirements as outlined in the CASE Partnership Project application, including the areas of: Operations, Programming & Program Administration, Data Collection & Evaluation, and Fiscal Management. Final awardees are determined based on the final application score. Final application scores for first-time applicants are derived solely from reviewer scores of the application and supporting documents. Returning applicants final application scores come from both the continuation rubric (40%) and reviewers scores (60%). PROGRAM PHILOSOPHIES CASE actively promotes its values of 1) Engagement and Enrichment, 2) Safe Environments and Nurturing Relationships, 3) Community and Family, 4) Collaboration and Service, and 5) Choice and Voice for all of its after-school programs (see Figure 2). CASE campuses are encouraged and trained to follow best practices in the field of after-school, including providing opportunities for hands-on learning, providing a variety of activities and group settings that appeal to different interests, abilities, and learning styles, and incorporating authentic real world activities that help students accomplish something worthwhile and gain real-life skills. 4

22 CASE does not dictate which specific activities must take place at centers; instead they support site-level flexibility for campuses to develop programs according to their defined campus needs and within funding guidelines. However, CASE closely monitors program implementation, observes and ranks program quality based on nationally-recognized standards, and provides in-depth technical assistance and coaching for programs at all stages of development. CASE provides Site Coordinator training, professional development, grant compliance tools, and data support to ensure that programs are able to appropriately deliver high-quality programming to students. Figure 2. CASE After-school Best Practices and Core Values 5

23 PROGRAM OBJECTIVES Based on CASE s agreement with TWC, they established two overarching program goals and five program objectives (see Table 2). Performance on these goals and objectives is detailed in the Program Outcomes section of this report. Table 2. Program Objectives Goal Objectives Measures/Outputs Time Frame Low-income parents receive support that enables them to maintain full-time employment/training Offer regular and comprehensive child care services during afterschool hours to students Confirmed schedules Each term Programs provide quality activities that support school readiness, early learning, and literacy for students ages % of parents will report employment related benefits because of services received 90% of programs will achieve frequencies of 80% or higher on constructs related to after-school quality CASE developed survey Assessing After-school Program Practices Tool (APT-O) (National Institute on Out-of- School Time, 2007) April-May administration Once per term 75% of regularly attending students will demonstrate teacher-reported improvements in academic performance and homework completion and quality 75% of regularly-attending students will demonstrate teacher-reported improvements classroom conduct, participation, and motivation 21 st APR Teacher Survey (Learning Point Associates, n.d.) 21 st APR Teacher Survey (Learning Point Associates, n.d.) Once per term Once per term PROGRAM SUPPORT Professional Training CASE hosts monthly meetings for all its Site Coordinators called CASE Connections. These meetings allow for information sharing, training and networking. The training meetings highlight best practices in the field, and help centers translate after-school research into practice. Training topics by month are summarized below. 6

24 Table 3. CASE Connections Professional Development Summary, Training Topics August September October November January February March April May Welcome/Orientation Introductions Grant requirements Networking SETAA The Afterschool Zone Classroom Management The Afterschool Zone Emergency Operations Plan Nutrition/Health Programming Supporting English Language Learners Houston Food Bank Staff Motivation Branding, Marketing, and Connecting Your Program Project Plan End of Year Recognition of Best Practices Other training opportunities are also provided. With the support of the Houston Endowment, CASE coordinates the After-school Community Training Series (ACTS), which brings high-quality professional trainers to the Houston area to train the after-school community, including both center staff and non-profit and for-profit after-school service providers. Workshop themes were determined from needs assessments of CASE Site Coordinators, and the workshops are open for all Site Coordinators to attend. A summary of the ACTS series is presented in the table below. Table 4. After-school Community Training Series (ACTS) Professional Development, Session Title Date Description Presenter Freedom Writers Method August 30, 2011 Freedom Writers Method-based lesson plans for teachers of at-risk middle and high school students. Safety and Privacy for Social Networks, Texting and the Internet December 8, 2011 (2 sessions) Best practices for safely using the Internet, texting, and social media. Power of P.L.A.Y. February 17, 2012 High energy workshop that provides student and staff motivation techniques as well as behavior modification techniques for misbehaving or disengaged students Erin Gruwell, Author Change Agent Productions, Longbeach, CA, Staff and Youth Jim Mr. Happy Atkinson 7

25 Special Initiatives After-school Initiative Nonprofit Residencies With the generous support of the Houston Endowment, CASE also supports the After-School Initiative (ASI) Nonprofit Residency. This project provides funding and program support to local Houston 501(c)3 nonprofit organizations who provide quality direct services to existing or new after-school programs. By funding nonprofit organizations, ASI expands the Houston-area after-school service field, helps local nonprofit groups connect with schools, and provides students with unique program experiences. These exciting projects stimulate business for local nonprofits, help attract students to the programs, and provide experiences that may not otherwise be accessible to local families. Table 5. ASI Residencies, Nonprofit Resident Campus Multicultural Education and Counseling in the Art Wharton ES (MECA) Urban Harvest, Inc. Park Place Elementary Wharton Elementary School Young Audiences, Inc. of Houston Kennedy Elementary South Houston Elementary Rodriguez Elementary Raymond Academy CASE AmeriCorps With the generous funding of the OneStar Foundation and support from the Houston Endowment, CASE coordinates an AmeriCorps program that places volunteers in after-school programs for a 10-month period of service. CASE AmeriCorps members teach college and career -based curricula to small groups of students at CASE schools and other sites, and bring select groups of students to off-site culminating events at the conclusion of each Kids Day model. Members also lead students in service learning projects. In the school year, CASE AmeriCorps included 52 members. AmeriCorps-led projects were implemented throughout the grant period with 249 students at 10 Partnership campuses, including Goodman Elementary, Kate Bell Elementary, Park Place Elementary, Pleasantville Elementary, Pyburn Elementary, Raymond Academy, Ross Elementary, Scroggins Elementary, South Houston Elementary, and Whidby Elementary. A key component of the strategy of CASE is the sponsoring of Kids Days: culminating events for students that build upon enrichment curricula delivered in after-school settings. Kids Day curricula focus on providing students with exciting, hands-on activities designed to stimulate interest in different professional fields and subjects, while building and reinforcing academic skills. In the school year, CASE sponsored eight Kids Days: Kids Day Space Science, Kids Day Frost Bank, the Grand Slam Tennis Jam, Kids Day HP, Houston: Energy City of the Future 8

26 HCC Northeast, Kids Day Natural Legacy, Kids Day Healthy Northwestern Mutual, and Kids Hobby Center for the Performing Arts. Site Visits and Principal Walk-Throughs CASE staff also works to ensure sites have program feedback from direct observation. Sites are assessed each term by CASE staff with the Afterschool Program Practices Observation Tool (APT-O) developed by the National Institute on Out-of-School Time (NIOST), and campus principals are requested to visit and rate the program each term with the Principal Walk Through form. These observation assessments are not meant to be punitive, rather, the APT-O encourages continuous quality improvement, and the Principal walk-throughs encourage administrative buy-in and investment in the program. Site and CASE staff review the principal assessments, discuss areas for concern, and implement action plans for any program weaknesses. APT-O assessments are returned to the CASE Evaluator, who generates detailed feedback forms that are sent back to the site and the principal. The APT-O is discussed further in the program outcomes section of this report. PROGRAM OUTCOMES GOAL: Low-income parents receive support that enables them to maintain full-time employment/training OBJECTIVE: Offer Regular and Comprehensive Child Care Services During After-school Hours STATUS: Met All Partnership Project programs are required to operate Monday through Friday for a minimum of fifteen hours per week. Five hours of program offerings must be dedicated to academic assistance, and five hours must be dedicated to structured enrichment. Partnership Project campuses served a total of 2,750 students in Programs served an average of 65 students per day. 9

27 Local Media Coverage In , CASE programs and staff were featured several times in the local media venues. Press the Control key and click the screen shots to be directed to each clip (requires internet connection). A radio interview with CASE Director, Dr. Lisa Caruthers is also available at this link. Also included below is a feature of after-school music programs shown at the CASE End of Year celebration. After-school Information Features: - August 17, 2011 Fox Houston May 2, 2012, Univision Project Features: December 13, 2011Telemundo Kids Day Entrepreneurship May 11, 2012, Telemundo Kids Day Healthy Living June 27, 2012, Fox Houston Kids Day Hobby Center May 17, 2012, CASE End of Year Celebration Music in After-school 10

28 Table 6. Students Served, Campus Total Students Served Average Students per Day Academy of Accelerated Learning Alief Montessori Bastian Elementary Berry Elementary Bonner Elementary Boone Elementary Briscoe Elementary Davila Elementary DeZavala Elementary Emerson Elementary Field Elementary Frazier Elementary Frost Elementary Goodman Elementary Kate Bell Elementary School Kennedy Elementary Martinez Elementary Park Place Elementary School Patterson Elementary Pleasantville Elementary Pyburn Elementary Raymond Academy Rodriguez Elementary Roosevelt Elementary Ross Elementary Scroggins Elementary South Houston Elementary Stehlik Intermediate Stevenson Middle School Wharton Elementary Whidby Elementary Young Elementary

29 GOAL: Low-income parents receive support that enables them to maintain full-time employment/training OBJECTIVE: 85% of parents will report employment-related benefits because of services received. STATUS: Met Using a CASE-developed scale, parents were asked to retrospectively rate the degree to which the program helped them worry less about work-related child care issues related to employment. Not all campuses returned parent surveys. Overall, 90.9% of responding parents reported that the programs helped them worry less about employment-related issues, including getting into trouble at work, work productivity, and getting or keeping a job. Table 7. Parent Reports of Employment-Related Benefits, % Who Worry Less (N=113) Getting in trouble at work because of a child care issue 93.4% Having to leave work early to take care of your child after-school 91.2% Getting less done at work because of a child care issue 91.4% Getting/keeping a job because of a child care issue 87.6% Overall 90.9% GOAL: Programs provide quality activities that support school readiness, early learning, and literacy for students aged 4-12 OBJECTIVE: 90% of programs will achieve frequencies of 80% or higher on constructs related to afterschool quality STATUS: Not Met CASE implements quality control and improvement processes that consist of a series of both announced and unannounced site visits. Since 2009, CASE has used the Assessment of Afterschool Program Practices Observation Tool (APT-O) developed by the National Institute on Out-of-School Time (NIOST). This tool concretely defines the quality after-school practices that research studies indicate are associated with positive youth outcomes. The APT-O assesses the frequency of aspects of program quality that are associated with positive outcomes, including staff behavior, activity organization, 12

30 program flow, program offerings and space, and youth interactions. Site Coordinators are presented with the tool each fall, and are encouraged to share it with their staff. Based on guidance from NIOST, the Evaluator meets with CASE staff on an annual basis to determine which constructs of the NIOST tool will be used and rated for the grant year. In , the following constructs were selected: Overall Ratings of Program Schedule and Offerings program is organized but flexible, program offers a variety and balance of activities, instructional approaches, and group sizes Homework Organization homework time is organized and well staffed for individual assistance Transition times transitions between activities are relaxed and organized and display clear evidence of procedures and behavioral expectations Organization and Nature of Activity Observed activity is well-planned, organized, and cognitively challenging Staff Promote Youth Engagement and Stimulate Thinking staff are energetic and engaged and use techniques to help youth think through problems and reflect on learning Youth Participate in Activity Time youth are busy and engaged, enjoy the activities, follow behavioral expectations, work and solve problems independently and in groups, demonstrate choice and voice in activity After-school activities must be observed for a minimum of twenty minutes, and at least two activities must be observed per visit. Homework and transition times are not always occurring at the time of the visit, and they are only rated if observed. CASE staff assesses each site with the tool two to three times during the year. The CASE staff member provides the Evaluator with the raw data assessments, and the Evaluator then generates an in-depth feedback form based on program performance. CASE staff uses the feedback form to discuss the frequency of observed research-based high-quality practices and to formulate plans to strengthen specific areas of their program. With guidance from NIOST trainers, CASE and its Evaluator developed benchmarks for performance based on frequency of high-quality practices. These are defined as follows: 80-%100% High-Quality Practices are Consistently Demonstrated 60%-79% High-Quality Practices are Frequently Demonstrated 40%-59% High-Quality Practices are Occasionally Demonstrated Under 40% High-Quality Practices are Rarely Demonstrated 13

31 Every center should operate at a minimum of 60% achievement score, and all centers should achieve a score of 80% by the spring term. Recognizing that some centers may have anomalous days or events during observation, CASE staff members employ a strengths-based coaching approach to target areas of low performance. All sites met quality thresholds for the fall term, and 68.8% (n=22) met quality thresholds for the spring term. GOAL: Programs provide quality activities that support school readiness, early learning, and literacy for students aged 4-12 OBJECTIVE: 75% of regularly attending students will demonstrate teacher-reported improvements in academic performance and homework completion and quality STATUS: Met After-school researchers 4 often recommend the measurement of school-related outcomes directly impacted by after-school programs in order to more completely capture mediating factors of school-day success. One of these factors is homework quality and completion. School-day teachers are surveyed annually to assess their views of changes in students homework over the course of the year or semester. Teachers are asked to rate changes only for students who have room for improvement. Students who are already functioning optimally are marked Did Not Need to Improve. CASE sites are instructed to only distribute surveys to teachers for students who have attended the program for 30 days or more. In general, a threshold of 75% for teacher-reported improvements was set for teacher surveys. This threshold is based on national performance targets for 21 st Century Community Learning Centers Performance Indicators. 5 4 Piha, S. (2006). Holding California Afterschool Programs Accountable. Temescal Associates. Community Network for Youth Development, in Partnership with the California Committee on Afterschool Accountability. Available: 5 Naftzger, N., & Vinson, M. (2011). 21 st Community Learning Centers (21 st CCLC) analytic support for evaluation and program monitoring: An overview of 21 st CCLC performance data: Learning Point Associates. Available: 14

32 Across the centers overall, teachers reported improvements in homework quality and completion for 80.8% of students who had room for improvement. Teachers are also asked to rate their perceptions of changes in academic performance for students who have need for improvement. Teachers reported that 77.9% of their after-school students demonstrated increases in classroom academic performance. Teacher surveys are also used to assess changes in student behavior. In addition to homework and academic performance, the survey measures behavioral constructs related to class participation/motivation, and classroom conduct. For students who have room for improvement, teachers are asked to rate any changes they observe in students over the course of the academic year or semester. Results of the survey are presented in the table below. Across the centers overall, teachers reported improvements in participation/motivation for 87.9% of students who had room for improvement. Teachers reported improvements in classroom conduct for 81.0% of students who had room for improvement. CONCLUSION Partnership Project campuses are provided with robust support from CASE staff, including training, coaching, and opportunities for additional services through the After-school Initiative and AmeriCorps. Programs are universally offered during the critical after-school hours, and monitored closely for quality implementation. These programs are seen as highly beneficial by the parents utilizing them. Over 90% of responding parents report that the programs help them have fewer concerns about employment-related childcare issues overall. Sites largely offer quality programming, but some sites appear to lose momentum in the spring term, scoring lower on national standards of quality than the fall term, and below established CASE benchmarks. The area with scores most frequently often falling below benchmarks was Overall Ratings of Program Schedule and Offerings, defined as the program being organized but flexible, and offering a variety and balance of activities, instructional approaches, and group sizes. Campus schedules are approved by CASE staff, so the performance in this area is probably not related to a lack of variety of 15

33 offerings, but rather to inconsistent implementation of these offerings. Sites scoring below established thresholds are provided with coaching, mentoring, and opportunities for professional development. School-day teachers rated positive changes overall for students in need of improvement, and met established thresholds for performance. However, not every campus met the benchmarks. Of the teacher-reported areas, the highest performing are was for improved student motivation, and the lowest was for improved academic performance. RECOMMENDATIONS Based on these findings, the following recommendations are made: As 12% of parents worry more or the same amount about getting/keeping a job even with the childcare support, CASE should consider providing additional parent support resources related to employment support within current funding constraints. CASE currently offers priority scoring points on the RFP if campuses offer parent classes. To best align with the intentions of TWC funding, these offerings should likely align specifically with parent workforce resources. Over 30% of programs scored lower in the spring term than they did in the fall on a nationally validated tool measuring after-school quality, appearing to lose momentum. CASE staff should provide additional support and coaching in the spring term to help program retain high levels of quality, and help programs be resilient to staff and students burnout. School-day teachers observe positive changes in their participating students, particularly for classroom motivation and school-day conduct. As lesser results are observed in homework quality/completion and academic performance, CASE and site-level staff should review offerings to ensure that programs are maximizing opportunities for academic gains. This is particularly true for sites falling far below national benchmarks of 75% improvement. Overall, Partnership Project programs are providing quality services that are valued by parents and appear to deliver positive results for students. Refinements in program delivery and management can help build upon existing strengths, and improve areas in need of improvement. 16

34 YMCA of Greater Houston Report provided by: Patricia Osborn Director of Youth Development YMCA of Greater Houston 2600 N Loop West, Suite 300 Houston, TX

35 YMCA of Greater Houston Outcomes The YMCA of Greater Houston served over 8,000 youth each day during out of school time hours in Goals and performance include: Children will demonstrate a commitment to learning. 70 % of the 855 children surveyed scored 25 points or higher on the YMCA Commitment to Learning Survey. Children will demonstrate knowledge of the YMCA Core Values. 72% of the 897 children scored 12 points or higher on a scale of 15 on the YMCA Character Development Checklist. Children will demonstrate basic problem solving skills. 75 % of the 897 children surveyed scored a 25 or higher on the YMCA Problem Solving Checklist. Data provided by Patricia Osborn, Director of Youth Development, YMCA of Greater Houston

36 Texas 21st Century Community Learning Centers: Year 2 Evaluation Report Report prepared by: American Institutes for Research 1120 East Diehl Road, Suite 200 Naperville, IL

37 Texas 21st Century Community Learning Centers Year 2 Evaluation Report Prepared for: Texas Education Agency FEBRUARY 2013

38 Copyright Notice: The materials are copyrighted and trademarked as the property of the Texas Education Agency (TEA) and may not be reproduced without the express written permission of TEA, except under the following conditions: 1) Texas public school districts, charter schools, and Education Service Centers may reproduce and use copies of the Materials and Related Materials for the districts and schools educational use without obtaining permission from TEA. 2) Residents of the state of Texas may reproduce and use copies of the Materials and Related Materials for individual personal use only without obtaining written permission of TEA. 3) Any portion reproduced must be reproduced in its entirety and remain unedited, unaltered and unchanged in any way. 4) No monetary charge can be made for the reproduced materials or any document containing them; however, a reasonable charge to cover only the cost of reproduction and distribution may be charged. Private entities or persons located in Texas that are not Texas public school districts, Texas Education Service Centers, or Texas charter schools or any entity, whether public or private, educational or non-educational, located outside the state of Texas MUST obtain written approval from TEA and will be required to enter into a license agreement that may involve the payment of a licensing fee or a royalty. For information contact: Office of Copyrights, Trademarks, License Agreements, and Royalties, Texas Education Agency, 1701 N. Congress Ave., Austin, TX ; copyrights@tea.state.tx.us. Texas Assessment of Knowledge and Skills (TAKS ) and State of Texas Assessments of Academic Readiness (STAAR ) are registered trademarks of the Texas Education Agency. Other product and company names mentioned in this report may be the trademarks of their respective owners.

39 Table of Contents Executive Summary... x Overview of the Evaluation...xi Overview of Findings... xiii Chapter 1 Introduction and Evaluation Overview Evaluation Overview and Summary of Year 1 Findings Approach to Addressing the Research Objectives in Year Data Sources Organization of the Report Chapter 2 Quality Activities Features of High Quality, Engaging Activities Examples of High Quality Activities Summary Chapter 3 Organizational Processes Center Intentionality Organizational Practices Summary Chapter 4 The Influence of Center Quality on Student Participation and Outcomes Approach for Developing Quality Profiles and Variables Used in the Analyses The Relationship of Center Quality to Participation The Relationship of Participation to Center and Student Characteristics The Relationship of Center Quality to Student Outcomes Summary of Findings Chapter 5 Program Impact on Student Outcomes Program Impact: Comparing High Attendees with Low Attendees Program Impact: Comparing Participants with Non-Participants Summary Chapter 6 Summary of Findings and Recommendations References Appendices Appendix A Moments of High Engagement: High vs. Low Quality Centers

40 Appendix B. Measures Used to Create Quality Profiles Appendix C. Rasch Models: Survey and Observational Data Appendix D. Hierarchical Linear Models Employed in Program Quality Analyses Appendix E. Propensity Score Matching Methods Appendix F. Descriptive Analysis of Student Outcome Data for ACE Program Participants and Non-participants

41 List of Acronyms 21st CCLC ACE AIR AEIS APT-O CSF CSM ESEA HLM MRFM NCLB OCE PAMS PEIMS POS PPICS PQA SEA TAKS TEA TX21st 21st Century Community Learning Center Afterschool Centers on Education American Institutes for Research Academic Excellence Indicator System Assessment of Afterschool Practices Observation Tool Critical Success Factor Critical Success Model Elementary and Secondary Education Act Hierarchical Linear Modeling Multi Facet Rasch Measurement No Child Left Behind Observation of Child Engagement Profile Analysis via Multi-dimensional Scaling Public Education Information Management System Point of Service Profile and Performance Information Collection System Program Quality Assessment State Education Agency Texas Assessment of Knowledge and Skills Texas Education Agency TX21st CCLC Student Tracking System

42 [intentional blank page]

43 Texas 21st Century Community Learning Centers Year 2 Evaluation Report February 2013 Prepared for: Texas Education Agency 1120 East Diehl Road, Suite 200 Naperville, IL

44 CREDITS American Institutes for Research 1120 East Diehl Road, Suite 200 Naperville, IL Phone: Gibson Consulting Group, Inc South MOPAC Expressway, Suite 355 Austin, TX Phone: David P. Weikart Center for Youth Program Quality 124 Pearl St., Suite 601 Ypsilanti, MI Phone: Contributing Authors American Institutes for Research Neil Naftzger David Manzeske, Ph.D. Mary Nistler Andrew Swanlund Gibson Consulting Group, Inc. Amie Rapaport, Ph.D. Joseph Shields David P. Weikart Center for Youth Program Quality Charles Smith, Ph.D. Anna Gersh Samantha Sugar Submitted to Texas Education Agency 1701 North Congress Avenue Austin, Texas Phone:

45 [intentional blank page]

46 Executive Summary The 21st Century Community Learning Centers (21st CCLC) program, authorized under Title IV, Part B, of the Elementary and Secondary Education Act (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB), supports the creation of community learning centers that provide academic enrichment opportunities during non-school hours for children, particularly students who attend high-poverty and/or low-performing schools. 1 The federal grants are awarded to state education agencies (SEAs), which, in turn, make competitive awards to eligible grantees to support afterschool and summer learning programs. 2 In July 2002, the federal government awarded the Texas Education Agency (TEA) $24.5 million to fund TEA's first cohort of 21st CCLC grantees for the school year. As of , this and subsequent federal funding has resulted in 281 grants being awarded in Texas over seven funding cycles. 3 All centers funded by the Texas 21st CCLC program, known in Texas as the Afterschool Centers on Education (ACE), 4 are expected to provide programs and services designed to support student performance in the following areas: academic performance, school attendance, school behavior, promotion rates, and graduation rates. 5 These five areas were examined by the evaluation team, and findings on gains made toward meeting the performance objectives will be presented in the Year 2 Evaluation Report. To ensure that grantees funded by the ACE program are positioned to achieve program objectives, TEA has developed a research-based Critical Success Model (CSM). This model includes four Critical Success Factors (CSFs) which represent behavioral changes that should be demonstrated by students and families enrolled in the program, or by the adults working on their behalf, to ensure success in meeting programmatic goals and objectives. The CSFs and corresponding milestones (key strategies that establish the foundation on which critical success factors are built) follow. 6 1 For more information see 2 Grantees include local education agencies, non-profits, for-profit organizations, institutions of higher education, and city or county government agencies. 3 A cycle represents a cohort of grantees that receive funding for five years. Cycle 5, for example, represents the fifth such cohort to receive funding since TEA began funding for this grant. 4 In Texas, the 21st CCLC program has its own unique brand that communicates the characteristics of the program and creates statewide awareness so that all Texas centers can identify themselves as part of a bigger picture. While 21st CCLC is the federal funding source, the programs in Texas are referred to as Afterschool Centers on Education, or Texas ACE. The term ACE will be used throughout the report to refer to the programs in Texas unless reference is made to the federal funding source, in which case the term 21st CCLC will be used. 5 For more information review the authorizing legislation as part of the Elementary and Secondary Education Act (2001), Title IV, Part B at 6 Beginning in , with the sixth funding cycle, program guidelines were revised to require grantees to develop and implement programs in alignment with TEA s research-based CSM. Cycle 6 grantees were required to use this model to establish program goals and implement their programs. In addition, Cycle 6 grantees must collect and report performance measure data to TEA based on milestones and CSFs. Cycle 5 grantees are not subject to these requirements, although they were made aware of the American Institutes for Research Texas 21st CCLC Final Evaluation Report x