Wyman s National Network: Bringing Evidence-Based Practice to Scale. COA Innovative Practices Award Case Study Submission. Emily Black.

Transcription

1 Running Head: WYMAN S NATIONAL NETWORK Wyman s National Network: Bringing Evidence-Based Practice to Scale COA Innovative Practices Award Case Study Submission Emily Black Wyman Center

2 Abstract Wyman Center is a nonprofit organization with a mission to enable teens from economically disadvantaged circumstances to lead successful lives and build strong communities. Wyman's Teen Outreach Program (TOP ) is an evidence-based best practice youth development program that provides critical supports and opportunities to young people during adolescence. Based on success in St. Louis, Wyman recognized the potential for replicating TOP on a national scale and committed to building a National Network of partners and providers who can deliver TOP - and other evidence-based practices with fidelity and at such scale that teens and communities at-risk are positively impacted by the proven outcomes. To realize this goal, Wyman employed an innovative and comprehensive replication approach which advances promising practices through four key principles: fidelity, flexibility, affordability and sustainability. In July of 2009, Wyman s National Network launched, beginning partnerships with a variety of organizations with the need, desire, and capability to reach large numbers of teens at-risk with effective programming. The innovative capacity building strategy includes partner identification and recruitment, intensive program training, periodic on-site program observations and audits, and technical assistance by Wyman staff. In 2013, the National Network has the capacity to reach 50,100 teens in 32 states and Washington D.C.

3 Background Wyman Center is a nonprofit organization with a mission to enable teens from economically disadvantaged circumstances to lead successful lives and build strong communities. With a commitment to delivering real results, Wyman bridges theory and practice, using evidence-based and evidence-informed approaches to positively impact the lives of young people providing the supports and opportunities necessary to thrive in life, work and learning. Wyman's Teen Outreach Program (TOP ) is an evidence-based best practice youth development program that provides critical supports and opportunities to young people during adolescence. TOP is grounded in contemporary research, which promotes the positive development through curriculum-guided, interactive group discussions, positive adult guidance, and community service learning. It is specifically designed for young people in 6th to 12th grade and lowers risk of negative behaviors 1 (e.g. course failure, school suspension, pregnancy, etc.) while delivering on three essential goals: healthy behaviors, life skills and a sense of purpose (see Appendix, TOP Logic Model). As one of Wyman s most successful, scalable and affordable programs, TOP helps teens acquire the skills needed for positive outcomes and successful transitions into adulthood. Wyman recognized the potential impact in replicating this program on a national scale and committed to building a National Network of partners and providers who can deliver TOP - and other evidence-based practices - with fidelity to teens at-risk nationwide. Goals Wyman s goal is to build a National Network of partners that supports faithful and affordable replication of evidence-based practices at such scale that teens and communities atrisk are positively impacted by the programs proven outcomes. Beginning with Wyman s Teen Outreach Program, Wyman aims to reach 250,000 teens cumulatively by As the National 1 Philliber Research, Who Benefits Most from a Broadly Targeted Prevention Program?

4 Network grows, and partners begin to ramp up their reach, intermediary measures include the number of partners, and the number of states in which TOP is currently being replicated. Approach To realize this goal, Wyman is employing an innovative and comprehensive replication approach which advances promising practices through four key principles: fidelity, flexibility, affordability and sustainability. To ensure the fidelity of the program as it expands, National Network partners must agree to adhere to strict program delivery, data collection and evaluation guidelines. Wyman s approach to fidelity and certification process is designed to assure faithful replication with fidelity monitoring tools from training to implementation, supporting partners to maintain certification status, sustain funding, and deliver high quality programs (See Appendix, Approach to Fidelity). The replication strategy balances the need for program fidelity with a degree of customization or tailoring to the local culture and context in which the partner operates. Wyman leverages the experience and reach of partners and allows them to tailor the program to the specific needs of the teens they serve. For example, the TOP framework is flexible enough to be used in the school setting, in after-school programs, or an out-of-school enrichment program. Wyman s National Network offers a highly affordable program for teen pregnancy prevention and youth development. The cost per participant in TOP is significantly less than the cost of the only other comprehensive youth development program with strong evidence of pregnancy risk reduction. 2 The average cost of $620 includes the assumed salary of facilitators, which may already be covered in school or agency budgets, or may not exist if volunteers are used as facilitators, thus making the costs even lower and more sustainable (See Appendix, Economic Benefits). To grow to scale (defined as minimum of partners), Wyman is investing in people, technology, marketing, and government relations, using build, learn and grow 2 The Brookings Institution, Cost Effective Investments in Children

5 investments to expand systems and processes and add other resources necessary to enable sustainable, affordable scale. Implementation Wyman s National Network in July Wyman began identifying appropriate funding and pursuing partnerships with a variety of organizations with the need, desire, and capability to reach large numbers of teens at-risk with effective programming. The innovative capacity building strategy includes regional partner identification and recruitment, intensive program training, periodic on-site program observations and audits, and technical assistance by Wyman staff. Partner Identification and Recruitment Primary target partners for scale are school systems and large regional nonprofits and/or government agencies. These organizations are large and cost-effective, and can pick up many of the functions required to ensure delivery with fidelity. Secondary target partners are small to mid-sized local nonprofits with the capacity to serve several hundred hard-to-reach (due to location or other barriers) youth annually who are at-risk and in need of services. Replication partners must be willing to follow the TOP model of fidelity, and have secured or identified a funding source which will cover start-up and first year of operational expenses (See Appendix, Partner Screening Tool). Wyman s National Network regularly identifies and shares funding opportunities in the field of health, juvenile justice, and child welfare, with potential partners. Once funding is secured and contractual obligations are met, training and delivery begin. Training and Monitoring for Fidelity The network is evaluated and maintained by Wyman and training is a vital first step that infuses program content in a consistent way to all partners. Wyman s train the trainer model

6 teaches partners how to deliver an evidence-based program with fidelity. Wyman conducts fiveday comprehensive training sessions with experiential learning elements and "teach-back" opportunities. The training is designed to support partners ability to develop deeper skills in training staff, in monitoring and coaching for quality, and in overall strategies in youth development. It provides partners with skill-building lessons that will make them more effective in all areas of program management, delivery and evaluation, teen engagement, and program facilitation. Oftentimes partners and their program providers are inexperienced working with the structure and disciplined approach of a curriculum like TOP. Wyman s training and support helps them increase their skills in delivering the type of program that will lead to significant and sustainable results. Training occurs prior to program implementation and as needed. To measure program impact, each Wyman partner measures teen progress through a pre-survey at the beginning of the program year and a post-survey at the end of the program year. The teen survey measures such outcomes as course failure, suspension, pregnancy, attitude, physical and emotional safety, and engagement with community. Facilitators are surveyed at the mid-point and end of the program year, regarding implementation of the program (e.g. dosage, number of teens, etc.). Metrics from teen and facilitator reports determine evidence based outcomes and patterns for successful implementation (See Appendix, TOP Club and Partner Survey Metrics). TOP s requirements with regard to data collection and reporting further instill both the knowledge and a level of discipline and analysis that help partners maximize the impact of their efforts. The National Network s proprietary systems also enable partners to generate instant reports that they may use for program evaluation, grant seeking, and related purposes. When partners understand and put into practice this disciplined approach, they quickly recognize the benefits and seek to adopt similar approaches with all of their youth development programs. Field visits, audits and technical assistance by Wyman staff help partners hone their skills, identify and address issues quickly, and ensure continuous quality improvement at the

7 point of program delivery. Each certified partner receives eight hours of technical assistance per year in order to foster quality program replication. This service is included in their annual membership fee. Wyman follows up with TOP providers through partner audits, which may consist of additional club observation, data, reports, or review of records to ensure efficiency of systems. Certified partner audits are based on Wyman discretion, and are conducted as needed. All TOP partners are certified by Wyman and undergo recertification on a bi-annual basis. This ensures that the partner organization maintains high standards, meet program requirements, and deliver quality programming. Required certification site visits ensure fidelity to program and contract agreements. Specific criteria determine if a partner becomes re-certified, provisionally certified or de-certified (See Appendix, Certification Criteria Rubric). Site visits occur during the first year of implementation and every two years thereafter. In addition, to share best practices, field observations occur when Wyman determines that a certified partner s successful program or systems provide learning opportunities for the National Network. Community of Practice National Network partners are part of a community of practice through which they may interact with peers who are committed to delivering excellence. As seen in the California Institute of Mental Health (CIMH) 3 Community Development Team Model, forming peer-to-peer networks to support easy exchange of information about implementation challenges and solutions is a key component of an effective evidence-based program implementation approach. To facilitate this approach The National Network is bound together through an online community. Certified replication partners are able to connect directly to Wyman with questions, comments or concerns, and benefit by receiving updates on curriculum, share lessons learned 3 Todd Sosna, PhD, and Lynn Marsenich, LCSW, Community Development Team Model, Supporting the Model Adherent Implementation of Programs and Practices, The California Institute of Mental Health, October 2006

8 and program delivery tips, access information on funding sources and other resources, and download a library of useful materials to support implementation. As a result, they learn from their participation in the National Network and are able to develop workable solutions to issues and challenges through their interaction with their peers across the country. Through TOP s community service learning requirement, partners become more engaged with other community organizations and develop stronger affiliations and collaborations with other service providers, funders, policy makers and local officials. This has the added benefit of raising awareness of youth development needs and forming proactive collaborations across a broad spectrum of community organizations that can promote and advance a pro-teen culture and attitude in the local community. Results Over the past three and a half years, Wyman s National Network has grown to include 57 certified replication partners that include school districts, nonprofit organizations, public health agencies, and other youth-serving community and government agencies. These partners, in turn, work with an estimated providers whose program facilitators run 550 registered TOP clubs in 32 states and the District of Columbia (See Appendix, National Network Map). During the school year, Wyman completed 20 certification site visits, five audits, three field observations, and five on-site trainings or technical assistance visits aimed at helping partners improve both their administrative and programmatic skills. In 2013 Wyman will conduct 8 trainings with up to 15 participants at each session, up to a 120 total participants. Our current partners and our projected new 2013 partners will train as many as 2,500 TOP Facilitators, who have the capacity to serve 50,100 teens in In terms of training quality and feedback Wyman is achieving program goals. 90% of trainers answered all the post-test knowledge-based questions correctly. In addition the average participant confidence score was 3.53/4.0; the average satisfaction score related to training quality was 3.98/4.0.

9 Additionally, Wyman s TOP is currently the subject of eight separate, ongoing research projects studying both the process and outcomes of this program. As these projects reach their conclusions, findings and recommendations will be incorporated as appropriate. Identification and Mitigation of Risk Factors/Obstacles TOP certification serves to help identify and address, in a timely manner, many challenges and ensures that the partners commitment to fidelity, and their ability to achieve it, continues to grow. However, based on experience to date, the National Network s major challenges include the following: Many partners have little or no experience in implementing evidence-based programs such as TOP. Often they have the right idea and focus, but lack the capacity, tools and organizational structure to implement a proven practice (like TOP ) in a way that is sustainable. Consequently, they may have difficulty with the need for a rigorous approach to fidelity. For example, some partners have suggested that a seven- or eight-month program would fit better with their school schedule or other priorities; even though the TOP research has shown that a nine-month dosage is needed to achieve the desired impacts. Other partners have struggled with the need to provide a minimum of 20 hours of community service learning, citing a lack of staff support, community coordination problems, safety issues, and other concerns. Still others have struggled to maintain a minimum facilitator to participant ratio of 1:25. The administrative requirements on National Network partners have proved challenging. Some partners have found that start-up can be labor intensive. Partners are responsible for ensuring that all program providers submit their data into the online system on a regular basis. Many of these providers have limited experience collecting and reporting the type of data required. Some even have little or no experience using online systems and require extra instruction on the use of the technology. Other administrative issues concern the need to obtain permissions and other forms from participants and their families, ensuring that required online

10 surveys are completed by participants and facilitators in a timely manner, and understanding both the process and the value of the data and the reports that are used for program quality assurance and improvement. All of these components are important to maintaining fidelity and achieving desired outcomes for teens. To address these issues, Wyman is working with partners to provide them with more technical assistance and more frequent site visits than anticipated in the original business model. As Fixsen notes, ongoing consultation and coaching, staff and program evaluation, and fidelity assessments are important drivers that help create and support partner behavior and lead to sustainable and effective delivery of evidence-based practices. 4 During site visits and observations, program quality is generally excellent, even as partners continue to work to meet administrative requirements. The National Network is adding additional training sessions and encouraging greater use of the online Community of Practice website to assist partners and strengthen their effectiveness. Conclusion As Wyman continues to grow the National Network, the primary focus remains on identifying partner needs and helping them improve their effectiveness. Wyman remains in learning and adaptive mode, using the lessons gathered from trainings, site visits and field observations, program audits and other partner feedback to guide thinking and program enhancements. Wyman s National Network aims to create a collective voice that will drive resources for sustainability and on-going capacity building, to effectively support teens at-risk nationwide. 4 Fixsen et al, 2005

11 Running Head: WYMAN S NATIONAL NETWORK 11 Appendix: Supporting Documents

12 Running Head: WYMAN S NATIONAL NETWORK 12

13 13

14 14

15 15

16 16

17 17