First 5 Sonoma County Program Evaluation Brief

Transcription

1 First 5 Sonoma County Program Evaluation Brief 07/1/ /30/2012 AVANCE Parent-Child Education Program February 2013 Prepared For First 5 Sonoma County Prepared By LFA Group: Learning for Action LFA Group: Learning for Action enhances the impact and sustainability of social sector organizations through highly customized research, strategy development, and evaluation services.

2 Executive Summary Program Name Contractors Program Details AVANCE Parent-Child Education Program (PCEP) Community Action Partnership and Petaluma Adult School Terms of Grant July 1, 2010 to June 30, Strategic Plan Goal Area Goal 2: Supported Families and Nurturing Families Priority Outcome Strategic Plan Core Program Outcomes (First 5 Sonoma County Pathways to Results) First 5 Sonoma County Funding Amount Leveraged Funding Received During First 5 Grant Period Level of Evidence at Start of Grant Priority Outcome 2B: Strengthen family literacy Increase in parent knowledge of child development and parenting skills Increase in family literacy skills, including parent-child read aloud time Increase in parent knowledge of and access to community resources Increase in parent interactions with children and encouragement of language acquisition Community Action Partnership: $$555,000 Petaluma Adult School: $555,000 No additional funding has been leveraged Evidence-based Program Recognized as Tier 1 in the Portfolio of Model Upstream Programs Key Accomplishments AVANCE PCEP has achieved three of its four evaluation targets as well as two additional targets during the fiscal year: Implemented program with fidelity to the AVANCE PCEP model: Three sites have implemented the program with fidelity to the AVANCE PCEP model, indicating that the programs are on track to increase parent knowledge of and access to community resources and increase parent interactions with children and encouragement of language acquisition. The fourth site has not yet been rated for fidelity. Increased participants knowledge of child development and parenting skills: 90% of program graduates reported improvement in their knowledge of child development and parenting skills. First 5 Sonoma County Evaluation Brief: AVANCE LFA Group: Learning for Action February

3 Key Challenges and Lessons Learned Program Challenges: 63% of graduates reported that they increased the frequency with which they read to their children aloud, falling short of the 75% annual target. Program Adjustments: The program experienced lower than hoped for completion/graduation rates, with 61% of its participants attending 75% of classes required to graduate in , due in part to low enrollment and attendance in an afternoon class. Thirty-three of 43 participants completed the program in , making a total of 94 graduates (65%) for the two program years. Program staff may want to consider ways to address their participants barriers to regular attendance, such as transportation, work schedule challenges, and family obligations. This evaluation brief provides an update on AVANCE PCEP progress toward key evaluation targets including populations served and outcomes achieved during the reporting period. This brief is intended to be a resource to guide program development implementation and improvement. It is also intended to inform the First 5 Sonoma County Commission of the impact of its investments and lessons learned for future funding decisions. 1 Program Description The AVANCE Parent-Child Education Program (PCEP), a 40-year old program headquartered in San Antonio, Texas, provides education and support to Latino parents with children under the age of three. It is founded on the belief that a parent is a child s first and most important teacher, so it is critical to train parents to have the skills to support optimal child development, literacy, and school readiness. The program operates in housing projects, community centers, and schools. AVANCE PCEP instructors guide parents through their children s stages of physical, cognitive, social, and emotional development with special attention to the importance of reading, nutrition, expressing affection, and effective discipline. Parents attend weekly three-hour workshops for nine months. Community Action Partnership conducts the workshops in Spanish and Petaluma Adult School s program is presented in a bilingual English and Spanish format. Both programs provide early child education programs. During the first hour, parents make an educational toy using typical household items and are taught to use the toy in activities that will promote their child s development. Parents receive a possibility sheet with suggested activities and vocabulary to use at home. During the second hour, the program covers a section of a 27-lesson sequence based on community college curriculum on early childhood development. During the third hour parents are introduced to an array of community resources, services and providers. Parents receive monthly home visits beginning in the third month. After the first year, parents are encouraged to continue on to classes in literacy, English, obtaining a GED, and enrolling in college or career training. During the program year, First 5 Sonoma County funded two contractors to administered the AVANCE PCEP: Community Action Partnership of Sonoma County (Community Action): The federally designated anti-poverty agency in Sonoma County that partners with low-income families and individuals to help them achieve economic and social stability, build community, and advocate for social and economic justice. Community Action operated the AVANCE PCEP at: o Kawana: An elementary school in Santa Rosa 1 For a complete description of the First 5 Sonoma County evaluation approach, please see the Evaluation Plan at First 5 Sonoma County Evaluation Brief: AVANCE LFA Group: Learning for Action February

4 o The Mountain Shadows campus (Mountain Shadows): A campus comprised of a Title 1 funded preschool program operated by the YMCA, the district s alternative education programs and the district administration office. Petaluma Adult School (PAS): Part of the Petaluma City School Districts, PAS offers an opportunity for students to continue their education on a high school or post-high school level suitable for adults. PAS operated the AVANCE PCEP at Petaluma s two sister schools serving a combined school attendance area: o McDowell Elementary School (K-3) o McKinley Elementary School (4-6) Program Theory Need for the Program AVANCE PCEP meets a pressing need for programming that will promote the long-term educational success of Latino children in Sonoma County. Latino children demonstrate lower educational outcomes than their white, non-latino counterparts; in 2011 only 27% of Latino third graders in Sonoma County scored proficient at reading, compared to 61% of white students in Sonoma County. 2 In Sonoma County public schools, the number of English Learner (EL) students increased by 52% from 2001 to How the Intervention Links to Outcomes AVANCE PCEP is an evidence-based program that supports children s school readiness by strengthening family literacy. A significant body of research has linked the AVANCE PCEP to the core outcomes addressed within this report, as well as to longer-term positive outcomes. The Carnegie Corporation of New York found, in a 1987 evaluation, that AVANCE PCEP mothers provided a more stimulating, organized, and responsive home environment, talked more frequently with their children, spent more time teaching their children, and increased their knowledge of parenting skills. 4 Additionally, a study by Project CAN PREVENT found that the AVANCE PCEP effectively improved conditions which are often linked to the prevention of child abuse and neglect, such as parents self-esteem, social support network, and knowledge of child development. 5 As an early childhood intervention program, AVANCE PCEP is well-positioned to support children s school readiness and achievement. Research shows that the positive effects of parental involvement increase when involvement starts early in a child s education, and engages parents in working directly with their children at home. 6 The AVANCE PCEP encourages early parental involvement and promotes family literacy skills that support children s school readiness. According to a 2006 study conducted by the Dallas chapter of AVANCE, 95% of parents reported that they play developmental games with their children and read to them always or sometimes, while 98% of parents reported discussing their child s school experiences with them daily. 7 Furthermore, Pre-K 2 California Department of Education. (2012). Standardized Testing and Reporting (STAR) Results. Retrieved from 3 California Department of Education, Educational Demographic Unit. DataQuest EL Report: Time Series Number of English Learners for Sonoma County. Report viewed on 2/22/12. 4 As cited in Rodriguez, G. (2003). Connecting with parents: The AVANCE experience. Children s Research Center at University of Illinois-Champaign. Retrieved from Report viewed on 2/4/13. 5 Ibid. 6 Cotton, K., Wikelund, K. Parent Involvement in Education. Northwest Regional Educational Laboratory, School Improvement Research Series. 7 (2006). Maternal Attitudes and Behaviors Supporting Education. Retrieved from Report viewed on 2/4/2013. First 5 Sonoma County Evaluation Brief: AVANCE LFA Group: Learning for Action February

5 to kindergarten teachers in a 2000 survey reported that AVANCE children are ready for school. 8 The teachers reported that 82% of AVANCE children knew their letters, 86% knew their numbers, and 89% could express themselves verbally. Many preschool and kindergarten accomplishments, such as the oral language skills and letter identification skills, reported in AVANCE children, are closely related to later success in conventional literacy tasks. 9 The AVANCE program s promotion of long-term academic achievement is further supported by research which indicates that reading difficulties can be prevented by ensuring families and preschools provide experiences for language and literacy development. 10 Long-Term Cost Savings By engaging Spanish-speaking families in activities that support children s school readiness, AVANCE PCEP will likely generate public sector cost savings by helping to close the achievement gap between white and Latino students. According to the Alliance for Excellent Education, if students of color graduated at the same rate as white students, more than $101 billion would be added to California s economy by 2020 as a result of an increase in taxable income. 11 Reaching the Target Population AVANCE PCEP is a broadly targeted program designed to mitigate the effects of poverty on children in targeted school areas in Sonoma County. The program provides services for low-income, Latino families with children ages 0-3 who traditionally enter the school system with limited schoolreadiness. Number Served Programs that serve the neediest families with children ages 0-3 invariably experiences steep attrition rates. Taking this into account, First 5 has set high recruitment and enrollment targets of AVANCE PCEP participants. The program sites have a local target to initially recruit 30 parents with children 0-3 years of age and ultimately enroll 25 families. 12 McDowell (PAS), which enrolled 30 parents, and McKinley (PAS), which enrolled 26 parents met the program target, while Kawana and Rohnert Park (Community Action), which enrolled 22 parents each, fell short of the target. Despite falling short of the enrollment target, the two Community Action sites actually had the highest completion rates among the four sites. While the program also serves children, it does not have a target number of children each year. During , the four sites served a combined total of 131 children; 84 of whom where ages 0-3 and 47 of whom where ages (2000). School Readiness of AVANCE Children. Retrieved from Report viewed on 2/4/ Snow, C. E., Burns, M. S., & Griffin, P. (Eds.). (1998). Preventing reading difficulties in young children. Washington, DC: National Academy Press. As cited in Zygouris-Coe, V. (2000). Emergent literacy. Florida Literacy and Reading Excellence Center at University of Central Florida. 10 Ibid. 11 Ibid. 12 For , AVANCE National has clarified its definition of enrollment to mean families who attend a minimum of three sessions before the enrollment period ends six weeks into the program year. 13 Although the program is designed to serve children 0-3, it also serves the older siblings of the 0-3 children enrolled. First 5 Sonoma County Evaluation Brief: AVANCE LFA Group: Learning for Action February

6 Exhibit 1: Families Enrolled at all Four Sites (FY ) 140 Number Enrolled CollectiveTarget = Parents/Caregivers Race, Ethnicity and Language of Participants AVANCE PCEP targets Spanish-speaking, Hispanic/Latino parents. During , all parents and almost all children (97%) identified as Hispanic/Latino (Exhibit 2 and 3). Exhibit 2: Parents Race/Ethnicity (FY ) Exhibit 3: Children s Race/Ethnicity (FY ) Multi- Racial 3% Hispanic /Latino 100% Hispanic /Latino 97% n=131 n=96 n=131 Note: Although 100 parents enrolled in the program, program staff collected race/ethnicity data for only 96. During , almost all parents (98%) and almost all children (98%) identified Spanish as their primary language (Exhibit 4 and 5). First 5 Sonoma County Evaluation Brief: AVANCE LFA Group: Learning for Action February

7 Exhibit 4: Parent s Primary Language (FY ) Exhibit 5: Children s Primary Language (FY ) English 2% English Other 1% 1% Spanish 98% Spanish 98% n=93 n=131 Note: Although 100 parents enrolled in the program, program staff collected primary language data for only 96. Special Needs Population Of the 131 children served by AVANCE PCEP in , eight (6%) have special needs. First 5 Sonoma County Evaluation Brief: AVANCE LFA Group: Learning for Action February

8 Progress Achieved In implementing the AVANCE program over the first two years of their grant, the Community Action and PAS sites have worked toward meeting the four First 5 Sonoma County Strategic Plan core outcomes that the program was intended to address: Increase in parent knowledge of child development and parenting skills Increase in family literacy skills, including parent-child read aloud time Increase in parent knowledge of and access to community resources Increase in parent interactions with children and encouragement of language acquisition Progress toward these outcomes is highlighted in the table below. Core Program Outcome Increase in parent knowledge of and access to community resources Increase in parent interactions with children and encouragement of language acquisition Increase in parent knowledge of child development and parenting skills Increase in family literacy skills, including parent-child read aloud time Progress Achieved Toward Core Outcomes 14 07/01 / / 31 / 2012 Specific Target 100% of sites will implement the program with fidelity to the AVANCE PCEP model, indicating that they are on track to meet these outcomes demonstrated by previous research 100% of sites will implement the program with fidelity to the AVANCE PCEP model, indicating that they are on track to meet these outcomes demonstrated by previous research 75% of parents show improvement in these areas 75% of parents show improvement in these areas Actual Results 100% (3 of 3) sites monitored for fidelity demonstrate fidelity to the model % (3 of 3) sites monitored for fidelity demonstrate fidelity to the model 16 90% (57 of 63) 63% (26 of 41) 17 Progress Toward Target Achieved Achieved Exceeded Not Achieved Three sites monitored in implemented the program with fidelity to the AVANCE PCEP model: This indicates that the programs are on track to increase parent knowledge of and access to community resources, and to increase parent interactions with children along with encouragement of language acquisition. The McDowell site was rated for fidelity in spring 2011 and the program improvements suggested they were implemented in Throughout this brief, progress toward targets is measured using the following definition: Not Achieved (more than 5% below target), On Track (1-5% below target), Achieved (0-5% above target), Exceeded (more than 5% above target). 15 The McDowell, PAS site did not receive a fidelity rating in FY because the national office did not complete a monitoring visit at this site. However, the site achieved fidelity when monitored in FY , and the monitor observed in FY that staff conducted home visits at the highest level of fidelity.. 16 The McDowell, PAS site did not receive a fidelity rating in FY because the national office did not complete a monitoring visit at this site. However, the site achieved fidelity when monitored in FY , and the monitor observed in FY that staff conducted home visits at the highest level of fidelity.. 17 This analysis excludes the 22 parents who reported already reading to their child seven days a week at the beginning of the program. First 5 Sonoma County Evaluation Brief: AVANCE LFA Group: Learning for Action February

9 AVANCE PCEP participants increased their knowledge of child development and parenting One of the fathers shared how skills. To assess their graduates retention of the AVANCE has changed his program s lessons, sites administered the AVANCE expectations and his family s life. He Parenting Questionaire (APQ) at the beginning and told us that before AVANCE, his focus end of the program. According to the APQ s results, was on material things like a house, at the end of the program, 90% of graduates car, and technology for his children, increased their knowledge of child development and and having money to spend as he parenting skills, thus exceeding the program target that 75% of parents show improvement in these wanted. He didn t see time spent with areas. The pre-post questionnaire measures parents his children as important or a priority. retention of the lessons taught during the program s Now, he values and looks forward to weekly three hour parent education classes and is spending time with his children therefore used as the measure of improvement. playing, reading or just being together. The program helped increase the frequency with which parents read aloud to their children, but did not reach its target. Parents at each site receive book bag containing Spanish, English and bilingual board books to support reading at home. This is a He knows that what he has learned has made him a better father and is the best advantage he can give his family. Program Staff Community Action enhancement of the AVANCE PCEP model that has been adopted across all sites. At the end of the program, 63% of graduates reported that they increased the frequency with which they read to their children aloud, falling short of the annual target that 75% of parents increase their reading to their children. While the parents who reported already reading to their child daily at the beginning of the program were excluded from this measurement, the single question about daily reading does not adequately measure the change in the quality of book sharing with children by parents. In addition to the key accomplishments specifically related to First 5 Sonoma County s Strategic Plan core outcomes described above, the Community Action and PAS sites also worked to make progress toward the followiong targets: Additional Progress Achieved 07 / 01 / / 30 / 2012 Program Outcome Specific Target Actual Results Staff trained in AVANCE model High-need parents are recruited and enrolled into program through door-to-door visits in school attendance areas High-need parents are recruited and enrolled into program through door-to-door visits in school attendance areas High-need parents are recruited and enrolled into program through door-to-door visits in school attendance areas High-need parents are recruited and enrolled into program through door-to-door visits in school attendance areas 100% of staff attend AVANCE training 100% (20 of 20) Progress Toward Target Achieved 25 enrolled per site McDowell, PAS 30 enrolled Achieved 25 enrolled per site McKinley, PAS 26 enrolled Achieved 25 enrolled per site 25 enrolled per site Kawana, Community Action Mountain Shadows, Community Action 22 enrolled Not Achieved 22 enrolled Not Achieved First 5 Sonoma County Evaluation Brief: AVANCE LFA Group: Learning for Action February

10 Program Outcome Specific Target Actual Results Program staff address barriers to parent participation to ensure parents weekly attendance and retention in the program 85% of parents successfully retained and complete the program 61% (61 of 100) Progress Toward Target Not Achieved Conduct monthly home visits 90% of families receive at least seven home visits At least 4 home visits N/A 18 AVANCE PCEP staff focused on continual learning and professional development. All site staff received training and were certified in the AVANCE PCEP model at the beginning of FY The day-long training was a review for most attendees, with all ten PAS staff and seven of ten Community Action staff returning for a second year. Community Action followed up this training, with another training that focused on aspects of the AVANCE PCEP model that are unique to each site. Additionally, site staff had many other opportunities to engage in continual learning and professional development. Before the beginning of the program year, Community Action s and PAS s Parent Educators attended a train-the-trainer workshop hosted by AVANCE National in Santa Barbara at which they learned how to develop AVANCE program staff and improve implementation at their sites to assure fidelity to the AVANCE PCEP model. Additionally, Early Childhood Education (ECE) and core program staff at the Community Action sites participated in a training hosted by an EL education specialist with expertise on how to intentionally encourage English and Spanish language development. Furthermore, two PAS staff members received training and certification on the Pasitos model, a program that engages Spanish-speaking parents and their 3- and 4-year old children in activities that encourage language and literacy development to prepare children to be successful in school. In addition to these trainings, site staff participated in two weekly meetings throughout the program year; one to debrief following the first class of the week and the second, to train and plan for the following week s classes. These meetings, which included the entire program staff, helped clarify and solidify roles and responsibilities and reinforced the interconnection of program elements within the AVANCE PCEP model. The all-staff meetings also promoted crossplanning among the core program activities: the parent and early childhood development education, and toy-making instruction, the community resource education, and the home visits. Two of the four sites did not meet their enrollment target despite an extensive, on-theground recruitment campaign. Only two of the four sites met their target to enroll 25 parents into the program. The two Community Action sites fell short of this target despite an intensive recruitment effort that included visiting neighborhood apartment complexes and knocking on hundreds of doors, distributing flyers outside the local markets, and placing posters at the local library, markets, churches, and Petaluma Health Center. All sites did, however, attract a number of new participants through referrals and word of mouth, indicating that the program has developed a positive reputation among parents in the community and may enjoy stronger enrollment moving forward. Participant retention is an ongoing challenge for all sites, and program staff members have worked hard to mitigate barriers to participants successful graduation. The sites did not meet their target to have 85% of their participants graduate: the two Community Action sites graduated 80% of its participants (17 of 18 at Kawana and 18 of 22 at Rohnert Park), while the two PAS sites graduated only 46% of its participants (8 of 26 at McKinley and 18 of 30 at McDowell). 18 Because home visits are not a requirement for graduation in the AVANCE PCEP, program staff did not record in detail parents level of participation in this aspect of the program. First 5 Sonoma County Evaluation Brief: AVANCE LFA Group: Learning for Action February

11 According to Sonoma County First 5 staff, many low-income participants struggled to regularly attend the weekly program because of transportation costs, irregular work schedules, and family obligations. The afternoon session offered at McKinley proved to be problematic for families, conflicting with early-release days at schools and husbands expectations that dinner be prepared when they returned home from work. Additionally, despite the on-site childcare offered, a number of children found it difficult to remain separated from their parent during the three hour sessions. Program staff worked hard to reduce these barriers to attendance. For families dealing with the issue of separation anxiety, sites created a personalized integration plan. At the McDowell site, staff provided parents with access to an on-site supervised counseling intern through their partnership with the McDowell Family Resource Center. Despite these supports, some children were ultimately deemed not ready to participate in the program. To address other attendance challenges, sites set up make-up sessions and gave participants the option to attend class at a different site to accommodate their changing schedules. Furthermore, program staff regularly called participants to encourage them to attend upcoming classes and to check in if they missed a class. Home visits provide an opportunity for program staff to see parents implement what they learn, and to provide parents with individual attention and feedback. The monthly home visits allow the program staff to observe how parents are putting the weekly lessons into practice. The Home Visitor observes firsthand how parents are using the toys made in class, games, and language to interact with, encourage, and nurture their children. At regular intervals, Home Visitors videotape the parent and child interaction during the visit and show the recording to all participants at subsequent weekly sessions as a learning tool to initiate group discussion. Participation in the home visits varied by contractor; however, every participating family received at least four home visits across all sites. PAS parents consistently honored the agreed-upon home visit schedule. However, Community Action parents regularly made last minute cancellations or missed their appointments. Despite the differences in the number of home visits, many Community Action and PAS parents reported to program staff that the personalized attention they received at the home visits was extremely helpful. Because home visits are not a requirement for graduation in the AVANCE PCEP, program staff did not record in detail parents level of participation in this aspect of the program. Learning for Action: Building on Success and Lessons Learned In the two years that Community Action and PAS have been implementing the AVANCE PCEP, and over the course of the evaluation, program staff and the evaluators have learned key lessons, including factors that contribute to a successful implementation of the AVANCE model and consideration for future programming. Select parents most in need of support to engage in family literacy activities. At the beginning of the program year, 22 parents reported that they already read to their children every day and 51 parents reported already reading to their children three times per week or more. Given that this is a primary focus of the program is to serve the neediest families, AVANCE national acknowledges that the greatest change will be seen among parents most in need of support, and with the most room to grow in terms of regularly reading to their children. Explore ways in which the program can better support families experiencing challenges to regular reading time at home. While the book bags provide age-appropriate books, parents still face challenges of family and household obligations that may prevent daily reading. AVANCE National is reviewing the AVANCE Parent Questionnaire to make sure questions about reading are also reflective of the improved quality of the book sharing and reading experience. Maximize on-the-ground recruitment efforts by working with partner schools and First 5 Sonoma County Evaluation Brief: AVANCE LFA Group: Learning for Action February

12 program alumni. The sites that enrolled the most parents into the program had the help of their partner school sites to identify the neighborhoods and apartment complexes where they should focus recruitment. All sites should consider taking advantage of their relationships with partner schools to maximize their recruitment efforts. Additionally, given the apparent effectiveness of word-of-mouth referrals, sites many want to consider inviting program alumni to participate in their on-the-ground recruitment drives. Develop flexible and customized solutions to address participants attendance barriers early on. The elimination of the afternoon time schedule will support improved attendance and completion rates in Program staff may also want to consider setting up a system at the beginning of the program to identify their participants possible barriers to regular attendance, such as transportation, work schedule, and home obligations. This information can help inform the time and place of the program s regular and make-up sessions. Regular phone calls and reminders to families who miss class also help families to be reminded that they are missed and welcomed back to class even if they have missed a couple of sessions. First 5 Sonoma County Evaluation Brief: AVANCE LFA Group: Learning for Action February