Evaluation of the Seattle Public School s Family Support Program

Evaluation of the Seattle Public School s Family Support Program The 17 th Annual Conference on Advancing School Mental Health October 26, 2012 Michael Pullmann, Ph.D Eric Bruns, Ph.D. Ericka Weathers, M.A. Spencer Hensley, B.A. University of Washington School of Medicine, Seattle, WA Isabel Muñoz-Colón, MPA City of Seattle Office for Education, Seattle, WA Acquinetta Williams and Micheal Melonson, BA Family Support Program, Seattle Public Schools, Seattle, WA 1

2011 Families & Education Levy: Goals Goals of Levy Kindergarten readiness Improving academic achievement and closing the achievement gap Students graduate from high school college/career ready Performance Measures (Outcomes/Indicators) Academic growth Passing courses Attendance Gains in English language acquisition 2011 Families & Education Levy 2

2011 Families & Education Levy: Investment Areas Administration: 9,364,377 4.0% Evaluation: $1,400,000 0.6% Health: $44,365,128 18.8% Early Learning: $61,050,064 25.9% High School: $20,728,408 8.8% Middle School: $44,593,762 18.9% Elementary School: $54,007,694 22.9% 2011 Families & Education Levy 3

Family Support Program Former Outcomes & Indicators Attendance Homework Behavior Mobility Parent Involvement Current Outcome & Indicators Attendance Reading

Family Support Program ACADEMIC SUPPORT TEAM Links w/ teachers, school and staff Communication w/ families Support for family events ACADEMIC SUPPORT TEAM Meets weekly or bi-weekly COMMUNITY BASED ORGANIZATION Cultural/Linguistic Support Outreach Specific Indentified work w/ sub groups of focus student Connection w/ community services and resources Monitors student progress and assessments Plans academic events for families Updates student/family data

FSW Special Projects Latino Achiever Academy African American Male Leaders Club Primary Learning and Intervention Project Parent and Student Learning Libraries Kindergarten Transition Nights Pre-K Referral Process MSW Project Project which will work closely with FSW at Tier 3 schools providing intensive/wrap around support to case load students.

The African-American Male Project s goal is to provide interventions that inspire, promote and instill character, along with strategies and lifestyle standards that enable these young men to aspire and achieve their dreams and reach their destiny.

Community Service Partnerships Boeing Employees CU Washington State 4- H Junior Achievement Therapeutic Health Services/Central Youth and Family Services Rainier Vista Boys & Girls Club Councilmen Larry Gossett, King County Government Seattle Fire Department Training Facility Blacks at Microsoft (BAM) Other Committed Individuals & Volunteers

Purpose of the Family Support Program Evaluation Funded by Seattle s Office for Education (OFE) To better understand the Family Support Program, the activities and the services the program provides, and the outcomes experienced by students and families. 10

FSP Evaluation Questions Is there a clear understanding and shared vision among stakeholders (administrators, program staff, others) for the rationale, activities, and measurable goals of the FSP? If so, what is it? Where are there differences in opinion, and why? Are there national models or best practices for providing support in school settings? How is the FSP similar to or different from these practices? Which practices might be beneficial to add? 11

FSP Evaluation Questions Do students served by the FSP show more improvements in the following outcomes than statistically matched comparison students who did not receive FSP services: Increased attendance Decreased disciplinary actions Improved standardized test (MAP) scores English Proficiency scores Mobility What are the mechanisms and moderators of change, i.e.: Particular types of FSP services/fsw activities Dosage effects, e.g., increased support is related to more improvement? 12

Methodology Literature Review Advisory and Leadership Committees Logic Model Site Observations Quantitative Data Analysis Focus Groups 13

From Family Support to Student Success: The Theory Identify children who are struggling Identify service gaps and family resource needs that are barriers to children s learning Build and promote connections among school, community resources, and families to help fill gaps and meet needs Promote family engagement, involvement, and family voice in education Address family needs in an empowering way Increased family access to school and community resources Increased family involvement in education at home and at school Increased family self-efficacy, self-esteem, and advocacy skills Reduced barriers to academic success Family better able to meet basic needs Reduced mobility Improved attendance Improved homework completion Improved behavior in school Increased academic performance, social skills, and health of students Altered life trajectory of atrisk youth and families Note: This model is a slightly adapted version of a model described in Kalafat, 2004 14

RESULTS Focus Groups 15

Focus group research methods In February and March we held 7 focus groups: 2 groups with FSWs (10 FSWs) 1 group with principals (2 principals) 2 groups with school staff (6 teachers, 3 counselors, 1 student support specialist) 1 group with parents (5 parents) 1 group with Office for Education and debriefing conversations with FSP administration and the Office for Education 15 schools were represented 16

Main findings from focus groups Much support for the program from Teachers, Principals, Counselors, and Parents If it wasn t for the FSP, I d Quit Have to move into my office I don t know what I would do We heard dozens of anecdotal stories from Teachers, Parents, Counselors, FSWs, and Principals about families that were helped by the program Most stories tied the work of the FSP to improved academic functioning, even without us asking about this directly 17

Main findings from focus groups Parents reported feeling like they had an advocate and someone they could trust, in spite of distrust of the school district or animosity with teachers and principals If it wasn t for the FSW, I would be in jail for assault. 18

Main findings from focus groups System for recording FSW activities was not considered useful FSWs wanted involvement in restructuring data collection system and system for case management 19

Main findings from focus groups Agreed-upon indicators of program success: Parent involvement in schools Attendance Disciplinary actions Teacher ratings of student behavior Mobility Measures of child academic progress and growth based on standardized tests Minor disagreement on: Homework completion Strong disagreement on: Standardized test cutoff scores 20

Main findings from focus groups FSWs asked for some additional support Continue to build and expand networks of support among the FSWs; Desire for professional development seminars that are more targeted towards collecting and using data for decision making, and activities focused on academic success; More individual consultation and help with complex cases and navigating the school district. 21

RESULTS Students Served Activities of FSWs Relationship to Academic Outcomes (Individual and School Level) 22

Focus schools 2010-2011 26 focus schools Mean Range Total Enrollment 406.04 264 762 % Special Education 14.9 8.0 28.4 % Free/Reduced Lunch 71.3 39.2 94.6 % Teachers w/ at Least Master s Degree 62.6 48.6 79.1 Unemployment % in community 8.0 1.6 17.2 % Below Poverty Line in community 16.8 4.7 40.0 23

Students served in 2010-2011 1,083 focus students served (K-5) 1,545 non-focus students served (K-5) Total of 2,628 students served (about 10% of SPS students K-5) 23,923 other students K-5 who did not receive services 31% of these focus students were also focus students in 2009-2010 17% were focus students in 2008-2009 Race/Ethnicity FSP All others students % % -White 5% 47% -Hispanic 24% 12% -Asian 12% 17% -Native American 3% 1% -Black 51% 16% -Multi-Racial 4% 6% -Native Hawaiian/Pacific Islander 1% 1% Free/Reduced Lunch Eligibility 87% 37% Male 54% 51% Special Education 21% 12% English Proficiency 57% 79% Living with two parents 41% 74% 24

Parent surveys showed significant improvement Surveys conducted by the FSP at the beginning and end of the year found statistically significant small-to-moderate-sized improvements in: Parent reports of involvement with the school, communication with teachers, helping children with homework, connecting children with resources (mentors & tutors), & more. 25

Limitations of comparison Analyses limited by weak methods Could not develop a statistically valid matched sample Many students touched by the program Can be thought of as a school-wide program Context of the schools, students, and communities is vital Because of these challenges, we use a variety of comparisons Most often, we look at individual change from the prior year 2009-2010 2010-2011 Focus students 2009-2010 2010-2011 All other students 26

Individual-level analyses Attendance No change in % attended between years Suspensions Higher % suspended in FSP between years 27

Individual-level analyses Academic performance Math standardized tests No change in % meeting standards between years 67.4% of FSP group met standards for progress, compared to 75.8% of all other students Reading standardized tests No change in % meeting standards between years 64.9% of FSP group met standards for progress, compared to 72.7% of all other students 28

Individual-level analyses Mobility No differences between FSP and all other students in % of youth moves Reasons for moves were approximately identical for both groups Focus students n=818 All other SPS students n=26,015 0 moves 67% 68% 1 move 31% 30% 2 moves 2% 2% 3 moves 0.6% 0.5% 4+ moves -- 0.1% 29

School-level analyses The FSP was not related to academic progress after controlling for poverty 30

The FSP was not strongly related to attendance after controlling for poverty r = -.526, p <.001 31

The FSP was not related to suspensions after controlling for poverty r =.579, p <.001 32

There was a trend that the FSP is related to reduced mobility after controlling for poverty r =.630, p <.001 33

There was a trend that the FSP is related to increased resources in the school, after controlling for poverty 34

Summary of overall findings The FSP Theory of change aligns with theory about facilitating positive academic outcomes by providing learning supports and addressing barriers FSWs engage in activities theoretically linked to academics Basic needs (food, transportation), communicating with parents, connections with community resources, etc. Parents report improvements in their involvement in the school and connecting their child to academic activities Some evidence for a positive impact on proximal outcomes of reduced mobility and increased resources in the school 35

From Family Support to Student Success: Testing the Theory Identify children who are struggling Identify service gaps and family resource needs that are barriers to children s learning Build and promote connections among school, community resources, and families to help fill gaps and meet needs Promote family engagement, involvement, and family voice in education Address family needs in an empowering way Increased family access to school and community resources Increased family involvement in education at home and at school Increased family self-efficacy, selfesteem, and advocacy skills Reduced barriers to academic success Family better able to meet basic needs Reduced mobility Improved homework completion Improved behavior in school Improved attendance Increased academic performance Increased social skills and health of students Altered life trajectory of at-risk youth and families Note: This model is a slightly adapted version of a model described in Kalafat, 2004 36

Summary of findings, continued These proximal outcomes do not appear to be measurably related to improved academic performance. Activities should strive to make this connection. However, methodological limitations and the lack of a control group hinder our conclusions (i.e. FSP students may have gotten worse without the program) 37

From Family Support to Student Success: Testing the Theory Identify children who are struggling Identify service gaps and family resource needs that are barriers to children s learning Build and promote connections among school, community resources, and families to help fill gaps and meet needs Promote family engagement, involvement, and family voice in education Address family needs in an empowering way Increased family access to school and community resources Increased family involvement in education at home and at school Increased family self-efficacy, selfesteem, and advocacy skills Reduced barriers to academic success Family better able to meet basic needs Reduced mobility Improved homework completion Improved behavior in school? Improved attendance Increased academic performance Increased social skills and health of students Altered life trajectory of at-risk youth and families Note: This model is a slightly adapted version of a model described in Kalafat, 2004 38

Implementation Team Team made up of staff of Seattle Public Schools and Office for Education, Family Support Workers, King County Public Health stakeholders, and principals Charged with developing and implementing an Action Plan based on recommendations from UW study Team will make recommendations on performance measures, data collection, and professional development

Recommendations: Performance Measures Narrowed number of performance measures reported to the Office for Education Focus now on one academic content area (Reading) Academic performance outcomes based on growth Removed measures that were time intensive to be collected but not closely related to student academic outcomes

Recommendation: Data Collection Removed data collection that was not useful Teacher Survey Updated data collection tools to be more aligned with what FSWs need and with performance measures Service Plans Parent Survey Streamlined data collection process Case Management Notes

Recommendation: Professional Development Include training focused on academic support of students Training on understanding student assessments and communicating information to parents and guardians

Thank you! Please contact us if you have any questions: Eric Bruns ebruns@uw.edu Mike Pullmann pullmann@uw.edu Ericka Weathers ewiggins@uw.edu Isabel Muñoz-Colón Isabel.Munoz-Colon@seattle.gov Acquinetta Williams ahebertwilli@seattleschools.org Michael Melonson mmelonson@seattleschools.org 43