Evaluation of the Seattle Public Schools Family Support Program

Transcription

1 Evaluation of the Seattle Public Schools Family Support Program Final Report June 19, 2012 Michael Pullmann, Ph.D. Ericka Weathers, M.A. Spencer Hensley, B.A. Eric Bruns, Ph.D. DIVISION OF PUBLIC BEHAVIORAL HEALTH AND JUSTICE POLICY DEPARTMENT OF PSYCHIATRY AND BEHAVIORAL SCIENCES UNIVERSITY OF WASHINGTON SCHOOL OF MEDICINE 1

2 Table of Contents Executive Summary 3 Introduction 5 Methods 7 Leadership and Advisory Committees 7 Literature Review 7 Logic Model 8 Staff Training, Supervision, Consultation, and Evaluation 8 Site Observations 8 Family Support Program Data 8 Outcome Analysis: Individual-Level 8 Outcome Analysis: School-Level 8 Focus Groups 9 Results 10 Literature Review 10 Logic Model 15 Staff Training, Supervision, Consultation, and Evaluation 17 Site Observations 21 Outcome Analysis: Individual-Level Analyses 22 Student Outcomes: Individual Level Analysis and Findings 26 Outcome Analysis: School-Level Analyses and Findings 39 Focus Groups 47 Comparing the Partners for Success Program with the Family Support 58 Program Summary and Recommendations 61 Appendices 65 Appendix A: Data Source and Evaluation Variable Crosswalk 65 Appendix B Leadership and Advisory Committee Membership 68 Appendix C Logic Model 69 Appendix D FSW Activity Report Categories 72 Appendix E Case Management Review Appraisal Form 73 Appendix F Training Details 75 Appendix G Site Visit Report Form 85 Appendix H FSW Evaluation Form 86 References 90 2

3 Acknowledgements We would like to acknowledge and thank our program partners and key stakeholders for their hard work and considerable efforts in making this evaluation possible. In particular, we greatly appreciate the efforts of: Thelma Payne and Janet Preston Linn Luu-Hibbert for her extremely hard work in facilitating our numerous requests All of the Family Support Workers, especially the many who talked with us, allowed us to shadow them during their work day, contributed to the development of the evaluation, and participated on the Advisory and Leadership Committees The members of our Advisory and Leadership Committees Seattle s Office for Education, in particular Isabel Munoz-Colon, Sid Sidorowicz, and Holly Miller, for their generosity and thoughtfulness Pegi McEvoy for her leadership and support Seattle s Alliance for Education for generously sharing their data on external resources in Seattle Public Schools & All of the parents, principals, teachers, school staff, and Family Support Workers who participated in our focus groups 3

4 Family Support Program Evaluation Final Report Executive Summary The Seattle Public Schools Family Support Program (FSP) is a school-based family support program financed by the City of Seattle s Families & Education Levy that seeks to address barriers to learning, particularly for students who are struggling academically. FSP works to inspire and motivate Seattle Public School children to succeed academically by advocating for increased family involvement in education, facilitating family access to supplementary academic resources and providing support to help families overcome social, emotional, and physical barriers that may hinder student academic progress (Family Support Program, 2010). The core of the program is the involvement of Family Support Workers (FSW) in public schools. These workers interact closely with at-risk students, their families and the schools, acting as a liaison between these parties and providing information about policies and services. The City of Seattle s Office for Education (OFE) contracted with the Division of Public Behavioral Health and Justice Policy (PBHJP) at the University of Washington (UW) to conduct an evaluation of the outcomes experienced by the students and families who receive FSP services. This external evaluation engaged in a broad range of activities to examine the program and its functioning. These activities included conducting a broad literature review of family support, developing a logic model of the FSP, conducting site visits with family support workers, quantitative analyses of academic outcome data, and conducting focus groups with stakeholders including family support workers, principals, teachers, parents, counselors, school support workers, and staff from Seattle s Office for Education. Results indicated that the FSP provides a broad variety of services to families and students with high needs, principally by providing non-academic support such as basic needs assistance and connection to community resources. The FSP was also described as acting as a liaison, and sometimes a mediator, between parents and schools. Many school staff members indicated that they so valued the program that they would not know what they would do without the FSP. Our focus group participants viewed the FSP activities as connected to positive changes in family functioning, parent engagement, and academic outcomes. However, our quantitative analyses, though limited by unavoidable weaknesses in design, did not find any strong relationships between the FSP and improvements in standardized test scores, attendance, or disciplinary actions. There was some indication that the FSP was positively related to reductions in mobility and increased availability of external resources and supportive programs available to students. Therefore, the activities of the FSP are highly valued by many, but not strongly linked to indicators of academic success. Synthesizing these findings with our literature review and focus group discussions, which are described in the full report, leads to a number of recommendations that we expect would improve the functioning and outcomes of the program: 1. Come to agreement on a way to evaluate the impact of the FSP on students and families while simultaneously a) satisfying the goals of the Families and Education Levy, and b) honoring the values and experience of the FSP staff. 4

5 2. Develop a comprehensive plan for continuous quality improvement that integrates the goals of the OFE, the SPS, individual schools, families, and family support workers. 3. Develop quality data collection and feedback systems. 4. Continue to support FSWs to be collaborative components of their school by involving them in school assessments, planning, and programming activities. 5. Work with FSWs to fine-tune trainings and other professional development opportunities to help meet FSW needs. 6. Provide additional consultation and support to FSWs, particularly for families with complex needs, particularly in terms of connecting FSW activities to improving academic outcomes. 7. The OFE and Seattle Public Schools should work together to develop contracts in a timely manner. 8. Maintain and ensure the private nature of the relationships between FSWs and families. 5

6 Family Support Program Evaluation Final Report Introduction The Seattle Public Schools Family Support Program (FSP) is a school-based family support program financed by the City of Seattle s Families & Education Levy that seeks to address barriers to learning, particularly for students who are struggling academically. FSP works to inspire and motivate Seattle Public School children to succeed academically by advocating for increased family involvement in education, facilitating family access to supplementary academic resources and providing support to help families overcome social, emotional, and physical barriers that may hinder student academic progress (Family Support Program, 2010). The core of the program is the involvement of Family Support Workers (FSW) in public schools. These workers interact closely with at-risk students, their families and the schools, acting as a liaison between these parties and providing information about policies and services. The FSW is a flexible and individualized position, and the men and women in these roles provide a wide range of services: mentoring services, academic assistance, home visits, transportation, organization of events, promotion of resources, and others as needs arise. The program receives lists of students who are not meeting standards, and from this list students are selected to receive services from a FSW based on their academic performance, school attendance, behavioral concerns, and parental involvement. Particular focus is given to Hispanic and African-American males. The program is financed by the City of Seattle s Families & Education Levy (FEL). The FEL was first passed by Seattle voters in November 1990 with the goal of helping to ensure children are safe, healthy, and ready to learn. The FEL has since been renewed by Seattle voters in 1997, 2004, and In 2004, the goals of the levy were expanded to include school readiness, academic achievement and reduction of the academic achievement gap, and dropout reduction as assessed by standardized measures of academic performance implemented by the district and/or state and the number of students who stay in school and graduate. In 2011, the goals were slightly expanded to state that all children will graduate from high school college and/or career ready. There are several principles that guide the 2011 FEL implementation: 1. Use an accountability structure based on student outcomes, indicators, and performancebased contracts. 2. Encourage course corrections to improve outcomes; defund projects that do not achieve outcomes, 3. Report on student performance at least annually, 4. Continue data-sharing agreement with Seattle Public Schools, 5. Maximize partnerships to achieve outcomes, 6. Provide support for innovative academic strategies aimed at dramatically improving academic achievement, and 7. Help support academic elements of place-based community strategies for transformation of schools or feeder patterns of schools (Ordinance , 2004; Ordinance , 2011). The City of Seattle s Office for Education (OFE) contracted with the Division of Public Behavioral Health and Justice Policy (PBHJP) at the University of Washington (UW) to conduct an evaluation of the outcomes experienced by the students and families who receive FSP 6

7 services. In addition to OFE, staff from Seattle Public Schools and representatives of a range of community organizations involved with the SPS have been involved in this evaluation. PBHJP s external evaluation focuses on four main types of questions: 1. Is there a shared vision among stakeholders (administrators, program staff, teachers, parents, others) for the rationale, activities, and measurable goals of the FSP? If so, what is it? Where are there differences in opinion, and why? 2. Are there national models or best practices for providing support in school settings? How is the FSP similar to or different from these practices? Which practices might be beneficial to add? 3. Do students served by the FSP show more improvements in the following outcomes than statistically matched comparison students who did not receive FSP services: a. Increased attendance b. Decreased disciplinary actions c. Increased grades d. Improved standardized test scores e. Health care utilization (referrals and consultations) f. Reduced mobility 4. What are the mechanisms and moderators of change, i.e.: a. What particular types of FSP services are related to improvements in the outcomes described above? b. Are there dosage effects, such that increased support is related to sharper improvements? c. Are there particular subgroups of students (gender, race/ethnicity, academic profiles) that seem to benefit more than other subgroups? d. Are there particular school contexts or environments that seem to be conducive to more effective FSP delivery? 7

8 Methods We have taken a collaborative approach to the evaluation, seeking input from representatives of the Family Support Program and the City of Seattle Office for Education through a series of monthly planning meetings in leadership and advisory committees. These meetings have helped the University of Washington researchers learn more about the Family Support Program and have also helped to determine the most effective approaches in carrying out the tasks of the evaluation. The evaluation consisted of several activities: 1) Forming Leadership and Advisory Committees to provide structure and input into the design of the evaluation, and to help interpret our findings; 2) Conducting a literature review of family support in schools (broadly defined); 3) Codeveloping a program logic model from the input of program supervisors, staff, and key stakeholders; 4) Conducting site visits with family support workers at various schools; 5) Quantitative analysis of individual-level student academic records and family support worker activity reports; 6) Quantitative analysis of school-level outcomes; and, 7) Conducting focus groups with family support workers, principals, teachers, parents, school counselors, school support workers, and staff from Seattle s Office for Education. Below, we describe the methods we used for each of these activities, and this is followed by our results for each area. Appendix A provides a crosswalk of the various sources of data for the indicators and outcomes of interest to the evaluation. The sections below describe in more detail these sources of data and the methods used to carry out the evaluation. Leadership and Advisory Committees With input from staff of the Family Support Program and Isabel Munoz-Colon, we developed a Leadership Committee and an Advisory Committee to provide oversight of the evaluation approach and activities, facilitate connections in the elementary schools served by the program, and provide feedback on evaluation materials. The Leadership Committee is made up of representatives from the Family Support Program, City of Seattle Office for Education, University of Washington, community organizations that work closely with the FSP, and former parents and students of the program. Similar in overall composition, the Advisory Committee is comprised of a larger number of individuals; with over half of these individuals also serving on the Leadership Committee. Appendix B lists the Leadership and Advisory Committee Members. Literature Review A comprehensive review of the literature on family support was conducted to determine if there are national models or best practices for providing support in school settings. This review is described in detail in our earlier report. We conducted the literature review by examining manuscripts from academic literature such as journal articles and book chapters, as well as program evaluations, summaries, and reports from programs around the nation. We searched for manuscripts about school-based family support, family engagement and involvement in schools, school supports for positive student behavior, and complementary learning. We examined publications about theory, programs, and evidence. The literature review has helped to inform the evaluation approach and parameters and has been used to make several recommendations for the City of Seattle Office for Education and the FSP. 8

9 Logic Model A Family Support Program Theory of Change (also referred to as a Logic Model) was created with input from the FSP supervisor, manager, FSWs, and the evaluation Advisory Committee. This input occurred over the course of approximately five formal meetings and several impromptu discussions. The logic model was reviewed by nearly all FSWs and approved by the Leadership and Advisory Committees. The logic model was designed to depict the program model, facilitate consistent communication about the program model and its expectations, and guide the evaluation. The evaluation team used the logic model to develop the research questions, the specific questions for focus groups, the outcomes for quantitative analysis, and to compare the program to other models of family support that were found in the literature review. The logic model provides an overall map of mission of the FSP, purpose, program operation, services the program provides, program resources, and expected outcomes for students, parents, and schools. Staff Training, Supervision, Consultation, and Evaluation In order to describe the types of training, supervision, consultation, and evaluation the FSWs receive, the research team conducted several informal interviews with program leadership and reviewed documents pertaining to each of these areas. Site Observations During the months of November and December, researchers at the University of Washington conducted four separate site visits shadowing FSWs for four hours at the following schools: Bailey Gatzert, West Seattle, Northgate, and Emerson. These four schools were selected based on their geographic location, student body characteristics, and their use of specialty programs such as the Latino Achievers Club and the African American Male Models Program. Family Support Program Data Monthly activity reports from Family Support Workers, which tally the number and types of services provided to focus students and non-focus students, were provided by the FSP. Results of parent and teacher surveys administered by FSWs during the and school years are also included in this report. The teacher survey is administered quarterly and assesses student progress in the areas of school readiness, homework, and behavior. The parent survey measures parent involvement in their child s education and is administered twice per school year. Outcome Analysis: Individual-Level We integrated data provided by the City of Seattle Office for Education and the Seattle Public Schools Family Support Program on FSW activities and academic outcomes for individual youth. FSW activity data was collected by the FSWs as part of their day-to-day responsibilities. Academic data, including disciplinary actions, attendance, standardized test scores, and mobility, were originally collected by Seattle Public Schools. For this analysis, we investigated students from the most recent school year available, In some analyses, as described below, we compare these students data to their data from the prior school year(s). Outcome Analysis: School-Level Data was obtained from Seattle Public Schools on parent and school staff surveys about their school s climate, including staff ratings of the professional culture and school leadership, and 9

10 parent ratings of family engagement and family satisfaction. These surveys are conducted annually by Seattle Public Schools as part of a broad quality improvement and school assessment process. These data were integrated with data on school-wide demographics and school-wide outcomes in academic performance, disciplinary actions, mobility, and attendance, in order to assess the relationship among the FSP, academic outcomes, school climate, and family engagement. These analyses provide a different lens and help triangulate and contextualize findings from the individual-level analyses. We also examined data generously provided by Seattle s Alliance for Education describing the various outside agencies providing services to each school in the SPS. Focus Groups Focus groups were conducted during February and March 2012 to determine if there was a shared vision among stakeholders for the rationale, purpose, activities, and measurable goals of the FSP. Seven focus groups were held with school staff, FSWs, parents, and officials from the Office for Education. 10

11 Literature Review Our literature review included over 60 publications on programs, research, and theory salient to family support in schools. Our prior report that focused solely on the literature review describes the literature review in more detail, though some updates have been included in the brief summary reported here. Students face internal and external barriers to learning. Internal barriers include biologicallybased learning disabilities. External barriers stem from a variety of widely discussed societal, neighborhood, familial, school, and personal conditions that interfere with success at school and beyond. More specifically, these barriers include things such as lack of home involvement in education, lack of peer support for education, negative peer influences, lack of positive recreational opportunities, lack of community involvement, and inadequate school, social, and health support services (Adelman & Taylor, 1997, 2000). Internal and external barriers to learning and teaching contribute to active disengagement from classroom learning and lead to significant learning, behavior, and emotional problems. These barriers contribute to poor academic performance, behavior problems, and absenteeism. For families in poverty and families of color, these barriers are directly related to the opportunity gap. Chronic and persistent barriers prevent sustained student involvement, positive classroom behavior, and learning. These barriers prevent youth from fulfilling their cognitive potential and, researchers argue, frequently contribute to incorrect diagnoses of learning disabilities and attention problems. This likely has a disproportionate impact on marginalized communities (Adelman & Taylor, 2000, 2010; Kalafat, 2004). Figure 1. Connecting family support to student success (Adapted from Kalafat, 2004) Addressing these barriers can be facilitated by a variety of strategies. Providing supports which do not appear to be directly related to academic outcomes is often marginalized. Support programs often function in isolation or are not coordinated. School districts and states should 11

12 work to both de-marginalize and coordinate efforts to provide school-based student and family support (Kalafat, 2004). Forming collaborative partnerships among schools, families, and communities; opening lines of communication among families, teachers, supportive school staff, and community resources; and supporting and removing barriers to family participation and involvement in education in school and at home are also strategies to address barriers to learning. Ultimately, as depicted in Figure 1, by building collaborative partnerships, promoting family engagement, and addressing family needs, a school-based family support program can increase access to resources, increase family involvement in education, and increase family self-efficacy and empowerment. This is expected to lead to improved academic performance for students and an altered life trajectory. There are many types of family support programs provided in schools, including family resource centers, home visiting programs, and comprehensive family-based preschool and Kindergarten programs (Head Start and similar programs). Some of these programs include the Chicago Child- Parent Center and Expansion Program and the Parent Information and Resource Centers. However, the literature is relatively scarce on school programs that provide the broad range of emotional, practical, and resource support to families as does the Seattle Public Schools Family Support Program. 360 Communities Partners for Success in Minnesota was the only program that we were able to find that delivered similar types of services as the FSP. This program is based in Minnesota s Dakota and Scott Counties Elementary and Secondary Schools. Its goals are to help families address challenges to success in school, particularly through creating a home environment that values and encourages learning and achievement. Students are prepared for transition to post-secondary education and/or a career. Family Support Workers interact directly with families, teachers, and school administration to help families address their challenges. Students are referred to the program mostly due to problems with academics, attendance, behavior, homework completion, or family struggles with food, financial difficulties, or family crises. The Family Support Workers collaborate with parents, the student, and teachers to develop Family Learning Plans with concrete goals, strategies to achieve those goals, and scheduled follow-ups with parents, teachers, and students to evaluate whether the plan is being implemented by the family and school, and whether progress towards goals is being made. If progress is not being made, then barriers to progress are identified, and strategies in the learning plan are modified to address those barriers. In Elementary schools, the Family Support Workers: partner with school staff to identify students who are struggling, especially with reading; develop plans to help families implement home-based strategies; act as a communication liaison between school and families about academic and social progress; assist families in acquiring basic needs (defined as food, housing, safety, and community resources); and provide ongoing support and sustained follow-up to ensure plans are implemented. In Secondary Schools, the Family Support Workers: partner with the school to identify students who are not progressing adequately; educate and inform families about the value of academic success, graduation requirements, credit accumulation, options for credit recovery, and the progress of their child; act as a communication liaison between school and families; assist 12

13 families with basic needs; and provide ongoing support and sustained follow-up to ensure plans are implemented. Results from the evaluation of the program in Minnesota indicate that the program is highly valued by school staff and parents (Kundin, 2011). Data supports the notion that program staff communicates and collaborates with parents and teachers in order to build relationships and work towards improving students educational performance. Parents and teachers reported high satisfaction with the program. However, the data supporting the notion that the program improves student performance and parent involvement in the schools was relatively weak due to methodological problems as well as insubstantial findings. Methodologically, the sample size with complete data is small (n=40), it is likely influenced by selection bias, and there is no comparison group; rather, pre- and post-measures are used to determine change over time. In terms of the actual findings, only half of the students in the sample met their growth target on a standardized test. Without a comparison group, it is difficult to evaluate the significance of this finding, but this does seem relatively small. Additionally, the results do not indicate that there were large impacts on parents level of involvement in school activities. Out of 15 items in a parent survey that measured parent involvement in school related activities, only two items showed statistically significant improvement (statistically, when analyzing 15 items separately, at least one item would be expected to be significant due to random chance.) The two items that were statistically significant were whether the parent used educational materials sent home with the child, and whether the parent talked with the family support worker about their child s progress. Other, longitudinal studies of multicomponent early education programs that featured family involvement and support as one component have demonstrated long-term impacts of early intervention. Studies of early intervention programs from the 1960 s and 1970 s found benefits of Kindergarten and preschool comprehensive education and support programs that continued through age 28. These included higher academic achievement, higher SES, less criminal involvement, and more stable social situations. Every dollar invested resulted in over seven dollars of savings to society (Reynolds, Temple, & Suh-Ruu, 2003). Several studies have found multiple connections between non-academic support and student success. A research review of several programs found that income, economic support, and wage supports for parents was consistently related to academic improvements (Greene & Anyon, 2010). A plethora of studies has established a connection between parent involvement in education (in both school and at home) and academic success, positive behaviors, and improved attendance (Christenson, Rounds, & Gorney, 1992; Fehrman, Keith, & Reimers, 1987; Institute of Education Sciences, 2007; Kratochwill, McDonald, Levin, Scalia, & Coover, 2009; Sheldon, 2003; Sheldon & Epstein, 2002, 2004; Steinberg, Lamborn, Dornbush, & Darling, 1992; Stevenson & Baker, 1987). Research consistently disproves the outdated and narrow notion that certain types of parents parents in poverty, parents without a high school diploma, immigrants, and others cannot be engaged in their child s education (Green, Walker, Hoover-Dempsey, & Sandler, 2007; Van Voorhis, 2001). Certain parents and families may have barriers to participation, but working to 13

14 reduce these barriers is connected to successful engagement (Greene & Anyon, 2010; Kratochwill, et al., 2009). Research on multicomponent school-based family support programs is extremely limited and in need of development. The minimal research that has been done indicates that multicomponent family support programs benefit greatly from ongoing evaluation of outcomes. Research on multi-component family support interventions in schools is hampered by several methodological issues. However, there has been one very high-quality evaluation of a multicomponent, flexible, Family Resource Center program in Kentucky. This evaluation found that better implementation of the program was related to better academic outcomes, improved behavior, and increased attendance. Program implementation was assessed by six qualitative domains: 1. Whether needs assessments were conducted on students who were served; 2. The quality of the relationship between the center and the school; 3. The quality of the relationship between the center and the community; 4. The quality of the relationship between the center and families; 5. The active involvement of an advisory council; and 6. Whether activities were focused on the mission of addressing barriers to learning. Better implementation of the program, as measured by more positive responses on the six domains above, was related to better classroom behavior, more positive peer relationships, less risk for dropout, improved standardized test scores, and achieving above grade level (Kalafat, 2004; Kalafat, Illback, & Sanders, 2007). Better implementation of the program was directly related to understanding the ultimate mission of the program (improving academic success through removing barriers to learning by providing comprehensive supports) and ensuring that the flexible activities by the program are directly tied to the mission. Activities should avoid mission drift. This review suggests several recommendations for the Seattle Office for Education and the Seattle Public Schools Family Support Program. Removing barriers to student learning through providing appropriate, empowering forms of family support should be a central aspect of all schools, and should be as highly valued as teaching and administration. Multicomponent and flexible programs need to establish a clear, well-understood mission. While program activities can be flexible and tailored to the unique context of each school, all activities should work towards that mission. Regular evaluation of activities is needed to avoid mission drift. Family involvement and community collaboration is key to the process of determining what types of support and what approaches to providing support are most appropriate in their communities. This helps to ensure culturally relevant and maximally effective activities. Communication and family engagement in education is consistently related to improved academic outcomes. When considering family engagement and involvement practices, an array of types of involvement and types of approaches to encouraging involvement should be considered in order to enhance the maximum breadth of family participation. Overall, this literature review provides theoretical and empirical justification for the proposition that the services provided by the FSP will have a positive impact on student achievement, behavior, and attendance, while reducing the opportunity gap for families. The existence of at least one program with similarities in mission and function also provides an external benchmark 14

15 (albeit limited) for outcomes of the Seattle FSP. Thus, there are several findings which can directly influence policy, practice, and contextualization of the current program. At the same time, the limited research on programs that are highly similar to the FSP presents an opportunity for this evaluation to have a national influence on our understanding of the connection between supportive activities, and student and family outcomes. 15

16 Logic Model The logic model is displayed in Appendix C and described in detail below. The model depicted conveys the assertion that the Family Support Program is needed because many students in the Seattle Public School System experience significant social, emotional, and behavioral needs. These social, emotional, and behavioral needs can impede student learning and result in poor academic performance. Without intervention, these students consistently have inadequate academic outcomes, higher rates of absenteeism, low grades, disciplinary problems, are likely to drop out of school, experience unemployment, and experience issues with the law. There is often an absence of partnership between schools, communities, and families and many families often lack understanding of school expectations. The mission statement of the FSP is The Family Support Program strives to inspire and motivate Seattle Public School children to succeed academically by advocating increased family involvement in education, facilitating family access to supplementary academic resources and providing support to help families overcome social, emotional, and physical barriers that may hinder student academic progress. The logic of the program is that early intervention with highrisk students and families may promote years of good outcomes, benefits, and cost-savings. The FSP operates under the belief that supporting families by providing basic needs and connecting to community resources will help remove barriers to attendance, learning, and performance. In addition, the program was designed with the belief that supporting families connections to the school will help increase the following: communication between teachers and parents, parent participation in school activities, parental understanding of academic standards and school policies, and parental support for child s education outside of school setting. To carry out their mission, the logic model states that the FSP employs highly experienced, dedicated, and knowledgeable FSWs that are located in all public elementary schools, one middle school, and the Seattle Public Schools Bilingual Orientation Center. The FSP uses a weighted eligibility formula to determine what level of services elementary schools will receive. FSWs are then assigned to schools based on the needs of the school and the skills of FSWs. According to the FSP, placing FSWs in the schools rather than housing them off-site allows FSWs to be more accessible, intervene more consistently with students, and reduce stigmatization of families seeking help. The program boasts their effectiveness in providing individualized and flexible approaches to meeting the needs of schools, children, and families. The FSP serves children who are experiencing challenges with academics, attendance, homework completion, mobility, parent involvement, and behavior. Due to the large disparities in academic outcomes for Hispanic, African American, and African students in the SPS, the program has a particular focus on Hispanic and African American males. FSWs work with community organizations, schools, and others to perform a variety of tasks to assist parents, students, and families. Some of these tasks are highlighted below. For a more comprehensive list of activities, please see the Logic Model in Appendix C. 16

17 Parents: Serve as a liaison to the school and as a navigator to information, policies, and services of the school system. Provide basic needs such as housing, food, and utilities. Organize parent educational events. Provide emotional support, mentoring, encouragement, and motivation. Connect parents to community resources, services, and programs. Support educational endeavors of eligible parents. Students: Provide mentoring and/or connections to mentoring services. Assist individual students with school work and academic support. Provide case management for homework and attendance concerns. Connect youth to services that will contribute to their academic success. Meet basic needs (e.g. recreation, mental health, and nutrition). Play a direct role in behavior or academic plans as defined in plans developed by the school s SIT team. Families: Provide case management Help in times of crisis Make home visits Provide transportation Facilitate access to culturally appropriate school and community resources According to the program, their work has long-term and short-term impacts on students, families, and schools. In the short term, students served by the FSP will make improvements in the areas of attendance, school readiness, school behavior, grades and academic performance, and homework completion. In the short term, families will be better able to meet their basic needs, are more involved in their child s education, have an increased sense of hope and responsibility, and increased connection to the community. In the short term, the work of the FSP will improve teacher and school administration perceptions and understanding of students and families and will improve the climate of the school. In the long term, the program boasts increased high school graduation rates, increased college enrollment, increased vocational and technical school enrollment, increased self-respect, improved life skills, and prevention of incarceration. For a more comprehensive list of outcomes, please see the Logic Model in Appendix C. 17

18 Staff Training, Supervision, Consultation, and Evaluation To describe the program activities of training, supervision, consultation, and evaluation, the research team conducted several informal interviews with program leadership and reviewed program documents. For a description of the types of experiences and qualifications of Family Support Workers, see the table below: FSW experience N % Bilingual -Yes 5 13% -No 33 87% Education -Some college 2 5% -Certificate 1 3% -Associates 8 21% -Bachelors 19 50% -Masters 8 21% Years of Experience (Mean=14, SD=6) % % % % % Family Support Program Training In order to determine the types of training to be offered to FSWs during the course of the school year, the FSP has a process for identifying the needs of schools and FSWs. Each cluster representative speaks with FSWs in their cluster to determine their needs, to discuss difficult cases, and identify barriers associated with helping families. Cluster representatives are FSWs who serve as a team lead for other FSWs in a specific cluster. There are six clusters, which are typically grouped by geographic location. This information is then used to prepare a cluster report that is shared with the program manager and supervisor. The cluster report is comprised of information directly reported by the FSWs (e.g. FSWs may report having a large number of families on their caseload residing in tent city, an area where many homeless persons and families reside). The extent to which quantitative data is used to inform the cluster reports appears to be minimal. The program manager and supervisor meet with the cluster representatives to discuss the identified needs and determine the types of training that would be beneficial to all FSWs. The program then works to form a training committee to plan appropriate trainings. There are instances where identified needs are cluster specific. When this happens, specific training is developed to meet those needs. In addition to the cluster report, the program supervisor completes two site visits at each school served by the program. Information from the site visits is used to determine training needs. To further assist with identifying training needs, the program supervisor reviews SOAP notes, activities/contacts, and service plans as a part of a case management review on randomly 18

19 selected focus students by grade level at each school. The data and information recorded on the case management review appraisal form provided by the program focuses on whether certain activities have been completed (e.g. service plans turned in to the main office, two monthly contacts per student, etc.). For more information about the form, please see Appendix D. The program supervisor works with the program s data manager to monitor incoming data. The number of home visits, completion of SOAP notes, and service plans are examples of data that are monitored. As a part of the collective bargaining agreement between the School District and the FSWs, FSWs are required to complete approximately 40 hours of professional development per school year in order to improve their abilities and skills. The FSP occasionally relies on professional development opportunities within the school district and community to meet some of the individual training needs of the FSWs. According to the information provided to the University of Washington (UW) research team by the FSP, FSWs participate in two large training institutes each school year. The first training institute takes place in June and the second training institute occurs in August. In addition, FSWs occasionally receive training on specific issues throughout the course of the school year. During the academic year, training during the June Institute spanned the course of two days. FSWs received approximately 11 hours of training on the following: 1. Accessing available resources from the Department of Social and Health Services (DSHS) and developing ways to work in partnership with DSHS in order to provide services 2. Reporting intake and processing for Child Protective Services (CPS) 3. Working with students and families around domestic violence 4. Understanding and navigating the foster care system The August Institute for the school year was conducted over the course of four days. FSWs received approximately 20 hours of training on a variety of topics including: 1. Accountability and teamwork 2. Agency resources and partnering 3. Fathers engagement and resource support 4. Safety advice for staff during home visits 5. Community/personal safety 6. Risk management 7. Data and Medicaid 8. Homeless and transitional housing 9. Site fund 10. Gang prevention Based on what the UW research team observed during the August Institute, the training that took place on the third day was delivered in a lecture style format. During the school year, FSWs participated in a day-long joint training with elementary school counselors on the impact of mental health issues on the family system. Participants were provided with information on families struggling with mental health issues and 19

20 received support on how to cope with the experiences that come along with being a FSW or school counselor. In addition, they attended concurrent breakout sessions about working with families in recovery, depression and the family system, and the impact of situational trauma on children, families, and communities from a cultural lens. During the school year, FSWs received a day-long training on strategies for speaking with youth about race. The objectives of the training were for FSWs to learn how school-aged youth see and understand race; learn ways to support a healthy ethnic identity development in youth; and learn strategies for talking with youth about encountering bias. FSWs also received training on cultural competence and organizational change. Due to the pending results of the evaluation, planning for the June and August Institutes for the school year have been placed on hold. The FSP staff will meet in June to discuss the evaluation findings and recommendations. As a result of the meeting in June, an implementation team will be formed and the team will be responsible for planning the August Institute taking in to consideration the recommendations provided by the University of Washington. After reviewing the training materials provided by the FSP, training appears to be focused on accessing resources and services, partnering with other agencies and organizations, understanding race and culture, and working with various populations of students and families. However, there does not appear to be a focus on how to help students meet academic goals. For more information about FSW trainings, please see Appendix F. Family Support Program Supervision, Consultation, and Evaluation As a part of supervision and consultation, the program supervisor completes two site visits per year (also described above, because these site visits also inform training needs) and case management reviews for each school. The first site visit is done during the beginning of the school year to highlight areas of concern and figure out the types of support FSWs need to work through the school year. The program supervisor also helps the FSWs select focus students during the site visit. The second site visit usually occurs in January. The purpose of this site visit is to determine if FSWs are compliant with data entry (e.g. is the data being entered correctly and is the data applicable) and to ensure the FSWs are meeting the minimum requirements for seeing families. The program manager uses a site visit report form to guide the site visit. The form is included in Appendix G. In our evaluative opinion, there are important elements of the site visit that were described to us but that are not included on the report form. For example, there is not a designated space for identified training needs, though we were told this was an important aspect of the site visit. We believe the site visit report form should be revised to fully encompass the range of important issues discovered during the site visits. The program manager and supervisor also have an open-door policy, allowing FSWs to contact them for consultation if they have specific needs and/or questions. In addition, the program supervisor meets with cluster representatives on a monthly basis to discuss issues and develop strategies. Cluster representatives also meet with their cluster group on a monthly basis and meet with individual FSWs in their cluster weekly. By mid-april of each year, FSWs are evaluated as required by the Seattle School District using the Seattle Association of Education Office Professionals (SAEOP) and Paraprofessional 20

21 Classified Evaluation Form. The form is completed by the supervisor of the program and it is kept in the FSW s school district personnel file. The program supervisor uses information from site visits, case management reviews, and principals evaluations of FSW performance to complete the evaluation form. When necessary, the evaluation form is completed with input from the program manager. The form is split into three major sections; completion of the third section is optional. The first section is designed to measure FSWs performance in several areas using a four point Likert scale, with scores ranging from excellent to unsatisfactory. The specific areas of assessment are: 1. Functional/technical knowledge 2. Accountability 3. Collaboration 4. Communication/interpersonal relationship skills 5. Critical thinking, decision quality, and problem solving 6. Initiative/action oriented 7. Time management 8. Quality of work. If the FSW receives a score of unsatisfactory in any one area, they are placed on an improvement plan. The improvement plan spells out the concerns, identifies the expectations, and highlights the required resources for the FSW to address the concerns and meet expectations. The FSW is given six weeks to improve. During this six week period, the FSW works with the program supervisor on weekly basis to figure out how to make improvement. If improvement has not been made during this time period, the FSW can be terminated. The second section requires an overall rating and summary of the FSWs performance. The overall rating is based on the ratings listed in the first section. The evaluator is required to provide a summary of overall strengths as well as a summary of areas for development and critical improvement. The third section of the evaluation form relates to goals for the next review period. The evaluator assists the FSW in identifying one or two individual or professional development goals, explaining why the goal is important, and identifying action steps and necessary resources to achieve the goal. For more information about the form, please see Appendix H. 21

22 Site Observations Overall, we saw that FSWs had a very visible relationship with school staff and students. On many occasions while walking through the hallway or sitting in on a classroom session, many FSWs were stopped by teachers, students, and other school staff to say hello, to discuss needs of students, to assist in providing resources for a family in need, or to request their presence at a meeting or parent teacher conference. During these site visits, we witnessed FSWs work to help families meet their basic needs through promoting and connecting parents to community resources, services, and programs. FSWs worked with a variety of community organizations to help families obtain food for the holidays, eyeglasses and clothing for students in need, and rental assistance for parents. We also observed FSWs work to connect youth to specific programs and services that will help their academic success. For example, focus students in these schools participate in culturally appropriate afterschool programs, receive tutoring during lunch, as well as tutoring assistance from students at local colleges and universities. At three of the schools visited, we found evidence that FSWs served as a liaison to the school and as a navigator to information, policies and services of the school system. This was especially true around making sure parents and guardians understood school attendance policies. For example, FSWs explained to parents the policies around what time a student should arrive to school and helped parents to understand excused and unexcused absences from school. Many of the schools visited are experiencing funding cuts and scarce resources in combination with a student population that has high needs. This has resulted in FSWs having to come up with creative ways to address the needs of students served in their schools. For example, during our visit we were told that West Seattle does not have art or music programs so an FSW works with students one-on-one to provide drumming lessons. At Emerson we were told they no longer have access to tutoring from AmeriCorps so an FSW is partnering with a teacher that has volunteered her time to offer tutoring to students twice per week. A number of themes emerged from the site observations that aligned with the procedures, proposed outcomes, and mechanisms of change of the FSP information as reported by stakeholders and included in the Logic Model. Other themes were relatively novel and/or centered on challenges inherent in implementing the FSP and required further exploration. Some of the emergent themes included pressure to conduct more academic focused activities, operating with limited resources, and the blurring of FSW roles and responsibilities. It is also clear that there are unique contextual needs of each school that directly impact the type of work the FSWs perform. These relate to student demographic needs, school resources, school culture, support from principals and other administrative staff, and relationships with teachers. These themes were further explored during focus groups with school professionals (described below). 22

23 FSP Activities and Student Outcomes: Individual-Level Analyses Focus students are chosen early in the school year based on students with academic and family support needs. The City of Seattle s Office for Education considers any student whom the FSP has set a goal for improving their homework, attendance, mobility, and/or suspensions to be a focus student. In , there were 985 focus students K-5 and 96 focus students 6-8, for a total of 1,081 focus students K-8. Many students are informally referred to or present themselves to Family Support Workers as needing assistance. When these activities are recorded, we consider these to be non-focus supported students. In , there were 1,545 non-focus supported students/families. The table below depicts the demographic characteristics of focus students who received services from the FSP during the school year K-5 and compares these demographics to all other students in Seattle Public Schools grades K-5. Family support students were more likely to be Black, Hispanic, or Native American, and less likely to be white or Asian. They were much more likely to be eligible for free or reduced lunch (92% total vs. 39%). They had higher rates of students in special education (21% vs. 12%), lower rates of English proficiency (57% vs. 79%), less likely to be living with two parents (39% vs. 73%), and more likely to be living with their mother only (51% vs. 22%). These demographics paint a picture of a high-needs and high-risk group. Demographic Characteristics of Seattle Public Schools Students in Grades K Variable Family Support Program Focus Students (N=985) All other students (N=24,450) N % N % Race/Ethnicity -White Hispanic Asian Native American Black Multi-Racial Native Hawaiian/Pacific Islander Age Grade -Kindergarten First Second Third Fourth Fifth Free Lunch Eligibility 23

24 -Not eligible for free or reduced price lunch Prepaid lunch tickets at full price Eligible for reduced price lunch Eligible for free lunch Gender -Male Female Special Education Bilingual Fluency -Knows only English Understands/speaks no English Understands first language more Knows English equally to first language Knows more English than first language Residential Status Missing=6 Missing=131 -Both parents Father only Mother only Guardian(s) <1 -Agency/social services Spouse/partner <1 -Foster parent(s) 5 <1 53 <1 -Grandparent(s) <1 -Other relatives 8 <1 45 <1 The context of the schools and the communities the schools are immersed in is also an important consideration when examining the FSP. The table below provides information about the schools who have a FSW working within them. The mean percentages illustrate a high level of need in schools and their communities. Examining the column with the range illustrates that the need is greatly varying, schools that have up to 95% of students qualified for free or reduced lunch, 28% of students qualifying for special education, 17% of people in the community being unemployed, and 40% of people in the community living below the poverty line. These factors likely play a major role in the challenges that students, schools, and the FSP face. 24

25 Characteristics of Focus Schools (n = 26) Mean (StdDev) Range Total Enrollment (104.1) % Special Education 14.9 (4.9) % Free/Reduced Lunch 71.3 (17.7) % Teachers w/ at Least Master s Degree 62.6 (8.3) Unemployment % in community* 8.0 (3.8) % Below Poverty Line in community* 16.8 (9.9) Source: OSPI Washington State Report Card, 2011: d=100 *Note: Community data calculated based on the 2010 census tract in which each school is located, Source: American Community Survey Washington/prepared by the U.S. Census Bureau, Developing the comparison sample An extremely important issue for any study of intervention effectiveness involves the choice of what to use as a comparison to indicate success. Currently, the FSP is evaluated by the City of Seattle s Office for Education using a collection of performance targets. These targets include the numbers and percentages of students expected to have fewer than 10 absences, engaging in academic activities, turning in homework, meeting standards or making progress according to standardized tests, decreasing mobility, decreasing disciplinary actions, and other benchmarks. For our approach to examining change, we were interested in the extent of student development as compared to what would have occurred without the Family Support Program. This is commonly referred to as the counterfactual, or what would have happened to the student if the factual event had not occurred (in this case, receipt of FSP services). Researchers traditionally obtain a comparison group representing the counterfactual by randomly assigning individuals to experimental or control groups. However, other acceptable counterfactuals exist. Even setting a performance target is a representation of a counterfactual of sort, as it implies that performance at or above the benchmark is related to successfully changing student s trajectories. The choice of what comparison group will represent the counterfactual is one of the most important decisions in determining the impact of an intervention. Choosing an inappropriate counterfactual will lead to incorrect conclusions. It could be, for instance, that students in the FSP do not flourish as compared to an average student they may have higher mobility, worse disciplinary actions, and so on but these students may still be doing much better than they would have done in the absence of the program. Comparing them to changes from average students would therefore be inappropriate as a counterfactual and would lead to a misunderstanding about the effectiveness of the program. Similarly, unless based on research about the expected functioning and development of program youth in the absence of intervention, outcome-based performance targets can also misrepresent program impact. Additionally, if not based on research, performance targets can seem arbitrary to program staff. Randomized and controlled experiments which randomly assign qualified students to receive the intervention (or not to receive it) are the gold standard for determining the amount of program impact. However, for practical, ethical, or political reasons, it is often very difficult to conduct a true randomized and controlled experiment in a community setting. In the absence of a 25

26 randomized experiment, other methods exist for obtaining an appropriate comparison group to represent the counterfactual, such as matching students who received services to students who did not receive services based on important characteristics such as demographics and need. We attempted to do this through a sophisticated technique known as propensity scoring, but we were unable to conclusively obtain a sufficiently matched group. We were additionally uncomfortable with using this technique because the FSP provides services to students beyond those listed as focus students, and there is a logical and theoretical reason to believe that the services provided by the FSP may impact the school s functioning as a whole. Therefore, it would be impossible to obtain any true matches from within schools that have the FSP, because all students in those schools could be considered touched by the FSP. For this reason, even an experiment that randomly assigns individual students to the FSP would be inappropriate; randomization would likely have to occur at the level of the school. As it currently operates, selecting schools or students within schools that do not use the FSP as comparison matches would also be inappropriate, because there important predictors of success that are correlated with whether a school is chosen to have the FSP. Because we were unable to identify one appropriate comparison group, we use a variety of sources as our comparison analyses below, depending on the available data and the appropriateness of the research question. In most cases, we examine the change in the focus student relative to his/her performance in the prior school year. There are several limitations to this approach. Student s prior performance may not be an adequate indicator of future performance. If students are on a downward trajectory in reality but become stabilized due to the FSP, we may mistakenly conclude that the FSP had no effect. Or, if students are identified for the program when in a lull but in reality are on an upswing that would have occurred without the FSP, we may mistakenly conclude that the FSP was successful when it actually had no effect. Additionally, we examine the relationship between services that students received from the FSP and changes in academic functioning. This is challenging because certain services may be delivered due to the needs of the youth over the school year. There may be reverse causality in that it is the functioning of the student that determines the services and not the impact of the services on the functioning of the student. For instance, a student who is struggling in a variety of areas may receive a large number and wide variety of services from the FSP as a reaction to their challenges. We may inappropriately conclude that receiving a wide variety and wide number of services is related to poor functioning. In spite of these limitations, we have taken the most appropriate methods and approach available to us. These limitations do not invalidate the conclusions of this evaluation, but they should add caution to any interpretation. 26

27 Student Outcomes: Individual Level Analysis and Findings FSP enrollment and services provided From school year to school year, there is a high proportion of repeat focus students. Of the 1,083 focus students in the school year, 335 (31%) were also focus students in the school year, and 180 (17%) were focus students in the school year. One hundred and twenty-eight (12%) were focus students all three years. The table below provides statistics on the amount of services the focus students and non-focus students received, as indicated by the family support worker activity logs from the school year, and is categorized into eight groups (Appendix D, provided by the FSP, depicts the various activities that make up the categories, and is important to examine in order to understand the differences between similar categories.) One important caution is that our conversations with Family Support Workers indicated that the services they provided are underreported. This is a common problem with data of this nature. However, we believe this data may still be useful for providing a picture of the array and extent of services provided by the program. These data are not necessarily reflective of the amount of time spent in each category, because certain categories (such as a medical visit) may take much more of a FSW s time than contacting a parent on the phone. Additionally, single events may be coded as multiple activities if they fall into more than one category. FSWs recorded a total of 27,268 activities. The most common service for focus student families was parent contact, which made up 42% of the activities for focus families. This was followed by direct service (which includes advocacy, intervention, tutoring, observing students, and other activities) at 25%, family involvement at 14% and meeting basic needs (food, clothing, etc.) at 10%. This differed from non-focus students, who received relatively little parent contact, and instead received relatively high amounts of basic needs (47%), family involvement services (31%), direct service (10%), and crisis assistance (6%). Interestingly, most crisis service activities were done with non-focus families (72% of the crisis activities were directed to nonfocus families). Combining activities for focus students and non-focus students, the majority of services were in the categories of parent contact (32%), direct service (22%), family involvement (22%) and basic needs (15%). 27

28 Activity categories (see Appendix D for specific activities in each category) Focus students n=1,083 Non-focus students n=1,545 All students (total) n=2,628 Parent contact 8,552 42% 116 2% 8,668 32% Direct service/ miscellaneous advocacy or intervention 5,193 25% % 5,883 22% Family Involvement 2,869 14% 3,204 47% 6,073 22% Basic needs 2,121 10% 2,103 31% 4,224 15% Attended SIT/IEP/MDT meetings* 709 3% % 712 3% Attended Parent % 4 0.1% 259 1% Teacher conferences* Social service 344 2% 197 3% 541 2% Physical health 194 1% 128 2% 322 1% Crisis assistance 169 1% 417 6% 586 2% Sum total of activities 20, % 6, % 27, % *Note: Our conversations with FSWs and our examination of patterns in the data lead us to believe that SIT team involvement and Parent-Teacher conferences were dramatically underreported. Data in these categories may be especially unreliable and misleading. SIT=School Intervention Team; IEP=Individualized Education Plan; MDT=Multi-Disciplinary Team We examined data on the percentage of focus families who received services and the amount of services they received (see the table below). As described in our previous report, the activity most widely and commonly received during the school year was home visits or other parent contact, with 95% of focus families receiving at least one parent contact, an average of 8.8 parent contacts throughout the year, and the number of contacts ranging from 1 to 68 over the course of the year. Nearly all focus families also received direct service or other advocacy (91%), with an average of 5.6 services. Most received family involvement (64%) and basic needs (63%) services at least once. There were smaller proportions of students who received other services. No other activity was recorded more than an average of once per student. 28

29 Activity categories (see Appendix D for specific activities in each category) # / % of focus students receiving at least once n=1,083 Average times received (of those who received any) Range Home visits and parent contact 973 / 95% Direct service/ miscellaneous 931 / 91% advocacy Family involvement 688 / 64% Basic needs 638 / 62% Family SIT/IEP/MDT teams* 264 / 25% Family Parent-Teacher conferences* 197 / 20% Social service 152 / 15% Physical health 129 / 12% Crisis assistance 108 / 11% *Note: Our conversations with FSWs and our examination of patterns in the data lead us to believe that SIT team involvement and Parent-Teacher conferences were dramatically underreported. Data in these categories may be unreliable and misleading. The number of activities received was not evenly distributed across the focus family population. The bar chart below displays the distribution of the amount of activities (a sum of the number of activities received in the school year) across focus families. The 10% of focus families who were the focus of the most activities accounted for 25% of the total number of services recorded by FSWs. 29

30 Attendance We ran two statistical tests including only students who had data from both and in order to examine whether there were changes in the average percentage of days of school attended from year to year. Our sample includes 818 students who had complete data from both years, out of 1,083 focus students from the year , and 26,015 non-focus students who had data from both years. Though it is inappropriate to directly compare focus students to nonfocus students (because of the heightened needs of focus students), using the non-focus students as a benchmark does allow us to examine whether there are any historical or maturational changes that need to be considered in interpretation. As the table below indicates, we found no differences improvement or deterioration for FSP students or non-fsp students. Both groups remained at exactly their level of attendance from the prior year. # in sample Mean % days attended Mean % days attended FSP students % 93.4% All other SPS elementary school students 26, % 95.1% We also ran several statistical tests to examine whether there were relationships between whether focus students received any particular types of services or had particular goals, as indicated by the family support worker activity reports, and whether there were attendance improvements over time. The table below indicates that there were no statistically significant relationships between having certain goals or receiving certain activities and improvement. The goal of attendance is removed from this table because virtually all focus students had increased attendance as a goal. t p 30

31 Any FSP activity in year # students Mean % days attended Mean % days attended Interaction p Basic Needs No Yes Direct Service No Yes Social Service No Yes Physical Health No Yes Crisis No Yes Home visits/parent No contact Yes SIT team No Yes Parent-teacher No conference Yes Any FSP Goal in year Class preparation No Yes Homework No Yes Mobility No Yes Suspensions No Yes We also ran statistical tests to examine whether there were correlations between the amount of service that focus students received (defined by the number of activities endorsed in FSP activity reports) and their change in attendance (defined as their percentage of attendance in subtracted from their percentage of attendance in We found a small but statistically significant relationship such that students who received more services were more likely to have their attendance improve (Pearson s r =.072, p =.04). This is a very small effect the number of activities accounted for approximately 0.5% of the change in attendance rates, and may be related to the fact that students who attended more days were more available to receive additional FSP services, rather than services being related to increased attendance. This is depicted in the graph below. The dots represent individual students and the red line represents the expected change in attendance percentage as the number of activities received over the year increases. 31

32 Standardized tests Math There were 356 focus students from who had data for the WASL (Washington Assessment of Student Learning)-Math and Reading tests for both school years from We analyzed whether these students experienced any change in the proportion who met standards in each year, and in the proportion who improved (did not meet standards in , met standards in ), deteriorated (met standards in , did not meet standards in ), remained positively stable (met standards both years) or remained negatively stable (did not meet standards either year). Of these students, 22% met standards for math in , and 23% met standards from , so there was no significant change in the overall proportion of students meeting math standards. However, some students improved while others declined. Results depicted in the table below indicated that, out of the 356 focus students, 8.4% improved from not meeting standards in to meeting standards in , 14.6% remained positively stable by meeting standards both years, 7.3% deteriorated by meeting standards in the prior year but not meeting standards in , and 69.7% remained negatively stable they did not meet standards in either year. These improvements and deteriorations are not statistically significant in other words, the changes from year to year are no greater or less than that expected by chance (McNemar Chi-square test p =.69). Therefore, 32

33 there was no evidence of improvement or deterioration in meeting math standards from the year prior to being in the FSP to the year the student was in the FSP. WASL Math Score Changes n= did not meet met standards standards did not meet standards 69.7% 8.4% met standards 7.3% 14.6% We also examined MAP (Measures of Academic Progress) scores for 135 focus students who had valid scores in This test may be more useful for students in the FSP because it takes into account students performance levels at the beginning of the school year. This reveals whether the student is demonstrating sufficient progress, instead of comparing the student to a benchmark defined by grade. Since FSP focus students are, by definition, performing poorly in school, the MAP may be a better indicator of the impact of the FSP. The results were that 67.4% of the focus students met standards for progress in math. This compares to 75.8% of all other students. We also explored whether there were any relationships between meeting standards and the goals and activities of the family support worker. We compared whether the percentage of students who improved, deteriorated, remained positively stable, or remained negatively stable were different if they did or did not have any of the following: a FSP goal of class preparation, homework, mobility, or suspensions, or a FSP activity of basic needs, direct service, social service, physical health, crisis intervention, home visits, SIT team, or parent-teacher conferences. Using the 134 focus students for whom we had valid data, we ran 12 statistical tests (not depicted here for space reasons) and found no statistically significant relationships between any of these goals or activities and meeting standards for the WASL math test. We also ran 12 additional tests exploring whether there was a relationship between goals and activities, and academic growth as measured by the MAP. These are not depicted here for space reasons. We found no statistically significant relationships between any of these goals or activities and meeting standards for growth on the MAP math test. We explored whether there was a relationship between academic growth in math ability during as measured by the MAP and the number of activities received by the student over the course of the school year. Using a logistic regression, we found no statistically significant relationship between amount of services received and growth on the MAP math test (chi-square=2.37, p =.124) Reading Of the 356 focus students with WASL data from both years, 30.6% met standards in and 30.3% met standards in % improved from not meeting standards to meeting standards, 19.9% remained positively stable by meeting standards both years, 10.7% deteriorated from meeting standards to not meeting standards, and 59.0% remained negatively stable by not meeting standards either year. These minor changes are not statistically significant (McNemar Chi-square test p =.99). In other words, there was no evidence of improvement or deterioration 33

34 in meeting reading standards from the year prior to being in the FSP to the year the student was in the FSP. WASL Reading Score Changes n= did not meet met standards standards did not meet standards 59.0% 10.4% met standards 10.7% 19.9% We also examined MAP scores for 135 students who had valid scores in The results were that 64.9% of the focus students met standards for progress in reading. This compares to 72.7% of the non-focus students. Although it is difficult to interpret comparisons to students in a different family support program in another state, it is worth noting that the percent of SPS FSP students who met reading academic progress standards is 14.9 percentage points higher than the 50% of the students in the Minnesota 360 Communities study who met their targeted growth (Kundin, 2011). As above, we also explored whether there were any relationships between the goals and activities of the family support workers and meeting standards for reading, or growth on the reading MAP test. We examined the percentages of students who improved, deteriorated, remained positively stable, and remained negatively stable, and explored whether these percentages varied by whether they had any of the goals or activities of the FSP. These included a FSP goal of class preparation, homework, mobility, or suspensions, or a FSP activity of basic needs, direct service, social service, physical health, crisis intervention, home visits, SIT team, or parent-teacher conferences. Using the 134 focus students for whom we had valid data, we ran 24 statistical tests (not depicted here for space reasons) and, as with the math analyses above, found no statistically significant relationships between any of these goals or activities and improvement, deterioration, or stability, or for growth on the MAP reading test. We explored whether there was a relationship between academic growth in reading ability during as measured by the MAP, and the number of activities received by the student over the course of the school year. Using a logistic regression, we found no statistically significant relationship between amount of services received and growth on the MAP reading test (chi-square =.001, p =.975) Disciplinary Actions As described in our interim report, there were very few long-term suspensions (less than 1%) and no expulsions in the group of focus students. Hence, our analyses focus on changes in short-term suspensions. Because the number of short term suspensions may increase over time in the general population due to maturation effects (i.e. youth who are a year older may be more likely to be suspended), for these analyses we compare changes over time between the FSP focus students and all other students. This is different than our approach to the standardized test score analysis above because standardized testing is weighted by grade. We have also found that more students tend to be suspended in the Spring semester than the Fall semester. Therefore, our analyses compare the changing percentage of students with a suspension in Fall 2009 to Fall 2010, and Spring 2010 to Spring

35 The graph below depicts the percentage of students with at least one suspension. The percentage of focus and other students with suspensions in 2011 rises dramatically for both groups, but the increase is much greater for focus students than all other students of this age in the district. 12% 10% 8% 6% 4% 2% 0% Focus students All other students Fall 2009 Spring 2010 Fall 2010 Spring 2011 Percentage of students with at least one suspension during semester However, the chart above only provides information on the overall percentage of students with a short term suspension. Another set of analyses examined individual trajectories, which vary. As above, we categorized students into four groups: Improved: suspended in one semester of , not suspended in the respective semester of Deteriorated: not suspended in one semester of , suspended in respective semester of Positive stability: not suspended in both Fall-Fall or both Spring-Spring semesters, to Negative stability: suspended in both Fall-Fall or both Spring-Spring semesters, to Results displayed in the table below indicated that both groups (focus students and all others) at both pairs of timepoints were more likely to deteriorate than to improve, as expected due to the maturation effect. This was more pronounced for both groups in the Spring pair than the Fall pair, again as expected because students are more likely to be suspended in the Spring semester. In the Fall pair, 4.4% of the focus students improved, 6.2% deteriorated, 88.6% remained positively stable, and.8% remained negatively stable. In the Spring pair, 3.8% of the focus students improved, 9.1% deteriorated, 85% remained positively stable, and 2.2% remained negatively stable. The Change Ratio illustrates the ratio of the change in focus students as compared to all other students (mathematically: Focus % / other student %). This change ratio shows that, for the Fall Semester pair, focus students were 2.9 times more likely than other students to have improvement, but they were also 3.2 times more likely to have deterioration. In the Spring Semester pair, focus students were 2.0 times more likely to show improvement, but 3.0 times more likely to deteriorate. 35

36 Focus students n=825 Fall 2009 & Fall 2010 Spring 2010 & Spring 2011 All other Change Focus All other students Ratio students students n=26,034 n=825 n=26,036 Change Ratio Improved 4.4% 1.5% % 1.9% 2.0 Deteriorated 6.2% 1.9% % 3.0% 3.0 Positive 88.6% 96.1% % 94.2% 0.9 stability Negative 0.8% 0.6% % 0.9% 2.4 stability Note: all Chi-square test significant at p <.05 We also examined whether there were any relationships between FSP goals and activities during the year and changes in suspensions. We ran 24 statistical tests examining these relationships (12 goals or activities for the Fall pair and 12 goals or activities for the Spring pair), only including the 825 students who had data at all timepoints. Findings were inconsistent and scattered across goals and activities, with little clear meaning. Due to this we do not present detailed findings here, but these are available upon request. Finally, we examined whether there was a relationship between the amount of FSP activity received over the school year and whether or not a student was suspended in Spring semester We ran a binary logistic regression and found no statistically significant relationship between amount of service and suspension (chi-square = 2.56, p =.110). Mobility The FSP is intended to decrease the school mobility (transfers, dropouts, disappearances, etc.) of the families it serves. The table below depicts the percentage of focus group students over two years for those on whom we had complete data from both years. This is compared to all other students in SPS. The data indicates that approximately the same percentage of focus students had at least one move as non-focus students in the school year (34% vs. 33%). Both groups had fewer mobility events occur in the prior year, , but this was due to sample selection because we only included people in the analysis who had data from both years. If someone left the school district or was unable to locate during the school year, by definition they would not have data from both school years. Focus students in school year n=818 All other SPS students from school year n=26, moves 86% 67% 82% 68% 1 move 10% 31% 16% 30% 2 moves 3% 2% 2% 2% 3 moves 0.9% 0.6% 0.3% 0.5% 4+ moves 0.1% % 0.1% 36

37 The table below compares the types of moves for focus students when compared to all other students in the school year. There were no significant differences in the reasons for being mobile. These results indicate that the FSP focus students did not have mobility outcomes which differed from students who were not in the FSP. Mobility Reasons in Focus students n=818 All other students n=26,015 No mobility 67% 68% Confirmed transfer to SPS school 27% 26% Confirmed transfer to school outside Seattle 4% 3% Year-end withdrawal without re-enrollment 1% 1% Unknown, unable to locate 0.5% 0.2% Confirmed transfer to private school or home 0.4% 0.9% instruction Absent 20 days or more 0.5% 0.1% No show new student who did not show up 0.1% 0.3% Suspension for major infraction % Finally, we examined mobility for all students who were in school in the school year (not restricting to those for whom we had data in both years, as we did above). We found that focus students and all other students had very high rates of mobility, mostly due to transferring to another SPS school, with 40.3% of focus students and 41.7% of all other students leaving the school. The difference between focus students and all other students was not statistically significant. We were also interested in examining the relationship between specific goals and activities and mobility. These analyses were conducted for all focus students from , not just those with data from the prior year. The table below depicts the proportion of students with at least one mobility episode during the school year, stratified by whether they had received at least one service in the indicated activity category. We found several statistically significant results. In all cases, focus students who received the service were less likely to be mobile. This included students who had received basic needs, direct service, physical health, family involvement, parent contact, parent attending a SIT team, and parent attending a parent-teacher conference. These findings should be tempered by one serious limitation inherent to studying mobility: if a student left the school for a school that did not have the Family Support Program during the school year, they would not have been able to receive the service. Hence, it may be that, instead of the FSP activities impacting the mobility of students, it is the mobility of students that impacts whether or not they received an activity. A more complex and thorough approach to analyzing this question is required, and we will conduct this for our final report. Finally, for this same reason, we did not examine the relationship between number of activities and mobility. 37

38 Any FSP activity in # students % mobile p year Basic Needs No % <.001 Yes % Direct Service No % <.001 Yes % Social Service No %.790 Yes % Physical Health No %.005 Yes % Crisis No %.094 Yes 93 48% Family involvement No % <.001 Yes % Home visits/parent No % <.001 contact Yes % SIT team No % <.001 Yes % Parent-teacher No % <.001 conference Yes % Summary of Individual-Level Academic Analyses Our initial analysis of the mechanisms of change has several limitations, as described in the introduction. Any consideration of our conclusions should be tempered by these limitations. However, given these constraints, we found the following: The Family Support Workers logged a large number of activities across a broad array of activity types for both focus students and non-focus students. o For focus students, activities were centered on parent contact, direct services or intervention, family involvement, and providing basic skills. Focus students received a lot of parent contact, with an average of nearly 9 contacts between the family support worker and parents over the course of the school year. o Activities are somewhat concentrated on a few students, 10% of focus students who used the most services receiving 25% of the recorded activities. o For non-focus students, activities were centered on family involvement and providing basic needs. There was some evidence of a weak but statistically significant relationship between receipt of family support program services and improved attendance from the prior school year. There did not appear to be any specific link between individual services or activities and improvement in attendance, though we are unable to stratify this analysis by individual need. We found that focus students experienced no overall improvement in meeting math or reading standards (as measured by the WASL) from one year to the next. o However, we found that 67% of focus students met the standard for progress in math (as measured by the MAP), as compared to 76% of the regular student body. While the percentage of focus students making adequate progress was lower than 38

39 the regular student body, given the unique and serious challenges experienced by focus students, this might be considered a positive outcome. o We had similar findings for reading, with 65% of focus students making adequate progress as compared to 73% of non-focus students. This compares favorably with the only other study available, Minnesota s 360 Communities Partners for Success, which found that only 50% of students who were served met their targeted growth in a standardized reading MAP. Overall, focus students had many more suspensions during their focus student year, with a sharp increase during the last semester of the year. o Focus students were both more likely than all other students to improve and more likely to deteriorate in terms of decreased or increased suspensions. Focus students had nearly equivalent rates of mobility as all other students, and the reasons for mobility were nearly identical. This is impressive, given the serious circumstances surrounding their lives, especially the large number of focus students who are homeless and transient. 39

40 Student Outcomes: School-Level Analyses While the analyses above examine individual youth, we often heard that the activities of the FSP were likely to impact entire schools. Therefore, we conducted analyses using readily-available school-level detail. These analyses were intended to ascertain whether the FSP was related to 1) parent ratings of family engagement, 2) school-wide percentages of youth meeting standards on the Measurement of Student Progress, 3) school-wide percentages of suspensions, 4) schoolwide attendance, 5) school-wide mobility rates, and 6) the availability of external resources and programs within the school. While all schools have access to FSWs through the Referral Service Model, Tier 2 and Tier 3 schools have much more regular Family Support Worker involvement. Therefore, we were interested in examining the difference between these FSP schools (Tier 2 and Tier 3) and non-fsp schools (Tier 1). However, FSP schools are not randomly assigned and tend to have contextual characteristics (covariates) that are associated with poorer family engagement and academic outcomes (hence their need for FSP-type services). We created subscale scores for several school climate measures. Family Engagement indicated the average percentage of positive responses to six questions about parents perceptions of opportunities for involvement, engagement, and support for families (e.g. I feel welcome whenever I visit the school, and The school staff is knowledgeable and respectful of different cultures, ). Family Parent Satisfaction indicated the average percentage of positive responses to six questions about the parents levels of satisfaction across a variety of school characteristics such as quality of instruction and leadership (e.g. Adults at school care a lot about my child s academic success and personal well-being. ) Professional Culture indicated the average percentage of positive responses to nine questions about school staff s perceptions of the interactions among staff and their work on instructional-related topics (e.g. Continuous professional learning is highly valued by staff at this school. ) School Leadership indicated the average percentage of positive responses to seven questions about staff s perceptions of principal behaviors and effectiveness (e.g. The principal is an effective manager of school operations. ) The table below provides correlations among our variables of interest, including demographic and climate data, for the elementary schools with complete data. 40

41 Mean % meeting MSP 1 standards Pearson r bivariate correlations Suspensions % Family Engagement mean % positive Attendance % Mobility % Outcomes % mean meeting MSP -- Family Engagement.292* -- % Suspensions -.617** -.368** -- % Attendance.735** ** -- % Mobility -.760** ** -.713** -- School Climate Survey Family Satisfaction ** Professional Culture ** -.359** School Leadership * School Demographics # Enrollment ** -.379** % American Indian or Alaska -.586** ** -.563**.604** Native % Asian -.373** % Pacific Islander -.571** ** ** % Asian/Pacific Islander -.399** % African American -.808** -.389**.675** -.504**.508** % Hispanic -.540** -.270* ** % White.854**.379** -.550**.412** -.514** % Male % Transitional Bilingual -.761** * -.275*.526** % Special Education -.399** **.352** % Free/Reduced Lunch -.926** -.384**.579** -.526**.630** % Section ** ** Average Years Teacher Experience % Teachers with at least a Master s Degree * p <.05 ** p <.01 1 Measures of Student Progress Family engagement. We explored whether the FSP was related to family engagement after controlling for other school climate and context variables. Data was obtained from Seattle Public Schools on parent and school staff interviews about their school s climate (an example of these questions can be found in the school climate reports downloadable through These were integrated with data on school-wide demographics in order to assess the relationship among the FSP, school climate, and family engagement. We ran an Analysis of Covariance predicting the relationship between having a FSW (i.e. being a Tier 2 or Tier 3 school) and Family Engagement while controlling for the percentage of African American students, percentage of students qualified for free or reduced lunch, Professional Culture scores, and School Leadership scores. Results indicated that FSP status was not significantly related to Family Engagement after controlling for these variables. Predicted Family Engagement scores (after controlling for these variables) for Tier 2 and 3 schools, or those with a FSP, was 87%, compared to 89% for Tier 1 schools, which is not a statistically 41

42 significant difference. Therefore, the FSP did not appear to be related to Family Engagement scores after controlling for important covariates. Academic Achievement. Unlike Family Engagement, which required multiple covariates to control for school selection factors, only one covariate was necessary to control for school differences in Academic Achievement (as measured by the mean percentage of students meeting state standards on the Measurement of Student Progress, including Grade 3, 4, and 5 scores on math, reading, writing, and science). The correlation between Academic Achievement and the percent of youth eligible for free and reduced lunch (a proxy indicator of poverty) was a massive r = -.926, which is one of the highest correlations our research team has ever found. Higher percentages of youth in poverty was related to decreased percentage of youth meeting academic standards. This relationship is depicted in the scatterplot below. Interestingly, no significant correlations were found between Academic Achievement and scores on School Leadership or Professional Culture, the percentage of teachers with at least a Masters Degree, or the average years of teachers experience (see table above). Other variables predictive of lower percentages of Academic Achievement included the percentage of any minority group (which was also highly related to poverty). The lines on the chart below indicate regression lines, independent for each group (FSP schools and non-fsp schools). The similarity and proximity of these lines indicates that the FSP was not related to school-wide MSP scores after controlling for poverty. An ANCOVA confirmed this. (Interestingly, when stratifying School Leadership and Professional Culture into high and low groups and controlling for school poverty rates, these variables were also unrelated to Academic Achievement.) 42

43 Disciplinary Actions. We conducted similar analyses to examine if the FSP was related to Disciplinary Actions after controlling for poverty. We define Disciplinary Actions as the % of youth suspended. As depicted in the chart below, we found that the FSP had no relationship to 43

44 the percentage of school-wide Disciplinary Actions after controlling for poverty. This was confirmed with an ANCOVA. Attendance. We conducted similar analyses, and confirmed with an ANCOVA, that there were no significant relationships between the FSP and Attendance, as depicted in the figure below. Though these lines begin to diverge, the ANCOVA confirmed that the apparent difference does not approach statistical significance. 44

45 Mobility. Similar analyses conducted on Mobility found a main effect (approach statistical significance) for FSP such that, when controlling for poverty, schools that had a FSW had slightly lower mobility than would be expected. This is in agreement with our individual-level findings, above, which indicated that youth in the FSP were no more or less mobile than youth not in the FSP, a surprising finding given increased needs and risk. 45

46 External Resources. Finally, we examined whether there was a relationship between the FSP and the availability of external resources within the school. This analysis was done as a result of several comments from Advisory Group members (particularly representatives from outside agencies) indicating a belief that the FSP was effective at securing additional resources for schools. Data was generously provided by Seattle s Alliance for Education. This data listed the name of the organization, the type of service they provided, the schools they provided it to, and the location (in school, out of school, or both) for every school in the SPS. We summed the total for each school, and then standardized this among schools by calculating the number of resources per 1,000 youth. After controlling for school poverty, we found some indication (approaching statistical significance) that the FSP was related for increased availability of resources in the school. 46

47 Summary The school-level analyses provide additional support to our findings from the student-level analyses. After controlling for an indicator of poverty, the FSP does not appear to be related to school-wide standardized tests (Measurements of Student Progress), attendance, or disciplinary actions. Additionally, the FSP does not appear to be related to parent-rated school-wide Family Engagement ratings. However, there is some indication that the FSP is related to decreased mobility rates and increased school availability of external service agencies. 47

48 Focus Groups Focus groups were conducted during February and March 2012 to determine if there was a shared vision among stakeholders for the rationale, purpose, activities, and measurable goals of the FSP. Seven focus groups were held with school staff, FSWs, parents, and officials from the Office for Education. Twenty-two teachers, 16 principals, 12 counselors, and 15 other school staff were nominated by FSWs from 26 elementary schools in the Seattle Public School District, and invited to participate via and phone by researchers at the University of Washington. It is important to note that, as school staff were recommended by the family support workers and not randomly selected, their comments cannot be considered necessarily representative of the opinions of all of the staff in the district. Of the 43 school staff that were nominated and invited, two principals, six teachers, three counselors, and one student support specialist participated in three focus groups. Thirty-five FSWs from 26 elementary schools were invited to participate. Of the 35 FSWs invited, 10 participated in two focus groups. Fifty parents were randomly selected to participate in a focus group and received an invitation letter in the mail. Response to these letters was minimal, and FSWs were asked to remind invited parents to respond. Of the 50 parents invited, five parents participated in the focus group. For these reasons, parent viewpoints also cannot be considered necessarily representative of the program as a whole. One final focus group was held with three participants who are staff in the Office for Education. These participants are responsible for various contractual and financial oversight of the Family Support Program. School staff, FSWs, and parents represented the following 18 schools: Arbor Heights, Beacon Hill, Concord, Dearborn Park, Dunlap, Emerson, Gatewood, Hawthorne, Highland Park, John Muir, Martin Luther King Jr., Thurgood Marshall, Van Asselt, West Seattle, Sanislo, Madrona K-8, and Wing Luke. In all seven focus groups, participants were asked questions about their experiences with and knowledge of the Family Support Program, such as, What is your understanding of the Family Support Program? and What kinds of outcomes do you see as a result of the Family Support Program? Each of these focus groups lasted two hours. School staff, FSW, and parent focus group participants received a light meal. A $50 Visa gift card was raffled during each of the five focus groups with school staff and FSW participants. The five parent participants each received a $25 Visa gift card. Alignment between the logic model and focus group participant responses Purpose and Role of the Family Support Program Participants were asked to describe the purpose and role of the FSP, and their description of their understanding largely matched the logic model developed by program staff at the outset of the evaluation project. There was a great deal of consistency among participants (teachers, counselors, principals, parents, FSWs, and Office for Education staff) in their understanding of the role of the FSWs, their purpose, their activities, and the outcomes they were meant to promote. Nearly all participants understood the program as offering non-academic support to students and families, such as basic needs assistance or connection to community resources. 48

49 Participants also mentioned educating families about the educational system, cultural competency with minority youth in the community, and the provision of a unique perspective to the educational team as important aspects of the FSP role, all of which are also included in the logic model. Several participants framed the program as one of crisis management, assisting families in emergencies, or as acting as last resorts in the educational system, serving the children and situations that would otherwise fall through the cracks. One teacher phrased this idea as, [FSWs] do anything to help the youth succeed that someone else at the school doesn t do. The idea that FSWs act as liaisons was among the most often-cited themes. FSWs were understood to be liaisons between the school and families, the school and the community, and the school and the district. Relationships were a similar and additionally popular theme; many participants felt that the FSP builds important relationships between parents, schools, and communities that otherwise go untended. Parents echoed this idea, by stressing the strength of their relationships and trust in the Family Support Workers. Many participants, particularly parents and FSWs, directly expressed that this trust is enhanced by the greater racial and cultural similarity between FSWs and target families than for other school staff (administrators, teachers) and FSP target families. Many stressed that the kind of support offered by FSWs should, conceptually, lead to increased academic outcomes. There was, however, some disagreement over the degree to which the program should be responsible for academic change, whether such change can be measured, and whether an academic focus leads to unintended negative consequences. The issue of academic focus will be discussed at greater length below. Flexibility and Variability That the FSWs provide flexible, highly-variable services from school to school or even student to student was often mentioned by participants. Most of the FSWs, parents, and other staff felt that this was a strength of the program. Teachers reported having never seen a FSW turn down a child in need, and praised this aspect of the program. However, during our focus group with the Office for Education staff, participants reported believing that the high variability was sometimes a distraction to the program. These participants felt that the program would be put to best use if it was more highly-focused on case management. One official would like to see the FSWs operate as part of a system wherein they could intervene at appropriate levels with appropriate levels of expertise, and identify kids at the top of the pyramid [ ] with intense needs, [for whom] the FSW might need to seek support outside of the school. This discrepancy, the reasons for it, and the challenges associated with it, are described in more detail in our sections below. Activities of the Program Participants were asked to describe the particular activities of the FSWs. These, again, largely matched the activities listed in the logic model: meeting basic needs (such as food, financial aid, housing, clothing, transportation, school supplies, or hygienic supplies), providing home visits to families, referring families to community resources, providing mentoring/emotional support for 49

50 children and families, attending SIT team meetings, educating parents, and generally going above and beyond the official requirements of their position. There were, however, some instances where a program component was listed in the logic model, but was not mentioned in focus groups. Two of these are worth mentioning. Neither teachers, principals, counselors, parents, city officials nor the FSWs themselves brought up administering surveys [to parents] about their involvement in the child s education, nor looking at schoollevel indicators to determine the best use of FSW time, though these were listed in the logic model. In some ways this is unsurprising, as the use of data within the program was somewhat problematic. We will discuss data at greater length below. We did see some indication of using school-level indicators to determine the best use of the FSWs time during our shadowing of FSWs. Types of Students and Families Served Participants described the program as reaching a wide range of students. The descriptions generally matched the logic model, or lined up with the services provided by the program: children who are poor, homeless, ELL, or marginalized, or have attendance issues, academic issues, behavior issues, or low self-esteem, or are in crisis. Participants also mentioned refugee populations, large families, and families with bad educational experiences as groups who are best served by the services of the program. One principal mentioned that the FSW at his school sought families who were not only in need, but who were also likely to make good use of the services. This worker spoke to the principal about looking for students who will take the support and then make something happen with it. [ ] Not every family is that kind of family all of the time. Perceived Outcomes Participants were asked to describe any outcomes they had perceived as a result of the FSP on students, parents, schools and communities. Several participants were quick to note the difficulty in isolating the work of the FSP; many of the students who the program reaches receive other services as well. Nevertheless, many participants provided anecdotal examples of perceived outcomes as a result of the FSP. Some participants felt that they had seen improved behavior, and one principal specified that he experienced fewer behavioral referrals as a result of the program. Participants mentioned increased attendance, improved academics, increased self-esteem, improved hygiene, and increased classroom participation as outcomes of the program on students. Some teachers felt that the program resulted in increased parental trust in the educational system or the particular school at which the parent s child is enrolled. Trust, generally, was a popular theme, and many parents, teachers and FSWs considered it an essential aspect of the program. However, several parents mentioned negative experiences with the educational system, either as a parent or as a student themselves. These parents did not necessarily express an increased trust in Seattle Public Schools (one parent explicitly praised the FSPs autonomy from the district), but instead felt that the program had enabled them to navigate that system in spite of some distrust. This is described in more detail in sections below. 50

51 The logic model specified a number of long term outcomes expected by the program. These included increased rates of graduation, increased college enrollment, decreased incarceration rates, and improved life skills. Long-term outcomes were mentioned only briefly during some of the focus groups. For the school staff and parents, this isn t entirely surprising. They don t have an opportunity to witness long-term outcomes. One FSW expressed a desire for the program to measure and focus more on the outcomes of students beyond their stay in the program, suggesting that the program does not currently do so. Some participants expressed the belief that the impact of the program for families with the most problems might not be felt until years later, because the current context of the student s life would overshadow any immediate causal relationship on outcomes. What Would You Do Without the FSP? In trying to understand the role of the FSP in the schools, researchers sometimes asked participants to imagine what would change if the program did not exist (participants frequently answered this question before researchers had a chance to ask it). Many school staff members indicated that they simply couldn t imagine or didn t know what they would do. Several participants indicated that their workload would increase or change. One counselor suggested she would need to move into her office to keep up with the additional work. Principals imagined they would have to spend more time fielding calls from parents, and may have to deal with more irate parents. Teachers felt as though they would have to pick up some of the responsibilities otherwise undertaken by the FSWs. One teacher said she would be less able to focus on academics because her time would be spent managing other aspects of the students lives. Parents also felt children would suffer academically, socially, and behaviorally without the support of FSWs. One parent credited her FSW with helping to mediate difficult situations between her family and school, Without the FSW there, I would have been in jail, honestly. 51

52 Focus Groups: Important Emergent Findings and Themes In this section, we summarize main themes and detailed content that emerged across the range of focus group questions and participants. Focus groups were not randomly selected and are not necessarily representative of those associated with the program. 1. Nearly everyone agreed that the current structure for data-based decision-making is burdensome, underutilized, and not particularly useful for planning, implementing, or evaluating services. Many FSWs expressed a desire to be involved in restructuring the data collection and feedback system, and the system for case management planning (currently exemplified by SOAP notes.) In our experience, databases that are complete, accurate, and valid are rare. Good worker activity data usually comes from billing databases or databases that are deemed useful by program staff. Our focus groups and our quantitative analysis indicate that the Activity Report database is not believed to be useful, and data entry is compliance-driven. FSWs reported that the database does not feature categories that adequately describe their work, it is difficult to navigate and burdensome (one reported that it took her two full days of data entry for every month of work), and the information is not returned to them in a timely or useful manner. In addition to these issues, the data was frequently reported as unreliable or inconsistent from one FSW to another the understanding of the meaning of various activity categories differ among FSWs, and the number of activities or categories endorsed for one specific event is thought to vary among FSWs. The FSWs in our focus groups wanted the database to strike a balance between capturing important data while being easy to complete, especially because the wide variety and constantly changing nature of their work meant that it was difficult to capture all of their activities. They felt that the burdensome data entry requirement took time away from performing their duties. One frequently cited problem by multiple focus group participants is that the activity data is not integrated with other important sources of data, most importantly the SOAP notes, which are used to determine and track case management activities. FSWs described carrying large amounts of paper with them, particularly those FSWs serving more than one school, because SOAP notes were not available on a central network (in fact, one FSW had a rolling suitcase full of paperwork, which she described as necessary to bring with her in order to keep track of her focus families between two schools). Other important data is also missing, such as data compiled by the Seattle Public Schools on attendance, disciplinary actions, grades, standardized tests, and other important academic data. During our shadowing visit, we observed a FSW using a real-time list highlighting students with the most significant attendance problems; she used this sheet to track who to intervene with and what progress she had made. This was an excellent use of data to drive activities. We also heard about FSWs who had attempted to integrate data across multiple sources by hand. They wrote important information from other data sources on their SOAP notes forms. Lastly, there was some concern expressed that integrating FSP activity and SOAP notes data with the academic dataset would violate confidentiality. 52

53 FSWs reported that they did not receive summary information about their activities until the end of the school year, which was too late for them to modify their approaches. Because this data isn t directly linked to outcomes for individual students, FSWs felt that it was not useful for modifying their work or understanding how their work impacted families. This revealed a need for more timely and useful data-based feedback about activities and performance. Many FSWs consider the SOAP notes not helpful. Most of the FSWs in our focus groups felt that some sort of structure for managing their cases was helpful in order to plan goals and track progress towards those goals. However, they felt the current structure of the SOAP notes was not helpful and not well-integrated with the type of work they were doing. This is very similar to reports from the 360 Communities program in Minnesota, which also found SOAP notes to not be very helpful, and switched to a different form of case management note taking. 2. Most Family Support Workers, teachers, and counselors in our focus groups strongly disagree with the Office for Education using standardized test scores as a benchmark for success. Fair indicators of success that were agreed to by most participants included attendance, disciplinary actions, parent involvement in their child s education (broadly defined), and mobility. There was less agreement about homework completion because that is tied to the context of individual classes and teachers. A paradox in this theme is that nearly all of the FSWs, teachers, counselors, principals, and parents felt that their actions had a positive impact on students academic improvement, including standardized test scores. However, they felt that being held to this as a benchmark was unfair, because academic functioning is tied to the context of the child s life in many domains that cannot be impacted by the FSW. In this line of reasoning, some FSWs felt that measuring growth on standardized tests (rather than passing scores) would be fairer. We heard from several FSWs a variation on the theme that, Even the teachers aren t held to the kinds of standards we are being held to. As we describe below, there may be several unintended consequences of using standardized test scores as a principal benchmark. In particular, the families chosen as focus families may be different than those who would be chosen based primarily on need and circumstance. 3. There is a strong philosophical division among stakeholders in how the FSP should serve families in need, particularly the division between focus families and non-focus families. Staff at the Office for Education desire the program to have more targeted support of families with moderate- to high-needs, while those with the most extreme needs are referred to other supportive services. Staff in the FSP want to be able to provide some form of assistance to everyone who needs help. According to the participants in our focus groups, the historical context of the development of the FSP has contributed to a philosophical division. One point of controversy is the distinction between focus families/students and other students or families who may receive support from the FSP. Historically, the FSP provided support to all families who were referred or otherwise identified as needing support (and data from our focus groups indicate 53

54 that for the most part, they still do provide services to nearly everyone who is referred or found to be in need). This created a program that was considered by the Office for Education and others to be nebulous and difficult to evaluate in terms of its impact and effectiveness. The school district and the Office for Education desired more careful monitoring of the program s impact on academic functioning, as improving academic success is the major goal of the Families and Education Levy funding. Additionally, successful interventions and positive outcomes are based upon good management (case management or otherwise) of students and families goals, progress, and needs. Therefore, serving all students on an as needed basis 1) is difficult to monitor and evaluate, and 2) may be less effective than targeted, monitored approaches. On the other hand, creating a program structure that is easier to evaluate and more focused has its own challenges, particularly because it conflicts with the longstanding philosophy and practice of the FSP, and the difficult personal decision-making that would have to occur by FSWs in order to prioritize individual students. o The shift of the FSP from its prior structure to the current case management approach remains a point of contention for many FSWs, teachers, and other school staff philosophically and historically. As one FSW said, Part of our dilemma is the way we started out in most cases, school staff and families still want that socialservice type of FSW. It is difficult to go from direct services for any kids that need and want it in your school to all of a sudden saying you must have an academic focus, and a limited number of children you can work with. It s been like this for seven years, but you can t change people s minds in seven years. We still try to do the other part, because how do you not? Several FSWs said that the change in approach made them feel like they were no longer experts at their job. Several FSWs appreciated the structure and guidance that a case management approach provided, as they believed the program had become less crisis-oriented and more focused. However, the case management approach didn t always fit with their values and expertise for the following reasons: 1) FSWs felt they could not turn families away from services due to a full caseload, because of their personal and ethical commitment to children ( I am a human being, I can t say no to a kid who is hungry ) and because they felt it might negatively impact their relationships with referring teachers, principals, and other staff; 2) Students may stop needing intensive support through the school year, but continue to be considered focus families ; 3) Selecting the right caseload for an entire school year is a nearly impossible task, given the limited information FSWs have at the beginning of the school year; 4) The increased time and paperwork required for documenting services to focus families takes services away from families; and 5) Using focus families as the basis for benchmarking outcomes results in some FSWs selectively choosing students for their caseload who they expect to show academic improvement, rather than choosing those most in need of their particular services. This last point was surprising to us, but was confirmed across many different interviews, and illustrated by this quote, When I am looking for students [to put on my caseload], I am not looking for someone who has passed the MSP, but rather someone who is close to passing it. Because I have [specialty programs] in my school, I am not going to take a child in those programs. I might help them anyway, but I won t put them on my caseload because there are so many outside influences on these students that I know they aren t going to meet academic 54

55 criteria. On the other hand, the FSWs felt that having an official caseload resulted in teachers pushing kids on us because it makes it easier for them so they don t have to deal with a challenging kid all of that time we do take challenging kids, but we try to balance it out. The unintended consequence of relying on academic benchmarking as an indicator of program success may change the FSWs caseloads in unpredictable and possibly negative ways. o Our quantitative analysis confirmed that there were many more individual non-focus students who received services from the FSP than focus students. Our focus groups with all parties reinforced this. For instance, two Principals in our focus group had the following exchange: Principal #1: [The FSW] serves 2.5 to 3 times as many kids as are on her caseload Principal #2: There s a kind of triage that goes on. The caseload is just a list of names to provide evidence for the program they re the same types of kids, but the number on a caseload is small compared to the need. The principals felt that the FSWs treated case planning for focus families differently, in that they were more likely to talk about academics, more likely to plan for the families, and less likely to be reactive to crises. This seems borne out by our data on activities, which shows that non-focus families received 3 times more crisis services than focus families. o One related concern that has been expressed is that the FSW efforts are being misspent on a small minority of families who receive the most services. We examined the activity data and found that the 10% of the families who received the most services used 25% of the total services recorded by FSWs. 4. The level of coordination of FSWs with the goals and activities of the school appears highly variable. Our shadowing of FSWs and our focus groups revealed that, for many schools, FSWs are core members of the school team we saw or heard about FSWs who sit in on family meetings such as IEP meetings, are members of Family Engagement Action Teams, are core members of Continuous School Improvement Plans, who plan evening school-based events for families, and other similar indicators of integration with the goals and activities of the school. On the other hand, we heard anecdotes that some FSWs are not fully coordinating with their schools vision, activities, and goals. One approach to ensuring coordination that has been discussed is having FSWs as formal school staff who are under the supervision of the principal. We heard varying opinions weighing the benefits and consequences of this approach, and these are described below. Nearly all teachers, counselors, principals, and FSWs agreed that having FSWs available within the school, rather than as an outside agency with no visible school presence, was essential to conduct effective family support. However, opinions are mixed about the degree of autonomy FSWs should have. While the principals in our group expressed a belief that FSWs were well-integrated in the functioning of their school (one said, Our FSW is always involved in the SIT meetings. We consider her a part of our core office group ), the principals in our group still expressed some desire for increased supervision of the activities of the FSW. (There were only two principals in our focus groups.) They felt this would increase integration with the goals of the 55

56 school and provide more accountability to the FSW role. One principal wanted someone who was more plugged into the school. They also wanted to be kept better informed of the FSW schedule and to have their availability in the school be more predictable (we should mention that, according to the FSP, the FSWs are supposed to sign in and out whenever they arrive at or leave the school). However, when we asked more specific questions about their responsibilities, principals were less willing to take on the full responsibilities of managing FSWs. They did not embrace managing FSW budgets, providing professional development/training and other skill development, and providing guidance in family support. One principal mentioned that other principals wanted more control, but this was because they didn t like the particular FSW they were working with, I know that some of my colleagues have expressed not really caring for the program, and when you get down to it, they don t care for the person. It s the person, not the program. They don t feel like they re effective. I always say don t throw out the program because you have a bad FSW. I think those principals concerns would be addressed by some supervisory control. Another reason principals provided for wanting more supervisory control is that some principals (not those in our focus group) may not understand the role and priorities of the FSW and may want to use the FSW for activities such as playground duty, and sometimes people get a blinder about the program because of little things like that part of it is respect for the skill set, it s like No, lunch duty is NOT the best use of their time. Another possible reason for wanting increased supervision provided by principals was the belief that sometimes FSWs and principals may disagree about what actions the school should take, that they butt heads over too much advocacy for this or that and once you get there, they re like The program is no good. Principals said that the most important part was that FSWs were integrated into the school s work towards collective goals. They believed that parents saw FSWs as part of their community and that parents did not know or care about supervisory relationships; however, this was contradicted in our parent focus groups. We had not planned to ask parents about the supervisory structure of FSWs; however, they volunteered information unprompted by us. Parents agreed that a major strong point of the FSP was that it was not managed by Seattle Public Schools and that FSWs were not supervised by principals. They felt that FSWs often play a role of mediator or advocate for parents, and that this role would be compromised by a different supervisory structure. This seemed particularly true for parents of color. As one parent, who happened to be African American, said, There are a lot of negative things about the district as far as dealing with our children insensitivity, disproportionate funding, that type of thing. [The FSP] has autonomy, it can take a stand that might be contradictory to the school district, which the district might [otherwise] more or less hush up That s one of the strong points, my FSP might tell me that on the record or off the record, I don t think they re treating your son right. They would bring stuff to my attention because I have the right to know it, without fear of retribution by the school district they can provide neutral support. That s necessary. Two mothers each described similar stories about conflicts with school staff or principals that nearly resulted in physical aggression (one mother said she had been banned from the grounds of her child s school for a year). They both said that it took the neutral, mediating role of the FSW to resolve their conflicts. Parents were mixed on whether this trust of the FSW contributed to any increased trust of the school as a whole. 56

57 o The FSWs also provided many examples of situations where they provided a mediating role between staff and parents that may have been difficult to provide if they were school staff, and they strongly argued that FSWs need to remain independent of principals in order to most effectively do their job. As one remarked, The reason why the program was set up this way is that it is very difficult for a FSW to be an advocate for kids and families and a liaison between the school, when sometimes what is in the best interest of the child is sometimes not what the school or the principal wants. Ultimately it was set up to avoid this conflict of interest. They also felt that, if the FSP was part of the budget of the school, the FSWs would be one of the first programs to be cut in times of tight budgets. o Teachers and counselors, too, felt that it was important for the FSP to have autonomy from the school; however, like the principals, they felt that they wanted additional information on the whereabouts of FSWs, and some of them felt that some balance of responsibilities (especially lunch duty and recess duty) may be reasonable in order to be fair to all workers in the school. o While community organizations were not formally a part of our focus groups, during our presentation to the evaluation Advisory Committee the issue of supervisory structure came up. The community organization representatives who were present felt that if individual schools were responsible for the administration of the program, it would be much more difficult for them to work with schools individually rather than working with the single source of the FSP. For these organizations, the FSW acts as a sort of distributor of the services and resources they provide. 5. The unintended consequence of partitioning limited family support resources into tiered schools is that some principals, teachers, and counselors feel punished for success by having their FSW removed if student outcomes improve. This was also believed to be related to anxiety about the future, which led to decreased job performance. Many focus group participants reported frustration with the current design for allocating FSWs. A typical comment, this from a principal, is Rewarding success with the reduction of resources is something that happens throughout the district and it always makes us shake our heads. You re almost better off to be NOT successful. Similarly, there was some frustration with a lack of understanding about how schools are tiered though they said it had been explained to them, principals and school staff felt that the decision criteria were confusing and that decision making was sometimes based on inaccurate data. Many participants felt that anxiety about year-to-year changes in FSW staffing by schools was tied to decreased job performance, as one principal expressed, It seems like for the last several years there have been anxieties about who is going to get laid off, and that adds a layer of anxiety and makes it harder for people to function because then THEY are in the place of worrying about their basic needs. 6. The Family Support Workers asked for additional support in terms of consultation and professional development. 57

58 The FSWs wanted the program to continue to build upon and expand networks of support among the FSWs the MSW pilot project and the use of cluster representatives are steps in the right direction, but the FSWs expressed that they felt isolated in their schools, without people who could provide regular consultation and feedback. Similarly, they asked for additional supervision by people who could help keep them on track and provide case consultation for difficult cases, particularly in terms of navigating the school district. One mentioned a desire for interdisciplinary teamwork to talk out difficult cases. FSWs also asked for trainings that are more specific to their current work, Our trainings are more relevant to the way we used to operate as opposed to our current operation. Since we have this more academic focus, there should be more emphasis on how to achieve this we need trainings focused on that. The emotional, social, and behavioral components of the training have all been relevant. The academic part is something I ve had to learn through the school. 58

59 Comparing the Partners for Success Program with the Family Support Program The Partners for Success Program is located in Dakota and Scott counties in Minnesota and provides basic needs assistance to students and families in 10 elementary schools, with the ultimate goal of removing barriers in order to improve students learning. During our literature review, we were unable to find a program more similar to Seattle s FSP than the Partners for Success program. As a result of requests by the Office for Education, we conducted a comparison of the activities and outcomes between the two programs. Our source for information on the Partners for Success Program is a report from September 2011 that evaluates the program for the school year (Kundin, 2011). This comparison should be read with caution, as it compares programs with different settings, histories, populations served, staff, and outcome measures. The table below provides a summary of the comparison. Partners for Success Seattle FSP Demographics of students served Number of youth served FSW characteristics Reach Organizational support trainings Organizational support additional needs voiced by staff Process for monitoring family and student change Primary outcomes of interest Most frequent activities Not provided Not provided Not provided 10 elementary schools in urban and suburban Minnesota 9 in-service trainings, once a month, three hours each, 27 hours total Updated database and database management skills Balanced workload Needing additional materials to share with families Dissatisfaction with SOAP notes led to the development of Family Learning Plans, with a focus on academic goals, input from parents, students, teachers about progress, and regular follow-up Reading ability Communication with parents Providing basic needs focused on academic 75% African American or Latino 87% poverty 21% special education 53% single parent 2,628 (1,083 focus students, 1,545 nonfocus students) 71% with a Bachelors or Masters degree; 81% more than 10 years experience Located in 28 elementary schools, available to all schools (including middle and high) for purchase 2 in-service trainings, 31 hours total Updated database and database management skills Trainings focused on improving academic outcomes with family support Balanced workload Needing additional case consultation availability SOAP notes, but FSWs express some dissatisfaction with this approach A variety of outcomes as depicted in the Logic Model (see appendix) and described as tailored to each family and school s need Communication with parents Direct service, advocacy, or intervention 59

60 Qualitative comments interviews with stakeholders Outcomes standardized student progress Outcomes other academic indicators Outcomes external resources Outcomes family interviews Satisfaction outcomes Communication with teachers Communication and collaboration with parents, teachers, and other school staff is a very important part of the job Help provided to parents is most often focused on academics helping with homework plans, reading materials, referral to special education 50% of 40 randomly selected students showed growth on Measures of Academic Progress for reading Math scores not available No information provided No information provided 2 out of 15 pre-post questions to 41 parents showed statistically significant improvement (with small to moderate effect sizes, see Appendix E) Parents and teachers expressed high levels of satisfaction Family involvement Providing basic needs Communication and collaboration with parents, teachers, resource providers, and other school staff is a very important part of the job Help provided to parents is most often focused on meeting basic needs, identifying and resolving challenges, connecting to community resources, and being a communication liaison and/or mediator with the school 65% of all focus students for whom data was available showed growth on Measures of Academic Progress for Reading 67% of focus students showed growth on MAP math scores School- and individual-level analyses do not demonstrate support for the notion that the FSP has a positive impact on standardized test scores, academic progress, disciplinary actions, or attendance Trends indicate that the FSP may be related to lower mobility Trends indicate that the FSP may be related to more external resources in schools 15 out of 17 pre-post questions to approximately 800 parents in the FSP showed statistically significant improvement (with small effect sizes, see Appendix E) Parents and teachers expressed high levels of satisfaction Of the information we can compare, and based on our conversations with staff from PfS and FSP, the two programs are quite similar along several dimensions. In both programs, family support providers work to try to reduce the barriers to learning. We cannot provide comparisons between the programs on information related to demographics of those served, numbers of people served, or numbers of staff, because that information was not provided by the Partners for Success report. Total amount of training time over the school year was roughly the same for both programs, but the PfS program met monthly, whereas the Seattle FSP had two in-service trainings per year. The FSP has a greater reach, with FSWs located within 28 elementary schools and available for purchase by all Seattle Public Schools, compared to 10 schools in the PfS program. 60

61 Staff in both programs have voiced concerns that the database they use to collect and manage their client caseload is ineffective, that they have a difficult time balancing their workload (particularly juggling handing case management, crises, and data collection/entry). The SPS FSP asked for trainings to be focused on improving academic outcomes, and needing some additional availability for case consultation. Interestingly, at one time the PfS program used SOAP notes as a case management tool, just like the SPS FSP does currently, but due to staff dislike of the process these were changed to the development of Family Learning Plans (FLPs). The SPS FSP have expressed a desire to change their case management tool. The programs also differ in their focus on outcomes of interest. PfS is clear that removing barriers in order to improve children s reading is the primary goal of the program. The SPS FSP on the other hand is focused on a broad variety of outcomes, and while they assert that the type of work they do is logically linked to academic improvement, there was definitely quite a bit of voiced concern that their work was broader. The types of activities are very similar, usually involving communication, liaison/mediation, and providing basic needs, though we have the sense that the activities of the PfS are more highly focused on supports that can improve learning. In terms of outcomes, both programs did not find any strong connection to academic success. The FSP analyses provide much greater detail on a wide array of possible outcomes, which is fitting given the different goals of the two programs (described above). Parent surveys at the beginning and end of the year were conducted with 41 PfS parents and approximately 800 SPS FSP parents. Questions asked about constructs such as the parents level of involvement with the school and supporting their child s education. The PfS surveys were provided to parents by external evaluators, whereas the FSP surveys were provided by FSWs. The PfS study only found 2 out of 15 items which demonstrated statistically significant improvement, both with small to moderate standardized effect sizes. This compares with 15 out of 17 items for the SPS FSP, with small effect sizes. Detailed analyses from the parent survey can be found in our appendices of our interim report. Finally, both programs did find very high satisfaction from parents and teachers during interviews or focus groups. 61

62 Summary and Recommendations As with all real-world evaluations, this study has a number of limitations. Most importantly, families and schools are selected to receive family support due to characteristics which make them less likely to experience success. Without an appropriate comparison group or precise statistical control variables, comparing individuals can be misleading. We attempted to build a statistically-controlled comparison group but due to limitations in the data and due to the broad coverage of the program, this proved to be an ineffective exercise. We attempted to validate our findings by triangulating our analyses using school-level data, focus group data, and familyreport data and, in fact, there was a great deal of consistency in our findings. Additionally, our focus groups were hampered by relatively low participation rates. This is particularly true for principal participation, as only 2 principals responded to our requests, despite numerous contacts. As with all non-randomly selected study designs, responses from focus group members may have been biased based on their self-selection to participate. Despite these limitations, our findings suggest that the Seattle Public School s Family Support Program is engaging in a broad number of activities in an attempt to provide supports and remove barriers to academic success. Based on activity logs and countless anecdotes from families, teachers, FSWs, and FSW leadership, it is clear that many FSWs do whatever it takes to provide support and assistance to families in need. These activities received a great deal of enthusiasm from the parents, school staff, and principals who were in our focus groups. Quantitative analyses did not indicate that the FSP is related to improved academic performance (as measured by standardized Measures of Student Progress), attendance, or disciplinary actions. However, there was some indication that the FSP is related to decreases in school mobility and increases in the availability of external resources to students. It appears that the FSP can have an impact on these proximal outcomes, but that these are not related to more distal academic success. Certainly, our finding that there is an extremely strong negative correlation between school-wide academic success and school-wide poverty supports the notion that youth in poverty are facing daunting challenges to learning. This raises the question of if and how family support can be structured to actually impact academic indicators. Existing theories of family support to remove barriers in education suggest several recommendations: 1) Programs should strive to function as core components of the educational mission of the school; 2) Programs should develop an array of innovative methods of engaging, involving, and communicating with families from all backgrounds; 3) Programs should seek family and community input on needs and preferences; 4) Support activities should be empowering, related to removing barriers to learning, comprehensive, coordinated with the school, culturally relevant, and continually monitored and evaluated; 5) All program activities should be linked to the mission of the program. Synthesizing the literature, theory, and our findings findings leads to a number of recommendations for the FSP, the OFE, and the Seattle Public School District. 1. Come to agreement on a way to evaluate the impact of the FSP on students and families while simultaneously a) satisfying the goals of the Families & Education Levy (FEL) and b) honoring the values and experience of the FSP. 62

63 Based on our interviews and meetings with the entire range of stakeholders, including the FSP, OFE, school staff, and families, it is clear that there is still general disagreement about the purpose of the FSP in terms of impacting academic success. When pressed, all parties agree that academic success is an appropriate outcome. As our focus group findings illustrate, the only academic indicator viewed as inappropriate is benchmarked achievement scores. Standardized measures of progress are viewed as appropriate outcomes to measure. Stakeholders must come to agreement on what outcomes to measure, how to measure them, how to determine success, and how to accurately describe the activities of the FSP. Many FSP staff value an open door policy to serving families, innovation in service delivery, and responsiveness, and they feel constrained by case management protocols that restrict who and how they can serve families in need. At the same time, the FEL demands accountability for demonstrable improvement in student outcomes, academic achievement, school readiness, and dropout prevention. The tension created between the necessary constraints on who and how the FSP can serve families that is created by this need for evaluability on the one hand and the desire for openness and innovation on the other must be resolved. 2. Develop a comprehensive plan for continuous quality improvement that integrates the goals of the OFE, the SPS, individual schools, families, and family support workers. Continuous quality improvement activities should be focused on integrating the recommendations from existing family support literature with the current limitations of the program. In order to become a core component of the school, the FSP must align its activities with the needs and goals of schools and students. This requires a systematic view that integrates school- and individual-level needs assessments and monitoring of progress, good data on family support activities being done by the program, timely and relevant child and family outcome data, and ongoing support, supervision, and training of FSWs, with timely consultation and feedback. Much of this work is already being done by the FSP, particularly in certain schools, but could be tightened up and better distributed across the program. 3. Develop quality data collection and feedback systems. Continuous quality improvement requires ensuring that needs, activities, and outcomes are closely monitored and readily accessible. Data should be used to drive activities and ensure that the activities of the FSP are consistent with the needs of the school and families. Conducting needs assessments that drive family support activities has been shown to be related to improved academic performance (Kalafat, 2004). Systematic needs assessments for schools and individuals should be conducted on a regular basis, incorporating consistent and systematic data from principals, teachers, and other school staff, as well as school demographic data and School Improvement Plans. The FSP does conduct needs assessments for schools, but the extent to which data regularly informs this process is unclear. The current site visit report form, for instance, does not clearly and 63

64 consistently capture the range of important factors related to school need. For individuals, the FSWs expressed that the current needs assessment process is inadequate. Similarly, the current system for collecting data on FSW activities is somewhat antiquated and FSWs uniformly expressed dissatisfaction with the process. We recommend that data tracking systems be built which are 1) easy to use and flexible, 2) integrate data among family needs, family support activities, family outcomes, and academic outcomes, 3) provide instant reports on needs, activities, and outcomes at the level of the overall program, individual schools, individual FSWs, and individual families. This system can be used to flag cases that need additional intervention and provide information to supervisors so they can address FSW needs. 4. Continue to support FSWs to be collaborative components of their school by involving them in school assessments, planning, and programming activities. As described above, we recommend that the FSW activities are coordinated with the school s needs and goals and the families needs and goals. FSWs should be involved in school needs assessments, planning, outcome assessments, and programming. This is facilitated when FSWs are provided with regular feedback on school data such as attendance, disciplinary actions, and homework completion. Using data to drive activities and to integrate the activities of the FSW with the school is already being done at some levels in some schools. For instance, in one school we visited, based on data and feedback from the school district, school attendance was deemed an extreme level of need. To address this, school staff provided data to FSWs regarding the weekly levels of attendance for all students in the school. In order to prioritize those individuals with the most need, students were sorted and color-coded by levels of poor attendance. During meetings occurring every Monday morning, the FSW met with school administration, teachers, and school staff to plan activities to address those youth with poor attendance. It is this type of coordination of data and feedback that is likely to lead to improved outcomes. 5. Work with FSWs to fine-tune trainings and other professional development opportunities to help meet FSW needs, particularly in terms of connecting FSW activities to improving academic outcomes. FSWs are asking for more assistance in conducting activities that are directly related to improving academic outcomes. FSW trainings should strive to specifically connect family support activities and academic outcomes. This does not mean training on how to do activities such as tutoring. It means providing training to help FSWs identify which needs are directly impacting academic success, and helping FSWs learn how to work with families to address those needs. Trainings should also incorporate best practices for adult learning. While lecture-style formats are a good way to convey information, they are extremely poor at teaching skills and new behaviors. Role plays and other behavioral rehearsal or practice can greatly facilitate learning new skills. 6. Provide additional consultation and support to FSWs, particularly for families with complex needs. 64

65 FSWs feel like they need additional support in working with families with the most complex needs. They appreciate the MSW case consultation program and want it to expand. The program should ensure that speedy and effective case consultation is easily available and coordinated with the Central Office. 7. The OFE and Seattle Public Schools should work together to develop contracts in a timely manner. Participants in our focus groups repeatedly stated that regular end-of-year anxieties about FSW allocations caused inefficiencies in their work. Additionally, principals and teachers reported feeling punished for success if their school improved to the extent that they were no longer selected for the FSP. Timely contracts can reduce the length of time and growing end-of-year anxiety that results from these issues. 8. Maintain and ensure the private nature of the relationships between FSWs and families. Our focus groups with families made it clear that they highly value the confidential nature of their use of family support services. Any family support structure should strive to ensure that families can trust the FSW even during periods of discord with the school or school staff. FSWs often play the role of mediator, and confidentiality is necessary to allow them to play that role without feeling undue pressure from school or district leadership. Additionally, families felt that there were possible feelings of shame or stigma that could be related to the use of family support services if their confidentiality was violated. 65

66 Appendix A: Data Source and Evaluation Variable Crosswalk FG-ALL = ALL TYPES OF FOCUS GROUPS FG-FSW = FOCUS GROUPS, FAMILY SUPPORT WORKERS FG-TEACH = FOCUS GROUPS, TEACHERS AND STAFF FG-PARENT = FOCUS GROUPS, PARENTS FG-PRIN = FOCUS GROUPS, SCHOOL PRINCIPALS ACAD DATA = ACADEMIC DATA (FROM SEATTLE PUBLIC SCHOOLS) TEACH SURV = TEACHER SURVEY (FROM FAMILY SUPPORT PROGRAM) PARENT SURV = PARENT SURVEY (FROM FAMILY SUPPORT PROGRAM) FAM SUPP = FAMILY SUPPORT DATA, FROM SEATTLE PUBLIC SCHOOLS, COLLECTED BY FSP SCH CLIM = SCHOOL CLIMATE SURVEY (FROM SEATTLE PUBLIC SCHOOLS WEBSITE) CENSUS = CENSUS DATA Data Source ACAD DATA--DEMOG FAM SUPP ACAD DATA GPA ACAD DATA MAP ACAD DATA DRA ACAD DATA WASL_MSP_HSPE ACAD DATA--Discipline ACAD DATA School Health ACAD DATA DEMOG ACAD DATA ELL Student characteristics Demographics Family Support Program Goals Academic Functioning grades, test scores, disciplinary actions Student Health ELL status Data Source School Characteristics Neighborhood characteristics CENSUS o Demographics: Rates of race/ethnicity, poverty, employment Length of time the school has had an FSW Length of time the current FSW had been an FSW Rate of homelessness Rate of ELL Rate of free/reduced lunch FG PRINCIPALS FG TEACH FG-FSW Rate of special ed Principal and other administration support of FSW Data Source FG-ALL FAM SUPP Data Source FG-ALL FG-ALL FAM SUPP PARENT SURV FAM SURV FAM SURV FG-ALL FAM SURV FG-ALL Program Activities Description of types and intensity of program activities Outcomes Short Term Family Family better able to meet basic needs (food, clothing, shelter) Increased family involvement and accountability in the education of the child o Help with homework o Volunteering at school 66

67 FAM SURV FG-ALL FAM SUPP FAM SURV FG-ALL FG-ALL FG-ALL FG-ALL FG-ALL FG-ALL FAM SUPP FAM SURV ACAD DATA MOBILITY FG-ALL FG-ALL FG-ALL FG-ALL FG-ALL FG-ALL FAM SUPP ACAD DATA ATTEND FG-ALL FAM SUPP TEACH SURV FG-ALL TEACH SURV FG-ALL FAM SUPP TEACH SURV ACAD DATA Discipline FG-ALL FAM SUPP TEACH SURV FAM SURV FG-ALL FG-ALL FAM SURV FG-ALL FG-ALL FAM SUPP FAM SURV ACAD DATA MOBILITY ACAD DATA ATTEND FG-ALL TEACH SURV ACAD DATA GPA FG-ALL ACAD DATA MAP ACAD DATA DRA ACAD DATA WASL_MSP_HSPE o Participating in school events o Participating in school meetings Strengthened self-efficacy for parents Increased sense of hope for parents Increased sense of parent responsibility Increased effective advocacy skills for parents Increased connection to the community Lowered family mobility Family has increased networks with community organizations and other supports Improved social skills for students Increased parent awareness and understanding of role in child s education Improved emotional stability Increased safety of child and family Short Term School Related Improved attendance Improved school readiness Improved behavior in school o Decreased suspensions/expulsions Improved homework completion Improved relationships between parent and school o Increased parent-school communication Improved teacher and school administration perceptions and understanding of student and family Increased retention in school Improved grades and academic performance o Improvement on standardized tests 67

68 FG-ALL SCH CLIM FG-ALL ACAD DATA MOBILITY FG-ALL FG-ALL FG-ALL FG-ALL FG-ALL FG-ALL FG-ALL FG-ALL Improved school climate Long Term Increased graduation Increased college enrollment Increased continuing education (e.g., vocational and technical) Increased employment Prevention of incarceration Increased self-respect (students and parents) Increased community respect (students and parents) Improved life skills (e.g., through community learning) 68

69 Leadership Committee Appendix B: Leadership and Advisory Committees Membership Name Organization Role Thelma Payne Family Support Program Family Support Program Manager Janet Preston Family Support Program Family Support Program Supervisor Isabel Munoz-Colon Seattle Department of Neighborhoods Program and Policy Advisor Aisha Cathcart Family Support Program Family Support Worker Mike Pullmann University of Washington Researcher Eric Bruns University of Washington Researcher Erika Wiggins University of Washington Researcher Anne Fitzpatrick Kimball Elementary School Principal Norma Zavala Concord International Elementary School Principal ElDorris Turner Van Asselt Elementary School Principal Jeff Imel Bailey Gatzert Principal -- Parent -- Parent -- Student Clyde Walker Advisory Committee Name Organization Role Thelma Payne Family Support Program Family Support Program Manager Janet Preston Family Support Program Family Support Program Supervisor Isabel Munoz-Colon Seattle Department of Neighborhoods Program and Policy Advisor Aisha Cathcart Family Support Program Family Support Worker Mike Pullmann University of Washington Researcher Eric Bruns University of Washington Researcher Erika Wiggins University of Washington Researcher Anonymous -- Parent Anonymous -- Parent Clyde Walker Mohomed Roble Seattle Public Schools Family Engagement TJ Cosgrove School-Based Health Services Sid Sidorowicz Seattle Department of Neighborhoods Pat Bates Assistance League Elise DeGooyer NW Harvest Norm Johnson Therapeutic Health Services Executive Director Jennifer Murray -- School Counselor/Former FSW Sean Robinson -- School Counselor/Former FSW Dinah Ladd Seattle Public Schools Home Coordinator Anonymous -- Student Glen Bafia Seattle Education Association Head of Union representing FSWs Dannette Smith Human Services Department, City of Seattle Director Narcita Eugenio ELL and International Programs Student and Family Advocate 69

70 Appendix C: Logic Model Seattle Public Schools Family Support Program Evaluation Logic Model Mission: The Family Support Program strives to inspire and motivate Seattle Public School children to succeed academically by advocating increased family involvement in education, facilitating family access to supplementary academic resources and providing support to help families overcome social, emotional, and physical barriers that may hinder student academic progress. Conditions: Why the Family Support Program is needed Students who have high needs and poor past academic performance tend to continue to do poorly in school, and have much higher rates of poor attendance, low grades, disciplinary problems, dropout, unemployment, and problems with the law Large disparities in academic performance for Latino/a, African-American, and African students Economic, basic needs, and other stressors can act as barriers to learning Poorly performing students often have disciplinary problems which create barriers to other student s learning Lack of partnership between schools, communities, and families Language and cultural barriers experienced by an increasing immigrant presence in the Seattle Public School System Families often lack understanding of school expectations Strengths and capacity: What we have to build on Highly experienced, dedicated, and knowledgeable staff most FSWs have been in this role for over 10 years The ability to cultivate and retain talent to perform this unique work FSWs are organized within clusters to better network with one another and share best practices Training is delivered on topics relevant to emerging issues for the students who are served Strategically located in individual schools to better access and serve students Long-lasting, positive relationships with families, staff, communities, businesses, and funders Infrastructure Located in all public elementary schools, one middle school, and the SPS Bilingual Orientation Center, and available to middle schools and high schools for purchase 45 staff and 40 FSWs in 2011 Schools are tiered based on need and types of services provided FSWs are assigned based on the needs of the school and the skills of the FSW; tasks are tailored for each school The FSP uses a weighted eligibility formula to determine what level of services elementary schools will receive Our philosophy/theory of change: Early intervention with high-risk students may promote years of good outcomes, benefits, and cost-savings Supporting families by providing basic needs and connecting to community resources will help remove barriers to attendance, learning, and performance Supporting families connections to the school will help increase teacher-parent communication, parent participation in school activities, parental understanding of academic standards and school policies, and parental support for child s education outside the school setting Being located in the schools allows FSWs to be more accessible, intervene more consistently with students, and reduce stigmatization of family help seeking. Allowing each FSW flexibility to provide services that meet the individual needs of schools, children, and families makes help that is provided more individualized and effective Program participants: Who we reach Program receives list of students not meeting standards, from this list, the school planning team determines focus group students based on the following criteria: Children who are experiencing academic challenges, attendance challenges, lack of homework completion, high mobility, low parent involvement, behavioral concerns Particular focus on Hispanic and African American males 70

71 Program components: What we do For Parents: Serve as a liaison to the school and as a navigator to information, policies and services of the school system Help parents meet their families basic needs (housing, food, utilities, and health) to reduce impact on child s learning Help parents understand basic expectations of the school, such as around attendance and homework Organize parent educational events Provide emotional support, mentoring, encouragement, and motivation Promote and connect parents to community resources, services, and programs Support educational endeavors of eligible parents (e.g., through the Seattle Milk Fund Grant Program that provides partial education and childcare funding) Promote connection to substance and alcohol abuse treatment and recovery programs Administer surveys about involvement in child s education For Students: Provide mentoring and/or mentoring resources Provide assistance with school work/one-one-on-one academic support Provide case management for homework and attendance concerns Connect youth to specific programs and services that will help their academic success (e.g., tutoring) Meet basic needs specific to the child (e.g., supervision, recreation, physical and mental health, and nutrition) Support students via special culturally appropriate projects sponsored by the program: o African American Male Project (AAMP)- Selected three schools with high proportion of AA males, features small group sessions, mentoring, and male leaders club o East African Tutoring and Mentoring Program (EAASTMP) - Implemented at Aki Kurose Middle School. Includes tutoring, field trips, and workshops. o Latino Achievers Academy-Students participate in activities and workshops to enhance their social and educational experiences o Wraparound- coordinating care and services to improve outcomes for students Play direct role in behavior or academic plans as defined in plans developed by the school s SIT team Advocate for foster care placements in the community when a child needs to be placed out of the home For Families (Parents and Students): Provide case management Help families in crisis Make home visits Provide transportation Facilitate families access to culturally appropriate school and community resources Provide families with monetary assistance and resources if needed. Some of which include gift cards, bus tokens, rental assistance, utility assistance, clothing through Operation School Bell, and summer camp scholarships Mentoring and modeling Developing behavior and academic plans Support families via special projects sponsored by the program: o Primary Learning and Intervention Project- Engage parents and students in early learning, provide parent and student learning libraries, collaborate with community early learning programs. Connections are made with Pre-K teachers Organize school events (e.g., family night) and recruit families to attend events Advocate for families For Schools: Look at school-level indicators (e.g., mobility, parent involvement, homework completion, CPS involvement) to determine the best use of FSW time in that school Serve active role in behavior and academic plans and partnering in SIT teams and IEP teams Surveys administered to teachers to assess student performance three times per year Serve as a liaison to connect teachers with parents Assist in the development of Individualized Education Programs (IEPs) With the Community: 71

72 Build partnerships with community members and organizations to support the program and program participants. o Community partner agencies, corporations, and programs help provide clothing, mental health counseling, food, and other assistance. These include, but are not limited to Windermere Foundation, Northwest Harvest, Therapeutic Health Services, Assistance League, Seattle City Lights, Big Brothers/Big Sisters, Friends of the Children, West Seattle Rotary Club, Boys and Girls Club, Communities in Schools, Treehouse, Costco, Atlantic Street Center, Central Area Youth Association, Friends of the Children, neighborhood churches, and community centers. Provide linkages between state agencies such as DSHS and schools/students Outcomes Short Term Family Family better able to meet basic needs (food, clothing, shelter) Increased family involvement and accountability in the education of the child o Help with homework o Volunteering at school o Participating in school events o Participating in school meetings Strengthened self-efficacy for parents Increased sense of hope for parents Increased sense of parent responsibility Increased effective advocacy skills for parents Increased connection to the community Lowered family mobility Family has increased networks with community organizations and other supports Improved social skills for students Increased parent awareness and understanding of role in child s education Improved emotional stability Increased safety of child and family Short Term School Related Improved attendance Improved school readiness Improved behavior in school Improved homework completion Improved relationships between parent and school o Increased parent-school communication Improved teacher and school administration perceptions and understanding of student and family Increased retention in school Improved grades and academic performance Improved school climate Long Term Increased graduation Increased college enrollment Increased continuing education (e.g., vocational and technical) Increased employment Prevention of incarceration Increased self respect (students and parents) Increased community respect (students and parents) Improved life skills (e.g., through community learning) 72

73 Appendix D: FSW Activity Report Categories Food Clothing/Shoes Housing Basic Needs Heat/Utilities Child Care Transportation Assistance Job/Employment Assistance Advocate for Family s/ School Admin. Or Teaching Staff Observe Student in Classroom Advocate/Intervene Other Direct Services/ Educational Supplies Miscellaneous Tutoring After School Advocacy or Tutoring During School Intervention Homework Club WA State Test Preparation Other Academic Enrichment Program Crisis Assistance Crisis Assistance Medical Dental Vision Physical Health Health Insurance (Includes Medicaid) State/Federal Agency Program Other Health Issue Emotional/Social Counseling Mental Health Counseling Social Services Drug/Alcohol Program State/Federal Agency Program Family Involvement Home Visits + Parent Contact Attended SIT/IEP/MDT Attended Parent Teacher Conference Support School Policy/ Procedures (Attendance, Behavior, etc.) Support Student Ed. Outside School Support State/District Academic Standards Academic Team Meeting Other School/Teacher Meeting School Family Literacy Night Family Math Night Family Health Night Other Academic Workshop WA State Test Prep. Workshop Non-Academic School Activities (e.g. PTA, Family, Parent Night) Home Visit (Family Home) Home Visit (Off-Site Location) Home Visit (School) Phone Call Letter/ Flyer/Information Bilingual/IA Interpretation SIT Meeting IEP Meeting MDT Meeting Annual Parent/Teacher Conference 73

74 Paired samples t-tests How often Went to school meetings or conferences held at the school Volunteered for activities at the Appendix E: Parent Interview Questions FSP Fall 2010 FSP Spring 2011 FSP p Effect size (Cohe n s d) PfS Study p PfS Effect Size study d school Communicated with my child's teacher about his or her school progress 1 Participated in daytime activities such as assemblies, events, or field trips Went to SIT/IEP/MDT Meetings Participated in School Family Night OR other evening school events Participated in organized family events around academic subjects; Math, Literacy, and/or WASL Prep My child completed and turned in his/her assigned work Chi-Square McNemars Fall 2010 %Y Spring 2011 %Y p MN Study p MN study d Attended my child's Parent- Teacher Conference I personally helped my child with homework I arranged for someone to help my child with homework. My child's teacher assigned homework Enrolling them in an after-school program Connecting them with a mentoring program Regularly participating in a Reading/Learning activity Enrolling them in an in-school or after-school tutor Visiting a library or cultural center 1 or more times

75 Additional significant PfS items How often do you use educational materials that are sent home with your child (e.g., books, flashcards, websites)? talk with your Family Support Worker about your child s educational progress? 1 MN Study: How often do you communicate with your child s teacher? 75

76 Appendix F: Training Details FAMILY SUPPORT PROGRAM JUNE INSTITUTE Tuesday, June 28, 2011 & Wednesday, June 29, 2011 Mount Zion Baptist Church th Avenue Seattle, WA DISTRICT ADMINISTRATION Pegi McEvoy Director of Safety & Health Thelma Payne Family Support Program Manager Janet Jones Preston Family Support Supervisor OFFICE OF EDUCATION CITY OF SEATTLE Holly Miller Director of Office of Education Isabel Munoz-Colon Program and Policy Advisor 76

77 ADMINISTRATIVE STAFF Linn Luu-Hibbert Faith Howard Violeta Plaza logo Funded by OFFICE OF EDUCATION CITY OF SEATTLE FAMILY SUPPORT WORKERS Carmen O Reilly Virginia Owens Dominique Pie Natalie Long Yolanda McGhee Michael Melonson Mary F Miller Helen Mitchell Suldan Mohamoud Schelley Moulton Paula Murray Pamela Rago Michelle Raine Sandra Smith Nebiat Tessema Tracey Thompson Acquinetta Williams Toni Williams 77

78 Tuesday s Agenda 8:30 Breakfast, Social Time & Registration (Hosted by West Seattle & South Seattle Clusters) 9:15 Greetings & Introductions 9:30 Department of Social & Health Services Economic Services Administration Community Services Division 12:00 Lunch 1:00 Rhoda Ramirez Department of Social & Health Services Child Protection Services Supervisor CPS REPORTING INTAKE & PROCESSING WITH THE NEXT STEPS 4:00 Closing Wednesday s Agenda 8:30 Breakfast & Social Time (Hosted by South, Central & Traditional Clusters) 9:15 Greetings 9:30 Sandi Schram M.C., LMHC WORKING WITH STUDENTS OF DOMESTIC VIOLENCE 12:00 lunch 1:00 Michelle Hetzel, MSW Department of Social & Health Services CPS Social Worker *Child & Family Welfare Service THE IN AND OUTS OF FORSTER CARE 4:00 Closing 78

79 FAMILY SUPPORT WORKERS Nickolaus Adams Carla Arellano Sharon Baez Garcia Margaret Balderama Barbara Blake-Hodge Nina Bowman Guillermo Carvajal Aisha Cathcart Sherryl Cox Cedric Dennard Gerald Donaldson Keith Ervin Delores Evans Akim Finch Jesselyn Finkley Vallerie Fisher Marcel Hauser Tracie Holiday-Robinson Alesia Jessie Daphne Jones 79

80 Best Western Executive Inn 8:30 AM 9:00 AM: Registration and Breakfast 9:00 AM 9:30 AM: Welcome & Introduction Antowaine Richardson 9:30 AM 11:30 PM: Accountability and Team Work Noon 1:00 PM: Lunch 1:15 4:00: Accountability and Team Work continued FSWs report to assigned school site and prepare for work; connect to community; Complete short biography for website by Wednesday Best Western Executive Inn 8:30 AM 9:15 AM: Registration and Breakfast 9:15 AM 9:30 AM: Greetings & Introductions Atlantic Street Sue Siegenthaler 9:15 AM 10:30 AM: Agency Resources and Partnering Department of Social & Health Services John Allen 10:45 AM 12: 15 PM: Fathers Engagement and Resource Support 12:15 PM 1:00 PM: Lunch Seattle Public Schools Judy Simon, Safety Supervisor 1:00 PM 2:20 PM: Safety advice for staff during home visits Seattle Police Department Terrie Johnston 2:30 PM 4:00 PM: Community/Personal Safety Qs & As 80

81 Best Western Executive Inn 8:30 AM 9:00 AM: Registration and Breakfast 9:00 AM 9:15 AM: Greetings & Introductions Seattle Public Schools Catherine Erickson 9:15 AM 9:50 AM: Employee Assistance Program Seattle Public Schools Barbara Martinson 10:00 10: 30: Risk Management Family Support Program - Faith Howard 10:45 11:30: Data and Medicaid Family Support Program - Linn Luu-Hibbert 11:30 Noon: Site Fund, Nuts & Bolts Noon 1:00 PM: Lunch Seattle Public Schools Dinah Ladd 1:00 PM 1:25 PM: Homeless and Transitional Housing City of Seattle Isabel Munoz-Colon 1:30 2:15: City Report & Introduction UW-Public Behavioral Health & Justice Policy Team University of Washington - Dr. Michael Pullmann, Dr. Eric Bruns and Ericka Wiggins, Research Coordinator 2:30 PM 4:00 PM: Family Support Program Evaluation Seattle University Pigott Building 8:30 AM 3:45 PM: Gang Prevention Training Mandatory First Aid and CPR Training 81

82 82

83 Mental Health and Family System Conference Program 9:15 AM Key Note Speaker: Barbara Bennett, PhD Families In Crisis: Struggling with Mental Health Issues Barbara Bennett is a School Psychologist and Special Education Consultant who has worked with Seattle Public Schools since 2007 providing professional development and coaching to teachers. She has taught workshops throughout the school district and region about the impact of trauma on both students and school staff. In 1989 she co-founded First Place, a non-profit agency and school for children and families who have experienced the trauma of homelessness and other family crises. Dr. Bennett has been an instructor in psychology with the University Behind Bars at Monroe Correctional Complex for the past three years. She received her M. Ed and PhD from the University of Washington. 10:00 11:45 AM BREAKOUT SESSIONS: Working with using families toward recovery and families in recovery: Anthony Austin & Kathi Kuciemba This workshop will explore the impact of drug and alcohol addiction on families. It will address best practices in helping families cope, seek help, and utilize resources in order to rebuild the family s ability to support their children. Anthony Austin: Anthony Austin has worked with families at Therapeutic Health Services for the past 5years. He is currently the lead facilitator for THS Focus on Family. Focus on Family is a parenting curriculum designed to assist families engaged in drug and alcohol recovery services. Anthony is also the program supervisor for the Making a Difference at Madrona Program. Anthony has a Master s degree in Education, over ten years of experience working with at-risk youth in the Seattle Public Schools, and several years of experience working as a parent educator. Kathi Kuciemba: Kathi Kuciemba has worked with Pregnant and Parenting Women and Families at Therapeutic Health Services for 30 years. She is the Manager of Services for the Pregnancy and Family Recovery Program and Childcare at THS Summit, Seneca, Shoreline, Everett and Eastside Branch s. Kathi became passionate about working with women, children and families developing strong recovery and parenting skills after teaching a class to women in prison who all had been addicted to drugs with children before they were sent to prison. Depression and the Family System: Dr. Kendra C. Roberson Objectives: Identify, review components of family systems theory and the use of Systems Analysis to assess cognitive, physical and emotional symptoms of depression. Dr, Kendra C. Roberson: Kendra Roberson graduated from the Bush School, Seattle, WA in 1986; earned her BA in Psychology from Spelman College, Atlanta, GA in 1990 her and Master of Social Work from Boston College Graduate School of Social Work, Chestnut Hill, MA in Kendra earned her PhD in Social Work from the U. of Maryland in 2003, and, in 2005 completed a 2 year post-doctoral research fellowship at the U.W. Social Development Research Group. The majority of her practice (clinical) experience has been with children, adolescents and their families. Her experience includes working with pregnant and post-partum teens and their babies in Syracuse, NY; and adolescent psychiatric in-patients at a Boston psychiatric hospital 83

84 The Impact of Situational Trauma on Children and Family Communities Looking at Trauma through a Cultural Lens : Dr. Barbara Casey Situational trauma is defined as a recent or past stress in a child s or family s life resulting in social, economic, emotional, physical, mental, cognitive and academic impacts resulting in a deficit in a child s or families interaction with their global world. In today s workshop, you will learn about situational trauma through the impact and influence of cultural variables (traditions, communication, socio-economics, family influences, parenting, physical and social environments, learning styles etc.) in how children and family communities present, understand the world and interpret experiences in the complexity and aftermath of trauma. Providing strategies and articulation of the issue from a cultural competency lens will engage staff in the discussion on best practices to support students and families who have been impacted by situational traumatic stress. Dr. Barbara L. Casey: Dr. Barbara Casey is an Administrator in Seattle Public Schools, Seattle, WA. She is currently serving as an Assistant Principal on Special Assignment with Systemic Intervention. In addition, she is a freelance writer and former managing editor with Brighthub.com and author of 244 educational articles. Dr. Casey is also a private consultant on state assessments, education/ business software, student academic performance measures for students at risk and doctorate dissertations. She has served as an Adjunct Professor at Antioch University and City U in Bellevue, WA and is currently serving as Adjunct Professor at Ashford University teaching online graduate education courses and is the Professor on record at Seattle Pacific University for three classroom management courses which she coauthored. Dr. Casey is an educational consultant/trainer who holds a doctorate in Educational Leadership from Seattle University. She is also the author of the book, "Resiliency Factors and Performance on the Washington State Essential Learning Exam: Psychosocial Issues and Academic Achievement of Students at Risk." Dr. Casey enjoys playing classical piano and is an activist in being the change she wants to see on the planet in closing the achievement gap for all students. 12:00 1:30 Lunch and Speaker: Laura van Dernoot Lipsky: Transforming Trauma - How to do this work and not completely lose our minds. A discussion on, How to reconcile what we experience in our work and the world around us in a way that is compassionate, honest yet accountable & sustainable. Laura van Dernoot Lipsky is an internationally known trauma worker who has been engaged in this work for 24 years. She has worked locally, nationally and internationally and has been involved in some of our nation s largest disasters and community crises. Her work has included front-line work with homelessness, domestic violence, sexual assault, child abuse, the sex trade, natural disasters, environmental destruction, and other acute trauma. She is known as a pioneer in the field of trauma exposure and its toll on those working in trauma related fields. 1:45 3:45 Workshop: Transforming Trauma, cont. - Laura van Dernoot Lipsky 3:45 - Wrap-Up 4:00 - Adjourn Training Registration: Yes, I will be attending the 4/29/11 Training: NAME: 84

85 POSITION: SCHOOL: Breakout Workshop Choices: Break Out Workshop Sign-Up Sheet Please indicate your 1 st, 2 nd, and 3 rd workshop choices on the attached form. Please reply early each workshop s sign-up will close when it reaches its official capacity. WORKSHOP SIGN-UP FORM: Workshop 1 st Choice 2 nd Choice 3 rd Choice 1. Impact of Drug Addiction on the Family System Anthony Austin & Kathi Kuciemba 2. Depression and the Family System Dr. Kendra C. Roberson 3. The Impact of Situational Trauma on Children and Family Communities Looking at Trauma through a Cultural Lens Dr. Barbara Casey Please return this registration form to Linn Luu-Hibbert at Family Support & Community Partnerships Program; JSCEE ; or Fax to Thelma Payne, Program Manager Family Support & Community Partnerships John Stanford Center REGISTRATION DEADLINE: April 22,

86 Appendix G: Site Visit Form FAMILY SUPPORT WORKER PROGRAM SITE VISIT REPORT School FSW Principal Other REFERRAL PROCESS FOR FOCUS FAMILIES: Number of Focus Families Process Number of Parent Signatures Concerns QUALITY OF WORK: E S NI E=Excellent S=Satisfactory NI=Improvement FOLDERS: Service Plan: Case Notes/SOAP: Parent Survey Plan: Number In In File Number In Yes No Random Moments: Planning Team Input: Number In Current Up-to-Date Yes No CBO Number Uncompleted - ATTENDANCE: Daily SIT Meetings Cluster Meetings Comments: (FSW Signature) (Supervisor Signature) (Date) (Principal Signature) 86

87 Appendix H: SAEOP and Paraprofessional Classified Evaluation Form 87

88 88

89 89

90 90