Baltimore s After School Strategy. Assessing After-School Program Quality in Baltimore: Interim Findings of An In-Depth Study of Eight Programs

Transcription

1 Baltimore s After School Strategy Assessing After-School Program Quality in Baltimore: Interim Findings of An In-Depth Study of Eight Programs Prepared by Carolyn Marzke with Lee Pearson September 2003 Policy Studies Associates 1718 Connecticut Avenue, NW Washington, DC Under contract to The Family League of Baltimore City Baltimore, Maryland

2 Table of Contents Page Executive Summary... i 1. Introduction Study Goals and Methods Overview of Participating Programs: Program Structure and Activity Schedules Sponsorship, Location, Facilities, and School Affiliation Program Budgets and Cost Per Youth Estimates Staffing Enrollment Size and Ages Served Attendance and Utilization Rates Hours of Operation and Activity Schedules Youth Characteristics, Reasons for Attending, and Other After-School Destinations School Participation and Performance Risk Indicators Youths Self-Reported Participation Patterns and Reasons for Attending Quality of Interactions Among Youth and Staff Standards and Observation Measures Related to Interactions Group Size and Youth Interactions Staff-Youth Interactions Summary and Conclusions About the Quality of Youth and Staff-Youth Interactions Extent of Opportunities for Skill-Building and Mastery Standards and Observation Measures Related to Skill-Building and Mastery Mastery Orientation Higher-Order Skill-Building Literacy-Building Opportunities Summary and Conclusions About Skill-Building Opportunities Youth Engagement and Satisfaction Youth Engagement in Observed Activities Youth Satisfaction Program Quality, Youth Engagement, and Utilization Youth Perceptions of Program Impact Summary and Conclusions References Appendix A: Summary List of Standards Appendix B: Observation Instrument, Spring 2003 Appendix C: Number of Observations Conducted in Each Site Appendix D: Youth Survey Scales Appendix E: Overview of Analysis Datasets Appendix F: Summary Tables of Results

3 Executive Summary Baltimore s After School Strategy is a collaborative, city wide initiative led by the Safe and Sound Campaign (SAS), the Family League of Baltimore City (FLBC), and The After-School Institute (TASI) to expand and improve the quality of after-school programs for children and youth in Baltimore City. Launched in early 1998, the After School Strategy has directed more than $29 million from a combination of private and public sources to support (1) the development of an after-school system and infrastructure in Baltimore, and (2) expansion and quality improvement in more than forty afterschool programs operating in more than one hundred sites around the city (Evaluation Overview, 2003). To date, the Strategy has funded over 40 provider organizations operating more than 80 afterschool program sites. To guide after-school program quality improvement efforts, the After-School Strategy developed the Standards for Baltimore After-School/Out-of-School Opportunities in YouthPlaces. The Standards are divided into two parts: (1) Organizational Standards, which address the quality of human relationships and the overall program infrastructure; and (2) Program Standards, which address program content and opportunities for skill-building and mastery. The goal of improving quality is integral to the Strategy s theory of change as both a means of drawing more youth to participate in after-school programs and for increasing the likelihood that programs will affect outcomes for participating youth. The In-Depth Study on After-School Program Quality was designed to inform the After School Strategy s efforts to assess and improve program quality by examining quality in a sample of eight programs against the Standards and refining a systematic approach to quality assessment based on structured observations of program activities. The study was guided by the following research questions: What is the quality of these eight programs assessed against the Standards for Baltimore After-School/Out-of-School Opportunities in YouthPlaces? What is the quality of interactions among youth and staff, as defined in the Organizational Standards? What is the quality of program content and opportunities for skillbuilding and mastery, as defined in the Program Standards? What are the links between quality and site features that may affect quality, such as cost or staffing strategies? What is the relationship between quality and program utilization? Are youth more likely to attend higher-quality programs? What is the relationship between program quality, other program features, and youths self-reported experiences and perceptions of the benefits of participating in after-school programs? The findings in this interim report are based on data collected in the eight study sites during the program year, including: i

4 Structured observations of 189 after-school program activities in the fall of 2002 and the spring of 2003, during which observers rated the prevalence of indicators of the quality of youth interactions, staff-youth interactions, mastery orientation, opportunities for higher-order skill-building, and youth engagement. Surveys of 241 youth participants the fall of 2002, which included items about youth characteristics; participation levels and reasons for attending; other after-school destinations; and perceptions of youth interactions, staff-youth interactions, and the benefits of participation. Data about program enrollment, attendance, utilization, and cost from the Family League of Baltimore City Management Information System for the program year Program Features Likely to be Associated With Quality The eight programs included in the study illustrate the diversity of after-school programs funded by the strategy. They are sponsored by large and small agencies, some public and some private, and three of the eight programs are school-based. Three programs serve elementary grades only, one serves middle school students only, two serve elementary and middle school grades, and two are open to youth at all grade levels. On Average, Programs Served Fewer Students Than They Were Contracted to Serve While the range of enrollment target levels was wide (30 to 125), six of the eight programs included in the study were contracted to serve between 40 and 60 youth. The average daily attendance (ADA) for study sites between September 2002 and January 2003 was 49 youth. Compared to the total number of youth the study sites were contracted to serve, these attendance levels translated into a mean utilization rate of 0.78 (i.e., 78 percent of contracted slots were filled on any given day). At the extremes, the utilization rate in one site was 0.26, and another site served more than the contracted number of youth for a utilization rate of 1.3. Program Budgets Vary Widely Among the Eight Sites, As Do Estimates of Programs Cost per Youth The average FY03 budget among the study sites was $132,463. The average cost per youth, based on average daily attendance (ADA) for September 2002 through January 2003, was $4,648, with a low of $583 and a high of $10,348. Program Staff Vary in Their Size and Qualifications The study sites employ an average of nine full- and part-time staff, although staff sizes range from four to 16. Five of the eight sites employ certified teachers as after-school program staff. In the ii

5 absence of staff survey data for the program year, the presence of certified teachers on staff was the sole measure available for staff qualifications. Program Schedules, Mix of Activities Offered, and Extent of Enrichment Programming The after-school components of the programs included in the study operate on varying schedules for an average of 11 hours per week, with a range of six to 15 hours of operation per week. Five sites operate five days a week, one is open four days a week, and two meet twice a week. For the most part, youth in these programs have relatively little choice about the activities in which they participate. Five programs maintain group-based assigned schedules in which all activities are required, and another three have menu-based schedules. Programs offered a mix of the types of activities specified in the Standards: homework help, other academic/cognitive enrichment, arts, sports, workforce development, civic development, and open time. Many programs also offered activities that were difficult to place in one of the program areas defined in the Standards, such as cooking and modeling. The study team computed both the number of different activities offered and an average number of hours per week to describe not only the richness of programming but also the level of exposure youth receive to a particular type of activity. The study sites offered an average of seven different kinds of activities per week. Overall, sites offered more hours per week of academic/cognitive development activities than any other type of activity, including homework help. Arts activities were also prevalent, with the remaining categories less wellrepresented across programs. All sites but one offered both academic and non-academic activities as required by the Standards. In addition to estimating the hours per week of activities offered in each program area, we also estimated the average total hours of enrichment activities offered across program areas, excluding homework time and open time. The amount of enrichment available varied tremendously among sites, from a low of one hour per week (rounding up) to a high of 16 hours per week. Sites fell into two categories based on this estimate those offering 11 or more hours per week and those offering 4 ½ or fewer hours per week of enrichment. School-based programs offered substantially more enrichment hours per week on average 13 hours per week compared to 3 hours per week in community-based programs. Quality of Youth Interactions Observed Interactions Among Youth Were Generally Positive, While Youth Survey Respondents Offered More Mixed Assessments of Peer Interactions During the activity observations conducted for the study, observers recorded the number of youth present and rated youths interactions with one another, documenting both positive and negative indicators of the quality of these interactions. These ratings are summarized in an Observed Youth Interactions Index. In addition, the youth survey included items about peer interactions, which are summarized in a Youth Survey Peer Interactions Scale. iii

6 The average number of youth present in observed activities was 12, and the median group size was 10. These are well within the guidelines established by the Standards, and very few activities exceeded those recommendations. Youth interactions in observed activities were generally positive and typically characterized by warm and friendly exchanges in the course of socializing. Other types of interactions demonstrating specific interpersonal skills such as sharing materials or listening respectfully to peers during discussions or when they are talking in front of the group were less prevalent. While observers recorded very few overtly negative interactions among youth, youth themselves indicated some ambivalence about the extent to which their peers cared about each other, got along well, and treated each other with respect. The majority of youth offered positive responses on these survey items, but were more likely to agree a little than to agree a lot, and a substantial minority offered negative responses to these items. The Quality of Observed Youth Interactions Was Positively Associated with Youth Engagement Levels Youth interactions were somewhat more positive in activities marked by high levels of youth engagement and in activities observed in the spring. Youth interactions were consistent across different types of activities, however, and youth engagement and time of year explained relatively little of the variation among sites in youth interactions observation index scores. Youths Assessments of Peer Interactions Were Positively Associated with the Degree of Mastery Orientation in Observed Activities Youth in programs whose activities were notably more mastery-oriented offered more positive assessments of peer interactions. Older youth and boys also reported more positive views about peer interactions. Quality of Staff-Youth Interactions Observers and Youth Offered Positive Assessments of Staff-Youth Interactions Overall, Although the Extent to Which Staff Provide Individualized Attention to Youth Was Less Clear During the activity observations conducted for the study, observers also recorded the number of staff present during the activity and rated staff-youth interactions, documenting both positive and negative indicators of the quality of these interactions. These ratings are summarized in an Observed Staff-Youth Interactions Index. The youth survey also included items about staff-youth interactions, which are summarized in a Youth Survey Staff-Youth Interactions Scale. The average staff:youth ratio in the activities observed for the study was 1:7 one staff member for every seven youth well below the guidelines of between 1:10 and 1:15 established by the Standards. Staff-youth ratios were at or below 1:10 in 80 percent of observed activities, and at or below 1:15 in 90 percent of activities. iv

7 The quality of staff-youth interactions across observed activities was generally high, reflecting the prevalence of positive interactions and the relative absence of overtly negative interactions among staff and youth. Staff consistently used positive language and tone of voice, and in many activities their interactions with youth were warm and friendly. Informal conversations among staff and youth and individualized attention and encouragement were observed less often than group-based, task-focused interactions. Likewise, youth offered relatively positive overall assessments of their own interactions with staff, and positive perceptions of how staff treat youth. There was more variation, however, in the extent to which youth reported individualized, personal interactions with adults in these programs. In particular, forty percent of youth survey respondents reported that they had never talked one-on-one with a staff member in their after-school programs. There Were No Clear Patterns of Association Between the Quality of Observed Staff-Youth Interactions, Youths Assessments of the Quality of Staff-Youth Interactions, and Site Features Observed staff-youth interactions were higher in spring than in the fall, but this was the only significant factor associated with variations in this measure across sites. The quality of staff-youth interactions was relatively consistent across different types of activities, and was not associated with site features such as location, cost, or the presence of certified teachers on staff. Similarly, youths views about the quality of staff-youth interactions did not differ based on observation measures of quality, nor were there any associations between site features and youths assessments of staff-youth interactions, whether they would talk to staff about a serious personal concern, or whether they had ever talked one-on-one with a staff member. The Qualities That Influence Youths Attitudes Toward Staff May Not Be Well-Captured in Snapshot Observations The results of observation and survey measures of staff-youth interactions did not correspond well with one another. In particular, there was more variation in observers assessments of staff-youth interactions than in youths responses to survey items about how staff treat youth. Moreover, youths responses to questions about more individualized or personal interactions with staff were negatively associated with observation measures. The survey results are based on non-representative samples of youth at the site level, which may also explain the low correspondence between the survey and observation measures. Interviews with youth are likely a better way to assess the qualities of programs and staff that promote positive staff-youth relationships, and may generate more sensitive survey items and observation indicators in the future. Extent of Opportunities for Skill-Building and Mastery The Extent of Opportunities for Skill-Building and Mastery Was Low Overall The degree to which observed activities were designed to help youth master skills was low overall, but varied considerably across sites and types of activities. In general, when staff were focused on helping youth to learn something, staff-centered instructional strategies were more prevalent than other more youth-centered and individualized strategies, such as providing feedback to individual youth v

8 or offering opportunities for youth to produce something or to perform. Higher-order skill-building opportunities such as cooperative activities and activities that require youth to think strategically or solve complex problems, answer thought-provoking questions, or engage in extended discussions with their peers were observed infrequently during observed activities. Measures of Mastery Orientation and Higher-Order Skill-Building Were Higher in Enrichment Activities and in Sites with Certified Teachers on Staff The extent of opportunities for skill-building and mastery varied substantially across sites, and the type of activity and the presence of certified teachers on staff were both strongly associated with ratings of the mastery orientation and higher-order skill-building in observed activities. Intentional skill-building opportunities were most pronounced in arts, academic enrichment activities, and the collection of activities that were difficult to categorize by program area, such as cooking, gardening, and chess club. Academic and other enrichment activities also incorporated more opportunities for higher-order skill development. Not surprisingly, these types of activities were more mastery-focused than sports or open time activities. Efforts to build cross-cutting skills such as literacy, critical thinking, or cooperation in the context of arts, sports, or other apparently recreational activities were the exception rather than the rule in observed activities. In most sites, arts activities targeted artistic skills and products were individual rather than collective, and academic activities generally sought to develop individual youths basic reading and math skills. Open time and sports activities were unstructured opportunities for youth to relax, play, and socialize informally with one another in a supervised setting. While these activities provided opportunities for youth to build some skills on their own, the skill-building was not facilitated or enhanced much by the adults in the room (i.e., these were not intentional learning environments). Other Findings Related to Quality Observed Quality Was Higher in the Spring Than in the Fall Interactions among youth and staff were slightly more positive at the end of the program year than near the beginning. In addition, observers saw more evidence of higher-order skill-building, regardless of the mix of activities observed, although the extent of such opportunities remained quite limited overall. The small improvement in observed interactions may reflect the fact that youth and staff had had time to get to know each other better, and staff may have relaxed somewhat in their approach to leading activities, leading to warmer interactions overall. The difference in interactions may also be explained by the departure of youth who did not get along well with peers or staff between the fall and spring rounds of observations. To the extent that embedding opportunities for higher-order skill-building into activities can be challenging, staff may have gained confidence over the year and stepped up their efforts to make mastery-oriented activities more challenging. Program Cost Was Inversely Associated With Measures of Program Quality Relationships between cost and the various measures of quality developed for the study were consistently negative. These findings run counter to expectations that programs spending more than the vi

9 typical per-youth cost estimates circulating in the after-school literature will be of higher quality. They are particularly surprising given the relationship between certified teachers and measures of skillbuilding and mastery and the fact that it is often more expensive to hire certified teachers than other types of staff. In fact, the presence of certified teachers on staff was itself inversely associated with program costs per youth. One hypothesis based on this finding is that the following trade-off exists between location, cost, and the quality of program content: school-based programs have lower facilities costs and a ready pool of enrollees, and can therefore afford to hire teachers at a relatively low cost per youth, while community-based programs overhead costs and higher recruiting costs and challenges make such staffing strategies less affordable. There Were Differences Among Sites in Overall Levels of Quality Analyses of differences in the observation measures for youth interactions, staff-youth interactions, mastery orientation, and higher-order skill-building indicate that some sites are consistently high and others consistently low in quality across these domains. Two sites received notably high index scores on three of the four observation measures compared to other sites, with both receiving particularly high scores for youth interactions and mastery orientation in addition to one other area. Two other sites received notably low scores in at least three domains compared to other sites. Both of these sites received particularly low scores for staff-youth interactions, mastery orientation, and higher-order skill-building, and one scored lower than other sites on all four measures. Youth Engagement and Satisfaction A central hypothesis in the After School Strategy s theory of change is that youth will find higher-quality programs more interesting, fun, and engaging, and consequently will attend more often. The study assessed youth engagement during activity observations, and asked youth survey respondents to report their overall satisfaction with their programs. Youth Were On-Task and Appeared To Be Enjoying Themselves in Most Observed Activities, Particularly in Arts Activities and Activities with Smaller Group Sizes Overall, youth were fairly engaged in most observed activities, and rarely were disruptive to the point that such behavior affected the overall climate of the activity. Youth were somewhat more engaged in arts activities and less engaged (although on-task) during homework time and other academically-focused activities. In addition, activities with smaller group sizes had higher youth engagement index scores. Most Youth Liked Their Programs, and Youth Who Offered More Positive Assessments of Peer and Staff-Youth Interactions Also Reported Higher Levels of Satisfaction Survey responses indicate that nearly two-thirds of youth really like their programs, and nearly one-third sort of like their programs. Few youth survey respondents indicated that they did not like attending their after-school programs. Younger youth reported somewhat higher levels of satisfaction than older youth. In addition, there was a strong positive relationship between youths vii

10 perceptions of the quality of peer and staff-youth interactions and their overall satisfaction with their programs. Youth satisfaction was not associated with site features such as the presence of certified teachers on staff, cost, or location. Likewise, none of the five measures of observed quality was a significant predictor of youths self-reported satisfaction. Associations Between Quality and Utilization School-Based Sites and Sites Offering Fewer Than Five Hours Per Week of Enrichment Had Higher Utilization Rates Than More Enrichment-Oriented Programs in Other Settings Sites offering more hours of enrichment per week seemed to have slightly lower utilization rates than sites offering fewer enrichment hours, controlling for program location (school- or communitybased). This suggests that youth may prefer more down time after school, and are more likely to attend recreation-oriented programs. School-based sites had higher utilization rates, suggesting that youth are more likely to attend programs that are easier to get to after school. There were no clear relationships between utilization and other site features or measures of quality based either on observations or youth survey responses. The analysis of relationships between site features, observation measures, and utilization was constrained by the small number of sites in the sample, and the association between location and utilization was very sensitive to the treatment of two sites with very different utilization rates from the other six. Associations Between Quality and Youths Perceptions of the Benefits of Participation There Were Few Clear Relationships Between Site Features, Measures of Observed Quality, and Youths Perceptions of the Benefits of Participation The youth survey included 18 items asking youth whether they had learned to do something really well in their after-school programs and whether their programs had helped them in school, socially, with decision-making skills, or with career or college exploration. The average Youth Survey Program Impact Scale score was 8.9 out of 18 total possible points, which means that, on average, youth identified between eight and nine areas in which the after-school program had helped them to learn or improve in some way. Nearly all respondents reported that their after-school program had helped them learn to do something other than academics really well, and almost all youth survey respondents indicated that they had learned to work together with others at the program. Nearly two-thirds of youth indicated that they had learned to do fine arts and to speak or perform in front of a group at their programs, and almost half reported learning to act, dance, or sing. Most youth reported that the program had helped them to get their homework done and to do better in school, to make new friends, and to learn resistance skills. Many youth reported that the program had helped them to enjoy school more or to enjoy reading more, and to avoid getting into fights. Despite the dearth of programming in this area, the majority of youth also reported learning about jobs or careers or about college in their programs. viii

11 There were no significant variations in youths overall self-reported program impact based on respondents age gender, or length of participation, nor were there any significant differences in these scores based on site features or measures of observed quality. The lack of association between measures of observed quality and youths assessment of the benefits of participation persisted in analyses of individual survey items. While site features were associated with a few items, no clear patterns of association between site features, observed quality, and youths views about how their programs have helped them emerged from these analyses. The lack of association between measures of observed quality and youths assessment of how the program has helped them suggests that outside views about program quality have little bearing on what youth believe (or at least report) about the benefits of participation. This study did not collect data to explore whether observed quality is associated with third-party assessments of youth outcomes. The findings based on youths reports therefore should not be interpreted as evidence that quality is irrelevant to program impact. ix

12 1. Introduction This report describes findings from the first year of a two-year study of program quality in eight after-school programs supported by Baltimore s After School Strategy. The findings are based on activity observations conducted in fall 2002 and spring 2003, as well as youth surveys administered in fall Baltimore s After School Strategy Baltimore s After School Strategy is a collaborative, citywide initiative led by the Safe and Sound Campaign (SAS), the Family League of Baltimore City (FLBC), and The After-School Institute (TASI) to expand and improve the quality of after-school programs for children and youth in Baltimore City. The Strategy is one element of the city s Urban Health Initiative, funded by the Robert Wood Johnson Foundation and spearheaded by the Safe and Sound Campaign to improve the health and safety of children in Baltimore City. Launched in early 1998, the After School Strategy has directed more than $29 million from a combination of private and public sources to support (1) the development of an after-school system and infrastructure in Baltimore, and (2) expansion and quality improvement in more than forty afterschool programs operating in more than one hundred sites around the city (Evaluation Overview, 2003). To date, the Strategy has funded over 40 provider organizations operating more than 80 afterschool program sites. Figure 1 summarizes the core elements of the After School Strategy (Halpern & Horton, 2003). The goal of improving program quality is integral to the Strategy s theory of change as both a means of drawing more youth to participate in after-school programs and for increasing the likelihood that programs will affect outcomes for participating youth. As Figure 1 illustrates, many of the Strategy s core components the development of standards and quality control mechanisms, the formation of a training and technical assistance organization, linking funding and accountability to quality improvement relate to quality. The In-Depth Study of After-School Program Quality was designed to inform the After School Strategy s efforts to assess and improve program quality by examining quality in a subset of grantees and refining a systematic approach to quality assessment based on structured observations of program activities. 1

13 Figure 1 Principal Elements of Baltimore s After-School Strategy Adapted from The Design and Implementation of Baltimore s After-School Strategy: An In-Process Look, by Robert Halpern and Carol Horton (May 2003) Identifying and seeking out new funding sources Developing and promoting quality standards Linking program funding to commitment to move toward and ultimately achieve standards Using a fiscal intermediary (The Family League of Baltimore City) Establishing a training, technical assistance and network-building intermediary (The After-School Institute) Implementing an initiative to provide skilled instruction and apprenticeship in arts, sports, and academics (A-Teams) Building capacity to collect and use information for planning, decision-making, quality control, program improvement and other purposes Developing a committee in which stakeholders work together on system-building tasks Deepening the public s, government and elected officials understanding of the value of after-school programs, through a communication and public education campaign Baltimore s Program Quality Standards One of the first activities undertaken by the After-School Strategy team in 1998 was the development of the Standards for Baltimore After-School/Out-of-School Opportunities in YouthPlaces. The Standards are based on the National School-Age Care Alliance s (NSACA) Standards for Quality School-Age Care, with additions designed to (1) ensure that the standards reflect the full age range of children and youth served by after-school programs, and (2) address the quality of program content. The Standards are divided into two parts Organizational Standards and Program Standards (a full list of the Standards is included in Appendix A). The Organizational Standards address the quality of YouthPlaces' overall program infrastructure and socioemotional climate in five sections, all of which apply to all programs: 2

14 Human Relationships Indoor Environment Outdoor Environment Health, Safety, and Nutrition Administration These five categories contain a total of 34 standards, each of which is defined by one or more indicators for a total of 172 indicators. The Program Standards address the quality and richness of program activities and content. They include standards that apply across all program activities, as well as standards specific to each of six program content areas. There are 17 Program Standards, which are defined by a total of 66 indicators. The six program areas articulated in the Program Standards are: Cognitive development Recreation Workforce development Artistic development Civic development Open time Recent reviews of research in child and youth development emphasize the importance of opportunities for skill-building and mastery for positive development (see, for example, Eccles & Gootman, 2002). The Program Standards represented a unique contribution to the emerging field of quality standards for school age care by expanding the focus on content as a dimension of program quality. As written, they provide for somewhat more flexibility of application than do the Organizational Standards, allowing for the kind of diversity in content across programs that is desirable within a community if youth are to have a variety of choices. For example, the Standards do not require programs to offer activities in all six program areas. Instead, they require that all programs provide skill-building and mastery opportunities in at least one non-academic program area (e.g., arts, recreation, civic, employment). All programs are expected to promote cognitive and academic development, however, directly through academic activities as well as indirectly through an infusion of opportunities for cognitive development throughout the program. 3

15 Assessing Program Quality Against the Standards As the fiscal intermediary for the Strategy s grants to providers, the Family League of Baltimore City (FLBC) has promoted use of the Standards through its after-school grant application, contracting, and contract monitoring procedures. For example, applicants are required to conduct selfassessments against the Standards, report the extent to which they meet each standard, and describe priorities and plans for addressing unmet standards. However, the Strategy did not provide guidelines for how programs should conduct these self-assessments. As Halpern and Horton observed in their recent evaluation of the Strategy s implementation, there was no single, specified process that grantees had to adhere to when conducting their initial self-assessments Consequently, there is no way of knowing precisely how each of the grantees determined where it initially stood with regard to the Standards without going back and carefully re-reading each individual set of proposals. 1 Indeed, a review of the proposals submitted by the eight programs included in the In-Depth Study of Program Quality yielded little or no information about the self-assessment processes in most of these programs. FLBC Contract Managers responsible for monitoring each funded program assess Standards adherence during periodic site visits. Again, however, the process for assessing whether programs meet the Standards is not defined clearly enough to enable Contract Managers to be confident in the event that their conclusions differ from programs self-assessment results (Halpern & Horton, 2003). Reflecting on the subjectivity of the self-assessment and monitoring processes, one site leader interviewed for this study noted, If they said you didn t meet this [standard], I d say you should have been here yesterday. Another commented, Standards are subjective. I think I ve met the standard, but someone else might think I m not meeting the standard. Program directors and site leaders also commented on the tension between honoring programs individual missions and priorities and satisfying what they interpret individually as the requirements represented in the Standards. As one program director observed, The influx of training and money are great, but we have a mission, history, and culture of our own. Certain things are more important to us. Another commented, There are [lots of] centers under my jurisdiction. They can t be mirror images of each other, they are not all sponsored by Safe and Sound. I have to think about the standards as they apply to a particular situation. Finally, another noted, We need to validate the issue of variety in programming. The problem with the Standards was that there was an expectation that we would all do the same thing. Requiring that programs meet a certain number of standards is okay, but not all standards should be required. These comments suggest that some questions remain among program leaders not only about how to judge whether programs do meet the standards, but also about whether all programs are or should be expected to meet all standards. 1 Halpern & Horton, p.15. 4

16 Finally, any assessment of program quality must consider external factors likely to affect quality. The unstable funding environment for after-school programs and the resulting instability in staffing often constrain programs quality improvement efforts. One program director interviewed for the quality study commented directly on the influence of funding on priorities, noting that certain program elements [related to the Standards] had their day but are not focused on right now, and some are funding-contingent. 5

17 6

18 2. Study Goals and Methods The In-Depth Study of After-School Program Quality is designed to assess quality and explore factors affecting quality in eight after-school programs funded by Baltimore s After School Strategy. The study design reflects the Strategy s interest in developing a more systematic approach to assessing program quality against the Standards. An overview of the theory of change for the study is presented in Figure 2. The theory guiding the study expands on the After School Strategy s theory of change to identify site characteristics and inputs, elements of program and activity quality as outlined in the Standards, and relationships between these program features and utilization and youth outcomes. Specifically, the goals of the study are to: Develop a systematic and valid approach to assessing after-school program quality across the range of program features included in the Standards for After-School Opportunities in Baltimore Assess the quality of eight after-school programs funded by Baltimore s After School Strategy against the Standards for Baltimore After-School/Out-of-School Opportunities in YouthPlaces Explore links between quality and program attributes and resources that may affect quality in these eight programs Describe the relationship between quality, program utilization, and youths selfreported experiences and perceptions of the benefits of participating in after-school programs Given the small number of sites included in the sample, this is a case study rather than a comprehensive evaluation of the quality of after-school programs supported by the After School Strategy. Moreover, the study is intended to describe patterns in quality across these eight sites, and does not report on the quality of individual programs. To protect the confidentiality of individual sites, this report identifies sites by letters or numbers, randomly assigned to prevent linking sites to their identifiers based on descriptive data. 7

19 Figure 2 Theory of Change for the In-Depth Study of After-School Program Quality Site Characteristics Activity Characteristics Activity Quality Total budget Cost per youth Location Hours of operation per week Enrollment size Ages served Attendance/Utilization Program Standards Number and types of activities offered Use of certified teachers Type of activity Organizational Standards Number of adults Number of youth Child-adult ratio Organizational Standards Youth Interactions Staff-Youth Interactions Program Standards Mastery Orientation Higher-Order Skill-Building Outcomes

20 Site Selection The study sample includes eight of the more than 40 organizations and 80 after-school program sites funded by the Strategy in fiscal year It is a purposive sample of sites selected to include (1) a mix of large and small, public and private sponsoring organizations; (2) school-based and community-based sites; (3) sites in a variety of neighborhoods targeted by the Strategy; (4) sites serving a mix of age ranges; and (5) sites with enrollment levels of 30 or higher to increase the chances of obtaining a large enough survey sample to allow for site-level analyses and comparisons. 2 A comparison of the study sample and all Strategy-funded sites according to enrollment size, utilization rates, and budget and cost figures is provided in Figure 3. Site Capacity, Utilization, Budget, and Cost Figure 3 Study Sites All Funded Sites Total Sites 8 83 Average Number of Youth Contracted by FLBC to Serve 3 ADA Sept 02 Jan Average Utilization Rate Average Cost per Contracted Slot $3,109 $2,573 Average Cost per Youth $4,648 $4,027 Average Total Program Budget $132,463 $125,017 Average Total Strategy Funds $83,728 $66,690 Average Percent of Budget Represented by Strategy Funds 65% 60% Source: FLBC MIS for September 2002 through April 2003 The sites in this study sample are contracted to serve nearly the same number of youth on average as funded sites overall, reported similar ADA between September 2002 and January 2003, and therefore have the same average utilization rate (just shy of 80 percent). Average costs and budget figures are higher for sample sites than for Strategy-funded sites as a whole, however. The average 2 Sites serving older youth were disproportionately represented in the group of sites excluded for small enrollment levels. As a result, the study sample does not include any sites that focus on high-school age youth. PAL centers participating in a separate study also were excluded to prevent imposing additional burden on these sites. 3 Total contracted to serve represents the total number of youth FLBC expects programs to enroll. 4 ADA as a percent of number contracted to serve. 9

21 total budget for study sites is nearly $7500 higher than the total pool of sites. Study sites received more Strategy funds on average, and those funds represent a slightly larger proportion of their budgets than funded sites as a whole. Likewise, the average cost per slot is a little more than $500 higher for sample sites, and the average cost per youth is more than $600 higher. This reflects the fact that one study site whose budget is particularly large reported very low average daily attendance levels between September 2002 and April Study Timeline and Data Collection Activities The study began in the summer of 2002 and will conclude in the summer of During the summer and early fall of 2002, PSA staff interviewed program directors and site leaders and refined and tested the observation instrument. In late fall 2002, teams of observers from PSA conducted twoday visits to each program. PSA also administered surveys to participating youth in the third grade or higher in separate visits; survey administration typically spanned two consecutive afternoons. In spring 2003, observers conducted a second round of visits over two afternoons in each site. This report draws from the observations and surveys conducted between October 2002 and May 2003, describing the results of the fall and spring observations and the initial round of youth surveys. Observation Instrument and Methods A team of observers visited each site twice during the program year, interviewing site leaders and conducting structured activity observations over the course of two consecutive afternoons. To structure the observation visits, PSA developed a set of data collection instruments and a method for conducting activity observations. The observation instruments include data collection forms designed to capture both program-level information and data about the quality of individual activities. A copy of the observation instrument can be found in Appendix B. This section of the report provides considerable detail about the observation instrument, including an overview of the content, description of the methods, and summary of scoring procedures. While such details generally are relegated to a report appendix, the observation instrument has become a source of considerable interest to the After School Strategy as a potential tool for contract managers, TASI staff, and programs themselves to use for a variety of purposes beyond research and evaluation. Moreover, the observations were central to the study s efforts to assess program quality, and the concept of structured observations is less familiar than other research methods such as surveys and interviews. For all of these reasons, this section of the report is designed to help readers understand the observation instrument and our approach to conducting the observations. 10

22 Overview of the observation instrument. The observation instrument includes a site leader interview guide, program overview form, and activity observation instrument. The Site Leader Interview Guide includes questions designed to capture basic descriptive information about the program s goals, activity schedules, and staffing, as well as site leaders perspectives about the quality of their program and implementation issues affecting quality. Site visitors record information about program facilities and activity schedules on the Program Overview Form. The heart of the site visit is a set of activity observations using an After-School Activity Observation Instrument (AOI). The AOI was structured to provide site visitors with a framework for thinking about and recording what they see in ways that yield quantitative as well as qualitative data about the quality of program activities. The AOI builds on the Standards for After-School Opportunities in Baltimore and is adapted from existing observation instruments developed for assessing school age care programs and school classrooms and instruction. The instruments consulted during the development process include the NSACA Program Observation Tool, a draft of the instrument developed by Public/Private Ventures for the evaluation of the Extended-Service Schools Initiative, the School Observation Measure (Smith & Ross, 1999), and the School-Age Care Environment Rating Scale (Harms et al, 1996). The quality of each observed activity is recorded using an Activity Observation Coding Form, which asks observers to mark whether specific activities, behaviors, and opportunities occurred consistently during each observation. These activities, behaviors, and opportunities are grouped into six categories, each of which contains a list of quality indicators (see Figure 4): Youth Interactions Staff-Youth Interactions Opportunities for Skill-Building and Mastery Youth Engagement Activity Organization Activity Setting and Resources Most indicators in the Activity Observation Coding Form describe positive behaviors or activities, but as Figure 3 illustrates the instrument also includes negative indicators. Observers mark positive and negative indicators if they see consistent evidence of them during the course of observation. Some indicators do not apply unless a particular event occurs (these are in italics in Figure 4). For example, observers may not have an opportunity to assess how staff handle serious conflicts among youth if such conflicts do not occur. Likewise, an activity may not include materials, or there may not be more than one staff member present, so ratings of materials or staff interactions are irrelevant. Observers mark indicators of this type NA if the triggering event does not occur or the precondition does not exist. 11

23 The AOI also includes an Activity Context Coding Form for capturing basic descriptive information about each activity observed. This form includes space to briefly describe the activity and a set of checklists of activity types, types of spaces, materials used, number of staff/adults, grade levels served, number of youth, group size(s), and grouping strategies. The AOI is a living document that has been refined over the course of pilot tests and between rounds of data collection. The instrument was piloted in early fall of 2002, and the observation team made revisions based on this initial pilot. In addition, site visitors conducted an extra day of observations in the first three site visits conducted for the study during fall of 2002 for purposes of reliability testing. Rather than overwhelming activities with a team of five observers, we conducted paired reliability tests in which every observer was paired with every other observer for two activity observations over the course of these three site visits. Some indicators were dropped after the fall 2002 round of observations and a few others added to capture information that observers recorded in their notes as missing from the form and important to their overall assessments of quality. Additional reliability checks occurred prior to and during the spring 2003 round of observations, although weather-related program closures precluded a full round of tests. Inter-observer agreement based on the fall and spring tests was 86 percent overall across all indicators in the instrument. This means that pairs of observers marked indicators in the same ways 86 percent of the time. Agreement varied slightly across the rating categories, from 83 percent for Youth Interactions to 87 percent for Skill-Building and Mastery and for Youth Engagement. Observation methods. Each observation visit occurred over two consecutive afternoons, and included an interview with the site leader, a tour of the program facilities, and a set of activity observations. Site visitors stayed in each activity for about 25 minutes long enough to conduct two ten-minute observation segments using the Activity Observation Coding Form and to record additional descriptive information about the activity using the Activity Context Coding Form. Site visitors observed as many different program activities as possible given programs hours of operation and activity schedules 12

24 Figure 4 Activity Observation Form Categories and Indicators Youth Interactions Youth: + Generally have warm, friendly interactions with each other + Cooperate/share with each other + Listen to each other/treat each other with respect + Discuss/resolve disagreements/disputes constructively - Tease each other in a clearly unfriendly way - Threaten/bully/intimidate each other Staff-Youth Interactions Staff: + Are somewhat engaged with youth + Are highly engaged with youth + Have warm, friendly interactions with youth + Engage informally in conversations with youth + Use positive language and tone of voice with youth + Listen actively and attentively to youth +Praise/encourage youth + Use positive behavior management techniques + Help youth to resolve disagreements constructively - Have few or no interactions with youth - Appear bored or distant - Use harsh disciplinary methods - Belittle youth or embarrass them in front of peers - Leave youth unsupervised - Are unaware of teasing/bullying or other serious conflicts among youth Youth: + Interact constructively/positively with staff + Seek out positive contact/interactions with staff - Ignore directions/tune staff out - Are rude/actively negative toward staff Youth Engagement + Youth are actively engaged, focused, interested, enjoying the activity - Youth are disruptive and/or out of control Opportunities for Skill-Building and Mastery Staff: + Are clearly focused on instruction/helping youth to master a skill or learn something new + Provide direct instruction/lecture/demonstrate/give directions + Engage youth in brief question/answer exchanges + Ask youth why, how, and what if questions that require complex answers + Facilitate discussion among youth + Critique/offer constructive feedback to individual youth - Criticize youths work without offering guidance - Discourage appropriate/reasonable questions/discussion/ requests Youth: + Write (not for homework) + Read or are read to (not for homework) + Work cooperatively with each other to achieve a goal + Work on projects with culminating products or events + Choose what or how they do something or help determine the direction of an activity + Lead activities or groups of peers + Tutor/mentor other youth + Think strategically/analyze/solve complex problems + Use information to accomplish a goal/make a decision + Participate in structured discussions Activity Organization + Instructions are clear/youth know what to do + Activity is appropriate for youths skill levels + Staff have prepared/assembled needed materials + Staff work well together Activity Setting and Resources + Materials/equipment are functional/in reasonably good condition + There are enough materials + There is enough room for the activity + External interruptions/distractions are minimal and/or manageable - Space is unsafe for the activity - Other safety concerns 13

25 Observers selected activities based on both the breadth and depth of programming across the program areas defined in the Standards (e.g., academics, arts, sports, open time, etc.). In programs offering a wide variety of activities across multiple program areas, observers selected as many different types of activities as possible. In programs offering many activities in a particular program area and a few in others, observers accordingly selected a larger proportion of activities from the program area most heavily represented in the schedule, while making an effort to observe the outliers as well. Observers also attempted to visit activities for all age groups served, although it was not always possible to see every type of activity for every age group. Although the observation team attempted to conduct the same number of activity observations in each site, the number of observations conducted in each site ultimately varied from 20 to 27. Programs hours of operation vary even from their written schedules, with some open 2 hours and others 2 ½ or 3 hours each afternoon, which affected the amount of time available for observations. The fluid nature of program schedules also affected the actual number of observations conducted each day. It was not uncommon to encounter activities cancelled due to staff absence or poor attendance. To the extent possible within the study budget, the observation schedule was adjusted in an effort to increase the number of activity observations in sites with shorter or more unpredictable schedules. The total number of activities observed in each site during each round of observations is provided in Appendix C. Activity scoring and analysis. Each activity was rated twice over 20 minutes, in two 10-minute observation segments. At the end of each ten-minute segment, observers checked all indicators of quality that were consistently observed during that time period in each of the six rating categories on the Activity Observation Coding Form. These forms were then scored according to the following rules: for each segment, a positive indicator received a score of 0 if it was not marked, and a score of 1 if it was marked or rated NA. Negative indicators were reverse scored a negative indicator received a score of 1 if it was not marked or if it was rated NA, and a score of 0 if it was marked. Reverse scoring therefore credits activities for the absence of negative behaviors. The indicator scores for each of the two ten-minute segments were then added together to generate a total score for the activity, so that each indicator received a total score of either 0 (not marked in either segment), 1 (marked in one segment only), or 2 (marked in both segments). The sum of the indicator scores within each category represents the total quality index score for that category. The mean of the indicator scores within each category represents the mean quality score for that category. The total possible score for each category, based on the total number of positive and negative indicators, is provided in Figure 5. Total index scores vary according to the number of indicators in each category. Mean indicator scores can only vary between 0 and 2, since each indicator received a score of either 0, 1, or 2. 14

26 Figure 5 Quality Index Range of Possible Scores Total Item Score Mean Item Score Youth Interactions Index (5 items) Staff-Youth Interactions Index (11 items) Opportunities for Skill-Building and Mastery Mastery Orientation Index (4 items) Higher-Order Skill-Building Index (7 items) Youth Engagement Index Activity Organization Index (4 items) Activity Setting and Resources Index (6 items) Youth Survey The study used a revised version of the youth survey instrument developed and administered for the Strategy by PSA in The survey included questions about the following topic areas: Youth demographics Risk status of participating youth Program participation (enrollment duration, frequency of attendance) Youths sense of safety in the program Youths perceptions of how program staff treat youth participants Youths perceptions of how program participants treat each other Activity participation patterns Extent of opportunities for skill-building and mastery Extent of opportunities for youth leadership, influence, and service The Standards served as a framework for the survey instrument. To the extent possible, PSA adapted items from surveys developed and used in other major after-school evaluations. PSA developed new survey items to capture information about areas of the Standards not addressed in existing instruments. 5 A copy of the survey and information about survey scales is included in Appendix D. 5 The instruments from which PSA adapted items include surveys from PSA s evaluation of the TASC After- School Program in New York, the Academy for Educational Development s evaluation of the Beacons Schools in New York, and Public/Private Ventures study of the contributions of voluntary youth organizations to healthy adolescent development. 15

27 PSA administered the survey over the course of two to three afternoons in each site to youth age 9 or older who had parental permission to participate in the study, based on lists generated by the Family League MIS. A total of 241 youth completed surveys, which represents approximately 73 percent of the youth of any age who had consent to participate in evaluation data collection, and 53 percent of the number of youth these sites were contracted to serve. 6 Age-specific enrollment levels were not available for these sites, so it is not possible to calculate a true survey response rate for enrolled youth age 9 or older. Other Data Sources In addition to data collected specifically for the study, PSA obtained data from the Family League of Baltimore City s management information system (FLBC MIS) about program budgets, enrollment levels, and average daily attendance. All FLBC MIS data used in the analyses for this report are for the program year; attendance and utilization figures were available for September 2002 through January This estimate of a survey response rate is based on November 2002 enrollment data from the FLBC MIS. 16

28 3. Overview of Participating Programs: Program Structure and Activity Schedules The eight programs included in the study illustrate the diversity of after-school programs funded by the Strategy. They are sponsored by large and small agencies, some public and some private, and offer a wide variety of activities. This chapter provides an overview of key characteristics of participating programs sometimes referred to as structural program features. Structural program features establish the setting and context for positive relationships and high-quality activities, and include staff qualifications and support, program size and group configuration, financial and physical resources, and a program s external affiliations. 7 The structural program features described in this chapter include sponsorship, location, and affiliation with schools; enrollment levels; budgets and costs per youth; and staffing. This chapter also describes the types of activities offered and activity schedules across the eight study sites. These data enable a rough assessment of the mix of academic and non-academic opportunities represented in these programs schedules, and the breadth and depth of opportunities available across program areas. Sponsorship, Location, Facilities, and School Affiliation Figure 6 provides a summary of programs structural features, including the type of organization that sponsors the after-school site, location of services, and school affiliation. Two of the study sites are sponsored by large public agencies, four are sponsored by large private organizations, and two are sponsored by small private organizations, as defined by the Family League. Five operate in community-based settings, and three are located in schools, including one elementary school, one middle school, and one private school. One community-based site is strongly affiliated with a school, offering support to students at school during the school day, although the after-school program itself operates in another location. These programs operate in facilities used for a variety of other purposes in addition to afterschool activities. The three school-based programs have access to all available facilities in their buildings. The five community-based programs operate in a variety of facilities, including two recreation centers, a community center, a facility owned by a private foundation, and an arts building. 7 Marzke et al,

29 Figure 6 Summary of Structural Program Features Number of Youth Contracted Actual Cost Per Youth (Based on ADA) Study Site Sponsorship Location and School Affiliation Total Budget Use of certified teachers to Serve Contracted Cost Per Youth ADA Grade Levels Served A Large private Community $258,704 No 30 $8, $10, B Large private Community $217,530 Yes 40 $5, $6, Number of staff C Small private Community $149,983 No 45 $3, $3, N/A D Large public Community $127,435 No 55 $2, $3, E Small private Community $108,500 Yes 40 $2, $9, F Large private School $97,438 Yes 125 $ $ G Large public School $59,416 Yes 60 $ $1, H Large private School $40,701 Yes 60 $ $ Average $132, $4,

30 Most programs have adequate indoor space that includes classrooms or meeting rooms as well as other specialized space. Six of the eight programs have access to computer labs, and six have some form of indoor recreation area or gym. One program located in an arts building has access to a dance studio, art room, and several rehearsal and performance spaces. As is typical for urban after-school programs, study sites generally lacked access to safe, inviting outdoor spaces for youth to play games or sports. In an effort to give youth some time outdoors, one program allows participants to play on the sidewalk outside the building, although it is located on a very busy road; older youth play on a vacant lot across the street that is strewn with broken glass and sloped downhill so that balls continually roll into the road. One program must cope with limited indoor space as well as lack of access to outdoor space all activities are confined to one of two rooms. Among the activities observed for the study, 60 percent occurred in a classroom or general activity room, 9 percent in a computer lab, 7 percent in a gym, 4 percent in a school cafeteria, 3 percent in some form of outdoor space such as a playground or playing field, 3 percent in a school auditorium, 1 percent in a school library, and 10 percent were in other types of rooms or spaces (e.g., dance studio, teachers lounge/kitchen). Program Budgets and Cost Per Youth Estimates To inform resource allocation decisions, after-school program funders and policymakers need information about the relationship between funding levels and program quality. To what extent do you get what you pay for in after-school programs? To enable exploration of this question, the study obtained data about program budgets and costs from the FLBC MIS for the program year. Program budgets vary widely among the eight study sites, as do the most recent available estimates of programs costs per youth. 8 The average FY03 budget among the study sites was $132,463; budgets ranged from $40,701 to $258,704. The average cost per youth, based on average daily attendance (ADA) for September 2002 through January 2003, was $4,648, with a low of $583 and a high of $10,348. The cost per youth reflects both variations in program budgets and in program ADA. The site with an estimated cost of $583 per youth reported attendance levels well in excess of the total number of youth it was contracted to serve, while the highest-cost site reported an exceptionally low ADA during the fall of The cost figures come from estimates generated by FLBC based on the total number of youth programs were contracted to serve and reported average daily attendance figures for September 2002 through January

31 Staffing Site leaders interviewed for the study attributed the quality of their programs in large measure to the quality of the staff they are able to hire. As one commented, If you get good staff, the rest is not important. Staff is the absolute key The better staff you have, the better the program. According to programs grant proposals, the study sites employ an average of nine full- and part-time staff, although staff sizes range from four to 16. Two sites assign group leaders who stay with a particular group of youth and organize most activities for them. In four sites, staff serve instead as content specialists, and groups of youth rotate through their rooms or spaces for activities in a particular content area. Two sites have both group leaders and activity specialists group leaders stay with one group of youth, and typically supervise homework help, snack, and attendance, while youth also travel to content specialists for certain activities. Five sites employ certified teachers, including all four of the school-based or strongly schoollinked programs as well as one community-based program. Other information about staff characteristics and qualifications is not available for the staff employed in the program year; the last round of staff surveys were administered by FLBC in spring 2002, at the close of the program year. Given the level of staff turnover reported in these programs, it is unlikely that these data serve as an accurate estimate of staff characteristics across program years. Enrollment Size and Ages Served The number and ages of program participants may affect program content, participation patterns, and the quality of interactions among youth and staff. For example, research suggests that the quality of interactions between staff and children is lower in programs with larger enrollment (Rosenthal & Vandell, 1996). Evaluations of after-school programs in New York and other cities have documented consistently lower attendance rates for older youth than for elementary school age children (e.g., Reisner et al, 2002). To explore the relationships between enrollment size, quality, and attendance, PSA obtained data from the FLBC MIS about ages served, enrollment size, and attendance rates among the eight programs included in the study. Overall, the study sites were contracted to serve a total of 455 youth for the program year. The average number of youth that programs were contracted to serve was 57. While the range of enrollment target levels was wide (30 to 125), six of the eight programs included in the study were contracted to serve between 40 and 60 youth. The study sites serve a mix of elementary, middle school, and high school grades in various configurations. Three programs serve elementary grades 20

32 only, one serves middle school students only, two serve elementary and middle school grades, and two are open to youth at all grade levels. Attendance and Utilization Rates The average daily attendance (ADA) for study sites between September 2002 and January 2003 was 49 youth (Figure 7). On the extremes, one site reported that an average of only 11 youth attended the program each day during this time period, while another reported an ADA of 167. Compared to the total number of youth the study sites were contracted to serve, these attendance levels translated into a mean utilization rate of 0.78 (i.e., 78 percent of contracted slots were filled on any given day), with a low of 0.26 and a high of 1.3. Figure 7 Participation Levels Number of Youth Grade Levels Study Site Contracted to Serve ADA Utilization Rate Served A B C D E F G H Average Source: FLBC MIS September 2002 January

33 Hours of Operation and Activity Schedules The after-school components of the programs included in the study operate on varying schedules for an average of 11 hours per week, with a range of six to 15 hours of operation per week. Five programs are open five days a week for 2-3 hours each day. One program operates four days a week (M-Th) for two hours per day, although its activity schedule is organized into modules of activities offered twice a week (M/W or T/Th). The remaining two programs are open three days a week, for 2 to 2 ½ hours each day. For the most part, youth in these programs have little choice about the activities in which they participate. The programs use one of two models for structuring their activity schedules: group-based assigned schedules and menu-based schedules. Five of the eight programs assign youth to grade-level groups, which rotate through a pre-determined schedule of activities each day. The groups stay together, and are required to keep to the assigned schedule. The schedule of activities varies from day to day to ensure a certain amount of variety over the course of the program week, although homework help or academic skill-building linked to the school day are daily activities in four of these five sites. The remaining three programs allow youth to choose from a menu of activities without regard to age, and no activity is required. Based on the program areas defined in the Standards, we computed the number and average hours offered per week of the following types of activities: homework help, other academic/cognitive enrichment, arts, sports, workforce development, civic development, open time, and other. These figures are based on information obtained in site leader interviews and from a review of program schedules obtained during each observation round. Although the Standards combine homework help with other academic activities in a single cognitive development program area, we separated these two types of activities because homework often predominates other types of academically focused activities in after-school programs. We also created an other category for (1) activities that did not fit the available program areas as defined in the Standards (e.g., modeling), and (2) time periods in program schedules that were not reserved for a particular content focus, but for which group leaders planned a wide range of activities that varied day by day. We computed both the number of different activities offered and an average number of hours per week to describe not only the richness of programming but also the level of exposure youth receive to a particular type of activity. Hours per week for each activity were computed separately for each grade level served and then averaged across grade levels, because in some sites the types of activities available vary somewhat across grades. By calculating hours available for each grade level, 22

34 we describe the activity schedule from the perspective of an individual youth, who, depending on his or her grade level, may not be able to participate in the entire list of activities offered in the program. 9 Academic/Cognitive Development and Artistic Development Were the Most Well-Represented Program Areas The study sites offered an average of seven different kinds of activities per week. The number and titles of available activities varied little between the fall and spring rounds of observations, although the content within these activities varied from semester to semester and, in some cases from day to day. Overall, sites offered more hours per week of academic/cognitive development activities than any other type of activity, even separating homework help from other academically-focused activities (Figure 8). Arts activities were also prevalent, with the remaining categories less well-represented across programs. All sites offered at least some of both academic and non-academic activities as required by the Standards, although one site provided no homework help and minimal academic/cognitive support. Figure 9 illustrates the extent of underlying variation among programs in the hours offered per week of each type of activity. Figure 8 Average Hours of Activities Offered Per Week By Type of Activity, (N=8) Average Hours Per Week Across Grade Levels Other academic/cognitive Arts Homework Other Civic Sports Open time Workforce Source: Program activity schedules and site leader interviews. 9 Since one site offers a large menu of activities that are open to all grade levels, we constructed a rough estimate of the total hours each youth would be able to spend on activities in a given program area given that many of these activities are offered simultaneously. 23

35 Figure 9 Average Hours Per Week of Different Types of Activities By Site (N=8) Average Hours Per Week Across Grade Levels Workforce Open time Other Sports Civic Arts Homework Other academic/cognitive Site Source: Activity Observation Forms, Fall 2002 and Spring 2003 The eight study sites offered an average of almost 3 ½ hours per week of academic or cognitive enrichment activities, compared to a little over 2 hours per week of homework time. Every site offers at least one activity with an academic or cognitive focus for at least some subset of participants. Homework time is consistently available for most or all participating youth in four of the eight study sites. In two programs, time is set aside but youth are not required to work on homework during that time period (i.e., other options are available). In the remaining two programs, there is no time set aside for homework. The study team observed a variety of academically-focused activities in addition to homework, including math, language arts, computer skill-building classes, and a few activities that included recreational reading or tutoring. Arts activities also are prevalent across these programs, which offer an average of 2 ½ hours per week of such activities. While this reflects the influence of one arts-focused site, our observations indicate that three other sites offer either creative or performing arts instruction. Another three sites offer arts and crafts on an ad hoc basis as part of other activities; one site does not appear to incorporate arts activities into its programming. Sports, civic development, and workforce development activities were less prevalent in these programs, and only one program offered much in the way of open time. Four programs provide some form of civic development, for an average of one hour per week. Only four of the eight study sites 24

36 provided opportunities for youth to play recreational sports, for an average of less than one hour per week. One of these fielded sports teams a boys basketball team and a girls volleyball team while the other three programs offered recreational sports and physical games (although in one program this consisted primarily of pick-up basketball, with little or no adult facilitation). The four programs that did not offer sports activities were limited by a lack of appropriate indoor or outdoor space none had gyms or outside playgrounds. One of these programs occasionally allows children to play outside on the sidewalk or in a vacant lot across the street, despite safety concerns, just to provide youth with some time outdoors. Another offers dance on a regular basis, affording participants some opportunity for physical exercise. Open time for hanging out, socializing, playing games, etc. is more difficult to characterize, since it is often combined with other scheduled activities such as snack or homework time (e.g., as an option once youth completed homework). Only one program provided large amounts of open, unstructured time for youth to use as they chose. In the rest of the study sites, snack time was the only real open time available. Workforce development is not a focus across the eight programs included in the study. The average hours in this area are generated entirely by one site, which provides enrichment in this area in the form of youth-run business ventures. This is not necessarily surprising given the age distribution of youth served. Only one program serves older youth to the exclusion of other age groups. That said, the program that places an emphasis on this area serves only elementary school age children. Programs offer other types of activities that could not be characterized by any of the above categories, for an average of about 1 ½ hours per week. In two sites, these other activities vary day by day at the discretion of the group leader. In other sites, they include regularly-occurring activities such as cooking classes, modeling classes, and mentoring groups. A comparison of the types of activities observed across sites based on site features revealed that a larger proportion of observed activities in school-based programs were academic enrichment, sports or physical recreation, and activities that were hard to categorize by program area. Observers were less likely to see homework help, arts, or open time in these programs. Total Hours Per Week of Enrichment Activities Varied Considerably Among Sites In addition to estimating the hours per week of activities offered in each program area, we also estimated (1) the average total hours of enrichment activities offered across program areas, excluding homework time and open time; and (2) the total number of different activities offered within each 25

37 enrichment-oriented program area. The amount of enrichment available varied tremendously among sites, from a low of one hour per week (rounding up) to a high of 16 hours per week (Figure 10). Sites fell into two categories based on this estimate those offering 11 or more hours per week and those offering 4 ½ or fewer hours per week of enrichment. School-based programs offered substantially more enrichment hours per week on average 13 hours per week compared to 3 hours per week in community-based programs. Figure 10 Average Total Hours of Enrichment Activities Offered Per Week By Site, (N=8) 18 Average Hours Per Week Across Grade Levels Site Source: Activity Observation Forms, Fall 2002 and Spring 2003 On average, programs offered two different academic/cognitive enrichment activities per week (not counting homework time), and between one and two arts activities. Programs offered an average of one other type of activity. Relationship Between Program Funding and Amount of Enrichment Provided One possible explanation for differences in the amount of enrichment activities offered is the wide variation in program budgets: programs with less money to spend may offer fewer enrichment options. Figure 11 illustrates the relationship between program budgets and the number of hours of enrichment activities offered per week among the eight study sites. Figure 12 accounts for variations in the number of youth programs are contracted to serve, and illustrates the relationship between 26

38 programs expenditures per youth (based on the number they were contracted to serve). While the sample size is too small to allow statistical comparisons, these figures seem to suggest that, with one notable exception, lower-cost study sites offer more rather than fewer hours of enrichment per week. Program expenditures may buy quality rather than quantity when it comes to enrichment, however, and staff may facilitate considerable skill-building in the context of seemingly content-free activities. We explore this possibility in Chapter 7, which reports the activity observation results related to opportunities for skill-building and mastery. Figure 11 Average Total Hours Per Week of Enrichment Activities By Total Program Budget, (N=8) Hours per Week $- $50,000 $100,000 $150,000 $200,000 $250,000 $300,000 Program Budget Source: Program schedules, site leader interviews, and FLBC MIS 27

39 Figure 12 Average Total Hours of Enrichment Activities Offered per Week By Cost per Contracted Youth, (N=8) Hours per Week $0.00 $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, Cost per Youth Source: Program schedules, site leader interviews, and FLBC MIS 28

40 4. Youth Characteristics, Reasons for Attending, and Other After-School Destinations Although enrollment and attendance data by gender were not available for these programs, survey and observation data suggest that in general, girls are somewhat more likely to participate in the study programs than boys. More than half of survey respondents were girls (59 percent), and on average girls comprised 55 percent of youth in the activities observed by the study team. 10 Thirty-nine percent of youth survey respondents live with both parents, and 44 percent indicated that they live with one parent. Seventeen percent reported living with other combinations of relatives or non-relative adults. Almost all youth survey respondents (94 percent) identified themselves as Black or African- American. School Participation and Performance Overall, youth survey respondents reported getting passing grades and completing their homework on time, and were optimistic about their chances of finishing high school. Most respondents (81 percent) indicated that they had not received any Ds or Fs on their last report card. Many respondents (74 percent) reported getting their homework done on time either always or often, while 20 percent reported sometimes getting their homework done and 7 percent reported rarely or never completing their homework on time (Figure 13). Eighty-two percent of respondents rated their chances of finishing high school as very high (59 percent) or high (23 percent); 10 percent gave themselves a fifty-fifty chance of finishing, and 8 percent reported their chances of finishing high school as very low. Risk Indicators Few youth survey respondents reported engaging in risk behaviors such as theft or substance use. However, 43 percent of youth reported involvement in physical fights in the past year, onequarter of respondents reported that they had hurt someone physically, and 22 percent reported that they had damaged or destroyed property at least once in the past year (Figure 14). Boys were more likely than girls to report that they had hurt someone physically in the past year. Moreover, in contrast to generally positive reports about homework completion and school performance, more than one- 10 This figure for observed activities excludes one site that groups youth by gender as well as by grade. 29

41 quarter (27 percent) of youth survey respondents indicated that they had been suspended from school in the past year. Overall, nine percent of respondents reported that they had ever used tobacco and four percent that they had ever used drugs in the last month. A slightly higher percentage of youth (14 percent) reported drinking alcohol in the last month, and older youth were more likely to report using tobacco or alcohol (Figure 15). Figure 13 How Often Do You Get Your Homework Done on Time? (N=238) Rarely 5% Never 2% Sometimes 20% Always 44% Often 30% Source: Youth Surveys, Fall

42 Figure 14 Frequency of Risk Behaviors More than 5 Times Percent of Youth Times Once Never Been involved in a physical fight (N=230) Been suspended from school (N=229) Hurt someone physically (N=224) Damaged or destroyed property (N=230) Stolen money or things (N=230) Source: Youth Surveys, Fall 2002 Figure 15 Substance Use by Age Percent of Youth Age 9-10 Age Used tobacco (N=211) 4 16 Drunk alcohol (N=207) Source: Youth Surveys, Fall

43 Youths Self-Reported Participation Patterns and Reasons for Attending About two-thirds of respondents (66 percent) had been coming to their after-school program for a year or more, and nearly one-third (30 percent) had been attending for 3 years or more (Figure 16). The youth surveys were administered in the fall, which means that many respondents had participated in prior school years. Figure 16 How Long Have You Been Coming to This After-School Program? (N=238) 3 or more years 30% 3 months to a year 34% 1-2 years 36% Source: Youth Surveys, Fall 2002 The ADA and utilization rates described earlier are a more accurate reflection of attendance patterns in these programs than youth survey data, since surveys are more likely to capture frequent attenders. However, age patterns in reported attendance among youth survey respondents mirrored a commonly reported trend of less frequent attendance among older youth. Three-quarters of respondents ages 8-10 reported coming at least three days a week, compared to 58 percent of youth ages Why Do Youth Attend? Youth survey respondents indicated a variety of reasons for coming to their YouthPlaces (see Figure 17). The youth survey listed nine possible reasons for coming, allowing respondents to select as 32

44 many as applied, and providing the option to write in others. The most frequently identified responses were Staff here are helpful and nice (98 percent) and I have fun here (96 percent of respondents circled this response). Other frequently selected reasons were I learn new things here (selected by 88 percent of respondents) and I am safe here (87 percent). Figure 17 Why Do You Come Here? The staff here are helpful and nice 98 I have fun here 96 I learn new things 88 I am safe 87 Most of my friends are here 67 I get help doing my homework 61 I get to do things here that I do not get to do elsewhere 48 There's nothing else to do 36 Other 30 My parents make me come Source: Youth Surveys, Fall 2002 Percent of Youth Where Do Youth Go After School When They Do Not Attend These Programs? More than half of respondents (53 percent) reported that they typically go to their own home when they do not come to the program, and 7 percent reported going to someone else s home (Figure 18). Although far less prevalent, the next most frequently selected alternative destination was going outside to hang out with my friends (11 percent). Eighteen percent of respondents reported that they also participate in some other organized activity or program after school, such as another recreation center or program (5 percent), a school-based activity or program (8 percent), private lessons or classes (3 percent), a job (1 percent), or a day care center (1 percent). There were no significant differences in after-school destinations by age or gender. 33

45 Figure 18 When You Do Not Come to this Program, Where Do You Go After School? (N=212) My Home 53 Outside to hang out with my friends 11 Stay at school for an activity 8 Someone else's home 7 To another recreation center or after school program 5 Private lessons or classes 3 To a job 1 Day care center Source: Youth Surveys, Fall 2002 Percent of Youth Fifty-seven percent of youth survey respondents reported that during the last month they had never been without adult supervision after school (Figure 19). Among the youth who had been unsupervised after school in the last month, 15 percent reported that this happened less than once a week, and 28 percent reported being unsupervised after school for at least one day a week. The youth survey also asked how many hours youth typically spent after school with no adult present, and 60 percent reported less than one hour (Figure 20). Twenty percent of youth survey respondents indicated that they typically spend 1-2 hours after school without an adult, and 20 percent spend 3-4 hours unsupervised. Younger children were more likely to report spending less than one hour per day unsupervised, and older children were more likely to report spending 2 hours without an adult (Figure 21). Roughly equal proportions of younger and older children reported spending 3 or more hours of unsupervised time after school. 34

46 Figure 19 Last Month, How Often Were You Somewhere After School Without An Adult? (N=237) 3 or more days a week 19% 1-2 days a week 9% Never 57% Less than once a week 15% Source: Youth Surveys, Fall 2002 Figure 20 Hours per Day After School With No Adult (N=227) 4 or more hours 13% 3 hours 7% 2 hours 10% Less than 1 hour 60% 1 hour 10% Source: Youth Surveys, Fall

47 Figure 21 Hours Per Day Without Adult by Age (N=206) Percent of Youth years old years old Less than 1 hour 1 hour 2 hours 3 hours 4 or more hours Source: Youth Surveys, Fall

48 5. Quality of Interactions Among Youth and Staff Child and youth development practitioners and researchers agree that caring, supportive relationships with adults, as well as positive relationships with peers and opportunities to belong, are critical to positive development (Eccles & Gootman, 2002; Gambone et al., 2003). This chapter of the report describes findings from the observations and youth surveys about the quality of interactions among youth and staff in the eight study sites. During the activity observations conducted for the study, observers rated youths interactions with one another and staff-youth interactions, documenting both positive and negative indicators of the quality of these interactions. These ratings are summarized in an Observed Youth Interactions Index and an Observed Staff-Youth Interactions Index. In addition, the youth survey included items about peer interactions and staff-youth interactions, which are summarized in a Youth Survey Peer Interactions Scale and a Youth Survey Staff-Youth Interactions Scale. Standards and Observation Measures Related to Interactions The After School Strategy s Standards for Human Relationships include the following: Staff relate to all youth in positive ways Staff respond appropriately to the individual needs of youth Staff use positive techniques to guide the behavior of youth Advocate support for youth is provided Youth generally interact with one another in positive ways The observation instrument includes two measures of the quality of youth and staff-youth interactions. The Observed Youth Interactions Index is based on five indicators in the activity observation form: Youth generally have warm, friendly interactions with each other Youth cooperate/share with each other Youth listen to each other/treat each other with respect Youth tease each other in a clearly unfriendly way Youth threaten/bully/intimidate each other 37

49 The Observed Staff-Youth Interactions Index includes 11 indicators: Staff are engaged with youth Staff have warm, friendly interactions with youth Staff engage informally in conversations with youth Staff use positive language and tone of voice with youth Staff listen actively and attentively to youth Staff praise/encourage youth Staff appear bored or distant Staff use harsh disciplinary methods Staff belittle youth or embarrass them in front of peers Staff leave youth unsupervised Staff are unaware of teasing/bullying or other serious conflicts among youth As described in Chapter 2, each of these indicators received a score of 2, 1, or 0, depending on whether the indicated behavior or opportunity was prevalent during both, one, or neither of the two tenminute scoring periods. The total index score is computed by adding together the scores for the indicators comprising the index. Group Size and Youth Interactions According to the Standards, the total group size can vary among activities based on the type and complexity of the activity, but should not exceed 30 youth. The average number of youth present in observed activities was 12, and the median group size was 10. Fifty-seven percent of observed activities involved 10 or fewer youth, and 77 percent involved 15 or fewer youth (Figure 22). Group size exceeded 30 youth in only six observed activities (3 percent). These standards are consistent with research in early childhood and after-school care indicating that large child-adult ratios are associated with more negative staff-child interactions (e.g., Rosenthal & Vandell, 1996). While there were variations across sites in the average group size for observed activities, the average did not exceed 30 in any site. Group size was lowest in arts activities, with an average of 8 youth, and highest in homework help activities, which involved 18 youth on average. Of the six observed activities in which group size exceeded 30, five were homework help and one was an academic enrichment activity. 38

50 Figure 22 Group Size (Total Number of Youth) (N=180) More than 30 3% % 10 or fewer 57% % Source: Activity Observation Forms, Fall 2002 and Spring Prevalence of Youth Interactions During the fall round of data collection, observers noted anecdotally for a small number of activities that youth interacted so infrequently they were unable to assess the quality of youth interactions. In some cases, the activities involved youth in a joint endeavor that required them to work together without interacting directly such as an orchestra rehearsal. Other activities with low levels of youth interaction were structured to discourage interactions among youth youth were assigned tasks to be completed individually, and encouraged by staff to stay on task and not to interact with their peers. These activities were excluded from the analysis. PSA added items to the observation instrument for the spring round of data collection to capture more systematically whether youth interacted frequently or infrequently. Observers were still allowed to mark the category as not applicable if youth interactions were too infrequent to form a judgment. These activities were not scored or included in analyses of youth interactions. Otherwise, observers indicated whether youth interactions over the observation period were frequent or infrequent. Youth interactions were frequent in 80 percent of activities observed in spring Observers noted frequent interactions throughout the twenty-minute observation period in 62 percent of observed activities; in 18 39

51 percent of observations youth interactions were frequent during only one of the two ten-minute segments of the observation (Figure 23). Figure 23 Percent of Activities Observed in Spring 2003 in Which Youth Interactions Were Frequent (N=119) Neither segment 20% One segment 18% Both segments 62% Source: Activity Observation Forms, Spring Quality of Youth Interactions in Observed Activities Out of a possible 10 points, the average Youth Interactions Index score for activities observed for the study was 7.6 points, with a low of 2 and a high of 10 points. Youth Interaction Index scores did not vary by the number of youth present in an observed activity, which is not surprising given the relatively small group size for most activities. Youth interactions were typically characterized by warm and friendly exchanges in the course of socializing. As Figures 24 and 25 illustrate, youth interactions were warm and friendly in 87 percent of activities observed, although they were prevalent throughout the observation in only 74 percent of observed activities. Other types of interactions demonstrating specific interpersonal skills-- such as sharing materials or listening respectfully to peers during discussions or when they are talking in front of the group were less prevalent. While there were minor disagreements or conflicts between at least a few youth in many observed activities, such negative interactions rarely persisted to the point of affecting observers assessments of the overall climate. Youth cooperated with each other and 40

52 shared materials or supplies for at least part of the observation period in 67 percent of observed activities, and listened to each other respectfully in 62 percent of activities. Interactions in which youth teased or threatened each other were relatively uncommon (these behaviors occurred in 15 and 7 percent of observations, respectively). Figure 24 Average Frequency of Youth Interactions During a 20-Minute Activity Observation (N=189) Percent of Observed Activities Neither segment One segment Both segments Warm, friendly interactions Cooperate and share with each other Treat each other with respect/listen 2 1 Tease each other in unfriendly way Threaten/bully each other Source: Activity Observation Forms, Fall 2002 and Spring Figure 25 Youth Interactions - Mean Indicator Scores (N=189) Warm, friendly interactions 1.6 Cooperate and share with each other 1.2 Treat each other with respect/listen 1.1 Tease each other in unfriendly way 0.2 Threaten/bully each other Source: Activity Observation Forms, Fall 2002 and Spring

53 The average Observed Youth Interactions Index score varied significantly across the eight study sites. The average site-level Youth Interactions Index scores ranged from a low of 6.6 points in one site to a high of 8.5 points in another (Figure 26). 10 Figure 26 Youth Interactions Observation Index Score by Site (N=189) Index Score Site Source: Activity Observation Forms, Fall 2002 and Spring To explore possible explanations for variations in the quality of observed youth interactions, the study conducted analyses guided by the following questions: What activity features seem to influence the quality of youth interactions? Is the quality of youth interactions influenced by the number of youth participating in the activity? Does the quality of youth interactions vary across different types of activities (e.g., arts, open time, sports)? Does the extent to which activities are mastery-oriented, focused on building higher-order skills (e.g., cooperation, group discussion), or generally more engaging affect the quality of youth interactions? 42

54 What program features seem to influence the overall quality of youth interactions within a program site? Is the quality of youth interactions influenced by the number of youth who attend the program overall? Is the amount of enrichment programming sites offer overall associated with the quality of youth interactions? Is the quality of youth interactions in programs located in schools different from youth interactions in other program settings? Is the quality of youth interactions different at the end of the school year than in the beginning? Bivariate and subsequent multivariate analyses were conducted to examine the relationships among the activity and program characteristics described above and the quality of youth interactions. The relatively small number of observations conducted in each site limited the extent of multivariate analysis possible with the interim observation dataset. In particular, the small sample size limited the number of independent variables allowed in regression analyses, so we conducted separate analyses to examine activity-level and site-level factors that might influence quality. The first analysis examined the joint effects on the Observed Youth Interactions Index score of the type of activity observed (i.e., arts, homework help, academic enrichment, sports, open time, and other ), the number of youth in the activity (group size), the mastery orientation and higher-order skill building scale scores for the activity, the level of youth engagement in the activity, and whether the observation was conducted in the fall or spring round of data collection. Controlling for both activity type and timing of data collection was particularly important for assessing the relative influence of each of these two factors, since the mix of activities observed varied somewhat between data collection rounds. A second analysis focused on the joint effects of the average number of hours per week of activities offered in each program area, the average daily attendance per site, and whether the observations were conducted in fall or spring. The approach described above guided the analysis of all of the observation index scores. The analysis of each observation index was framed by questions about the relationships one might expect to find between that aspect of observed quality and potentially relevant activity features, program features, and time of year. These questions were explored first in bivariate analyses, followed by selected multivariate regression analyses based on the bivariate results. 43

55 Factors Associated with Variations in the Quality of Observed Youth Interactions The quality of youth interactions observed in the spring was slightly more positive than in the fall (Figure 27). The average Youth Interactions Index score increased from 7.2 points in the fall to 8.2 points in the spring. This difference persisted in multivariate analyses controlling for the type of activity, group size, level of youth engagement, and extent of mastery orientation and higher-order skill-building opportunities. It is not necessarily surprising to find more positive youth interactions near the end of the school year, after youth have had time to get to know one another. In the absence of final attendance data for the spring semester, it is difficult to assess whether the change in youth interactions reflected changes in the overall number of youth attending the program and possibly attrition among youth who did not get along with their peers in the program. In at least one site, however, the site director informed observers that an entire group of highly disruptive youth had been asked to leave the program after the fall round of observations. Activities with higher levels of youth engagement were associated with more positive youth interactions. The correlation between the Youth Engagement and Youth Interactions Index Scores was 0.44 (p<.01), and this substantial association persisted in multivariate analyses controlling for the type of activity, extent of mastery and higher-order skill-building opportunities, and timing of the observation. Figure 27 Youth Interactions Observation Index Score by Observation Round (N=189) Index Score Fall '02 Spring '03 Observation Round Source: Activity Observation Forms, Fall 2002 and Spring

56 The Youth Interactions Index score did not differ significantly across different types of activities, nor was it associated with the overall number of hours of enrichment activities offered per week in sites. Although a few observers noted that youth interactions seemed less positive in competitive sports than in other types of activities, these differences were not significant in the analyses conducted for the study. Likewise, there was no association between group size within an activity and the quality of interactions among youth, although there was a small positive association between Youth Interactions Index scores and programs average daily attendance levels. It is not surprising to find that group size was not associated with the quality of youth interactions in observed activities, since there was so little variation in group size across activities and sites and the numbers were generally small. It appears that within the range of attendance levels represented in these programs, the number of youth in the building on any given day also has relatively little influence on the quality of youth interactions. Youths Perceptions of the Quality of Peer Interactions The Youth Survey Peer Interactions Index included the following statements, with which youth could agree a lot, agree a little, disagree a little, or disagree a lot: Kids/youth get along with each other here Kids/youth really care about each other here Kids/youth treat each other with respect here Overall, the average Youth Survey Peer Interactions Scale score was 8.3 points on a scale of 3 to 12, which suggests that on average, participants had modestly positive views of the quality of peer interactions in their after-school programs. Figure 28 describes youths responses to each of the items within the Youth Survey Peer Interactions Scale. The item-specific responses reveal somewhat more mixed assessments of peer interactions among youth. By item, 69 percent of survey respondents agreed (either strongly or a little) that youth at their program get along with each other, 65 percent agreed that youth care about each other, and 59 percent agreed that youth treat each other with respect at their program. However, only 27 to 32 percent of respondents strongly agreed with these statements across sites. These patterns varied across age groups and by gender, with older students and boys offering slightly more positive assessments of peer interactions than younger students or girls. 45

57 Figure 28 How Do Kids Treat Each Other at this After School Program? Percent of Youth Disagree a lot Disagree a little Agree a little Agree a lot Kids here really care about each other (N=231) Kids here get along with each other (N=232) Kids treat each other with respect (N=227) Source: Youth Surveys, Fall 2002 Factors Associated with Variations in Youths Assessments of Peer Interactions Due to the small number of youth survey respondents in four of the eight sites (N<20), we were unable to conduct comparisons of survey responses among individual sites. Comparisons of youths self-reported perceptions of peer interactions based on site features included the following dichotomous variables: (1) whether the site offered five or more hours of enrichment activities per week, (2) whether the cost per contracted youth was $2000 or higher, (3) whether sites employed certified teachers as after-school program staff, and (4) whether the program was school-based. All of these program features were associated with youths assessments of peer interactions in their programs in bivariate analyses as well as multivariate analyses controlling for youth survey respondents gender and age (see Figure 29). 11 Youth in sites that offered five or more hours of enrichment per week reported more positive assessments of peer interactions than youth in sites offering less enrichment. 11 The positive correlations among enrichment hours, employment of certified teachers, and location precluded multivariate analyses using all of these variables in a single regression. 46

58 Youth in sites operating for less than $2000 per youth reported more positive peer interactions than youth in higher-cost programs, mirroring the inverse relationship between cost and enrichment noted earlier in this report. Peer Interactions Survey Index scores were higher in sites that were school-based compared to sites located in other settings. Figure 29 Youth-Reported Peer Interactions Scale Scores by Site Characteristic Moderate or low cost per contracted youth ($2000 or less) 8.9 High cost per contracted youth (More than $2000) 7.5 Moderate or low enrichment (<8 hrs/wk) 7.5 High enrichment (8+ hrs/wk) 8.8 No certified teachers on staff 7.3 Certified teachers on staff 8.8 Community-based 7.5 School-based Youth Survey Peer Interactions Scale Score In addition to comparing youth survey responses based on these site characteristics, the study also examined relationships between observers and youths assessments of the quality of youth interactions. Specifically, analyses compared Peer Interactions Survey Index scores for youth in programs that received notably high average observation index scores with Peer Interactions Survey Index scores in sites that received lower observation index scores (see Figures 30 and 31). 12 Youth survey respondents in four sites that received notably high mastery orientation observation index scores offered more positive assessments of peer interactions than youth in sites that received lower observation scores in these areas (Figure 30). This suggests that youth experience more positive interactions with their peers in programs that offer more mastery-focused activities. The association between mastery orientation and youth-reported peer interactions persisted in multivariate 11 Notably high observation index scores were defined as those with effect sizes of 0.2 or higher. See Appendix E for a more complete discussion of this variable. 47

59 analyses controlling for age, gender, observed youth interactions, and observed level of higher-order skill-building. Youth survey respondents in two sites that received notably high youth interactions observation index scores offered more positive assessments of peer interactions than youth in sites that received lower observation scores in these areas. This suggests some level of congruence between survey- and observation-based measures of youth interactions, although the association does not appear in multivariate analyses including other measures of observed quality such as mastery orientation and higher-order skill-building. Figure 30 Youth-Reported Peer Interactions Scale Score by Observed Mastery Orientation Scale Score Youth Survey Peer Interactions Scale Score High observed mastery orientation score Observation Measure Moderate or low observed mastery orientation score 48

60 Figure 31 Youth-Reported Peer Interactions Scale Score by Observed Youth Interactions Scale Score Youth Survey Peer Interactions Scale Score High youth interactions score Moderate or low observed youth interactions score Observation Measure Staff-Youth Interactions The Standards allow staff-youth ratios to range between 1:10 to 1:15 for activities that involve youth over the age of six. The average staff:youth ratio in the activities observed for the study was well below this standard at 1:7 one adult for every seven youth. Staff-youth ratios were at or below 1:10 in 80 percent of observed activities, and at or below 1:15 in 90 percent of activities (Figure 32). Most activities (58 percent) were staffed by one adult, so these ratios reflect the fact that most activities were attended by relatively small numbers of youth. There was some variation in the average staff:youth ratio across sites, although it never exceeded 1:10. Arts activities had the lowest average adult-youth ratios (1:5), and academic enrichment/literacy activities and open time had the highest (1:9), mirroring patterns in group size across different types of activities. 49

61 Figure 32 Youth:Adult Ratio (N=179) > 1:15 10% Between 1:11 and 1:15 10% <= 1:5 35% Between 1:6 and 1:10 45% Source: Activity Observation Forms, Fall 2002 and Spring Prevalence of Staff-Youth Interactions in Observed Activities In both rounds of observations, observers recorded whether staff were engaged with youth as one indicator of the quality of staff-youth interactions. The observation results for this indicator are an important measure on their own of the extent to which youth have opportunities to interact with adults a fundamental prerequisite to forming meaningful relationships with adults. In addition to receiving separate attention as a measure of the prevalence of staff-youth interactions, this indicator was incorporated into the Observed Staff-Youth Interactions Index score as part of a measure of the overall quality of interactions. Staff were engaged with youth to some degree in nearly all activities observed for the study (95 percent). Rarely did staff spend the entire observation period sitting on the sidelines, passively monitoring youth rather than interacting with them. At the same time, in the fall round of data collection observers noted that there were qualitative differences between the levels of engagement among staff that warranted a more refined measure than engaged vs. not engaged. In the spring round of observations, the study team therefore refined the activity observation form to distinguish between staff who were highly engaged, those who were somewhat engaged, and those who had 50

62 few or no interactions with youth. Staff did not have to interact with lots of youth to be highly engaged. Rather, they had to be consistently engaged with one or more youth, while monitoring and responding appropriately to the needs and behavior of other youth in the room. Staff were considered only somewhat engaged with youth if they spent more time in a passive monitoring role than interacting with youth. These were not mutually exclusive categories observers could mark staff as both highly and somewhat engaged within an activity if there was more than one staff member in the room, for example. Across the activities observed in spring 2003, staff were highly engaged with youth throughout the observation period in 69 percent of observed activities, and highly engaged during one of the two ten-minute observation segments in 12 percent of activities (Figure 33). In 19 percent of activities, observers did not consider staff to be highly engaged with youth. Observers also described staff as consistently somewhat engaged with youth in 25 percent of activities. In 12 percent of activities, observers recorded that staff consistently had few or no interactions with youth. 100 Figure 33 Staff Engagement with Youth in Activities Observed in Spring 2003 (N=81) Percent of Observed Activities Neither segment One segment Both segments Highly engaged Somewhat engaged Few or no interactions Level of Engagement Source: Activity Observation Forms, Spring

63 The average level of staff engagement with youth varied across sites. The clearest way to illustrate site variations in the quality of staff-youth interactions is with mean scores for this indicator. As described in Chapter 2, mean indicator scores account for the extent to which the behavior was observed during neither, one, or both segments on an activity observation. An indicator score can range from 0 to 2 based on the number of segments in which it was marked, so mean indicator scores across activities also vary from 0 to 2. Across all sites, the mean score for the indicator Staff are highly engaged with youth was 1.8, but ranged from a low of 0.5 in one site to 2 or nearly 2 in several sites (Figure 34). Figure 34 Mean Staff Engagement Indicator Scores by Site Spring 2003 Observations (N=81) Mean Indicator Score Highly engaged Somewhat engaged Site Source: Activity Observation Forms, Spring It is reasonable to expect different levels of staff-youth interaction across different types of activities. Youth need time to hang out and socialize with peers in a supervised setting, and do not necessarily expect or want to interact with adults in every activity. In keeping with this expectation, there were significant differences among different types of activities observed in spring 2003 in the level of staff engagement with youth (Figure 35). 52

64 Staff were more often highly engaged with youth in academic enrichment and arts activities, and in activities categorized as other. Staff were less likely to be highly engaged with youth in activities categorized as open time, when youth are socializing, playing games, and generally relaxing. The average score across activities on the indicator, Staff are highly engaged with youth was 1.5 (out of 2); the mean score for open time activities was 0.7, compared to 1.8 to 1.9 for academic enrichment, arts, and other activities. For sports and homework help activities, average scores on this indicator were near the average at 1.6. Likewise, staff in open time activities were more likely to be scored as somewhat engaged with youth and to have few or no interactions with youth than staff in other types of activities. Staff in sports and physical activities also were more likely to have few or no interactions with youth. Many of the sports activities observed for the study were really open time in a gym or other large indoor space, during which youth could choose among a variety of games or sit by the sidelines and hang out. Staff served mostly as monitors in these types of activities. Figure 35 Staff Engagement with Youth Indicator Scores by Activity Type (N=80) Highly engaged Somewhat engaged 1.4 Few or no interactions Mean Indicator Score Arts Academic Enrichment Other Homework Help Sports/Fitness/Physical Open Time/Games 53

65 Quality of Staff-Youth Interactions in Observed Activities Observed staff-youth interactions were positive in general, reflecting both the prevalence of positive and the marked absence of negative interactions. The average Observed Staff-Youth Interactions Index score was 16.9 out of a total possible score of 22, with a range of 5 to 22 points. The frequency with which each positive indicator within this index was observed varied considerably, however (Figures 36 and 37). In 75 percent of activities, staff consistently used positive language and spoke to youth in a positive tone of voice throughout the entire observation period. In 65 percent of activities, observers characterized staff-youth interactions as warm and friendly in both segments of the observation period. However, while staff were observed engaging in informal conversations between staff and youth in 40 percent of activities, such conversations occurred consistently over the entire observation only 22 percent of the time. Similarly, staff offered praise or encouragement to youth in 39 percent of activities, but this type of interaction was consistently evident in only 18 percent of activities. In observation notes and debriefing discussions, observers commented that in many activities, staff used positive language and tone of voice but were not particularly warm or personal in their interactions with youth. Figure 36 Average Frequency of Staff-Youth Interactions During a 20-Minute Activity Observation (N=189) Use positive language and tone of voice Warm, friendly interactions Listen actively and attentively to youth Engage informally in conversation Praise/encourage youth Both segments Appear bored or distant One segment Neither segment Leave youth unsupervised Are unaware of teasing/bullying/other serios conflicts 0 Belittle/embarrass youth Use harsh disciplinary methods Percent of Observed Activities Source: Activity Observation Forms, Fall 2002 and Spring

66 Figure 37 Staff-Youth Interactions - Mean Indicator Scores (N=189) Use positive language and tone of voice 1.6 Warm, friendly interactions 1.5 Listen actively and attentively to youth 1 Engage informally in conversation 0.6 Praise/encourage youth 0.6 Use harsh disciplinary methods 0.1 Appear bored or distant 0.1 Belittle or embarrass youth Are unaware of teasing/bullying/other serious conflicts Source: Activity Observation Forms, Fall 2002 and Spring There was significant variation in the Observed Staff Youth Interactions score from site to site, with average scores per site ranging from 15.3 to 18.5 points (Figure 38). At the indicator level, the frequency with which staff offered praise or encouragement to youth varied significantly among sites. The average score on this indicator overall was 0.6 out of 2. The lowest score was 0.09 in a site that offered relatively few mastery-oriented activities, and the highest score was 1.1 in a site with a rich array of skill-building opportunities. While overall variations among sites in the frequency of informal conversations between staff and youth were not significant at p<.05, one site did stand in marked contrast to the rest on this measure. This site received an average score of 1, compared to the crosssite average of 0.6 around which the other sites clustered. This program is fairly unstructured, with relatively few enrichment activities and lots of open time in its daily and weekly schedule, perhaps allowing more natural opportunities for staff and youth to interact informally and socially than the other more highly-scheduled programs. 55

67 Figure 38 Staff-Youth Interactions Observation Index Score by Site (N=189) Index Score Site Source: Activity Observation Forms, Fall 2002 and Spring The study conducted a similar set of analyses of possible explanations for variations in the quality of observed staff-youth interactions to those conducted for youth interactions. Analyses of factors associated with variations in staff-youth interactions were guided by the following questions: What activity features seem to influence the quality of staff-youth interactions? Is the quality of staff-youth interactions influenced by the ratio of youth to staff in the activity? Does the quality of staff-youth interactions vary across different types of activities (e.g., arts, open time, sports)? Does the extent to which activities are mastery-oriented, focused on building higher-order skills (e.g., cooperation, group discussion), or generally more engaging affect the quality of staff-youth interactions? 56

68 What program features seem to influence the overall quality of staff-youth interactions within a program site? Is the quality of staff-youth interactions influenced by the number of youth who attend the program overall? Is the amount of enrichment programming sites offer overall associated with the quality of staff-youth interactions? Is the presence of certified teachers on staff associated with the quality of staffyouth interactions? 13 Was the quality of staff-youth interactions different at the end of the school year than in the beginning? Bivariate and subsequent multivariate analyses examined the relationships among the activity and program characteristics described above and the quality of staff-youth interactions, with separate analyses of activity-level and site-level factors that might influence staff-youth interactions. The first analysis examined the joint effects on the Observed Staff-Youth Interactions Index score of the type of activity observed (i.e., arts, homework help, academic enrichment, sports, open time, and other ), youth-staff ratio, presence of certified teachers on staff, and whether the observation was conducted in the fall or spring round of data collection. A second analysis focused on the joint effects of the average number of hours per week of activities sites offered in each program area, the average daily attendance per site, and whether the observations were conducted in fall or spring. Factors Associated with Variations in the Quality of Observed Staff-Youth Interactions Time of year was the most persistent factor associated with variations in the quality of observed staff-youth interactions, similar to the findings for youth interactions. The average Observed Staff-Youth Interactions index score was slightly higher in the spring (at 17.6) than the fall round of data collection (with a score of 16.4) (Figure 39). The difference between the fall and spring scores on the Staff-Youth Interactions index was the only significant factor associated with observed staff-youth interactions in multivariate analyses. 13 We were not able to include other staff characteristics, because the most recent staff survey data were from a prior program year and there was subsequent staff turnover in most programs. We used the presence of certified teachers on staff as a proxy for activity-level data about staff qualifications, which were not collected as part of the activity observations. 57

69 . Figure 39 Staff-Youth Interactions Observation Index Score by Observation Round (N=187) Index Score Fall '02 Spring '03 Observation Round Source: Activity Observation Forms, Fall 2002 and Spring Staff-youth interactions did not vary across different types of activities, nor was it associated with the presence of certified teachers on staff, the youth-adult ratio, or average daily attendance. In bivariate analyses, staff-youth interactions were more positive overall in arts, homework help, and academic enrichment activities than in other types of activities. However, the differences in staff-youth interactions index scores across activity types disappeared in multivariate analyses that controlled for the timing of data collection. Observed Staff Youth Interactions index scores were positively associated with the presence of certified teachers on staff in initial analyses, but these differences also disappeared when the timing of data collection, activity type, and youth-adult ratio were included in the analysis. Similarly to findings for youth interactions, neither the staff-youth ratio for activities nor the average daily attendance at the site was a significant predictor of the quality of observed staff-youth interactions. Again, this is likely a reflection of the lack of variation in staff-youth ratios across observed activities ratios were consistently low and well within the guidelines contained in the Standards. 58

70 Youth Perceptions of the Quality of Staff-Youth Interactions Youth offered relatively positive overall assessments of their own interactions with staff and positive perceptions of how staff treat youth. There was more variation, however, in the extent to which youth reported individualized, personal interactions with adults in these programs. To assess staff-youth relationships, the survey included a Youth Survey Staff-Youth Interactions scale comprised of the following statements, with which youth could agree a lot, agree a little, disagree a little, or disagree a lot: The staff here really care about me The staff here always try to be fair The staff let kids/youth have a say in how things work here The staff here expect a lot of me I learn a lot from staff here The staff here tell me when I do a good job The average score on the Youth Survey Staff-Youth Interactions Scale was 20.8 on a scale of 6 to 24, which indicates generally positive views among survey respondents about staff-youth interactions. Looking at youth responses to each statement individually, most respondents agreed (either strongly or a little) that staff care about them (93 percent), tell them when they do a good job (91 percent), and try to be fair (89 percent) (see Figure 40). Most youth also indicated that learned a lot from staff (84 percent) and that staff let youth have a say in how things work in their YouthPlaces (82 percent). Girls and boys responses were similar, and no clear pattern of differences emerged among age groups. The survey also asked several questions about the extent to which youth have developed supportive relationships with one or more adult staff. Specifically, the survey asked youth how often they talk one-on-one with a staff member about what is going on in their lives and whether or not they would talk to a staff member about a serious issue or concern. Youths responses on these items were mixed, and relatively consistent regardless of respondents ages, gender, or length of participation: Thirty-one percent of respondents reported that they talked one-on-one at least once a week with a staff member about things that were happening in their lives, and 26 percent reported doing so almost every day (see Figure 41). At the same time, fully 43 percent of respondents had never had a one-on-one conversation with a staff member about what was going on in their lives. Fifty-seven percent of respondents reported that they would or probably would talk to a staff member about a serious issue, 28 percent said they would not or probably would not, and 15 percent said they were unsure (Figure 42). 59

71 Figure 40 How Do Staff Treat You and Other Kids? Percent of Youth Disagree a lot Disagree a little Agree a little Agree a lot Staff here really care about me (N=228) Staff here always try to be fair (N=228) Staff let kids/youth have a say in how things work here (N=223) Staff here expect a lot from me (N=221) I learn a lot from staff here (N=223) Staff here tell me when I do a good job (N=224) Source: Youth Surveys, Fall 2002 Figure 41 How Often Do You Talk One on One with a Staff Member About What's Going on in Your Life? (N=237) Almost every day 26% Never 43% Once or twice a week 14% Less than once a month 9% Once or twice a month 8% Source: Youth Surveys, Fall

72 Figure 42 Would You Talk to a Staff Member About Personal Concerns? (N=234) No 21% Probably not 7% Yes 41% Not sure 15% Probably 16% Source: Youth Surveys, Fall 2002 Factors Associated with Variations in Youths Assessments of Staff-Youth Interactions There was little association between site features and youths assessments of staff-youth interactions, whether they would talk to staff about a serious personal concern, or whether they had ever talked one-on-one with a staff member. Youth in two sites with notably high scores on the higher-order skill-building observation index offered slightly more positive overall assessments of staffyouth interactions in their programs than youth in the other six sites, although scores for each group were high. In the two sites offering more higher-order skill-building opportunities, the average Staff- Youth Interactions survey index score was 21.8 (out of 22) compared to 20.5 in the other sites. Likewise, youths views about the quality of staff-youth interactions did not differ based on measures of observed youth interactions, staff-youth interactions, mastery orientation, or higherorder skill-building opportunities. This suggests that youths perceptions of staff-youth interactions do not correspond well with observers assessments, and are independent of the extent to which staff are focused on providing skill-building opportunities for youth. However, the low correspondence between survey and related observation measures may also be due to the small and non-representative nature of the youth survey sample in half of the sites included in the study. 61

73 A deeper exploration of relationships between the observation measure for the quality of staffyouth interactions included two additional sets of analyses: (1) comparisons of the observation index to individual survey items asking youth how often they talk one-on-one with staff and whether they would talk to staff about a serious personal matter, and (2) an analysis of correlations among specific indicators that comprise the observed staff-youth interactions index and survey items comprising the corresponding survey scale. The results of these analyses yielded additional evidence of the contradictions between the observation and youth survey results. There is a significant negative correlation between the Staff-Youth Interactions observation index score and (a) whether youth reported ever talking one-on-one with staff and (-0.26, p<.01), and (b) whether youth would talk to staff about a serious personal concern (-0.28, p<.01). These negative correlations are present at the indicator level as well these two survey items are negatively associated with observers assessments of whether staff have warm interactions with youth, use positive language or tone of voice, and listen actively and attentively to youth. There is no relationship between youths reports of talking one-on-one with staff and how often observers saw staff engaged in informal conversations with youth. Moreover, none of the other observation scales were associated with the likelihood that youth had talked one-on-one with staff or would talk with them about personal concerns. This suggests that the qualities that promote individualized and more personal interactions among staff and youth may not be well-captured in snapshot observations. There are no significant positive correlations between survey scale items about staff-youth interactions and positive observation indicators. There are small and somewhat weak correlations between some individual survey items and some of the negative observation indicators, despite the very low incidence of negative interactions during observations. In particular, youths assessments of whether staff cared about them and were fair were more positive in sites with fewer activities in which staff seemed bored or belittled or embarrassed youth in front of their peers. Youth were also more likely to report that staff had high expectations for them in sites with fewer activities during which staff appeared bored or distant. One possible interpretation of these results is that the infrequent incidence of overtly negative interactions during observations is a stronger indicator of youths experiences with staff than positive indicators. In other words, even if most interactions are positive most of the time, it may be that a few negative interactions can make or break youths perceptions of how staff treat youth. 62

74 Summary and Conclusions About the Quality of Youth and Staff-Youth Interactions Taken together, the observation and survey results offer a mixed picture of youth interactions with their peers and with staff. While observers generally rated youth interactions positively, there was not as much interaction as expected in many activities, and there was considerable variation in the average quality of observed youth interactions among sites. For their part, youth were more likely to agree than to disagree with positive statements about how youth treated one another in their programs, but tended to agree a little more often than a lot. Similarly, observers typically recorded positive staff-youth interactions, and on average youth responded positively to survey items about staff-youth interactions. However, observers saw less evidence of personal, informal interactions between staff and youth, and a substantial proportion of youth survey respondents reported that they had never had a one-on-one conversation with a staff member. Analyses of various factors that might be expected to account for variations among sites in measures of peer and staff-youth interactions offer some evidence that: Youths interactions are more positive when youth are participating in highly engaging, mastery-focused activities. Perhaps youth have more opportunities to get into disagreements or other types of negative interactions when they are bored and unchallenged. Observers and youths assessments of peer interactions were both positively associated with mastery orientation. The link between mastery and youths assessments of peer interactions was less marked, but youths views were also higher in sites offering more than five hours of enrichment activities per week. Youths perceptions of peer interactions are somewhat more positive in school-based programs. This result is difficult to interpret, although it is possible that the school hosting these programs have positive expectations and strategies for managing youths behavior toward one another that carry over into school-based after-school programs, while the community-based programs have less well-developed approaches to facilitating positive youth interactions. This result may be an artifact of the small number of sites generating data about site features, however, and thus is perhaps best framed as an implication or a hypothesis worth exploring further rather than a conclusion. It does not necessarily cost more to create an after-school environment characterized by positive interactions among youth and staff. Youths assessments of staff-youth interactions were not associated with program cost. Youths perceptions of peer interactions were actually somewhat more negative in programs with per-youth costs of $2000 or higher compared to lower-cost sites. This is consistent with conventional wisdom about staffing in youth programs that adults with the interpersonal qualifications for creating positive social and emotional climates are drawn to these jobs more by a desire to work with and be around young people rather than by salary, the quality of the facility, or other factors that drive program costs. Again, however, given 63

75 the small number of sites, this result should be treated as a strong hypothesis rather than a firm conclusion. The qualities that influence youths attitudes toward and relationships with staff may not be well-captured in snapshot observations. The survey results are based on nonrepresentative samples of youth at the site level, which may also explain the low correspondence between the survey and observation measures. Nonetheless, these instruments seem to be measuring different aspects of staff-youth interactions. 64

76 6. Extent of Opportunities for Skill-Building and Mastery After-school programs represent one of many potential sources of opportunities for young people to master a variety of skills, including a range of cognitive, interpersonal, and emotional skills required for success in school and in the workplace (Eccles & Gootman, 2002; Forum for Youth Investment, 2001). Advocates of after-school and youth development programs note the potential value to youth of opportunities to explore talents and develop skills that are not necessarily nurtured or rewarded in school or at home. This seems especially important in an era when schools are reducing or eliminating arts and physical education classes in the face of pressure to increase academic instructional time and test scores. At the same time, educators are also turning to after-school programs as a potential source of additional academic support and skill-building for children who are not succeeding in school. In some cases, schools may view after-school programs as a place to provide more school, and in other cases after-school programs may be viewed as opportunities to try alternative strategies for engaging and teaching youth who have become discouraged and disengaged from school. The eight programs participating in this study vary in the extent to which they provide opportunities for skill-building and mastery and in their relative emphasis on academic and nonacademic skill-building. As described earlier, the types and amount of enrichment activities programs offer each week varies considerably. However, it is possible that programs providing few activities labeled in a way that suggests they are enrichment programs nonetheless afford youth with a rich array of opportunities for skill-building and mastery. In a study of community-based programs for youth, Milbrey McLaughlin concluded that successful programs are knowledge-centered, and that a key feature of knowledge-centered programs is the embedded curriculum : The adults within a successful CBO recognize the many kinds of knowledge and skills their youth need to succeed in school and life, and they deliberately try to provide them. Embedded within the organization s programs are activities that build a range of academic competencies and life skills. Youth leaders take every opportunity to extend these skills. (McLaughlin, 2000) This chapter describes the extent to which adults in the activities observed for the study created opportunities for youth to build skills and demonstrate skill mastery, both formally by structuring skillbuilding activities and informally in their interactions with youth. 65

77 Standards and Observation Measures Related to Skill-Building and Mastery The Standards set the expectation that every program will offer some academic support and some skill-building and mastery opportunities in at least one non-academic program area (e.g., arts, recreation, civic, employment). In addition, programs are expected to facilitate literacy and cognitive skill development across the range of activities offered. The absence of explicit skill-building within a particular activity does not mean the activity is low quality youth need opportunities to relax, hang out, and socialize after school. The standard is not that every activity and interaction will be skillfocused, but rather that skill-building opportunities are part of the mix. The activity observation instrument designed for this study assumes that these types of skillbuilding opportunities will take many different forms and that teachable moments can occur any time in any setting. Although the instrument includes indicators of various teaching strategies, activities and interactions that build academic and cognitive skills do not necessarily look like school. The observation instrument includes three measures for skill-building and mastery: the Mastery Orientation Index, the Higher-Order Skill-Building Index, and two indicators of literacy-building opportunities. The Mastery Orientation Index is based on four indicators in the activity observation form: Staff are clearly focused on helping youth to master a skill or learn something new Staff provide direct instruction, lecture, give directions, demonstrate, and/or engage youth in brief question-and-answer exchanges (combined three separate indicators) Staff critique and offer constructive feedback to individual youth Youth work on projects with culminating products or events The Higher-Order Skill-Building Index includes seven indicators: Staff ask youth why, how, and what if questions that require complex answers Staff facilitate discussion among youth Youth work cooperatively with each other to achieve a goal Youth lead activities or groups of peers Youth think strategically or analyze or solve complex problems Youth use information to accomplish a goal or make a decision Youth participate in structured discussions To capture literacy building opportunities, observers also indicated when youth (1) wrote stories, essays, letters, etc. for some reason other than to complete a homework assignment, or (2) read a book or other materials for fun or some purpose other than to complete a homework assignment (or were read to by someone else). 66

78 As described in Chapter 2, for each activity, each of these indicators received a score of 2, 1, or 0, depending on whether the indicated behavior or opportunity was prevalent during both, one, or neither of the two ten-minute scoring periods. The total index score is computed by adding together the scores for the indicators comprising the index. Mastery Orientation Mastery orientation the degree to which activities are designed to help youth master skills was low overall across observed activities. The average aggregate Mastery Orientation Index score for the activities observed for the study was 3.4 points out of a possible total of 8, with low of 0 and a high of 8 points. In 36 percent of activities, observers described staff as clearly and consistently focused on instruction or on helping youth to learn something in the course of their interactions with youth. Staff-centered instructional strategies were more prevalent than other more youth-centered approaches, such as feedback to individual youth or opportunities for youth to produce something or to perform (Figures 43 and 44). For example, staff provided direct instruction in 51 percent of observed activities for at least part of the observation period. By contrast, youth worked on projects in 20 percent of observed activities. In general, observers saw more frequent evidence of staff efforts to Figure 43 Average Frequency of Mastery-Oriented Staff-Youth Interactions During a 20-Minute Observation (N=189) Staff clearly focused on instruction/helping Staff provide direct instruction Staff engage youth in brief Q&A Both segments One segment Staff demonstrate or model a concept/skill Neither segment Youth work on projects with culminating events/products Staff critique/offer constructive feedback Percent of Observed Activities Source: Activity Observation Forms, Fall 2002 and Spring

79 Figure 44 Average Mastery Orientation - Mean Indicator Scores (N=189) Staff instruct, direct, demonstrate, and/or engage youth in brief Q&A 1.2 Staff clearly focused on instruction/helping 0.9 Staff provide direct instruction 0.7 Staff demonstrate or model a concept/skill 0.6 Staff engage youth in brief Q&A 0.5 Staff critique/offer constructive feedback 0.5 Youth work on projects with culminating events/products Source: Activity Observation Forms, Fall 2002 and Spring provide various forms of instruction than of staff critiquing or providing constructive feedback to individual youth. Project-based or performance-oriented activities were relatively uncommon overall, and observers were most likely to see such opportunities in the context of arts activities. The average Mastery Orientation Index Score varied considerably across the eight study sites, from a low of 0.3 to a high of 5.7 (Figure 45). To explore the relative influence of activity type, presence of certified teachers on staff, and time of year on the level of mastery orientation, we conducted a multivariate analysis of Mastery Orientation Index scores that controlled for these activitylevel features. We also explored the relative influence on Mastery Orientation Index scores of site-level features, including the presence of certified teachers on staff, overall enrichment focus (measured by the number of hours of enrichment activities offered per week), average daily attendance, cost per contracted youth, and timing of data collection. These analyses were designed to answer the following questions: Are some types of activities generally more mastery-focused than others? Does the presence of certified teachers on staff influence the degree of mastery orientation in program activities? Do programs that hire teachers do more teaching? 68

80 Does the level of mastery orientation vary depending on the time of year? For example, are programs less mastery-oriented (i.e., more recreational) in spring? Is mastery orientation associated with program cost? Finally, in our observation team debriefings and review of observers notes, we explored the extent to which staff in these programs appeared to embed cross-cutting skill-building opportunities in a wide range of activities. Figure 45 Mastery Orientation Index Score by Site (N=189) Index Score Site 0.3 Source: Activity Observation Forms, Fall 2002 and Spring Factors Associated with Variations in Mastery Orientation Index Scores As might be expected, the degree of mastery orientation varied substantially across different types of activities. In contrast to the Youth Interactions and Staff-Youth Interactions observation measures, the aggregate Mastery Orientation Index score did not change between the fall and spring rounds of data collection. Arts activities, academic activities other than homework help, and activities that were difficult to categorize by program area were more mastery-focused than sports activities, homework time, or open time (see Figure 46). The differences among different activity types were consistent across analyses controlling for the timing of data collection and child-adult ratio, and generally followed expected patterns of variation based on qualitative notes about observed activities. 69

81 Open time generally was structured to provide youth with needed time to unwind, hang out, and have fun rather than to help youth master skills, although these blocks of time did provide opportunities for social development. The low mastery orientation in sports and homework time reflects the approach to these activities in several sites, in which adults played largely supervisory rather than coaching or instructional roles. Often, the goals in the sports activities were to give youth opportunities to get some exercise and blow off steam physically rather than to build particular athletic skills or fitness habits. Likewise, the goal in many homework sessions was to give youth time to complete their homework and an opportunity to seek assistance, rather than a more proactive effort to ensure that youth were mastering the skills targeted by the homework assignments. Overall, while open time, sports, and homework activities provided opportunities for youth to build some skills on their own and thus received some credit on the mastery orientation index the skill-building was not facilitated or enhanced much by the adults in the room (i.e., these were not intentional learning environments). Figure 46 Mastery Orientation Index Score by Activity Type (N=189) Index Score Arts Other academic/cognitive Other Sports/fitness/physical Homework help Open time/games Source: Activity Observation Forms, Fall 2002 and Spring

82 Skill-Building in Arts Activities While many of the arts activities observed were recreational arts and crafts, there were also examples of staff using the arts to provide opportunities for a mix of artistic and other skill development. In one activity, a drama instructor held a debriefing with youth the day after a performance. He offered feedback, praising aspects of each youths performance and suggesting things to work on in preparation for the next one. He also asked youth to reflect on their own experience and performance. In another site, an art instructor discussed the concept of symmetry with youth, and then asked them to draw the person sitting across from them. During the course of the activity, youth made comments about one anothers appearance, and the teacher responded to these remarks with low-key questions and comments about diversity and treating each other with respect. In a session of an African drumming class, an instructor showed youth a video of a drumming performance on an island off the west coast of Africa that was a departure point in the slave trade. He often stopped the video to point out or ask youth to find aspects of the performance that resembled western symphony performances. He also talked about the historical and cultural significance of the location. Youth were deeply absorbed by the video and the conversation. And finally, after a discussion about violence, a staff member in another program asked youth to express their feelings about violence in drawings. Cooking Class: Learning More Than How to Make Doughnuts Many of the activities that were difficult to categorize in a particular program area were creative combinations of recreation and skill development. Two sites offered cooking classes, one as a drop-in activity and another as an integral part of a larger, theme-based curriculum. In one meeting of the drop-in cooking class, a group of middle-school youth were making doughnuts. They learned to follow a recipe, measure ingredients, and work together, taking turns and helping each other with guidance from the teacher. Youth and the teacher socialized with one another as they worked. While it was a pleasant and engaging activity that allowed youth to practice some basic cooking skills, the teacher did not take advantage of the opportunity to engage youth in some practical math problem-solving. There were so many youth participating compared to the available measuring and mixing tasks that dividing portions into several parts to allow more turns could have challenged participants to solve two kinds of problems at once. Even Snack Time Can Offer Skill- Building Opportunities In one site, staff use every opportunity to help youth build employment-related skills. During snack time, youth rotate through a set of jobs, including preparing the snack, serving it, and cleaning up after snack. Youth in one snack period observed for the study seemed to enjoy these roles and to take them seriously. 71

83 In addition to varying by activity type, mastery orientation scores were higher in sites that employed certified teachers. The average Mastery Orientation Index Score was 4.1 in sites that employed teachers, compared to 1.3 for sites with no teachers on staff (Figure 47). The difference held true regardless of the activity type, and taken together these two features activity type and the presence of certified teachers on staff were very strong predictors of Mastery Orientation Index scores. The size of the difference reflects the fact that the presence of certified teachers on staff was positively associated with every indicator comprising the index. During observed activities, staff in these sites were consistently more likely to focus on helping youth to learn something, providing some form of instruction, and offering feedback to individual youth. Figure 47 Mastery Orientation Index Score By Whether Site Employs Certified Teachers (N=189) Index Score Certified teachers on staff No certified teachers on staff Source: Activity Observation Forms, Fall 2002 and Spring In an analysis of site-level features associated with mastery orientation, the presence of certified teachers was again positively associated with mastery opportunities, but the number of hours per week of enrichment activities was negatively associated with mastery orientation. Cost per contracted youth and ADA were not significant predictors of mastery orientation in this analysis. To unpack the puzzling negative relationship between enrichment hours per week and mastery, we explored the influence of the mix of activities offered by replacing the total number of hours of enrichment per week with variables for the number of hours per week offered in each program area (with separate variables for homework help and academic/cognitive enrichment). Activities in sites offering more hours of 72

84 academic activities and activities that were difficult to categorize by program area had lower Mastery Orientation Index scores than sites with fewer hours in these areas. These results are not consistent with the activity-level analyses. In the process of conducting various versions of this analysis, however, it became apparent that enrichment hours overall and by program area are less reliable predictors of mastery orientation than the type of activity observed. Although sites attended by a larger number of youth offered less mastery-oriented activities on average, the youth-adult ratio within activities was not associated with mastery orientation, nor was group size when the type of activity was taken into consideration. It is possible that sites with higher attendance levels are less able to provide as many mastery-focused opportunities for youth due to higher costs for such activities. However, mastery orientation was in fact negatively associated with cost per contracted youth activities in lower-cost sites were more mastery-oriented than activities in sites with high per-youth costs. Finally, the Mastery Orientation Index score did not change between the fall and spring rounds of data collection. Higher-Order Skill-Building Opportunities for youth to participate in cooperative activities, think strategically or solve complex problems, answer thought-provoking questions, or engage in extended discussions with their peers were observed infrequently during the observation periods (see Figures 48 and 49). The total possible score for the seven-item Higher-Order Skill-Building Index was 14 points. The average aggregate Higher-Order Skill-Building Index score for the activities observed for the study was 1.1 points, with a range of 0 to 8 points. 73

85 Figure 48 Average Frequency of Higher-Order Skill-Building Opportunities During a 20-Minute Activity Observation (N=189) Youth work cooperatively to achieve a goal Staff ask youth how/why/what-if questions Youth think strategically/solve problems Both segments Staff facilitate youth discussion One segment Neither segment Youth use information Youth participate in structured discussions Youth lead activities/peers Percent of Observed Activities Source: Activity Observation Forms, Fall 2002 and Spring Figure 49 Average Higher-Order Skill-Building - Mean Indicator Scores (N=189) Youth work cooperatively to achieve a goal 0.3 Youth think strategically/solve problems 0.2 Staff ask youth how/why/what-if questions 0.2 Youth use information 0.1 Staff facilitate youth discussion 0.1 Youth participate in structured discussions 0 Youth lead activities/peers Source: Activity Observation Forms, Fall 2002 and Spring

86 Variation among sites in the average Higher-Order Skill-Building Index Scores was significant, although these types of skill-building opportunities occurred infrequently among observed activities in every site (Figure 50). The same analyses exploring factors associated with variations in mastery orientation were conducted for the Higher-Order Skill-Building index. Figure 50 Higher-Order Skill-Building Index Score by Site (N=189) Index Score Site Source: Activity Observation Forms, Fall 2002 and Spring Factors Associated with Variations in Higher-Order Skill-Building Index Scores There was a small but significant degree of variation among different types of activities in the observed frequency of higher-order skill-building opportunities (see Figure 51). Similarly to mastery orientation, higher-order skill-building opportunities were observed more frequently in academic activities other than homework help, and in activities that did not fit neatly into a particular program area. The analysis of joint effects for site-level factors suggest that activities in sites that offer more hours per week of enrichment activities other than arts, academic enrichment, and sports also offer more frequent opportunities for higher-order skill development. 75

87 Figure 51 Higher-Order Skill-Building Index Score by Activity Type (N=189) 8 6 Index Score Other Other academic/cognitive Open time/games Arts Sports/fitness/physical Homework help Source: Activity Observation Forms, Fall 2002 and Spring Higher-order skill-building scores also were higher in sites that employed certified teachers (Figure 52). The difference between the average Higher-Order Skill-Building Index score for sites that employed teachers and those with no teachers on staff was less pronounced than for the mastery orientation index, but the difference was still significant in activity-level analyses controlling for the type of activity and time of year. Finally, the average aggregate score on the Higher-Order Skill-Building Index was slightly higher overall in the spring than the fall round of data collection. This was true both in activity-level and site-level analyses, although the difference was less significant in the activity-level analysis. In the fall, the average score across sites was 0.8, compared to 1.4 in the spring. This change reflects an increase from fall to spring in this score in two sites; one site received a lower average score in the spring, but it was not a large enough change to offset the increases in the other two sites. 76

88 Figure 52 Higher-Order Skill Building Index Score By Whether Site Employs Certified Teachers (N=189) Index Score Certified teachers on staff 0.4 No certified teachers on staff Source: Activity Observation Forms, Fall 2002 and Spring Literacy-Building Opportunities Overall, opportunities aside from homework to read or write occurred infrequently in the activities observed for the study. Youth had opportunities to read books or other materials not assigned for homework in 15 percent of observed activities. As Figure 53 illustrates, such activities typically did not occur throughout the entire 20-minute observation period. In 12 percent of observations, reading activities were observed during only one of the two 10-minute observation segments; reading activities persisted through the full observation period in only 3 percent of observed activities. Writing activities outside the context of homework help were even less common, observed in only 6 percent of activities. Similarly to reading, in only 3 percent of observed activities did youth have writing opportunities throughout the 20-minute observation period. 77

89 Figure 53 Literacy Skill-Building Opportunities (N=189) Percent of Observed Activities Neither segment One segment Both segments Youth read (not for homework) Youth write (not for homework) Source: Activity Observation Forms, Fall 2002 and Spring Summary and Conclusions About Skill-Building Opportunities Most programs offered some academic assistance as well as activities in at least one nonacademic program area. Arts activities were more common than activities in other non-academic program areas defined by the Standards, and several sites offered an eclectic blend of activities, many of which varied on a daily basis or were difficult to categorize by program area. Overall, the extent of opportunities for skill-building and mastery varied substantially across sites. Taken together, the analyses of factors that might be expected to influence these patterns indicate that: Opportunities for skill-building and mastery varied in expected ways across different types of activities, and the mix of purely recreational and more mastery-oriented activities varied among sites. Intentional skill-building opportunities were most pronounced in arts, academic enrichment activities, and the collection of activities that were difficult to categorize by program area, such as cooking, gardening, and chess club. These types of activities were more mastery-focused than other activities, and academic and other enrichment activities also incorporated more opportunities for higher-order skill development. Open time and sports activities were unstructured 78

90 opportunities for youth to relax, play, and socialize informally with one another in a supervised setting. Skill-building strategies typically resembled traditional, staff-centered instructional practices, which may reflect the ages served by these programs and views about what is possible with elementary school age children. Staff provided direct instruction or demonstrated a particular task or skill, and engaged youth in brief question-and-answer exchanges to test their recollection of basic facts and instructions. Staff rarely engaged youth in more extended discussions based on open-ended questions that asked youth to synthesize information or think critically about and express their own views. Likewise, staff were less likely to offer proactive individual feedback to youth in the course of an activity than to provide group instruction and then respond to individual questions or requests for assistance. Finally, project-based approaches that engaged youth in a cooperative enterprise over more than one activity period or day were less commonly observed, and tended to be concentrated in particular sites prevalent in some sites and all but absent in others. Sites with certified teachers on staff offered more mastery-focused activities and more opportunities for higher-order skill-building than sites that did not employ certified teachers. This may reflect differences in professional orientation among staff who are teachers and staff with other qualifications or experience teachers are simply more likely to extend what they do during the rest of the day into their work in after-school programs. Programs offering more opportunities for skill-building and mastery did not have higher costs per youth. This finding is consistent with the larger pattern of inverse relationships between cost and other measures of quality in these sites. 79

91 80

92 7. Youth Engagement and Satisfaction A central hypothesis in the After School Strategy s theory of change is that youth will find higher-quality programs more interesting, fun, and engaging, and consequently will attend more often. The study assessed youth engagement during activity observations, and asked youth survey respondents to report their overall satisfaction with their programs. Youth Engagement in Observed Activities To capture the level of youth engagement in observed activities, the activity observation instrument included two indicators marking the possible extremes: (1) youth are actively engaged, focused, interested, enjoying the activity; or (2) youth are disruptive and/or out of control. Observers could mark either of these indicators, or, by leaving both blank, indicate that youth were neither actively engaged nor were they disruptive or out of control. Youth Were Consistently Engaged in Two-Thirds Of Observed Activities The average Youth Engagement Index score for observed activities was 3.5 out of a possible 4 points, indicating that youth were fairly engaged much of the time and rarely disruptive to the point that such behavior affected the overall climate of the activity. Overall youth engagement levels were similar across observation rounds. As Figure 54 illustrates, in 69 percent of activities youth were engaged throughout the twenty-minute observation period. Youth engagement was less consistent in 19 percent of activities, and was at best neutral in 13 percent of activities. Observers rarely indicated that youth were out of control or disruptive. Most disruptive behavior observed was relatively brief and typically confined to one or two youth. 81

93 Figure 54 Most Youth Are Engaged, Focused, and Enjoying Activity (N=189) Neither segment 13% One segment 19% Both segments 69% Source: Activity Observation Forms, Fall 2002 and Spring The extent to which observers indicated that youth were actively engaged and enjoying activities varied somewhat across sites. The lowest average score on this indicator was 1.1 (on a scale of 0 to 2), and the highest score was 1.9. The low score was at a site offering little variety in activities, while the high score was at a site that provides extensive enrichment opportunities. However, in a multivariate analysis of youth engagement levels, neither the number of enrichment activities overall nor the number of activities offered in specific program areas was associated with scores on this indicator. Factors Associated with Variations in Youth Engagement in Observed Activities Activities with smaller group sizes and more positive youth interactions had higher youth engagement index scores than activities with more participants and less positive youth interactions. These factors were the only significant predictors of positive youth engagement in multivariate analyses controlling for the type of activity, round of observation, and other observation measures. The association between youth engagement and group size is interesting given the relatively narrow range of variation in this feature across observed activities and lack of statistical association with other measures of quality. Observers remarked in debriefing discussions that more youth seemed bored or at loose ends during open time involving relatively large groups of youth. 82

94 Youth engagement also varied across different types of activities, with somewhat lower levels of engagement in homework time and other academic activities. The mean score for the positive indicator of youth engagement ranged from 1.3 in homework help and 1.4 for other academic activities to 1.8 in arts activities; the mean scores for sports, open time, and other activities clustered around 1.6 (Figure 55). These differences were not significant in multivariate analyses including observation measures for youth interactions, staff-youth interactions, mastery orientation, and higher-order skillbuilding. In practice, observers initially found it challenging to score youth engagement consistently across the wide variety of activities observed for the study. In particular, homework help, academic skill-building, and open time activities represented very different contexts for assessing youth engagement than other types of activities. Observers noted that youth were generally on task and compliant in homework help and academic skill-building activities, but did not show signs of particular enthusiasm for the activity. During open time, youth engagement was a measure of the extent to which youth were able to find something enjoyable to do and did not appear to be bored or at loose ends. Observers commented that youth typically appeared to be enjoying themselves during open time, which generally involved some combination of socializing and playing board games. Figure 55 Positive Youth Engagement Indicator Score by Activity Type (N=187) Mean Indicator Score Homework help Other academic/cognitive Arts Sports/physical Open time/games Other Source: Activity Observation Forms, Fall 2002 and Spring

95 Youth Satisfaction In addition to recording youth engagement during activity observations, the study included an item on the youth survey asking youth how they liked the program overall. In response to this question, 66 percent of youth reported that they really liked the program and 32 percent responded that they sort of liked the program. Only 2 percent indicated that they did not like coming to the program. There was a strong positive relationship between youths overall satisfaction with their programs and their age and views about the quality of peer and staff-youth interactions. Younger youth reported somewhat higher levels of satisfaction than older youth, as did youth who offered positive assessments of peer and staff-youth interactions. These factors remained significant predictors of satisfaction in multivariate analyses controlling for site characteristics, observation measures, and respondents gender, and perceived staff-youth interactions were particularly strong predictors of satisfaction. Youth satisfaction was not associated with site features such as the presence of certified teachers on staff, cost, or location. Likewise, none of the five measures of observed quality were significant predictors of youths self-reported satisfaction. 84

96 8. Program Quality, Youth Engagement, and Utilization The After School Strategy s focus on program quality is based on the assumptions that (1) youth will benefit more from their after-school programs if they attend more often, and (2) youth are more likely to come regularly to programs that offer a variety of high-quality, engaging activities and opportunities. This section of the report assesses relationships between the quality of observed activities, youth engagement levels in these activities, and program utilization rates. The small number of sites poses a challenge to the analysis of factors affecting site-level variables such as program utilization rates, because a single unusual case can exert such a powerful effect on the results. The findings presented in this chapter should therefore be treated as tentative. To address the small sample of sites, we conducted our analyses using activity- and youth-level data by assigning site-level utilization rates to the activity observation and youth survey records. For reasons described earlier in the report, we conducted separate analyses of the observation and survey data. As a result, we can describe the relative influence of various observation-based quality measures on utilization, and the relative influence of the youth survey measures of quality on utilization rates, but we cannot assess the joint effects of observed and youth-reported quality. School-Based Sites and Sites Offering Fewer Than Five Hours Per Week of Enrichment Had Higher Utilization Rates Than More Enrichment-Focused Programs in Other Settings The analysis of utilization in the context of the observation data explored the joint effects of (1) program location (school or community), (2) youth engagement levels in observed activities, and (3) index scores for youth interactions, staff-youth interactions, mastery orientation, and higher-order skill building opportunities in observed activities. Consistent with the hypothesis that programs that are easier to get to will be better attended, school-based programs had higher utilization rates than community-based programs. By contrast, utilization rates were slightly lower in sites for which activities received higher mastery orientation scores and that offered more hours per week of enrichment. Neither youth interactions nor staff-youth interactions index scores were associated with utilization rates. These results suggest that youth enrolled in these eight programs may prefer less skillbuilding and more recreation after school. To explore the sensitivity of these results to the influence of the two outliers (a site with a very high utilization rate and another with a remarkably low rate) we conducted a second analysis that excluded these sites. In this analysis, school-based programs become significant negative predictors of utilization utilization was lower in school-based programs and mastery orientation scores are no 85

97 longer associated with utilization. However, the relationship between hours per week of enrichment activities and utilization rates remained consistent across both analyses. Survey Measures of Peer and Staff-Youth Interactions Were Not Associated with Utilization The survey-based analysis of interactions between quality and utilization included the youth survey peer interactions and staff-youth interactions scales, and controls for youths age and gender. Whether programs were school-based was also a significant predictor in the survey analysis using data from all sites, of course, but no other variables were significantly related to utilization, either positively or negatively. When the two outlier sites were excluded from the analysis, the results again were reversed for school-based programs, and survey respondents age emerged as a significant positive predictor of utilization, but the quality measures remained insignificant. 86

98 9. Youth Perceptions of Program Impact The youth survey included 18 items asking youth whether they had learned to do something really well in their after-school programs and whether their programs had helped them in school, socially, with decision-making skills, or with career or college exploration. This chapter reports aggregate responses for these questions and explores the extent to which youths responses differ based on their age, gender, or length of participation; the features of the programs they attend; or their programs scores on observation-based measured of quality. To capture information about youths perceptions of mastery opportunities, one survey question asked youth to circle whether they had learned to do the following in their after-school programs: Do a sport Act, dance, or sing Play a musical instrument Do fine arts (painting, drawing, sculpture, etc.) Speak or perform in front of a group Work together with others Another question asked youth the extent to which they agreed or disagreed (a little or a lot) that the program had helped them to: Enjoy reading more, finish their homework, enjoy school more, or do better at school Make new friends, avoid getting into fights, or learn ways to say no when someone asks them to do things that are wrong or dangerous Learn about different jobs or careers or about how to get into college Think about the possible good and bad results of different choices To assess youths overall perceptions of the benefits of participating in these after-school programs, we constructed a program impact scale from these survey items. The Youth Survey Program Impact Scale is an 18-item scale based on the number of things youth reported learning to do really well in the program, and the number of statements about how the program had helped them with which youth agreed a lot. 87

99 The average Youth Survey Program Impact Scale score was 8.9 out of 18 total possible points, which means that, on average, youth identified between eight and nine areas in which the after-school program had helped them to learn or improve in some way. There were no significant variations in youths overall self-reported program impact based on respondents age gender, or length of participation, nor were there any significant differences in these scores based on site features or measures of observed quality. Although self-reported program impact as a whole was slightly higher in two sites with notably high higher-order skill-building index scores compared to the other six sites, this difference did not persist in multivariate analyses of relationships among all of the observation measures, the presence of certified teachers on staff, enrichment hours, and respondents age and gender. The lack of association between measures of observed quality and youths assessment of the benefits of participation persisted in analyses of individual survey items. The results of these analyses are described in detail in the remainder of the chapter, and suggest that outside views about program quality have little bearing on what youth believe (or choose to report) about the benefits of participation. Youths Perceptions of Mastery Opportunities in Non-Academic Areas Nearly all respondents (96 percent) reported that their after-school program had helped them learn to do something other than academics really well, and almost all youth survey respondents (90 percent) indicated that they had learned to work together with others at the program (Figure 56). Nearly two-thirds of youth indicated that they had learned to do fine arts (63 percent) and to speak or perform in front of a group (61 percent) at their programs, and almost half (48 percent) reported learning to act, dance, or sing. Forty percent of youth overall reported learning a sport, despite the relative dearth of sports activities involving skill-building observed by site visitors. There were no differences in youths responses to these items based on youth characteristics such as age, gender, or length of participation. Likewise, these survey responses did not vary based on site features, nor was the mastery orientation observation index associated with youths perceptions of mastery opportunities Youth in sites with certified teachers and in high-cost sites were more likely to report learning to play a musical instrument. This is likely explained by the single site offering such instruction. 88

100 Figure 56 What Have You Learned to Do Really Well at This After School Program? Percent of Youth Work together with others (N=222) Do fine arts (N=220) Speak or perform in front of a group (N=220) Act, dance or sing (N=223) A Sport (N=228) Something else (N=240) Source: Youth Surveys, Fall 2002 School-Related Benefits of Participation Eighty percent of youth agreed either a lot or a little that their after-school programs had helped them do better at school (Figure 57). Nearly the same proportion of respondents (77 percent) indicated that their programs helped them to finish their homework. About two-thirds reported that their programs helped them to enjoy school more (69 percent), and to enjoy reading more (67 percent). Younger youth were more likely to report that the program helped them to enjoy reading more, and to indicate that the program had helped them to do better in school. Youths responses did not vary according to whether they reported receiving any Ds or Fs on their last report card, or whether they usually got their homework done on time. Youth in sites offering more than 5 hours per week of enrichment activities were more likely to report that the program helped them to enjoy school more, and youth in high-cost sites were more likely to report that the program had helped them to finish their homework. There were no differences in responses to these school-related items based on whether youth attended school-based programs or programs that employed certified teachers. Likewise, more positive scores on the mastery 89

101 and higher-order skill-building observation measures were not associated with more positive responses on youth survey items related to school performance or enjoyment. Figure 57 Percent of Youth Who Agreed That Their After-School Program Had Helped Them in School or Reading The after school program has helped me to Percent of Youth Do better at school (N=225) Finish my homework (N=228) Enjoy school more (N=220) Enjoy reading more (N=225) Source: Youth Surveys, Fall 2002 Youths Perceptions of How Programs Helped Them Socially Most youth survey respondents (80 percent) agreed that they had made new friends because of the after-school program (Figure 58). In addition, most youth (77 percent) reported that the afterschool program had helped them to learn resistance skills, indicating that they had learned ways to say no to things that I know are wrong or dangerous. More than two-thirds of respondents (68 percent) also reported that the program had helped them avoid getting into fights. Younger youth and youth attending higher-cost sites were more likely to report that their programs had helped them to make new friends, and cost was also positively associated with the likelihood that youth believed the program had helped them to avoid getting into fights. There were no differences in youths responses on these items based on their self-reported involvement in physical fights in the past year. Likewise, observation measures of youth interactions and staff-youth interactions were not associated with youths reports of interpersonal benefits of participation. 90

102 Figure 58 Percent of Youth Who Agreed That Their After-School Program Had Helped Them Socially The after school program has helped me to Percent of Youth Make new friends (N=222) Learn ways to say "No" (N=227) Avoid getting into fights (N=218) Source: Youth Surveys, Fall 2002 Other Self-Reported Benefits of Participation Despite the lack of explicit programming in workforce development, youth survey results suggest that youth view their after-school programs as a source of information about careers or jobs and about college requirements. Nearly two-thirds of youth survey respondents indicated that the afterschool program had helped them to learn about different careers or jobs (Figure 59). Sixty percent of youth agreed that their after-school program had helped them learn about how to get into college. In addition, most youth (88 percent) indicated that their programs had helped them to think about the consequences of different choices. Younger youth were more likely to report learning about careers, but age was not related to whether youth reported learning about college. There were no gender-related differences in youths responses to these items, nor did length of participation make a difference. Youth attending a site with an explicit workforce development component were more likely to agree a lot that the program had helped them to learn about different jobs or careers. Seventy-one percent of respondents in this site agreed a lot, while in other sites the proportion of youth who agreed a lot ranged from 25 to 43 percent. This may also account for the fact that younger youth were more likely to report learning about careers, since this site serves only elementary school age youth. 91

103 Youth in high-cost sites were more likely to report learning about different jobs or careers. Observation measures were not positively associated with youths responses to items related to learning about careers or college. Figure 59 Percent of Youth Who Agreed That Their After-School Program Had Helped Them Think About Consequences or Learn About Careers or College The after school program has helped me to... Percent of Youth Think about the possible results of different choices (N=224) Learn about different careers or jobs (N=224) Learn about how to get into college (N=225) Source: Youth Surveys, Fall

104 10. Summary and Conclusions The In-Depth Study of After-School Program Quality was designed to answer the following questions: What is the quality of these programs assessed against the Standards for Baltimore After-School/Out-of-School Opportunities in YouthPlaces? What is the quality of relationships among youth and staff, as defined in the Organizational Standards? What is the quality of opportunities for skill-building and mastery, as defined in the Program Standards? What are the links between quality and site features that may affect quality, such as cost or staffing strategies? What is the relationship between quality and program utilization? Are youth more likely to attend higher-quality programs? What is the relationship between program quality, other program features, and youths self-reported experiences and perceptions of the benefits of participating in after-school programs? The findings in this interim report are based on data collected during the program year, which included observations of 189 activities in the fall of 2002 and the spring of 2003, and surveys of 241 youth participants the fall of Assessment of Program Quality and Factors Associated with Observed and Youth-Reported Measures of Quality The Quality of Observed Interactions Among Youth and Staff Were More Positive Overall Than Observers Ratings of Opportunities for Skill-Building and Mastery Overall, as illustrated in Figure 60, the quality of youth interactions and staff-youth interactions were more positive and consistent than the extent of opportunities for skill-building and mastery in activities observed across the eight study sites. The average score for indicators of youth and staffyouth interactions was 1.5 on a scale of 0 to 2, compared to average scores of 0.8 for mastery orientation and 0.2 for higher-order skill-building. 93

105 Figure 60 Mean Observation Indicator Scores for Each Quality Index (N=189) Mean Indicator Score Youth Interactions Staff-Youth Interactions Mastery Orientation Higher-Order Skill- Building Youth Engagement Source: Activity Observation Forms, Fall 2002 and Spring Observed Interactions Among Youth Were Generally Positive, While Youth Survey Respondents Offered More Mixed Assessments of Peer Interactions Youth interactions in observed activities were generally positive and typically characterized by warm and friendly exchanges in the course of socializing. Other types of interactions demonstrating specific interpersonal skills such as sharing materials or listening respectfully to peers during discussions or when they are talking in front of the group were less prevalent. While observers recorded very few overtly negative interactions among youth, youth themselves indicated some ambivalence about the extent to which their peers cared about each other, got along well, and treated each other with respect. The majority of youth offered positive responses on these survey items, but were more likely to agree a little than to agree a lot, and a substantial minority offered negative responses to these items. 94

106 The Quality of Youth Interactions Was Associated with Levels of Youth Engagement and the Degree of Mastery Orientation in Observed Activities The quality of observed youth interactions was positively associated with youth engagement levels, and youths assessments of peer interactions were positively associated with the degree of mastery orientation in observed activities. Youth interactions were somewhat more positive in activities marked by high levels of youth engagement. Youth interactions were consistent across different types of activities, however, and youth engagement and time of year explained relatively little of the variation among sites in youth interactions observation index scores. Youth in programs whose activities were notably more mastery-oriented offered more positive assessments of peer interactions. These results suggest that youth have more positive interactions with each other in settings that are highly engaging and mastery-oriented. Observers and Youth Offered Positive Assessments of Staff-Youth Interactions Overall, Although the Extent to Which Staff Provide Individualized Attention to Youth Was Less Clear The quality of staff-youth interactions across observed activities was generally high, reflecting the prevalence of positive interactions and the relative absence of overtly negative interactions among staff and youth. Staff consistently used positive language and tone of voice, and in many activities their interactions with youth were warm and friendly. Informal conversations among staff and youth and individualized attention and encouragement were observed less often than group-based, task-focused interactions. Likewise, youth offered relatively positive overall assessments of their own interactions with staff, and positive perceptions of how staff treat youth. There was more variation, however, in the extent to which youth reported individualized, personal interactions with adults in these programs. In particular, 40 percent of youth survey respondents reported that they had never talked one-on-one with a staff member in their after-school programs. There Were No Clear Patterns of Association Between the Quality of Observed Staff-Youth Interactions, Youths Assessments of the Quality of Staff-Youth Interactions, and Site Features Observed staff-youth interactions were higher in spring than in the fall, but this was the only significant factor associated with variations in this measure across sites. The quality of staff-youth interactions was relatively consistent across different types of activities, and was not associated with site features such as location, cost, or the presence of certified teachers on staff. Similarly, youths views about the quality of staff-youth interactions did not differ based on observation measures of quality, nor were there any associations between site features and youths 95

107 assessments of staff-youth interactions, whether they would talk to staff about a serious personal concern, or whether they had ever talked one-on-one with a staff member. The Qualities That Influence Youths Attitudes Toward Staff May Not Be Well-Captured in Snapshot Observations The results of observation and survey measures of staff-youth interactions did not correspond well with one another. In particular, there was more variation in observers assessments of staff-youth interactions than in youths responses to survey items about how staff treat youth. Moreover, youths responses to questions about more individualized or personal interactions with staff were negatively associated with observation measures. The survey results are based on non-representative samples of youth at the site level, which may also explain the low correspondence between the survey and observation measures. Interviews with youth are likely a better way to assess the qualities of programs and staff that promote positive staff-youth relationships, and may generate more sensitive survey items and observation indicators in the future. The Extent of Opportunities for Skill-Building and Mastery Was Low Overall The degree to which observed activities were designed to help youth master skills was lower overall than the quality of interactions among youth and staff. Likewise, higher-order skill-building opportunities such as cooperative activities and activities that require youth to think strategically or solve complex problems, answer thought-provoking questions, or engage in extended discussions with their peers were observed infrequently during observed activities. In general, when staff were focused on helping youth to learn something, staff-centered instructional strategies were more prevalent than other more youth-centered and individualized strategies, such as providing feedback to individual youth or offering opportunities for youth to produce something or to perform. Measures of Mastery Orientation and Higher-Order Skill-Building Were Higher in Enrichment Activities and in Sites with Certified Teachers on Staff The extent of opportunities for skill-building and mastery varied substantially across sites, and the type of activity and the presence of certified teachers on staff were both strongly associated with ratings of the mastery orientation and higher-order skill-building in observed activities. Intentional skill-building opportunities were most pronounced in arts, academic enrichment activities, and the collection of activities that were difficult to categorize by program area, such as cooking, gardening, and chess club. Academic and other enrichment activities also incorporated more opportunities for 96

108 higher-order skill development. Not surprisingly, these types of activities were more mastery-focused than sports or open time activities. Efforts to build cross-cutting skills such as literacy, critical thinking, or cooperation in the context of arts, sports, or other apparently recreational activities were the exception rather than the rule in observed activities. The presence of an embedded curriculum was difficult to detect in all but a small minority of activities, and was clearly evident program-wide in only one site. In most sites, arts activities targeted artistic skills and products were individual rather than collective, and academic activities generally sought to develop individual youths basic reading and math skills. Open time and sports activities were unstructured opportunities for youth to relax, play, and socialize informally with one another in a supervised setting. While these activities provided opportunities for youth to build some skills on their own, the skill-building was not facilitated or enhanced much by the adults in the room (i.e., these were not intentional learning environments). Observed Quality Was Higher in the Spring Than in the Fall Interactions among youth and staff were slightly more positive at the end of the program year than near the beginning. In addition, observers saw more evidence of higher-order skill-building, regardless of the mix of activities observed, although the extent of such opportunities remained quite limited overall. The small improvement in observed interactions may reflect the fact that youth and staff had had time to get to know each other better, and staff may have relaxed somewhat in their approach to leading activities, leading to warmer interactions overall. The difference in interactions may also be explained by the departure of youth who did not get along well with peers or staff between the fall and spring rounds of observations. To the extent that embedding opportunities for higher-order skill-building into activities can be challenging, staff may have gained confidence over the year and stepped up their efforts to make mastery-oriented activities more challenging. Program Cost Was Inversely Associated with Measures of Program Quality Relationships between cost and the various measures of quality developed for the study were consistently negative. These findings run counter to expectations that programs spending more than the typical per-youth cost estimates circulating in the after-school literature will be of higher quality. They are particularly surprising given the relationship between certified teachers and measures of skillbuilding and mastery and the fact that it is often more expensive to hire certified teachers than other types of staff. In fact, the presence of certified teachers on staff was itself inversely associated with program costs per youth. 97

109 One hypothesis based on this finding is that the following trade-off exists between location, cost, and the quality of program content: school-based programs have lower facilities costs and a ready pool of enrollees, and can therefore afford to hire teachers at a relatively low cost per youth, while community-based programs overhead costs and higher recruiting costs and challenges make such staffing strategies less affordable. However, the small number of sites in the sample, the extreme range of costs represented, and the high degree of correlation among site features related to cost such as location and employment of certified teachers, dictate caution in drawing conclusions or forming recommendations based on this result. There Were Differences Among Sites in Overall Levels of Quality Analyses of differences in the observation measures for youth interactions, staff-youth interactions, mastery orientation, and higher-order skill-building indicate that some sites are consistently high and others consistently low in quality across these domains. Two sites received notably high index scores on three of the four observation measures compared to other sites, with both receiving particularly high scores for youth interactions and mastery orientation in addition to one other area. Two other sites received notably low scores in at least three domains compared to other sites. Both of these sites received particularly low scores for staff-youth interactions, mastery orientation, and higher-order skill-building, and one scored lower than other sites on all four measures. Quality, Utilization, and Youths Assessments of the Benefits of Participation School-Based Sites and Sites Offering Fewer Than Five Hours Per Week of Enrichment Had Higher Utilization Rates Than More Enrichment-Oriented Programs in Other Settings The analysis of relationships between site features, observation and survey measures, and fall 2002 utilization was constrained by the small number of sites in the sample. Sites offering more hours of enrichment per week seemed to have slightly lower utilization rates than sites offering fewer enrichment hours, controlling for program location (school- or community-based). This suggests that youth may prefer more down time after school, and are more likely to attend recreation-oriented programs. School-based sites had higher utilization rates, suggesting that youth are more likely to attend programs that are easier to get to after school. However, this result was very sensitive to the treatment of the two sites with very different utilization rates from the other six. There were no clear relationships between other measures of quality and utilization. Similarly, youth survey measures for the quality of youth and staff-youth interactions were not associated with utilization. 98

110 There Were Few Clear Relationships Between Site Features, Measures of Observed Quality, and Youths Perceptions of the Benefits of Participation There were no significant variations in youths overall self-reported program impact based on respondents age, gender, or length of participation, nor were there any significant differences in the overall measure of self-reported impact based on site features or measures of observed quality. Within specific domains such as school affiliation and success or interpersonal skills: Younger youth were more likely to report that the program helped them to enjoy reading more, and to indicate that their after-school program had helped them to do better in school. Youth in sites offering more than 5 hours per week of enrichment activities were more likely to report that the program helped them to enjoy school more. Youth in sites with a cost per youth of $2000 or higher were more likely to report that the program had helped them to finish their homework. Observation measures of youth interactions and staff-youth interactions were not associated with youths reports of interpersonal benefits of participation. More positive scores on the mastery and higher-order skill-building observation measures were not associated with more positive responses on youth survey items related to school participation, performance, affiliation, or enjoyment. The lack of association between measures of observed quality and youths assessment of how the program has helped them suggests that outside views about program quality have little bearing on what youth believe about the benefits of participation. This study did not collect data to explore whether observed quality is associated with third-party assessments of youth outcomes. The findings based on youths reports therefore should not be interpreted as evidence that quality is irrelevant to program impact. Looking Ahead: Re-Assessing the Likely Benefits of a Second Full Round of Data Collection This report is based on the first phase of the In-Depth Study of After-School Program Quality, which occurred within the context of a single program year and assessed within-year changes in measures of observed quality. The second phase of the study includes plans for (1) a follow-up youth survey to assess changes in youths experiences in and self-reported benefits of participation in their after-school programs, and (2) a final round of activity observations to assess changes in observed 99

111 quality across program years. Additional data about utilization during the program year also would be available for analysis, since these analysis variables were created before the full set of spring utilization figures were available. The new fiscal challenges facing the After-School Strategy, and the likely attrition in the youth survey sample, warrant a reassessment of the potential value added of the second phase of the study. The primary research questions to be addressed by the second phase of the study relate to stability and change in program quality across program years, and would include: To what extent does program quality change across program years? Are changes across years more or less pronounced than changes observed between fall and spring during phase one? What factors are associated with changes in quality? Is staff turnover or changes in whether programs employ certified teachers associated with changes in quality? Are changes in funding levels associated with changes in quality? Do high-quality programs attract higher levels of participation over time? Are utilization patterns over the course of an entire program year associated with program quality? Is program quality in one year associated with changes in utilization in the following year? Do youth offer more positive assessments of the benefits of participation after a year of participation? Potential Benefits and Costs of Assessing Change Across Program Years and Factors Associated With These Changes Assessing factors associated with changes in quality across program years could provide some insight into the extent and nature of the effects of well-known challenges posed by staff turnover to the stability of programs overall. This information could also yield additional information about the potential effects of one particular type of change in staffing, namely the presence of certified teachers on staff, provided there are changes in this feature in enough sites. For example, a finding that staff turnover has little effect on quality could influence how decision-makers approach staffing policies thought to affect turnover, such as pay scales, benefits, and opportunities for advancement. Similarly, additional information about the association between quality and the presence of certified teachers on staff would strengthen the findings from the first phase of the study. 100

112 The small number of sites makes it more difficult to draw firm conclusions about the relationship between site-level features and quality, however, and this constraint may well limit the additional benefits of another round of data collection. Regardless of the number of activity observations included in the analysis, there are only eight sites contributing to variations in site features, which increases the likelihood that an unusual set of circumstances in one program will skew the findings. An additional round of observation data would certainly be required to assess change in quality across program years. However, additional survey data likely would be less useful and therefore less critical, since attrition is to be expected and the sample is already limited. If the focus of youth surveys was limited to youths perspectives on the quality of peer and staff-youth interactions, the survey instrument could be scaled back and treated as a cross-sectional rather than a longitudinal measurement (i.e., there would be no need to track down the same set of youth surveyed in the first phase). However, there is no reason to believe that sites will obtain higher consent rates than last year, and some reason to believe that sites will have less incentive to comply with this funding requirement in the last year of the grant cycle. The costs associated with this set of study questions for phase two could therefore be reduced, although the survey administration costs in phase one were relatively small compared to the costs associated with the observations. Potential Benefits and Costs of Conducting a More Complete Assessment of the Relationship Between Utilization and Quality The small number of sites was most problematic for these analyses in phase one, and would remain a constraint in phase two. However, the utilization data for phase one were limited to a single semester, and included data submitted by sites in the early months of the program year, which typically are less accurate and reliable. Moreover, assessment of relationships between utilization patterns and quality based on a more complete set of data about utilization does not necessarily require additional data collection. The additional data required to assess within-year patterns for the program year should have become soon available after the draft interim report was completed, and could be incorporated into a new set of analyses at any time. An assessment of the lagged effects of quality in one year on utilization patterns in the next year could be conducted once the fall 2003 semester of utilization data become available for the program year. The benefits of such analyses might therefore be more reliable findings at relatively little cost. 101

113 Assessing Relationships Between Quality and Changes in Youths Assessments of Program Benefits Efforts to answer questions related to changes in individual youths perceptions of the program are not likely to add much to the knowledge base. The capacity of the study to address questions about the impact of program quality on youth outcomes was extremely limited from the beginning in the absence of resources to field a comparison group of youth survey respondents. The challenges posed by the limitations of the initial youth survey sample and the anticipated attrition in the follow-up sample all but eliminate the study s already limited capacity to explore these issues. A more effective use of resources toward this end would be to (1) obtain and school records for participants in these eight sites in the program year and a comparison group of youth, (2) obtain participants attendance data from the FLBC MIS, and (3) analyze the relationships between youths program participation ( dosage ranging from none to high), features and quality of programs attended, and average school performance. 102

114 References American Youth Policy Forum. (1997, 1999). Some things do make a difference for youth: A compendium of evaluations of youth programs and practices. Washington, DC: American Youth Policy Forum. Baron, R.M., & Kenny, D.A. (1986). The moderator-mediator variable distinction in social psychology research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 29, Beckett, M., Hawken, A., & Jacknowitz, A. (2001). Accountability for after-school care: Devising standards and measuring adherence to them. Los Angeles: RAND Corporation. Bransford, J.D., Brown, A.L., & Cocking, R.R. (Eds.). (1999). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press. Bronfenbrenner, U., & Morris, P. (1998). The ecology of developmental process. In W. Damon & R. Lerner (Eds.), Handbook of child psychology (5 th ed.). New York: John Wiley. Catalano, R.F., Berglund, M.L., Ryan, J.A.M., Lonczak, H.S., & Hawkins, J.D. (1998). Positive youth development in the United States: Research findings on evaluation of positive youth development programs. Seattle, WA: University of Washington, Social Development Research Group. Eccles, J., & Gootman, J.A. (Eds.). (2002). Community programs to promote youth development. Washington, DC: National Academy Press. Forum for Youth Investment (August 2001), FYI Newsletter, Volume 1, Issue 11. Baltimore, MD: The Forum for Youth Investment. Grossman, J.B., Price, M.L., Fellerath, V., Jucovy, Linda Z., Kotloff, L.J., Raley, R., Walker, K.E. (2002). Multiple choices after school: findings from the extended-service schools initiative. Philadelphia: Public/Private Ventures & Manpower Demonstration Research Corporation. Grossman, J.B., Walker, K., & Raley, R. (2001). Challenges and opportunities in after-school programs: Lessons for policymakers and funders. Philadelphia: Public/Private Ventures & Manpower Demonstration Research Corporation. Harms, T., Jacobs, E.V., White, D.R. (1996). School-Age Care Environment Rating Scale. New York: Teachers College Press, Columbia University. MacDonald, G.B., & Valdivieso, R. (2000). Measuring deficits and assets: How we track youth development now, and how we should track it. In Youth Development: Issues, Challenges and Directions (pp ). Philadelphia: Public/Private Ventures. 103

115 McLaughlin, M.W. (2000). Community counts: How youth organizations matter for youth development. Washington, DC: Public Education Network. Merry, S. (2000). Beyond home and school: The role of primary supports in youth development. Chicago: Chapin Hall Center for Children. National Institute of Child Health and Development Early Child Care Research Network. (2002). Child care structure process outcome: Direct and indirect effects of child care quality on young children s development. Psychological Science, 13, National School-Age Care Alliance. The NSACA Standards for Quality School-Age Care. Boston, MA: Author. National School-Age Care Alliance and the National Institute for Out-of-School Time (1998). Guide to NSACA Program Accreditation: NSACA Program Observation Tool. Boston, MA: NSACA. Pierce, K.M., Hamm, J.V., & Vandell, D.L. (1999). Experience in after-school programs and children s adjustment in first-grade classrooms. Child Development, 70, Pittman, K., Irby, M., & Ferber, T. (2001). Unfinished business: Further reflections on a decade of promoting youth development. In Youth development: Issues, challenges, and directions. Philadelphia: Public/Private Ventures. Policy Studies Associates, Inc. (2001). School Literacy Observation Instrument. Developed for The Qualitative Evaluation of the Achievement First Initiative, Fund for Educational Excellence, Baltimore, Maryland. Policy Studies Associates, Inc. (2002). The TASC After-School Classroom Observation Instrument. Developed for the Evaluation of the TASC After-School Program. Washington, DC: Policy Studies Associates, Inc. The Safe and Sound Campaign (1999). Standards for Baltimore After-School Opportunities in Youth Places. Baltimore, MD: Author. Posner, J., & Vandell, D.L. (1994). Low-income children s after-school care: Are there beneficial effects of after-school programs? Child Development, 65, Rosenthal, R., & Vandell, D.L. (1996). Quality of school-aged child care programs: Regulatable features, observed experiences, child perspectives, and parent perspectives. Child Development, 67, Scales, P.C., & Leffert, N. (1999). Developmental assets: A synthesis of the scientific research on adolescent development. Minneapolis, MN: Search Institute. Smith, L.J., & Ross, S.M. (1999). School Observation Measure Observer s Manual. Memphis, TN: Center for Research in Educational Policy, The University of Memphis. 104

116 Vandell, D.L., & Pierce, K.M. (2001, April). Experiences in after-school programs and child wellbeing. In J.L. Mahoney (Chair), Protective aspects of after-school activities: Processes and mechanisms. Paper symposium conducted at the biennial meeting of the Society for Research in Child Development, Minneapolis, MN. Vandell, D.L., & Posner, J.K. (1999). Conceptualization and measurement of children s after-school environments. In S.L. Friedman & T.D. Wachs, (Eds.), Assessment of the environment across the lifespan. Washington, DC: American Psychological Association. Vandell, D.L., & Wolfe, B. (2000). Child care quality: Does it matter and does it need to be improved? (Institute for Research on Poverty Special Report No. 78). University of Wisconsin Madison, Institute for Research on Poverty. 105

117 Appendix A Summary List of the Standards for Baltimore After-School/Out-of-School Opportunities in YouthPlaces

118 Organizational Standards Human Relationships Staff relate to all youth in positive ways Staff respond appropriately to the individual needs of youth Staff encourage youth to make choices and to become more responsible Staff interact with youth to help them learn Staff use positive techniques to guide the behavior of youth Youth generally interact with one another in positive ways Staff and families interact with each other in positive ways Staff work well together to meet the needs of youth Indoor Environment The program's indoor space meets the needs of youth The indoor space allows youth to take initiative and explore their interests Outdoor Environment The outdoor play area meets the needs of youth, and the equipment allows them to be independent and creative Safety, Health, and Nutrition The safety and security of youth are protected The program provides an environment that protects and enhances the health of youth The program staff try to protect and enhance the health of youth Youth are carefully supervised to maintain safety The program serves foods and drinks that meet the needs of youth Staff-youth ratios and group sizes permit the staff to meet the needs of youth Youth are supervised at all times Administration Staff support families' involvement in the program Staff, families, and schools share important information to support the well-being of youth The program builds links to the community The program's indoor space meets the needs of staff The outdoor space is large enough to meet the needs of youth and staff Staff and youth work together to plan and implement suitable activities, which are consistent with the program's philosophy Program policies and procedures are in place to protect the safety of youth Program policies exist to protect and enhance the health of all youth Advocate support for youth is provided All staff are professionally qualified to work with youth Staff (paid, volunteer, substitute) are provided orientation to the job before working with youth The training needs of the staff are assessed, and training is relevant to the responsibilities of each job Staff receive appropriate support to make their work experience positive The administration provides sound management of the program Emergency information for each child is on file and accessible Program policies and procedures are responsive to the needs of youth and families in the community A-1

119 Program Standards 1. All programs/activities are designed to allow youth to participate and develop a mastery of at least one program component skill, knowledge or competency in both academic and non-academic activity. 2. Throughout the entire organization there is an infusion of opportunities for cognitive development Activities The daily schedule is flexible Youth choose from a wide variety of activities Activities reflect the mission of the program and promote the development of all youth in the program There are sufficient materials to support program activities Content Areas Cognitive Development Competitive and non-competitive Academic assistance Weekly small-group instruction in basic subjects using cooperative learning and other researchbased instructional strategies Recreation Team sports Workforce Development Education/exploration/planning/apprenticeships offered Experiential opportunities Artistic Development Meaningful competencies and in-depth knowledge Civic Development Community service opportunities Civic education/leadership development linked to communities, race/gender issues Open Time Opportunity for unstructured time Regular, structured opportunities to exchange ideas and think critically about issues that are important to youth A-2

120 Appendix B Observation Instrument Spring 2003 Version

121 Activity Context Coding Form Site I.D.: Date: Room #: Activity Number: Start: End: Observer Initials: Description: FOCUS SKILL AREAS TOTAL ADULTS/STAFF # Single activity/whole group Interpersonal Number of activity leaders Primary focal activity/opt-out activity(ies) Physical/athletic Number of assistants Multiple activities/no focal activity Artistic Number of teen staff ACTIVITY TYPE (Circle focal activity) Math/numeracy Guest speakers/instructors Homework assistance Reading/language arts GRADE LEVELS Tutoring Problem-solving PreK K 1 Reading/language arts enrichment Decision-making Math/science enrichment Other (Specify on back) Recreational reading/ listening to story/book None Computer skill-building TYPE OF SPACE Computer games Classroom/activity room NUMBER OF YOUTH # Sports tournament/game Gym Total number of youth Organized sports practice/drills Auditorium Number of boys Organized physical games Cafeteria Number of girls Fitness/exercise class Library GROUPING PATTERNS Recreational arts & crafts Computer room Large (15+) Fine arts instruction Outside playground/field Medium (6-14) Performing arts rehearsal/instruction Other (Specify on back) Small (3-5) Board/table/card games or puzzles MATERIALS USED Pairs (2) Open time/free play Computers Individuals working alone TV/video watching Art supplies GROUPING STRATEGY Job readiness/training Sports equipment Age or grade Health/well-being Reading materials Child s choice/interest Service/civic (in community or program) Writing materials All program attendees College/career planning Musical instruments Other (Specify on back) Other: Snack Arrival/dismissal Cultural awareness Other (Specify on back) Science materials Other (Specify on back) None B-1

122 Activity Context Other/Not Applicable Descriptions Activity Type Skill Areas Type of Space Materials Used Grouping Strategy B-2

123 Activity Observation - Coding Sheet Site ID Date Room #: Activity # Start Time: End Time: Observer Initials: Category Rating 1. YOUTH INTERACTIONS Indicator Checklist (+) Segment Indicator Checklist (-) Segment YOUTH 1 2 YOUTH 1 2 Have frequent interactions with one another Tease each other in a clearly unfriendly way Have infrequent interactions with one another Generally have warm, friendly interactions with each other Cooperate/share with each other Listen to each other/treat each other with respect Discuss/resolve disagreements/disputes constructively Threaten/bully/intimidate each other Category 2. STAFF-YOUTH INTERACTIONS Rating Indicator Checklist (+) Segment Indicator Checklist (-) Segment STAFF 1 2 STAFF 1 2 Are somewhat engaged with youth Are highly engaged with youth Have warm, friendly interactions with youth Engage informally in conversations with youth Use positive language and tone of voice with youth Listen actively and attentively to youth Use positive behavior management techniques Praise/encourage youth Have few or no interactions with youth Appear bored or distant Use harsh disciplinary methods Belittle youth or embarrass them in front of peers Leave youth unsupervised Are unaware of teasing/bullying or other serious conflicts among youth Actively encourage/facilitate participation from disengaged youth Help youth to resolve disagreements constructively YOUTH 1 2 YOUTH 1 2 Interact constructively/positively with staff Seek out positive contact/interactions with staff Ignore directions/tune staff out Are rude/actively negative toward staff B-3

124 Site ID Date Room #: Activity # Start Time: End Time: Observer Initials: Category 3. OPPORTUNITIES FOR SKILL-BUILDING AND MASTERY Rating Indicator Checklist (+) Segment Indicator Checklist (-) STAFF 1 2 Are clearly focused on instruction/helping youth to master a skill or learn something new Communicate goals, purpose, or expectations for activity Provide direct instruction/lecture/give directions Engage youth in brief question-response exchanges Demonstrate or model a concept or skill Ask youth why, how, and what if questions that require complex answers Facilitate discussion among youth Respond to youths requests for assistance in a timely manner Notice youth who are having difficulty and offer assistance Critique/offer constructive feedback to individual youth Challenge youth to push themselves intellectually, creatively and/or physically Link activity to an ongoing theme/thematic unit Encourage/facilitate cultural awareness/ethnic identity development Segment STAFF 1 2 Criticize youths work without offering guidance Discourage appropriate/reasonable questions/discussion/requests YOUTH 1 2 YOUTH 1 2 Write (not for homework) Read or are read to (not for homework) Work cooperatively with each other to achieve a goal Work on projects with culminating products or events Choose what or how they do something or help determine the direction of an activity Lead activities or groups of peers Tutor/mentor other youth Think strategically/analyze/solve complex problems Use information to accomplish a goal/make a decision Participate in structured discussions B-4

125 Site ID Date Room #: Activity # Start Time: End Time: Observer Initials: Category Rating Indicator Checklist (+) Youth are actively engaged, focused, interested, enjoying the activity 4. YOUTH ENGAGEMENT Segment 1 2 Indicator Checklist (-) Youth are disruptive and/or out of control Segment 1 2 Number of youth engaged/ on task at end of each observation segment Number of youth present at end of each observation segment Category Rating Indicator Checklist (+) Instructions are clear/youth know what to do Activity is appropriate for youths skill levels Staff have prepared/assembled needed materials Staff work well together 5. ACTIVITY ORGANIZATION Segment 1 2 Indicator Checklist (-) Segment 1 2 Category Rating Indicator Checklist (+) Materials/equipment are functional/in reasonably good condition There are enough materials There is enough room for the activity External interruptions/distractions are minimal and/or manageable 6. SETTING AND RESOURCES Segment 1 2 Indicator Checklist (-) Space is unsafe for the activity Other safety concerns Segment 1 2 B-5

126 Program Overview Data Collection Form Type of Space # Classroom/activity room # Observed Comments Facility Description List and describe the space(s) available to the program and how they are used Gym Auditorium Cafeteria Library Computer room Outside playground/field Other Activities Offered Cognitive/academic development Ages/ Grades Req d? (Y,N) # Days/W k # Mins/ Day Obsvd Activity Schedule Artistic development Describe the activities offered in each program area. For each activity listed, indicate the ages/grades served, number of days per week it is offered, duration of activity in minutes, and check if the activity was observed. Sports Civic development Workforce development Open time/general recreation Other B-6

127 Indicators Very True Somewhat True Not True Description Physical Environment There is comfortable furniture available for relaxing, reading, etc. There is quiet space/time for youth to read or do quiet work The program has access to a play area that is sufficiently large to allow youth vigorous physical play/exercise There are no observable safety hazards Literacy Environment Age-appropriate reading materials (books, magazines) are readily accessible to youth There are enough reading materials for youth There are printed materials on the walls Overall Program Organization Arrival, dismissal, and transitions are reasonably smooth B-7

128 Appendix C Number of Observations Conducted in Each Site Hours Open Number of Activity Observations Site Per Day Fall 2002 Spring 2003 Total A B C D 2 ½ E 2 ½ F G H Varies daily C-1

129 Appendix D Youth Survey Scales

130 How do kids/youth treat each other at this after-school program? (For each statement, circle if you agree a lot, agree a little, disagree a little, or disagree a lot.) a. Kids/youth here really care about each other here Alpha =0.83 b. Kids/youth here get along with one another c. Kids/youth treat each other with respect here How do staff at the after-school program treat you and other kids/youth? (For each statement, circle if you agree a lot, agree a little, disagree a little, or disagree a lot.) a. The staff here really care about me Alpha =0.79 b. The staff here always try to be fair c. The staff let kids/youth have a say in how things work here d. Staff here expect a lot of me e. I learn a lot from staff here f. Staff here tell me when I do a good job Below are some ways in which the after-school program may have helped you. (For each statement, circle if you agree a lot, agree a little, disagree a little, or disagree a lot.) The after-school program has helped me to: Alpha = 0.81 a. Learn to do a sport b. Learn to act, dance, or sing c. Learn to play a musical instrument d. Learn to do fine arts (painting, drawing, sculpture, etc.) e. Learn to speak or perform in front of a group f. Learn to work together with others g. Enjoy reading more h. Finish my homework i. Feel more like a part of my school j. Enjoy school more k. Do better at school l. Make new friends m. Learn about different jobs or careers n. Learn about how to get into college o. Avoid getting into fights p. Learn ways to say no when someone asks me to do things that I know are wrong and/or dangerous q. Think about the possible good and bad results of different choices before I make decisions D-1

131 Appendix E Overview of Analysis Datasets

132 Data Sources The In-Depth Study on After-School Program Quality was designed to inform the After School Strategy s efforts to assess and improve program quality by examining quality in a sample of eight programs. The major source of data for the study was a series of structured observations of program activities within the eight sites. PSA also administered surveys to participating youth in the third grade or higher to capture information such as student satisfaction, perceptions of quality, and program impact. In addition to data collected specifically for the study, PSA obtained data from the Family League of Baltimore City s management information system (FLBC MIS) about program budgets, enrollment levels, and average daily attendance. Data Linkages To facilitate the analysis of data across sources, we linked survey, observation and programlevel FLBC MIS data together and created two master databases. The first database was used in analyses exploring the relationship between observed quality and FLBC MIS program characteristics, such as average daily attendance or the use of certified teachers. Using each observation s program ID, we were able to link program-level characteristics to each observation record. The second database was created for analyses involving the interaction between survey responses, observed quality and program characteristics. These three data sources were linked by aggregating observation measures to the program level and appending them, along with program characteristics, to each student s survey responses. Analytic Approach To explore possible explanations for variations in the quality of observed youth interactions, the study conducted analyses guided by the following questions: What activity features seem to influence the quality of youth interactions? Is the quality of youth interactions influenced by the number of youth participating in the activity? Does the quality of youth interactions vary across different types of activities (e.g., arts, open time, sports)? E-1

133 Does the extent to which activities are mastery-oriented, focused on building higherorder skills (e.g., cooperation, group discussion), or generally more engaging affect the quality of youth interactions? What program features seem to influence the overall quality of youth interactions within a program site? Is the quality of youth interactions influenced by the number of youth who attend the program overall? Is the amount of enrichment programming sites offer overall associated with the quality of youth interactions? Is the quality of youth interactions in programs located in schools different from youth interactions in other program settings? Is the quality of youth interactions different at the end of the school year than in the beginning? Bivariate and subsequent multivariate analyses were conducted to examine the relationships among the activity and program characteristics described above and the quality of youth interactions. The relatively small number of observations conducted in each site limited the extent of multivariate analysis possible with the interim observation dataset. In particular, the small sample size limited the number of independent variables allowed in regression analyses, so we conducted separate analyses to examine activity-level and site-level factors that might influence quality. The first analysis examined the joint effects on the Observed Youth Interactions Index score of the type of activity observed (i.e., arts, homework help, academic enrichment, sports, open time, and other ), the number of youth in the activity (group size), the mastery orientation and higher-order skill building scale scores for the activity, the level of youth engagement in the activity, and whether the observation was conducted in the fall or spring round of data collection. Controlling for both activity type and timing of data collection was particularly important for assessing the relative influence of each of these two factors, since the mix of activities observed varied somewhat between data collection rounds. A second analysis focused on the joint effects of the average number of hours per week of activities offered in each program area, the average daily attendance per site, and whether the observations were conducted in fall or spring. The approach described above guided the analysis of all of the observation index scores. The analysis of each observation index was framed by questions about the relationships one might expect to find between that aspect of observed quality and potentially relevant activity features, program features, and time of year. These questions were explored first in bivariate analyses, followed by selected multivariate regression analyses based on the bivariate results. The selection of dependent variables in E-2

134 addition to the observation index scores was guided by these bivariate analyses as well as reasonable theories, based on our experiences in the programs as well as research in the field of youth development, about what relationships may exist. Use of Effect Sizes For this report, a focus on the analysis has been the observations conducted in the eight afterschool projects selected for this study. Indicators were combined to create total quality index scores for each category of the observation instrument. Analyses of the observation data show that the projects vary in the quality scores generated by indicators. To enhance our ability to interpret the differences in the observed quality by site, we compared the mean quality index score at a site with the mean quality index score across all eight sites, in order to compute an effect size. 15 We interpreted an effect size of as a notable positive difference between an individual project and the average project, and as a notable negative difference. The identification of projects that have notably positive or negative effect sizes is useful in directing the analysis of the other data collected for this project. In particular, it permits us to identify aspects of youths self-reported experiences in their programs that may be associated with observed quality. The small number of survey responses in four sites precluded the use of site-level quality scores in the analysis. The computation of quality-based categories was a solution to this problem, and the use of effect sizes to determine the categories provided a standardized basis for creating these categories. 15 The effect size is computed as: ((mean index score of observations at a site) minus (mean index score of observations at all eight sites) divided by (the standard deviation of the mean index score of observations at all eight sites)). E-3

135 Appendix F Summary Tables of Results

136 Utilization Rate Activity type ADA Activity type ** ** Group size NS NS Certified teachers Location Enrichment hours/wk ** 0.18* Cost per youth NS ADA OBSERVED QUALITY - BIVARIATE ANALYSES Observation Dataset Observation Measures Youth Interactions Staff-Youth Interactions Mastery Orientation Higher-Order Skill-Building Youth Engagement Data Round (Spring) Activity Characteristics Activity type Group size Certified teachers on staff Location (Schoolbased) Site Characteristics CORRELATIONS AMONG ACTIVITY AND SITE CHARACTERISTICS Observation Dataset Location Enrichment (School- hours Cost per Group size Based) per week youth Enrichment hrs/wk Cost per youth ADA Utilization Rate Youth Interactions Staff- Youth Interactions Observation Measures Mastery Orientation Higher- Order Skill- Building ++ NS NS NS NS 0.16* NS 0.35** 0.44** NS NS ++ ** NS ++ NS NS NS NS NS 0.37** NS 0.31** NS ** -0.33** ** -0.23** NS -0.20** 0.39** 0.20** ++ ** -0.19** ** -0.16** NS NS 0.16** NS * -0.25** NS NS NS NS NS NS Youth Engagement ** significance level of p<.01 * significance level of p < positive association at a significance level of p <.01 + positive association at a significance level of p <.05 negative association at a significance level of p <.01 negative association at a significance level of p <.05 F-1

137 OBSERVED QUALITY - REGRESSION ANALYSES Observation Dataset DEPENDENT VARIABLES Youth Interactions Staff-Youth Interactions Mastery Orientation Higher-Order Skill-Building Youth Engagement Data Round (Spring) INDEPENDENT VARIABLES Activity characteristics Site Characteristics Observation Measures Activity type Group size Youthadult ratio Certified teachers on staff Enrichment hrs/wk Cost per youth ADA Location Youth Interactions Staff- Youth Interactions Mastery Orientation Higher- Order Skill- Building ++ NS NS NS NS NS NS NS NS ++ NS NS ++ NS NS NS 16 NS ++ NS 17 NS ++ NS NS + 18 NS NS NS NS NS NS NS NS NS NS NS Youth Engagement Utilization Rate ++ NS NS NS NS 16 For open time, sports, and homework activities 17 For open time, sports, and homework activities 18 For arts, sports, and homework activities 19 For academic enrichment activities F-2

138 CORRELATIONS BETWEEN SITE CHARACTERISTICS Survey Dataset Enrichment hours per week >5 Cost per youth >=$2000 Certified teachers on staff Location (Schoolbased) Certified teachers Location (School-based) ++ Enrichment hours/wk > ** Cost per youth >=$2000 YOUTH-REPORTED QUALITY, SATISFACTION, IMPACT BIVARIATE ANALYSES Survey Dataset Survey Measures Youth- Reported Peer Interactions Survey Scale Youth- Reported Staff-Youth Interactions Scale Youth- Reported Program Satisfaction Youth- Reported Program Impact Scale Activity Characteristics Age Gender (Girl) Length of Participartion Certified teachers on staff Site Characteristics Location (Schoolbased) Enrichment hours per week >5 Cost per youth >=$2000 Utilization Rate Youth- Reported Peer Interactions Survey Scale Survey Measures Youth- Reported Staff-Youth Interactions Scale Youth- Reported Program Satisfaction Youth- Reported Program Impact Scale NS NS 0.26** 0.26** 0.22** -0.26** NS 0.41** 0.28** 0.30** NS NS NS NS NS NS NS 0.36** 0.44** NS NS NS NS NS NS NS 0.38** -0.26** NS NS NS NS NS NS NS F-3

139 YOUTH-REPORTED QUALITY, SATISFACTION, IMPACT BIVARIATE ANALYSES (Cont d) Survey Dataset Survey Measures Youth-Reported Peer Interactions Youth-Reported Staff-Youth Interactions Youth-Reported Program Satisfaction Youth-Reported Program Impact Notably High Observed Youth Interactions Notably High Observed Staff-Youth Interactions Observation Measures Notably High Observed Mastery Orientation Notably High Observed Higher-Order Skill- Building ++ NS ++ NS NS NS NS + NS NS NS NS NS NS NS + F-4

140 YOUTH-REPORTED QUALITY, SATISFACTION, IMPACT REGRESSION ANALYSES Survey Dataset DEPENDENT VARIABLES Youth-Reported Peer Interactions Survey Scale Youth Characteristics Age Gender (Girl) Certified teachers on staff Site Characteristics Location (schoolbased) NS + + Enrichment hrs/wk >5 NS ++ NS NS NS INDEPENDENT VARIABLES Cost per youth >=$2000 Notably High Obsv d Youth Interactions Observation Measures Notably High Obsv d Staff-Youth Interactions Notably High Obsv d Mastery Orientation Notably High Obsv d Higher- Order Skill- Building NS NS NS NS NS Youth-Reported Staff- Youth Interactions Scale Youth-Reported Program Satisfaction Youth-Reported Program Impact Scale NS NS + NS NS NS NS NS NS NS + NS NS NS NS NS NS NS NS NS NS NS NS NS NS NS Utilization Rate NS NS ++ NS NS NS NS NS F-5

141 OBSERVED YOUTH INTERACTIONS INDEX SCORES AND EFFECT SIZES BY STUDY SITE Observation Dataset Program-wide Study Site Site-level Average Observed Youth Interactions Index Score Average Observed Youth Interactions Index Score Effect Size A (+) B C D (-) E (-) F G H (+) OBSERVED STAFF-YOUTH INTERACTIONS INDEX SCORES AND EFFECT SIZES BY STUDY SITE Observation Dataset Study Site Site-level Average Observed Staff-Youth Interactions Index Score Program-wide Average Observed Staff-Youth Interactions Index Score Effect Size A (+) B C (+) D E (-) F (-) G H (-) + notably positive effect size - notably negative effect size F-6

142 MASTERY ORIENTATION INDEX SCORES AND EFFECT SIZES BY STUDY SITE Observation Dataset Site-level Study Site Average Mastery Orientation Index Score Program-wide Average Mastery Orientation Index Score Effect Size A (+) B (+) C (+) D E (-) F (-) G (-) H (+) HIGHER-ORDER SKILL-BUILDING INDEX SCORES AND EFFECT SIZES BY STUDY SITE Observation Dataset Study Site Site-level Average Higher-Order Skill-Building Index Score Program-wide Average Observed Higher-Order Skill- Building Index Score Effect Size A B C D (+) E (-) F (-) G (-) H (+) F-7

143