The only research-proven online game that assesses and teaches social and emotional skills

Transcription

1 The only research-proven online game that assesses and teaches social and emotional skills

2 Introduction The Problem Research demonstrates that positive social and emotional skills and relationships in childhood are associated with children s positive behavioral, emotional, and academic wellbeing. However, many children struggle socially throughout their school years. And without early intervention, these children can face a wide range of social, emotional, and academic problems, including school dropout, drug abuse, depression, anxiety, physical aggression, and antisocial behavior and these problems can persist into adulthood. Social and emotional learning helps children develop the skills they need to thrive socially, but conducting in-person training presents a variety of challenges for clinicians, schools, and parents. Logistical barriers, particularly the financial costs to families and schools, demanding resource needs, and time and travel requirements for children to participate, can deny many children the benefits of participating in a social and emotional learning program. Determining which children have social and emotional skills deficits can also be problematic. Traditional assessment methods such as behavioral observation and behavior rating scales can be costly and require extensive time and training to implement. In addition, they also pose challenges such as unreliable or biased observers, lack of social comparison data, inability to capture a student s behavior in specific situations, potential for students to modify their behavior because they re being observed, and inappropriate recording techniques. The Solution 1 in 4 children experience some type of social problem. 10% of children report having no friends in their class and no one to play with at school. 30% of children are involved in bullying on a regular basis as a bully, victim, or both. social scenarios. With funding from the U.S. Department of Education, 3C Institute developed Zoo U, an online game-based social and emotional skills assessment and skill-building program for children aged 7 12 ( In Zoo U, the only online game that assesses and teaches social and emotional skills, children become students at a virtual school for future zookeepers and build their social and emotional skills by working through a series of common Zoo U leverages powerful technology to eliminate the barriers of traditional social and emotional skills assessment and training methods (Figure 1). For example: Administration of Zoo U requires minimal training. Subjective bias and recording errors are eliminated because the assessment system rather than observers scores the child s behaviors. Social comparison data can be collected efficiently from a large group of children. 1

3 Situations that are important for assessment but unlikely to be observed because they occur infrequently can be incorporated into the assessment. Stealth assessment techniques in which assessments are embedded into a game and students aren t even aware they re being assessed greatly reduce the likelihood that children will alter their behavior to please an observer. Figure 1. Advantages of Zoo U over traditional social and emotional skills assessment methods Behavior Rating Scale Behavioral Observation Minimal training required ü ü Time-efficient ü ü Produces data about less frequent behaviors ü ü Objected report of behavior ü ü Situational specificity ü ü Reporter/recorder objectivity ü ü Stealth observation ü ü Assigns appropriate weight to behaviors ü= applicable; ü= somewhat applicable Zoo U Zoo U offers an engaging educational experience for students, an affordable method for broadscale everyday use by schools, and a seamless means of integrating data-driven decision making into school-based social interventions. And in addition to strong endorsements by both children and school professionals, research shows that Zoo U is effective in improving children s social and emotional skills. ü about the product The Child Experience: Assessment In Zoo U, players are students in a school for future zookeepers, where Principal Wild and a host of friendly animals help them learn essential social and emotional skills as they navigate common school-based social scenarios. Social and emotional skills addressed in Zoo U include: The child s Zoo U experience begins by creating a character, or avatar (Figure 2), which promotes engagement by helping students visualize themselves in the social scenarios the game presents. 2

4 Next, the child enters the assessment component and chooses one of the assessment scenes (Figure 3). Each scene focuses on a specific social skill area. Throughout the assessment, the child encounters six scenarios similar to those they might experience at school. To navigate these scenes, the child makes dialogue, action, and behavior choices (Figure 4). Figure 2. Character Creator Figure 3. Scene Selection Figure 4. Sample scene from Zoo U Assessment The Student Experience: Skill Builder Once the assessment is complete, the child enters the skill-building component of the game (Figure 5). Throughout the skill builder, the child completes 30 social scenes five in each social skill area. As with the assessment, the child makes dialogue, action, and behavior choices to navigate these scenes, and the child can earn virtual coins for making good choices. Following each scene, Principal Wild reviews the child s choices, giving praise and reinforcement for skills the child demonstrated well and constructive feedback on areas that the child hasn t quite mastered (Figure 6). Zoo U also helps children practice and strengthen their newly acquired skills through mini-games and other activities, such as deciphering a code with a classmate to improve cooperation skills (Figure 7). 3

5 Figure 5. Samle skill-builder scene Figure 6. Principal Wild feedback sample Figure 7. Interactive activity sample The Educator Experience Educators have access to an online dashboard (Figure 8) and downloadable graphic assessment and skill builder reports (Figures 9 and 10) full of useful information about their students progress and recommendations for specific resources based on the student s strengths and weaknesses. Reports can also be shared and saved. Zoo U also includes a Resource Center with supplemental social and emotional learning resources and activities. Educators can create classes or small groups and upload student information for easy, organized program administration. Reports can be generated for individual students or entire classes. 4

6 Figure 8. Educator Dashboard Alex Herman 5

7 Figure 9. Graphic assessment report sample with interpretation guidelines 6

8 Figure 10. Graphic skill builder report sample with interpretation guidelines 7

9 RESEARCH BEHIND ZOO U Zoo U is Effective in Assessing and Building Social and Emotional Skills 3C Institute researchers investigated the effectiveness of Zoo U with a group of children aged Parents completed questionnaires about their child s social and emotional skills and behaviors both prior to and after their child played Zoo U, and children also completed questionnaires before and after playing the game. Analyses revealed that children who played Zoo U Showed significant improvements in controlling impulses, initiating conversation, and managing emotions (see Table 1 in "Summary Statistics Section) Showed less aggression in social interations (see Table 1) Reported feeling more confident about social interactions and more accepted by peers (see Table 2) Made significant gains in social and emotional skills knowledge, especially in the areas of communication, cooperation, and empathy (see Table 3) What children learned from Zoo U: I learned how to use better communication and how to treat others better. I learned to control my feelings and that I should be nice to people. I learned more about considering other people and how they would react to a situation. I learned that sometimes I need to take a moment to relax and think about my actions before I jump right to the next thing. I learned how to deal with tough situations in real life. A H S The Zoo U Assessment Produces Reliable Data Researchers at 3C Institute conducted three field tests including one study that spanned 27 classrooms and 12 states to determine the validity of the scoring algorithms in the Zoo U assessment component. These studies enabled the game developers to refine the algorithms throughout the course of development and ultimately demonstrate that the assessment data produced through Zoo U gameplay accurately reflect the player s social skill level in each of Zoo U s six social skill areas. To test Zoo U s assessment validity, researchers compared Zoo U scores with teachers independent external assessment of their students social and emotional skills using established rating scale measures such as the Social Skills Improvement System. 8

10 These field tests revealed strong correlations between the students Zoo U assessment scores and the scores of the measures completed by the students teachers (see Tables 4 and 5 in Summary Statistics section). In addition, a national field test supported how differential performance on Zoo U (high, moderate, low) predicted teachers ratings of children s social and emotional skills at school (see Table 6) as well as teacher-rated real-world adjustment at school, including discipline problems and academic performance (see Table 7). Zoo U Also Helps Identify Potential Bullies Study data also revealed that Zoo U is effective in identifying potential bullies. Students who consistently selected the most negative dialogue option were significantly more likely than other children to be identified as a bully: 53% of these children were considered bullies by their teacher. In addition, students who scored poorly in the cooperation, empathy, and impulse control assessment scenes were significantly more likely than others to be identified as a bully by a teacher. In fact, 67% percent of children with low scores in all three areas were identified by their teacher as bullies. And 78% of the students who fell into both of these categories consistently negative responses and low scores in cooperation, empathy, and impulse control were identified as a bully by their teacher (Figure 11). The ability to identify bullies or a predilection for bullying behavior is especially powerful when it enables school staff to address the social and emotional skills deficits that lead to bullying before the child actually exhibits this behavior. Social-emotiona2 skills training such as Zoo U s skill-building component helps children learn how to interact appropriately with others and develop the capacity to resolve conflicts peacefully. Zoo U Engages Students To ensure usability with the target audience, 3C Institute researchers tested Zoo U s usability with a group of children aged In this study, children liked the game and thought their friends would also like it. Many of them were interested in playing Zoo U again or playing other games like Zoo U. Feedback from this study (Figure 12) reinforced the positive findings from an earlier field test conducted with the Zoo U prototype. (Also see Table 8 in "Summary Statistics" section.) 9

11 Figure 12. Child ratings of Zoo U 87% thought the game was fun 80% wanted to play more games like Zoo U 83% thought their friends would like it Children also shared feedback in their own words. Comments included: I liked the way you made learning social and emotional skills fun. I also liked the way you made the animals play an active part of Zoo U. The thing I liked the most about Zoo U was that it was a fun game to play, but it helped me learn about my feelings and how to control them. I liked that it was like a play just for fun online game and that it also teaches kids important lessons. It was fun to play, and I could not wait to see what was next. I loved the way you have an option of what you want to say. School Professionals Found Zoo U Innovative and Effective Observations during a Zoo U field test revealed high levels of student engagement with 96% on-task behavior. Students easily understood how to navigate the game with almost no instruction. During the game development process, 3C Institute researchers tested Zoo U with elementary school providers (teachers, counselors, and school administrators). Feedback about the game was extremely positive; school staff thought the software would be easy for teachers to use in the classroom, and the game was seen as a highly valuable tool for helping teachers assess students social skill levels and monitor student progress over time. They viewed Zoo U as highly innovative and effective compared to current social and emotional skills programs (Figure 13). (Also see Table 9 in Summary Statistics section.) Provider praise for Zoo U: This product is particularly useful because, unlike most assessments, it collects data directly from the student. Teachers need products like this that are easy to implement and will impact student learning. I love this approach when working with students. It's very engaging and appealing to students. This program would be an innovative way to collect baseline data and progress monitoring data on a child s social and emotional skills. A H S 10

12 Figure 13. Provider ratings for Zoo U 100% 75% % AGREED 50% 25% NOTE: See Summary Statistics section for detailed study data. summary Zoo U not only helps children build critical social and emotional skills, it gives educators reliable, affordable, easy-to-use tools for discovering children s specific social strengths and weaknesses and monitoring their progress throughout the intervention. Highly rated by school staff and children alike and proven effective through multiple research studies, Zoo U offers a compelling, interactive educational experience that enables children to safely explore different responses to social scenarios and the consequences of their choices. Future Directions Research: 3C Institute plans to conduct another randomized control trial with a larger sample of children in early 2015 to further explore the impact of Zoo U on children s social and emotional skills and behavior at home and at school. Product Development: Ongoing enhancements will be made to continually improve the gameplay and user interface experience as well as expand the supplemental resources. Additionally, data collected in the 2015 study will be used to refine Zoo U s scoring algorithms. The development team also plans to integrate emotional responses into the intelligent engine to further personalize the user experience. Additional enhancements will also be made based on user feedback once the product is commercialized. Commercialization: Zoo U is currently available for purchase by schools at 11

13 ABOUT THE GAME DEVELOPERS Zoo U was developed by a diverse team of researchers, content experts, game developers, web programmers, and artists at 3C Institute, an award-winning research and development company ( 3C Institute s mission is to bring research to practice to improve health and well-being in children, adolescents, and adults. 3C Institute's areas of expertise include evidence-based programs for social and emotional development, bullying prevention, and mental health; autism spectrum disorder; adaptive online assessment and game technology; and implementation science. The company is also developing several other social and emotional learning games. 3C Institute has partners with Personalized Learning Games ( to market and distribute Zoo U and other innovative technologies. The development of Zoo U was funded by the U.S. Department of Education through the following grants: ED-IES-10-P-0114 ($99,992) ED-IES-11-C-0039 ($849,989) PUBLICATIONS & PRESENTATIONS DeRosier, M. E. (Ed.). (2014). Social skills assessment through games: The new best practice. Cary, NC: Interchange Press. DeRosier, M. E. (2012, September). Making effective social skill assessment more accessible, informative, and effective for schools on a broad-scale through an interactive game-based platform for students. Presentation at the Society for Research on Educational Effectiveness (SREE) Fall Conference, Washington, DC. DeRosier, M. E., Craig, A. B., & Sanchez, R. P. (2012). Zoo U: A stealth approach to social skills assessment in schools. Advances in Human-Computer Interaction. doi: /2012/ Hehman, C. & Craig, A. (2012, June). Stealth assessment of social skills through interactive games. Presentation at the 8th Annual Games for Health Conference, Boston, MA. NOTE: 3C Institute researchers are in the process of publishing study results and are conducting additional testing on Zoo U to further support its effectiveness. 12

14 SUMMARY STATISTICS 13

15 Data supporting the effectiveness of Zoo U (continued): Table 1 (continued). Summary of Relevant Descriptive Information, ANCOVA Statistics, and Effective Sizes for Children s Social and Emotional Skills and Behaviors via Parents Reports on the Social and Emotional Skills Behavior Inventory Social Behaviors Internalizing Pre- Intervention Mean (SE) Post- Intervention Change Treatment 3.24 (.07) 3.93 (.05).69 Wait-list Control 3.21 (.10) 3.27 (.05).06 Externalizing Treatment 2.28 (.11) 2.12 (.12) -.16 Wait-list Control 2.58 (.15) 2.67 (.14).09 Assertion Treatment 3.52 (.24) 3.78 (.21).26 Wait-list Control 3.33 (.22) 3.17 (.22) -.16 Note. ᵻ p <.10, *p <.05, **p <.001 ANCOVA F ɳ * * *.090 Table 2. Summary of Relevant Descriptive Information, ANCOVA statistics, and Effect Sizes for Child-Reported Self-Efficacy and Peer Acceptance. NOTE: Two measures were used: (1) The Self-Efficacy and Outcome Expectancy Scale (SEOES; Ollendick & Schmidt, 1987), was used to assess children s social self-efficacy. This 20-item measure asks children to rate on a 5-point Likert-type scale how sure they felt about engaging in social behaviors and how sure they are that other children would respond in a specific way. (2) The Loneliness and Social Dissatisfaction Scale (LSDS; Cassidy & Asher, 1992) is a 19-item measure of children s self-reported feelings of loneliness and dissatisfaction with peer relationships. Pre- Intervention Mean (SE) Post- Intervention Change ANCOVA F ɳ 2 Self-Efficacy (based on a 5-point scale) Treatment (.11) (.09).374 Wait-list Control (.13) (.13) *.089 Peer Acceptance (based on a 4-point scale) Treatment (.04) (.13).917 Wait-list Control (.11) (.10) **.160 Note. *p <.05, **p <.001

16 Data supporting the effectiveness of Zoo U (continued): Table 3. Summary of Relevant Descriptive Information, MANOVA Statistics, and Effective Sizes for Child Performance on the Achieved Learning Questionnaire (ALQ) NOTE: Each unit of Zoo U is accompanied by a unit quiz, which, when compiled, comprises the Achieved Learning Questionnaire (ALQ), a 36-item measure of children s social literacy across six social-emotional domains. Children were asked to demonstrate their knowledge in each of these domains by answering multiple choice and true/false questions. Mean (SE) (% correct on ALQ) MANOVA F ɳ 2 Impulse Control Treatment.841 (.05) Wait-list Control.746 (.05) Communication Treatment.978 (.03) Wait-list Control.819 (.03) **.285 Cooperation Treatment.964 (.02) Wait-list Control.884 (.02) 5.569*.112 Social Initiation Treatment.949 (.03) Wait-list Control.899 (.03) Empathy Treatment.920 (.03) Wait-list Control.681 (.03) **.445 Emotion Regulation Treatment.725 (.03) Wait-list Control.826 (.03) 4.445*.092 Overall Treatment.896 (.02) Wait-list Control.809 (.02) 9.131**.172 Note. *p <.05, **p <.001

17 Data supporting the reliability of the Zoo U assessment: To establish the scoring algorithms for Zoo U s six social emotional skills, 3C Institute researchers conducted a field test with 50 3 rd and 4 th grade students in one central North Carolina elementary school. Students completed the six scenes of the Zoo U assessment while research staff supervised and observed. The students teachers independently completed online ratings of the social and behavioral adjustment of the students in their classroom using the Social Skills Improvement System (SSIS), a widely used behavioral rating scale with considerable evidence supporting its reliability and validity. 3C researchers computed correlations between the Zoo U and SSIS subscale scores. Results are displayed in Table 4: Table 4. Correlations between SSIS and Zoo U Scores. SSIS Subscales Impulse Control Emotion Regulation Zoo U Social Skill Areas Empathy Social Initiation Cooperation Attentive Self-Control Communication Empathy Engagement Cooperation Note. All correlations.20 were significant at the p<.05 level. As Zoo U development progressed, 3C researchers conducted a second field test to further refine the algorithms. A total of rd and 4 th grade students within 14 classrooms in two central North Carolina schools participated with their teachers. For this test, to gather teacher opinions of social behaviors that more closely mapped onto the social emotional skill constructs assessed in Zoo U, they consulted established social skill rating scale measures, including the Teacher Checklist, the Social Skills Improvement System, and the Social Competence Scale-Teacher Version, to create the Social Skills Behavior Inventory (SSBI). 3C researchers computed correlations between the Zoo U and SSBI subscale scores. Results are displayed in Table 5: Table 5. Correlations between SSBI and Zoo U Scores. SSBI Subscales Impulse Control Emotion Regulation Zoo U Social Skill Areas Communication Communication Empathy Social Initiation Cooperation Impulse Control.29***.31***.15*.16*.17*.16* Emotion Regulation.15*.30***.08.17*.25**.17* Communication.26***.28***.20**.19*.27***.20** Empathy.26***.22**.10.23**.24**.17* Social Initiation.17*.14*.08.15*.27***.18* Cooperation.23**.31***.12.17*.16*.21** Note. *p<.05, **p<.01, ***p<.0001level.

18 3C Institute researchers conducted a third field test to evaluate Zoo U with a broader selection of schools and teachers. The participating sample included rd and 4 th graders in 27 classrooms across 12 states. Teachers independently completed the SSBI for each student in their class, rating children for each of the SSBI social skill subscales as well as subscales assessing children s adjustment for school-related outcomes. Researchers investigated whether differential performance on Zoo U predicted teachers ratings of children s social skills at school (Table 6) and teacher-rated real-world adjustment at school (Table 7). Results are displayed in Tables 6 and 7: Table 6. Means and Mean Comparisons for Teacher-rated SSBI Social and Emotional Skills by Zoo U Categories. Zoo U Category Social Skill Area High Moderate Low Impulse Control.38 A -.04 B -.21 C Emotion Regulation.24 A -.01 A -.38 B Communication.17 A.00 AB -.19 B Empathy.34 A -.12 B -.17 B Social Initiation.20 A -.04 A -.42 B Cooperation.20 A.05 AB -.22 B Note. Means within a row with different superscript letters are significantly different from one another. Table 7. Significant Prediction of School Outcomes by Grade Level. SSBI School Outcome Subscale Internalizing behavior problems Externalizing behavior problems Disruptive behavior problems Impulse Control Emotion Regulation Communication Empathy Social Initiation 4 th 4 th 4 th 3 rd & 4 th 4 th 3 rd & 4 th 3 rd & 4 th 4 th 4 th 4 th Cooperation Discipline actions 4 th 3 rd & 4 th 4 th 3 rd Academic performance 3 rd & 4 th 4 th 3 rd & 4 th 3 rd & 4 th

19 Table 8 (Data for Figure 10). 3C Institute researchers tested Zoo U s usability with 29 children aged 7 12, who provided feedback through a brief evaluation survey. Table 8 shows child ratings of Zoo U using a 5-point scale from 1=Strongly disagree to 5=Strongly agree. Area rated Mean % Agreed or Strongly Agreed Liked the game % Thought the game was fun % Wanted to play more games like Zoo U % Thought their friends would like it % Wanted to play Zoo U again % Table 9 (Data for Figure 11). 3C Institute researchers tested Zoo U with 60 elementary school providers (teachers, counselors, and school administrators), who provided feedback through a brief evaluation survey. Table 9 shows school staff ratings of Zoo U using a 5-point scale from 1=Strongly disagree to 5=Strongly agree. Area Rated Mean (SD) % Agreed or Strongly Agreed Engaging for students 4.60 (.69) 95% Easy to use in the classroom 4.43 (.67) 90% Easy to use with individual students 4.63 (.52) 98% Feasible for use in schools 4.55 (.62) 94% A valuable tool for teachers 4.52 (.70) 92% A valuable assessment tool 4.55 (.59) 95% Useful for monitoring progress 4.50 (.62) 94% High overall quality 4.45 (.75) 95% More effective than current methods 4.48 (.70) 88%