The only research-proven online game that assesses and teaches social and emotional skills

Transcription

1 The only research-proven online game that assesses and teaches social and emotional skills

2 INTRODUCTION The Problem Research demonstrates that positive social and emotional skills and relationships in childhood are associated with children s positive behavioral, emotional, and academic wellbeing. However, many children struggle socially throughout their school years. And without early intervention, these children can face a wide range of social, emotional, and academic problems, including school dropout, drug abuse, depression, anxiety, physical aggression, and antisocial behavior and these problems can persist into adulthood. Social and emotional learning helps children develop the skills they need to thrive socially, but conducting in-person training presents a variety of challenges for clinicians, schools, and parents. Logistical barriers, particularly the financial costs to families and schools, demanding resource needs, and time and travel requirements for children to participate, can deny many children the benefits of participating in a social and emotional learning program. 1 in 4 children experience some type of social problem 10% of children report having no friends in their class and no one to play with at school. 30% of children are involved in bullying on a regular basis as a bully, victim, or both. Determining which children have social and emotional skills deficits can also be problematic. Traditional assessment methods such as behavioral observation and behavior rating scales can be costly and require extensive time and training to implement. In addition, they also pose challenges such as unreliable or biased observers, lack of social comparison data, inability to capture a student s behavior in specific situations, potential for students to modify their behavior because they re being observed, and inappropriate recording techniques. The Solution With funding from the U.S. Department of Education, 3C Institute developed Zoo U, an online game-based social and emotional skills assessment and skill-building program for children aged 7 12 ( In Zoo U, the only online game that assesses and teaches social and emotional skills, children become students at a virtual school for future zookeepers and build their social and emotional skills by working through a series of common social scenarios. Zoo U leverages powerful technology to eliminate the barriers of traditional social and emotional skills assessment and training methods (Figure 1). For example: Administration of Zoo U requires minimal training. Subjective bias and recording errors are eliminated because the assessment system rather than observers scores the child s behaviors. Social comparison data can be collected efficiently from a large group of children. Situations that are important for assessment but unlikely to be observed because they occur infrequently can be incorporated into the assessment. Stealth assessment techniques in which assessments are embedded into a game and students aren t even aware they re being assessed greatly reduce the likelihood that children will alter their behavior to please an observer.

3 Figure 1. Advantages of Zoo U over traditional social and emotional skills assessment methods Behavior Rating Scale Behavioral Observation Minimal training required Time-efficient Produces data about less frequent behaviors Objected report of behavior Situational specificity Reporter/recorder objectivity Stealth observation Zoo U Assigns appropriate weight to behaviors = applicable; = somewhat applicable Zoo U offers an engaging educational experience for students, an affordable method for broadscale everyday use by schools, and a seamless means of integrating data-driven decision making into schoolbased social interventions. And in addition to strong endorsements by both children and school professionals, research shows that Zoo U is effective in improving children s social and emotional skills. ABOUT THE PRODUCT The Child Experience: Assessment In Zoo U, players are students in a school for future zookeepers, where Principal Wild and a host of friendly animals help them learn essential social and emotional skills as they navigate common schoolbased social scenarios. Social and emotional skills addressed in Zoo U include: The child s Zoo U experience begins by creating a character, or avatar (Figure 2), which promotes engagement by helping students visualize themselves in the social scenarios the game presents.

4 Next, the child enters the assessment component and chooses one of the assessment scenes (Figure 3). Each scene focuses on a specific social skill area. Throughout the assessment, the child encounters six scenarios similar to those they might experience at school. To navigate these scenes, the child makes dialogue, action, and behavior choices (Figure 4). Figure 2. Character Creator Figure 3. Scene Selection Figure 4. Sample scene from Zoo U Assessment The Student Experience: Skill Builder Once the assessment is complete, the child enters the skill-building component of the game (Figure 5). Throughout the skill builder, the child completes 30 social scenes five in each social skill area. As with the assessment, the child makes dialogue, action, and behavior choices to navigate these scenes, and the child can earn virtual coins for making good choices. Following each scene, Principal Wild reviews the child s choices, giving praise and reinforcement for skills the child demonstrated well and constructive feedback on areas that the child hasn t quite mastered (Figure 6). Zoo U also helps children practice and strengthen their newly acquired skills through mini-games and other activities, such as deciphering a code with a classmate to improve cooperation skills (Figure 7).

5 Figure 5. Samle skill-builder scene Figure 6. Principal Wild feedback sample Figure 7. Interactive activity sample The Educator Experience Educators have access to an online dashboard (Figure 8) and downloadable graphic assessment and skill builder reports (Figures 9 and 10) full of useful information about their students progress and recommendations for specific resources based on the student s strengths and weaknesses. Reports can also be shared and saved. Zoo U also includes a Resource Center with supplemental social and emotional learning resources and activities. Educators can create classes or small groups and upload student information for easy, organized program administration. Reports can be generated for individual students or entire classes.

6 Figure 8. Educator Dashboard

7 Figure 9. Graphic assessment report sample with interpretation guidelines

8 Figure 10. Graphic skill builder report sample with interpretation guidelines

9 RESEARCH BEHIND ZOO U Zoo U is Effective in Assessing and Building Social and Emotional Skills 3C Institute researchers investigated the effectiveness of Zoo U with a group of children aged Parents completed questionnaires about their child s social and emotional skills and behaviors both prior to and after their child played Zoo U, and children also completed questionnaires before and after playing the game. Analyses revealed that children who played Zoo U Showed significant improvements in controlling impulses, initiating conversation, and managing emotions (see Table 1 in Summary Statistics Section) Showed less aggression in social interations (see Table 1) Reported feeling more confident about social interactions and more accepted by peers (see Table 2) Made significant gains in social and emotional skills knowledge, especially in the areas of communication, cooperation, and empathy (see Table 3) The Zoo U Assessment Produces Reliable Data Researchers at 3C Institute conducted three field tests including one study that spanned 27 classrooms and 12 states to determine the validity of the scoring algorithms in the Zoo U assessment component. These studies enabled the game developers to refine the algorithms throughout the course of development and ultimately demonstrate that the assessment data produced through Zoo U gameplay accurately reflect the player s social skill level in each of Zoo U s six social skill areas. To test Zoo U s assessment validity, researchers compared Zoo U scores with teachers independent external assessment of their students social and emotional skills using established rating scale measures such as the Social Skills Improvement System.

10 These field tests revealed strong correlations between the students Zoo U assessment scores and the scores of the measures completed by the students teachers (see Tables 4 and 5 in Summary Statistics section). In addition, a national field test supported how differential performance on Zoo U (high, moderate, low) predicted teachers ratings of children s social and emotional skills at school (see Table 6) as well as teacher-rated real-world adjustment at school, including discipline problems and academic performance (see Table 7). Zoo U Also Helps Identify Potential Bullies Study data also revealed that Zoo U is effective in identifying potential bullies. Students who consistently selected the most negative dialogue option were significantly more likely than other children to be identified as a bully: 53% of these children were considered bullies by their teacher. In addition, students who scored poorly in the cooperation, empathy, and impulse control assessment scenes were significantly more likely than others to be identified as a bully by a teacher. In fact, 67% percent of children with low scores in all three areas were identified by their teacher as bullies. And 78% of the students who fell into both of these categories consistently negative responses and low scores in cooperation, empathy, and impulse control were identified as a bully by their teacher (Figure 11). Figure 11. Predictability of bullying behavior based on Zoo U gameplay responses The ability to identify bullies or a predilection for bullying behavior is especially powerful when it enables school staff to address the social and emotional skills deficits that lead to bullying before the child actually exhibits this behavior. Social-emotiona2 skills training such as Zoo U s skill-building component helps children learn how to interact appropriately with others and develop the capacity to resolve conflicts peacefully. Zoo U Engages Students To ensure usability with the target audience, 3C Institute researchers tested Zoo U s usability with a group of children aged In this study, children liked the game and thought their friends would also like it. Many of them were interested in playing Zoo U again or playing other games like Zoo U. Feedback from this study (Figure 12) reinforced the positive findings from an earlier field test conducted with the Zoo U prototype. (Also see Table 8 in Summary Statistics section.)

11 Figure 12. Child ratings of Zoo U Children also shared feedback in their own words. Comments included: I liked the way you made learning social and emotional skills fun. I also liked the way you made the animals play an active part of Zoo U. The thing I liked the most about Zoo U was that it was a fun game to play, but it helped me learn about my feelings and how to control them. I liked that it was like a play just for fun online game and that it also teaches kids important lessons. It was fun to play, and I could not wait to see what was next. I loved the way you have an option of what you want tosay. School Professionals Found Zoo U Innovative and Effective During the game development process, 3C Institute researchers tested Zoo U with elementary school providers (teachers, counselors, and school administrators). Feedback about the game was extremely positive; school staff thought the software would be easy for teachers to use in the classroom, and the game was seen as a highly valuable tool for helping teachers assess students social skill levels and monitor student progress over time. They viewed Zoo U as highly innovative and effective compared to current social and emotional skills programs (Figure 13). (Also see Table 9 in Summary Statistics section.)

12 Figure 13. Provider ratings of Zoo U NOTE: See Summary Statistics section for detailed study data Zoo U not only helps children build critical social and emotional skills, it gives educators reliable, affordable, easy-to-use tools for discovering children s specific social strengths and weaknesses and monitoring their progress throughout the intervention. Highly rated by school staff and children alike and proven effective through multiple research studies, Zoo U offers a compelling, interactive educational experience that enables children to safely explore different responses to social scenarios and the consequences of their choices. Future Directions Research: 3C Institute plans to conduct another randomized control trial with a larger sample of children in early 2015 to further explore the impact of Zoo U on children s social and emotional skills and behavior at home and at school. Product Development: Ongoing enhancements will be made to continually improve the gameplay and user interface experience as well as expand the supplemental resources. Additionally, data collected in the 2015 study will be used to refine Zoo U s scoring algorithms. The development team also plans to integrate emotional responses into the intelligent engine to further personalize the user experience. Additional enhancements will also be made based on user feedback once the product is commercialized. Commercialization: Zoo U is currently available for purchase by schools at

13 ABOUT THE GAME DEVELOPERS Zoo U was developed by a diverse team of researchers, content experts, game developers, web programmers, and artists at 3C Institute, an award-winning research and development company ( 3C Institute s mission is to bring research to practice to improve health and well-being in children, adolescents, and adults. 3C Institute s areas of expertise include evidence-based programs for social and emotional development, bullying prevention, and mental health; autism spectrum disorder; adaptive online assessment and game technology; and implementation science. The company is also developing several other social and emotional learning games. 3C Institute has partners with Personalized Learning Games ( to market and distribute Zoo U and other innovative technologies. The development of Zoo U was funded by the U.S. Department of Education through the following grants: ED-IES-10-P-0114 ($99,992) ED-IES-11-C-0039 ($849,989) PUBLICATIONS & PRESENTATIONS DeRosier, M. E. (Ed.). (2014). Social skills assessment through games: The new best practice. Cary, NC: Interchange Press. DeRosier, M. E. (2012, September). Making effective social skill assessment more accessible, informative, and effective for schools on a broad-scale through an interactive game-based platform for students. Presentation at the Society for Research on Educational Effectiveness (SREE) Fall Conference, Washington, DC. DeRosier, M. E., Craig, A. B., & Sanchez, R. P. (2012). Zoo U: A stealth approach to social skills assessment in schools. Advances in Human-Computer Interaction. doi: /2012/ Hehman, C. & Craig, A. (2012, June). Stealth assessment of social skills through interactive games. Presentation at the 8th Annual Games for Health Conference, Boston, MA. NOTE: 3C Institute researchers are in the process of publishing study results and are conducting additional testing on Zoo U to further support its effectiveness.

14 SUMMARY STATISTICS Data supporting the effectiveness of Zoo U: 3C Institute researchers investigated the effectiveness of Zoo U with 47 children aged Study participants were recruited nationally via parenting and educational listservs and social media sites. Parents completed an eligibility questionnaire that included a demographic survey and the parent rating scales of the Behavior Assessment System for Children, Second Edition (BASC-2; Reynolds and Kamphaus, 2004.) Study results are presented in the tables below. (continued)