Are badges useful in education?: it depends upon the type of badge and expertise of learner

Transcription

1 Education Tech Research Dev DOI /s DEVELOPMENT ARTICLE Are badges useful in education?: it depends upon the type of badge and expertise of learner Samuel Abramovich Christian Schunn Ross Mitsuo Higashi Ó Association for Educational Communications and Technology 2013 Abstract Educational Badges are touted as an alternative assessment that can increase learner motivation. We considered two distinct models for educational badges; merit badges and videogame achievements. To begin unpacking the relationship between badges and motivation, we conducted a study using badges within an intelligent-tutor system for teaching applied mathematics to middle-school students. Our findings indicate that badge earning could be driven by learner motivations and that systems with badges could have a positive effect on critical learner motivations. However, badge acquisition patterns were different across learners with different levels of prior knowledge. Different badge types also affected different learners motivation. Additionally, we believe that our findings are compatible with the research finding that extrinsic motivators have a negative influence on learning. The implication for educational badge designers is that they must consider the ability and motivations of learners when choosing what badges to include in their curricula. We believe our findings exist as one piece of the large research base needed to understand educational badges. keywords Badges Alternative assessment Motivation Intelligent tutors Introduction For designers, modern educational assessment is an example of the proverbial doubleedged sword. Assessments can contribute to learning and proper understanding (Shepard S. Abramovich (&) C. Schunn Learning Research and Development Center, University of Pittsburgh, Suite O Hara St, Pittsburgh, PA 15260, USA sja24@pitt.edu C. Schunn schunn@pitt.edu R. M. Higashi Robotics Academy, The National Robotics Engineering Center, Carnegie Mellon University, Ten 40th St., Pittsburgh, PA 15201, USA rhigashi@andrew.cmu.edu

2 S. Abramovich et al. 2000) through providing necessary formative feedback as well as additional ways to gauge student learning. But over-assessment and inaccurate assessment have clear negative consequences on motivation (Stiggins 2002). Students who are over-assessed can become motivated to seek mastery of exams rather than genuine learning. Inaccurate assessment is dangerous in that it can cause a drop in motivation to learn for a student who thought they were learning but is provided feedback that contradicts their self-assessment. Because motivation is known to be key to learning (Clark et al. 2006), the potential benefit of an assessment is determined by its ability to both maintain learning motivation and accurately communicate a student s learning. Educational reformers have looked to alternative assessments as a means to maximize the benefits of assessments while minimizing negative effects (Stiggins 2005). Moving away from the culture of standardized testing allows more options for instructional designers in the construction of assessments that permit a wider variety of feedback and data to students and teachers. One alternative assessment that has begun to gain traction among reformers and instructional designers is educational badges (Alberts 2010). Badges, much like their counterparts in scouting and videogames, are seen as a way to assess learning outside of formal schooling. The issuers of educational badges an educator or educational organization can give a symbolic award for any type of skill, knowledge, or achievement similar to how they currently provide degrees or certificates. The symbol, in the form of a badge, can then be displayed by the learner to let others know of their mastery or knowledge. Therefore, instructional designers can use educational badges to influence engagement and learning. For example, badges can provide focused goals, challenging tasks, clear standards, affirmation of performance, novelty, choice, and authenticity (Dickey 2005). The implementation of educational badges is already underway by many organizations such as Mozilla (Peer 2 Peer University and Mozilla Foundation 2011) and the Khan Academy. However, there is little research that examines how badges interact with student motivation. While badges might provide the type of formative feedback valued in alternative assessment, badges could also be a negative influence through decreasing a student s motivation to learn. For example, earlier research about other kinds of rewards has regularly found that external rewards are bad motivators for learning (Deci et al. 1999). If learners interpret badges as external rewards, then they could possibly lower a student s motivation to learn or cause the student to focus on earning badges to the exclusion of the learning goals. In order to begin unpacking the possible bi-directional relationships between educational badges and learning motivation, we conducted a study using badges within an intelligent-tutor system for teaching applied mathematics to middle-school students. We investigated how learner motivation changed with exposure to badges and how learner motivations shaped badge acquisition. Theoretical background Alternative models of badges Badge advocates claim that badges can be offered as an alternative assessment that will increase learner motivation while maintaining high-quality feedback (Davidson 2011). To understand where these claims originate, let us consider the two distinct models for educational badges: merit badges and videogame achievements.

3 Are badges useful in education? Merit-badges, a feature of the United States Boy and Girl Scout organizations, offers participants the chance to earn certification (i.e. a Badge) of specific knowledge or skills not usually addressed by formal educational systems. Children select which badges they want to earn, under the theory that the goal of earning a badge will trigger an increase in motivation for those learners who want formalized recognition. Additionally, a scout s display of their earned badges represents a type of curriculum vitae of their learning and allows others to learn both what a scout knows and what the scout values. The second model of badges is based on meta-gaming features common to videogames. Many best-selling videogame systems allow players to earn recognition of their in-game achievements outside of the game itself. For example, Xbox players have a virtual profile displaying their different game accomplishments for others to see. These achievements (i.e. badges) are awarded to players while playing a game but have no direct effect on play other than being designed to encourage more play. Within the Xbox ecosystem, a player can compare their achievements with those earned by their friends and peers. Player can choose to earn specific badges but also can earn a badge incidentally through normal game play. Educational badges share many of the same features of the merit-badge and video-game models. Like merit badges, educational badges are commonly offered for learning that occurs outside of traditional educational institutions (e.g. community building, online literacy). Educational badges are commonly viewable on a learner s online profile by a learner s peers, similar to how videogame badges are viewable to other players and how scouts display badges on sashes. Similar to videogame achievements, badges can be awarded for incidental activity in addition to mastery of skills or demonstration of knowledge. An example of an educational badge system is Carnegie Mellon University s Computer Science Student Network (CS2N; CS2N is an online learning system in which participants can earn badges while developing computer-science skills and knowledge. Although computer science courses can be found in some public schools, CS2N cover aspects of computer science that are often skipped (e.g. animation, robotics). Each CS2N user has a profile page on which they can see the badges they have earned as well as potential badges they might wish to earn. The variety of educational applications available in CS2N (e.g. intelligent tutors, virtual robots, learning management systems) on a variety of topics (e.g. algebra, programming, movie making) provides students a breadth of badges to pursue. Within each application, badges are awarded for progress toward mastery as well as continued participation. Psychological theories of learner motivation Because motivation and assessment are intertwined, to understand the educational potential of badges requires understanding of how badges affect learning motivation. In the past 15 years, a particularly successful theory for studying motivation and learning is achievement goal theory (Elliot 1999). Achievement goal theory frames motivational learning goals into four different types organized into a matrix of mastery and performance crossed with approach and avoidance. Mastery approach goals reflect a desire to achieve mastery based on one s own interest. Performance approach goals reflect a desire to perform demonstrably better. Performance avoidance goals reflect the desire to avoid the appearance of underperforming. Mastery avoidance goals, while existing theoretically, do not manifest in most real-world contexts. Research typically finds a positive correlation between mastery learning and academic performance outcomes and a negative

4 S. Abramovich et al. correlation between performance avoidance and academic outcomes (Elliot et al. 2006). Performance approach goals have had mixed outcomes, depending on other variables (Elliott et al. 2005). Overall, achievement goal orientation has proven to be a good predictor of academic performance in various academic subjects (Pajares et al. 2000). For example, mastery goals were found to positively correlate with student use of cognitive strategies while performance approach goals positively correlated with teacher-assigned grades (Wolters 2004). Because badges, at first glance, have performance and mastery elements, achievement goal theory may be a useful way of unpacking the effects of educational badges on learning-relevant motivations. It is important to note that achievement goal theory is only one well-established theory of learning motivation. Another established theoretical framing of motivation is expectancy-value theory (Wigfield and Eccles 2000). How a student expects to perform in a subject is known to be an independent predictor of achievement. For example, students with low confidence in their math ability are more likely to guess an answer to a math question rather than work to solve it (Beal et al. 2008). Additionally, the level of interest of a student has in a subject (intrinsic value) can also independently predict future learning. For example, students who are interested in math will continue to enroll in math related courses, increasing their opportunity to learn relative to students less interested in math (Pintrich 2003). Educational badges, in addition to potentially changing a learner s achievement goals, might also change how much the learner values a subject (by having them choose goals to pursue) or what their expectations are for success (by regularly flagging success or lack thereof). Interactions between learner prior knowledge and badges The ways in which learners react to badges may depend upon a range of prior experiences with the domain being badged. Prior knowledge and its interaction with other student characteristics can have a major influence on assessment performance (Dochy et al. 1999). In addition to having built up particular achievement goals, interests, and expectations for success, a learner will have prior knowledge levels that will shape how quickly and easily badges are earned, which then may also shape how much the learner values these badges. Therefore we examine whether prior ability levels moderate the relationships of badge earning with learner motivations. Methods Participants 36 seventh graders and 15 eighth graders at a charter school serving a low-income suburb of a large east coast city in North America participated in our study. Measures To measure achievement goal orientation, we used a version of the patterns of adaptive learning scales (PALS) survey instrument (Midgley et al. 2000) modified to measure motivation related to the content area focus of the study, math. The measures of Mastery goals (e.g. One of my goals this year is to master a lot of new skills in Math. ),

5 Are badges useful in education? Performance Approach goals (e.g. It s important to me that I look smart compared to others in Math. ), and Performance Avoidance goals (e.g. One of my goals in Math is to avoid looking like I have trouble doing the work. ) contained four questions each with answers in a Likert Scale from 1 5. Pre Cronbach alphas for mastery goals, performance approach, and performance avoidance were 0.82, 0.81, and 0.68 and post alphas were 0.81, 0.84, and To measure the key constructs from expected value theory, we used five questions with answers in a Likert Scale related to expectancy (e.g. If you had to take an important math test today, would you do well? ), and two questions related to interest (intrinsic value, e.g. In general, I find working on math assignments very interesting. ) that were adapted from a variety of studies (Beal et al. 2008; Wigfield 1994; Wigfield and Eccles 2000). Motivation and interest alphas, respectively, were 0.81 and 0.83 for pre measures and 0.78 and 0.82 for post measures. To determine if badge interaction with learning motivation was different for students with different levels of skills, we used a pretest of proportional reasoning that has been validated for this age group and focuses on the exact skills taught in the tutor (Weaver and Junker 2004). We divided the students in the study into two groups based on a mean split of the number of pre-assessment math problems correct. The low-performing students had a range of correct problems from 3 to 10. High-performing students had a range of problems correct. Students opinions on badges were measured using questions previously established in an earlier study (Abramovich et al. 2011) which in turn were created through several pilot studies. Because of the relative newness of educational badges, we are unaware of any other badge specific opinion questions used in prior research. Procedure We measured pre-ability and pre and post motivational levels around the use of the CS2N intelligent tutoring system. Students earned various forms of badges in the CS2N intelligent tutoring system and were notified of all awarded badges through a dialogue box within the tutor interface (Fig. 1). Students were initially unaware of the available badges but Fig. 1 Examples of participation (left) and skill (right) badges in the CS2N intelligent tutor

6 S. Abramovich et al. would compare and converse with each other about the different type of badges they had earned. The intelligent tutoring system builds upon a Cognitive Tutor approach developed at Carnegie Mellon University and deployed in thousands of schools (Anderson et al. 1995). The tutor could provide hints when students were stuck and kept track of skills being mastered based on successfully solving particular solution steps associated with each skill, but doing so without error or hint. The content area focus of the particular unit called robots in motion was proportional reasoning, understood to be a foundation for mathematics, science, and engineering education (Silk et al. 2010). Students spent on average 20 min each school day with the tutor over *1 month period. They learned how to use various mathematical reasoning strategies to program a robot to move certain distances and turn certain angles in the context of trying to solve a larger problem involving controlling a robot on a distant asteroid, eliminating the guessand-check methodology commonly employed by novice roboticists (see Fig. 2 for an example task page). In the particular version of CS2N and the intelligent tutor that was studied, badges were granted based on metrics related to progress within the tutor and mastery of measured skills. Accordingly, we classified the badges awarded by the tutor into two distinct categories (Table 1): those indicating mastery of skills and those reflecting participation in the system. Prior testing had indicated the potential for badges to re-motivate students whose frustration level was increasing while using the tutor (Abramovich et al. 2011). Fig. 2 Example tutor page. On this page, students respond via pull-down menus, whereas on some other pages, students type in number responses or free text explanations. When students click on the hint button, hints are shown in the hints window

7 Are badges useful in education? Consequently, we theorized that badges awarded for participation, regardless of the quality of the user s performance, would provide increased motivation to all users. Badges that were awarded based on mastery of skills within the tutor were aligned with the theory that badges can act as an alternative assessment. Because of their different theoretical functions with respect to motivation and learning, we separately examined correlations with number of participatory badges and number of skill badges. In addition, exploratory factor analyses on badge earning data suggested there are two underlying dimensions of skill and participation (see Table 1 for the list of each). Our research questions were: 1. Do badges have a different motivation relationship for different ability learners? 2. Do different types of badges have different motivation relationships with learners? Table 1 Badges available in the CS2N Intelligent tutoring system and number earned overall and by level of learner Description Category Number earned HP LP Correct a mistake in less than 2 min Skill Complete a problem with no errors in less than 5 min Skill Complete a problem with no errors in under 1 min Skill Find distance using scale factor Skill Find distance using unit rate Skill Find power using scale factor Skill Find power using unit rate Skill Count half of a turn Skill Count a whole turn Skill Measure with non-zero aligned start Skill Measure with zero aligned start Skill Measure remainder Skill min of cumulative cognitive tutor time Participatory min of cumulative cognitive tutor time Participatory min of cumulative cognitive tutor time Participatory min of cumulative cognitive tutor time Participatory min of cumulative cognitive tutor time Participatory Answer 25 questions correctly Participatory Answer 50 questions correctly Participatory Answer 75 questions correctly Participatory Answer 100 questions correctly Participatory Answer 200 questions correctly Participatory Answer 5 questions correctly Participatory Correct a mistake in under 5 min Participatory Complete a page with a mistake in at least 10 min Participatory Complete a page with an error in at least 15 min Participatory Getting a hint Participatory HP badges earned by high math performers, LP badges earned by low math performers

8 S. Abramovich et al. Table 2 Mean pre, post, and change in motivation measures (with S.D.) Measure Pre Post Change Low High All Low High All Low High All Mastery 4.5 (0.5) 4.6 (0.7) 4.6 (0.7) 4.5 (0.8) 4.6 (0.7) 4.5 (0.8) 0.0 (0.7) 0.0 (0.4) 0.0 (0.5) Performance approach 3.4 (1.0) 3.2 (1.3) 3.3 (1.2) 3.4 (1.0) 2.8 (1.3) 3.1 (1.2) 0.0 (0.9) -0.4 (1.0) -0.2 (1.0) Performance avoidance 3.5 (1.0) 2.8 (1.2) 3.1 (1.2) 3.0 (0.9) 2.7 (1.2) 2.9 (1.1) -0.5** (0.7) -0.1 (0.9) -0.2* (0.8) Expectancy 3.8 (0.8) 4.6 (0.4) 4.2 (0.7) 3.9 (0.7) 4.6 (0.3) 4.3 (0.7) 0.1 (0.5) 0.0 (0.4) 0.1 (0.4) Interest 3.4 (1.2) 4.2 (1.0) 3.8 (1.2) 3.8 (1.2) 4.3 (0.9) 4.1 (1.1) 0.4* (0.9) 0.1 (0.7) 0.3* (0.8) * p \ 0.05; ** p \ 0.01

9 Are badges useful in education? Fig. 3 Mean pre- and postscores (with SE bars) on performance avoidance and interest measures for low and high math pre-test groups.**p \ 0.01, *p \ 0.05 Findings Aggregate analyses across all learners Analysis of all students (n = 51) revealed a few small overall changes in motivations. Paired sample t-tests of pre to post changes revealed a 0.25 decrease (on the 1 to 5 scale) in performance avoidance motivation (t = 2.06, p \ 0.05, Cohen s d = 0.29) and a 0.25 increase in interest (t = 2.34, p \ 0.05, Cohen s d = 0.33). Overall, students who used the cognitive tutor became less concerned about poor performance relative to other students and became more interested in math. To discover if badges interacted with learning motivation (i.e. were predicted by pre motivation levels or were associated with changes in motivation levels), we conducted Pearson correlations between the number of participatory badges earned and the different measures of pre motivation, prior ability, and pre-post changes in motivation. Pre measures of motivation and number of math problems correct did not correlate with the total number of badges earned by students. However, the total number of badges earned did correlate with an increase in performance avoidance motivation (r = 0.30, p \ 0.05). The more badges a student earned indicated less of an overall decrease in the same student s concern about their performance. Based on these limited findings, we might have concluded that while the cognitive tutor might have had a positive effect on some learning motivation, the inclusion of badges possibly has a negative effect on learning. However, we further unpacked our findings, looking for whether the effects are actually specific to particular student subgroups and/or badges types. Do lower performing students and higher performing students see different motivation changes? As shown in Table 2, for the low-performing students (n = 23), we discovered that performance avoidance motivation (e.g. One of my goals in Math is to avoid looking like I have trouble doing the work.) decreased by 0.48 (t = 3.49, p \ 0.01, Cohen s d = 0.73)

10 S. Abramovich et al. Table 3 Pearson correlations (and p values) between motivation pre measures and skill and participatory badges earning for all users and separately by math performance levels Pre measures Skill badges Participation badges All users HP LP All users HP LP Mastery 0.19 (0.18) 0.13 (0.50) 0.24 (0.26) 0.19 (0.21) 0.16 (0.43) 0.19 (0.39) Performance (0.68) (0.53) 0.23 (0.30) 0.06 (0.68) (0.38) 0.48* (0.02) approach Performance -0.34* (0.01) -0.39* (0.04) 0.09 (0.67) (0.12) (0.10) 0.10 (0.65) avoidance Expectancy 0.14 (0.34) (0.06) 0.07 (0.74) 0.18 (0.21) (0.72) 0.08 (0.73) Interest 0.27 (0.06) 0.27 (0.16) (0.96) 0.06 (0.70) 0.09 (0.64) (0.47) HP high math performers, LP low math performers * p \ 0.05 Table 4 Correlations between changes in Motivation measures and Badges Badges Increase in performance avoidance Increase in expectancy All users HP LP All users HP LP Skill 0.40** (0.00) 0.37 (0.05) 0.22 (0.31) 0.20 (0.17) 0.49** (0.01) (0.72) Participation 0.30* (0.03) 0.13 (0.50) 0.46* (0.03) 0.04 (0.76) 0.21 (0.28) (0.84) HP high math performers, LP low math performers Pearson correlations, * p \ 0.05; ** p \ 0.01 and interest in math increased by 0.43 (t = 2.40, p \ 0.05, Cohen s d = 0.50) over the course of using the cognitive tutor. There were no statistically significant changes in motivation for the high-performing students (n = 28). That is, the changes in motivation that we previously uncovered for all students are, in actuality, only for low-performing students (see Figure 3). Is badge earning predicted by different motivational factors in high and low math performers? For the low-performing students, the pre measure of performance approach motivation (e.g. It s important to me that I look smart compared to others in Math.) correlated with the total badges earned (r = 0.47, p \ 0.05) and total badges earned correlated with an increase in performance avoidance (r = 0.42, p \ 0.05). Based on this data, we hypothesize that among low-performing students, those with a higher desire to outperform other students earned more badges. However, the more badges earned by low-performing students also indicated less of an overall decrease in concern about their performance relative to other students. For the high-performing students, there was a negative correlation between pre performance avoidance motivation and total badges earned (r =-0.39, p \ 0.05). This negative correlation indicates that, among the high performing students, those who were less concerned about having poor performance relative to other students were the ones who earned more badges.

11 Are badges useful in education? Fig. 4 For low-performing students, change in performance avoidance goals from pre to post as a function of # of participation badges earned Fig. 5 For high-performing students, change in expectancies for success from pre to post measures as a function of number of skill badges earned Having discovered that different ability students motivations have different relations to badges, we looked to discover whether different types of badges have different relationships with student motivation. Specifically, we performed correlations between the

12 S. Abramovich et al. different motivation measures and the two classifications of badges: participation and skill (see Table 3). For low-performing students, pre performance approach motivation (e.g. It s important to me that I look smart compared to others in Math. ) correlated with earning participation badges (r = 0.48, p \ 0.05) but not skill badges. That is, the prior correlations we detected for low-performing students can be further limited to only participation badges. For the high-performing students, pre performance avoidance (e.g. t s important to me that I don t look stupid in Math. ) had a negative correlation with skill badges (r =-0.39, p \ 0.05) but no correlation with participation badges. In fact, earning participation badges did not correlate with any pre measure for the high-performing students. Just as we can limit the correlations to low-performing student to only participatory badges, we can as well limit the correlations detected for high-performing students to just skill badges. Does badge earning predict motivational changes? As shown in Table 4 and Fig. 4, in low-performing students, earning participatory badges correlated with an increase in performance avoidance motivation (r = 0.46, p \ 0.05) while earning skill badges had no correlation with any measure. Note that relatively few students had an absolute growth in performance avoidance goals. Rather, the Table 5 Mean ratings (and SD) for each question in the badge opinion survey for all students and separately for high and low performing students Questions All students HP LP I understand why I earned all of my badges/achievements 4.3 (1.0) 4.4 (1.1) 4.2 (0.8) The badges/achievements were more important to me than other parts 2.6 (1.4) 2.6 (1.5) 2.7 (1.4) of the cognitive Tutor I think that the badges/achievements are a good addition to the 3.6 (1.4) 3.6 (1.4) 3.6 (1.4) cognitive tutor I knew what badges/achievements were before I stated working 2.8 (1.6) 2.9 (1.7) 2.6 (1.4) on the cognitive tutor I wanted to earn more robots in motion tutor badges/achievements 3.3 (1.5) 3.4 (1.5) 3.3 (1.4) I don t care about the robots in motion tutor badges/achievements 2.6 (1.4) 2.4 (1.5) 2.8 (1.3) I like earning badges/achievements but not the ones in the robots in 2.8 (1.4) 2.5 (1.4) 3.3 (1.4) motion tutor I wish the robots in motion tutor badges/achievements were harder 2.7 (1.5) 2.9 (1.6) 2.4 (1.2) to earn I wish the robots in motion tutor badges/achievements were easier 2.9 (1.5) 2.8 (1.7) 3.0 (1.4) to earn I want to look at all of the badges/achievements I earned in the robots 3.5 (1.5) 3.6 (1.7) 3.3 (1.4) in motion tutor I told others about my robots in motion tutor badges/achievements 2.4 (1.4) 2.3 (1.4) 2.6 (1.3) The robots in motion tutor badges/achievements made me want to keep working 3.5 (1.4) 3.6 (1.4) 3.3 (1.4) HP high math performers, LP low math performers

13 Are badges useful in education? low-performing students that earned the most participatory badges had a mixture of increases and decreases in performance avoidance goals. The low-performing students that earned few such badges consistently had decreases in performance avoidance goals. By contrast, as show in Table 4, for high-performing students earning more participation badges had no relationship to changing motivations. Instead it was the earning of skill badges that was associated with motivational changes and, in particular, the number of skill badges earned was highly correlated with an increase in expectancy to do well at math (r = 0.49, p \ 0.01; see Fig. 5). Here we see that students earning few skill badges tended to experience a drop in expectancies for success, whereas those who earned many skill badges tended to experience a growth in expectancies for success. Do lower performing students and higher performing students have different opinions about badges? Because educational badges are often a new experience for students, we also surveyed the students about their experience with the intelligent tutoring system badges. Overall, the students responses were favorable, with means towards the positive end of each scale for all questions (i.e. above 3 for positive statements, below 3 for negative statements; see Table 5). Further, this positive view was held by both subgroups, and there were also no statistically significant differences (p \ 0.05) in the mean ratings between high and lower performing students on any of the statements except for I like earning badges/achievements but not the ones in the robots in motion tutor (t = 2.05, p \ 0.05, Cohen s d = 0.58). Low performing students indicated more agreement with this statement, possibly related to changes in their performance avoidance motivation. To try and unpack some of the differences we had detected between low-performing and high-performing students on motivational changes and relationships to badges, we looked for correlations between these survey opinions and their scores on the motivation measures. For low-performing students, an increase in performance avoidance correlated with I wish tutor badges were harder (r = 0.42, p \ 0.05) and I knew about badges before I started the tutor (r = 0.48, p \ 0.05). For high-performing students, an increase in expectancy to do well at math correlated with I wish tutor badges were harder (r = 0.46, p \ 0.05) but also I understood why I earned all of my badges (r = 0.42, p \ 0.05). All other correlations with opinion questions were non-significant. Discussion Overall, we found evidence of improvements in interest and decrease in counter-productive motivational goal from a system using educational badges. Further, we find evidence that earning various badges can be associated in increases in expectations for success but also increases in counter-productive educational goals. Thus, in contrast to what might be expected from conceptualizing badges as only being extrinsic rewards (and therefore only bad for learning), we find evidence suggesting both positive and negative effects. However, it is also very salient that effects of educational badges vary with different ability learners: badge acquisition patterns were quite different across learners with different levels of prior knowledge. Only for the low-performing students in our study did a higher desire to outperform other students, the performance approach goal, correlate with earning more badges. But the more badges earned by low-performing students also indicated less of an overall decrease in concern about their performance relative to other

14 S. Abramovich et al. students, the performance avoidance goal. Because we did not find these same patterns for high-performing students, we conclude that educational badges adhere to the research finding that prior knowledge can have a major influence on assessment performance (Dochy et al. 1999). Different badge types also affected different learners motivation. The motivation relationship between badges and motivation for low performing students was limited to participatory badges. Skill badges earned by the low-performing students did not correlate with the change in performance avoidance goals. Not only do we conclude that different types of badges will have different effects on student motivation to learn but we also conclude that different types of badges will also affect learning performance since motivation can predict future learning performance (Pajares et al. 2000). Additionally, we believe that our findings are compatible with the research finding that extrinsic motivators can have a negative influence on learning (Deci et al. 2001). Participation badges, earned eventually by all learners, had minimal explicit connection to individual measure of skill. Consequently, learners might not be able to make a connection between internal motivation and participatory badges, perceiving of them as only external motivators. Skill badges, awarded based on direct student performance, perhaps could be more easily connected to internal motivation, and thus considered by learners as an intrinsic motivator. As an alternative assessment technique, we conclude that the design specifics of educational badges in addition to the targeted students will be the main predictors of badge influence on learning motivation. The implication for instructional designers of badges is that they must consider the ability and motivations of learners when choosing what badges to include in their curricula. If badges are offered for learners who might not excel in the content area of the badges, then there is the potential, depending on the design of the badges, for a negative motivational effect. Lowering or removing the number of participatory badges or increasing the number of skill related badges might mitigate some of the negative motivational effects. Based on our opinion survey, we also conject that providing detail to learners about how to earn the badges and what actions by resulted in earning a badge will also mitigate some negative motivational effects while preserving the assessment goal of badges. We believe that our findings represent a significant first step in empirically establishing how educational badges affect learning motivation. There is little existing empirical work on the effects of badges, despite much theoretical work and considerable system design work in many contexts. However, more research is necessary to understand the broader motivational impacts of badges under different circumstances, across other kinds of badges, at other ages, and for different kinds of learning environments. Advocates of educational badges need to further understand the interplay between different type of learners and different type of badges. Future research should replicate our findings and examine additional differentiation between badges and learners as well as the impact of choosing which badges to earn. Additionally, our findings do not allow us to determine the temporal effects of badges on motivation. Consequently, we believe our findings exist as one piece of the large research base needed to understand educational badges. Acknowledgments Work on this project was funded through grants from the Digital Media and Learning competition and the Defense Advanced Research Projects Agency.

15 Are badges useful in education? References Abramovich, S., Higashi, R., Hunkele, T., Schunn, C., & Shoop, R. (2011). An Achievement System to Increase Achievement Motivation. Paper presented at the games learning society 7.0, Madison, WI. Alberts, B. (2010). An education that inspires. Science, 330(6003), 427. Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4(2), Beal, C., Qu, L., & Lee, H. (2008). Mathematics motivation and achievement as predictors of high school students guessing and help-seeking with instructional software. Journal of Computer Assisted Learning, 24(6), Clark, R. E., Howard, K., & Early, S. (2006). Motivational challenges experienced in highly complex learning environments. In J. Elen & R. E. Clark (Eds.), Handling complexity in learning environments: Theory and research (pp ). Oxford: Elsevier. Davidson CN (2011). Could badges for lifelong learning be our tipping point? Retrieved November 1, 2011, from Deci, E. L., Koestner, R., & Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin, 125(6), 627. Deci, E. L., Koestner, R., & Ryan, R. M. (2001). Extrinsic rewards and intrinsic motivation in education: Reconsidered once again. Review of Educational Research, 71(1), 1. Dickey, M. (2005). Engaging by design: How engagement strategies in popular computer and video games can inform instructional design. Educational Technology Research and Development, 53(2), Dochy, F., Segers, M., & Buehl, M. M. (1999). The relation between assessment practices and outcomes of studies: The case of research on prior knowledge. Review of Educational Research, 69(2), Elliot, A. J. (1999). Approach and avoidance motivation and achievement goals. Educational psychologist, 34(3), Elliot, A. J., Cury, F., Fryer, J. W., & Huguet, P. (2006). Achievement goals, self-handicapping, and performance attainment: A mediational analysis. Journal of Sport and Exercise Psychology, 28, Elliott, A. J., Shell, M. M., Henry, K. B., & Maier, M. A. (2005). Achievement goals, performance contingencies, and performance attainment: An experimental test. Journal of Educational Psychology, 97(4), 630. Midgley, C., Maehr, M. L., Hruda, L. Z., Anderman, E., Anderman, L., Freeman, K. E., et al. (2000). Manual for the patterns of adaptive learning scales (PALS). Ann Arbor: University of Michigan. Pajares, F., Britner, S. L., & Valiante, G. (2000). Relation between achievement goals and self-beliefs of middle school students in writing and science. Contemporary Educational Psychology, 25(4), Peer 2 Peer University, & Mozilla Foundation. (2011). An open badge system framework. Retrieved July 14, 2011, from Pintrich, P. R. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology, 95(4), Shepard, L. A. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), Silk, E. M., Higashi, R., Shoop, R., & Schunn, C. D. (2010). Designing technology activities that teach mathematics. The Technology Teacher, 69(4), Stiggins, R. J. (2002). Assessment crisis: The absence of assessment for learning. Phi Delta Kappan, 83(10), Stiggins, R. J. (2005). From formative assessment to assessment for learning: A path to success in standardsbased schools. The Phi Delta Kappan, 87(4), Weaver R & Junker B (2004). Model specification for cognitive assessment of proportional reasoning: Department of Statistics Technical Report. Wigfield, A. (1994). Expectancy-value theory of achievement motivation: A developmental perspective. Educational Psychology Review, 6(1), Wigfield, A., & Eccles, J. S. (2000). Expectancy-value theory of achievement motivation. Contemporary Educational Psychology, 25(1), Wolters, C. A. (2004). Advancing achievement goal theory: Using goal structures and goal orientations to predict students motivation, cognition, and achievement. Journal of Educational Psychology, 96(2), 236.

16 S. Abramovich et al. Samuel Abramovich is a Doctoral Student at the Learning Research and Development Center at the University of Pittsburgh. Prior to that he was a researcher at the Center for Technology in Education at Johns Hopkins University, a Technology Coordinator for a Boston Independent School, and a Social Studies Teacher. His most current research projects include assessing how Badges can be constructed to motivate and assess in a variety of in-school and out-of-school contexts and how large-scale online feedback impacts teacher practice. His past research looked at the use of games and virtual-reality to teach at-risk youth and how design-based instruction is best integrated with technology education. Christian Schunn is a Senior Scientist at the Learning Research and Development Center and a Professor of Psychology, Learning Sciences and Policy, and Intelligent Systems at the University of Pittsburgh. He directs a number of research projects in science, mathematics, and engineering education. This work includes studying expert engineering and science teams, building innovative technology-supported STEM curricula, and studying factors that influence student and teacher learning. Ross Higashi is a Robotics Education Specialist at the Robotics Academy, a Carnegie Mellon educational outreach program based at the National Robotics Engineering Center in Pittsburgh, PA. He works on the design of CS2N the computer science student network ( and co-leads the development of its badge system and cognitive tutored curricular units. He graduated with a B.S. in Logic and Computation from CMU, where he also studied computer science and human computer interaction.