Tuturistic Online Enggement Tutorials

Transcription

1 Engging with Mssive Online Courses Ashton Anderson Dniel Huttenlocher Jon Kleinberg Jure Leskovec Stnford University Cornell University Cornell University Stnford University {dph, ABSTRACT The Web hs enbled one of the most visible recent developments in eduction the deployment of mssive open online courses. With their globl rech nd often stggering enrollments, MOOCs hve the potentil to become mjor new mechnism for lerning. Despite this erly promise, however, MOOCs re still reltively unexplored nd poorly understood. In MOOC, ech student s complete interction with the course mterils tkes plce on the Web, thus providing record of lerner ctivity of unprecedented scle nd resolution. In this work, we use such trce dt to develop conceptul frmework for understnding how users currently engge with MOOCs. We develop txonomy of individul behvior, exmine the different behviorl ptterns of high- nd low-chieving students, nd investigte how forum prticiption reltes to other prts of the course. We lso report on lrge-scle deployment of bdges s incentives for enggement in MOOC, including rndomized experiments in which the presenttion of bdges ws vried cross subpopultions. We find tht mking bdges more slient produced increses in forum enggement. Ctegories nd Subject Descriptors: H.2.8 [Dtbse mngement]: Dtbse pplictions Dt mining. Keywords: MOOCs; online enggement; bdges. 1. INTRODUCTION Mssive open online courses, or MOOCs, hve recently grnered widespred public ttention for their potentil s new eductionl vehicle. There re now multiple MOOC pltforms (including the edx consortium, Courser nd Udcity) offering hundreds of courses, some of which hve hd hundreds of thousnds of students enrolled. Yet, despite their rpid development nd the high degree of interest they ve received, we still understnd remrkbly little bout how students engge in these courses. To reson bout MOOCs we generlly pply our intuitions from universitylevel courses in the offline world, thinking of students who enroll in course for credit or s n uditor, nd who prticipte on n ongoing bsis over 1-14 week time period. Yet there is rel- Copyright is held by the Interntionl World Wide Web Conference Committee (IW3C2). IW3C2 reserves the right to provide hyperlink to the uthor s site if the Mteril is used in electronic medi. WWW 14, April 7 11, 214, Seoul, Kore. ACM /14/4. tively little quntittive evidence to support or refute whether such intuitions hold in the cse of MOOCs. Understnding how students interct with MOOCs is crucil issue becuse it ffects how we evlute their efficcy nd how we design future online courses. For students who tret MOOCs like trditionl courses, which run t fixed pce over fixed time period, it mkes sense to tlk bout students flling behind or dropping out. But for students who might tret MOOCs s online reference works or textbooks, completely different set of expecttions would pply, in which the nturl interction style my consist of bursts of synchronous enggement nd selective smpling of content. For students who might use MOOCs to shrpen nd test their skills in n re, it would be resonble to see them undertking portions of the course work without ever viewing lecture content. Despite the high level of interest in both the populr press nd recent cdemic literture, there hve been reltively few quntittive studies of MOOC ctivity t lrge scle, nd reltively little understnding of different wys in which students my be engging with MOOCs (in Section 6 we review recent work in the reserch literture.) Without this understnding it hs been hrd to rigorously evlute either optimistic clims mde bout student experience in MOOCs, or concerns rised bout low rtes of completion. And to the extent tht different types of student behviors hve been necdotlly identified in MOOCs, it hs been hrd to ssess how prevlent they re. In this pper we propose frmework for understnding how students engge with mssive online courses, bsed on quntittive investigtions of student behvior in severl lrge Stnford University courses offered on Courser (one of the mjor MOOC pltforms). Students in these courses cn engge in rnge of ctivities wtching lectures, tking quizzes to test their understnding, working on ssignments or exms, nd engging in forum where they cn seek help, offer dvice, nd hve discussions. We find first of ll tht ech student s ctivities in course cn be usefully described by one of smll number of enggement styles, which we formlize in txonomy bsed on the reltive frequency of certin ctivities undertken by the student. These styles of enggement re quite different from one nother, nd underscore the point tht students disply smll number of recurring but distinct ptterns in how they engge with n online course. We lso consider the issue of performnce nd grdes in the course, nd show tht thinking bout grdes in terms of different styles of enggement cn shed light on how performnce is evluted. Our second min focus in this pper is the course forums, nd how to increse enggement in them. They provide interesting ptterns of interction, in which students engge not just with the course mteril but with ech other. To shift the wys in which students engge in the forums, we designed two lrge-scle inter- 687

2 ventions. In prticulr, we report on our deployment of bdges in the discussion forum of one of the lrgest MOOCs, with the bdges serving s incentives to increse forum ctivity. We find through rndomized experiment tht different wys of presenting the bdges to emphsize socil signls nd progress towrd milestones cn hve n effect on the level of enggement. We now give n overview of these two themes the styles of enggement, nd the interventions to modify enggement. A txonomy of enggement styles In chrcterizing the predominnt styles of enggement, we consider the two fundmentl ctivities of (i) viewing lecture nd (ii) hnding in n ssignment for credit. (Additionl ctivities including ungrded quizzes nd forum prticiption will be considered lter in the pper, but won t be explicitly used in ctegorizing enggement.) One bsic ttribute of ny given student s behvior is the extent to which her overll ctivity is blnced between these two modlities. A nturl wy to ddress this question of blnce between ctivities is to compute student s ssignment frction: of the totl number of lectures nd ssignments completed by the student, wht frction were ssignments? Thus, student with n ssignment frction of only viewed lectures, while student with n ssignment frction of 1 only hnded in ssignments, without viewing ny of the course content. We computed the ssignment frction for ech student in six Courser clsses: three successive offerings of Mchine Lerning (which we nme ML1,, nd ML3) nd three successive offerings of Probbilistic Grphicl Models (which we nme PGM1, PGM2, nd PGM3). Figure 1 shows histogrms of the number of students with ech ssignment frction in these six courses. It is striking tht the histogrm for ech course hs three nturl peks: left mode ner the frction, right mode ner the frction 1, nd centrl mode in between. This suggests three nturl styles of enggement, one clustered round ech mode. 1. Viewers, in the left mode of the plot, primrily wtch lectures, hnding in few if ny ssignments. 2. Solvers, in the right mode, primrily hnd in ssignments for grde, viewing few if ny lectures. 3. All-rounders, in the middle mode, blnce the wtching of lectures with the hnding in of ssignments. The number of distinct enggement styles is lrger, however, due to two other types of students who re not pprent from the three modes of the histogrm. First, subset of the students in the left mode re in fct downloding lectures rther thn viewing them on the site; this distinction is importnt becuse someone who only downlods content my or my not ever ctully look t it. We therefore seprtely define fourth style of enggement: 4. Collectors, lso in the left mode of the plot, primrily downlod lectures, hnding in few ssignments, if ny. Unlike Viewers they my or my not be ctully wtching the lectures. Finlly, there re students who do not pper in the plot, becuse they undertook very few ctivities: 5. Bystnders registered for the course but their totl ctivity is below very low threshold. These form our five styles of enggement: Viewers, Solvers, Allrounders, Collectors, nd Bystnders. While there re not shrp boundries between them, we see from the simple plot of reltive ctivity levels tht there re nturl behviorl modes tht correspond to these cses. Moreover, we cn formlize precise versions l (ML1) l () l (ML3) l (PGM1) l (PGM2) l (PGM3) Figure 1: Distribution of ssignment frction (rtio of ssignments,, to overll ctivity of ssignments nd lectures, + l). of these styles simply by defining nturl thresholds on the x-xis of the histogrms in Figure 1 (to define clusters round the three modes), then seprting out downloding from wtching lectures (to resolve the Collectors from the Viewers) nd seprting out students with very low totl ctivity levels (to identify the Bystnders). Alredy from this simple txonomy we cn see tht intuitions bout styles of student enggement from the offline world do not mp precisely onto the world of MOOCs. For instnce, those just viewing lectures re not necessrily the sme s uditors, becuse if they re downloding rther thn streming they re quite possibly collecting the lectures for future use rther thn wtching them, much s one would downlod n online textbook or other resource for possible future use. And we find severl other fine-grined distinctions even within the different types, lthough we do not seprtely distinguish them s distinct styles of enggement. For exmple, subset of the All-rounders first hnd in ll the ssignments nd then downlod ll the lectures. Such students re in sense first behving s Solvers, then subsequently s Collectors, but re certinly not prticipting in ny wy tht one might recognize s common in n offline course. Moreover, this rnge of enggement styles shows tht while the issue of students dropping out of MOOCs points to genuine nd importnt distinction in types of student ctivity, it is rgubly distinction being mde t too superficil level. Indeed, even sking whether student completes n online course is question lredy bsed on the ssumption tht there is single notion of completion. While our txonomy is bsed on ctivity trces nd does not record student s internl motivtions or intentions, we see tht students re reching very different kinds of completion in their interctions with these courses. In prticulr, the reltively lrge numbers of Viewers shows tht focusing primrily on those who engge with both the lecture content nd the grded ssignments, s the mjority of students do in offline courses, overlooks rel- 688

3 tively lrge portion of those who re ctively engged in MOOCs. Solvers similrly represent style of enggement tht is different from the underlying intuition in this re; mny of them re likely students who hve previously lerned the mteril or re lerning it elsewhere. Both these types of students re presumbly getting vlue from the course, bsed on the fct tht mny complete most of the ssignments or view (or downlod) most of the lectures. Enggement nd Grdes. Our nlysis of enggement styles lso reltes to the issue of students performnce nd grdes; here too we find tht one must be creful in dpting intuitions from offline courses. First, we find in ll the clsses tht most students receive grde of zero; however, the fct tht mny Viewers spend non-trivil mount of time wtching lectures mens tht grde of zero should not be equted with filure to invest ny effort in the course. Second, we see n intriguing difference between the ML nd PGM clsses; the PGM clsses, which re more chllenging, include students who do ll the coursework but hve wide vrition in their grdes (s one sees in chllenging off-line courses), wheres in the ML clsses we find tht student s grde hs more liner reltionship to the number of ssignments nd lectures. This suggests useful distinction between MOOCs in which student s grde is reflection primrily of effort expended (s in ML), versus differentil mstery of the work hnded in (s in PGM). Forums nd Bdges The second min focus in our work here is on the forums in online courses, nd the design of mechnisms to increse ctivity in them. Just s much of the discussion bout student behvior on MOOCs hs been bsed on intuitions imported from other domins such s off-line teching, much of the resoning bout online course forums to dte hs proceeded by nlogy with forum-like modlities in other online settings. The chllenge in drwing such nlogies, however, is tht forums re used in very diverse set of contexts cross the Web. Should we think of n online course forum s behving like discussion forum, where people engge in bcknd-forth interction; or like question-nswering site, where some people come with questions nd others try to nswer them; or like product review forum, where mny people ech individully rect to specific topic? Our nlysis shows tht threds in the course forums proceed in reltively stright-line fshion; ech thred grows much more through the rrivl of brnd-new contributors thn through repeted interction mong the initil contributors. Moreover, we find n interesting pttern in which the ctivity level nd verge grde of the student mking the initil post is substntilly lower thn tht of the students mking subsequent posts. This suggests tht we my be observing dynmic in which better students re helping out others in the clss by joining the threds they initite. Deploying Bdges in the Forums. We now discuss our work on designed interventions to shift the level of student ctivity. We did this in the context of forum ctivity, introducing bdges into the forum for the third offering of Courser s Mchine Lerning clss (ML3). Our bdges were bsed on milestones for reching certin ctivity levels bsed on contributing to threds, reding content, nd voting on content. When student reched one of these milestones she received bdge. It is interesting to note first of ll tht the ML3 course hd significntly higher enggement compred to erlier offerings of Mchine Lerning (ML1 nd ) in which bdges were not used. While we cnnot necessrily estblish tht this is due to the introduction of bdges, it is the cse tht the first two offerings of the Mchine Lerning course were extremely similr to ech other in their fo- Clss Students HWs Quizzes Lectures Posts Strt ML1 64, ,52 1,486,566 3,222,74 15,274 4/212 6,92 488,554 1,563,31 3,66,189 15,763 8/212 ML3 112, ,569 2,76,354 4,742,864 32,2 4/213 PGM1 3, , ,29 1,564,87 14,572 3/212 PGM2 34,693 21, ,29 1,59,464 7,44 9/212 PGM3 25,93 172, , ,899 4,32 7/213 Tble 1: Bsic course sttistics. Posts re forum posts nd Strt is when the clss strted. Clss Bystnder Viewer Collector All-rounder Solver ML1 28,623 (.47) (.25) 8,85 (.15) 8,67 (.13) 378 (.1) 27,948 (.49) 13,92 (.21) 7,314 (.11) 9,298 (.19) 55 (.1) ML3 62,2 (.54) 24,411 (.21) 15,282 (.13) 13,417 (.12) 786 (.1) PGM1 13,486 (.47) 6,742 (.23) 6,147 (.21) 2,365 (.8) 25 (.) PGM2 22,767 (.62) 6,689 (.18) 5,727 (.16) 1,57 (.4) 116 (.) PGM3 15,92 (.61) 4,816 (.19) 3,772 (.15) 1,287 (.5) 157 (.1) Tble 2: Number (frction) of students of different types. rum behvior, nd the third ws quite distinct: the distribution of forum contributions in ML3 developed much hevier til, nd group of high-volume forum contributors emerged who followed up much more ctively on initil posts. Moreover, the chnges in ML3 were much more prominent on forum ctivities where bdges were introduced, nd much less pronounced where they weren t. We were lso interested in whether different wys of mking the bdges slient to the students could produce different incentives nd hence different levels of forum ctivity, so we developed rndomized experiment tht presented the bdges differently to different sub-popultions of the students. As we discuss in Section 5, mking the bdges more slient produced n ggregte increse in forum ctivity; the strongest effect cme from design tht mde student s own progress towrd the bdge visible nd explicit, but we lso sw effects from more socil mechnisms for creting slience displying student s current set of bdges next to her or her nme for others to see. It is striking tht these reltively subtle differences produced non-trivil effect on forum contribution; students hd knowledge of nd ccess to the sme bdges in ll cses, nd the only difference ws the extent to which the bdge ws emphsized in the user interfce. Dt. As noted bove, our dt comes from six Stnford clsses offered on Courser: three successive offerings of Mchine Lerning (ML1-3) nd three of Probbilistic Grphicl Models (PGM1-3). ML3 is the course in which we dded bdge-bsed incentives to the forum. Tble 1 summrizes the number of ctions of ech type in ech of the three offerings of the two courses. 2. PATTERNS OF STUDENT ACTIVITY Enggement styles. We begin by putting numericl estimtes on the five styles of enggement outlined in the introduction. We will bse our estimtes on the structure of Figure 1, since the three modes of the depicted distributions were the initil motivtion behind our five ctegories: the left mode is comprised of mixture of Viewers nd Collectors; the middle mode of All-rounders; nd the right mode of Solvers. Finlly, students who only took very few ctions re Bystnders. Clerly there re not perfectly shrp distinctions between these ctegories, but we cn produce working pproximtion by simply dividing up the x-xis in Figure 1 into three intervls round the three modes. More concretely, we first define numericl thresholds 689

4 # of users # of course items engged with # of ssignment questions # of lectures # of registrnts Oct 212 Dec 212 Feb 213 Apr 213 Jun 213 Aug 213 Oct 213 Figure 2: () Distribution over number of course items (lectures nd ssignment questions) engged with in ML3; (b) Sctter plot of the sme dt. c 1 nd < θ < θ 1 < 1. Then, for given student who did ssignment questions nd consumed l lectures, we cll them: Bystnder if ( + l) c ; otherwise, they re Viewer or Collector if /( + l) θ ; depending on whether they primrily viewed or downloded lectures, respectively, n All-rounder if θ < /( + l) < θ 1; Solver if /( + l) θ 1. To produce the five sets, we choose the threshold c = 2, θ to be the midpoint between the left nd center mode, nd θ 1 to be the midpoint between the center nd right mode. The resulting sizes of the five ctegories re shown in Tble 2. (The prtition ppers to be reltively robust numericlly, in tht firly different pproches to dividing the students into these five ctegories produce roughly comprble numbers.) As noted erlier, lthough the Solvers re smll ctegory in reltive terms, it is nevertheless remrkble tht severl hundred such students should be present in these courses. These distinct modes of behvior show up in other wys of looking t student ctivity. Figure 2 shows simply the distribution of the totl number of items (ssignment questions nd lectures) engged with in the course. One immeditely notices two lrge spikes in the curve, locted t the totl number of lectures nd the totl number of lectures plus ssignment questions, respectively. We cn lso look t vritions in these ptterns of ctivity. Figure 2b depicts the following sctter plot for ML3 (plots for the other courses re similr): for ech student who hs ttempted ssignment questions nd consumed l lectures, we put point t (, l). Different styles of enggement occupy different prts of the (, l) plne, but severl other things stnd out cross ll the courses s well. First, there is dense prt of the sctter plot corresponding to digonl line of positive slope: in ggregte, nd l grow monotoniclly in one nother. Second, the horizontl stripes correspond to collection of students who ll stopped fter prticulr ssignment question. And third, there is corresponding lck of verticl stripes; this suggests tht students do not tend to stop t clerly defined brek-points in the sequence of lectures, s they do in the sequence of ssignments. Time of interction. Our clssifiction of students into five enggement styles is bsed on the totl number of ssignment questions they ttempted nd lectures they consumed. However, we lso find tht when student intercts with course is n importnt correlte of their behvior. In the Courser clsses we study, students could register months in dvnce, much like mny trditionl courses. But unlike most offline courses, registrtion ws often left open until months fter the Courser clsses ended. Figure 3 shows tht significnt frction of students mde use of this policy: Figure 3: CDF of registrtion times in ML3. Verticl brs indicte the course s strt nd end dtes. Frction of popultion Bystnder Viewer Collector All-rounder Solver Nov 212 Dec 212 Jn 213 Feb 213 Mr 213 Apr 213 My 213 Jun 213 Jul 213 Aug 213 Sep 213 Oct 213 Figure 4: How the distribution of enggement styles vries with registrtion time. Verticl brs indicte the course s strt nd end dtes. only bout 6% of the students registered before the clss officilly begn, nd 18% registered fter it ended. This implies tht significnt portion of the students re intercting with the clss solely fter it ends. We cll these students rcheologists. Precisely, student is n rcheologist if her first ction in the clss is fter the end dte of the clss (note tht student cn register for the clss t ny time nd still be n rcheologist, the only criterion is for her first ction nd thus ll her ctions to come fter the end dte). We keep this clssifiction of students orthogonl from the enggement styles described bove; wht ctions students tke, nd when they tke them, re seprte types of behvior. Tht so mny students re intercting with courses fter they end is n unexpected wy in which MOOCs differ from trditionl clsses. A student who registers months in dvnce my hve very different motivtions nd intentions from one who registers the dy before it begins, or one who signs up months fter it ends. In Figure 4, we plot the distribution of enggement styles s function of registrtion time. The effect of registrtion time on enggement is strikingly lrge: for exmple, the frction of Bystnders is s high s 7% for students who sign up six months erly, then sinks to just 35% round the clss s strt dte, nd rises gin to 6% in the months fter the clss ends. We lso see tht lte registrnts re more likely to be Viewers nd Collectors thn erly registrnts re. One cn lso exmine the distribution of enggement styles s function of first ction time. Doing this, we discover tht there is crucil moment in determining students enggement style: significnt mount of distributionl mss bruptly shifts from Allrounders to Bystnders nd Collectors the dy fter the first ssignment is due. 3. GRADES AND STUDENT ENGAGEMENT Hving considered the bsic styles nd temporl ptterns of enggement, we turn to n investigtion of the grdes tht students 69

5 Student grde Student rnk 7 Student grde PGM2 Student rnk Figure 5: Finl course grde distribution in nd PGM2. 35 # of users Zero grde # of lecture consumptions # of users High chievers # of lecture consumptions Figure 7: Lecture wtching behvior of students with finl course grde of (left) nd 1 (right). The plots shows dt for ML3 but we observe very similr behvior in ll courses. Assignment Q submissions Lecture consumptions PGM Finl grde Finl grde Quiz submissions Forum thred reds Finl grde Finl grde Figure 6: Medin number of ctions of students with given finl grde in PGM2 nd. receive for their work. In prticulr, we re interested in quntifying the reltion between student s grde nd the wy she engges with the course. We find tht student enggement ptterns re qulittively similr mong different versions of the sme clss, but there re interesting differences between ML nd PGM. Thus, without loss of generlity, in this section we will focus on the second itertions of both clsses, nd PGM2. Grde distribution. First we exmine the overll finl grde distribution in the two clsses, shown in Figure 5. Notice tht the grde distributions in the courses re hevily skewed; out of the 6, students who registered for, two-thirds of them ( 4,) get finl grde of, 1% of students ( 5,) chieve perfect score, nd the remining 2% of students ( 1,) receive grde in between these two extremes. This brekdown is stble cross ll the versions of ML. The distribution of grdes in PGM2 is even more skewed: there is no lrge contingent of students with perfect grde (only two students chieve this distinction), nd only 1% (3,3 out of 35, students) chieve non-zero grde. In both clsses there is lrge number of students who chieve score of zero in the course. However, this doesn t men tht these students re not engging or putting effort in the course. In fct, mny of the zero-grde students re Viewers who spend non-trivil mounts of time wtching lectures. Figure 7 shows the distribution of lecture views for zero-grde students. About 5% of the registered students wtch t lest one lecture, nd 35% wtch t lest 1. The spike in the figure t x = 12 corresponds to the totl number of lectures in the clss. Student s finl grde nd her enggement with the course. We now investigte how student s grde is relted to her enggement nd ctivity levels. Here we think of the finl grde s n independent vrible; our gol is not to predict student s grde from her ctivity but rther to gin insight into how high-grde nd lowgrde students distribute their ctivities differently cross the site. Figure 6 plots the medin number of ctions of given type users tke s function of their finl grde. Overll, the grde is generlly proportionl with their ctivity. In, the medin number of ctions of given type (ssignment submissions, quiz submissions, lectures viewed, forum thred views) linerly increse with the student s finl grde. PGM2 devites from this generl trend in n interesting wy: the liner reltionships only hold up to certin point. Clss ctivity increses until grde of round 8%, but then decreses. For exmple, PGM students with ner-perfect grdes wtch round the sme number of lectures s students who got 2%. Perhps this is due to PGM being highly technicl course where students who hve seen the content before hve n esier time, while others seem to struggle. Finlly, both courses devite from the liner trend in forum reding. We observe tht students with perfect grdes red less on the forums thn those with lower grdes (Figure 6d). Behvior of high-chievers. We now exmine the ctivity of highchievers, the students who scored in the top 1 th percentile of the clss. A key trit of high-chievers is tht they consume mny lectures. Most of them hve more lecture wtches thn there re lectures in the course (indicting some re-wtching behvior); lthough the mode of the distribution is t exctly the number of videos in, the distribution is skewed to the right (see Figure 7b). In the plot we lso clerly observe the popultion of Solvers, who wtch very few (or no) lectures. While lecture wtching is chrcteristic of high-chievers, the number of ssignment question ttempts, however, is surprisingly vrible (Figure 8). Out of totl of 42 ssignment questions, the men number of submissions high-chievers hnded in ws 57 (mode 42), while some submitted more thn 2. Although it is no surprise tht in order to finish the course with perfect grde one hs to submit ll the ssignments, it is interesting to observe the bimodl distribution of the number of quizzes students submitted (Figure 8b). Here we observe tht most students submitted round 12 quizzes, while the first mode of the distribution is t round 3. This is consistent with the explntion tht the popultion of nerperfect students is composed of two subgroups: Solvers, who perhps lredy know the mteril, nd All-rounders, who diligently wtch the lectures, finish the quizzes, nd do ssignments. 691

6 # of users # of users # of ssignment questions submitted # of quiz ttempts Figure 8: Number of hnded-in ssignments (left) nd quizzes (right) for high-chievers. # distinct contributors ML1 ML3 PGM1 PGM2 PGM Thred length Figure 9: Number of distinct contributors s function of thred length. 4. COURSE FORUM ACTIVITY We now move on to our second min focus of the pper, the forums, which provide mechnism for students to interct with ech other. Becuse Courser s forums re clenly seprted from the course mterils, students cn choose to consume the course content independently of the other students, or they cn lso communicte with their peers. Following our clssifiction of students into enggement styles, our first question is simple one: which types of students visit the forums? To nswer this, we compute the distribution of enggement styles for the popultion of students who red t lest one thred on ML3 (shown in the top row of Tble 3). The representtion of enggement styles on the forum is significntly different from the clss s whole, with more ctive students overrepresented on the forum; for exmple, Bystnders comprise over 5% of registered students but only 1% of the forum popultion. We lso compute the frction of ech enggement style present on the forum (the bottom row of Tble 3). It is striking tht 9% of All-rounders re forum reders, mening tht the two popultions hevily overlp. While numericlly the forum is used by smll frction of the full popultion of registered students, this is superficil mesure; using our enggement txonomy it is pprent tht lrge mjority of the most engged students re on the forum. The composition of threds. The forum is orgnized in sequence of threds: ech thred strts with n initil post from student, which is then potentilly followed by sequence of further posts. Threds cover vriety of topics: discussion of course content, question followed by proposed nswers, nd orgniztionl issues including ttempts by students to find study groups they cn join. Forum threds re feture of wide rnge of Web sites socil networking sites, news sites, question-nswer sites, product-review sites, tsk-oriented sites nd they re used quite differently cross domins. Thus one hs to be creful in dpting existing intuitions bout forums to the setting of online courses in principle it would Bystnder Viewer Collector All-rounder Solver P (S F ) P (F S) Tble 3: How enggement styles re distributed on the ML3 forum. P (S F ) is probbility of enggement style given forum presence (reding or writing to t lest one thred); P (F S) is probbility of forum presence given enggement style. be plusible to conjecture tht the forum might be plce where students engge in bck-nd-forth discussions bout course content, or plce where students sk questions tht other students nswer, or plce where students weigh in one fter nother on clss-relted issue. Our gol here is to develop n nlysis frmework tht cn clrify how the forums re in fct being used. In prticulr, we d like to ddress the following questions: Does the forum hve more converstionl structure, in which single student my contribute mny times to the sme thred s the converstion evolves, or more stright-line structure, in which most students contribute just once nd don t return? Does the forum consist of high-ctivity students who initite threds nd low-ctivity students who follow up, or re the threds initited by less centrl contributors nd then picked up by more ctive students? How do stronger nd weker students interct on the forum? Cn we identify fetures in the content of the posts tht indicte which students re likely to continue in the course nd which re likely to leve? The course forums contin mny threds of non-trivil length, nd we sk whether these threds re long becuse smll set of people re ech contributing mny times to long converstion, or whether they re long becuse lrge number of students re ech contributing roughly once. As first wy to ddress this question, we study the men number of distinct contributors in thred of length k, s function of k. If this number is close to k, it mens tht mny students re contributing; if it is constnt or slowly growing function of k, then smller set of students re contributing repetedly to the thred. We find tht the number of distinct contributors grows linerly in k (see Figure 9): thred with k posts hs roughly 2k/3 distinct contributors. Moreover, this slope is mrkedly consistent cross ll six courses in our dt. The liner growth in distinct contributors forms n interesting contrst with discussion-oriented sites; for exmple, on Twitter nd Fcebook, the number of distinct contributors in thred of length k grows sublinerly in k [9, 2]. Now, it is possible for the ctul number of distinct contributors to exhibit two modes; for exmple, long threds on Fcebook hve this multi-modl behvior, s long converstionl threds mong few users co-exist with guest-book style threds tht bring in mny users [2]. In our domin, however, we find single mode ner the men number of distinct users; long converstionl threds with very few contributors re extremely rre in these course forums. Properties of thred contributors. Even if we know the forum is dominted by threds with mny distinct contributors, there re still severl possible dynmics tht could be t work for exmple, top-down mechnism in which high-ctivity forum user strts the discussion, or n inititor-response mechnism in which less ctive user begins the thred nd more ctive users continue it. One wy to look t this question is to plot student s verge forum ctivity level s function of her position in the thred tht 692

7 Forum usge ML1 ML Thred position Course grde ML1 ML Thred position Figure 1: () Averge forum usge s function of thred position, (b) Averge grde s function of thred position (ML). is, let f(j) be the lifetime number of forum contributions of the student in position j in the thred, verged over ll threds. We do this in Figure 1, finding cler inititor-response structure in which the second contributor to the thred tends to hve the highest ctivity level, recting to the initil contribution of lower-ctivity student. This effect is especilly pronounced in ML3, where bdges were present; it suggests hypothesis tht s students sought bdges, common strtegy ws to rect quickly to new threds s they emerged nd mke the next contribution. In the ML clsses, we lso see reflection of this distinction between the thred inititor nd lter contributors when we look t the grdes of the thred contributors. In prticulr, let g(j) be the finl course grde of the student in position j in the thred, verged over ll threds. We find (Figure 1b) tht the inititor of thred tends to hve significntly lower grde thn the lter contributors to the thred. In the first nd second ML clsses, this ws prticulrly true of the second contributor; the effect ws reduced in the third ML clss, where the second contributor tended to be extremely ctive on the forum but less distinctive in terms of grdes. Overll, however, this suggests vluble structure to the forum, with stronger students responding to the posts of weker students, s one would hope to find in peer-interction eductionl setting. In the PGM clsses, the sme pttern holds for forum ctivity (the function f(j)), but the grde function g(j) is essentilly flt: the thred inititor hs roughly the sme grde s the lter contributors. It would be interesting to relte this to the other grde distinctions between the ML clsses nd PGM clsses tht we sw in Section 3. Lexicl nlysis. Finlly, we consider the content of the posts themselves. As discussed bove, bsic question is to understnd whether we cn nlyze student enggement bsed on wht they write in their posts cn we estimte student s eventul ctivity level from their erly forum posts? One wy to mke this question precise is s follows. Consider ll forum posts for the first two weeks of the course; nd for ech word w tht occurred sufficient number of times, let h(w) be the finl number of ssignments hnded in by student who uses the word w in their post, verged over ll posts in the first two weeks tht contin w. Do certin words hve prticulrly high or low vlues of h(w) reltive to the overll distribution? In ll six clsses, there ws consistent pttern to the extremes of h(w). For illustrtion, Tble 4 shows the words chieving the highest nd lowest h(w), over ll words w occurring t lest 15 times. The high end of h(w) contins number of terms suggesting the uthor is lredy fmilir with some course terminology, nd hence potentilly some of the course content. Perhps more interestingly, the low end of h(w) ws consistently chrcterized by two points. First, mny of the words with low Highest number of ssignments Lowest number of ssignments vlues, your, error, different, x, using, mtrix, cost, function, grdient que, de, computer, study, m, interested, I m, me, hello, new Tble 4: For ech word w, we look t the verge finl number of ssignments hnded in by students who use the word w in post in the first two weeks. The tble depicts the words whose usge is ssocited with the highest nd lowest verge number of ssignments submitted. h(w) re ssocited with students using the forum to form study groups or to find study prtners. Second, mny of the words with low h(w) re non-english words. This finding suggests tht the pltform could potentilly be improved by more effective mechnisms for helping students form study groups, nd mking the course more ccessible to spekers of lnguges other thn English. 5. A LARGE-SCALE BADGE EXPERIMENT We ve discussed mny fcets of the behvior we observe in our dt: the vrious types of enggement styles students exhibited, the grdes they received, nd their ctivity on the forums. Now we report on two lrge-scle interventions we crried out on the third run of Courser s Mchine Lerning clss (ML3). First, we designed nd implemented bdge system on the forums; nd second, we rn rndomized experiment tht slightly vried the presenttion of the bdges to different groups of users. As result of these two interventions, we find compelling evidence tht introducing bdge system significntly incresed forum prticiption, nd tht even smll chnges in presenttion cn produce non-trivil differences in incentives thereby helping to shed light on how bdges generte their incentive effects. We ll first discuss the bdge system we implemented, nd then conclude with the rndomized experiment. 5.1 Bdge system In previous work, we studied the role of bdges s incentives both through theoreticl model nd n nlysis of user behvior on Stck Overflow, question-nswering site with thred structure similr to wht we hve observed in the course forums here [1]. We studied milestone bdges, where users win bdges once they perform pre-specified mount of some ctivity. We found tht s users pproched the milestone for bdge, they steered their effort in directions tht helped chieve the bdge. In other words, the bdge ws producing n incentive tht guided their behvior. Bdges re similrly prominent feture of mny thriving online forums, nd re generlly viewed s producing incentives for prticiption. The extent to which they ctully do so, however, remins uncler. Are bdges indeed significnt drivers of enggement, or re they minly incidentl to behvior on the site? As Courser ws interested in incresing forum enggement, we hd the opportunity to design nd implement bdge system for ML3 s forums, which provided n idel setting to study this question. Bdge types. Our bdge system operted entirely on the ML3 forums; there ws no connection to course-relted ctions like doing ssignments or wtching lectures. We primrily used milestone bdges, the sme type we studied in our previous work, but we lso introduced some other types described below. One of the min design principles suggested by our prior work is tht suite of severl, less-vluble bdges trgeting the sme ction, crefully plced in reltion to ech other, cn be more effective s group thn one single super-bdge. Following this principle, s well s 693

8 some previous industry implementtions, we designed four bdge levels bronze, silver, gold, nd dimond which were ssocited with incresing milestones, nd were correspondingly difficult to ttin. We decided to wrd bdges for some ctions but not others. We wrded simple cumultive bdges for reding certin numbers of threds, nd voting on certin numbers of posts nd comments, but we wrded no such bdges for uthoring posts or threds. This ws to void incentivizing the cretion of low-qulity content solely to win bdges. To discourge low-qulity votes, users hd to wit smll period of time between votes, to prevent users from quickly voting on succession of posts or comments t rndom. Insted of cumultive bdges for uthoring posts nd threds, we insted implemented gret chievement bdges to incentivize highqulity content, which were wrded for uthoring posts or threds tht were up-voted certin number of times by other forum members. The bronze, silver, gold, nd dimond levels corresponded to incresing numbers of up-votes s milestones. We lso creted cumultive gret chievement bdges to rewrd users who consistently uthored high-qulity content, s judged by their peers. We clled post or thred good if it received t lest three up-votes, nd bronze, silver, good, nd dimond bdges were wrded for uthoring certin numbers of good posts nd threds. Finlly, we creted number of one-time bdges for different purposes. Community Member nd Forum Newbie simply welcomed users to the min course nd course forums respectively (nd introduced users to the bdge system). The Erly Bird bdge ws given for prticiption in the forums in the erly dys of the clss, to help the forums get off to good strt, nd the All-Str bdge ws given to users who were ctive on the forums during every week of the clss. All the bdges re shown in Figure 12c. Effects of the bdge system on forum enggement. As first contrst between the forums with nd without bdges, we nlyze the distribution of the number of ctions users took on ML3, nd compre it to the sme distributions in previous runs of the clss, ML1 nd, neither of which hd bdge system in plce. If bdges incentivized users to be more engged in the forums, then we should observe shift in the distribution of the number of ctions in ML3 towrd hevier til indicting tht more users took more ctions reltive to the number of users who took fewer ctions. As we discuss more fully below, we cnnot necessrily conclude tht the differences exhibited by ML3 reltive to ML1 nd re the result of the bdge system, but close look t the distinctions between the courses provides evidence for the impct of bdges. (And in the next subsection we discuss the results of our rndomized experiment within ML3, where we could vry the conditions more precisely.) In Figure 11, we show the complementry cumultive distribution function (CCDF) on votes, where the point (x, y) mens y frction of users voted t lest x times (s frction of the totl number of users who voted t lest once in tht run of the clss). We show the normlized vlues (the frction of students) insted of the bsolute mgnitudes (the number of students) becuse the different runs of Mchine Lerning hd different numbers of students. By normlizing ech curve by the totl number of students who voted t lest once in tht run, we enble comprisons between the shpes of the distributions, which is wht we re interested in. First, observe tht the distribution in ML3 is clerly more hevytiled thn the distributions of both ML1 nd, indicting tht lrger frction of users voted mny times in ML3 thn in ML1 nd. This indictes tht users were more engged in voting in ML3. For exmple, reltive to the number of students who voted t lest once, the frction of students who voted t lest 1 times ws Frction of users Frction of users ML1 ML () Number of votes (c) Number of posts (b) Number of reds (d) Number of comments Figure 11: Normlized CCDFs of ction counts. Enggement on ctions with cumultive bdges (voting, reding) is significntly higher in ML3 thn in ML1 nd ; enggement on ctions without bdges (posts, comments) is essentilly the sme cross the three runs. 1 times lrger in ML3 thn in the other two runs. Second, notice tht the distributions in ML1 nd re essentilly identicl, despite these two runs of the course hving been offered t different times. This suggests tht certin fetures of the distribution were stble prior to the striking difference exhibited by ML3, in which bdges were offered. Similrly, the distribution of threds viewed is strikingly different in ML3 compred to ML1 nd (see Figure 11b). Agin, reltive to the number of students who red t lest one thred, the number of students who red t lest 1 threds ws 1 times lrger in ML3 thn in the other two runs. Students were substntilly more engged in viewing threds nd voting on posts nd comments in the ML3 run thn they were in ML1 nd. Wht bout the other forum ctions tht were vilble to students? In Figure 11c, we show the distributions of the number of posts users uthored. Although they vry slightly, the distributions re lrgely similr. The differences between ML3 nd re on the sme scle s the differences between ML1 nd. And s cn be seen in Figure 11d, the differences in the distributions of the number of comments re even smller. Thus, the ctions tht didn t hve cumultive bdges, uthoring posts nd comments, didn t show qulittively significnt differences in enggement between the three runs of the clss. This is in strk contrst with the ctions with cumultive bdges, voting nd reding threds, where users were much more engged in ML3 thn in ML1 nd. Finlly, we exmine whether content qulity ws different in the version of the course with bdges, by compring the normlized CCDFs of votes per item. The distributions re gin very similr cross clsses if nything, the distribution in ML3 is slightly hevier thn the other two, suggesting posts were more likely to receive more votes in ML3. Thus, by this mesure of voting, post qulity didn t suffer in ML3. The observtionl comprisons in this section ren t sufficient to mke definitive clim tht the bdge system we designed nd 694

9 implemented ws responsible for the striking increse in user enggement on the forum. There could be other fctors, unobservble by us, tht re responsible. But there re severl consistent points in support of the hypothesis tht our bdge system plyed n importnt role. First, we compred our results to two seprte controls, the enggement levels on two previous runs of the clss. These controls were lwys very similr to ech other, which both vlidtes them s resonble controls nd renders less plusible lternte hypotheses positing tht different clss runs re incomprble. Second, nd even more striking, is tht the forum enggement on ML3 incresed in specific wys tht completely prlleled the portions of the course trgeted by the bdge system. The ctions with cumultive bdges, voting nd thred reding, sw big increses in enggement, wheres the ctions without cumultive bdges, post nd comment uthoring, didn t significntly chnge. Any lternte hypothesis thus requires some difference between ML3 nd the other two runs tht didn t exist between ML1 nd, nd furthermore this difference must hold for the trgeted ctions (voting nd thred reding) but not for the control ctions (post nd comment uthoring). We don t hve nturl lternte hypotheses for how these prticulr prts of the course chnged significntly while the others were unffected. 5.2 Bdge Presenttion Experiment A big open question from our previous work ws to understnd how nd why bdges were producing their incentive effects. Were users viewing the bdges s gols to be chieved for intrinsic personl resons? Or were they viewed s signls of socil sttus, with the incentive effect correspondingly coming from the bdge s visibility to others? These questions re importnt for determining how best to design bdge systems, but without the bility to experiment, there is no cler wy to distinguish mong these hypotheses. We performed n experiment in which we rndomly prtitioned the student popultion of ML3 nd presented them with bdges in slightly different wys. Some of the bdge presenttions were designed to mplify the personl gol-setting spect of bdges: the sequence of milestones nd the student s prtil progress were highlighted nd mde slient. Other spects of the bdge presenttions were designed to mplify their socil function: students bdges were displyed next to their nme, creting sense of socil sttus tht might hve motivted them nd/or other students. In the end, ll of the modified presenttions were quite subtle, nd none of them shifted the ctul milestones for chieving the bdges. Thus, there is no priori reson to be confident tht they should hve hd ny effect t ll. But in fct, we will see tht they produced non-trivil effects, nd for one of presenttions in prticulr they produced quite significnt effect. Bdge tretment conditions. We seek to study the effect of bdges on student behvior nd thus to investigte severl dimensions in the bdge design spce. We im to nswer questions including: Wht is the effect of the bdges on forum prticiption? Are students motivted more when they see their own bdges, or does seeing others bdges hve lrger impct? Does it help to keep the student informed bout their future prospects for erning bdges? With the bove questions in mind we designed the following three different tretment conditions, which re outlined in Figure 12: () Top byline: In the heder of Courser webpge student sw byline showing the counts of how mny different levels of bdges (bronze, silver, etc.) she hs lredy won. Figure 12 shows n exmple of the top byline of student who erned 1 bronze () Top-byline: bdge byline shown in heder of every pge: (b) Thred-byline: bdge byline displyed on every forum post: (c) Bdge-ldder: bdge ldder on student s profile pge: (d) Bdge-ldder control: bdge list on student s profile pge: Figure 12: Three bdge experimentl conditions (,b,c) nd the bdge-ldder control (d). bdge. This tretment ws imed to test whether students chnge their behvior when the bdges they lredy hve is more visible to them. Students in the control group sw the sme heder without the bdge byline. (b) Thred byline condition is similr to Top byline, except here every post on the forum is nnotted with the byline of bdges erned by the uthor of tht post. In this experimentl condition, bdges re more visible, nd re more strongly linked with user identity on the forum since nmes re lwys ccompnied by bdge counts (Figure 12b shows n exmple). Students in the control group didn t see ny bdge bylines ccompnying uthor nmes. (c) Bdge ldder tested the effect of keeping students informed bout their progress towrds bdges nd the bdges they cn win in the future. Figure 12c shows n exmple of bdge ldder, where we see bdges erned by the student, ll other locked bdges, s well s conditions for obtining them. In this experimentl condition, upon winning bdge the student would receive pop-up ( tost ) with congrtultory messge nd, crucilly, informtion bout how mny more ctions were needed to win the bdge t the next level (e.g. gold, fter winning silver bdge). In the control group, the student could only see the bdges she obtined so fr on her profile pge (see Figure 12d), nd the congrtultory messge displyed upon bdge win wsn t ccompnied with informtion bout future bdges. Overll, we creted 2 3 = 8 experimentl buckets of students: one for ech possible combintion of control nd tretments for the three independent experiments. Effect of bdge tretment conditions. Now we turn to mesuring the tretment effects nd their corresponding sttisticl signifi- 695