Social Comparisons and Contributions to Online Communities: A Field Experiment on MovieLens

Transcription

1 Socia Comparisons and Contributions to Onine Communities: A Fied Experiment on MovieLens Yan Cen F. Maxwe Harper Josep Konstan Serry Xin Li November 18, 2008 We woud ike to tank Race Croson, Jon Duffy, Caterine Ecke, Steffen Huck, Sara Kieser, Gueorgi Kossinets, Nancy Kotzian, Robert Kraut, Vai-Lam Mui, Jon List, Cares Pott, Tanya Rosenbat, Roberto Weber, and seminar participants at Arizona, Community Lab (communityab.org), Edinburg, Innsbruck, Pittsburg, Micigan, Texas A & M, Yaoo! Researc and ASSA 2006 (Boston, MA), ESA 2006 (Atanta, GA) for epfu discussions, two anonymous referees for teir constructive suggestions, and Xinzeng Si for exceent researc assistance. Te financia support from te Nationa Science Foundation troug grants no. IIS is gratefuy acknowedged. Cen: Scoo of Information, University of Micigan, 1075 Bea Avenue, Ann Arbor, MI Emai: yancen@umic.edu. Harper and Konstan: Department of Computer Science and Engineering, University of Minnesota, 200 Union Street SE, Minneapois, MN Emai: konstan@cs.umn.edu, arper@cs.umn.edu. Li: Scoo of Economic, Poitica and Poicy Sciences, University of Texas at Daas, 800 W. Campbe Road, Ricardson, Texas ; Emai: serry.xin.i@utdaas.edu. 1

2 Abstract We design a fied experiment to expore te use of socia comparison to increase contributions to an onine community. We find tat, after receiving beaviora information about te median user s tota number of movie ratings, users beow te median demonstrate a 530% increase in te number of monty movie ratings, wie tose above te median do not necessariy decrease teir ratings. Wen given outcome information about te average user s net benefit score, above-average users mainy engage in activities tat ep oters. Our findings suggest tat effective personaized socia information can increase te eve of pubic goods provision. Keywords: socia comparison, socia information, conformity, socia preference, pubic goods, embedded onine fied experiment JEL Cassifications: C93, H41 2

3 1 Introduction Wit te increasing popuarity of te Internet, information tecnoogy is canging te way we interact, entertain, communicate and consume. In onine communities, groups of peope meet to sare information, discuss mutua interests, pay games and carry out business. Users of communities suc as SourceForge (ttp://sourceforge.net/) and Wikipedia contribute information goods, wic are typicay sared as pubic goods. However, despite te popuarity of onine communities, many suc communities fai due to nonparticipation and under-contribution. For exampe, Buter (2001) found tat 50% of socia, obby, and work maiing ists ad no traffic over a 122 day period. Under-contribution is a probem even in active and successfu onine communities. For exampe, in MovieLens (ttp:// an onine movie recommendation website tat invites users to rate movies and, in return, makes personaized recommendations and predictions for movies te user as not aready rated, under-contribution is common. More tan 22% of te movies isted on te site ave fewer tan 40 ratings, so few tat te software cannot make accurate predictions about wic users woud ike tese movies (Cosey, Ludford and Terveen 2003). Simiary, Eureka, a Xerox Corporation onine information saring system, wic enabes its 20,000 wordwide customer service engineers to sare repair tips, aso suffers from under-contribution. Wie many service engineers downoad macine repair tips from Eureka, ony an estimated 20% ave submitted a vaidated tip to te system (Bobrow and Waen 2002). Many onine communities are popuated wit peripera users, wo observe te community and use te contents created by oters witout contributing to te community content or discussions. In te P2P fie saring site Gnutea, in 2000, 25% of users sare 98% of te content wie 66% of users sare noting (Adar and Huberman 2000). By 2005, 85% of users sare noting (Huges, Couson and Wakerdine 2005). Tus, a key caenge to te onine community designer is to motivate te peripera participants to become active contributors, and te core participants to sustain and improve teir contributions. To resove te probem of under-contribution, economists migt turn to te teories of incentivecompatibe mecanisms for pubic goods provision. However, most mecanism design teories regarding pubic goods rey on tax-subsidy scemes. 1 Tus, tey cannot be directy appied to onine communities, as tese communities rey on vountary participation and contribution of time and effort rater tan monetary transfers to encourage contributions. Furtermore, compared to traditiona communities, onine communities ave distinct caracteristics, wic give te mecanism designer a new set of options. Most notaby, te designer as more information tan is traditionay assumed in mecanism design teory, wic enabes er to personaize te motivation information to eac user. 2 For exampe, some software can track te 1 See Groves and Ledyard (1987) for a survey of te teoretica iterature and Cen (fortcoming) for a survey of te experimenta iterature. 2 In dominant strategy and Nas impementations, it is usuay assumed tat te designer knows noting about te 3

4 detaied activities of eac user, incuding a user s cick stream and a time stamp for eac activity. From tese data, te designer can infer important underying user preferences and te time cost of eac activity. Suc information as been used to target customers in e-commerce, as in Amazon.com s book recommendations. 3 In tis paper, we expore ow a designer migt be abe to motivate users to contribute contents to an onine community by using personaized socia information. Te idea tat socia information migt affect beavior is teorized in bot socia psycoogy and economics. In socia psycoogy, Festinger (1954) teorizes tat we compare ourseves to oters wo are better off for guidance, and to oters wo are worse off to increase our sef-esteem. We conjecture tat onine community participants aso ave a tendency to compare temseves to oters wen te socia information is avaiabe. Since Festinger s semina work, a arge body of iterature in socia psycoogy sows tat socia comparisons affect beavior, since individuas gain information on wat constitutes te rigt beavior in various contexts. Furtermore, socia comparison teory suggests tat peope ean toward socia comparisons in situations tat are ambiguous (see Buunk and Mussweier (2001), Sus, Martin and Weeer (2002) for recent surveys), a condition wic is true in many onine communities. Atoug we are not aware of a matematica formaization of socia comparison teory, tree specia cases of tis teory ave been formaized in economics. In te first case, wen information regarding prevaent beavior is avaiabe, peope exibit te tendency to copy tis beavior, a penomena referred to as conformity (Asc (1956), Akerof (1980), Jones (1984), Berneim (1994)). In te second case, wen outcome information regarding oter peope s payoffs is avaiabe, peope sow distributiona concerns, suc as inequaity aversion (Fer and Scmidt (1999), Boton and Ockenfes (2000)). In tis case, participants in te aboratory act to reduce payoff inequaities. A tird reated iterature mode interdependent preferences, were utiity functions depend not ony on te absoute vaue of consumption, but aso on eiter te average eve of consumption (Duesenberry (1949), Poak (1976)), or te ordina rank in te distribution of consumption (Frank (1985), Robson (1992), Hopkins and Kornienko (2004)). Samueson (2004) s evoutionary mode provides a justification for preferences tat incorporate reative consumption effects in order to compensate for incompete environmenta information. Most empirica studies of te impact of socia information in economic decision making are conducted in te aboratory, using variants of te dictator games (e.g., Cason and Mui (1998), Krupka and Weber (2005), Duffy and Kornienko (2007)), te utimatum bargaining games (e.g., underying distribution of preferences or te production tecnoogy, wie in Bayesian impementation, it is usuay assumed tat te designer knows te distribution of agent preferences, but not te reaization in individua agents. 3 For exampe, te book Toucing te Void (Simpson 1988), a mountain cimber s account of near deat in te Peruvian Andes, received good reviews and modest success wen it was first pubised, and was soon forgotten. Years ater, anoter mountain-cimbing tragedy, Into Tin Air (Krakauer 1999), became a pubising sensation. Amazon began to recommend Toucing te Void to readers wo bougt Into Tin Air. Eventuay Toucing te Void outsod Into Tin Air more tan two to one (Anderson 2004). 4

5 Knez and Camerer (1995), Duffy and Fetovic (1999), Bonet and Zeckauser (2004), Ho and Su (fortcoming)), or coordination games (Ecke and Wison 2006). A more detaied discussion of te findings is reported in Section 3.2. In comparison, two natura fied experiments examine te effects of socia information on contribution to pubic goods. A natura fied experiment provides a bridge between a aboratory experiment and direct fied observations (Harrison and List 2004). In a university fundraising campaign, Frey and Meier (2004) find tat information about te average contribution in te past as a significant impact on contribution. In contrast, Sang and Croson (2005) finds tat, in a pubic radio fundraising campaign, te most infuentia socia information is contribution beavior drawn from te 90t to 95t percentie. Like tese two studies, we designed a natura fied experiment to compare te effects of different types of socia information to motivate user contributions. Our study differs from tese two studies in bot te type of contributions (time vs. money) and te medium of impementation (onine vs. offine). We impement our experiment troug a combination of emai newsetters and direct modification of te MovieLens website. Furtermore, we personaize our socia information to separatey investigate te effects of socia information on underperformers and overperformers, and find tat tey are drasticay different. For designers of onine communities, te effects of personaization to motivate different types of users are important and tecnicay feasibe. Additionay, tis paper contributes to an emerging body of fied experiment iterature about eiciting participation in onine communities. For exampe, Beenen, Ling, Wang, Cang, Frankowski, Resnick and Kraut (2004) appy te socia psycoogy teories of socia oafing and goa setting (Karau and Wiiams 1993) to contributions in MovieLens. Tey find tat individuas contributed more wen tey were reminded of teir uniqueness and wen tey were given specific and caenging goas. Ludford, Cosey, Frankowski and Terveen (2004) conduct a fied experiment using MovieLens and test te effects of simiarity and perceived uniqueness on participation in discussion groups. Doakia, Bagozzi and Pearo (2004) use survey data to investigate te effects of group norms and socia identity on participation in onine communities, suc as web-based cat rooms and onine games. Lasty, Cosey, Frankowski, Terveen and Ried (2007) conduct a fied experiment wit SuggestBot, software tat recommends work to Wikipedia editors, sowing tat suggesting work consistent wit teir previous edits yieds significanty more tota work done. Common to a onine communities, a user s activities can ave bot private and pubic benefits. A user s benefit from performing an activity is caed er private benefit from tat activity. Tis private benefit incudes private consumption (e.g. aving a more ceany fied and easiy searcabe bookmarks on de.icio.us), or te fun of paying onine games. If, in addition, er activity aso benefits oters, i.e., it is non-rivarous and nonexcudabe, we say tat tis activity as pubic benefits. For exampe, in te de.icio.us community, adding bookmarks as ony private benefit if a user does not revea er bookmarks to oters. Se benefits from an easiy searcabe bookmark 5

6 system. If, in addition, se makes er bookmarks pubic, so tat oters can browse or subscribe, tey become a pubic good, and tus ave pubic benefits as we (Was and Rader 2007). We wi define tese terms more precisey in Section 4. In tis paper, we report a randomized fied experiment on MovieLens were we send users an emai newsetter wic directs tem to perform activities wit varying degrees of private and pubic benefits. Te newsetter contains one of two types of socia information: te median number of ratings or te net benefit score of an average user in er coort. 4 Te contro group receives information about ony teir own past rating beavior. We ten modify te interface for eac user, wit new sortcuts tat ead to different types of contributions, incuding rating popuar or rare movies, updating te database, inviting a buddy or just visiting MovieLens. We ten track user beavior for a mont after te reease of te newsetter. From tis experiment, we find tat, after receiving beaviora information about te median user s tota number of movie ratings, users beow te median ave a 530% increase in te number of monty movie ratings, wie tose above te median do not necessariy decrease teir ratings. Wen given outcome information about te average user s net benefit score, above-average users mainy engage in activities tat ep oters. Our findings suggest tat effective personaized socia information can increase te eve of pubic goods provision. Te rest of tis paper is organized as foows. In Section 2, we introduce MovieLens. In Section 3, we present our experimenta design. Section 4 presents a teoretica framework for onine recommender systems and a mode of socia comparison. Section 5 presents te resuts. In Section 6, we summarize te resuts and discuss teir impication in te design of onine communities. 2 MovieLens: An Overview MovieLens (ttp:// is an onine movie recommender system tat invites users to rate movies and in return makes personaized recommendations and predictions for movies te user as not aready rated. It is run by a researc group in te Department of Computer Science and Engineering at te University of Minnesota. It is one of te most popuar noncommercia movie recommender sites, and as been featured extensivey by Te New York Times, ABC News Nigtine, and Te New Yorker. Specificay, as of Apri 30, 2006, MovieLens as over 13 miion user ratings of 9043 movies. Tese ratings come from just over 100,000 users, of wom approximatey 15,000 were active witin te past year. Since most readers are famiiar wit Netfix, it is important to point out te main difference between te two sites. Unike Netfix, MovieLens does not ave any DVD renta service. To determine personaized recommendations, MovieLens uses coaborative fitering tecno- 4 Te net benefit score is rougy te difference between te benefit a user receives from using MovieLens, and te time and effort se puts in. It is expained more precisey in Section 4. 6

7 ogy an agoritmic approac to personay evauate items for users based on te opinions of bot tat user and te entire community of users. Te underying assumption for tis tecnoogy is tat tose wo agreed in te past tend to agree again in te future. Te agoritm matces togeter users wit simiar opinions about movies, and for eac user, generates a neigborood of oter ike-minded users. Personaized recommendations for eac user is generated from te ratings of tese neigbors. Appications of te coaborative fitering tecnoogy incude Amazon.com s book recommendation system (users wo bougt x aso bougt y), and Netfix s movie recommender system. 5 In an age of information exposion, a recommender system eps individuas find desired information. For exampe, in MovieLens, a user can ask MovieLens to recommend movies, eiter overa or witin a searc, and te site wi return a ist of movies tat fufi te user s searc criteria sorted in te order of tose te user is most ikey to enjoy. Aternativey, te user can enter specific movies and receive a prediction of enjoyment on a 1/2- to 5-star scae. MovieLens encourages users to rate movies tey ave seen. Rating as two significant benefits: (a) it improves te user s profie by giving te agoritm more information about te user, and tereby may improve te quaity of recommendations and predictions generated for er; and (b) it adds to te overa database of ratings, and terefore may improve te recommendations and predictions generated for oters. Terefore, rating is an impure pubic good. In rating movies, tere are distinctions in effort and vaue. Movie ratings ave a skewed distribution. 6 For exampe, te most popuar movie in te system, Pup Fiction, as been rated by neary 50,000 users. By contrast, te bottom ten movies ave zero ratings, and 75% of te movies in te system ave fewer tan 1100 ratings. Rating a rare movie 7 takes more work a user needs to identify from te database one tat se as seen, and most users ave seen very few of tem. Terefore, in te rating process, a user migt need to go troug many more screens of movie tites before finding one se as seen and can rate. On te oter and, rating a rare movie adds greater vaue to oters in te community. MovieLens currenty as penty of data from wic to recommend popuar movies, but sti needs more data to accuratey and personay recommend rare ones. Furtermore, popuar movies ave aready been seen and rated by many system users, and terefore wi not be recommended to tem, no matter ow many new ratings tey receive; in contrast, rare movies ave not been seen or rated by many users (by definition) and terefore may be recommended to neary a of te users in te system (depending on ow muc te system estimates tose users wi ike tem). Terefore, rating a rare movie generates iger pubic benefit 5 Te recent 1 Miion Doar Netfix Prize for improving te accuracy of its movie recommendations underscores te importance of recommendation quaity in onine business appications. Reed Hastings, te CEO of Netfix, beieves tat recommendations are one of Netfix s most important advantages, especiay for its non-bockbusters (Anderson 2006). 6 Te best fit distribution for te current movie ratings in te database is ognorma(2016.1, 17410), atoug te Komogorov-Smirnov test rejects it at te 5% eve. 7 In te experiment, we define a rare movie as one wit fewer tan 250 ratings. 7

8 tan rating a popuar movie. Te vaue to a user of rating a rare versus popuar movie is ess cear. A user s profie is made more accurate wen te user s rating: (a) differentiates te user s taste from oter tastes consistent wit er prior ratings, and (b) associates te user wit a different set of simiar users. Tis improved accuracy is greatest wen te movie being rated as a ig variance in ratings (i.e., many peope ike it, and many disike it), and wen tat movie aso as been rated by many oters. Rare movies can improve a profie by associating te user wit oters wo viewed tat movie, but rarey wi ave as arge an effect as rating a divisive popuar movie. 8 For an overview of te recommender systems and te tecnica detais of te metods, we refer te reader to Gediminas Adomavicius (2005). In sum, rating a popuar movie takes ess time tan rating a rare movie. Terefore, in our mode deveoped in Section 4, we wi assume tat te margina cost of rating a popuar movie is ess tan tat of rating a rare movie. Furtermore, wie te private benefit from rating a popuar versus a rare movie migt be movie-specific, te pubic benefit from rating a rare movie is muc iger. Terefore, in te experimenta design, we empasize te private benefit from rating a popuar movie, and te pubic benefit from rating a rare movie (Appendix A #2 and #3). In addition to rating movies, MovieLens users can contribute in oter ways to benefit temseves or te community as a woe. For exampe, users can invite a buddy into te system buddies are peope wo can coaborate by accessing eac oter s recommendations and by receiving joint recommendations (i.e., movies tey eac woud ike, and terefore migt wis to see togeter). Adding a buddy is a good way of enancing te user s experience (movies, and movie recommenders, are more fun wit a friend). However, ony 2500 MovieLens users (about 5%) ave buddies in te system. Inviting a buddy is primariy vauabe to te user ersef, toug bringing a new person to te community certainy benefits te community as a woe. In te mode in Section 4, we wi assume tat aving a buddy contributes to te nonrating fun of using MovieLens. We aso note tat of te users wo as a buddy, 69% as one buddy. 9 Terefore, in our mode, for simpicity, we wi assume tat te fun of aving a buddy is not determined by te number of buddies. Furtermore, we assume tat finding a buddy is costess, as it requires tat a user fis out te name and e-mai address of te person. Based on our knowedge of MovieLens, we conjecture tat neary a buddies are rea-ife acquaintances. More recenty, te movie database as been opened up to te community, 10 so users can ep 8 We recognize tat many popuar movies are not divisive, and terefore ave itte information-teoretic vaue to add to a profie. 9 Te distribution is te foowing: te proportion of users wit one buddy is 69.36%, two buddies 15.98%, tree buddies 6.33%, four and above 8.34%. 10 Prior to 2005, te database was maintained by a singe user, wo did a meticuous job of database entry, but was sow in getting new movies into te database. Te ist of user-suggested movies to be entered into te database was so ong tat it became a major source of dissatisfaction among users. 8

9 maintain te database by entering new movies directy into te database or by vaidating detais of existing entries (see Appendix A #5 for an exampe). Tis task provides itte direct benefit to te user, but instead benefits te community as a woe. Wie it is possibe tat some users migt fee tat updating te database is fun or get a warm gow from te act (Andreoni 1990), based on user feedback, we view tis activity as an atruistic service activity. 11 Consequenty, in our mode in Section 4, we make te simpifying assumption tat tere is no private benefit from te act of updating te database. We assume tat updating te database provides a pubic good to te community. Furtermore, in te experiment, we empasize te pubic benefit of tis activity by teing users tat updating te database wi improve te quaity of information in te system. In sum, MovieLens is representative of many onine communities in tat te underying coaborative fitering tecnoogy draws on user-provided information to serve eac individua user and te community as a woe. Te probem in suc a system is ow to motivate users to contribute to te (impure) pubic goods witout using monetary incentives. Tis study expores te effects of socia information to motivate users to contribute to te community. 3 Experimenta Design In June 2005, we aunced a fied study of 398 MovieLens users in order to test te effects of socia information on contribution beavior. In tis section, we describe our experimenta design. Our experiment focuses on te impact of a personaized emai newsetter sent to eac of te subjects. Te emai newsetter contained messages tat compared eac subject s rating or net benefit in MovieLens wit tat of oter users in te system. We aso conducted two onine surveys wit our subjects before and after te experiment. [Figure 1 about ere.] Figure 1 summarizes te experiment time ine. To determine te extent to wic members coud understand te content of our newsetters, we conducted 14 pone interviews wit Movie- Lens members before auncing te experiment. In genera, members were abe to understand te information in te emai newsetter. Tese 14 members were not incuded in te experiment. We refer to tis pase as te Newsetter Apa Test, wic is comparabe to a piot session in a aboratory experiment. As of 12:00:00 on June 14, 2005, tere were 100,366 users in MovieLens, a of wom finised te sign-up process, wic requires at east 15 movie ratings. 12 To soicit vounteers for te study, 11 Users commented tat te activity was boring. Wy can t you scrape data from IMBD? (Cosey, Frankowski, Kieser, Terveen and Ried 2005). 12 See ttp://movieens.umn.edu/join. 9

10 we decided to use te poo of MovieLens users wo ad ogged in between June 2004 and June 2005, wo ad rated at east 30 movies, and wo ad given us permission to send tem emai. We used te ogin and ratings criteria to ensure tat we coud cacuate a user s net benefit score, wic we wi expain in detai in Section 4. Of a users, 5488 met our seection criteria, among wom we randomy cose 1,966 users and sent eac one a recruiting emai. Te emai contained a ink to a web page wit a consent form. A tota of 629 users cicked on te emai ink, of wom 398 consented to participate in te study. 13 [Figure 2 about ere.] Figure 2 presents te number of users and average number of movie ratings (in brackets) of te entire popuation, tose wo met our criteria, tose wom we invited, and tose wo participated in our study. Using two-sided Komogorov-Smirnov test of te equaity of distributions, we find tat tose wo met our criteria rated significanty more movies tan tose wo did not (p < 0.01). Comparing tose wo were invited wit tose not invited, te number of ratings are not significanty different (p = 0.114). However, of tose wo were invited, users wo vountariy participated in te study rated significanty more movies tan tose wo did not participate (p < 0.01). In sum, users in our sampe rated significanty more movies compared to te popuation. Tis difference between te beavior of experiment vounteers and te popuation being studied is known in te experimenta iterature as te vounteer effects (Rosnow, Rosenta, Mc- Conocie and Arms 1969). 14 A study participants ad te cance to earn up to tree entries (by competing te two onine surveys and participating in te study) in a prize drawing ed at te concusion of te study. We awarded one $100, two $50, and five $20 cas prizes to participants at te end of te study. Using prize drawing is a standard metod to induce users to compete onine surveys (Bosnjak and Tuten 2003). In comparison, users oter activities on MovieLens, suc as rating movies and inviting buddies, are part of teir natura activities on te site, wic we do not need to infuence wit a prize. We coect user beaviora data during te mont before te recruiting emai was sent out (weeks 1-4 in Figure 1) wen beavior ad not been infuenced by any experimenta 13 Based on te post-experiment survey of te participants, 75% are mae, 91% ave at east coege education, and 76% are between age 20 and In genera, we are ikey to see vounteer effects in an onine community study for two reasons beyond tose experienced in offine studies. First, frequent users are more ikey to encounter advertisements or recruitment messages. We avoided tis caenge by recruiting troug emai, a mecanism troug wic eac user receives a singe invitation weter tat user visits oury or infrequenty. Second, oya users are more ikey to fee affinity for te community and terefore migt be more ikey to participate in studies conducted on te site. We recognize tis imitation, but randomy assigned users among conditions so tat a conditions woud be simiary biased. We recognize tat vounteer recruitment may imit our generaizabiity to tose simiar to te vounteers, but argue tis imitation is true in most simiar experiments. 10

11 stimuus, and after te personaized newsetter was sent out (weeks 7-10 in Figure 1), eaving out te recruiting and pre-survey period (weeks 5 and 6). 3.1 Pre-Experiment Survey Users wo consented to participate in tis study were immediatey redirected to an onine 10- question survey. Te first purpose of tis survey was to eicit users perceptions of teir benefits and costs from using MovieLens, using questions drawn from our earier study of onine recommender systems (Harper, Li, Cen and Konstan 2005). We used tese survey responses in combination wit information on participants istorica usage of MovieLens to compute net benefit scores for tose in te Net Benefit treatment. Te second purpose of tis survey was to discover ow users beieved tey compared wit oter users in te study, in terms of ow many movies tey rated and teir net benefit from using te system. 383 of te 398 subjects in te experiment competed tis survey. A copy of te pre-experiment survey is posted at ttp:// 3.2 Personaized Emai Newsetter and Modified MovieLens Interface Approximatey two weeks after sending te initia invitation to participate in te study, we sent a personaized emai newsetter to eac subject. We randomy divided te 398 subjects into te tree experimenta groups. A user s experimenta group determined te type of emai newsetter te user woud receive in te study. Te first treatment group, Rating Info, received a newsetter indicating ow many movies tey ad rated compared wit te median user in teir group. Te second treatment group, Net Benefit, received a newsetter indicating ow muc net benefit tey obtained from using MovieLens compared wit oter users. Finay, te Contro group received a newsetter wit ony information about teir own ratings profie. 15 Findings from socia psycoogy ave suggested tat peope are more responsive to comparisons wit peope saring simiar reated attributes tan to comparisons wit dissimiar oters (Sus, Martin and Weeer 2002). In our study, we oped to avoid comparing a new user wit users wo ad been using te system for years. Tus, we furter subdivided te Rating Info and te Net Benefit groups into tree membersip coorts, New, Mid and Od, based on a user s date of registration wit MovieLens. Tabe 1 presents te caracteristics of eac of te tree membersip coorts. Atoug we did not divide te contro group into coorts in te experiment, wenever a treatment group is compared to te contro in te anaysis, we compare te corresponding mem- 15 Te exception to te random assignment of users to experimenta groups is te 15 users wo did not compete te pre-experiment survey. Tey were assigned to te Rating Info and te Contro groups, as we did not ave te information to compute teir net benefit score. In subsequent anayses, we incude a 398 users. We repeat a anayses excuding te 15 users wo did not compete te pre-survey and find tat te main resuts sti od. 11

12 bersip coorts respectivey. In te two treatments, tere are approximatey equa number of users in eac coort. Te numbers in brackets are te number of active users wo rated movies, updated te database or invited a buddy during te two-mont period of data coection, i.e., te monts before te recruiting emai and after te newsetter was sent out. [Tabe 1 about ere.] A tree newsetters are simiar in design. Eac is formatted in tm, atoug users wit textony emai cients received a text-ony version. 16 Eac design contained a eader, wit te Movie- Lens ogo, and some statistics about te number of MovieLens members, movies, and ratings. Beow te eader, tere were tree sections. Te first section contained personaized information according to te subject s experimenta group, as described beow. Te second section contained a sort news item about recent feature additions to MovieLens. Te fina section was a reminder about te researc study prizes. Sampe emai newsetters are incuded in Appendix A (#1 - #3). Te first section of te newsetter, wic contained personaized information about te subject, was te source of our experimenta manipuation. Wie a tree experimenta groups received different types of personaized information, a of te newsetters contained te same five inks: (1) rate popuar movies, (2) rate rare movies, (3) invite a buddy to use MovieLens, (4) ep us update te MovieLens database, and (5) just visit te MovieLens ome page. Tese inks were carified by neigboring text tat expained te effect of tese actions on a subject s own as we as oters experience in MovieLens. For exampe, te ink rate rare movies was foowed by te text rating rare movies wi ep oters get more movie recommendations. Wie a contained te same inks, te inks were grouped differenty according to te experimenta condition. Furtermore, depending on te participant s experimenta group, te emai contained one of tese additiona messages. Subjects in te Rating Info treatment received a message about ow many movies tey ad rated compared wit oter users in teir coort. Teir newsetter contained te foowing text: Ever wondered ow many movies you ve rated compared wit oter users ike you? You ave rated [ ] movies. Compared wit oter users wo joined MovieLens around te same time as you, you ve rated [more, fewer, about as many] movies tan te median (te median number of ratings is [ ]). Two main options foowed tis text, randomy ordered. One main option was to rate more movies, foowed by te inks to rate popuar movies and to rate rare movies. Te oter main option was to try new MovieLens features. Under tis eading we provided two inks, one to 16 Eac was sent in dua format, tm and text-ony. Te emai cient of te user automaticay cose wic one to dispay. 12

13 invite a buddy to use MovieLens and anoter to ep maintain te MovieLens database, again randomy ordered. Beow tese inks was te ink to te MovieLens ome page. Participants in te Net Benefit treatment received a message empasizing teir net benefits from using MovieLens compared wit te net benefits of oter users. Teir newsetter contained te foowing text: We ave cacuated te net benefit tat you get from MovieLens, a measure of te enjoyment and te vaue you receive minus te time and effort you put in. Your net benefit score is [ ]. Compared wit oter users wo joined MovieLens around te same time as you, your net benefit is [above, beow, about] average (te average net benefit score is [ ]). In a footnote in te emai newsetter, we expain te concept of net benefit: Te net benefit score is a measure of te tota benefit you receive from using MovieLens minus te time and effort you put in. Te tota benefit you receive incudes te vaue of movie recommendations you get from MovieLens, and your enjoyment from rating movies and oter fun activities, suc as browsing movies. Tis score is computed by using a matematica mode constructed in one of our earier studies. Te information used incudes your activities on MovieLens and your responses to reated questions in te survey. Te score ranges from 60 to 90. Tis score is cacuated based on Equation (1) in Section 4. Te benefit to a user maps into te private benefit a user receives from te system. We again provided two main options, randomy ordered. One main option was to increase your net benefit score, foowed by te inks to invite a buddy to use MovieLens and to rate popuar movies, randomy ordered. Te oter main option was to ep oters increase teir net benefit scores, foowed by inks to ep maintain te MovieLens database and to rate rare movies, again randomy ordered. Beow tese inks was te ink to te MovieLens ome page. An important design decision is te type of socia information provided in te experiment. In oter studies of socia comparison, different socia information as been seected and presented to te participants. Severa studies present te decision(s) of one oter participant and find mixed resuts. Cason and Mui (1998) find tat, in sequentia dictator games, atoug observation of beavior of one oter participant constraints subjects from moving towards sef-regarding coices, te effect is modest as beavior of one randomy cosen oter migt not cange individua beiefs about wat constitutes te appropriate beavior. Duffy and Fetovic (1999) find tat observation of beavior of one randomy cosen pair infuences beavior in different ways in te repeated utimatum and best-sot games. In a coordination game in Ecke and Wison (2006), observation of te move of one payer affects beavior of oter payers ony wen tis payer as ig status. In comparison, in te pubic radio fundraising fied experiment, Sang and Croson (2005) find tat te most infuentia socia information is contribution beavior of a donor drawn from te 90t to 13

14 95t percentie of previous contributions, atoug participants do not know te percentie of te comparison target. A second type of socia information is te compete ranking of a participants, suc as in Duffy and Kornienko (2007), wo find tat suc ranking information as significant effects on giving in dictator games, owever, it migt not be appicabe to a arge popuation suc as tat in our experiment. Finay, Bonet and Zeckauser (2004) present te average offer in utimatum bargaining games and find tat tis information activates te socia norm of equa spit. In a university fundraising fied experiment, Frey and Meier (2004) aso present information about te average contribution beavior of te student popuation in te past and find significant impact on contribution. In a cosey reated study of binary dictator games, Krupka and Weber (2005) et subjects observe te decisions of four payers from previous experiments and find a significant jump in saring wen te number of te oter payers wo sare increases from two to tree, consistent wit te effect of a socia norm. Based on findings in oter studies and te pubic goods nature of our experiment, we coose te median or average as te socia information presented to our participants. Note tat, in te Rating Info treatment, we use te median rating as te socia information rater tan te mean, as te distribution of te number of ratings is rigt skewed due to te presence of power users. Using te median rating rater tan te average rating ensures comparabe sampe sizes across above-, about, and beow-median groups and across membersip coorts. More importanty, information about te median aows users to infer te beaviors of te numerica majority used in conformity teory. In contrast, in te Net Benefit treatment, we use te average net benefit score, as te distribution of te net benefit scores is symmetricay centered. As a resut, te medians and te averages are amost te same across te tree membersip coorts of participants. Based on te resuts of our apa test, most of MovieLens users understood te concept of median, and ad intuitive knowedge about ow to interpret net benefit scores. A of tem understood te comparison of teir standing reative to tat of teir coorts. Finay, te Contro group received a message about teir participation in MovieLens witout any comparison to oter users. Teir newsetter contained te foowing text: Here are some statistics about your ratings beavior for one popuar movie genre. About [ ] of te movies tat you ve rated are comedies. Your average rating in tis genre is [ ]. Tis message was foowed by te same five inks and expanations offered to te Rating Info and Net Benefit treatments, atoug te inks were not grouped. Te order of te first four inks was randomized, wit te ink to visit te MovieLens ome page at te bottom. Comparing te treatments and contro newsetters, we note tat tey differ in more tan te socia information dimension. For exampe, in te Net Benefit treatment, te activity inks are proceeded by To increase your net benefit score or To ep oters increase teir net benefit 14

15 score, wie in te contro, tey are proceeded by Interested in getting more out of MovieLens? Here are some options. Tese different wordings were crafted to make te newsetter ook and fee natura to participants of our fied experiment. However, we do not rue out te possibiity tat tey migt cange ow peope interpret tese options. Consequenty, te comparison between a treatment and te contro migt be affected by tis confound. In contrast, te comparisons of users in te same treatment, e.g., above and beow average users in te Net Benefit treatment, are immune from tese potentia confounds. Subjects wo visited MovieLens, eiter by cicking on te newsetter s inks or oterwise, were given a sigty modified interface wit te four inks from te emai newsetter incuded in te sortcuts pane of te main MovieLens interface - visibe from eac page in te system (Appendix A #4). Tese four inks beaved exacty as tey did in te emai, but were ogged differenty so tat we coud differentiate between te different types of actions. Foowing sortcut conventions at MovieLens, te inks on te site were not annotated wit expanatory information. 3.3 Post-Experiment Survey We waited for one mont after we sent te emai newsetter to give te subjects a cance to use te system. At te concusion of te mont, we emaied te users again wit an invitation to take a second survey. Tis survey incuded MovieLens reated questions, questions modified from te Genera Socia Survey, te Big Five personaity survey, 17 and questions on demograpics. 310 of te subjects (78%) competed tis survey. A copy of te survey is posted at ttp:// 4 A Teoretica Framework In tis section, we first set up a static mode of onine recommender systems, wic extends te one deveoped in Harper et a. (2005) by incorporating new MovieLens features. We ten extend te static mode into a two-period mode wic incorporates socia comparisons based on our experimenta design. Te teoretica mode produces a set of ypoteses for te experiment. 4.1 A Static Mode We first outine a static mode in te neocassica framework wit sef-interested agents. Tis mode is appropriate for an onine community were socia information as been argey unavaiabe before te impementation of our experiment. Te MovieLens community is entirey virtua and neary anonymous. Unti recenty, users were not made aware of te presence of oters, except 17 Te Big Five measures five broad dimensions of personaity (Godberg 1993). It is now among te most widey accepted and used modes of personaity. 15

16 troug teir imited understanding of te recommendation process. For most users, tis recommendation system is a too tat eps tem keep track of, find, and recommend movies. 18 Terefore, absent of socia information, a neocassica mode captures te basic features and motivations in te MovieLens community. In our mode, tere are n users. Let X i be te tota number of ratings from user i, and X i = X p i + X r i, were X p i and X r i are te number of popuar and rare movies 19 user i as rated respectivey. Let d i be te number of movie entries updated by user i. Let d = n i=1 d i be te tota number of vaidated movie entries in te database, wic is a pubic good. Based on survey data (Harper et a. 2005), a user s benefit from using MovieLens comes from tree sources. Te most important benefit is te quaity of te movie recommendations, Q i (X i, j i X j), wic depends on one s own ratings tat te agoritm uses to infer a user s taste, and te stock of ratings in te system. Based on te caracteristics of te agoritm, we assume tat Q i (, ) is concave in bot its components, i.e., more ratings from a user increase te quaity of er recommendations, but at a decreasing rate. More tota ratings by oters in te system aso increase te quaity of recommendations, at a decreasing rate. For anaytica tractabiity, we assume tat Q i (, ) is additivey separabe. We denote te margina benefit from te quaity of recommendations as γ i. Te second source of benefit comes from rating fun, f i (X i ), as identified by te enjoyment derived from rating movies and voicing opinions. We assume tat f ( ) > 0, and f ( ) 0. Finay, users may aso enjoy non-rating activities, i, incuding enjoyment from browsing and aving a buddy. As expained in Section 2, for simpicity, we assume tat non-rating fun is independent of te number of webpages browsed, or te number of buddies, as 69% of te users ave ony one buddy. We aso assume tat finding a buddy is costess, because to invite a buddy invoves cicking a ink, and fiing out te name and emai address of te person, ten cicking a Submit button. As we opened up te database for te experiment, we add a fourt component of benefit derived from a vaidated database, v i (d), were v i ( ) is concave and twice continuousy differentiabe. For simpicity, we assume tat tis benefit is not determined by te quaity of te database. In our mode, we furter assume tat tere is a cost associated wit rating movies. Te (tota) cost function of rating movies, c i (X i ), measures te amount of time tat agent i needs to rate X i movies. Assume c i (X i ) is convex, i.e., te margina cost is positive, c i(x i ) > 0, and c i (X i ) 0 for a i N. Tis assumption captures te feature tat te margina cost of rating eiter remains constant or increases wit te number of ratings. A distinction between popuar and rare movies is tat te margina cost of rating a popuar movie is ess tan tat of rating a rare movie, i.e., dc i /dx p i < dc i /dx r i. Simiary, we assume tat te cost of updating te database is c d i (d i ), were 18 Since te experiment described in tis paper, a socia tagging system as been added to te site, wic increases te opportunity for socia visibiity. 19 Reca tat, in te experiment, we define a rare movie as one wit fewer tan 250 ratings. 16

17 c d i ( ) is aso convex. 20 Taking into consideration a benefits and costs of using MovieLens, we specify a user s neocassica utiity function as (1) π i (X i, j i X j ) = γ i Q i (X i, j i X j ) + f i (X i ) + i + v i (d) c i (X i ) c d i (d i ). We assume additive separabiity to get a cose-form soution for our empirica anaysis (Harper et a. 2005), were we caibrate Equation (1) wit survey and beaviora data, and parameterize various components of te benefit function. In our experiment, we use Equation (1) to compute a user s net benefit score from using MovieLens. 21 Reca tat in te pre-experiment survey of tis study, we eicited users perception on monetary cost and benefit of using te Movienens system. For exampe, subjects were asked about ow muc time it takes tem to searc for and rate 10 movies, ow muc tey are wiing to be paid to rate 10 movies for te system, and ow muc tey are wiing to pay for a ist of te top-ten movies tat MovieLens recommends. 22 Te cost and benefit information aong wit te data on user rating beaviors is fed into te parameterized mode to estimate user net benefit scores in tis study. Te scores presented in te newsetter are normaized so tey fa into te range. In wat foows, we extend te static neocassica mode to a two-period mode wic incorporates te two different kinds of socia information in our experiment treatments, and derive teoretica predictions for te experiment. 4.2 Beaviora Comparison: Rating Info Treatment We first extend te mode to incorporate te effect of socia information on beavior. Reca, in te Rating Info treatment, we give eac participant information about er own number of movie ratings and te number of ratings by te median user in er membersip coort. Based on te 20 Based on te time stamp of activities in our experimenta ogfies, we find tat rating a popuar movie takes a median user 9 seconds (based on 537 movie rating events), wie rating a rare movie takes a median user 11 seconds (based on 30 movie rating events). Note tat te atter migt be an underestimate of te actua time cost because of te sma sampe size. Updating a database entry, owever, takes a median user 90 seconds (based on 348 events). 21 We set d i = 0 wen computing te net benefit score prior to te start of te experiment, and used te number of database entries during te mont of te experiment wen computing te score at te end of te experiment. 22 Te exact wording of tese questions are How muc time do you tink it woud take you to searc for and rate ten movies tat you ave not yet rated in MovieLens?, Hypoteticay, if MovieLens paid peope to rate movies, ow muc money woud MovieLens ave to pay you to rate ten additiona movies?, and Hypoteticay, if you ived in a word were tere were no free movie review web sites, ow muc money woud you pay for a ist of te top-ten movies tat MovieLens tinks you woud ike to see? (Note: we ave no intention of ever carging MovieLens users for te services tat we provide!). See ttp:// for te compete pre-experiment survey. 17

18 socia comparison teory, and conformity teory in particuar, we expect tat tis information wi ave an effect on user beavior. Matematica modes of conformity eiter directy assume disutiity from non-conforming beavior (Akerof 1980) or derive equiibrium beavior from a signaing mode (Berneim 1994) were users care about teir intrinsic utiity as we as teir status. In a pooing equiibrium, wen status is sufficienty important, individuas wit eterogeneous preferences conform to a omogeneous standard of beavior. In tis subsection, we extend Akerof s (1980) reduced form mode to caracterize te effect of beaviora comparison wit te median user on individua beavior. In tis mode, te basic unit of time is one mont. Suppose te newsetter is reeased at te end of mont t. After te reease, users ave information to compare temseves wit te median user in teir coort. Let x τ i be user i s tota number of ratings in mont τ. Ten X t i = t τ=1 xτ i is te tota number of ratings from user i up to time t. Let X t m be te tota number of ratings from te median user at time t. We anayze te beaviora data in te mont foowing te reease of te newsetter, x t+1 i, and compare tis data to tat in te mont before, x t i. Wit a sigt abuse of notation, we use πi t to denote te net benefit in period t. A user s utiity function after earning te median user s rating information can be expressed as foows, (2) u i (X t+1 i, X t+1 j, Xm t+1 ) = π t+1 i g i ( X t+1 i Xm t+1 ), j i were (3) π t+1 i = γ i Q i (X t+1 i, j i X t+1 j ) + f i (x t+1 i ) + i + v i (d t+1 ) c i (x t+1 i ) c d i (d t+1 i ), and were g i ( ) captures te disutiity from deviating from te socia norm. We assume tat g i ( ) 0, for i m, indicating tat a user is eiter indifferent or suffers disutiity from deviating from te socia norm. We furter assume tat tis disutiity weaky increases wit greater deviation from te norm, i.e., g i( ) 0. Wie Equation (2) migt not be te most genera functiona form wic captures te effects of socia comparison, it maps into our experimenta design te best. In subsequent discussions, we index a user beow te median in te number of ratings as, and one above te median as. Observation 1. Comparing rating beavior in te mont before and after te reease of te newsetter, we ave te foowing resuts: (a) Te median user s beavior remains te same, i.e., x t+1 m = x t m, or x m = 0; 18

19 (b) Users beow te median wi rate more movies in te mont after compared to te mont before, i.e., x t+1 x t, or x 0; (c) Users above te median wi rate fewer movies in te mont after compared to te mont before, i.e., x t+1 x t, or x 0; and (d) Users in te contro group wi rate te same number of movies in te mont after compared to te mont before, i.e., x t+1 c = x t c, or x c = 0; Proof: See Appendix B. Observation 1 compares eac group s rating beavior in te mont after te newsetter wit its beavior in te mont before. Teory predicts tat users from bot ends of te spectrum wi cange teir rating beaviors. In our teoretica framework, users in te contro group do not receive any socia information about ratings, so teir rating beavior remains te same. However, in reaity, tere migt be spurious events not captured in our mode wic can cause te rating beavior of users to cange. An anaysis metod to address tis issue is to compare te difference in beavior in te treatment wit tat in te contro groups. Tis Observation provides a teoretica bencmark for suc anaysis in Section 5. Observation 1 is a common prediction tat can be made by severa teories, incuding te informationa socia comparison teory (Samueson 2004), conformity (Akerof (1980), Berneim (1994)), ancoring and priming, and mimicking. We present it in te context of MovieLens for competeness and for stating testabe ypoteses for our experiment. oter. In te foowing proposition, we compare te groups witin te Rating Info treatment wit eac Proposition 1. Wen conforming to te new socia norm is sufficienty important, i.e., wen g i( ) is sufficienty arge, (a) Users beow te median wi rate at east as many movies as te median user in te mont after receiving te newsetter, or x t+1 x t+1 m ; (b) Users above te median wi rate at most as many movies as te median user in te mont after receiving te newsetter, or x t+1 x t+1 m. (c) At te aggregate eve, we soud observe conformity to te median, X t+1 Xi t Xm. t Proof: See Appendix B. i Xm t+1 Proposition 1 indicates tat, if conforming to te socia norm is sufficienty important, te distance between a user s tota number of ratings and te tota number of ratings of te median user at time t + 1 is no greater tan te distance at time t wen te newsetter was reeased. In oter words, we expect te distribution to be tigter after te reease of te median rating information. Togeter, Observation 1 and Proposition 1 provide a teoretica bencmark for te data anaysis of our Rating Info treatment. 19

20 4.3 Outcome Comparison: Net Benefit Treatment In contrast to te Rating Info treatment, were te information regarding a median user s beavior is presented, in te Net Benefit treatment, we present te outcome information, i.e., te user s own net benefit score and tat of te average user. In te socia psycoogy iterature on socia comparison, peope compare temseves to oters in bot te beavior and outcome dimensions (e.g., Sus et a. (2002), Lockwood and Kunda (1997)). We now extend te mode deveoped in Subsection 4.2 to te comparison in outcomes, i.e., te net benefit scores. Rewriting Equation (2) in te outcome space, we get (4) u t+1 i = π t+1 i g i ( π t+1 i π t+1 a ). Again, g i ( ) captures te disutiity from deviating from te socia norm, i.e., te average net benefit score, wit te same assumptions on te properties of g i ( ). To avoid excessive notation, we use a, and to index users wit net benefit scores about, beow and above average, respectivey. We. first ook at an average user, i.e., π a = π. Equation 4 impies tat se maximizes er neocassica utiity function, (5) u t+1 a = π t+1 a. For a user wit a net benefit score beow average, er utiity function is: (6) u t+1 = π t+1 g (π t+1 a π t+1 ). Terefore, wen se is beow average, se suffers disutiity wic is increasing in te distance between er net benefit and te average user s net benefit. For a user wit a net benefit score above average, er utiity function is: (7) u t+1 = π t+1 g (π t+1 π t+1 a ). Terefore, wen se is above average, se again suffers disutiity wic is increasing in te distance between er net benefit and te average user s net benefit. In a specia case wen g i ( ) is inear, our mode as te same functiona form as te inequaity aversion mode of Fer and Scmidt (1999). 23 In tis case, te coefficient, g in Equation (6), can be interpreted as te degree to wic user envies te average user, wie te coefficient, g in Equation (7), can be interpreted as te degree of a user s carity concerns. Wie te Fer and 23 We do not attempt to review te arge iterature on socia preference ere. Rater, we refer te reader to severa key modes in tis iterature, incuding Rabin (1993), Levine (1998), Fer and Scmidt (1999), Boton and Ockenfes (2000), Carness and Rabin (2002), Fak and Fiscbacer (2006), Cox, Friedman and Gjerstad (2007). 20

21 Scmidt (1999) mode assumes tat g g and g [0, 1), we do not impose tese additiona assumptions, as empirica estimations of te mode do not support tese additiona assumptions (See, e.g., Engemann and Strobe (2004), Cen and Li (2009)). Te foowing proposition caracterizes user response to te outcome information wen g i ( ) is inear. Proposition 2. For te Net Benefit treatment, we expect te foowing resuts: (a) For an average or a beow-average user, it is a dominant strategy to rate popuar movies, and a dominated strategy to rate rare movies or to update te database. (b) For an above-average user, tere exists a g (0, 1), suc tat wen g < g, it is a dominant strategy to rate popuar movies, and a dominated strategy to rate rare movies or to update te database; wen g g, it is a dominant strategy to rate rare movies and to update te database, and a dominated strategy to rate popuar movies. Proof: See Appendix B. Proposition 2 predicts tat an average or a beow-average user is more ikey to rate popuar movies tan to rate rare movies or to update te database. For an above-average user, if se as competitive preferences (g < 0) or is sufficienty sefis (g < g ), se is more ikey to rate popuar movies tan to rate rare movies or to update te database. However, if se is sufficienty caritabe (g gi ), se is more ikey to coose activities wic benefit te community, i.e., rating rare movies or updating te database. Proposition 2 enabes us to compare beaviors across groups. If te fraction of users wit sufficient carity concerns is arge enoug, we expect tat te above-average users wi be more ikey to rate rare movies or to update te database compared to te average or beow-average users or tose in te contro group. Simiary, we expect tat te average or beow-average users are more ikey to rate popuar movies tan te above average group. Finay, we expect tat te average users wi beave simiary to te contro group. 5 Resuts In tis section, we present our data anaysis and main resuts. After tracking user beavior in te mont after receiving te emai newsetter, we find significant and interesting beaviora responses to te socia information we presented in te newsetter. Tere are some common features tat appy trougout our anaysis. First, since te median user s beavior can be idiosyncratic, in te anaysis, we compare te rating beavior of te beowand above-median groups wit tat of te median group, 24 rater tan te median user. Simiary, 24 Te median group is defined as te 1/6 of users wit ifetime ratings above and beow te median, i.e. te midde 1/3 of te users for eac membersip coort. It is kept constant over time. 21

22 in te Net Benefit treatment, we compare te above- and beow-average users wit tat of te average group, rater tan te average user. Second, we note tat te Invite-a-Buddy sortcut did not attract te attention of our users. 25 Tere were a tota of seven buddies invited for te entire subject poo, too sma for any meaningfu statistica comparisons across treatments. Terefore, in reporting te resuts, we focus on movies ratings and database updating. Lasty, since 275 out of 398 participants (see Tabe 1) were active in te two-mont period, we report separate resuts for a users vs. active users. We first verify tat te pre-experiment distributions of tota movie ratings between eac of te treatment groups and te contro group come from te same distribution. Te resuts of Komogorov-Smirnov tests cannot reject te equaity of distribution functions except for te comparison of od users between te Net Benefit treatment and te contro group. 26 [Figure 3 about ere.] Figure 3 presents an overview of user rating beavior in te Rating Info treatment and contro groups, comparing te mont before (te wite bar) and te mont after te newsetter (te back bar). Te eft pane incudes a users, wie te rigt one incudes ony active users. Compared to te mont before, te effects of socia information on post-newsetter beavior are striking. For te Rating Info group, users beow te median ave a 530% increase in te tota number of movie ratings, wie tose above te median decrease teir monty ratings by 62%. Movements from bot ends converge towards te median, atoug te effect of socia information is more dramatic for tose beow te median. In comparison, te about median group as a 290% increase in te number of ratings in te mont after compared to te mont before, wic is not predicted by conformity teory. However, a coser examination of te about median group reveas tat most of te increase comes from tose wo are actuay beow te median (88% for new users, 91% for mid users and 79% for od users), wic is consistent wit conformity teory. Te striking cange in post-newsetter beavior migt be attributed to te socia information, or to any spurious trends absent of te socia information, incuding a regression to te mean effect. To differentiate te two effects, we compare te cange in beavior in te Rating Info treatment and te contro group. If te cange in beavior in te Rating Info treatment is due to a regression to te mean effect, we expect to observe it in te contro group as we. Specificay, we compute te difference in te number of movie ratings in te mont before and after te reease of te newsetter, x E,i = x t+1 E,i xt E,i, for eac experimenta treatment E {R(ating Info), C(ontro)}, and for te beow-, about- and above-median groups i {, m, }. We ten ceck weter tere 25 We specuate tat tis migt be due to te demograpics of our subject poo. Based on te post-experiment survey, more tan 70% of our subjects are mae between te age of twenty and forty. 26 P-vaues from te Komogorov-Smirnov tests between te Rating Info treatment and te contro groups are 0.99 (overa), 0.84 (New), 0.97 (Mid) and 0.85 (Od). P-vaues of te same tests between te Net Benefit treatment and te contro groups are 0.62 (overa), 1.0 (New), 0.98 (Mid) and 0.02 (Od). 22

23 are significant differences between te corresponding treatment and te contro groups. Reca tat te contro group was never divided into te beow, about and above median subgroups in te experiment itsef. Tis division is ony used in te anaysis to investigate any regression to te mean effect. If te cange in beavior is due to te socia information, based on Observation 1, we expect tat, compared to te corresponding subgroups in te contro, te cange in movie ratings wi be arger for te beow-median group, about te same for users in te median group, and smaer for users in te above-median group. [Tabe 2 about ere.] Resut 1 (Rating Info vs. Contro). Compared to te contro group, te cange in movie ratings witin te Rating Info group is significanty arger for te beow-median and te median groups, and about te same for users in te above-median groups. Support. Tabe 2 presents te average difference in te monty ratings for eac group in te Rating Info treatment and contro groups, wit differentia effects on te new, mid and od users. Using te Wicoxon rank-sum test, we reject te nu ypotesis x R, = x C, in favor of x R, > x C, (p = 0.01 for a users, p = 0.00 for active users). Furtermore, we reject te nu x R,m = x C,m in favor of x R,m > x C,m (p = 0.05 for a users, p = 0.02 for active users). However, we fai to reject te nu x R, = x C, in favor of x R, < x C, (p = 0.41 for a users, p = 0.24 for active users). Resut 1 confirms tat te socia information in te Rating Info treatment group as a significant effect on beavior in te beow-median and te median groups. Compared to Observation 1, ony te prediction for te beow-median group is confirmed. Wie te outcome for te median group is different from te teoretica prediction, as we noted before, more tan 80% of te increase in te median group comes from users wo are actuay beow te median. We now proceed to anayze beaviora canges witin te Rating Info treatment. [Tabe 3 about ere.] Resut 2 (Conformity in ratings). In te mont after te reease of te newsetter, among active users in te Rating Info treatment, tose beow te median rate significanty more movies tan teir median counterparts. Among a users in te Rating Info treatment group, te above-median users rate significanty more movies tan te median users. At te aggregate eve, te average distance between an active user and er median counterpart s number of ratings is significanty smaer in te mont after tan tat in te mont before. Support. Tabe 3 presents our ypoteses and te corresponding Wicoxon rank sum test statistics. Te aternative ypoteses are derived from Proposition 1 in Section 4. Among active users (ower 23

24 pane), beow-median users rate more movies tan median users (p = 0.02 overa and p = 0.01 for new users). Among a users (upper pane), x t+1 = x t+1 m cannot be rejected in favor of x t+1 < x t+1 m (p = 0.97 overa and p = 0.98 for new users). However, we can reject x t+1 = x t+1 m in favor of x t+1 > x t+1 m (p = 0.03 overa and p = 0.02 for new users). To test part (c) of Proposition 1, we compute te distance between eac user i and er median counterpart s cumuative ratings in te mont before te newsetter, Xi t Xm, t and te mont after, X t+1 i Xm t+1. Pair-sampe t-tests reject te nu of equa distance in favor of te aternative ypotesis tat te distance in te mont after (347.9) is smaer tan tat in te mont before (354.5) for te active users (n = 99, p = 0.02). If we incude a users in te Rating Info treatment, owever, we fai to reject te nu in favor of te aternative (n = 134, p = 0.42). Since Proposition 1 ods wen conforming to te socia norm is sufficienty important, i.e., wen g i( ) is sufficienty arge, we expect te resuts to be different for users wit different tendencies to conform. To investigate tis difference, we partition te users into conforming and non-conforming types based on teir post-experiment survey responses to a pair of questions designed to measure te tendency to conform, i.e., I wi stick to my opinion if I tink I am rigt, even if oters disagree, and its reversed version, I wi cange te opinion I express as a resut of an onsaugt of criticism, even toug I reay don t cange te way I fee. Te correation of te responses to tese two questions is 0.5 (p < 0.01). We categorize a user as a conforming type if se responds disagree or strongy disagree to te first question, and agree or strongy agree to te second, and a non-conforming type oterwise. Out of te 83 users wo answered bot two questions, 58 are categorized as conforming and 25 non-conforming types. 27 Before presenting resuts combining te beaviora and survey data, we mention severa common caveats associated wit te post-experiment survey data. First, ony 78% of te users responded to te survey. Second, 71% of te users answered every question. Lasty, among users wo answered a set of questions and teir reversed versions, te answers are not aways consistent. 28 Terefore, one soud use caution interpreting te survey resuts. [Tabe 4 about ere.] Tabe 4 presents our two-type anaysis in two panes. Pane A, wic corresponds to parts (a) and (b) of Proposition 1, presents te comparison of te post-experiment monty ratings between te beow-median, median, and above-median users. Resuts indicate tat te prediction of Proposition 1 (a) is ony significant for te conforming types (p = 0.03), wie te prediction of Proposition 1 (b) is not significant for eiter type (p > 0.10). 27 As a robustness ceck, we construct two aternative measures of conforming tendency based on responses to eac of te two questions separatey, and find tat te resuts are consistent wit tose based on tis composite measure. 28 For exampe, for te two questions on conformity, consistent answers woud yied a correation of 1.0, rater tan 0.5, wic migt refect different interpretations of te same questions. 24

25 Pane B, wic corresponds to Proposition 1 (c), presents te average distance between eac user i and er median counterpart s cumuative ratings in te mont before, Xi t Xm, t and te mont after te newsetter, X t+1 i Xm t+1, for te conforming and non-conforming types respectivey. We find tat, for te conforming types, te distance from median in te mont after is significanty smaer tan tat in te mont before (p = 0.04, one-side paired-sampe t-tests). For te non-conforming types, atoug te average distance aso reduces, te reduction is not statisticay significant (p = 0.20). Resuts terefore support Proposition 1 (c). In sum, wie Proposition 1 predicts te beavior of users beow te median we, its prediction does not od for users above te median, wo rate significanty more movies tan te median users. Furtermore, te coorts most responsive to te median rating information are te new users, wo migt be more maeabe. Bot Resuts 1 and 2 suggest tat te median rating information as a more dramatic effect on te beow-median group (a 530% increase in te monty ratings compared to te mont before) compared to te above-median group (a 62% decrease in te monty ratings). We specuate tat tis disparity in effect migt be due to an interaction between conformity and competitive preferences. In MovieLens, te system exorts te users to rate more movies. For exampe, in te new user tour, one screen says Remember: te more movies you rate, te more accurate MovieLens predictions wi be. Terefore, rating more movies migt be perceived as a sociay desirabe course of action, wic coud, in turn, trigger competitive preferences, i.e., more ratings are better. For te beow-median group, conformity and competitiveness work in te same direction, wereas for te above-median users, conformity teory predicts a decrease in te number of monty ratings, wie competitive preference predicts an increase. User responses to te post-experiment survey are consistent wit tis specuation. [Figure 4 about ere.] Figure 4 presents te cange in ratings ( x i ) for te beow-, about- and above-median groups as a function of sef-reported competitiveness in te survey. 29 Te average number of ratings by beow-, median and above-median users is represented by wite, grey and back bars, respectivey. Wie beow-median users for a competitiveness eves increase teir number of ratings, te more competitive users increase teir number of ratings by a arger amount. By contrast, for abovemedian users, te cange in ratings is negativey correated wit teir competitiveness. Specificay, noncompetitive users ave te argest decrease in te number of ratings, foowed by te neutra group, wie te competitive users ave a sigt increase in teir number of ratings. Median users foow te same pattern, wit te exception of te competitive users in te group. 29 In te post-experiment survey, participants were asked to indicate teir agreement on a scae of 1 (strongy disagree) to 5 (strongy agree) wit te foowing statement, It s acievement, rater tan popuarity wit oters, tat gets you aead nowadays. Tey are considered to ave a noncompetitive preference if tey coose 1 or 2, a neutra preference if tey cose 3, and a competitive preference oterwise. 25

26 Reca tat, to keep te experimenta treatments and te contro strategicay comparabe, a users in te experiment are provided wit te same five sortcuts. Wie conformity teory predicts tat te number of ratings moves towards te median, it does not predict any systematic pattern for ow users migt differ in te number of database entries updated. Indeed, we find tat users beow-, about- and above-median are not significanty different in te number of database entries tey provide. Comparing te Rating Info treatment group wit te contro group, we find tat users in te contro group provide weaky significanty more entries in te database (p = 0.09, one-taied Wicoxon rank sum test). One pausibe expanation is tat updating te database is a reativey new feature in MovieLens and te novety of tis feature migt ave attracted te attention of te users in te contro group, since tey do not receive any socia information. In sum, in te Rating Info treatment group, socia information significanty canges user rating beavior. By reporting te median user s rating in eac reevant MovieLens membersip coort, we observe a sift of beavior towards te median from bot ends of te spectrum. Te effect is more dramatic for te beow-median users tan for te above-median users. For bot groups, owever, we observe an interaction between conformity and competitive preferences. For beowmedian users, more competitive users ave arger increases in te number of ratings, wereas for above-median users, more competitive users ave a smaer decrease in te number of ratings. In te Net Benefit treatment group, we provide net benefit information to investigate weter we can everage users distributiona preferences to contribute to ig-cost pubic goods suc as rating rare movies or updating te database. We now examine te resuts for tis group. [Figure 5 about ere.] Figure 5 presents an overview of user beavior in te Net Benefit treatment, comparing beavior in te mont before (te wite bar) and te mont after (te back bar) te newsetter. Te eft coumn presents te beavior of a users, wie te rigt coumn presents tat of te active users. Since updating te database was not avaiabe prior to te experiment, te ast row does not contain any wite bars. We first verify tat beaviora canges in te treatment group are due to user responses to te socia information in te newsetters by comparing canges in beavior in te treatment and contro groups. Since updating te database was not avaiabe prior to te experiment, we examine canges in popuar and rare movie ratings compared to te respective beaviors in te contro group. [Tabe 5 about ere.] Resut 3 (Net Benefit vs. Contro). Te increases in popuar movie ratings for te beow-average and te average groups are bot significanty greater tan te contro group. Support. Tabe 5 presents te average difference in te number of popuar movie ratings for eac group in te Net Benefit treatment and contro groups. Te increase in popuar movie ratings is 26

27 significanty greater for te beow-average group tan for te contro group (p = 0.02 for mid users among a users, and 0.07 for active users, one-sided Wicoxon rank sum tests). Furtermore, te increase in popuar movie ratings for te average users is aso significanty greater tan tat in te contro group (p < 0.01 for a and active users, one-sided Wicoxon rank sum tests). Resut 3 indicates tat te cange in popuar movie rating in te Net Benefit group is indeed caused by te socia information in te newsetter. We conduct simiar anaysis for te rare movie ratings. However, as tere are fewer rare movies rated, we cannot reject te ypotesis tat beowaverage, average, and above-average groups are te same as te respective contro groups (p = 0.72, 0.57 and 0.51 respectivey, two-sided Wicoxon rank sum tests). Terefore, compared to te contro group, te socia information provided induces te beow-average and average users to rate more popuar, but not more rare movies, i.e., among te rating options, tey prefer te more sefis to te more oter-regarding one. We next compare te beavior of different groups witin te Net Benefit treatment group in te mont after te newsetter. We examine tree activities: te number of popuar movies rated, te number of rare movies rated, and te number of database entries updated. We summarize te main findings in Resut 4. Resut 4 (Inequaity Aversion). In te mont after receiving te newsetter, users receiving different net benefit information ave significanty different activity eves: (a) Popuar movie ratings: Te average users rate significanty more popuar movies tan tose beow or above average; (b) Rare movie ratings: Te above-average users rate significanty more rare movies tan tose beow-average; (c) Database entries: Te above-average users contribute 94% of te new updates in te database from te Net Benefit treatment group, significanty more tan te average or te beow-average users. Support. A p-vaues presented are from Wicoxon rank sum tests: = x p,t+1 is rejected in favor of x p,t+1 a (a) Popuar movie ratings: x p,t+1 a users). Likewise, x p,t+1 a = x p,t+1 (b) Rare movie ratings: x r,t+1 is rejected in favor of x p,t+1 a = x r,t+1 > x p,t+1 is rejected in favor of x r,t+1 > x p,t+1 at p = 0.03 (a at p = 0.03 (active users). > x r,t+1 (c) Database entries: d t+1 = d t+1 is rejected in favor of d t+1 > d t+1 p = 0.01 (active users). Likewise, d t+1 = d t+1 a is rejected in favor of d t+1 > d t+1 a users), p = 0.01 (active users). at p = 0.01 (a users). at p < 0.01 (a users), at p < 0.01 (a Resut 4 indicates tat, overa, users wit above average net benefit scores mainy engage in activities tat raise te net benefit of oters, i.e., rating rare movies and updating te database. It does not, owever, separate users wit ig and ow piantropic concerns. Since Part (b) of Proposition 2 predicts tat above-average users wit sufficienty ig piantropic concerns wi 27

28 rate rare movies and update te database, we now use survey data to cassify users by teir atruism score. We construct an atruism score from subject responses to six of te questions designed to measure atruism in te post-experiment survey, were a iger score represents a greater sefreported atruistic preference. 30 [Tabe 6 about ere.] Tabe 6 reports te above-average user activities, teir atruism score (Low, Midde, Hig), ypoteses based on part (b) of Proposition 2, and te p-vaues for one-sided t-tests. Te resuts indicate tat (1) te average popuar movie ratings decrease wit te atruism score; (2) te average rare movie ratings increase wit te atruism score; and (3) te average database entries increase wit te atruism score. Tese resuts are argey consistent wit te teoretica predictions. Among tese tree activities, owever, ony rare movie rating comparisons between tose wit ow and ig (p = for a and p = for active above-average users), and tose wit midde and ig (p = for a and p = for active above-average users) atruism scores are statisticay significant. [Tabe 7 about ere.] Summarizing a treatments, Tabe 7 presents eigt Tobit specifications of contribution beavior using socia information categories as we as demograpic caracteristics as independent variabes. Te dependent variabes incude te number of movie ratings in te Rating Info treatment (specifications 1 and 2), te number of popuar (3 and 4) and rare movie ratings (5 and 6), and te number of database entries (7 and 8) in te Net Benefit treatment in te mont after te newsetter. Te independent variabes incude Pre-rating (te number of movies rated in te mont before), Above (users wit ifetime rating above te median or net benefit score above te mean), Beow, MovieLens Age, Mae, Age, 31 Education (years of education), and occupation variabes incuding 30 Participants were asked to indicate teir eve of agreement wit te foowing statements regarding teir personaities, I see mysef as someone wo a) is epfu and unsefis wit oters; b) can be cod and aoof; c) is considerate and kind to amost everyone; d) ikes to cooperate wit oters; e) is often on bad terms wit oters; f) fees itte concern for oters; g) is on good terms wit neary everyone. (For statements a), c), d) and g), we code te answers strongy agree, agree, neutra, disagree and strongy disagree as 2, 1, 0, -1, and -2, respectivey. For statements b), e), and f), we code te answers strongy agree, agree, neutra, disagree and strongy disagree as -2, -1, 0, 1 and 2, respectivey. Summing eac individua s responses across te above questions yieds a score tat ranges from -5 to 13 wit a mean of 4 and standard deviation of 3.8. We bin te scores into tree categories, were category 1 (Low) incudes tose wo are more tan one standard deviation beow te mean, category 2 (Midde) incudes tose witin one standard deviation of te mean, and category 3 (Hig) incudes tose wo are more tan one standard deviation above te mean. 31 Age is a categorica variabe in te post experiment survey. Te users are given severa categories: beow 15, 15-19, 20-29, 30-39, 40-49, 50 and above. 28

29 Student, CompMat (Computer and Mat occupations), EdTLib (Education, Training and Library occupations), and a constant. Omitted variabes incude Median, Femae, and oter occupations. In specifications (1) and (2), we find tat te beow-median users rated significanty more movies compared to te median group, consistent wit Resut 2. Unike in Resut 2, owever, te comparison between te above-median and te median users is not significant, wic migt be due to te smaer number of observations in te regression anaysis, or te incusion of demograpic variabes. Among te demograpic variabes, MovieLens Age is negativey correated wit te number of movies ratings in te mont after, consistent wit our previous resut tat new users are most responsive to te socia information. In te Net Benefit treatment, we find tat, compared to te average users, te above-average users rated significanty ess popuar movies (4), significanty more database entries (7 and 8), wie te beow-average users rated significanty ess rare movies (5 and 6), wic is argey consistent wit Resut 4. Wie gender as no effect on beavior, oder users rated significanty more rare movies. Lasty, users wit more education finised significanty ess database entries. Trougout te anaysis, we ave assumed tat te socia information provided in te newsetter causes te canges in user beavior. An aternative expanation is tat te effects migt be due to ancoring, i.e., simpy mentioning a number migt serve as a reference point (te ancor ) wic migt infuence beavior (Tversky and Kaneman 1974). To investigate tis possibiity, we use our contro data, were no socia information was provided. Instead, eac user in te contro group is provided wit teir persona statistics. If ancoring is te mecanism at work, we expect tat te persona statistics in te contro woud infuence and correate wit te beavior in te mont after. We find, owever, tat te correation between te information provided in te newsetter, i.e., te ancor, in te contro and te number of movies rated in te mont after, is sma and statisticay insignificant at te 10% eve. We repeat te same exercise for bot treatments and find simiar resuts. Tis indicates tat it is unikey tat canges in beavior are primariy caused by ancoring. Lasty, for bot te Rating Info and Net Benefit treatments, we compare te rankings in te mont before and after to ceck weter tere are any canges in te distribution. 32 More specificay, we are interested in weter te significant canges in te amount of movie ratings and database updating ave moved some beow-median (or beow-average) users to above te median (or average) in movie ratings (or net benefit scores), and vice versa. Using te Spearman and Kenda rank correation tests, we find tat te correation of rankings for movie ratings is cose to one in te Rating Info treatment, wereas te correation of te net benefit scores is strongy positive. Furtermore, we can reject te nu ypotesis of ranking independence at te 1% significance eve for a tests. Terefore, te reative ranking of users remains argey uncanged despite a significant amount of work by various groups of users during te mont after te newsetter, an 32 We tank Jon Duffy for suggesting tis part of te anaysis. 29

30 effect known as te Red Queen Effect, taken from Lewis Carro s (1871) Troug te Looking- Gass, were te Red Queen said, Now ere, you see, it takes a te running you can do, to keep in te same pace. At te aggregate eve, atoug te tota number of ratings in te Rating Info treatment does not cange from te mont before (2569) to te mont after (2556) te newsetter, we do observe a 530% increase in te beow-median group. For te Net Benefit treatment, wie te number of monty movie ratings as a 59% increase from 1216 in te mont before to 1928 in te mont after, above-average users rate more rare movies and contribute 94% of te new updates in te database, activities tat mosty benefit oters. In contrast, te contro group as a 33% decrease in te number of movies ratings (from 2431 to 1632), owever, users in te contro group contribute 273 new updates in te database. In our entire subject poo, te monty ratings ave a 1.6% decrease from before (6216) to after te intervention (6116), wie tere is a net increase of 417 new updates in te database. From a mecanism designer s perspective, to increase te overa contribution to onine communities, it is important to personaize te socia information, wic as disparate effects on different groups of peope. For exampe, te median rating information is effective to increase ratings for users wit a ow number of ratings, but not for tose wit a ig number of ratings. In comparison, te average net benefit score can motivate users wit above-average scores to increase te eve of costy activities wic mainy ep oters, and tose wit beow- and about-average scores to increase eves of activities wic mosty benefit temseves. Personaization is feasibe and ow-cost, especiay for onine communities. 6 Concusion Te Internet enabes te formation of onine communities and coaboration on a scae never seen before. Many popuar websites, suc as Wikipedia, MySpace and YouTube, are based entirey on content contributed by teir members. Te caenge facing designers and managers of suc onine communities is to motivate members to sustain and improve teir contributions. In tis study, we investigate te impact of socia comparisons as a natura, non-pecuniary incentive mecanism for motivating contributions to an onine community. Specificay, we use emai newsetters to et members of an onine movie recommender community know ow tey compare wit oter members in terms of movie ratings and net benefits. We find tat, after receiving beaviora information about te median user s tota number of movie ratings, users beow te median sow a 530% increase in te number of monty movies ratings, wie tose above te median decrease teir monty ratings by 62%. Furtermore, we find tat te effects of socia comparisons are most dramatic for te beow-median users, consistent wit an interaction between conformity and competitive preferences. Additionay, we find tat wen given outcome information about te 30

31 average user s net benefit score from te system, te average users rate significanty more popuar movies, wie users wit net benefit scores above average contribute 94% of te new updates in te database, consistent wit socia preference teories. Our findings ave significant impications for bot te mecanism designers and managers of onine communities. We demonstrate tat socia information as significant effect on user contribution to pubic goods. From te perspective of designers and managers of an onine community, our findings indicate tat one can effectivey cassify users and personaize teir messages to increase te amount of ig-vaue work done by members of an onine community. For exampe, in te case of MovieLens, for users wit a ow number of ratings, information on te median user s ratings can induce significanty more ratings. For users wit ig net benefit scores, information on teir scores and tose of an average user can trigger teir distributiona concerns and ead to an increase in contributions to te database updating and rating of rare movies. Wat is particuar intriguing is tat average users, upon earning tat tey are about average, can be caenged to increase teir ratings as we. Our findings aso contribute to te teoretica iterature on conformity and socia norms. Most existing modes ave te caracteristic tat agents suffer disutiity wen tey deviate from te socia norm (e.g., Akerof (1980), Berneim (1994)). Our resuts indicate tat an interaction between conformity and competition is an important factor wic as been overooked. Wen te socia norm, suc as movie ratings, contributes to te common good, conformity works in te same direction as competition for peope beow te median, wereas tey work in opposite directions for tose above te median, resuting in a more dramatic effect on te ow end of te spectrum tan on te ig end. In sum, our resuts indicate tat socia comparison can provide an effective non-pecuniary incentive to motivate contributions to onine communities. One imitation of tis study is tat Movie- Lens is argey a eisure community. It woud be interesting to examine weter we can repicate our resuts in work-oriented onine communities. To expore tis possibiity, we are conducting projects on onine reference communities, suc as Googe Answers. Furtermore, in our study, we investigate socia comparisons wit peers, troug information provided about te median or average user. In practice, we aso observe oter forms of socia comparisons, suc as eaderboards in te ESP game (ttp:// and contribution-based status eves at Sasdot (ttp://sasdot.org/). In future work, we ope to study different forms of socia comparisons and evauate teir effects on user beavior and te growt of onine pubic information goods. 31

32 APPENDIX A. Screen Sots In tis appendix, we incude one exampe newsetter for eac treatment. Oter newsetters ave te same format and ayout, except for te individua specific numbers and comparison prases. # 1. Emai Newsetter: Contro Group 32

33 # 2. Emai Newsetter: Rating Info Treatment (Beow Median) 33

34 # 3. Emai Newsetter: Net Benefit Treatment (Beow Average) 34

35 # 4. Modified MovieLens Interface: Sortcuts 35