Learning from Collective Behavior

Transcription

1 Lerning from Collective Behvior Michel Kerns Computer nd Informtion Science University of Pennsylvni Jennifer Wortmn Computer nd Informtion Science University of Pennsylvni Abstrct Inspired by longstnding lines of reserch in sociology nd relted fields, nd by more recent lrgepopultion humn subject experiments on the Internet nd the Web, we initite study of the computtionl issues in lerning to model collective behvior from observed dt. We define forml models for efficient lerning in such settings, nd provide both generl theory nd specific lerning lgorithms for these models. 1 Introduction Collective behvior in lrge popultions hs been subject of enduring interest in sociology nd economics, nd more recent topic in fields such s physics nd computer science. There is consequently now n impressive literture on mthemticl models for collective behvior in settings s diverse s the diffusion of fds or innovtion in socil networks [10, 1, 2, 18], voting behvior [10], housing choices nd segregtion [22], herding behviors in finncil mrkets [27, 8], Hollywood trends [25, 24], criticl mss phenomen in group ctivities [22], nd mny others. The dvent of the Internet nd the Web hve gretly incresed the number of both controlled experiments [7, 17, 20, 21, 8] nd open-ended systems (such s Wikipedi nd mny other instnces of humn peer-production ) tht permit the logging nd nlysis of detiled collective behviorl dt. It is nturl to sk if there re lerning methods specificlly tilored to such models nd dt. The mthemticl models of the collective behvior literture differ from one nother in importnt detils, such s the extent to which individul gents re ssumed to ct ccording to trditionl notions of rtionlity, but they generlly shre the significnt underlying ssumption tht ech gent s current behvior is entirely or lrgely determined by the recent behvior of the other gents. Thus the collective behvior is socil phenomenon, nd the popultion evolves over time ccording to its own internl dynmics there is no exogenous Nture being rected to, or injecting shocks to the collective. In this pper, we introduce computtionl theory of lerning from collective behvior, in which the gol is to ccurtely model nd predict the future behvior of lrge popultion fter observing their interctions during trining phse of polynomil length. We ssume tht ech gent i in popultion of size N cts ccording to fixed but unknown strtegy c i drwn from known clss C. A strtegy probbilisticlly mps the current popultion stte to the next stte or ction for tht gent, nd ech gent s strtegy my be different. As is common in much of the literture cited bove, there my lso be network structure governing the popultion interction, in which cse strtegies my mp the locl neighborhood stte to next ctions. Lerning lgorithms in our model re given trining dt of the popultion behvior, either s repeted finite-length trjectories from multiple initil sttes (n episodic model), or in single unbroken trjectory from fixed strt stte ( no-reset model). In either cse, they must efficiently (polynomilly) lern to ccurtely predict or simulte (properties of) the future behvior of the sme popultion. Our frmework my be viewed s computtionl model for lerning the dynmics of n unknown Mrkov process more precisely, dynmic Byes net in which our primry interest is in Mrkov processes inspired by simple models for socil behvior. As simple, concrete exmple of the kind of system we hve in mind, consider popultion in which ech gent mkes series of choices from fixed set over time (such s wht resturnt to go to, or wht politicl prty to vote for). Like mny previously studied models, we consider gents who hve desire to behve like the rest of the popultion (becuse they wnt to visit the populr resturnts, or wnt to vote for electble cndidtes). On the other hnd, ech gent my lso hve different nd unknown intrinsic preferences over the choices s well (bsed on cuisine nd decor, or the ctul policies of the cndidtes). We consider models in which ech gent blnces or integrtes these two forces in deciding how to behve t ech step [12]. Our min question is: Cn lerning lgorithm wtching the collective behvior of such popultion for short period produce n ccurte model of their future choices? The ssumptions of our model fit nicely with the literture cited in the first prgrph, much of which indeed proposes simple stochstic models for how individul gents rect to the current popultion stte. We emphsize from the outset the difference between our interests nd those common in multigent systems nd lerning in gmes. In those fields, it is often the cse tht the gents themselves re cting ccording to complex nd firly generl lerning l-

2 gorithms (such s Q-lerning [26], no-regret lerning [9], fictitious ply [3], nd so on), nd the centrl question is whether nd when the popultion converges to prticulr, nice sttes (such s Nsh or correlted equilibri). In contrst, while the gent strtegies we consider re certinly dptive in rective sense, they re much simpler thn generl-purpose lerning lgorithms, nd we re interested in lerning lgorithms tht model the full collective behvior no mtter wht its properties; there is no specil sttus given either to prticulr sttes nor to ny notion of convergence. Thus our interest is not in lerning by the gents themselves, but t the er level of n observer of the popultion. Our primry contributions re: The introduction of computtionl model for lerning from collective behvior. The development of some generl theory for this model, including polynomil-time reduction of lerning from collective behvior to lerning in more trditionl, single-trget I.I.D. settings, nd seprtion between efficient lernbility in collective models in which the lerner does nd does not see ll intermedite popultion sttes. The definition of specific clsses of gent strtegies, including vrints of the crowd ffinity strtegies sketched bove, nd complementry crowd version clsses. Provbly efficient lgorithms for lerning from collective behvior for these sme clsses. The outline of the pper is s follows. In Section 2, we introduce our min model for lerning from collective behvior, nd then discuss two nturl vrints. Section 3 introduces nd motivtes number of specific gent strtegy clsses tht re brodly inspired by erlier sociologicl models, nd provides brief simultions of the collective behviors they cn generte. Section 4 provides generl reduction of lerning from collective behvior to generlized PAC-style model for lerning from I.I.D. dt, which is used subsequently in Section 5, where we give provbly efficient lgorithms for lerning some of the strtegy clsses introduced in Section 3. Brief conclusions nd topics for further reserch re given in Section 6. 2 The Model In this section we describe lerning model in which the observed dt is generted from observtions of trjectories (defined shortly) of the collective behvior of N intercting gents. The key feture of the model is the fct tht ech gent s next stte or ction is lwys determined by the recent ctions of the other gents, perhps combined with some intrinsic preferences or behviors of the prticulr gent. As we shll see, we cn view our model s one for lerning certin kinds of fctored Mrkov processes tht re inspired by models common in sociology nd relted fields. Ech gent my follow different nd possibly probbilistic strtegy. We ssume tht the strtegy followed by ech gent is constrined to lie in known (nd possibly lrge) clss, but is otherwise unknown. The lerner s ultimte gol is not to discover ech individul gent strtegy per se, but rther to mke ccurte predictions of the collective behvior in novel situtions. 2.1 Agent Strtegies nd Collective Trjectories We now describe the min components of our frmework: Stte Spce. At ech time step, ech gent i is in some stte s i chosen from known, finite set S of size K. We often think of K s being lrge, nd thus wnt lgorithms whose running time scles polynomilly in K nd other prmeters. We view s i s the ction tken by gent i in response to the recent popultion behvior. The joint ction vector s S N describes the current globl stte of the collective. Initil Stte Distribution. We ssume tht the initil popultion stte s 0 is drwn ccording to fixed but unknown distribution P over S N. During trining, the lerner is ble to see trjectories of the collective behvior in which the initil stte is drwn from P, nd s in mny stndrd lerning models, must generlize with respect to this sme distribution. (We lso consider no-reset vrint of our model in Section 2.3.) Agent Strtegy Clss. We ssume tht ech gent s strtegy is drwn from known clss C of (typiclly probbilistic) mppings from the recent collective behvior into the gent s next stte or ction in S. We minly consider the cse in which c i C probbilisticlly mps the current globl stte s into gent i s next stte. However, much of the theory we develop pplies eqully well to more complex strtegies tht might incorporte longer history of the collective behvior on the current trjectory, or might depend on summry sttistics of tht history. Given these components, we cn now define wht is ment by collective trjectory. Definition 1 Let c C N be the vector of strtegies for the N gents, P be the initil stte distribution, nd T 1 be n integer. A T -trjectory of c with respect to P is rndom vrible s 0,, s T in which the initil stte s 0 S N is drwn ccording to P, nd for ech t {1,, T }, the component s t i of the joint stte s t is obtined by pplying the strtegy c i to s t 1. (Agin, more generlly we my lso llow the strtegies c i to depend on the full sequence s 0,..., s t 1, or on summry sttistics of tht history.) Thus, collective trjectory in our model is simply Mrkovin sequence of sttes tht fctors ccording to the N gent strtegies tht is, dynmic Byes net [19]. Our interest is in cses in which this Mrkov process is generted by prticulr models of socil behvior, some of which re discussed in Section The Lerning Model We now formlly define the lerning model we study. In our model, lerning lgorithms re given ccess to n orcle O EXP ( c, P, T) tht returns T -trjectory s 0,, s T of

3 c with respect to P. This is thus n episodic or reset model, in which the lerner hs the luxury of repetedly observing the popultion behvior from rndom initil conditions. It is most pplicble in (prtilly) controlled, experimentl settings [7, 17, 20, 21, 8] where such popultion resets cn be implemented or imposed. In Section 2.3 below we define perhps more brodly pplicble vrint of the model in which resets re not vilble; the lgorithms we provide cn be dpted for this model s well (Section 5.3). The gol of the lerner is to find genertive model tht cn efficiently produce trjectories from distribution tht is rbitrrily close to tht generted by the true popultion. Thus, let ˆM( s 0, T) be (rndomized) model output by lerning lgorithm tht tkes s input strt stte s 0 nd time horizon T, nd outputs rndom T -trjectory, nd let Q ˆM denote the distribution over trjectories generted by ˆM when the strt stte is distributed ccording to P. Similrly, let Q c denote the distribution over trjectories generted by O EXP ( c, P, T). Then the gol of the lerning lgorithm is to find model ˆM mking the L1 distnce ε(q ˆM, Q c ) between Q ˆM nd Q c smll, where ε(q ˆM, Q c ) s 0,, s T Q ˆM( s 0,, s T ) Q c ( s 0,, s T ). A couple of remrks re in order here. First, note tht we hve defined the output of the lerning lgorithm to be blck box tht simply produces trjectories from initil sttes. Of course, it would be nturl to expect tht this blck box opertes by hving good pproximtions to every gent strtegy in c, nd using collective simultions of these to produce trjectories, but we choose to define the output ˆM in more generl wy since there my be other pproches. Second, we note tht our lerning criteri is both strong (see below for discussion of weker lterntives) nd useful, in the sense tht if ε(q ˆM, Q c ) is smller thn ǫ, then we cn smple ˆM to obtin O(ǫ)-good pproximtions to the expecttion of ny (bounded) function of trjectories. Thus, for instnce, we cn use ˆM to nswer questions like Wht is the expected number of gents plying the plurlity ction fter T steps? or Wht is the probbility the entire popultion is plying the sme ction fter T steps? (In Section 2.4 below we discuss weker model in which we cre only bout one fixed outcome function.) Our lgorithmic results consider cses in which the gent strtegies my themselves lredy be rther rich, in which cse the lerning lgorithm should be permitted resources commensurte with this complexity. For exmple, the crowd ffinity models hve number of prmeters tht scles with the number of ctions K. More generlly, we use dim(c) to denote the complexity or dimension of C; in ll of our imgined pplictions dim( ) is either the VC dimension for deterministic clsses, or one of its generliztions to probbilistic clsses (such s pseudo-dimension [11], ft-shttering dimension [15], combintoril dimension [11], etc.). We re now redy to define our lerning model. Definition 2 Let C be n gent strtegy clss over ctions S. We sy tht C is polynomilly lernble from collective behvior if there exists n lgorithm A such tht for ny popultion size N 1, ny c C N, ny time horizon T, ny distribution P over S N, nd ny ǫ > 0 nd δ > 0, given ccess to the orcle O EXP ( c, P, T), lgorithm A runs in time polynomil in N, T, dim(c), 1/ǫ, nd 1/δ, nd outputs polynomil-time model ˆM such tht with probbility t lest 1 δ, ε(q ˆM, Q c ) ǫ. We now discuss two resonble vritions on the model we hve presented. 2.3 A No-Reset Vrint The model bove ssumes tht lerning lgorithms re given ccess to repeted, independent trjectories vi the orcle O EXP, which is nlogous to the episodic setting of reinforcement lerning. As in tht field, we my lso wish to consider n lterntive no-reset model in which the lerner hs ccess only to single, unbroken trjectory of sttes generted by the Mrkov process. To do so we must formulte n lterntive notion of generliztion, since on the one hnd, the (distribution of the) initil stte my quickly become irrelevnt s the collective behvior evolves, but on the other, the stte spce is exponentilly lrge nd thus it is unrelistic to expect to model the dynmics from n rbitrry stte in polynomil time. One nturl formultion llows the lerner to observe ny polynomilly long prefix of trjectory of sttes for trining, nd then to nnounce its rediness for the test phse. If s is the finl stte of the trining prefix, we cn simply sk tht the lerner output model ˆM tht genertes ccurte T -step trjectories forwrd from the current stte s. In other words, ˆM should generte trjectories from distribution close to the distribution over T -step trjectories tht would be generted if ech gent continued choosing ctions ccording to his strtegy. The length of the trining prefix is llowed to be polynomil in T nd the other prmeters. While spects of the generl theory described below re prticulr to our min (episodic) model, we note here tht the lgorithms we give for specific clsses cn in fct be dpted to work in the no-reset model s well. Such extensions re discussed briefly in Section Weker Criteri for Lernbility We hve chosen to formulte lernbility in our model using rther strong success criterion nmely, the bility to (pproximtely) simulte the full dynmics of the unknown Mrkov process induced by the popultion strtegy c. In order to meet this strong criterion, we hve lso llowed the lerner ccess to rther strong orcle, which returns ll intermedite sttes of smpled trjectories. There my be nturl scenrios, however, in which we re interested only in specific fixed properties of collective behvior, nd thus weker dt source my suffice. For instnce, suppose we hve fixed, rel-vlued outcome function F( s T ) of finl sttes (for instnce, the frction of gents plying the plurlity ction t time T ), with our gol being to simply lern function G tht mps initil sttes s 0 nd time horizon T to rel vlues, nd pproximtely minimizes [ G( s 0, T) E s T [F( s T )] ] E s 0 P

4 where s T is rndom vrible tht is the finl stte of T -trjectory of c from the initil stte s 0. Clerly in such model, while it certinly would suffice, there my be no need to directly lern full dynmicl model. It my be fesible to stisfy this criterion without even observing intermedite sttes, but only seeing initil stte nd finl outcome pirs s 0, F( s T ), closer to trditionl regression problem. It is not difficult to define simple gent strtegy clsses for which lerning from only s 0, F( s T ) pirs is provbly intrctble, yet efficient lerning is possible in our model. This ide is formlized in Theorem 3 below. Here the popultion forms rther powerful computtionl device mpping initil sttes to finl sttes. In prticulr, it cn be thought of s circuit of depth T with gtes chosen from C, with the only rel constrint being tht ech lyer of the circuit is n identicl sequence of N gtes which re pplied to the outputs of the previous lyer. Intuitively, if only initil sttes nd finl outcomes re provided to the lerner, lerning should be s difficult s corresponding PAC-style problem. On the other hnd, by observing intermedite stte vectors we cn build rbitrrily ccurte models for ech gent, which in turn llows us to ccurtely simulte the full dynmicl model. Theorem 3 Let C be the clss of 2-input AND nd OR gtes, nd one-input NOT gtes. Then C is polynomilly lernble from collective behvior, but there exists binry outcome function F such tht lerning n ccurte mpping from strt sttes s 0 to outcomes F( s T ) without observing intermedite stte dt is intrctble. Proof: (Sketch) We first sketch the hrdness construction. Let H be ny clss of Boolen circuits (tht is, with gtes in C) tht is not polynomilly lernble in the stndrd PAC model; under stndrd cryptogrphic ssumptions, such clss exists. Let D be hrd distribution for PAC lerning H. Let h H be Boolen circuit with R inputs, S gtes, nd depth D. To embed the computtion by h in collective problem, we let N = R + S nd T = D. We introduce n gent for ech of the R inputs to h, whose vlue fter the initil stte is set ccording to n rbitrry AND, OR, or NOT gte. We dditionlly introduce one gent for every gte g in h. If gte g in h tkes s its inputs the outputs of gtes g nd g, then t ech time step the gent corresponding to g computes the corresponding function of the sttes of the gents corresponding to g nd g t the previous time step. Finlly, by convention we lwys hve the Nth gent be the gent corresponding to the output gte of h, nd define the output function s F( s) = s N. The distribution P over initil sttes of the N gents is identicl to D on the R gents corresponding to the inputs of h, nd rbitrry (e.g., independent nd uniform) on the remining S gents. Despite the fct tht this construction introduces gret del of spurious computtion (for instnce, t the first time step, mny or most gtes my simply be computing Boolen functions of the rndom bits ssigned to non-input gents), it is cler tht if gte g is t depth d in h, then t time d in the collective simultion of the gents, the corresponding gent hs exctly the vlue computed by g under the inputs to h (which re distributed ccording to D). Becuse the outcome function is the vlue of the gent corresponding to the output gte of h t time T = D, pirs of the form s 0, F( s T ) provide exctly the sme dt s the PAC model for h under D, nd thus must be eqully hrd. For the polynomil lernbility of C from collective behvior, we note tht C is clerly PAC lernble, since it is just Boolen combintions of 1 or 2 inputs. In Section 4 we give generl reduction from collective lerning of ny gent strtegy clss to PAC lerning the clss, thus giving the climed result. Conversely, it is lso not difficult to concoct cses in which lerning the full dynmics in our sense is intrctble, but we cn lern to pproximte specific outcome function from only s 0, F( s T ) pirs. Intuitively, if ech gent strtegy is very complex but the outcome function pplied to finl sttes is sufficiently simple (e.g., constnt), we cnnot but do not need to model the full dynmics in order to lern to pproximte the outcome. We note tht there is n nlogy here to the distinction between direct nd indirect pproches to reinforcement lerning [16]. In the former, one lerns policy tht is specific to fixed rewrd function without lerning model of next-stte dynmics; in the ltter, t possibly greter cost, one lerns n ccurte dynmicl model, which cn in turn be used to compute good policies for ny rewrd function. For the reminder of this pper, we focus on the model s we formlized it in Definition 2, nd leve for future work the investigtion of such lterntives. 3 Socil Strtegy Clsses Before providing our generl theory, including the reduction from collective lerning to I.I.D. lerning, we first illustrte nd motivte the definitions so fr with some concrete exmples of socil strtegy clsses, some of which we nlyze in detil in Section Crowd Affinity: Mixture Strtegies The first clss of gent strtegies we discuss re ment to model settings in which ech individul wishes to blnce their intrinsic personl preferences with desire to follow the crowd. We brodly refer to strtegies of this type s crowd ffinity strtegies (in contrst to the crowd version strtegies discussed shortly), nd exmine couple of nturl vrints. As motivting exmple, imgine tht there re K resturnts, nd ech week, every member of popultion chooses one of the resturnts in which to dine. On the one hnd, ech gent hs personl preferences over the resturnts bsed on the cuisine, service, mbince, nd so on. On the other, ech gent hs some desire to go to the currently hot resturnts tht is, where mny or most other gents hve been recently. To model this setting, let S be the set of K resturnts, nd suppose s S N is the popultion stte vector indicting where ech gent dined lst week. We cn summrize the popultion behvior by the vector or distribution f [0, 1] K, where f is the frction of gents dining in resturnt in s. Similrly, we might represent the personl preferences of specific gent by nother distribution w [0, 1] K in which w represents the probbility this gent would ttend resturnt in the bsence of ny informtion

5 () (b) (c) Figure 1: Smple simultions of the () crowd ffinity mixture model; (b) crowd ffinity multiplictive model; (c) gent ffinity model. Horizontl xis is popultion stte; verticl xis is simultion time. See text for detils. bout wht the popultion is doing. One nturl wy for the gent to blnce their preferences with the popultion behvior would be to choose resturnt ccording to the mixture distribution (1 α) f + α w for some gent-dependent mixture coefficient α. Such models hve been studied in the sociology literture [12] in the context of belief formtion. We re interested in collective systems in which every gent i hs some unknown preferences w i nd mixture coefficient α i, nd in ech week t chooses its next resturnt ccording to (1 α i ) f t + α i w i, which thus probbilisticlly yields the next popultion distribution f t+1. How do such systems behve? And how cn we lern to model their mcroscopic properties from only observed behvior, especilly when the number of choices K is lrge? An illustrtion of the rich collective behvior tht cn lredy be generted from such simple strtegies is shown in Figure 1(). Here we show single but typicl 1000-step simultion of collective behvior under this model, in which N = 100 nd ech gent s individul preference vector w puts ll of its weight on just one of 10 possible ctions (represented s colors); this ction ws selected independently t rndom for ech gent. All gents hve n α vlue of just 0.01, nd thus re selecting from the popultion distribution 99% of the time. Ech row shows the popultion stte t given step, with time incresing down the horizontl xis of the imge. The initil stte ws chosen uniformly t rndom. It is interesting to note the drmtic difference between α = 0 (in which rpid convergence to common color is certin) nd this smll vlue for α; despite the fct tht lmost ll gents ply the popultion distribution t every step, revolving horizontl wves of ner-consensus to different choices re present, with no finl convergence in sight. The slight personliztion of popultion-only behvior is enough to drmticlly chnge the collective behvior. Brodly speking, it is such properties we would like lerning lgorithm to model nd predict from sufficient observtions. 3.2 Crowd Affinity: Multiplictive Strtegies One possible objection to the crowd ffinity mixture strtegies described bove is tht ech gent cn be viewed s rndomly choosing whether to entirely follow the popultion distribution (with probbility 1 α) or to entirely follow their personl preferences (with probbility α) t ech time step. A more relistic model might hve ech gent truly combine the popultion behvior with their preferences t every step. Consider, for instnce, how n Americn citizen might lter their nticipted presidentil voting decision over time in response to recent primry or polling news. If their first choice of cndidte sy, n Independent or Libertrin cndidte ppers over time to be unelectble in the generl election due to their inbility to swy lrge numbers of Democrtic nd Republicn voters, nturl nd typicl response is for the citizen to shift their intended vote to whichever of the front-runners they most prefer or lest dislike. In other words, the low populrity of their first choice cuses tht choice to be dmpened or erdicted; unlike the mixture model bove, where weight α is lwys given to personl preferences, here there my remin no weight on this cndidte. One nturl wy of defining generl such clss of strtegies is s follows. As bove, let f [0, 1] K, where f is the frction of gents dining in resturnt in the current stte s. Similr to the mixture strtegies bove, let w i [0, 1] K be vector of weights representing the intrinsic preferences of gent i over ctions. Then define the probbility tht gent i plys ction to be f w i, /Z( f, w i ), where the normlizing fctor is Z( f, w i ) = b S f b w i,b.

6 Thus, in such multiplictive crowd ffinity models, the probbility the gent tkes n ction is lwys proportionl to the product of their preference for it nd its current populrity. Despite their similr motivtion, the mixture nd multiplictive crowd ffinity strtegies cn led to drmticlly different collective behvior. Perhps the most obvious difference is tht in the mixture cse, if gent i hs strong preference for ction there is lwys some minimum probbility (α i w i, ) they tke this ction, wheres in the multiplictive cse even strong preference cn be erdicted from expression by smll or zero vlues for the populrity f. In Figure 1(b), we gin show single but typicl step, N = 100 simultion for the multiplictive model in which gent s individul preference distributions w re chosen to be rndom normlized vectors over 10 ctions. The dynmics re now quite different thn for the dditive crowd ffinity model. In prticulr, now there is never nerconsensus but grdul dwindling of the colors represented in the popultion from the initil full diversity down to 3 colors remining t pproximtely t = 100, until by t = 200 there is stnd-off in the popultion between red nd light green. Unlike the dditive models, colors die out in the popultion permnently. There is lso cler verticl structure corresponding to strong conditionl preferences of the gents once the stnd-off emerges. 3.3 Crowd Aversion nd Other Vrints It is esy to trnsform the mixture or multiplictive crowd ffinity strtegies into crowd version strtegies tht is, in which gents wish to blnce or combine their personl preferences with desire to ct differently thn the popultion t lrge. This cn be ccomplished in vriety of simple wys. For instnce, if f is the current distributions over ctions in the popultion, we cn simply define kind of inverse to the distribution by letting g = (1 f )/(K 1), where K 1 = b S (1 f b) is the normlizing fctor, nd pplying the strtegies bove to g rther thn f. Now ech gent exhibits tendency to void the crowd, moderted s before by their own preferences. Of course, there is no reson to ssume tht the entire popultion is crowd-seeking, or crowd-voiding; more generlly we would llow there to be both types of individuls present. Furthermore, we might entertin other trnsforms of the popultion distribution thn just g bove. For instnce, we might wish to still consider crowd ffinity, but to first shrpen the distribution by replcing ech f with f 2 nd normlizing, then pplying the models discussed bove to the resulting vector. This hs the effect of mgnifying the ttrction to the most populr ctions. In generl our lgorithmic results re robust to wide rnge of such vritions. 3.4 Agent Affinity nd Aversion Strtegies In the two versions of crowd ffinity strtegies discussed bove, n gent hs personl preferences over ctions, nd lso rects to the current popultion behvior, but only in n ggregte fshion. An lterntive clss of strtegies tht we cll gent ffinity strtegies insted llows gents to prefer to gree (or disgree) in their choice with specific other gents. For fixed gent, such strtegy cn be modeled by weight vector w [0, 1] N, with one weight for ech gent in the popultion rther thn ech ction. We define the probbility tht this gent tkes ction if the current globl stte is s S N to be proportionl to i:s i= w i. In this clss of strtegies, the strength of the gent s desire to tke the sme ction s gent i is determined by how lrge the weight w i is. The overll behvior of this gent is then probbilisticlly determined by summing over ll gents in the fshion bove. In Figure 1(c), we show single but typicl simultion, gin with N = 100 but now with much shorter time horizon of 200 steps nd much lrger set of 100 ctions. All gents hve rndom distributions s their preferences over other gents; this model is similr to trditionl diffusion dynmics in dense, rndom (weighted) network, nd quickly converges to globl consensus. We leve the nlysis of this strtegy clss to future work, but remrk tht in the simple cse in which K = 2, lerning this clss is closely relted to the problem of lerning perceptrons under certin noise models in which the intensity of the noise increses with proximity to the seprtor [5, 4] nd seems t lest s difficult. 3.5 Incorporting Network Structure Mny of the socil models inspiring this work involve network structure tht dicttes or restricts the interctions between gents [18]. It is nturl to sk if the strtegy clsses discussed here cn be extended to the scenrio in which ech gent is influenced only by his neighbors in given network. Indeed, it is strightforwrd to extend ech of the strtegy clsses introduced in this section to network setting. For exmple, to dpt the crowd ffinity nd version strtegy clsses, it suffices to redefine f for ech gent i to be the frction of gents in the locl neighborhood of gent i choosing ction. To dpt the gent ffinity nd version clsses, it is necessry only to require tht w j = 0 for every gent j outside the locl neighborhood of gent i. By mking these simple modifictions, the lerning lgorithms discussed in Section 5 cn immeditely be pplied to settings in which network structure is given. 4 A Reduction to I.I.D. Lerning Since lgorithms in our frmework re ttempting to lern to model the dynmics of fctored Mrkov process in which ech component is known to lie in the clss C, it is nturl to investigte the reltionship between lerning just single strtegy in C nd the entire Mrkovin dynmics. One min concern might be effects of dynmic instbility tht is, tht even smll errors in models for ech of the N components could be mplified exponentilly in the overll popultion model. In this section we show tht this cn be voided. More precisely, we prove tht if the component errors re ll smll compred to 1/(NT) 2, the popultion model lso hs smll error. Thus fst rtes of lerning for individul components re polynomilly preserved in the resulting popultion model. To show this, we give reduction showing tht if clss C of (possibly probbilistic) strtegies is polynomilly lernble (in sense tht we describe shortly) from I.I.D. dt,

7 then C is lso polynomilly lernble from collective behvior. The key step in the reduction is the introduction of the experimentl distribution, defined below. Intuitively, the experimentl distribution is ment to cpture the distribution over sttes tht re encountered in the collective setting over repeted trils. Polynomil I.I.D. lerning on this distribution leds to polynomil lerning from the collective. 4.1 A Reduction for Deterministic Strtegies In order to illustrte some of the key ides we use in the more generl reduction, we begin by exmining the simple cse in which the number of ctions K = 2 nd nd ech strtegy c C is deterministic. We show tht if C is polynomilly lernble in the (distribution-free) PAC model, then C is polynomilly lernble from collective behvior. In order to exploit the fct tht C is PAC lernble, it is first necessry to define single distribution over sttes on which we would like to lern. Definition 4 For ny initil stte distribution P, strtegy vector c, nd sequence length T, the experimentl distribution D P, c,t is the distribution over stte vectors s obtined by querying O EXP ( c, P, T) to obtin s 0,, s T, choosing t uniformly t rndom from {0,, T 1}, nd setting s = s t. We denote this distribution simply s D when P, c, nd T re cler from context. Given ccess to the orcle O EXP, we cn smple pirs s, c i ( s) where s is distributed ccording to D using the following procedure: 1. Query O EXP ( c, P, T) to obtin s 0,, s T. 2. Choose t {0,, T 1} uniformly t rndom. 3. Return s t, s t+1 i. If C is polynomilly lernble in the PAC model, then by definition, with ccess to the orcle O EXP, for ny δ, ǫ > 0, it is possible to lern model ĉ i such tht with probbility 1 (δ/n), Pr s D [ĉ i ( s) c i ( s)] ǫ NT in time polynomil in N, T, 1/ǫ, 1/δ, nd the VC dimension of C using the smpling procedure bove; the dependence on N nd T come from the fct tht we re requesting confidence of 1 (δ/n) nd n ccurcy of ǫ/(tn). We cn lern set of such strtegies ĉ i for ll gents i t the cost of n dditionl fctor of N. Consider new sequence s 0,, s T returned by the orcle O EXP. By the union bound, with probbility 1 δ, the probbility tht there exists ny gent i nd ny t {0,, T 1}, such tht ĉ i ( s t ) c i ( s t ) is less thn ǫ. If this is not the cse (i.e., if ĉ i ( s t ) = c i ( s t ) for ll i nd t) then the sme sequence of sttes would hve been reched if we hd insted strted t stte s 0 nd generted ech dditionl stte s t by letting s t i = c i( s t 1 ). This implies tht with probbility 1 δ, ε(q ˆM, Q c ) ǫ, nd C is polynomilly lernble from collective behvior. 4.2 A Generl Reduction Multiple nlogs of the definition of lernbility in the PAC model hve been proposed for distribution lerning settings. The probbilistic concept model [15] presents definition for lerning conditionl distributions over binry outcomes, while lter work [13] proposes definition for lerning unconditionl distributions over lrger outcome spces. We combine the two into single PAC-style model for lerning conditionl distributions over lrge outcome spces from I.I.D. dt s follows. Definition 5 Let C be clss of probbilistic mppings from n input x X to n output y Y where Y is finite set. We sy tht C is polynomilly lernble if there exists n lgorithm A such tht for ny c C nd ny distribution D over X, if A is given ccess to n orcle producing pirs x, c( x) with x distributed ccording to D, then for ny ǫ, δ > 0, lgorithm A runs in time polynomil in 1/ǫ, 1/δ, nd dim(c) nd outputs function ĉ such tht with probbility 1 δ, E x D Pr(c( x) = y) Pr(ĉ( x) = y) ǫ. y Y We could hve chosen insted to require tht the expected KL divergence between c nd ĉ be bounded. Using Jensen s inequlity nd Lemm of Cover nd Thoms [6], it is simple to show tht if the expected KL divergence between two distributions is bounded by ǫ, then the expected L 1 distnce is bounded by 2 ln(2)ǫ. Thus ny clss tht is polynomilly lernble under this lternte definition is lso polynomilly lernble under ours. Theorem 6 For ny clss C, if C is polynomilly lernble ccording to Definition 5, then C is polynomilly lernble from collective behvior. Proof: This proof is very similr in spirit to the proof of the reduction for deterministic cse. However, severl tricks re needed to del with the fct tht trjectories re now rndom vribles, even given fixed strt stte. In prticulr, it is no longer the cse tht we cn rgue tht strting t given strt stte nd executing set of strtegies tht re close to the true strtegy vector usully yields the sme full trjectory we would hve obtined by executing the true strtegies of ech gent. Insted, due to the inherent rndomness in the strtegies, we must rgue tht the distribution over trjectories is similr when the estimted strtegies re sufficiently close to the true strtegies. To mke this rgument, we begin by introducing the ide of smpling from distribution P 1 using filtered version of second distribution P 2 s follows. First, drw n outcome ω Ω ccording to P 2. If P 1 (ω) P 2 (ω), output ω. Otherwise, output ω with probbility P 1 (ω)/p 2 (ω), nd with probbility 1 P 1 (ω)/p 2 (ω), output n lternte ction drwn ccording to third distribution P 3, where P 1 (ω) P 2 (ω) P 3 (ω) = ω :P 2(ω )<P 1(ω ) P 1(ω ) P 2 (ω ) if P 1 (ω) > P 2 (ω), nd P 3 (ω) = 0 otherwise.

8 It is esy to verify tht the output of this filtering lgorithm is indeed distributed ccording to P 1. Additionlly, notice tht the probbility tht the output is filtered is ( P 2 (ω) 1 P ) 1(ω) = 1 P 2 (ω) 2 P 2 P 1 1. (1) ω:p 2(ω)>P 1(ω) As in the deterministic cse, we mke use of the experimentl distribution D s defined in Definition 4. If C is polynomilly lernble s in Definition 5, then with ccess to the orcle O EXP, for ny δ, ǫ > 0, it is possible to lern model ĉ i such tht with probbility 1 (δ/n), E s D [ s S Pr(c i ( s)=s) Pr(ĉ i ( s)=s) ] ( ǫ ) 2 (2) NT in time polynomil in N, T, 1/ǫ, 1/δ, nd dim(c) using the three-step smpling procedure described in the deterministic cse; s before, the dependence on N nd T stem from the fct tht we re requesting confidence of 1 (δ/n) nd n ccurcy tht is polynomil in both N nd T. It is possible lern set of such strtegies ĉ i for ll gents i t the cost of n dditionl fctor of N. If Eqution 2 is stisfied for gent i, then for ny τ 1, the probbility of drwing stte s from D such tht ( ǫ ) 2 Pr(c i ( s) = s) Pr(ĉ i ( s) = s) τ (3) NT s S is no more thn 1/τ. Consider new sequence s 0,, s T returned by the orcle O EXP. For ech s t, consider the ction s t+1 i chosen by gent i. This ction ws chosen ccording to the distribution c i. Suppose insted we would like to choose this ction ccording to the distribution ĉ i using filtered version of c i s described bove. By Eqution 1, the probbility tht the ction choice of c i is filtered (nd thus not equl to s t+1 i ) is hlf the L 1 distnce between c i ( s t ) nd ĉ i ( s t ). From Eqution 3, we know tht for ny τ 1, with probbility t lest 1 1/τ, this probbility is less thn τ(ǫ/(nt)) 2, so the probbility of the new ction being different from s t+1 i is less thn τ(ǫ/(nt)) 2 + 1/τ. This is minimized when τ = 2NT/ǫ, giving us bound of ǫ/(nt). By the union bound, with probbility 1 δ, the probbility tht there exists ny gent i nd ny t {1,, T }, such tht si t+1 is not equl to the ction we get by smpling ĉ i ( s t ) using the filtered version of c i must then be less thn ǫ. As in the deterministic version, if this is not the cse, then the sme sequence of sttes would hve been reched if we hd insted strted t stte s 0 nd generted ech dditionl stte s t by letting s t i = ĉ i( s t 1 ) filtered using c i. This implies tht with probbility 1 δ, ε(q ˆM, Q c ) ǫ, nd C is polynomilly lernble from collective behvior. 5 Lerning Socil Strtegy Clsses We now turn our ttention to efficient lgorithms for lerning some of the specific socil strtegy clsses introduced in Section 3. We focus on the two crowd ffinity model clsses. Recll tht these clsses re designed to model the scenrio in which ech gent hs n intrinsic set of preferences over ctions, but simultneously would prefer to choose the sme ctions chosen by other gents. Similr techniques cn be pplied to lern the crowd version strtegies. Formlly, let f be vector representing the distribution over current sttes of the gents; if s is the current stte, then for ech ction, f = {i : s i = } /N is the frction of the popultion currently choosing ction. (Alterntely, if there is network structure governing interction mong gents, f cn be defined s the frction of nodes in n gent s locl neighborhood choosing ction.) We denote by D f the distribution over vectors f induced by the experimentl distribution D over stte vectors s. In other words, the probbility of vector f under D f is the sum over ll stte vectors s mpping to f of the probbility of s under D. We focus on the problem of lerning the prmeters of the strtegy of single gent i in ech of the models. We ssume tht we re presented with set of smples M, where ech instnce I m M consists of pir f m, m. Here f m is the distribution over sttes of the gents nd m is the next ction chosen by gent i. We ssume tht the stte distributions f m of these smples re distributed ccording to D f. Given ccess to the orcle O EXP, such smples could be collected, for exmple, using three-step procedure like the one in Section 4.1. We show tht ech clss is polynomilly lernble with respect to the distribution D f induced by ny distribution D over sttes, nd so by Theorem 6, lso polynomilly lernble from collective behvior. While it my seem wsteful to gther only one dt instnce for ech gent i from ech T -trjectory, we remrk tht only smll, isolted pieces of the nlysis presented in this section rely on the ssumption tht the stte distributions of the smples re distributed ccording to D f. In prctice, the entire trjectories could be used for lerning with no impct on the structure of the lgorithms. Additionlly, while the nlysis here is gered towrds lerning under the experimentl distribution, the lgorithms we present cn be pplied without modifiction in the no-reset vrint of the model introduced in Section 2.3. We briefly discuss how to extend the nlysis to the no-reset vrint in Section Lerning Crowd Affinity Mixture Models In Section 3.1, we introduced the clss of crowd ffinity mixture model strtegies. Such strtegies re prmeterized by (normlized) weight vector w nd prmeter α [0, 1]. The probbility tht gent i chooses ction given tht the current stte distribution is f is then αf + (1 α)w. In this section, we show tht this clss of strtegies is polynomilly lernble from collective behvior nd sketch n lgorithm for lerning estimtes of the prmeters α nd w. Let I(x) be the indictor function tht is 1 if x is true nd 0 otherwise. From the definition of the model it is esy to see tht for ny m such tht I m M, for ny ction S, E[I( m = )] = αf + (1 α)w, where the expecttion is over the rndomness in the gent s strtegy. By linerity of expecttion, [ ] E I( m = ) =α f m, +(1 α)w M. (4)

9 Stndrd results from uniform convergence theory sy tht we cn pproximte the left-hnd side of this eqution rbitrrily well given sufficiently lrge dt set M. Replcing the expecttion with this pproximtion in Eqution 4 yields single eqution with two unknown vribles, α nd w. To solve for these vribles, we must construct pir of equtions with two unknown vribles. We do so by splitting the dt into instnces where f m, is nd instnces where it is low. Specificlly, let M = M. For convenience of nottion, ssume without loss of generlity tht M is even; if M is odd, simply discrd n instnce t rndom. Define M low to be the set contining the M/2 instnces in M with the lowest vlues of f m,. Similrly, define M to be the set contining the M/2 instnces with the est vlues of f m,. Replcing M with M low nd M respectively in Eqution 4 gives us two liner equtions with two unknowns. As long s these two equtions re linerly independent, we cn solve the system of equtions for α, giving us E[ α= ˆα= I( m =) ] I( low m =). f m, f m, low We cn pproximte α from dt in the nturl wy, using I( m =) I( low m =) f m, low f m,. (5) By Hoeffding s inequlity nd the union bound, for ny δ > 0, with probbility 1 δ, ln(4/δ)m α ˆα f m, f low m, where Z = 1 M/2 = (1/Z ) ln(4/δ)/m, (6) f m, 1 M/2 low f m,. The quntity Z mesures the difference between the men vlue of f m, mong instnces with vlues of f m, nd the men vlue of f m, mong instnces with low vlues. While this quntity is dt-dependent, stndrd uniform convergence theory tells us tht it is stble once the dt set is lrge. From Eqution 6, we know tht if there is n ction for which this difference is sufficiently, then it is possible to obtin n ccurte estimte of α given enough dt. If, on the other hnd, no such exists, it follows tht there is very little vrince in the popultion distribution over the smple. We rgue below tht it is not necessry to lern α in order to mimic the behvior of n gent i if this is the cse. For now, ssume tht Z is sufficiently lrge for t lest one vlue of, nd cll this vlue. We cn use the estimte of α to obtin estimtes of the weights for ech ction. From Eqution 4, it is cler tht for ny, w = E [ I( m = ) ] α f m, (1 α)m. We estimte this weight using m:i ŵ = I( m M m = ) ˆα m:i f m M m,. (7) (1 ˆα)M The following lemm shows tht given sufficient dt, the error in these estimtes is smll when Z is lrge. Lemm 7 Let = rgmx S Z, nd let ˆα be clculted s in Eqution 5 with =. For ech S, let ŵ be clculted s in Eqution 7. For sufficiently lrge M, for ny δ > 0, with probbility 1 δ, α ˆα (1/Z ) ln((4 + 2K)/δ)/M, nd for ll ctions, w ŵ ((1 ˆα)Z / 2 + 2) ln((4 + 2K)/δ) Z (1 ˆα) 2 M (1 ˆα) ln((4 + 2K)/δ). The proof of this lemm, which is in the ppendix, relies hevily on the following technicl lemm for bounding the error of estimted rtios, which is used frequently throughout the reminder of the pper. Lemm 8 For ny positive u, û, v, ˆv, k, nd ǫ such tht ǫk < v, if u û ǫ nd v ˆv kǫ, then u v û ǫ(v + uk) ˆv v(v ǫk). Now tht we hve bounds on the error of the estimted prmeters, we cn bound the expected L 1 distnce between the estimted model nd the rel model. Lemm 9 For sufficiently lrge M, (αf + (1 α)w ) (ˆαf + (1 ˆα)ŵ ) E f D f S 2 ln((4 + 2K)/δ) Z M { K(Z / 2 + 2) ln((4 + 2K)/δ) + min Z (1 ˆα) M ln((4 + 2K)/δ), } 2(1 ˆα). In this proof of this lemm, which ppers in the ppendix, the quntity (αf + (1 α)w ) (ˆαf + (1 ˆα)ŵ ) S is bounded uniformly for ll f using the error bounds. The bound on the expecttion follows immeditely. It remins to show tht we cn still bound the error when Z is zero or very close to zero. We present light sketch of the rgument here; more detils pper in the ppendix. Let η nd µ be the true medin nd men of the distribution from which the rndom vribles f m, re drwn. Let f be the men vlue of the distribution over f m,

10 conditioned on f m, > η. Let f be the empiricl verge of f m, conditioned on f m, > η. Finlly, let ˆf = (2/M) f m, be the empiricl verge of f m, conditioned on f m, being greter thn the empiricl medin. We cn clculte ˆf from dt. We cn pply stndrd rguments from uniform convergence theory to show tht f is close to f, nd in turn tht f is close to ˆf. Similr sttements cn be mde for the nlogous quntities f low low low, f, nd ˆf. By noting low tht Z = ˆf ˆf this implies tht if Z is smll, then the probbility tht rndom vlue of f m, is fr from the men µ is smll. When this is the cse, it is not necessry to estimte α directly. Insted, we set ˆα = 0 nd ŵ = 1 I( m = ). M Applying Hoeffding s inequlity gin, it is esy to show tht for ech, ŵ is very close to αµ + (1 α)w, nd from here it cn be rgued tht the L 1 distnce between the estimted model nd the rel model is smll. Thus for ny distribution D over stte vectors, regrdless of the corresponding vlue of Z, it is possible to build n ccurte model for the strtegy of gent i in polynomil time. By Theorem 6, this implies tht the clss is polynomilly lernble from collective behvior. Theorem 10 The clss of crowd ffinity mixture model strtegies is polynomilly lernble from collective behvior. 5.2 Lerning Crowd Affinity Multiplictive Models In Section 3.2, we introduced the crowd ffinity multiplictive model. In this model, strtegies re prmeterized only by weight vector w. The probbility tht gent i chooses ction is simply f w / b S f bw b. Although the motivtion for this model is similr to tht for the mixture model, the dynmics of the system re quite different (see the simultions nd discussion in Section 3), nd very different lgorithm is necessry to lern individul strtegies. In this section, we show tht this clss is polynomilly lernble from collective behvior, nd sketch the corresponding lerning lgorithm. The lgorithm we present is bsed on simple but powerful observtion. In prticulr, consider the following rndom vrible: χ m = { 1/fm, if f m, > 0 nd m =, 0 otherwise. Suppose tht for ll m such tht I m M, it is the cse tht f m, > 0. Then by the definition of the strtegy clss nd linerity of expecttion, [ ] ( ) E χ m 1 fm, w = f m, s S f m,sw s = w 1 s S f m,sw s, where the expecttion is over the rndomness in the gent s strtegy. Notice tht this expression is the product of two terms. The first, w, is precisely the vlue we would like to clculte. The second term is something tht depends on the set of instnces M, but does not depend on ction. This leds to the key observtion t the core of our lgorithm. Specificlly, if we hve second ction b such tht f m,b > 0 for ll m such tht I m M, then w = E [ ] χm w b E [ ]. χm b Although we do not know the vlues of these expecttions, we cn pproximte them rbitrrily well given enough dt. Since we hve ssumed (so fr) tht f m, > 0 for ll m M, nd we know tht f m, represents frction of the popultion, it must be the cse tht f m, 1/N nd χ m [0, N] for ll m. By stndrd ppliction of Hoeffding s inequlity nd the union bound, we see tht for ny δ > 0, with probbility 1 δ, [ χ m E χ m ] N ln(2/δ). (8) 2 M This leds to the following lemm. We note tht the role of β in this lemm my pper somewht mysterious. It comes the fct tht we re bounding the error of rtio of two terms; n ppliction of Lemm 8 using the bound in Eqution 8 gives us fctor of χ,b + χ b, in the numertor nd fctor of χ b, in the denomintor. This is problemtic only when χ,b is significntly lrger thn χ b,. The full proof ppers in the ppendix. Lemm 11 Suppose tht f m, > 0 nd f m,b > 0 for ll m such tht I m M. Then for ny δ > 0, with probbility 1 δ, for ny β > 0, if χ,b βχ b, nd χ b, 1, then if M N ln(2/δ)/2, then w w b χm χm b (1 + β) N ln(2/δ). 2 M N ln(2/δ) If we re fortunte enough to hve sufficient number of dt instnces for which f m, > 0 for ll S, then this lemm supplies us with wy of pproximting the rtios between ll pirs of weights nd subsequently pproximting the weights themselves. In generl, however, this my not be the cse. Luckily, it is possible to estimte the rtio of the weights of ech pir of ctions nd b tht re used together frequently by the popultion using only those dt instnces in which t lest one gent is choosing ech. Formlly, define M,b = {I m M : f m, > 0, f m,b > 0}. Lemm 11 tells us tht if M,b is sufficiently lrge, nd there is t lest one instnce I m M,b for which m = b, then we cn pproximte the rtio between w nd w b well. Wht if one of these ssumptions does not hold? If we re not ble to collect sufficiently mny instnces in which f m, > 0 nd f m,b > 0, then stndrd uniform convergence results cn be used to show tht it is very unlikely tht we see new instnce for which f > 0 nd f b > 0. This ide is formlized in the following lemm, the proof of which is in the ppendix.

11 Lemm 12 For ny M < M, for ny δ (0, 1), with probbility 1 δ, Pr f D f [, b S : f > 0, f b > 0, M,b < M] K2 2 ( M M + ln(k 2 /(2δ)) 2 M Similrly, if χ,b = χ b, = 0, then stndrd uniform convergence rgument cn be used to show tht it is unlikely tht gent i would ever select ction or b when f m, > 0 nd f m,b > 0. We will see tht in this cse, it is not importnt to lern the rtio between these two weights. Using these observtions, we cn ccurtely model the behvior of gent i. The model consists of two phses. First, s preprocessing step, we clculte quntity χ,b =,b χ m for ech pir, b S. Then, ech time we re presented with stte f, we clculte set of weights for ll ctions with f > 0 on the fly. For fixed f, let S be the set of ctions S such tht f > 0. By Lemm 12, if the dt set is sufficiently lrge, then we know tht with probbility, it is the cse tht for ll, b S, M,b M for some threshold M. Now, let = rgmx S {b : b S, χ,b χ b, }. Intuitively, if there is sufficient dt, should be the ction in S with the est weight, or hve weight rbitrrily close to the est. Thus for ny S, Lemm 11 cn be used to bound our estimte of w /w with vlue of β rbitrrily close to 1. Noting tht w w /w s S w = s s S w, s/w we pproximte the reltive weight of ction S with respect to the other ctions in S using χ, /χ, ŵ = s S χ, s, /χ,s nd simply let ŵ = 0 for ny S. Applying Lemm 8, we find tht for ll S, with probbility, w s S w ŵ s (1 + β)k N ln(2k 2 /δ) 2M (1 + β)k N ln(2k2 /δ), (9) where M is the lower bound on M,b for ll, b S, nd β is close to 1. With this bound in plce, it is strightforwrd to show tht we cn pply Lemm 8 once more to bound the expected L 1, [ ] E f D w f ŵ f f s S w sf s, S s S ŵsf s nd tht the bound goes to 0 t rte of O(1/ M) s the threshold M grows. More detils re given in the ppendix. Since it is possible to build n ccurte model of the strtegy of gent i in polynomil time under ny distribution D over stte vectors, we cn gin pply Theorem 6 to see tht this clss is polynomilly lernble from collective behvior. ). Theorem 13 The clss of crowd ffinity multiplictive model strtegies is polynomilly lernble from collective behvior. 5.3 Lerning Without Resets Although the nlyses in the previous subsections re tilored to lernbility in the sense of Definition 2, they cn esily be dpted to hold in the lternte setting in which the lerner hs ccess only to single, unbroken trjectory of sttes. In this lternte model, the lerning lgorithm observes polynomilly long prefix of trjectory of sttes for trining, nd then must produce genertive model which results in distribution over the vlues of the subsequent T sttes close to the true distribution. When lerning individul crowd ffinity models for ech gent in this setting, we gin ssume tht we re presented with set of smples M, where ech instnce I m M consists of pir f m, m. However, insted of ssuming tht the stte distributions f m re distributed ccording to D f, we now ssume tht the stte nd ction pirs represent single trjectory. As previously noted, the mjority of the nlysis for both the mixture nd multiplictive vrints of the crowd ffinity model does not depend on the prticulr wy in which stte distribution vectors re distributed, nd thus crries over to this setting s is. Here we briefly discuss the few modifictions tht re necessry. The only chnge required in the nlysis of the crowd ffinity mixture model reltes to hndling the cse in which Z is smll for ll. Previously we rgued tht when this is the cse, the distribution D f must be concentrted so tht for ll, f flls within very smll rnge with probbility. Thus it is not necessry to estimte the prmeter α directly, nd we cn insted lern single probbility for ech ction tht is used regrdless of f. A similr rgument holds in the no-reset vrint. If it is the cse tht Z is smll for ll, then it must be the cse tht for ech, the vlue of f hs fllen into the sme smll rnge for the entire observed trjectory. A stndrd uniform convergence rgument sys tht the probbility tht f suddenly chnges drmticlly is very smll, nd thus gin it is sufficient to lern single probbility for ech ction tht is used regrdless of f. To dpt the nlysis of the crowd ffinity multiplictive model, it is first necessry to replce Lemm 12. Recll tht the purpose of this lemm ws to show tht when the dt set does not contin sufficient smples in which f > 0 nd f b > 0 for pir of ctions nd b, the chnce of observing new stte distribution f with f > 0 nd f b > 0 is smll. This rgument is ctully much more strightforwrd in the no-reset cse. By the definition of the model, it is esy to see tht if f > 0 for some ction t time t in trjectory, then it must be the cse tht f > 0 t ll previous points in the trjectory. Thus if f > 0 on ny test instnce, then f must hve been non-negtive on every trining instnce, nd we do not hve to worry bout the cse in which there is insufficient dt to compre the weights of prticulr pir of ctions. One dditionl, possibly more subtle, modifiction is necessry in the nlysis of the multiplictive model to hndle the cse in which χ,b = χ b, = 0 for ll ctive pirs of ctions, b S. This cn hppen only if gent i hs

12 extremely smll weights for every ction in S, nd hd previously been choosing n lternte ction tht is no longer vilble, i.e., n ction s for which f s hd previously been non-negtive but suddenly is not. However, in order for f s to become 0, it must be the cse tht gent i himself chooses n lternte ction (sy, ction ) insted of s, which cnnot hppen since the estimted weight of ction used by the model is 0. Thus this sitution cn never occur in the no-reset vrint. 6 Conclusions nd Future Work We hve introduced computtionl model for lerning from collective behvior, nd populted it with some initil generl theory nd lgorithmic results for crowd ffinity models. In ddition to positive or negtive results for further gent strtegy clsses, there re number of other generl directions of interest for future reserch. These include extension of our model to gnostic [14] settings, in which we relx the ssumption tht every gent strtegy flls in known clss, nd to reinforcement lerning [23] settings, in which the lerning lgorithm my itself be member of the popultion being modeled, nd wishes to lern n optiml policy with respect to some rewrd function. Acknowledgments We thnk Nin Blcn nd Eyl Even-Dr for erly discussions on models of socil lerning, nd Duncn Wtts for helpful converstions nd pointers to relevnt literture. References [1] S. Bikhchndni, D. Hirshleifer, nd I. Welch. A theory of fds, fshion, custom, nd culturl chnge s informtionl cscdes. Journl of Politicl Economy, 100: , [2] S. Bikhchndni, D. Hirshleifer, nd I. Welch. Lerning from the behvior of others: Conformity, fds, nd informtionl cscdes. The Journl of Economic Perspectives, 12: , [3] G. W. Brown. Itertive solutions of gmes by fictitious ply. In T.C. Koopmns, editor, Activity Anlysis of Production nd Alloction. Wiley, [4] T. Bylnder. Lerning noisy liner threshold functions. Technicl Report, [5] E. Cohen. Lerning noisy perceptrons by perceptron in polynomil time. In 38th IEEE Annul Symposium on Foundtions of Computer Science, [6] T. Cover nd J. Thoms. Elements of Informtion Theory. John Wiley & Sons, New York, NY, [7] P. Dodds, R. Muhmd, nd D. J. Wtts. An experimentl study of serch in globl socil networks. Science, 301: , August [8] M. Drehmnn, J. Oechssler, nd A. Roider. Herding nd contrrin behvior in finncil mrkets: An Internet experiment. Americn Economic Review, 95(5): , [10] M. Grnovetter. Threshold models of collective behvior. Americn Journl of Sociology, 61: , [11] D. Hussler. Decision theoretic generliztions of the PAC model for neurl net nd other lerning pplictions. Informtion nd Computtion, 100:78 150, [12] P. Hedstrom. Rtionl imittion. In P. Hedstrom nd R. Swedberg, editors, Socil Mechnisms: An Anlyticl Approch to Socil Theory. Cmbridge University Press, [13] M. Kerns, Y. Mnsour, D. Ron, R. Rubinfeld, R.E. Schpire, nd L. Sellie. On the lernbility of discrete distributions. In 26th Annul ACM Symposium on Theory of Computing, pges , [14] M. Kerns, R. Schpire, nd L. Sellie. Towrds efficient gnostic lerning. Mchine Lerning, 17: , [15] M. Kerns nd R. E. Schpire. Efficient distribution-free lerning of probbilistic concepts. Journl of Computer nd System Sciences, 48(3): , [16] M. Kerns nd S. Singh. Finite-smple rtes of convergence for Q-lerning nd indirect methods. In Advnces in Neurl Informtion Processing Systems 11, [17] M. Kerns, S. Suri, nd N. Montfort. A behviorl study of the coloring problem on humn subject networks. Science, 313(5788): , [18] J. Kleinberg. Cscding behvior in networks: Algorithmic nd economic issues. In N. Nisn, T. Roughgrden, E. Trdos, nd V. Vzirni, editors, Algorithmic Gme Theory. Cmbridge University Press, [19] S. Russell nd P. Norvig. Artificil Intelligence: A Modern Approch. Prentice Hll, [20] M. Slgnik, P. Dodds, nd D. J. Wtts. Experimentl study of inequlity nd unpredictbility in n rtificil culturl mrket. Science, 331(5762): , [21] M. Slgnik nd D. J. Wtts. Socil influence, mnipultion, nd self-fulfilling prophecies in culturl mrkets. Preprint, [22] T. Schelling. Micromotives nd Mcrobehvior. Norton, New York, NY, [23] R. Sutton nd A. Brto. Reinforcement Lerning. MIT Press, [24] A. De Vny. Hollywood Economics: How Extreme Uncertinty Shpes the Film Industry. Routledge, London, [25] A. De Vny nd C. Lee. Qulity signls in informtion cscdes nd the dynmics of the distribution of motion picture box office revenues. Journl of Economic Dynmics nd Control, 25: , [26] C. Wtkins nd P. Dyn. Q-lerning. Mchine Lerning, 8(3): , [27] I. Welch. Herding mong security nlysts. Journl of Finncil Economics, 58: , [9] D. Foster nd R. Vohr. Regret in the on-line decision problem. Gmes nd Economic Behvior, 29:7 35, 1999.

13 A Proofs from Section 5.1 A.1 Proof of Lemm 8 For the first direction, Similrly, u v û ˆv û ˆv u v u v u ǫ v + kǫ = u uv ǫv v v(v + kǫ) = u uv + ukǫ ǫv ukǫ v v(v + kǫ) = u u(v + kǫ) ǫ(v + uk) v v(v + kǫ) ǫ(v + uk) ǫ(v + uk) = v(v + ǫk) v(v ǫk). = = = A.2 Proof of Lemm 7 u + ǫ v kǫ u uv + ǫv = v v(v kǫ) u v uv ukǫ + ǫv + ukǫ u v(v kǫ) v u(v kǫ) + ǫ(v + uk) u v(v kǫ) v ǫ(v + uk) v(v ǫk). We bound the error of these estimtions in two prts. First, since from Eqution 6 we know tht for ny δ 1 > 0, with probbility 1 δ 1, ˆα f m, α f m, 1 Z ln(4/δ1 ) M f m, 1 M ln(4/δ1 ), Z nd (1 ˆα)M (1 α)m 1 M ln(4/δ1 ), Z we hve by Lemm 8 tht for sufficiently lrge M Setting δ 2 = Kδ 1 /2, we cn gin pply Lemm 8 nd see tht for sufficiently lrge M m:i I( m M m = ) E [ m:i I( m M m = ) ] (1 ˆα)M (1 α)m M ln(4/δ1 )/2 ( (1 ˆα)Z M + 2M ) = Z (1 ˆα) 2 M 2 (1 ˆα)M M ln(4/δ 1 ) ln(4/δ1 )/2 ( (1 ˆα)Z + 2 ) Z (1 ˆα) 2 M (1 ˆα) ln(4/δ 1 ) ((1 ˆα)Z / 2 + 1) ln(4/δ 1 ) Z (1 ˆα) 2 M (1 ˆα) ln(4/δ 1 ). Setting δ 1 = δ/(1+k/2) nd pplying the union bound yields the lemm. A.3 Proof of Lemm 9 As long s M is sufficiently lrge, for ny fixed f, (αf + (1 α)w ) (ˆαf + (1 ˆα)ŵ ) S S α ˆα f + S α ˆα + S 2 α ˆα + (1 ˆα) S 2 ln((4 + 2K)/δ) (1 α)w (1 ˆα)ŵ ( α ˆα ŵ + w ŵ (1 ˆα) α ˆα w ŵ ) α ˆα S w ŵ w ŵ Z M { K(Z / 2 + 2) ln((4 + 2K)/δ) + min Z (1 ˆα) M ln((4 + 2K)/δ), 2(1 ˆα)}. ˆα m:i f m M m, α m:i f m M m, Notice tht this holds uniformly for ll f, so the sme (1 ˆα)M (1 α)m bound holds when we tke n expecttion over f. (1/Z ) M ln(4/δ 1 ) ( (1 ˆα)M + ˆα m:i f ) m M m, (1 ˆα) 2 M 2 (1 ˆα)M A.4 Hndling the cse where Z is smll M ln(4/δ 1 )/Z Suppose tht for n ction, Z < ǫ. Let η nd µ be the true medin nd men respectively of the distribution M ln(4/δ1 )((1 ˆα)M + ˆαM) Z (1 ˆα) 2 M 2 (1 ˆα)M from which the rndom vribles f m, re drwn. Let f M ln(4/δ 1 ) be the men vlue of the distribution over f m, conditioned on f ln(4/δ1 ) m, > η, nd similrly let f = Z (1 ˆα) 2 M (1 ˆα) ln(4/δ 1 ). low be the men vlue conditioned on f m, < η, so µ = ( ) f low + f /2. 1 Let f be the empiricl verge of f m, conditioned on Now, by Hoeffding s inequlity nd the union bound, for low f m, > η, nd f be the empiricl verge of f m, ny δ 2 > 0, with probbility 1 δ 2, for ll, conditioned on f m, < η. (Note tht we cnnot ctu- [ low lly compute f nd f since η is unknown.) Fi- I( m = ) E I( m = )] 1 Assume for now tht it is never the cse tht f m:i m, = η m M. This simplifies the explntion, lthough everything still holds if f m, M ln(2k/δ 2 )/2. cn be η.

14 nlly, let ˆf (2/M) low We show first tht f = (2/M) f m,. Notice tht Z = f m, nd ˆf low ˆf = low ˆf. f nd nd f low re close to f low respectively. Next we show tht f low nd f re low close to ˆf nd ˆf respectively. Finlly, we show tht this implies tht if Z is smll, then the probbility tht rndom vlue of f m, is fr from η is smll. This in turn implies smll L 1 distnce between our estimted model nd the rel model for ech gent. f To bound the difference between f nd, it is first necessry to lower bound the number of points in the empiricl smples with f m, > η. Let z m be rndom vrible tht is 1 if f m, > η nd 0 otherwise. Clerly Pr(z m = 1) = Pr(z m = 0) = 1/2. By strightforwrd ppliction of Hoeffding s inequlity, for ny δ 3, with probbility 1 δ 3, z m M 2 ln(2/δ3 )M, 2 f nd so the number of smples verged to get is t lest (M/2) ln(2/δ 3 )M/2. Applying Hoeffding s inequlity gin, with probbility 1 δ 4, f f ln(2/δ 4 ) f ˆf M 2 ln(2/δ 3 )M. Now, ˆf is n empiricl verge of M/2 vlues in [0, 1], while f is n empiricl verge of the sme points, plus or minus up to ln(2/δ 3 )M/2 points. In the worst cse, f either includes n dditionl ln(2/δ 3 )M/2 points with vlues lower thn ny point in M, or excludes the ln(2/δ 3 )M/2 lowest vlues points in M. This implies tht in the worst cse, ln(2/δ3 ) M/2 ln(2/δ3 ). By the tringle inequlity, f ln(2/δ 4 ) ˆf M 2 ln(2/δ 3 )M ln(2/δ3 ) + M/2 ln(2/δ3 ). The sme cn be show for f low nd ǫ, then f low ln(2/δ 4 ) ǫ + 2 f Cll this quntity ǫ. Clerly we hve µ ǫ f low low ˆf. Hence if Z M 2 ln(2/δ 3 )M 2 ln(2/δ 3 ) + M/2 ln(2/δ3 ). µ f µ + ǫ. Since f is n verge of points which re ll er thn f low, nd similrly f low is n verge of points ll lower thn f, this implies tht for ny τ 0, Pr( f m, µ ǫ + τ) Pr(f m, f ǫ /(ǫ + τ). + τ) + Pr(f m, f low τ) Recll tht when Z is smll for ll, we set ˆα = 0 nd ŵ = m:i I( m M m = ). Let w = αµ + (1 α)w. Notice tht E[ŵ ] = w. Applying Hoeffding s inequlity yet gin, with probbility 1 δ 5, ŵ w ln(2/δ5 )/2M. For ny τ > 0, [ ] (αf + (1 α)w ) (ˆαf + (1 ˆα)ŵ ) E f D f S = S E f D f [ αf + (1 α)w ŵ ] S E f D f [ αf + (1 α)w w + w ŵ ] α S E f D f [ f µ ] + K ln(2/δ 5 )/2M ) ) K ((1 ǫ ǫ (ǫ + τ) + ǫ + τ ǫ + τ +K ln(2/δ 5 )/2M = K (τ + ǫ ǫ + τ + ) ln(2/δ 5 )/2M. B Proofs from Section 5.2 B.1 Proof of Lemm 12 By the union bound, Pr f D f [, b S : f > 0, f b > 0, M,b < M] Pr f D f [f > 0, f b > 0, M,b < M],b S,b S: M,b <M Pr f D f [f > 0, f b > 0]. For ny fixed pir of ctions, b such tht M,b < M, it follows from Hoeffding s inequlity tht for ny δ (0, 1), with probbility 1 δ, Pr f D f [f > 0, f b > 0] M M + ln(1/δ). 2 M Noting tht the number of pirs is less thn K 2 /2 nd setting δ = 2δ/K 2 yields the lemm. B.2 Bounding the L 1 Here we show tht with probbility (over the choice of M nd the drw of f), if M is sufficiently lrge, w f ŵ f s S w sf s S s S ŵsf s 2(1 + β)kn N ln(2k/δ) 2M (1 + β)k(n + 1) N ln(2k/δ).

15 First note tht we cn rewrite the expression on the lefthnd side of the inequlity bove s w f ŵ f s S w s f s s S ŵ sf s, S where for ll, w = w /( s S w s). We know from Eqution 9 tht with probbility (over the dt set nd the choice of f), for ll S, w f ŵ f f (1 + β) S N ln(2k 2 /δ) 2M (1 + β) S N ln(2k 2 /δ), nd w s f s w s f s s S s S f s w s ŵ s s S (1 + β) S N ln(2k 2 /δ) 2M (1 + β) S N ln(2k 2 /δ). Thus, using the fct tht s S ŵ s f s 1/N, we cn pply Lemm 8 once gin to get tht w f ŵ f s S w s f s s S ŵ s f s (1 + β)kn ( ) ŵ N ln(2k/δ) f + P f s S ŵsfs, 2M (1 + β)k(n + 1) N ln(2k/δ) nd w f s S w s f s S ŵ f ŵ s f s s S 2(1 + β)kn N ln(2k/δ) 2M (1 + β)k(n + 1) N ln(2k/δ).