EMBARGO: December 4th, 2014, 11am Pacific/2pm Eastern/7pm UK. The Social Bayesian Brain: Does Mentalizing Make a Difference When We Learn?

Transcription

1 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK The Social Bayesian Brain: Does Menalizing Make a Difference When We Learn? Marie Devaine 1,2, Guillaume Hollard 3,4, Jean Daunizeau 1,2,5 * 1 Brain and Spine Insiue, Paris, France, 2 INSERM, Paris, France, 3 Maison des Sciences Economiques, Paris, France, 4 CNRS UMR, Paris, France, 5 ETH, Zurich, Swizerland Absrac When i comes o inerpreing ohers behaviour, we almos irrepressibly engage in he aribuion of menal saes (beliefs, emoions ). Such "menalizing" can become very sophisicaed, evenually endowing us wih highly adapive skills such as convincing, eaching or deceiving. Here, sophisicaion can be capured in erms of he deph of our recursive beliefs, as in "I hink ha you hink ha I hink " In his work, we es wheher such sophisicaed recursive beliefs subend learning in he conex of social ineracion. We asked paricipans o play repeaed games agains arificial (Bayesian) menalizing agens, which differ in heir sophisicaion. Criically, we made people believe eiher ha hey were playing agains each oher, or ha hey were gambling like in a casino. Alhough boh framings are similarly deceiving, paricipans win agains he arificial (sophisicaed) menalizing agens in he social framing of he ask, and lose in he non-social framing. Moreover, we find ha paricipans choice sequences are bes explained by sophisicaed menalizing Bayesian learning models only in he social framing. This sudy is he firs demonsraion of he added-value of menalizing on learning in he conex of repeaed social ineracions. Imporanly, our resuls show ha we would no be able o decipher inenional behaviour wihou a priori aribuing menal saes o ohers. Ciaion: Devaine M, Hollard G, Daunizeau J (2014) The Social Bayesian Brain: Does Menalizing Make a Difference When We Learn? PLoS Compu Biol 10(12): e doi: /journal.pcbi Edior: Jeff Beck, Duke Universiy, Unied Saes of America Received March 17, 2014; Acceped Ocober 18, 2014; Published December 4, 2014 Copyrigh: ß 2014 Devaine e al. This is an open-access aricle disribued under he erms of he Creaive Commons Aribuion License, which permis unresriced use, disribuion, and reproducion in any medium, provided he original auhor and source are credied. Daa Availabiliy: The auhors confirm ha all daa underlying he findings are fully available wihou resricion. All relevan daa are available for download a he following URL: hps:// = 0 Funding: This work was suppored by he European Research Council (JD), he IHU-A-ICM (MD, JD) and he French Minisère de l Enseignemen Supérieur e de la Recherche (MD). The funders had no role in sudy design, daa collecion and analysis, decision o publish, or preparaion of he manuscrip. Compeing Ineress: The auhors have declared ha no compeing ineress exis. * jean.daunizeau@gmail.com Inroducion Wha is so special abou he way we selec he mos appropriae acion in a social conex? We make decisions on he basis of heir expeced consequences, which we may have o learn from rial and error. However, when his involves predicing oher peoples over reacions, we almos irrepressibly engage in rich and complex represenaions of heir hidden menal saes, such as beliefs, emoions, inenions In fac, one of he mos criical aspecs of social inference may be our insigh ha people s behaviour is driven by heir beliefs raher han by physical realiy, even if hese beliefs happen o be false [1]. In his work, we ask wheher his specific aspec of social cogniion makes a difference when we learn. We acquire his insigh during early childhood [2], from our developing abiliy o aribue menal saes o ohers, known as "Theory of Mind" (ToM) or "menalizing" [3]. ToM is concerned wih he inerpreaion of social signals, from eye gazes and facial expressions o over behaviour and language, which is why i lies a he core of human social cogniion [4]. We know ha ToM engages large-scale specific brain neworks [5,6] and ha severe neuropsychiaric disorders such as schizophrenia or auism are associaed wih is impairmen [7,8]. However, curren research falls shor of an undersanding of he compuaional mechanisms underlying menalizing, or of a clear demonsraion of is addedvalue for decision making in social exchanges [1]. Here, we ake inspiraion from recen works in behavioural economics and experimenal psychology, which invesigae sophisicaed menalizing processes, of he sor ha adapive social skills such as persuading or deceiving proceed from. On he one hand, i has been shown ha decisions made in he conex of economic games enail recursive hinking of he sor "I hink ha you hink ha I hink, ec " [9,10]. This is essenially because if ohers reward depends upon your acion, wha hey believe you will do is relevan for you o predic heir behaviour. On he oher hand, i has been suggesed ha simple forms of acion undersanding conform wih Bayesian models of inenion recogniion [11,12]. This means ha our inerpreaion of ohers acions is opimal, under he insigh ha ohers behave according o common sense. Taken ogeher, hese ideas yield he "social Bayesian brain" hypohesis, namely: our (Bayesian) brain a priori assumes ha ohers are Bayesian oo (i.e. ohers also learn abou ourselves) [13 15]. In he conex of muual social exchanges, his implies ha menalizing may involve he updae of recursive beliefs from he repeaed observaion of ohers over behaviour. From a modelling perspecive, one can define opimal learning rules ha are rooed in informaion heory and are specific o he sophisicaion of menalizing agens (i.e., he deph k of heir recursive beliefs). This is imporan, because one can now evaluae he added value of some form of menalizing sophisicaion, in erms of is abiliy o decipher inenional behaviour. Criically, our k-tom model predics ha he performance of agens engaged in compeiive repeaed ineracions increases wih heir ToM sophisicaion [14]. PLOS Compuaional Biology 1 December 2014 Volume 10 Issue 12 e

2 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK Menalizing and Learning Auhor Summary A defining feaure of human social cogniion is our insigh ha ohers behaviour is driven by heir beliefs and preferences, raher han by wha is objecively rue or good for hem. In fac, a grea deal of our social ineracions are concerned wih guessing ohers menal saes. Bu is such "menalizing" of any help for predicing ohers behaviour? Afer all, mos animal species seem o cope wih his problem wihou appealing o any form of sophisicaed "Theory of Mind". Here, sophisicaion refers o he deph of recursive beliefs, as in "I hink ha you hink ha I hink " Alhough we are likely o engage in such recursive beliefs whenever our ineress are ied up wih ohers (e.g. in he aim of deceiving hem), i is unclear how hese beliefs are updaed and wheher his gives us any advanage when we learn. These are he quesions we address in his work, by combining compuaional and experimenal approaches. We es hese ideas in he following experimen: we had paricipans believe eiher ha hey were playing a compeiive game wih each oher, or ha hey were performing a gambling ask. In fac, in boh condiions, paricipans were compeing agains arificial k-tom agens wih differen ToM sophisicaion levels. Criically, he ask-relevan informaion (available acions and correc/incorrec feedback), is idenical in boh framings. Our predicion is wofold: (i) he social framing of he ask induces paricipans o menalize and hus o engage in recursive inference, and (ii) domain-general learning heurisics ha prevail in he nonsocial framing are vulnerable o arificial menalizing agens (whose sophisicaion people canno grasp). This implies ha people should perform beer in he social han in he non-social framing of he ask, because arificial ToM agens would ousmar learners who do no engage in menalizing. Maerials and Mehods Ehics saemen Our analysis involved de-idenified paricipans daa and was approved by he ehics commiee of he Laboraoire d Economie Expérimenale de Paris (LEEP, Paris Experimenal Economics Laboraory). In accordance wih he Helsinki declaraion, all subjecs gave an informed consen. Compuaional modelling 1) k-tom model. In his secion, we expose he key seps in he derivaion of he k-tom model in he conex of repeaed woplayer games (see also [14]). We used his model boh o generae he choices of he paricipans (arificial) opponens during he experimen, and in he analysis of paricipan choices. Firs, recall ha, in is simples form, a game is defined in erms of a uiliy able U a self,a op, which yields he payoff one ges when making decision a self while he oher player chooses a op. Incenives can be arbirarily chosen o capure differen forms of social exchanges or ransacions. In our experimen, we induced social compeiive ineracions by balancing he gain of he winner by he loss of he loser ( hide and seek game, cf. Table 1 below). By convenion, acions a op and a self ake binary values encoding he firs (a~1) and he second (a~0) available opions. According o Bayesian decision heory, agens aim a maximising expeced payoff V~E Ua self,a op, where he expecaion is defined in relaion o he agen s uncerain predicions abou his opponen s Table 1. "Hide and Seek" uiliy able as a funcion of he paricipan s acion a self and his opponen s a op. nex move (see below). Imporanly, his implies ha he form of he decision policy is he same for all agens, irrespecive of heir ToM sophisicaion. In his work, we consider ha choices may exhibi small deviaions from he opimal decision rule, i.e. we assume agens employ he so-called "sofmax" probabilisic policy: Pa self V 1 {V 0 ~1 ~s b ð1þ V i ~p op U a self ~i,a op ~1 z 1{p op U a self ~i,a op ~0 where Pa self ~1 is he probabiliy ha he agen chooses he acion a self ~1, s is he sigmoid funcion and b is he exploraion emperaure ha conrols he magniude of behavioural noise. Equaion 1 simply says ha he probabiliy of choosing he acion a self ~i increases wih is expeced payoff V i. Here, he criical variable is p op : he probabiliy ha he opponen will choose he acion a op ~1. The repeaed observaion of his opponen s behaviour a op gives he agen he opporuniy o learn his predicion. Theory of Mind comes ino play when agens consider ha he opponen s behavioural endency p op is moivaed by his hidden beliefs and desires. More precisely, our "social Bayesian brain" hypohesis implies ha ToM agens consider ha he opponen is himself a Bayesian agen, whose decision policy p op ~Pa ð op ~1Þ is formally similar o Equaion 1. In his siuaion, one has o rack one s opponen s predicion p self abou one s own acions. This makes ToM agens mea-bayesian agens [13], i.e. Bayesian observers of Bayesian agens. In line wih [13], his mea-bayesian inference is recursive ("I hink ha you hink ha I hink "). The recursion deph induces disinc ToM sophisicaion levels, which differ in how hey updae heir subjecive predicionp op. We define k-tom agens in erms of he way hey learn from heir opponen s behaviour, saring wih 0-ToM. By convenion, a 0-ToM agen does no aribue menal saes o his opponen. More precisely, 0-ToM agens simply assume ha heir opponens choose he acion a op ~1 wih probabiliy p op ~sx 0, where he log-odds x 0 varies across rials wih a cerain volailiy s0 (and s is he sigmoid funcion). Observing his opponen s choices gives 0- ToM informaion abou he hidden sae x 0, which can be updaed rial afer rial using he following Bayes-opimal probabilisic scheme: qx 0 z1 a op ~1 a op ~0 a self ~1 1,0 0,1 a self ~0 0,1 1,0 In he able enries, he lef-hand number if he paricipan s payoff (he "seeker") and he lef-hand number is his opponen s (he "hider"). doi: /journal.pcbi !pa 0 z1 Dx 0 z1 ð qx 0 px 0 z1 Dx 0 dx 0 ð2þ where px 0 z1 Dx0 encodes 0-ToM s prior belief on he volailiy of he log-odds, and qx 0 :px 0 Da op 1: is his poserior belief abou he log-odds x 0 a rial, having observed his opponen s behaviour a op up o rial. Under hese premises, one can derive PLOS Compuaional Biology 2 December 2014 Volume 10 Issue 12 e

3 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK Menalizing and Learning 0-ToM s learning rule, in erms of he change in his predicion abou his opponen s nex move (we refer he ineresed reader o Tex S1): qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ^p op z1 &s m0 1z S 0 zs0 3=p 2 m 0 &m0 {1 zs0 a op {s m 0 {1 S 0 & 1 1 S 0 zs m0 {1 zs0 {1 1{s m 0 {1 where m 0 (resp. S0 ) is he approximae mean (resp. variance) of 0- ToM s poserior disribuion qx 0. In oher words, m 0 is 0-ToM s esimae of he log-odds a rial, and S 0 is her subjecive uncerainy abou i. Insering ^p op ino Equaion 1 now yields 0- ToM s decision rule. Noe ha he erm a op {s m 0 {1 can be hough of as a predicion error, whose impac on learning accouns for changes in he subjecive uncerainy S 0. Here, he effecive learning rae is conrolled by he volailiy s 0. A he limi s 0?0, Equaion 3 converges owards he (saionary) opponen s choice frequency and Equaions 1-3 essenially reproduce "ficiious play" sraegies [16,17]. Equaions 1-3 describe how 0-ToM agens learn and decide, rial by rial. This is he saring poin for a 1-ToM agen, who considers ha she is facing a 0-ToM agen. This means ha 1- ToM has o predic 0-ToM s nex move, given his beliefs and he choices payoffs. The issue here is ha 0-ToM s priors (as well as his exploraion emperaure) are unknown o 1-ToM and have o be learned, hrough heir non-rivial effec on 0-ToM s choices. More precisely, 1-ToM agens assume ha 0-ToM chooses he acion a op ~1 wih probabiliy p op ~s0v 1 x 1, where he hidden saes x 1 lumps s 0 and b ogeher and he mapping v 1 is derived from insering Equaion 2 ino Equaion 1: ~1Þ~s0v 1 x 1 v 1 x 1 p self DU 1z 1{pself ~ Pa ð op p self b DU 0 qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ~s m 0 {1 1z S 0 {1 zs0 3=p 2 where DU i~u aop ~1,a self ~i {U a op ~0,a self ~i is he ne inciaion of 1-ToM s opponen o pick he firs opion if 1-ToM chooses opion a self ~i. Here, 1-ToM s esimae of p self is effecively a second-order belief, i.e. 1-ToM s be abou her opponen s predicion abou her own nex move. Similarly o 0- ToM agens, 1-ToM assumes ha he hidden saes x 1 vary across rials wih a cerain volailiy s 1, which yields a mea-bayesian learning rule similar in form o Equaion 3 (see Equaion 5 below). In brief, 1-ToM evenually learns how her (0-ToM) opponen learns abou herself, and acs accordingly. More generally, k-tom agens (k 2) consider ha heir opponen is a k-tom agen wih a lower ToM sophisicaion level (i.e.: kvk). Imporanly, he sophisicaion level k of k-tom s opponen has o be learned, in addiion o he hidden saes x k ha conrol he opponen s learning and decision making. The difficuly for a k-tom agen is ha she needs o consider differen scenarios: each of her opponen s possible sophisicaion level k ð3þ ð4þ yields a specific probabiliy p op,k ~s0v k x k ha she will choose acion a op ~1. The ensuing mea-bayesian learning rule enails updaing k- ToM s uncerain belief abou her opponen s sophisicaion level k and hidden saesx k : p op ~ X l k,k p op,k lvk &s0~v k m k,k p op,k 2 6 {1,Sk,k {1 l k,k & 6 lk,k {1 pop,k 4 P l k,k0 k 0 {1 pop,k0 vk m k,k &m k,k a op l k,k {1 ð1{p op,k Þ P l k,k0 k 0 {1 1{p op,k0 vk {a op {1 zlk S k,k W{1 k aop {s0v k m k,k {1 {1zs {1 & S k,k 0 0v k m k,k l k W{1 k T W{1 k S k,k {1 zsk where l k,k is k-tom s poserior probabiliy ha her opponen is k-tom, and W k is he gradien of v k wih respec o he hidden saes x k. Here, he mapping v k is obained by he recursive inserion of Equaion 5 ino Equaion 1 (as in Equaion 4), and ~v k is defined h implicily in i erms of he expecaion operaor, as follows: E s0v k x k,k {1 ~s0~v k m k,k {1,Sk,k {1. Equaion 5 is bu a compac formulaion of how he summary saisics (m k, S k and l k ) of k-tom s poserior disribuion qx k,k evolve from rial o rial. Boh Equaions 3 and 5 have been derived using a variaional approach o approximae Bayesian inference [18 20]. We refer he ineresed reader o a previous heoreical paper [14]. Alhough Equaion 5 is slighly more complex han Equaion 3, noe ha learning is sill driven by a simple predicion error erm. However, here is an ineracion beween he beliefs on he opponen s sophisicaion level and hidden saes. For example, one can see ha m k,k and S k,k are lef unchanged if he k-tom scenario is unlikely, i.e. if l k,k?0. Also, l k,k increases in proporion o how likely was he opponen s las choice under he k-tom scenario p op,k, which depends upon m k,k and S k,k. Finally, noe ha k-tom models do no differ in erms of he number of heir free parameers. More precisely, k-tom s learning and decision rules are enirely specified by heir prior volailiy s k (cf. Equaions 3 and 5) and behavioural emperaure b (Equaion 1). This concludes he mahemaical exposiion of our meabayesian model of ToM agens. A his poin, one may no have a clear inuiion abou how such k-tom agens reac o heir opponens choices. We hus performed Volerra decomposiions of simulaed choice sequences of arificial k-tom agens playing "hide and seek" agains a random opponen. In our conex, his means regressing k-tom s simulaed choices ono (i) her opponen s pas choices, and (ii) her own pas choices (see Tex S1). In brief, a posiive Volerra weigh capures a endency o reproduce or copy he corresponding acion. Fig. 1 shows he esimaed Volerra kernels of k-tom agens, averaged across a housand Mone-Carlo simulaions. Chance level was derived as he exremum Volerra weighs esimaed for a random choice sequence. We also evaluae Volerra s fi accuracy, in erms of he percenage of correc choice predicions. {1 ð5þ PLOS Compuaional Biology 3 December 2014 Volume 10 Issue 12 e

4 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK Menalizing and Learning Fig. 1. Volerra decomposiion of k-tom s response. Lef: impulse response o k-tom s own acion (x-axis: lag, y-axis: Volerra weigh v ). Righ: impulse response o k-tom s opponen s acion. ToM sophisicaion levels are colour-coded (blue: 0-ToM, green: 1-ToM, red: 2-ToM, magena: 3- ToM). The grey shaded area denoes chance level. doi: /journal.pcbi g001 One can see ha 0-ToM has a srong endency o imiae he behaviour of her opponen (posiive Volerra weighs v op of opponen s acions). In conradisincion, 1-ToM anicipaes his and hus ends o alernae her own choices (negaive Volerra weighs v self of own acions). 2-ToM depics a paern ha mixes he anicipaion of 1-ToM (picking his opponen s unchosen acion) and 0-ToM (alernaing his own choices). Finally, we noe ha Volerra s fi accuracy decreases wih ToM sophisicaion (from 86% o 72%). This is because nonlineariies in he behaviour of k-tom agens (as induced by, e.g., changes in heir belief abou heir opponen s sophisicaion) canno be compleely capured wihou higher-order Volerra kernels. 2) Oher agens models. The above k-tom model was used boh in he experimenal paradigm (arificial players), and in he saisical daa analysis (paricipans behaviour). In order o es our social Bayesian brain hypohesis, we need o compare our k-tom model wih oher non-bayesian and/or non-menalizing models of peoples choice sequences. Table 2 below summarizes he characerisics of he models we included in he comparison se. As can be seen, he comparison se can be pariioned ino eiher Bayesian (B+) versus non-bayesian (B-) model families, or Table 2. Summary of he models included in he comparison se. ToM (T+) versus no-tom (T-) model families. We will use his facorial srucure of he comparison se when performing grouplevel Bayesian model selecion. Le us now briefly describe he raionale behind hese agen s models: For all agen s models (including k-tom), he probabiliy of choosing he acion a self ~1 a rial can be wrien using he sofmax policy of Equaion 1, augmened wih an unknown bias erm. This formulaion is convenien because models only differ in erms of he underlying dynamics of hidden saes ha deermine eiher he agen s predicion abou heir opponen s nex move p op (as in, e.g., Equaion 3) or direcly opions values V (see below): N hbl (hierarchical Bayesian Learner): his model is a hierarchical exension of 0-ToM, which includes a Bayesian updae rule for he volailiy s 0 of he opponen s log-odds. This yields a sophisicaed non-menalizing agen ha can adap is learning rae over he course of he experimen. Augmening 0-ToM wih such a learning rule essenially cos wo addiional parameers ha conrol he coupling beween he volailiy and he log-odds. We refer he ineresed reader o [21]. Model s name Bayesian menalizing number of free parameers k-tom (1ƒkƒ3) yes (B+) yes (T+) 3 0-ToM yes (B+) no (T2) 3 HGF yes (B+) no (T2) 5 n-bsl (1ƒnƒ3) yes (B+) no (T2) 3 k-inf (1ƒkƒ2) no (B2) yes (T+) 3 (1-Inf), 4 (2-Inf) RL no (B2) no (T2) 3 WSLS no (B2) no (T2) 2 Nash no (B2) no (T2) 1 doi: /journal.pcbi PLOS Compuaional Biology 4 December 2014 Volume 10 Issue 12 e

5 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK Menalizing and Learning N n-bsl (Bayesian Sequence Learner): his is anoher exension of 0-ToM, which opimally racks he frequency of he opponen s choice sequences of lengh n. More precisely, n- BSL s predicion abou her opponen s nex move p op depends upon he previous n acions, i.e.: p op :P a op ~1Da op {n:{1. Alhough he number of her belief s sufficien saisics increases exponenially wih n (here are 2 n sequences of lengh n), n-bsl s corresponding updae rules are simple duplicaes of Equaion 2. N k-inf ( Influence model): his is a non-bayesian menalizing agen ha can be regarded as an analogous o k-tom, in ha she accouns for how her own acions influence her opponen s sraegy. For example, 1-Inf uses he following heurisic racking rule of her opponen s acions [22]: p op z1 ~pop zgða op {p op q ~ 1{b s{1 ðp op Þ 2 Þ{2lp op ð1{p op Þ a self {q where g (resp. l) conrols he relaive weigh of 1-Inf s predicion error (resp. he influence correcion erm). Noe ha he influence correcion erm is proporional o 1-Inf s esimae of her opponen s predicion error. Equaion 4 can be augmened wih a second-order correcion erm, which incorporaes he knowledge ha he opponen is iself using an influence model. This yields 2-Inf s updae rule: ð6þ p op z1 ~pop zgða op {p op Þ{2lp op h i ð7þ ð1{p op Þ a self {q z2v q ð1{q Þ where v now conrols he weigh of 2-Inf s opponen s (firsorder) influence correcion erm. Noe ha we did no consider higher order correcion erms. N RL (reinforcemen learning): a each rial, he agens updae he value of he chosen opion in proporion o he reward predicion error [23] ( Vz1 i ~V iza 2u {1{V i if acion a self ~i was chosen Vz1 i ~V i oherwise ð8þ where u ~U a self,a op is he las game oucome and a is he (unknown) learning rae. N WSLS (win-say/lose-swich): a each rial, he agen repeas her las choice if i was successful and alernaes oherwise [24]: ( Vz1 i ~2u -1 V i z1 ~1{2u if acion a self ~i was chosen oherwise N Nash: his is a probabilisic policy ha prevens he oher player from conrolling his expeced earnings. In "hide and seek", he (saic) Nash sraegy consiss in choosing any of he wo opions wih a fixed probabiliy of 1=2. This can be done by fixing he opions values as follows: V 1~V 0 ~0. Noe ha here, he Nash policy is sricly equivalen o a random chance ð9þ model (wih a poenial bias owards one of he alernaive opions). Experimenal mehods 1) Paricipans. In oal, n = 29 subjecs (15 females, mean age = 22.5, SD = 3.8) wihou hisory of neurological or psychiaric disease were recruied via wihin an academic daabase. Three paricipans were excluded from he analysis due o very low performance in a 3-back conrol ask (see below). Paricipans were paid a minimum of 5 J plus an addiional moneary bonus ha depended upon heir performance in he differen asks (see Tex S1). They were insruced abou moneary earnings prior o he experimenal session. 2) Main ask. In our main experimen, paricipans performed four games of hide and seek and four sessions of he Casino gambling ask. Our raionale for choosing he "hide and seek" game is wofold: (i) one can show ha, in his game (as opposed o, e.g., cooperaive games), expeced performance increases wih ToM sophisicaion [14], (ii) i lends iself easily o a non-social framing. Paricipans were divided in wo subgroups, each of which performed he experimen a he same ime in he same room. This was required o make hem believe ha hey were playing agains each oher (social framing). Since LEEP policy prevens deceiving insrucions, paricipans were no explicily old hey were playing agains each oher. Insead, in he social condiion, paricipans were insruced ha hey would play hide and seek agains four differen players and ha each of hem would be aribued one of he wo possible roles: hider or seeker. The respecive role of seekers and hiders were explained before he beginning of he experimen. Evenually, all paricipans were privaely informed (hrough insrucions on heir compuer screen) ha hey were seekers. In he non-social framing, paricipans were insruced hey would "perform four sessions of repeaed choices beween wo slo machines" and ha "only one slo machine would be winning on any given rial". In boh condiions, paricipans were given feedback (correc/incorrec) on heir choice a each rial. Sricly speaking, in boh framings, paricipans were no given any informaion regarding he rue feedback mechanism, apar from he fac ha here was a unique correc opion a each rial (i.e. hey knew he counerfacual oucome: if one opion led o success he oher one necessarily led o fail ). In fac, each game/session was played agains a specific algorihm (264 facorial design, cf. Fig. 2), namely: a random sequence wih a 65% bias for one opion (bias was counerbalanced beween he wo framings wihin paricipans), a 0-ToM agen, a 1-ToM agen and a 2-ToM agen. Criically, 0-ToM, 1- ToM and 2-ToM algorihms are all learning agens (i.e. hey adap o he paricipan s choices), bu only 1-ToM and 2-ToM engaged in (arificial) menalizing. Noe ha he random biased opponen (RB) serves as a conrol condiion for non-specific moivaional or aenional confounds on he performance difference beween he wo framings (e.g., people being more willing o engage in a game wih oher human players). The order of opponens was randomized for each paricipan. Each game/session included sixy rials in which paricipans had o choose beween wo opions (wo hiding places or wo slo machines) in less han 1300 msec. If hey were oo slow, he opponen s choice was no revealed (abou 0.5% of rials) and he poin was aribued o he oher player. Feedback was hen revealed for 1 sec afer which a new rial began and he oal number of correc rials was given a he end of each session. Before obaining heir final earnings, paricipans had o fill in a PLOS Compuaional Biology 5 December 2014 Volume 10 Issue 12 e

6 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK Menalizing and Learning Fig. 2. Main ask s experimenal paradigm. Lef: social framing ("hide and seek" game). Righ: non-social framing (Casino game). A each rial, paricipans have 1300 msec o pick one of he wo opions (social framing: wall or ree, non-social framing: lef or righ slo machine). Feedback is displayed for 1 sec, for boh framings his feedback includes if he subjec won or los and he acual winning opion by showing a characer picure (social framing) or hree idenical coins (non-social framing). doi: /journal.pcbi g002 debriefing form, in which hey could describe verbally heir impressions and sraegies. Noe ha paricipans expressed no suspicion regarding he ask framing. 3) Secondary asks. In addiion o he main ask, paricipans performed hree asks assessing execuive funcions. Firs, behavioural flexibiliy was assessed hrough he number of perseveraive responses in a modified card soring ask [25]. Second, inhibiory conrol was measured as he sensiiviy index d9 in a Go/No Go ask [26]. Finally, working memory capaciy was measured as he sensiiviy index d9 in a 3-back ask [27,28]. In addiion, paricipans compleed he Empahy Quoien es [29]. For compleeness, hey were also asked o perform hree sandard ToM asks. Firs, heir apiude o acknowledge he difference beween heir own and ohers beliefs was measured as he average probabiliy raing aribued o he correc response in Vicky s Violin false belief ask [30,31]. Second, heir abiliy o reason abou embedded narraives was measured by he raio of correc answers (menal saes vs conrol) in he Imposing Memory ask [32]. Finally, heir accuracy in discriminaing beween disinc inenional and emoional saes was scored using he caegorizaion accuracy in he Frih-Happé animaion ask [33]. Paricipans performed all he above asks in he following order: hide and seek, Vicky s Violin ask, he modified card soring ask, he Casino gambling game, he Frih-Happé animaion ask, he Go-No Go ask, he empahy quoien, he 3-back ask, Vicky s Violin ask (2) and finally, he imposing memory ask. In oal, he experimen lased roughly one hour and a half. Saisical daa analyses All saisical daa analyses (including ANOVAs) were performed using he VBA oolbox (hp://code.google.com/p/mbbvb-oolbox/) [34]. Noe: alhough we repor summary saisics ha are no correced for muliple comparisons, we indicae he family-wise error rae hreshold (FWER 5% ) when necessary. 1) Behavioural performance. Firs of all, we performed a design saniy check, i.e. we verified ha here was no difference in opponens biases across framing condiions (cf. Figure 1 in Tex S1). Tesing our main hypohesis hus reduces o asking wheher paricipans perform significanly beer in he social han in he non-social framing. Here, peoples performance or earning is defined as he difference P beween he numbers of correc and incorrec rials, i.e.: ð2u {1Þ, where he game oucome u ~U a self,a op a any given rial is eiher correc (1) or "incorrec" (0). Under he null (H 0 ), one is as likely o be correc as incorrec, i.e.: Pu ð ~1DH 0 Þ~Pðu ~0DH 0 Þ~1=2. I follows ha one can derive he probabiliy disribuion ðu DH 0 Þ of average cumulaive earnings u ~1=4n P n P 4 P i~1 j~1 ~1 2ui,j {1 as a funcion of rial index, where u i,j is he game oucome a rial for paricipan i agains opponen j. We used his o idenify he classical 5% false posiive rae hreshold, i.e. he criical average earning u ha yields Pðu u DH 0 Þv0:05. Classical significance esing of observed performance in he main ask hus reduces o a direc comparison wih u, which we did for earnings in boh framing condiions, as well as for he difference in earnings beween framings. Furher, we assessed he effec of framing, of opponen and heir ineracion using a pooled-variance ANOVA on final earnings. For he sake of simpliciy, we modelled he opponen s facor in erms of he linear effec of sophisicaion ono performance. In addiion, we also performed ess of condiion-specific effecs. Since he laer did no correspond o a priori hypoheses, we indicaed he correced hresholds for compleeness. PLOS Compuaional Biology 6 December 2014 Volume 10 Issue 12 e

7 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK Menalizing and Learning Finally, we analysed he impac of execuive funcions, empahy or (secondary) ToM asks ono peoples performance in each framing condiion of he main ask using a general linear model, which also included paricipans age and gender. More precisely, we used framing-specific omnibus F-ess o es for any effec of performances in he seven secondary asks on peoples final earnings (averaged across opponens). We also performed he same analysis on he difference in performance (beween framings). 2) Volerra decomposiions of choice sequences. Volerra series allow a sysemaic decomposiion of dynamical sysems inpuoupu relaionships, where he oupu is ypically a funcion of he hisory of pas inpus. In our conex, we assume ha each choice resuls from he (logisic) convoluion of boh players pas acions. This means ha Volerra decomposiions reduce o esimaing he impulse response o one s own and opponen s acions, respecively (see Tex S1 for more deails). We performed Volerra decomposiions of each paricipan s choice sequence, in each condiion of he main ask. We hen assessed he effec of framing, of opponen and heir ineracion using a pooledvariance ANOVA on each Volerra weigh separaely. In addiion, Volerra decomposiions of arificial k-tom agens (cf. Fig. 1) serve as a reference poin for inerpreing paricipans Volerra kernels. More precisely, hey define "bes k-tom responses" o each opponen ype (for insance, 1-ToM is a "bes k-tom response" o 0-ToM since she holds a correc model of her opponen), which one can compare each paricipan s response o. In paricular, he similariy o he "bes k- ToM response" is a proxy for he opimaliy of people s learning rule when playing agains ToM agens. 3) Bayesian model comparison. In oal, we included hireen agen models (see Table 2) and he Volerra decomposiion (for reference) in he saisical comparison. All hese models were augmened wih a poenial (session-specific) bias owards any of he wo opions, which was included in he logisic likelihood funcion (cf. Equaion 1). Noe ha hese models differed in he number of unknown parameers, which ranges from 2 parameers for WSLS, o 17 for Volerra. Since hese were allowed o vary beween subjecs (and, wihin subjecs, across condiions), one has o accoun for model complexiy when evaluaing how likely hese models are given he paricipans choice sequences. This was done by evaluaing he marginal likelihood or Bayesian model evidence, under a variaional Laplace approximaion [20]. Evenually, we obained = 2912 model evidences (14 models, 26 paricipans, 2 ask framings, 4 opponens). These were hen insered ino a group-level random-effec Bayesian model comparison (RFX-BMS) [35]. This analysis reas models as random effecs ha could differ beween subjecs, wih an unknown populaion disribuion (described in erms of model frequencies/ proporions). This is paricularly useful in our conex, because we assume ha differen individuals may have disinc ToM sophisicaion levels. In every analysis we repor he exceedance probabiliy (EP) associaed wih models (or family of models), which corresponds o he poserior probabiliy ha a given model is he mos frequen one in he populaion. Relevan mehodological deails of RFX-BMS are summarized in Tex S1. Firs, beween-condiion comparisons allowed us o ask wheher models were he same across ask condiions [36]. I confirmed ha, in conras o he opponen facor, he ask framing is likely o induce differences in model aribuions. We hen summed log-evidences over opponens (fixed effec across opponens), and performed framing-specific RFX-BMS. This allowed us o esimae model frequencies and ToM/no-ToM family exceedance probabiliies for boh ask framings. The proporion of ToM sophisicaion levels was derived by re-performing an RFX-BMS, having resriced he se of models o he winning family. Resuls Behavioural performance resuls Fig. 3 summarizes he group resuls on he behavioural performance in he main ask. Overall, he paern of mean performances follows our predicions. Le us firs consider he op-lef panel of Fig. 3, which depics he dynamics of he group mean cumulaive earnings (averaged across opponens) for boh framing condiions, overlaid on he chance 5% false posiive rae hreshold. One can see how he effec size unfolds over ime. In paricular, i is reassuring o see ha paricipans performance ends o reach saisical significance almos from he sar of he experimen onwards. When summarizing he performance in erms of final earnings: people significanly win in he social framing (ū 60 = 1.79, p = 0.008), whereas hey significanly lose in he non-social framing (ū 60 = 21.28, p = 0.047) despie posiive earnings agains RB in he nonsocial condiion (cf. Fig. 3, boom-lef panel). The framing effec is even clearer on he op-righ panel of Fig. 3, which depics he dynamics of he difference in average cumulaive earnings beween framings. In brief, he framing effec becomes significan a abou rial = 15, and increases in size as ime unfolds (o reach ū 60 = 3.07, p = a he end of he game). We refer he ineresed reader o Figure 2 in Tex S1 for furher informaion regarding he dynamics of condiion-specific earnings. Now, as one can see on he boom-lef panel of Fig. 3, paricipans final earnings seem o depend upon boh he framing and he opponen ype. More precisely, in he social framing, paricipans seem o win agains all arificial agens excep 2-ToM (null earnings). In conrary, in he non-social framing, paricipans seem o lose agains all menalizing opponens, be even wih 0- ToM, and win agains RB. This view is largely consisen wih resuls of he ANOVA on peoples final earnings: In addiion o he main effec of framing (F = 7.49, p = 0.007), paricipans performance significanly decreases wih he sophisicaion of heir opponen (F = 6.96, p = 0.009), bu show no ineracion of framing and opponen (F = 0.89, p = 0.35). Including paricipans performance in he seven secondary asks (as well as heir age and gender) as confounding facors in he ANOVA did no change hese resuls. When looking more closely a condiion-specific effecs (cf. Fig.3 boom-righ panel), we found ha he opponen, agains which paricipans performance showed he sronges framing effec was 1-ToM ( = 2.9, p = 0.003; FWER 5% = ). This makes sense, if we assume ha peoples effecive ToM sophisicaion is higher (resp. lower) han 1-ToM in he social (resp., non-social) framing. Noe ha he mean performance in he conrol condiion (RB) shows no difference beween he social and non-social framings ( = 0.1, p = 0.43). This is imporan, because i implies ha he difference in mean performance agains 1-ToM is unlikely o be due o moivaional or aenional confounds (which would also induce differences agains RB). A his poin, we looked a iner-individual differences o srenghen our resuls inerpreaion. Firs, we asked wheher any iner-individual variabiliy in peoples performance could be explained by iner-individual differences in he seven secondary cogniive asks. Ineresingly, we found no significan effec on peoples performance in he main ask, irrespecive of he ask framing or he opponen s sophisicaion (see Tex S1 for furher deails). This is imporan, because his implies ha peoples capabiliy o ousmar arificial menalizing opponens is no influenced by execuive funcions or empahy. Nex, we asked wheher idiosyncraic differences in moivaional and/or aenional saes could drive he iner-individual variabiliy in our main PLOS Compuaional Biology 7 December 2014 Volume 10 Issue 12 e

8 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK Menalizing and Learning Fig. 3. Group-level performance resuls. Top-lef: average cumulaive earnings ū (y-axis) in he social (blue) and non-social (red) framings, as a funcion of rials in he game (x-axis), overlaid on he chance 5% false posiive rae hreshold (grey shaded area). Top-righ: average difference in cumulaive earnings ū (social minus non-social) as a funcion of rials in he game, overlaid on he chance 5% false posiive rae hreshold. Boomlef: group average cumulaed earnings agains he four differen opponens (red: non-social framing, blue: social framing). Errorbars depic one sandard error. Boom-righ: group average difference (social minus non-social) in cumulaed earnings agains he four differen opponens. doi: /journal.pcbi g003 ask. We reasoned ha if his was indeed he case, people who win more han ohers in he social framing should also win more in he non-social framing. We hus focused on he correlaion beween peoples performance in he social and in he non-social framings. To begin wih, we found no correlaion beween average performances in he social and non-social framings (r = 0.24, p = 0.23). Furhermore, when esing he correlaion for each opponen s sophisicaion separaely, we found ha i was significan only in he conrol condiion (r = 0.48, p = , FWER 5% = ). Recall ha RB is he only opponen, agains which menalizing should yield no advanage. Agains oher opponens, differences in performance induced by individual variabiliy in aenional or moivaional saes are negligible, when compared o, e.g., differences induced by peoples ToM sophisicaion. In brief, he iner-individual variabiliy of peoples performance agains arificial menalizing agens is unlikely o be driven by cogniive requiremens (such as behavioural flexibiliy, working memory, inhibiory conrol, ec ) or aenional/ moivaional confounds. Raher, our analysis of peoples earnings seems o indicae ha peoples abiliy o reliably predic he behaviour of arificial menalizing agens criically depends upon wheher or no hey engage in (poenially auomaic) sophisicaed ToM inferences. Volerra decomposiions Nex, we asked wheher we could find evidence for framingspecific learning rules ha could explain he observed differences in peoples performances across framings. We hus performed Volerra decomposiions of peoples rial-by-rial choice sequences, i.e. we looked a how much rial-by-rial variance in peoples choice sequences can be explained by he hisory of boh players acions. Average Volerra s fi accuracy in each of he 462 condiions is given in Table 3 below. One can see ha Volerra decomposiions of paricipans and arificial ToM agens choices have similar fi accuracies. More precisely, hey yield abou 75% of correc choice predicions, which is significanly above chance level. This is a prerequisie for inerpreing he esimaed Volerra kernels as a Table 3. Average fi accuracy of Volerra decomposiions of paricipans choice sequences agains each opponen (columns) in each framing condiion (rows). RB 0-ToM 1-ToM 2-ToM social framing 74.5% 76.5% 76.3% 76.2% non-social framing 80.0% 82.2% 78.9% 79.6% doi: /journal.pcbi PLOS Compuaional Biology 8 December 2014 Volume 10 Issue 12 e

9 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK Menalizing and Learning summary of paricipans average response o he hisory of players acions. Fig. 4 depics he group mean Volerra kernels agains each opponen, in he social and in he non-social framing condiion. For each opponen, we superimposed he Volerra kernel of he corresponding "bes k-tom response", i.e. one ToM sophisicaion level above paricipans opponens. For compleeness, resuls of a parameric Volerra decomposiion are exposed in Figure 5 of Tex S1. In he non-social framing, i seems ha people have a srong endency o imiae heir opponen s las acion (cf. posiive Volerra weigh v op 1 ). They also end o perseverae, i.e. o reproduce heir las choice (cf. posiive Volerra weigh v self 1 ). In he social condiion, people raher seem o alernae heir own acions (cf. negaive Volerra kernels v self ) and o imiae heir opponen s choices less ofen han in he non-social framing (cf. small Volerra kernels v op ). In addiion, Volerra decomposiions of peoples choice sequences in he social framing seem much closer o he "bes k-tom response" han in he non-social framing (excep maybe in he conrol condiion). Firs, we consider he impac of our experimenal facors ono peoples response o feedback hisory. The ANOVA on peoples Volerra kernels confirms ha boh weighs v op 1 and v self 1 significanly decreased in he social framing, when compared o he non-social framing (v op 1 : F = 6.6, p = 0.01; vself 1 : F = 13.7, p = ). Also, peoples response o heir opponen s pas acions shows a main effec of opponen. More precisely, paricipans endency o replicae heir opponens acions decrease wih he sophisicaion of heir opponen (v op 1 : F = 11.5 p = 0.001, v op 2 : F = 6.8 p = 0.01). Noe ha here was no significan ineracion beween framing and opponen on Volerra weighs (irrespecive of he lag). This is ineresing, because his means ha our experimenal facors have a similar effec on behavioural performance and on peoples response o feedback hisory. Moreover, he observed change in Volerra kernels is consisen wih he idea ha peoples effecive ToM sophisicaion increases in he social framing, when compared o he non-social framing. This is because Volerra weighs of menalizing k-tom agens are sysemaically smaller han hose of 0-ToM (cf. Fig. 1). Nex, we focus on he similariy o he "bes k-tom response", which we ake as a proxy for he opimaliy of peoples response. We measured he correlaion beween each paricipan s Volerra kernel and he appropriae "bes k-tom response" in each of he 462 condiions. This analysis is summarized on Fig. 5. One can see ha he opimaliy score seems o mimic peoples final earnings (cf. Fig. 3, boom panels). In fac, people s opimaliy significanly correlaed wih heir final earnings (r = 0.25, p = ), even afer having removed he effec of he experimenal facors (p = 0.002). We hen performed an ANOVA on he Fisher-ransformed correlaion coefficiens. Resuls showed ha people s opimaliy significanly increased in he social framing, when compared o he non-social framing (F = 5.62, p = 0.02), and significanly decreased wih he opponen s sophisicaion (F = 18.5, p = ). There was no significan ineracion (F = 0.126, p = 0.723). Taken ogeher, hese resuls sugges ha he effec of our experimenal facors ono behavioural performance is mediaed hrough peoples similariy o he "bes k-tom response". A classical Sobel es [37] confirmed his for boh framing (p = 0.010) and opponen (p = 0.013) facors. In summary, our analysis of Volerra kernels demonsraes ha he social framing induces a sysemaic change in peoples behavioural response o feedback hisory. Imporanly, his change is reminiscen of sophisicaed mea-bayesian inference, i.e. peoples similariy o he "bes k-tom response" increases in he social framing, when compared o he non-social framing. This evenually drives peoples behavioural performance agains arificial menalizing agens. Fig. 4. Volerra decomposiion of paricipans responses. Top: impulse response o paricipans own acion (x-axis: lag, y-axis: Volerra weigh v ) agains each opponen (red: non-social framing, blue: social framing). Righ: impulse response o paricipans opponen s acion. Errorbars depic one sandard error on he mean. Black lines depic he "bes k-tom response" o each opponen ype. doi: /journal.pcbi g004 PLOS Compuaional Biology 9 December 2014 Volume 10 Issue 12 e

10 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK Menalizing and Learning Fig. 5. Opimaliy of paricipans response. Lef: group average correlaion beween paricipans Volerra kernels and he "bes k-tom response" o each of he four differen opponens (red: non-social framing, blue: social framing). Errorbars depic one sandard error. Righ: group average difference (social minus non-social) in he correlaion beween paricipans Volerra kernels and he "bes k-tom response" o each of he four differen. doi: /journal.pcbi g005 Model inversions Lasly, we performed a formal model-based analysis of peoples rial-by-rial choice sequences, in he aim of idenifying he mos likely learning scenario in boh social and non-social framings. In brief, we performed a group-level random-effec Bayesian model comparison (RFX-BMS, [36]) of foureen differen models (cf. Table 2). These include mea-bayesian ToM models (1-ToM, 2- ToM and 3-ToM), non-bayesian ToM models (1-Inf and 2-Inf), Bayesian no-tom models (0-ToM, hbl, 1-BSL, 2-BSL and 3- BSL), as well as non-bayesian no-tom models (RL, WSLS, Nash and Volerra decomposiions). In wha follows, we will exploi hese wo orhogonal pariions of our model se, namely: T+/T- (which refers o models ha include menalizing or no) and B+/B- (which refers o models ha rely upon Bayesian belief updaes or no). Noe ha all models include a bias erm ha can capure a sysemaic endency o prefer one alernaive opion over he oher (wihin games/sessions). Firs, we performed Bayesian hypohesis ess o assess he sabiliy of models aribuion across condiions. To begin wih, we esed he hypohesis ha he model family (T+ versus T-) used in he social framing was he same han in he nonsocial framing, for each opponen. Evidence for he null hypohesis was found for he conrol condiion RB (EP = 95%). However, evidence for a difference in model families across framings was found for boh 0-ToM (EP = 23%) and 1-ToM (EP = 0%) opponens. The es was inconclusive for 2-ToM (EP = 53%). Then, we esed wheher he same family of model was used across opponens in a given framing. In his case, we found srong saisical evidence in favour of sabiliy of model aribuions. More precisely, he null hypohesis was srongly suppored for all beween-condiions comparisons (EP.83%), wih he excepion of comparisons beween 2-ToM and RB in he social framing, which yielded weaker evidence (EP = 69%). Overall, his analysis indicaes ha people s learning rule is mosly framing-dependen (bu no opponen-dependen). This moivaes our final analysis, which essenially is a framing-specific RFX-BMS. The resul of his procedure is depiced on Fig. 6, which shows he exceedance probabiliy of model families in boh he social and non-social condiions. We refer he ineresed reader o Tex S1 for quaniaive diagnosics of he RFX-BMS approach (cf. fixedeffec analysis and confusion marices). One can see ha, in he social condiion, peoples rial-by-rial choice sequences are more likely o be explained by T+ models han by T- models (EP = 100%). In conradisincion, peoples behaviour in he non-social condiion is more likely o be explained by models ha do no rely on menalizing (EP = 96%). This is srong saisical evidence ha any realisic mechanisic descripion of peoples policy in he social framing has o rely upon recursive menalizing processes. We hen asked wheher we could find more specific evidence regarding he informaion-heoreic naure of peoples belief updaes. Thus, we furher divided our T+ and T- families ino B+ and B- subfamilies. We hen used RFX- BMS o perform a comparison of he wo corresponding subfamilies (T-B-, T-B+ in he non-social condiion T+B+ and T+B+ in he social condiion. We found ha T+B+ models were he mos likely explanaions o peoples rial-by-rial choices (EP = 98%) in he social condiion, whereas T-B- was he mos likely family in he non-social condiion (EP = 99%). This is imporan, because his means ha menalizing processes are likely o follow mea-bayesian belief updae rules (as opposed o oher non-opimal heurisics). In oher erms, he way we learn abou how ohers learn is near-opimal (from an informaion-heoreic poin of view). Le us now focus on he esimaed models frequency disribuion in he social condiion (cf. upper panel of Fig. 7). Firs, one can see ha 2-ToM is he mos prevalen model (well above reference models such as Nash or RL). Second, we resriced he model comparison o he T+B+ family, in he aim of deriving efficien esimaes of he disribuion of ToM sophisicaion in he human populaion. We found ha 2-ToM agens are abou wo imes more frequen han 1-ToM agens (3-ToM being almos negligible). This suggess ha he naural iner-individual variabiliy of ToM sophisicaion exiss bu is raher narrow. In addiion, i is likely o be upper-bounded. For compleeness, Fig. 7 also shows he equivalen esimaed models frequency disribuion in he non-social condiion (cf. boom panel). One may infer ha WSLS is he mos likely explanaion for peoples behaviour in his condiion. However, i urns ou ha RFX-BMS may confuse Bayesian sequence learning wih WSLS (more precisely: 2-BSL or 3-BSL). Alhough such saisical confusion does no compromise he inerpreaion of oher poenially likely models, i renders he comparison of he families T-B- and T-B+ slighly unreliable. Thus, he esimaion of model frequencies wihin he winning family (T-B-) is provided only as an indicaion (see Tex S1 for furher deails). Discussion Our sudy combined a compuaional modelling approach wih an experimenal invesigaion of Theory of Mind (ToM) in a siuaion of social ineracion. We demonsraed a srong social PLOS Compuaional Biology 10 December 2014 Volume 10 Issue 12 e

11 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK Menalizing and Learning Fig. 6. Bayesian model comparison. Lef: exceedance probabiliies of he no-tom (T-) and ToM (T+) model families (red: non-social framing, blue: social framing). Righ: exceedance probabiliies of he no-tom/non-bayesian (T-B-), no-tom/bayesian (T-B+), ToM/bayesian (T+B+) and Tom/non- Bayesian (T+B-) model families. doi: /journal.pcbi g006 framing effec, whereby he abiliy of paricipans o predic he behaviour of arificial menalizing agens was condiional on wheher or no hey believed hey were playing agains anoher human being. Using daa-driven analyses, we showed ha his social framing effec was due o a difference in peoples rial-byrial response o feedback. In addiion, we found ha our mea- Bayesian model is a more plausible explanaion of people s rialby-rial choice sequences han oher non-bayesian and/or nonsocial (non-menalizing) learning heurisics only in he social condiion. Finally, we found saisical evidence ha ToM sophisicaion is variable across people, and is likely o be upperbounded (2-ToM). Recall ha our experimen aimed a revealing he specificiy of social inference indirecly, by simulaing behavioural daa ha conform o peoples naural predicion of ohers acions, and hen measuring a difference in performance ha originaes from he Fig. 7. Disribuion of ToM sophisicaion. Top: Esimaed model frequencies in he social framing (dark grey: having resriced he models o he winning T+B+ family). Errorbars depic one poserior sandard error. Boom: Esimaed model frequencies in he non-social framing (dark grey: having resriced he models o he winning T-B- family). doi: /journal.pcbi g007 PLOS Compuaional Biology 11 December 2014 Volume 10 Issue 12 e

12 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK Menalizing and Learning ask framing. Here, he framing induces priors ha deermine how people process he feedback informaion, which shapes heir predicions regarding he nex bes move. Criically, such a manipulaion only works if (i) he underlying model realisically simulaes peoples hidden social prior beliefs, and (ii) people are unlikely o appeal o hese priors in he non-social framing. In our case, social priors essenially induce a sophisicaed inerpreaion of he game s oucome, which involves menalizing abou ohers beliefs. In urn, people engage in recursive belief updaes, which we claim is very specific o human social ineracions. To suppor his claim, we have provided wo complemenary pieces of evidence: (i) people could win over sophisicaed (arificial) menalizing agens only in he social framing condiion, and (ii) he mos likely explanaion for people s rial-by-rial choices involves menalizing only in he social condiion. Noe ha he qualiaive change in people s perspecive induced by he framing is confirmed by he shor debriefing we conduced a he end of he main experimen. In brief, mos paricipans repored "having ried o adap heir sraegy o heir opponen s" in he social framing, whereas hey were "looking for feedback emporal paerns" in he non-social framing. Some paricipans even repored ha hey perceived well ha hiders were "responding o heir own choices", whereas slo machines "followed complex, predeermined, sequences". Taken ogeher, hese resuls validae our mea-bayesian model of menalizing in repeaed social ineracions. Perhaps he mos shocking resul of his work is he fac ha people are clearly fooled by menalizing (arificial) agens in he non-social condiion. This happens despie repeaed negaive feedback ha signals persisen predicion error. Noe ha his does no mean ha people disregard his predicion error in he non-social condiion; however, predicion error does no serve o learn he relevan variables. Our analyses sugges ha he nonsocial framing of he ask induces implici priors ha obscure he evidence for inenional behaviour. This is imporan, because his may explain why we engage in menalizing as soon as we inerac wih social agens [1]. Noe ha one could argue ha wih sufficien raining, paricipans would evenually learn he bes response o heir opponen, wihou having o menalize. This is in principle possible, since k-tom agens are reducible (up o 80% accuracy) o a linear convoluion of compeing players acions (cf. Volerra decomposiions in Fig. 1). However, here is hardly any sign of performance improvemen over he enire session duraion (cf. Figure 2 in Tex S1). A slighly more severe criicism of our inerpreaion of he social framing effec appeals o some form of sysemaic order effec beween he social and he non-social condiions (he former was always performed afer he laer). An example of his is [12], which shows ha, e.g., pedagogical learning is faciliaed when people are primarily engaged in eaching ohers. In our conex, such order effec could no be driven by raining or priming, which would raher improve peoples performance in he non-social condiion. In oher words, our curren (imbalanced) design could deec a ne performance decline from he social o he non-social condiion, above and beyond poenial raining and/or priming effecs. Noe ha order effecs could also be due o he impac of cogniive faigue. Under he assumpion ha menalizing is an efforful menal aciviy, one could argue ha people may be less moivaed o engage in sophisicaed menalizing in he second (non-social) condiion, which would lead o performance losses. We will discuss moivaional confounds below. Even more problemaic is he concern ha he social framing effec migh be confounded by some rivial difference in he undersanding of he ask srucure (as induced by, e.g., peoples assumpions regarding he way casino slo machines work). In paricular, his implies ha paricipans migh have performed beer in he non-social condiion, had hey been "warned" abou he exisence of some form of hidden sophisicaed rule. Insead, we chose o favour a balanced design ha relied on raher noninformaive insrucions. Criically however, paricipans answers o our debriefing quesions seem o indicae ha hey were well aware of he exisence of some srucure in he feedbacks sequence (cf. above). Noe ha model comparisons of paricipans rial-byrial choices in he non-social framing yield ambiguous evidence eiher in favour of simple heurisics like "win-say/lose-swich" or in favour of more sophisicaed Bayesian sequence learning schemes (cf. confusion marix in Figure 10 of Tex S1). In addiion, our analyses show ha non-tom sophisicaed learning models do no seem o provide a likely explanaion for peoples rial-by-rial choices in he social condiion. This means ha sophisicaed inferences induced by he social framing were specifically semming from adoping he inenional sance [38], i.e. hey assumed ha he feedback sequence was he (poenially complex) resul of heir opponen s reacion o heir own choices. Alhough his is cerainly reassuring, we canno enirely rule such poenial confound ou. We will address his poenial design imbalance in forhcoming experimens. Le us now briefly discuss poenial aenional and/or moivaional confounds. In brief, one could argue ha he prospec of ousmaring some conspecifics (as opposed o some unineresing machine) incies us o inves he menal effor required for performing sophisicaed inferences (ypically: menalizing). In fac, our resuls raher speak agains such aenional/ moivaional effecs on peoples performance (e.g., no framing effec agains RB, no correlaion beween peoples performance in he social and in he non-social framings ). In addiion, we found no effec of framing on peoples reacion imes (see Tex S1), which is surprising under such moivaional inerpreaion (because one would expec people o respond faser in he social han in he non-social condiion). In any case, such poenial issues do no confound our main resul, namely ha one is unlikely o decipher inenional behaviour wihou a priori adoping he inenional sance [38]. Given he apparen added value of ToM sophisicaion, one migh be surprised by is apparen limiaion. In oher words, one may wonder why evoluion has no made all of us smarer. In fac, one can show ha, in heory, compeiive and cooperaive social ineracions induce boh a lower and an upper bound on ToM sophisicaion [14]. Ineresingly, he empirical esimae of he disribuion of ToM sophisicaion levels (cf. boom panel in Fig. 4) is very similar o he prediced equilibrium we derived from evoluionary game heory. Alhough his is cerainly reassuring, i is ye unclear how such resuls would generalize over conexs ha induce differen incenives for sophisicaed menalizing. For example, he effor cos incurred when menalizing in very complex seings migh overcome he expeced gain in performance. Thus, he cogniive process ha yields he bes complexiy/accuracy rade-off migh no involve ToM a all. This may explain why people end o resor o raher heurisic behavioural policies in some complex social ineracions. One can noe however, ha our upper bound on ToM sophisicaion (2-ToM) is consisen wih resuls from behavioural economics regarding limied deph in sraegic hinking. Experimenal invesigaions of he cogniive hierarchy model, for insance, ypically demonsrae ha only a small proporion of people (around 20%) would exceed 2 seps of recursive hinking in sraegic games (e.g., "beauy cones" games) [9]. Having said his, we would argue such PLOS Compuaional Biology 12 December 2014 Volume 10 Issue 12 e

13 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK Menalizing and Learning sraegic games are essenially differen from our main ask. This is because hey monior some form of explici reasoning abou ohers, whereas he ime limiaion on each rial of our main ask raher reveals paricipans inuiive "firs guess" on heir opponen (as is eviden from peoples shor reacion imes and he lack of effec of, e.g., working memory and inhibiory conrol on heir performance in he main ask). This relaes o he curren debae regarding he implici/explici dichoomy of menalizing processes [39]. Le us now briefly discuss how novel or consisen our resuls are, when compared o o exising sudies in boh experimenal psychology and behavioural economics. Firs, on he heoreical side, we bridged he gap beween he lieraures on sraegic hinking in games [9,10,40,41] and acion undersanding [11,42,43]. More precisely, we exended inverse planning models o siuaions of reciprocal social ineracions, which may induce recursive beliefs. We also exended cogniive hierarchy models o repeaed games, which may involve he (Bayesian) recogniion of ohers inenions and beliefs. The key poin is ha we can now mimic differen sophisicaions of menalizing. Second, on he experimenal side, our resuls are consisen wih he idea ha learning in a social conex relies on very specific cogniive processes, which are engaged for predicing ohers behaviour (see, e.g., [22,44]). In paricular, previous neuroscienific sudies have demonsraed ha specific neural sysems are acivaed when performing classical ToM asks [45,46] and during recursive hinking in games [22,47 50]. In his conex, our criical conribuion was o demonsrae he added-value of (some form of) sophisicaed menalizing, in erms of is abiliy o decipher inenional behaviour. Tha is, we showed ha, peoples abiliy o predic goal-oriened choices criically depends upon wheher hey adop he inenional sance [38] or no. This is no rivial, as one could hink ha domain-general learning heurisics could have performed well agains menalizing agens. Among he exising lieraure, he closes example o our work is [12], which shows ha learners who know hey are being explicily augh (by a eacher) learn more from he daa han when assuming oherwise. Taken ogeher, our work and his recen sudy end o conradic oher exising sudies ha concluded ha social learning (such as advice aking behaviour) was driven by non-specific reinforcemen-like References 1. Frih CD, Frih U (2012) Mechanisms of social cogniion. Annu Rev Psychol 63: doi: /annurev-psych Onishi K, Baillargeon R (2005) Do 15-monh-old infans undersand false beliefs? Science (80-) 308: doi: /science do. 3. Premack D, Woodruff G (1978) Does he chimpanzee have a heory of mind? Behav Brain Sci 1: doi: /s x Baron-Cohen S (1999) The evoluion of a heory of mind. In: Corballis MC, Lea SEG, ediors. The descen of mind Psychological perspecives on hominid evoluion. Oxford Universiy Press. pp Gallagher HL, Frih CD (2003) Funcional imaging of heory of mind. Trends Cogn Sci 7: Van Overwalle F (2011) A dissociaion beween social menalizing and general reasoning. Neuroimage 54: doi: /j.neuroimage Frih U, Happe F, Siddons F (1994) Auism and heory of mind in everyday life. Soc Dev 3: doi: /j b00031.x. 8. Brüne M (2005) Theory of mind in schizophrenia: a review of he lieraure. Schizophr Bull 31: doi: /schbul/sbi Camerer CF, Ho T, Chong J-K (2004) A cogniive hierarchy model of games. Q J Econ 119: Nagel R (1995) Unraveling in guessing games: An experimenal sudy. Am Econ Rev 85: Baker CL, Saxe R, Tenenbaum JB (2009) Acion undersanding as inverse planning. Cogniion 113: doi: /j.cogniion Shafo P, Goodman ND, Griffihs TL (2014) A raional accoun of pedagogical reasoning: Teaching by, and learning from, examples. Cogn Psychol 71: doi: /j.cogpsych processes [44,51]. Noe however ha no recursive learning models was considered for comparison purposes in hese works. Of course, our k-tom model does no embrace all menalizing processes. For example, i canno be used o model how people "read ohers mind" from low-level social signals such as eye gaze, bodily posure or facial expression [52]. Alhough i comprises he basic building blocks for modelling false beliefs (cf. beliefs abou beliefs), i would sill require some modificaion o capure he difference beween people who pass and people who fail he false belief es [53] (bu see [54]). We noe ha exending k-tom in order o explain he various phenomena observed across he lieraure is well beyond he scope of he presen sudy. We will pursue his in subsequen publicaions. Finally, we would like o highligh a few promising applicaions of his work. Given he simpliciy of he ask ha paricipans have o perform (namely: choosing beween wo alernaive opions, one of which is leading o a reward), one could argue ha i could be used o address hree aspecs of menalizing. Firs, one could assess is developmenal aspec by quanifying he drif in ToM sophisicaion ha occurs when we age. Second, our approach could be adaped o perform ehological iner-species comparisons of ToM sophisicaion (e.g. monkeys, grea apes and humans). Third, in line wih ideas from he emerging field of compuaional psychiary [55,56], one may wish o quanify pahological impairmens of menalizing in neuropsychiaric disorders such as auism or schizophrenia. We are currenly pursuing hese ideas. In hese conexs, he main addedvalue of our approach lies in is abiliy o capure quaniaive differences in ToM sophisicaion hrough is impac on behaviour, wihou being confounded by linguisic skills. Supporing Informaion Tex S1 This is a documen conaining supporing informaion regarding models, saisical mehods, experimenal deails, addiional daa analyses and model inversion diagnosics. (PDF) Auhor Conribuions Conceived and designed he experimens: MD JD GH. Performed he experimens: MD. Analyzed he daa: MD JD. Conribued reagens/ maerials/analysis ools: MD JD. Wroe he paper: MD JD GH. 13. Daunizeau J, Den Ouden HEM, Pessiglione M, Kiebel SJ, Sephan KE, e al. (2010) Observing he Observer (I): Mea-Bayesian Models of Learning and Decision-Making. PLoS One 5: Devaine M, Hollard G, Daunizeau J (2014) Theory of Mind: did evoluion fool us? PLoS One 9: e Yoshida W, Dolan RJ, Frison KJ (2008) Game heory of mind. PLoS Compu Biol 4: Fudenberg D, Levine DK (2009) Learning and Equilibrium. Annu Rev Econom 1: Brown G (1951) Ieraive soluion of games by ficiious play. In: T. C Koopmans, edior. Aciviy analysis of producion and allocaion. New York: Wiley. 18. Beal MJ (2003) Variaional Algorihms for Approximae Bayesian Inference. Universiy College London. 19. Daunizeau J, Frison K, Kiebel SJ (2009) Variaional Bayesian idenificaion and predicion of sochasic nonlinear dynamic causal models. Physica D 238: doi: /j.physd Frison K, Maou J, Trujillo-Barreo N, Ashburner J, Penny W (2007) Variaional free energy and he Laplace approximaion. Neuroimage 34: Mahys C, Daunizeau J, Frison KJ, Sephan KE (2011) A bayesian foundaion for individual learning under uncerainy. Fron Hum Neurosci 5: 39. doi: /fnhum Hampon AN, Bossaers P, O Dohery JP (2008) Neural correlaes of menalizing-relaed compuaions during sraegic ineracions in humans. PNAS 105: Rescorla RA, Wagner AR (1972) A heory of Pavlovian condiioning: Variaions in he effeciveness of reinforcemen and nonreinforcemen. In: Black AH, Prokasy WF, ediors. Classical Condiioning II Curren Research and Theory. PLOS Compuaional Biology 13 December 2014 Volume 10 Issue 12 e

14 EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK Menalizing and Learning Classical Condiioning II: Theory and Research. Appleon-Cenury-Crofs, Vol. 20.pp Nowak M, Sigmund K (1993) A sraegy of win-say, lose-shif ha ouperforms i-for-a in he Prisoner s Dilemma game. Naure 364: doi: / a Nelson HE (1976) A modified card soring es sensiive o fronal lobe defecs. Corex 12: Aron AR (2007) The neural basis of inhibiion in cogniive conrol. Neuroscienis 13: Pochon JB, Levy R, Fossai P, Lehericy S, Poline JB, e al. (2002) The neural sysem ha bridges reward and cogniion in humans: an fmri sudy. PNAS 99: Braver TS, Cohen JD, Nysrom LE, Jonides J, Smih EE, e al. (1997) A parameric sudy of prefronal corex involvemen in human working memory. Neuroimage 5: Baron-Cohen S, Wheelwrigh S (2004) The empahy quoien: an invesigaion of aduls wih Asperger syndrome or high funcioning auism, and normal sex differences. J Auism Dev Disord 34: Birch SAJ, Bloom P (2007) The curse of knowledge in reasoning abou false beliefs. Psychol Sci 18: Converse B a, Lin S, Keysar B, Epley N (2008) In he mood o ge over yourself: mood affecs heory-of-mind use. Emoion 8: Kinderman P, Dunbar R, Benall RP (1998) Theory-of-mind deficis and causal aribuions. Br J Psychol 89: Whie SJ, Conison D, Rogers R, Frih U (2011) Developing he Frih-Happé animaions: a quick and objecive es of Theory of Mind for aduls wih auism. Auism Res 4: Daunizeau J, Adam V, Rigoux L (2014) VBA: a probabilisic reamen of nonlinear models for neurobiological and behavioural daa. PLoS Compu Biol 10: e Sephan KE, Penny WD, Daunizeau J, Moran RJ, Frison KJ (2009) Bayesian model selecion for group sudies. Neuroimage 46: Rigoux L, Sephan KE, Frison KJ, Daunizeau J (2013) Bayesian model selecion for group sudies - Revisied. Neuroimage 84C: doi: / j.neuroimage Baron R, Kenny D (1986) The moderaor mediaor variable disincion in social psychological research: Concepual, sraegic, and saisical consideraions. J Pers Soc Psychol 51: Denne DC (1987) The Inenional Sance. U B, edior MIT Press. 39. Heyes CM, Frih CD (2014) The culural evoluion of mind reading. Science 344: doi: /science Hedden T, Zhang J (2002) Wha do you hink I hink you hink?: Sraegic reasoning in marix games. Cogniion 85: Sahl D, Wilson P (1995) On Players Models of Oher Players: Theory and Experimenal Evidence. Games Econ Behav 10: Shafo P, Kemp C, Bonawiz EB, Coley JD, Tenenbaum JB (2008) Inducive reasoning abou causally ransmied properies. Cogniion 109: doi: /j.cogniion Shafo P, Goodman N D, Frank M C (n.d.) Learning from ohers: The consequences of social conex for human learning: Suzuki S, Harasawa N, Ueno K, Gardner JL, Ichinohe N, e al. (2012) Learning o simulae ohers decisions. Neuron 74: doi: /j.neuron Saxe R (2006) Uniquely human social cogniion. Curr Opin Neurobiol 16: doi: /j.conb Amodio DM, Frih CD (2006) Meeing of minds: he medial fronal corex and social cogniion. Na Rev Neurosci 7: doi: /nrn Coricelli G, Nagel R (2009) Neural correlaes of deph of sraegic reasoning in medial prefronal corex. PNAS 106: Bha M, Camerer CF (2005) Self-referenial hinking and equilibrium as saes of mind in games: fmri evidence. Games Econ Behav 52: doi: /j.geb Yoshida W, Seymour B, Frison KJ, Dolan RJ (2010) Neural mechanisms of belief inference during cooperaive games. J Neurosci 30: Gallagher HL, Jack AI, Roepsorff A, Frih CD (2002) Imaging he Inenional Sance in a Compeiive Game. Neuroimage 16: Behrens TEJ, Hun LT, Woolrich MW, Rushworh MFS (2008) Associaive learning of social value. Naure 456: doi: /naure Baron-Cohen S, Wheelwrigh S, Hill J, Rase Y, Plumb I (2001) The Reading he Mind in he Eyes Tes revised version: a sudy wih normal aduls, and aduls wih Asperger syndrome or high-funcioning auism. J Child Psychol Psychiary 42: Wimmer H, Perner J (1983) Beliefs abou beliefs: Represenaion and consraining funcion of wrong beliefs in young children s undersanding of decepion. Cogniion 13: Goodman N, Baker CL, Bonawiz EB, Mansinghka VK, Alison G, e al. (2006) Inuiive heories of mind: A raional approach o false belief. Proceedings of he cogniive science sociey. 55. Xiang T, Ray D, Lohrenz T, Dayan P, Monague PR (2012) Compuaional Phenoyping of Two-Person Ineracions Reveals Differenial Neural Response o Deph-of- Though. PLoS Compu Biol. 8doi: /journal.pcbi Yoshida W, Dziobek I, Kliemann D, Heekeren HR, Frison KJ, e al. (2010) Cooperaion and heerogeneiy of he auisic mind. J Neurosci 30: PLOS Compuaional Biology 14 December 2014 Volume 10 Issue 12 e