Learning Paerns in Noise: Environmenal Saisics Explain he Sequenial Effec Friederike Schüür (fs62@nyu.edu), Brian Tam (bp218@nyu.edu), Laurence T. Maloney (lm1@nyu.edu) Deparmen of Psychology, 6 Washingon Place New York, NY 10003 USA Absrac Subjecs display sensiiviy o local paerns in simulus hisory (sequenial effecs) in a variey of decision-making, percepual, and moor asks. Sequenial effecs are ypically cas as a prime example of human irraionaliy. We propose a Bayesian model ha explains sequenial effecs as he naural consequence of one incorrec assumpion: insead of assuming a sable world, subjecs assume change. We es and confirm one specific predicion of our model in a 2- alernaive forced-choice reacion ime ask. We manipulaed paricipans beliefs abou he comparaive likelihoods of binary evens and hen showed ha hese beliefs deermine biases in sequenial effecs. We conclude ha he origin of sequenial effecs is belief in a changing world. Keywords: sequenial effecs, forced choice reacion ime ask, generaive model, decision-making, Bayes Sequenial Effecs in Decision Making, Percepion, and Moor Conrol In Mone Carlo in he summer of 1913, many los a forune in a game of roulee. Gamblers be millions of francs agains black when i came up 26 imes in a row. They commied wha is known as he Gambler s fallacy in believing ha he probabiliy of red increased afer many occurrences of black. The Gambler s fallacy is jus one example of how subjecs use local paerns in series of sochasic evens o ry o predic wha is going o happen nex. Sensiiviy o local paerns or sequenial effecs are a pervasive phenomenon in decision-making (Ayon & Fischer, 2004; Barron & Leider, 2010; Gilovich, Vallone, & Tversky, 1985; Kahneman & Tversky, 1972; Roney & Trick, 2009), percepion (Howarh & Bulmer, 1956; Maloney, Marello, Sahm, & Spillmann, 2005; Speeh & Mahews, 1961), and moor behavior (Cho e al., 2002; Soeens, Boer, & Hueing, 1985). Previous sudies have found sequenial effecs in, for example, 2-alernaive forced-choice reacion imes asks (Remingon, 1969) even afer more han 4000 rials (Soeens e al., 1985). If inervals beween rials exceed 500ms, hen sequenial effecs are no due o auomaic faciliaion bu reflec subjecive expecancy (Soeens e al., 1985; Soeens, 1998). Given saisical independence of sochasic evens, however, predicions based on local paerns are of no predicive value and ofen cosly. Why, hen, do we find sequenial effecs in a wide variey of asks and in human and non-human subjecs? Do we fall prey o supersiions (Skinner, 1948) and end up having o pay he price? Insead of casing sequenial effecs as a prime example of human failure and irraionaliy, we develop a raional, Bayesian accoun of sequenial effecs. We propose ha sequenial effecs are driven by mechanisms criical for adaping o a changing (i.e. dynamic) world (cf. Wilder, Jones, & Mozer, 2009; Yu & Cohen, 2008). We es one specific predicion of our model wih a novel 2-alernaive forced-choice reacion ime ask, aimed a firs manipulaing paricipans beliefs abou he comparaive likelihoods of binary evens. Predicing he Fuure by Using a Generaive Model If we could predic he fuure, hen we would no have o agonize abou which job offer o accep or where o go o college. We would make beer decisions because we would know which alernaive led o he preferred oucome. Similarly, foreknowledge abou upcoming sensory simuli or imperaive evens improves sensory percepion and moor behavior (Hyman, 1953; Klemmer, 1957), respecively. If we undersood he processes ha generaed pas, presen, and fuure evens (i.e. heir generaive model), hen we could predic he fuure. For deerminisic processes, we would know which course of acion we would have o ake o obain he desired oucome wih cerainy. For probabilisic processes, we could assign a probabiliy o each acion ha i leads o he desired oucome and hen pick he one acion wih he highes probabiliy. Take he oucome of a coin oss, for example. The coin migh come up heads wih probabiliy p and ails wih probabiliy 1 p. In fac, p fully specifies he generaive model of a coin oss where binary oucomes are drawn from a Bernoulli disribuion wih rae parameer p. If one knows p and p 0.5 hen one should be on he oucome wih p > 0.5. Furher, if evens drawn from a Bernoulli disribuion are sochasically independen, hen once p is known, he oucome of a previous coin oss should no affec he subjec s subsequen being behavior. In oher words, here should be no sequenial effecs. Similarly, he oucome of 2-forced choice reacion ime asks, for example, wheher a simulus is going o show up o he lef or righ of fixaion, is a Bernoulli process wih probabiliy p for lef and 1 p for righ. Insead of locaion, we may also rack wheher a simulus is likely o show up a he same (lef-lef / righ-righ), or alernae locaions (lef-righ / righ-lef) (cf. Cho e al., 2002; Soeens e al., 1985; Wilder e al., 2009; Yu & Cohen, 2008). If he oucomes lef / righ are sampled from a Bernoulli disribuion, hen wheher we observe a repeiion or alernaion is again a sochasic process wih a given probabiliy of repeiion γ (and alernaion 1 γ ). Therefore, once γ is known, here should be no sequenial effecs. The presence of sequenial effecs in reacion imes,
however, shows ha paricipans do no follow he opimal policy. Bu he sysemaic dependency on local simulus hisory also suggess ha do no simply fail o learn γ. Insead, hey mus follow some (incorrec) sraegy in esimaing γ ha underlies sequenial effecs. We propose ha paricipans make jus one misake: insead of sable worlds wih fixed γ hey assume dynamic worlds wih changing γ. Sable versus Dynamic Worlds In sable worlds, γ does no change over ime bu in dynamic worlds, γ may have differen values a differen imes γ. In Figure 1, we illusrae hree worlds: one sable world (black line) and wo dynamic worlds in which γ jumps from one value o he nex a random, Poisson-disribued ime-poins wih consan rae δ. Each new value of γ is drawn from a disribuion ha can be biased owards higher probabiliies (Figure 1b) or lower probabiliies (orange; Figure 1c). We propose ha sequenial effecs reflec paricipans effors o esimae γ over ime. By manipulaing he repeiion probabiliy γ, we ensured ha across rials, Elmo was equally likely o appear o he lef or righ of fixaion. As such, observed effecs canno be explained by sronger preparaion or an advanage (for example, due o handedness) of eiher of he lef or righ hand across sessions. Observed effecs of changing γ can be explained only by subjecs racking repeiions and alernaions (i.e. 2 nd order characerisics of he sequence) whils responding o simuli ha appear on he lef or righ o fixaion (1 s order characerisics). Figure 2: Task Figure 1: a Sable versus b dynamic environmens. Whack-a-Mole (elmo) Task Subjecs compleed a 2-forced choice reacion ime ask based on he arcade game Whack-A-Mole. Sesame Sree s Elmo appeared eiher o he lef or righ of fixaion. Subjecs were insruced o press a buon wih heir righ or lef index finger depending on Elmo s locaion as soon as i appeared (Figure 2). Subjecs compleed hree sessions. In Session 1 and 3, Elmo was equally likely o pop up a he same or alernae locaions relaive o he immediaely preceding rial. In oher words, he repeiion probabiliy was γ = 0.5 and did no change over ime. In Session 2, however, γ was resampled from a Bea-disribuion wih a sligh bias owards repeiions Bea(12,6) or alernaions Bea(6,12). Subjecs were randomly assigned o he repeiion-bias (N=12) or alernaion-bias group (N = 13). Resampling occurred randomly a consan rae δ = 0.18. Each change in γ was signaled explicily o he paricipans. To minimize effecs of Elmo s iming on RTs, fixaion changed color from whie (250ms), o red (250ms), o blue prior o Elmo s appearance. Afer he final color change, Elmo popped up afer a variable inerval sampled from a runcaed exponenial disribuion (mean = 500ms, max. 2s) o ensure consan probabiliy of appearance over ime. Also, inervals beween rials exceeded 500ms, which precludes any explanaion of sequenial effecs as auomaic faciliaion (Soeens e al., 1985). Reacion Times as a Measure of Implici Expecaion Reacion imes (RTs) increase monoonically wih he amoun of informaion conveyed by an even ha requires subjecs o respond (Hyman, 1953). Conversely, subjecs respond quickly o evens hey expec o occur, because predicable evens convey lile informaion. For example, subjecs respond quickly o frequen compared o infrequen simuli (Hyman, 1953) and if simuli occur a expeced compared o unexpeced imes (Klemmer, 1957). RTs are hus informaive abou paricipans esimaed probabiliies of simulus-occurrence. In he curren experimen, we measured and analyzed RTs as a funcion of simulus hisory o explore how local paerns in simulus hisory affeced paricipans esimaes of simulus probabiliies. Learning he Generaive Model Observed evens are informaive abou he probabiliy of repeiion γ. In Figure 3a, we show simulaion resuls, assuming a sable world, of rial-by-rial, ieraive updaing of γ as a funcion of observed evens. The blue line shows
esimaes of γ when subjecs had no esimae of γ prior o he experimen (blue). In oher words, hey considered each possible value of γ o be equally likely (i.e. uniform prior). The green line shows esimaes of γ for a non-uniform prior wih γ > 0.5 (i.e. repeiion bias). Wih or wihou bias, γˆ quickly converges o he rue value of 0.5. In Figure 3b, we show simulaion resuls of rial-by-rial, ieraive updaing of γ if paricipans incorrecly assumed a dynamic environmen eiher wih no bias (blue) or wih a repeiion bias (green). For a uniform prior, γˆ flucuae over ime around he rue value of 0.5. For repeiion-bias, γˆ are relaively less variable bu biased owards higher values han 0.5. owards repeiion, hen sequenial effecs reflec his bias (Figure 4bd, green line), similarly for alernaions (orange line). If paricipans have a uniform prior, hen sequenial effecs are symmeric (blue line). If paricipans assume a sable world, hen any belief in bias has lile effec on sequenial effecs (Figure 4ac). We were paricularly ineresed in he effecs of repeiion versus alernaion raining during Session 2 on sequenial effecs in Session 3. During Session 2, we resampled he probabiliy of repeiion from a biased Bea-disribuion and explicily signaled each change in probabiliy o he paricipan o induce he belief in a dynamic world wih a bias eiher owards repeiion or alernaion. If sequenial effecs indeed originae in he (incorrec) belief in a dynamic environmen, hen we would expec o find sequenial effecs in Session 3. The bias in sequenial effecs owards eiher repeiion or alernaion should correspond o he repeiion or alernaion raining ha paricipans received during Session 2. Firs, such bias would demonsrae ha paricipans can heir environmenal saisics. Second and more imporanly, such bias would reveal he rue naure of sequenial effecs as phenomenon ha originaes in he belief ha environmens are dynamic and change over ime. Sequenial effecs in Session 1, eiher wih or wihou bias, would hen inform us abou he beliefs paricipans held when hey walked ino our laboraory. I is imporan o ake noe ha here is no naïve observer (Henrich, Heine, & Norenzayan, 2010). Every subjec eners an experimen wih his or her own ideas and biases abou wha is going on based on personal hisory. Figure 3: Ieraive updaing assuming a a sable b dynamic environmen. This variabiliy is caused by discouning he evidence of informaion conveyed by pas evens. If subjecs assume change, hen evens in he disan pas are less likely o be informaive abou he curren probabiliy γ han evens in he recen pas. Consequenly, paricipans discoun pas evens. This resuls in higher variabiliy in esimaes, because discouning effecively reduces he number of evens paricipans use o derive esimaes (for deails of he model, see Model). Hypoheses In Figure 4 we show simulaion resuls of ˆ γ as a funcion of local simulus hisory assuming a sable world (Figure 4a) and assuming a dynamic world (Figure 4b). If subjecs learned ha γ was sable (and γ = 0.5 ), hen here should be no sequenial effecs (Figure 4a). By conras, if paricipans assumed a dynamic world, hen probabiliy esimaes reflec sensiiviy o local paerns in simulus hisory (Figure 4b). Simulaions resuls also reveal ha sequenial effecs should reflec subjecs repeiion or alernaion biases. If subjecs believe ha in a changing world and in γ biased Figure 4: Hypoheses if subjec assume a a sable b dynamic environmen. Resuls Figure 5 shows paricipans normalized RTs as a funcion of local simulus-hisory for Session 1 (Figure 5a) and Session 3 (Figure 5b). How quickly paricipans responded o a repeiion or alernaion (final even) was deermined by
local simulus-hisory (final even * simulus-hisory: F(1,24) = 92.40, p <0.001). If subjecs experienced a repeiion, we found ha RTs increased wih fewer occurrences of repeiions in recen hisory (posiive slope of linear fi: mean = 0.088, SE = 0.013) while we found a decrease in RTs wih more alernaions in recen hisory when he final even was an alernaion (mean = -0.084, SE = 0.011; (24) = 8.52, p < 0.001). In essence, occurrence of a repeiion or alernaion in local simulus-hisory leads o a decrease in RTs for repeiion and alernaion, respecively. RTs show clear sequenial effecs. Figure 5: Resuls for a Session 1 and b Session 3. The bias in sequenial effecs owards repeiion or alernaion changed from Session 1 o Session 3. Imporanly, his change depends on he raining paricipans received during Session 2 (session * final even * raining: F(1,24) = 6.39, p = 0.012). For boh groups, we found a bias owards alernaions prior o raining (mean = -0.132; SE = 0.022) compared o repeiions (mean = 0.124, SE = 0.024; final even: F(1,24) = 24.26, p < 0.001). Afer raining, bias was differen for each group (final even * raining group: F(1,24) = 8.89, p = 0.005). If repeiion rained paricipans experienced an alernaion, hen i ook hem longer o respond (mean = 0.076, SE = 0.063), compared o alernaion rained paricipans (mean = -0.092, SE = 0.045; (24) = -2.23, p = 0.036; Figure 5d). And conversely, repeiion rained paricipans responded faser when hey experienced a repeiion (mean = -0.011, SE = 0.057) compared o alernaion rained paricipans (mean = 0.118, SE = 0.043; (24) = 1.86, p = 0.076; Figure 5d). Experiencing a dynamic environmen wih a bias eiher owards repeiions or alernaions deermined he bias in sequenial effecs in a subsequen sable environmen. Discussion Resuls show ha paricipans can learn he saisics of heir environmen. Training paricipans in a dynamic environmen, in which hey experienced changing wih γ a bias oward eiher alernaion or repeiion, deermines he bias in sequenial effecs in a subsequen session wih consan γ and no bias. These specific effecs of raining sugges ha sequenial effecs are no due o human failure or irraionaliy. Insead, sequenial effecs are driven by one misake: insead of assuming a sable environmen, paricipans assume a dynamic environmen. Upon his assumpion, rial-by-rial ieraive updaing of he rae parameer γ of he generaive model leads o sequenial effecs. Any biases in sequenial effecs reflec paricipans assumpions abou he likely values of he rae parameer (raher han auomaic faciliaion) (Wilder e al., 2009). The abula rasa hypohesis in psychology The (incorrec) belief in a dynamic raher han sable environmen may be cas as human failure and irraionaliy. We argue, however, ha we canno and should no judge he beliefs ha paricipans hold when hey ener he laboraory. Paricipans may live in a world in which hey experience change. For example, hey may perform much beer on heir exams in he morning compared o lae afernoon. In fac, if changing environmens are more common han sable environmens, hen paricipans are more jusified o believe in change han sabiliy. Sequenial effecs may reflec ha paricipans are finely uned o he saisic of heir environmen, raher han supersiious beliefs. Indeed, our resuls show ha paricipans can quickly learn he saisics of heir environmen. We call he experimener s belief ha paricipans ener a laboraory wihou prior knowledge he abula rasa hypohesis. The abula rasa hypohesis is mos likely incorrec (see Sun & Perona, 1998 for an example from vision research). To deermine raional priors, one would have o sudy he saisics of naural environmens (cf. Simoncelli & Olshausen, 2001). To deermine he effecs of priors on behavior, one can manipulae hem during raining and hen measure effecs on subsequen behavior (Adams, Graf, & Erns, 2004). Training effecs hen reveal which aspecs of behavior are deermined by prior beliefs. Given he raining effec in Session 3, we can hus conclude ha he alernaion bias in Session 1 is also driven by prior beliefs: when paricipans enered our laboraory, hey hough alernaions were more likely compared o repeiions. We can only speculae why i is ha paricipans believed in alernaion. Perhaps hey regarded Elmo as an opponen rying o rick hem by varying raher han repeaing is locaion? We would have o sudy behavior in compeiive games in paricipans naural environmens o deermine wheher such bias is raional (cf. Tversky & Gilovich, 1989). Uniform versus non-uniform priors I may seem beer o ener a novel siuaion (e.g. an experimen) wih noninformaive, uniform priors raher han beliefs in repeiions or alernaions. In Figure 6a, we show ha in a sable environmen, assuming a non-uniform prior leads o a
reducion in summed squared error when paricipans assumed a dynamic environmen wih he same, consan rae of change δ. Figure 6b o 6d show he priors (i.e. all combinaions of he wo parameers a and b as inpus o he Bea-disribuion ha formalizes he prior) ha led o reduced error for hree differen change raes (area inside he black lines). Of course, assuming a prior wih a mean of 0.5 improves performance. However, priors ha differ subsanially from he rue 0.5 sill lead o an improvemen in paricipans esimaes. In essence, priors reduce he variabiliy in paricipans esimaes of γˆ bu may induce (consan) bias (if 0.5 ). There is a rade-off beween bias and variabiliy (cf. Jazayeri & Shadlen, 2010). A consan bias is someimes beer han he variabiliy caused by a uniform prior. disribuion p(γ x ), a Bea disribuion wih Β(a + r +1,b +1 r +1) (see Figure 3a). Ieraive Updaing assuming a Dynamic World When paricipans assume a dynamic world, he oucome of an even is weighed by how long ago i occurred. The weigh is deermined by w u = exp λu. u sands for he number of rials in he pas relaive o he curren rial. λ is a consan ha deermines how quickly informaion conveyed by pas evens is discouned. r = w u R u wih R for repeiion (and he equivalen for alernaions a ). Paricipans hus esimae γ based on curren evidence alone, which corresponds o he mean of he Beadisribuion Bea( r, a ). In addiion, paricipans have prior expecaion abou γ. The prior p(γ ) is a Bea disribuion wih p(γ ) = Β(a +1,b +1) and paricipans prior esimae is is mean. Paricipans combine hese wo esimae of γ in a weighed sum: w c p c (γ x ) + (1 w c ) p(γ ) (2) w c is deermined by paricipans confidence in heir esimae based on curren evidence and heir prior, which is inversely proporional o he variance of hese wo Beadisribuions. This resuls in a single unique esimae γˆ on rial (see Figure 3b). Figure 6: Effecs of non-uniform prior for ab c and d. Model Ieraive Updaing assuming a Sable World and Updaed esimae of γ are based on he se of binary { }, he number of observaions observed so far x 1,...,x observed repeiions r (and alernaions r ). In addiion, esimaes may be influenced by paricipans beliefs abou γ prior o he experimen. Bayes Rule ells us how o compue his esimae: p(γ x ) p(x γ ) p(γ ) = γ r +a (1 γ ) r +b (1) The prior p(γ ) is a Bea disribuion wih p(γ ) = Β(a +1,b +1) and he prediced probabiliy of seeing a repeiion on he nex rial is he mean of he poserior Acknowledgmens FS and LTM were funded by NIH #. References Adams, W. J., Graf, E. W., & Erns, M. O. (2004). Experience can change he ligh-from-above prior. Naure Neuroscience, 7(10), 1057 1058. Ayon, P., & Fischer, I. (2004). The ho hand fallacy and he gambler s fallacy: wo faces of subjecive randomness? Memory & cogniion, 32(8), 1369 1378. Barron, G., & Leider, S. (2010). The role of experience in he Gambler s Fallacy. Journal of Behavioral Decision Making, 23(1), 117 129. Cho, R. Y., Nysrom, L. E., Brown, E. T., Jones, A. D., Braver, T. S., Holmes, P. J., & Cohen, J. D. (2002). Mechanisms underlying dependencies of performance on simulus hisory in a wo-alernaive forced-choice ask. Cogniive, Affecive, & Behavioral Neuroscience, 2(4), 283 299. Gilovich, T., Vallone, R., & Tversky, A. (1985). The ho hand in baskeball: On he mispercepion of random sequences. Cogniive Psychology, 17(3), 295 314.
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdes people in he world? Behavioral and Brain Sciences, 33(2-3), 61 83. Howarh, C. I., & Bulmer, M. G. (1956). Non-random sequences in visual hreshold experimens. Quarerly Journal of Experimenal Psychology, 8(4), 163 171. Hyman, R. (1953). Simulus informaion as a deerminan of reacion ime. Journal of Experimenal Psychology, 45(3), 188 196. Jazayeri, M., & Shadlen, M. N. (2010). Temporal conex calibraes inerval iming. Naure neuroscience, 13(8), 1020 1026. Kahneman, D., & Tversky, A. (1972). Subjecive probabiliy: A judgmen of represenaiveness. Cogniive Psychology, 3(3), 430 454. Klemmer, E. T. (1957). Simple reacion ime as a funcion of ime uncerainy. Journal of Experimenal Psychology, 54(3), 195 200. Maloney, L. T., Marello, M. F. D., Sahm, C., & Spillmann, L. (2005). Pas rials influence percepion of ambiguous moion quares hrough paern compleion. Proceedings of he Naional Academy of Sciences of he Unied Saes of America, 102(8), 3164 3169. Remingon, R. J. (1969). Analysis of sequenial effecs on choice reacion imes. Journal of Experimenal Psychology, 82(2), 250 257. Roney, C. J. R., & Trick, L. M. (2009). Sympaheic magic and percepions of randomness: The ho hand versus he gambler s fallacy. Thinking & Reasoning, 15(2), 197 210. Simoncelli, E. P., & Olshausen, B. A. (2001). Naural image saisics and neural represenaion. Annual review of neuroscience, 24(1), 1193 1216. Skinner, B. F. (1948). Supersiion in he pigeon. Journal of Experimenal Psychology, 38(2), 168 172. Soeens, E. (1998). Localizing sequenial effecs in serial choice reacion ime wih he informaion reducion procedure. Journal of Experimenal Psychology: Human Percepion and Performance, 24(2), 547 568. Soeens, E., C, L., & E, J. (1985). Expecancy or auomaic faciliaion? Separaing sequenial effecs in wo-choice reacion ime. Journal of Experimenal Psychology: Human Percepion and Performance, 11(5), 598 616. Speeh, S. D., & Mahews, M. V. (1961). Sequenial effecs in he signal deecion siuaion. Journal of he Acousical Sociey of America, 33, 1046 1054. Sun, J., & Perona, P. (1998). Where is he sun? Naure Neuroscience, 1(3), 183 184. Tversky, A., & Gilovich, T. (1989). The Ho Hand : Saisical Realiy or Cogniive Illusion? CHANCE, 2(4), 31 34. Wilder, M., Jones, M., & Mozer, M. (2009). Sequenial effecs reflec parallel learning of muliple environmenal regulariies. Advances in neural informaion processing sysems, 22, 2053 2061. Yu, A. J., & Cohen, J. D. (2009). Sequenial effecs: Supersiion or raional behavior? Advances in neural informaion processing sysems, 21: 1873-1880.