psychology and economics lecture 9: biases in statistical reasoning tomasz strzalecki
failures of Bayesian updating how people fail to update in a Bayesian way how Bayes law fails to describe how people update
plan for today base rate fallacies: people under- or over- use their prior base rate neglect confirmation bias sample size fallacies: ppl are under-sensitive to sample size law of small numbers gambler s fallacy hot hand fallacy sampling variation fallacy non-belief in the law of large numbers
Bayes rule: tradeoff between signal and prior posterior = signal prior P(A B) P(notA B) = P(B A) P(B nota) P(A) P(notA) conditional odds = likelihood ratio unconditional odds base rate neglect: posterior not responsive to the prior confirmation bias: posterior is sticky
base rate neglect A cab was involved in a hit and run accident at night. Two cab companies, the Green and the Blue, operate in the city. You are given the following data: (a) 85% of the cabs in the city are Green and 15% are Blue (b) A witness identified the cab as Blue. The court tested the reliability of the witness under the same circumstances that existed on the night of the accident and concluded that the witness correctly identified each one of the two colors 80% of the time and failed 20% of the time what is the probability that the cab involved in the accident was Blue rather than Green? % many people say 80%
base rate neglect P(B b) P(b B) = P(G b) P(b G) P(B) P(G) = 0.8 0.2 0.15 0.85 < 1
confirmation bias the same information is interpreted in different ways, depending on the prior belief polarization Lord, Ross, and Lepper (1979) subjects either opposed (opponents) or favored capital punishment (proponents) read two articles that discussed whether the death penalty deters violent crimes one said yes, another no after reading: proponents were more in favor of capital punishment opponents less in favor of capital punishment proponents had greater belief in the deterrent effect of capital punishment opponents reported less belief in this deterrent effect
confirmation bias Plous (1991) replicated this study in the context of judgments about the safety of nuclear technology pro- and anti-nuclear subjects were given identical information and arguments regarding the Three Mile Island disaster belief change: 54 % of pronuclear subjects became more pronuclear only 7 % became less pronuclear 45 % of antinuclear subjects became more antinuclear only 7% became less antinuclear
sample size fallacies mathematical truth: noise goes down as sample size increases small samples are random in large samples more structure appears in the limit, frequecies converge to probabilities law of large numbers (LLN)
sample size fallacies noise true relationship 1 sample size general phenomenon: people misunderstand sample size
sample size fallacies noise true relationship LLN 1 sample size general phenomenon: people misunderstand sample size
sample size fallacies noise true relationship perception 1 sample size general phenomenon: people misunderstand sample size
sample size fallacies noise true relationship perception 1 sample size general phenomenon: people misunderstand sample size one extreme: law of small numbers (people think small samples are large and the LLN applies) in between: sampling variation fallacy (mispredict differences in randomness of samples of different size) another extreme: non-belief in the law of large numbers (underestimate how fast the LLN kicks in)
two manifestations of sample size fallacies noise informativeness true relationship true relationship 1 sample size 1 sample size people mis-predict randomness people mis-infer from samples
sample size and informativeness suppose that you have a coin with an unknown bias for simplicity, you assign equal probability to two scenarios: 3 5 Heads, 2 5 Tails (call this θ H) 2 5 Heads, 3 5 Tails (call this θ T ) toss 3 times and see HHT what are your posterior odds of θ H to θ L
sample size and informativeness so P(θ H HHHTT ) P(θ T HHHTT ) = P(HHHTT θ H) P(HHHTT θ T ) P(θ H) P(θ T ) = 3 5 3 5 3 5 2 5 2 5 2 5 2 5 2 5 3 5 3 5 1 2 1 2 = 1.5 P(θ H HHHTT ) = 0.6
sample size and informativeness P(θ H (HHHTT ) n ) P(θ T (HHHTT ) n ) = P((HHHTT )n θ H ) P((HHHTT ) n θ T ) P(θ H) P(θ T ) ( 3 5 = 3 5 3 5 2 5 ) 2 n 1 5 2 2 5 2 5 2 5 3 5 3 = 1.5 n 1 5 2 posterior belief in Θ H 0.9 0.8 0.7 0.6 5 10 15 20 25 30 35 40 45 50 sample size with frequency 3 of H 5
two manifestations of sample size fallacies noise informativeness true relationship true relationship 1 sample size 1 sample size people mis-predict randomness people mis-infer from samples
two manifestations of sample size fallacies noise informativeness true relationship perception perception true relationship 1 sample size 1 sample size people mis-predict randomness people mis-infer from samples
the law of small numbers the false belief that in the LLN holds in small samples example: sex of six babies born in a sequence in a hospital which sequence is most likely? BBBGGG GGGGGG BGBBGB under independence and P(B) = B(G) = 1 2 : P(BBBGGG) = 1 1 1 1 1 1 2 2 2 2 2 2 P(GGGGGG) = 1 1 1 1 1 1 2 2 2 2 2 2 P(BGBBGB) = 1 1 1 1 1 1 2 2 2 2 2 2
the law of small numbers suppose you are tossing a fair coin and heads came up 7 times in a row what is more likely in the next toss: heads? tails? gambler s fallacy
gambler s fallacy gamblers fallacy: the false belief that in a sequence of independent draws from a distribution, an outcome that hasnt occurred for a while is more likely to come up on the next draw if red came up in roulette four times in a row, a black is due mean reversion; regression to the mean
Gold and Hester s (1987) experiment subjects were told that a coin with one black and one red side would be flipped 25 times the experimenter actually reported a pre-determined sequence of flips: 17 mixed, then 1 black, and then 4 red on the 23rd flip, participants were given a choice between 70 points for sure 100 points if the next flip was their color, 0 pts otherwise half of the subjects color was red and half s was black results: 24 of 29 red subjects took the sure thing 8 of 30 black subjects took the sure thing
hot hand fallacy Gilovich, Vallone and Tversky (1985) 91% of fans agreed that a player has a better chance of making a shot after having just made his last two or three shots 68% of fans agreed that a player has a better chance of making a shot after having just made his last two or three free throws 84% of fans agreed that it was important to pass the ball to someone who has just made several (two, three, or four) shots in a row. Gilovich et al. analyzed basketball statistics basically, P(H HHH) = P(H HMH) = P(H MMM)
hot hand fallacy
hot hand fallacy coach of the Boston Celtics on the Gilovich et al study: Who is this guy? So he makes a study. I couldn t care less.
hot hand fallacy likely explanation: over-inference people have the following model in mind: hot streaks P(H) is higher cold streaks P(L) is lower if we observe HHH this means that the chances are that the player is in a good shape today so the next shot will be a hit if we observe MMM this means that the chances are that the player is in a bad shape yesterday so the next shot will be a miss so also a law of small numbers phenomenon!
sampling variation fallacy A study of the incidence of kidney cancer in the 3,141 counties of the United States reveals a remarkable pattern. The counties in which the incidence of kidney cancer is lowest are mostly rural, sparsely populated, and located in traditionally Republican states in the Midwest, the South, and the West. What do you make of this?
typical replies sparsely populated located in traditionally Republican states rural typical responses focus on clean living of the rural lifestyle: no air pollution, no water pollution, healthy food
sampling variation fallacy A study of the incidence of kidney cancer in the 3,141 counties of the United States reveals a remarkable pattern. The counties in which the incidence of kidney cancer is highest are mostly rural, sparsely populated, and located in traditionally Republican states in the Midwest, the South, and the West. What do you make of this?
typical replies sparsely populated located in traditionally Republican states rural typical responses focus on poverty of the rural lifestyle: no access to good medical care, climate, they drink too much
sampling variation fallacy jar with 50% black and 50% white stones David draws 4 stones at the time David wins if all stones are of one color Tomasz draws 7 stones at the time Tomasz wins if all stones are of one color not a zero-sum game! Who is more likely to win? how much more likely?
sampling variation fallacy possible states for David: BBBB, BBBW, BBWB,..., total of 2 4 = 16 states we are interested in two: BBBB and WWWW probability that David wins: 2 16 possible states for Tomasz: BBBBBBB, BBBBBBW, BBBBBWB,..., total of 2 7 = 128 states we are interested in two: BBBBBBB and WWWWWWW 2 probability that Tomasz wins: 128 David wins 2 16 2 128 = 8 times more frequently than Tomasz
sampling variation fallacy In a small town nearby, there are two hospitals. Hospital A has an average of 45 births per day; Hospital B is smaller, and has an average of 15 births per day. As we all know, overall the number of males born is 50%. Each hospital recorded the number of days in which, on that day, at least 60% of the babies born were male. which hospital recorded more such days: (a) hospital A (b) hospital B (c) both equal
sampling variation fallacy In a small town nearby, there are two hospitals. Hospital A has exactly 1000 births per day; Hospital B is smaller, and has exactly 1 birth per day. As we all know, overall the number of males born is 50%. Each hospital recorded the number of days in which, on that day, at least 90% of the babies born were male. Which hospital recorded more such days: (a) hospital A (b) hospital B (c) both equal
non-belief in the LLN suppose we toss a fair coin N times what is the probability that the fraction of heads is: between 45% and 55%? between 85% and 95%? etc...
Kahneman and Tversky (1971) reported probabilities
true probabilities Kahneman and Tversky (1971) reported probabilities
back to the coin example suppose that you have a coin with an unknown bias for simplicity, you assign equal probability to two scenarios: 3 5 Heads, 2 5 Tails (call this θ H) 2 5 Heads, 3 5 Tails (call this θ T ) you observed n heads and m tails what are your posterior odds of θ H to θ L
sample size P(θ H H n T m ) P(θ T H n T m ) = P(Hn T m θ H ) P(H n T m θ T ) P(θ H) P(θ T ) ) n ( 2 m 5) = = ( 3 5 ( 2 ) n ( 3 m 5 5) 1 2 1 2 ( 3 n m ( 5) 2 ) n m = (1.5) n m 5 this is a case where the sample size should not matter this is pretty sophisticated!
Griffin and Tversky (1992)
perspective biases in statistical reasoning base rate fallacies sample size fallacies people don t do exactly what the Bayes law says how to model their learning? work of Matthew Rabin and coauthors work of Andrei Shleifer and coauthors
perspective Bayes theorem is 300 years old there are new theorems in statistics proven every day does this mean that economic agents must know them? at what point do these experiments become IQ tests? difference between experiments and problem sets? our statistical intuitions often point in the right direction bounded rationality which word to emphasize? Heuristics and Biases
going forward... next two classes will be about behavioral game theory one way to think about it using what we learned: in strategic situations lots of inferences to make people mis-infer from the behavior of the others