Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
A Cold Shower for the Hot Hand Fallacy Joshua B. Miller (Bocconi University and IGIER) with Adam Sanjurjo (University of Alicante) CAGE Conference A QUICK TOUR OF THE HH FALLACY (12 SLIDES) Hot Hand: the belief and the fallacy • An individual believes in the hot hand if s/he: – considers a good outcome as more likely than usual after a streak of good outcomes, or – regards "hot" streaks of good outcomes to be too long or too frequent • And therefore attributable to some unobserved or transient factor (e.g. human agency). • The belief becomes a fallacy in an environment in which a good outcome always has the same chance of occurring. Some history • Research from the 1960s & 1970s: e.g. Wagenaar 1972 – People are not good at discriminating random (iid) sequences of outcomes from non-random ones. – People expect negative recency from random sequences (modal belief: P(H|T)=P(T|H)=0.6 ) – When the generating process is unknown, people mistakenly judge random sequences to be nonrandom in the absence of negative recency: • streaks are "too long" to be due to chance • the distribution does not balance in the near term. (i.e. law of small numbers, Kahneman & Tversky, 1971) Early "hot hand fallacy" results 1. Evidence is typically from artificial environments (lab, casinos) – Does it matter in more naturalistic decision making environments? 2. Decision makers are amateurs 3. The decisions may have real consequences for whomever the decision effects, but the marginal cost of indulging incorrect beliefs isn't high. – the opportunity cost of betting on black isn’t high assuming you are already gambling; if there is any “irrationality” it is gambling to begin with. 1985 • Gilovich, Vallone & Tversky (1985) had the novel insight that the sport of basketball was an excellent potential exhibit: – Clean and relatively controlled decision environment. – Experts (coaches and players) believe that streaks occur because sometimes an individual player exceeds their average ability level ("hot hand", "in the zone“, “in rhythm”) – The lab results suggest people may over-infer from streaks; are streaks exceptional when controlling for a player's ability? – There is a real economic cost to the experts for believing in the hot hand (ball allocation, substitution, defensive strategy, etc.). GVT 1985 • • • Validated that observers and players believe in the hot hand. Collected data from in game shooting situations, free throw shooting, and a controlled shooting experiment. Found that shooting performance was consistent with individual players always shooting with a fixed probability of success. (i.e. iid Bernoulli trials) Robustness of GVT results • Subsequent failure to detect the hot hand: – Larkey, et al. (1989), Wardrop (1999), Koehler & Conley (2003) , Avugos, Bar-Eli, Ritov, Sher (2013) • Persistence of beliefs & behavior: – Coaches and players believe and act on it (Rao 2009 , Cao 2011, Attali 2013) The Consensus • "Many researchers have been so sure that the original Gilovich results were wrong that they set out to find the hot hand. To date, no one has found it" Thaler & Sunstein, Nudge (2008) • "The hot hand [in basketball] is a massive and widespread cognitive illusion." Daniel Kahneman, Thinking , Fast and Slow (2011) Influence of the hot hand fallacy • With diverse applications: – Camerer (1989); Brown and Sauer (1993); Barberis, Shleifer, and Vishny (1998); Rabin (2002); Croson and Sundali (2005); DellaVigna (2009); Guryan and Kearney (2008); • More recently: – Rabin & Vayanos, (2010) have a model that generates the HH fallacy and can explain financial puzzles such as fund flows and active fund puzzles. Importance of the hot hand fallacy • Counter-intuitive psychologically. – Confidence and personal efficacy is influenced by performance, and there is evidence they can feedback into performance (Bandura 1982, Solomon 2001) – Noise in preparatory neurons vary and can explain variation in movement (Churchland 2006) – Natural fluctuations in attention/concentration But there are simpler reasons to doubt this.... HH fallacy is a severe challenge for rationality • HH fallacy is found in experts in their domain of expertise, with high stakes. – While systematic mistakes among experts is not unheard of, e.g. the “Moneyball” anomaly (Lewis 2004) – They tend to go away, e.g. Hakes & Sauer 2006 • HH fallacy goes further, because the HH belief is robust to learning and advice: – "Amos Tversky used to describe the robustness of the belief in the hot hand by saying that he has been in a thousand arguments over this topic, won them all, but convinced no one" (Gilovich, 2002) Our study • One should be skeptical of this degree of irrationality • We revisit HH fallacy in domain where the evidence is considered the strongest and most readily generalizable. • The evidence is not as strong as it seems, and this necessitates a discussion of the particularities of basketball shooting data (and an experiment)... Validity concerns for game data • "Live ball" shooting: – confounds: shot distance, angle, type; defense, incentives, fatigue, etc... – Shot frequency (fundamental) – Aggregation across games & shot selection bias • Free throw data: – Shot selection bias – Hot hand or cold hand? – Recent Evidence: Arkes 2010, Yaari & Eisenmann 2011 GVT's controlled Experiment • GVT recognized some of the aforementioned validity concerns. • Their results rely on a controlled shooting experiment. • Their design and analysis has severe limitations that persist into the subsequent literature. Design • Design: – A distance is determined where player is believed to hit 50% of shots. – Players take 100 shots, moving in an arc – Payment • paid fixed $2 • For each shot, players bet on own shot choosing either: – "high" [variance] bet ($0.05 Hit, $0.04 Miss), or – "low" [variance] bet ($0.02 Hit, $0.01 Miss) GVT Empirical Strategy: 4 statistics 1. Serial correlation (lag 1) 2. Runs: – – Streaky: 10 shots, 3 runs: 1100001111 Anti-Streaky: 10 shots, 8 runs : 0110101101 Problems: • Neither measure hotness • Both measure first order serial dependence : GVT Empirical Strategy: 4 statistics 3. Conditional probability – E.g. Prop(hit| 3+ hits) vs. Prop(hit| 3+ misses) – Could be driven by the cold hand. 4. Distribution of performance in four-shot window (test of statitionarity) OUR STUDY: Design Overview • Phase 1: – 10 players shooting for money. – Players complete multi-item questionnaire. • Inter-Phase – Between Phase 1 & Phase 2 – Subjects in Spain and Italy watch videos and predict player performance. • Phase 2: – 6 months after Phase 1. – 8 players return and shoot for money Our Shooters Phase 1: • 10 semi-professional male basketball players – Santo Domingo de Betanzos (former/current) • In the 5th of 7 categories (can move to 4th next year) – Recruited at practice • All players were interested in participating but some had scheduling conflicts so could not Phase 2: • 8 returned (+3 ad-hoc additions) THE SHOOTING TASK (VIDEO) Design: Details 1. Shooter warms up 2. Shooter always shoots from same spot – calibrated so they are expected 50% (3-pt) 3. 300 shots – Calibrated to avoid fatigue concerns. 4. No betting considerations for shooter 5. Paid 5 € show up fee + 6 € for each make from 10 randomly selected shots 3 hot hand statisitics • Hot Streak Length – Length of the longest run of hits (in 300 shots) : • Hot Streak Frequency – % of shots taken during a hit streak of 3+ : • Hot Streak Momentum – % hits if hit 3+ : The null reference model The null reference model Distribution of statistics under H0 • In principle, there is an exact distribution of each test statistic under the null hypothesis of exchangeability: – Calculate the test statistic for every permutation of the data, each permutation is equally likely. • We can approximate this distribution to arbitrary precision with Monte Carlo permutations of shots. • Exchangeability is consistent with more than Bernoulli – We find that our significance levels are the same or more conservative for Bernoulli shooters. • When aggregating we use subject and session strata (to avoid the influence of good day/bad day effects) STRATEGY TO DETECT THE EXISTENCE OF THE HOT HAND Selecting the hot player Phase 1: 1. Multi-item questionnaire of teammates: – – If hot hand belief has any validity, teammate opinion should predict the hottest player. If the task has any external relevance, in-game hot hand should predict the hot hand in the shooting task. 2. Hot Hand Statistics in the 300-shot task. 3. Student performance in a laboratory prediction task (watching videos of the player's 300 shots) – If observers have hot hand beliefs and a player is truly hot, observers should outperform chance in a prediction task. Identifying the hot player Phase 2: 1. Run additional sessions for the "hot player" in Phase 2. – If the hot hand is a player specific trait, it should persist across time . Phase one statistics... Bin(7,8)=.035 .7 .2 Phase 1 1 Momentum (H M) 1 .15 8 5 5 .5 3 2 8 3 .4 2 6 8 .45 .55 .5 7 2 Runs (R) 8 53 140 8 3 .4 150 10 7 .35 160 12 1 Length (H L) .3 .2 .15 .1 .05 0 4 6 .3 .05 4 2 4 5 6 130 1 4 6 120 6 4 Observed 7 7 .1 Observed .6 Frequency (H F) 5 6 7 Median 8 9 130 135 140 Median 145 150 RC is hot • (Stats) RC was the hottest player in phase 1 • (Teammates) RC was selected by teammates as hottest in games. • (Human Detectors) Observers predicted well with him, significantly better than chance. Will this predict what he does 6 months later? Bin(6,6)=.016 Player RC (subject 1) Not Cold hand! RC not the cold hand • Linear probability model of fg% – Base category: Hit 3+ PANEL OF 8 PLAYERS: PERSISTENCE .25 .7 Panel (averaged over sessions) Momentum (H M) .4 .5 .15 .3 .05 .1 Observed .2 .6 Frequency (H F) .15 .2 .25 .4 .45 150 .1 14 .05 .55 .6 .65 Runs (R) 145 135 6 8 140 10 Observed 12 Length (H L) .5 6 8 10 Median 12 140 142 144 146 Median 148 150 Aggregate analysis More data: Jagacinski, Newell, Issac (1979) More data: GVT Overall Discussion • Hot hand exists • Hot hand is substantial in some players. What about the hot hand fallacy? • The basketball result is the central exhibit of the HH fallacy being general and powerful. – It can no longer be used. • More generally, evidence appears to be thin that mistaken belief in the hot hand is powerful and costly. • Our view: Rabin has it right, HH belief is a product of a (conditionally) rational inference if you are uncertain about the possibility of a regime change (when in fact it cannot happen) Conclusion