Conversation

SBF
@SBF_FTX
3) Nate Silver said the election was 90% --> Biden. Prediction markets were 70%. Some people were 0%, some were 100%. Who was most right?
1
3
SBF
@SBF_FTX
5) The problem, really, is that we don't get 5,000 2020 presidential elections, so we can't measure what % of them TRUMP won. We only get one of those, and the winner is binary. So how can you update based on that?
1
3
SBF
@SBF_FTX
6) Well, you _could_ look at who won, and then credit whoever was most confident they would (yay, random internet trolls, and then 538!). But I think that's the wrong approach.
1
3
SBF
@SBF_FTX
7) If in fact we say 5,000 simulations and Biden won each one of them, _then_ 3 cheers for confidence. But we only see one simulation. How much does the binary outcome update? PM: 70% Biden. 538: 90% Biden.
1
1
SBF
@SBF_FTX
8) Bayesian update: Before: 50/50 between PM and 538 before After: 56/44 towards 538. Not much of an update! Why? Because both prediction markets and 538 thought Biden would probably win.
1
3
SBF
@SBF_FTX
9) Neither were shocked by the binary outcome; 538 was only a bit less slightly surprised. So is there basically no update here?
1
2
SBF
@SBF_FTX
10) Well, _in fact_, we don't think that the binary outcome was all people were predicting. We don't think 538's model was "90% exactly the map that happened, 10% that but with CA and NY for TRUMP so he wins". We actually _know_ their model because they published it.
1
2
SBF
@SBF_FTX
11) But even if we didn't, we could have a good guess. 538's model was Biden winning popular vote by ~8% and electoral college by ~5%, with 90% that Biden won EC. Prediction markets don't publish models, but we can guess. They were 30% for TRUMP.
1
1
SBF
@SBF_FTX
12) So _probably_, their implied model was something like: Biden winning popular vote by ~4%, EC by ~1%. Standard Deviation of ~3%. That's what you'd get if you assumed 538's model but shifted it all until it was 30% for Trump.
Twitter Web App
1
Like
SBF
@SBF_FTX
Replying to
@SBF_FTX
13) You can do this for lots of predictions: guess at the underlying model that generated them, and see how well it did. _now_ let's do a Bayesian update. Assuming normal distribution, SD of 3%, 538 predicted 8%, PM predicted 4%, and actual was 4%: 2.4x update for PM!
1
1
SBF
@SBF_FTX
14) If you were 50/50 before, now you'd be 70/30 for prediction markets. A big swing, and the opposite of what we got before! I basically think this approach is better. It's using a fuller guess at people's models.
1
2
SBF
@SBF_FTX
15) Basically: sure, 538 guessed the winner more confidently. But both guessed it right, and in fact prediction markets guessed the margin about right; 538 was off. The election was fairly close! Prediction markets nailed this; 538 didn't.
1
2
SBF
@SBF_FTX
16) All of this, BTW, is assuming that Biden did/does win; if you disagree with that, you have different updates, and in fact update much _further_ towards prediction markets. But even this approach can be flawed.
1
1
SBF
@SBF_FTX
17) What if, in an alternate universe, Biden won by 4%, but also 24h before the election he got caught in an armed robbery (unlikely, but you never know!). Sure, prediction markets would have been "right". But the truth is _no one_ was predicting burglary.
1
1
SBF
@SBF_FTX
18) Really he would have been about to win by 8% before that really unlikely event. An even which wasn't contributing to either 538's or PM's models. I think the *right* update would have been: 538 was about right with +8%, and then some really unlikely thing happened.
1
1
SBF
@SBF_FTX
19) So you can't even just look at the result and impute to models: you have to understand whether that was really part of what the models were modeling, and what the real update was. If you'd asked 538 and PMs how likely +8% --> burglary --> +4% was, 538 would have been higher!
1
1
SBF
@SBF_FTX
20) So in fact the _full_ Bayesian update would have favored 538 there, even though updating on the % vote would have favored prediction markets. In this case, though, it wasn't shocking. Polls were wrong in about the way some thought they might be. About the same as 2016.
1
1
SBF
@SBF_FTX
21) This was, in fact, pretty close to exactly what I think prediction markets were predicting. It wasn't *shocking* to 538 -- just over 1 standard deviation away from their mean! But it was a *little* bit surprising, at least more than it was to markets.
1
1
SBF
@SBF_FTX
22) And this doesn't mean markets are always right either! After results started coming out, all PMs went to ~80% for Trump. Trump going up was correct! Florida was good for him. But 80% was probably too high, and PMs probably overreacted.
1
2
SBF
@SBF_FTX
23) To be honest I'd be surprised if they *didn't*. Ask my rational brain and it'd have said "eh this takes it from 30/70 to 50/50".
1
3
SBF
@SBF_FTX
24) Tell my emotional brain, and: "so you're saying TRUMP was heavily favored to lose, and was about to lose Florida and thus the election, but then one late-reporting Florida county came in way better for Trump than expected so he won Florida instead"--
1
5
SBF
@SBF_FTX
25) My first instinct was "whelp, we've been here before". We hadn't, in fact! 2020 was different: Biden's lead was ~4% higher which was enough of a cushion. But prediction markets sure thought it looked the same.
4
8