Conversation

11:08 PM · Nov 18, 2020Twitter Web App

As important as getting good at estimating tails is getting good at knowing when to be super confident about tails -- most people are WAY too confident about them, and I’d recommend all the 0% and 100% people on this one could benefit from a little sanity-checking.

Show this thread

GIF

read image description

ALT

Replying to

2) NOT POLITICAL ADVICE. NOT INVESTMENT ADVICE.

3) Nate Silver said the election was 90% --> Biden. Prediction markets were 70%. Some people were 0%, some were 100%. Who was most right?

Cryptocurrency Derivatives Exchange

4) Well, here are some ways of estimating this! a) Biden won 100% of the 2020 presidential elections (1 out of 1!). People totally confident he would win were right! b) BUT WHAT ABOUT FRAUD???? 0% was right! c) ftx.com/trade/TRUMPFEB is at 15%; Silver was about right!

ftx.com

FTX

5) The problem, really, is that we don't get 5,000 2020 presidential elections, so we can't measure what % of them TRUMP won. We only get one of those, and the winner is binary. So how can you update based on that?

6) Well, you _could_ look at who won, and then credit whoever was most confident they would (yay, random internet trolls, and then 538!). But I think that's the wrong approach.

7) If in fact we say 5,000 simulations and Biden won each one of them, _then_ 3 cheers for confidence. But we only see one simulation. How much does the binary outcome update? PM: 70% Biden. 538: 90% Biden.

8) Bayesian update: Before: 50/50 between PM and 538 before After: 56/44 towards 538. Not much of an update! Why? Because both prediction markets and 538 thought Biden would probably win.

9) Neither were shocked by the binary outcome; 538 was only a bit less slightly surprised. So is there basically no update here?

10) Well, _in fact_, we don't think that the binary outcome was all people were predicting. We don't think 538's model was "90% exactly the map that happened, 10% that but with CA and NY for TRUMP so he wins". We actually _know_ their model because they published it.

11) But even if we didn't, we could have a good guess. 538's model was Biden winning popular vote by ~8% and electoral college by ~5%, with 90% that Biden won EC. Prediction markets don't publish models, but we can guess. They were 30% for TRUMP.

12) So _probably_, their implied model was something like: Biden winning popular vote by ~4%, EC by ~1%. Standard Deviation of ~3%. That's what you'd get if you assumed 538's model but shifted it all until it was 30% for Trump.

13) You can do this for lots of predictions: guess at the underlying model that generated them, and see how well it did. _now_ let's do a Bayesian update. Assuming normal distribution, SD of 3%, 538 predicted 8%, PM predicted 4%, and actual was 4%: 2.4x update for PM!

14) If you were 50/50 before, now you'd be 70/30 for prediction markets. A big swing, and the opposite of what we got before! I basically think this approach is better. It's using a fuller guess at people's models.

15) Basically: sure, 538 guessed the winner more confidently. But both guessed it right, and in fact prediction markets guessed the margin about right; 538 was off. The election was fairly close! Prediction markets nailed this; 538 didn't.

16) All of this, BTW, is assuming that Biden did/does win; if you disagree with that, you have different updates, and in fact update much _further_ towards prediction markets. But even this approach can be flawed.

17) What if, in an alternate universe, Biden won by 4%, but also 24h before the election he got caught in an armed robbery (unlikely, but you never know!). Sure, prediction markets would have been "right". But the truth is _no one_ was predicting burglary.

18) Really he would have been about to win by 8% before that really unlikely event. An even which wasn't contributing to either 538's or PM's models. I think the *right* update would have been: 538 was about right with +8%, and then some really unlikely thing happened.

19) So you can't even just look at the result and impute to models: you have to understand whether that was really part of what the models were modeling, and what the real update was. If you'd asked 538 and PMs how likely +8% --> burglary --> +4% was, 538 would have been higher!

20) So in fact the _full_ Bayesian update would have favored 538 there, even though updating on the % vote would have favored prediction markets. In this case, though, it wasn't shocking. Polls were wrong in about the way some thought they might be. About the same as 2016.

21) This was, in fact, pretty close to exactly what I think prediction markets were predicting. It wasn't *shocking* to 538 -- just over 1 standard deviation away from their mean! But it was a *little* bit surprising, at least more than it was to markets.

22) And this doesn't mean markets are always right either! After results started coming out, all PMs went to ~80% for Trump. Trump going up was correct! Florida was good for him. But 80% was probably too high, and PMs probably overreacted.

23) To be honest I'd be surprised if they *didn't*. Ask my rational brain and it'd have said "eh this takes it from 30/70 to 50/50".

24) Tell my emotional brain, and: "so you're saying TRUMP was heavily favored to lose, and was about to lose Florida and thus the election, but then one late-reporting Florida county came in way better for Trump than expected so he won Florida instead"--