Noah Haber

Thanks in particular to @danengber from http://Wired.com  for inviting me and working to edit this into something legible. Some entertaining back story: this was originally supposed to be musings on uncertainty, with masks as the example. Then DANMASK-19 dropped.

I really can't express how weird it is that this exists. Learned a LOT about op-eds thanks to the editors at Wired, and all the various friends who took a look at early versions of this.

Remember when being a pod person meant you were from outer space?

MORE OF THIS PLEASEhttps://twitter.com/LucyStats/status/1329850597718200321 …

And yeah, I am millennial AF.

Maybe confirmation bias at this point, but I constantly think about how the generational divides in science/metascience match so neatly with the generational divides in society and politics.

Sometimes, I critique causal inference. Other times, I causal inference critique. It's probably not healthy.https://twitter.com/NoahHaber/status/1329791259326750721 …

Yes, I do understand the meta-irony here. But then again, you probably shouldn't trust me, because I said this would be a brief thread. Honestly, you're probably out of salt by now.

Should we rest satisfied by that? ABSOLUTELY NOT. But it does mean that we should not be dismissing valid critique just because it's high profile or (partially) driven by the results. We should avoid spending more time casting doubt on the robust critique than the weak study.

More critique happens for higher profile studies. Weaker methods tend to produce more extreme results Weak methods -> extreme results -> out of line with expectations -> high profile We tend to (BUT NOT ALWAYS) get more critique when methods are worse, even when results-driven

Now, as for why this isn't as big a problem as some might assume, we have to get a little causal inference-y (draw a DAG if ya like), and think about WHY results-driven critique happens. There are a few steps here, so bear with me.

The DANMASK trial is a slightly unusual case, because the design of the trial completely determined the results. There is never a strict separation between methods and results, but for the DANMASK trial they were effectively one and the same. So, grain of salt again.

In the case of the DANMASK-19 trial, we made our critique on the methods and predictions, BEFORE the results came out (credit to the DANMASK trial authors for making their design available before results publication). I wish this was how it could always be done.

I'll compile a list of some folks who I've learned to trust with regard to fair and well-thought out critique done in good faith regardless of the results in the near future, so that others can seek them out. If you know who to look for, you can ignore the noise.

I've stated this before, but if I have a results-driven bias, it's more likely that I err on the side of overcompensation critiquing studies whose conclusions I agree with. But again, have a grain of salt, we are rarely the best judges of ourselves.