It’s a lot harder to data mine a credible looking graph.
Running a diff-in-diff for the first time in ages. Ccannot believe how many degrees of freedom I have to choose specifications! And just yesterday I watched an experimental paper get dinged for running an extra regression and not adjusting std errors. What a sham science we are.
-
-
-
yeah and its pretty easy for a researcher with deep pockets to run pilot RCTs until you get the exact detailed treatment design you're willing to scale up and write the pre-analysis plan with.
-
Has this ever been done?
-
And isn’t that called out of sample validation of a hypothesis?
-
but if the hypothesis is about a class of treatments that are supposed to be invariant to small details then you're not testing that hypothesis.
-
back in the days when Zvi Griliches had to invert his 6 x 6 matrices by hand, he would do only one regression...
-
Computational scarcity as a means of quality assurance? :)
- 2 more replies
New conversation -
-
-
@fburlig has a nice paper about doing pre analysis plans in non experimental work. Not the same slam dunk as w RCTs, but this is a great example of how they can still be valuable. -
Are you kidding me? I love this. I’m like a kid having sugary for the first time. Not sure I will ever publish this paper but I am giddy running 4 million different regressions until I get the little endorphin asterisks.
-
Chris! This is not Hollywood! Do not go looking for stars!
End of conversation
New conversation -
-
-
It's all a bit of a sham, Chris. We just like to pretend it isn't.
-
Robustness checks something something. No, I’m with you. Absent a credible way to preregister observational studies with existing data, huge problem.
-
Yeah, you never see the robustness tests that didn't work out. This is where I think open data is key -- preferably in the peer-review phase as well.
-
Related: "estimates suggest that 45% of failed tests remain in the "file drawer" rather than being published." https://www.nber.org/papers/w25058
-
yeah, we know from Ioannidis et al. that econ has huge false positive/inflated size problem too despite all the robustness tests https://onlinelibrary.wiley.com/doi/full/10.1111/ecoj.12461 …pic.twitter.com/FRWi4UhmSX
End of conversation
New conversation -
-
-
Someone should write a recommended "checklist" for diff-in-diff studies (like this one for RDD https://www.princeton.edu/~davidlee/wp/RDDEconomics.pdf …). My sense is many people have a reasonably standard way to proceed in mind, without too much freedom (check balance, check for pre-trends...)
-
This depends on how the checklist is used: I've seen ref reports ding papers for not following the RDD "checklist" even when an item wasn't applicable or reasonable.
-
Agreed. Checklists should serve as a guide, but not supersede common sense.
End of conversation
New conversation -
-
-
Just because you can get magic stars doesn’t mean you will get your ad hoc spec choices past editors and referees. Give the profession a little credit
-
My personal philosophy of significance testinghttps://youtu.be/_VJlHWESyLI
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.