Thomas Yeo

Look forward to that blog post. I'd add a 3rd variety who spend plenty of time designing and conducting replications and control experiments to make sure they are not mining noise before attempting to publish. I'm not saying you can't fool yourself with this strategy, but still.

Gah how dull. Where’s the flair in that.

I know, what a bunch of losers.

If you read my post it should be clear that I am NOT saying that what the field is doing now is fine. What I said is that standard practice is horrible, but that RR won't fix it. The first thing to do is to kill the idea that point-null hypothesis testing is sufficient.

RR seems to be primarily about reducing Type I error. But if you view PNHT as insufficient, merely a weak, poorly reasoned pretest of data quality, then it becomes obvious that the focus should be elsewhere. We need a revolution, not more paperwork.

I don't get this at all. RRs just make it so that papers are accepted based on question and methods, and not based on results and post-hoc storytelling. I'm surprised you're not in favor of that. they don't add much (if any) work, and they're completely orthogonal to Q of NHST

if anything, it should be much *easier* to criticize studies for using NHST when an RR is first submitted for review, than to wait until after the authors are happily trumpeting their p < .001 result and can say "but look, it's strong!"

The appropriate answer to "look my p < .001 is strong" is to explain the difference between significance and prediction out of set. RR could perversely give people cover to do what they shouldn't do to begin with (that is, to equate significance as effect size or prediction).