
… and the total number of papers published might drop precipitously if reviewers were more reluctant to recommend publication.
That would be good. Everyone agrees there’s WAY too much stuff published under the current system. A 90% reduction would be great.
-
Show this thread
-

I’d suggest that every PhD student be required to perform at least one attempt at Sokaling as a graduation requirement.
Learning what should count as unacceptably bad research is a critical part of learning how to do it well. And of spotting the difference in the lit.3 replies 12 retweets 65 likesShow this thread -

Initially, everyone would go for low-hanging fruit in Sokaling attempts: the easiest ways to get nonsense past reviewers.
Reviewers would quickly catch on to the simplest tricks… then subtler errors.
And I hope this would lead to a virtuous upward spiral of quality.2 replies 3 retweets 45 likesShow this thread -

Three questions:
𝔸) Would this make academia more adversarial, and thereby even more awful?
𝔹) Can a system develop that is adequately resistant to gaming (Goodhart’s Law)?
ℂ) Is it realistic to imagine something like this could actually happen?6 replies 6 retweets 54 likesShow this thread -
Replying to @Meaningness
a) Yes. I also worry it would make it more conservative - it's hard enough to get niche papers about weird ideas published already (speaking as someone who is trying) b) Probably, but not easily. c) No. It requires unilateral coordination from people it would hurt.
1 reply 1 retweet 4 likes -
Replying to @DRMacIver @Meaningness
Maybe that was too harsh. You might be able to do something with the idea on a sort of alt-peer-review basis - build a community of people who are actively working on improving their research together and use it as a sort of "certification" process by submitting preprints to it.
1 reply 0 retweets 1 like -
Replying to @DRMacIver @Meaningness
So people would continue to publish under the normal system, but people who have been sokal-certified get more kudos because it makes their paper look more respectable.
1 reply 0 retweets 0 likes -
Replying to @DRMacIver @Meaningness
Still has the conservatism bias problem though, and I think that's intrinsic to the proposal.
1 reply 0 retweets 1 like -
Replying to @DRMacIver @Meaningness
Another issue is that it's really easy to Sokal by going cross discipline (indeed this is how he did it) because it's very hard to review those well with only one expertise, which would create a large bias against legitimate cross disciplinary work
1 reply 2 retweets 9 likes -
Replying to @DRMacIver
Excellent point, yes. OTOH, it’s also exceptionally easy to get rubbish published that way (sprinkle some irrelevant math in a paper in any soft-science field). I know many people who have been extremely successful that way. So extra skepticism is warranted.
1 reply 1 retweet 1 like
Since much of my own work has been cross-disciplinary, this problem is acute for me…
-
-
Replying to @Meaningness
Yeah I think the current system is just very poorly set up for cross disciplinary work.
1 reply 0 retweets 2 likes -
Replying to @DRMacIver @Meaningness
TBH my preferred patch is probably to weaken rather than strengthen peer review and create better mechanisms for post publication checking. Publication needs to indicate interesting rather than true
1 reply 0 retweets 4 likes - 2 more replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.