Azhir Mahmood  🚀 🤖 🧬 🧠 🪐

Code review and stats check are real open issues in the current publication landscape! I'd love to hear ideas on how to institute this...https://twitter.com/neurocow/status/1366101710641463306 …

Do we really want to institute this and make the publishing process even more painful & time-consuming, or should we not just trust authors that they've done their best (which we have to do for experimental data anyway - we trust authors that they didn't just make up data)?

The question is to what degree we want such quality control. Could be automatic, i.e. you have to submit a notebook format paper and all figures / analyses need to compile...

Sure - and it's a very valid question. I'm just not sure what exactly you could detect with this unless you spent a lot of time checking the code. Supposedly the code ran when the authors used it to make the figs so it's going to run regardless of who's running it.

The code may well require access to some specific HPC system which neither reviewers nor readers will practically have access. Certainly I have written papers where that is the case. Plotting figures is one thing; proper number-crunching another.