Tal Yarkoni

Great! All that remains is for someone to pay for it...

the irony is that funding such a thing would cost a tiny fraction of what we all currently pay publishers for essentially no value. I think the fact that we *don't* already have such institutions everywhere kind of gives away where our values really lie

BREAKING NEWS: I just got a call from Elsevier. If Tal's tweet above gets 100 likes Elsevier will quit the publishing business and pivot to manufacturing and selling canned fish.

What sort of canned fish? Something sustainable like sardines, I hope.

Carp.

Well given we were talking about MRI, salmon might make more sense, no?

I think @whatthecarp may want a word with you

Ok. I will admit that this one flew over my head...

This will can go downhill many ways - Who selects papers to check (all clearly impossible) - Who decides b/w "not optimal" and "error"? - Consequences? How about public shaming (-> breaking ECR = pruning) - Legal action if people feel misjudgment ruined their hire/tenure

I'm not a fan of "it could conceivably go wrong, so we shouldn't do it" reasoning. here are direct analogs of your questions: * who decides what gets published? * who decides what's good enough to publish? * what about people winning praise and adulation for shitty work?

Just love to be the advocates diaboli ;-) More serious, big differences: There are many journals & alternatives for publishing. People getting praise for bad, high-IF work doesn't directly hurt you. A "science police" would be a unique power directly impacting lives

you must be joking. people getting praise for bad, high-IF work hurts *everyone*. it lowers the quality of science both by shaping the kind of work people strive to publish, and by actively rewarding (with jobs!) people who do bad work.

Yet, but it's bad in a more general way (as you mentioned) not targeting a specific person. Imagine an ECR being caught by the "science police" for a stupid mistake and publicly called out in the sake of openness. Given current levels of competition, that person is done

that sounds suspiciously like being an ECR who lucked into an undeserved publication in a high-IF journal, and screwed someone else who does better work out of a job. if your point is that life is unfair, I agree.

if you have some constructive ideas about how one is supposed to criticize specific papers without "targeting" the specific findings in those papers, I would love to hear them. otherwise you're just endorsing a system that incentivizes people to cut corners when producing papers.

1) Differentiate mistake vs. error vs. QRP vs. fraud 2) Not fraud: communicate w/ authors 2.1) Give chance for a correction (which should be seen as positive) 2.2) Disagree whether error/QRP: publish views side-by-side 3) Last resort / fraud: public Boring but effective & fair

The question is what they do when they are found...

you publicly document the problems and their consequences—just like we do with bugs in open-source software. how seriously a scientific community then takes those public bug reports is a pretty good indication of how much you should trust its science.

It would be great if all of this could be crowdsourced somehow. Might help with another weakness of the current system: Wild inconsistencies among the (tiny sample) of reviewers.