This is why I like Kaggle, or in general, competitions with a hidden test set, a reproducibility constraint, and a money incentive. It forces you to be honest with yourself and your peers. Academic incentives encourage intellectual dishonesty.https://twitter.com/egrefen/status/928897300675661824 …
-
Agreed, contests are nice that way. OTOH, they are surprisingly hard to run right. Often evaluation variance is too high, there is a data set leak, or the dataset is broken in some other way. Plus, there are no "good idea papers" just bad scores.
Hard to run, but when hundreds of participants give it a shot, any bug in the data/eval becomes apparent after a few weeks. Self-correcting system. Also, if you have a good idea that scores poorly, you can always write about it. The score forces you to be honest about its merits
