I debated at #DALI2019 about enforcing code release and #reproducibility in #AI #ML #conferences. This is the right thing to do: it is fair and it will ensure honest progress. Otherwise we are headed to next #AIwinter.https://anima-ai.org/2019/01/04/reproducibility-debate-should-code-release-be-compulsory-for-conference-publications/ …
-
Show this thread
-
Replying to @AnimaAnandkumar
IIRC, rapid progress in Robocup came because each year people published their code. This acted as a ratchet: the code from the best team in one year became the basis for the next year. So even "leaderboard chasing" improves with open code.
2 replies 1 retweet 22 likes -
Replying to @tdietterich
Indeed, enforcing code release will make leaderboard chasing even easier. That will allow us to make quick progress and community can move on to deeper and harder problems. Now we are spending enormous time+energy+contributing to climate change to replicate broken algorithms.
1 reply 1 retweet 11 likes -
Replying to @AnimaAnandkumar @tdietterich
It could be that publishing code promotes exploitative research, incrementally improving on last year's best, at the cost of radical explorative research that must write its own ab initio code. I don't know how serious a concern this is.
1 reply 0 retweets 3 likes -
Replying to @maosbot @tdietterich
I commented on that. Barrier to entry for such incremental research becomes low, so it won't be considered innovative. Also, academics can compete on a more equal footing. We can quickly figure out which algorithms are brittle and move to deeper questions
2 replies 0 retweets 5 likes -
Replying to @AnimaAnandkumar @tdietterich
I think the true target of my concern is not code sharing, but publication being contingent on performance superior to that of the best of a deeply-mined mode. It might take a while to build up a code base equivalent to that of the current best.
1 reply 0 retweets 3 likes -
Put another way: what if the next revolutionary classifier is not a good match for tensorflow or pytorch? Would it get off the ground?
1 reply 0 retweets 2 likes
Specification is just as important as code if not more so for progress.
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.