Most fields operate mainly in the s. repro space, and honestly I was a bit shocked when I entered machine learning and saw the intense focus on e. repro.
-
Show this thread
-
Nearly all the experiments done in my PhD lab were practically reproducible in only a handful of places in the world. But this doesn't invalidate the work, it just means that it'll take longer before we are certain of what we know.
1 reply 1 retweet 9 likesShow this thread -
A lot of criticism circles around things like the fragility of our algorithms (e. repro), and this is well founded. But we should also be more self-aware -- why were we so invested in that one algorithm in the first place after just a single study?
1 reply 0 retweets 8 likesShow this thread -
Both types of reproducibility are necessary and highly intertwined, but IMO s. repro is the most important, and proper s. repro takes patience, care, and aggregate effort.
1 reply 0 retweets 9 likesShow this thread -
So, to conclude: 1) State your scientific claims in your papers! 2) Proper science guarantees that good ideas stand the test of time. Code sharing alone does not. 3) A single paper should often only contribute little to the truth value of a claim.
1 reply 5 retweets 44 likesShow this thread -
4) We should not be shocked when we can't replicate results. 5) Let's be a bit more patient before declaring that the field is undergoing a "crisis". 6) And finally, let's all read a bit more philosophy of science :)
5 replies 4 retweets 57 likesShow this thread -
Replying to @santoroAI
Cool thread. I talked about this at ICML a little. "In this talk I will discuss the different types of replicability/reproducibility, focussing heavily on the two that involve software/computational models."https://figshare.com/articles/Varieties_of_Reproducibility_in_Empirical_and_Computational_Domains/6818018 …
1 reply 2 retweets 6 likes -
Replying to @o_guest @santoroAI
"However useful it is to be able to re-run code from the past, it is often secondary to doing good science because checking that the spec is generally correct — i.e., the theory is actually computationally captured — is more important to science."
1 reply 1 retweet 5 likes -
Replying to @o_guest @santoroAI
"Implementation-only details, for example, might need to be upgraded to theory-level if they turn out to be imperative to modelling a certain effect."
1 reply 1 retweet 4 likes -
Replying to @o_guest @santoroAI
"And vice versa, theory-level assumptions could be relaxed if it is found that other important aspects of the theory are nonetheless captured with a variety of implementations."
2 replies 1 retweet 3 likes
More here in this journal article: Cooper, R. P. & Guest, O. (2014). Implementations are not specifications: specification, replication and experimentation in computational cognitive modeling. http://dx.doi.org/10.1016/j.cogsys.2013.05.001 …
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.