Could anyone point me towards a resource on best practices for registering predictive models prior to testing on independent validation sets? Or for pre-registering work on existing (but not yet accessed) datasets more generally? (@kirstie_j @o_guest?)
Ah, no worries! If you're asking me, I don't really think prereg makes sense for my work and I don't think computational modelling has the same type of reproducibility issues (it has others). But I can totally see why it's needed for (some) empirical work like this case.
-
-
IMO, Imodelling has enormous reproducibility issues in terms of over fitting to datasets. Models that work well on imagenet may not generalize to other image sets. If pre-regging reduces over fitting, then it may help.
-
I don't disagree with the first part and by all means use any method. I don't think prereg can address this, but please try, of course! Correctly using the testing, training, and validation sets might help, I think. But even then that might not either. It's a big issue.
-
Just to be clear... Here you're using modelling to men machine learning, right? They don't use modelling that way and neither do I, so just making sure.
-
Yes, ML. Modelling is one of those terms with many meanings.
-
Yes, but I wasn't talking about ML, just so we're clear, in the side thread but about the modelling I do... Which is why I mistakenly assumed I was tagged. Then I realised you mean ML and mentioned what I did in my replies to you.
-
What I believe about my kind of modelling is slightly different to what I believe about ML modelling but there are overlaps which I've discussed at great length.
-
I totally agree.
End of conversation
New conversation -
-
-
The abstract here, below the slides, should explain my views.https://figshare.com/articles/Varieties_of_Reproducibility_in_Empirical_and_Computational_Domains/6818018 …
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.
