Could anyone point me towards a resource on best practices for registering predictive models prior to testing on independent validation sets? Or for pre-registering work on existing (but not yet accessed) datasets more generally? (@kirstie_j @o_guest?)
-
-
Replying to @AgnesNorbury @kirstie_j
Computational models? Never seen any coherent position on this. I'm not convinced pre-registration and Computational modelling are compatible per se more like parallels.
@BArslan_CogSci@bradpwyble@IrisVanRooij3 replies 1 retweet 3 likes -
Replying to @o_guest @kirstie_j and
In this case, just wondering if it would be possible to fit a predictive model (e.g. symptom change from baseline imaging and clinical data), then register that somewhere before testing on independent data. Issue is both datasets already exists so hard to prove date of access...
1 reply 0 retweets 1 like -
Replying to @AgnesNorbury @kirstie_j and
A statistical model? Like a multiple regression?
1 reply 0 retweets 0 likes -
Replying to @o_guest @kirstie_j and
Yes, sorry, in this case likely some kind of regression (but also could be e.g. some kind of classification algorithm).
1 reply 0 retweets 1 like -
Replying to @AgnesNorbury @kirstie_j and
Ah, so this is surely not really what I'm an expert in at all. FWIW this should be exactly what standard prereg is — although not an expert myself!
1 reply 0 retweets 0 likes -
Replying to @o_guest @AgnesNorbury and
That "surely" is superfluous/a mistake.
1 reply 0 retweets 1 like -
Replying to @o_guest @kirstie_j and
Ah sorry, my fault for assuming you would probably know about anything model-related :-). Interesting to think about the computational case too though. Maybe this is covered under standard pre-reg under some kind of time-lock system.
3 replies 0 retweets 1 like -
Replying to @AgnesNorbury @kirstie_j and
Ah, no worries! If you're asking me, I don't really think prereg makes sense for my work and I don't think computational modelling has the same type of reproducibility issues (it has others). But I can totally see why it's needed for (some) empirical work like this case.
2 replies 0 retweets 0 likes
The abstract here, below the slides, should explain my views.https://figshare.com/articles/Varieties_of_Reproducibility_in_Empirical_and_Computational_Domains/6818018 …
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.