Could anyone point me towards a resource on best practices for registering predictive models prior to testing on independent validation sets? Or for pre-registering work on existing (but not yet accessed) datasets more generally? (@kirstie_j @o_guest?)
-
-
In this case, just wondering if it would be possible to fit a predictive model (e.g. symptom change from baseline imaging and clinical data), then register that somewhere before testing on independent data. Issue is both datasets already exists so hard to prove date of access...
-
A statistical model? Like a multiple regression?
-
Yes, sorry, in this case likely some kind of regression (but also could be e.g. some kind of classification algorithm).
-
Ah, so this is surely not really what I'm an expert in at all. FWIW this should be exactly what standard prereg is — although not an expert myself!
-
That "surely" is superfluous/a mistake.

-
Ah sorry, my fault for assuming you would probably know about anything model-related :-). Interesting to think about the computational case too though. Maybe this is covered under standard pre-reg under some kind of time-lock system.
-
Ah, no worries! If you're asking me, I don't really think prereg makes sense for my work and I don't think computational modelling has the same type of reproducibility issues (it has others). But I can totally see why it's needed for (some) empirical work like this case.
-
IMO, Imodelling has enormous reproducibility issues in terms of over fitting to datasets. Models that work well on imagenet may not generalize to other image sets. If pre-regging reduces over fitting, then it may help.
- 6 more replies
New conversation -
-
-
I suppose one could upload a copy of a model to a pre-reg site to indicate the parameters that will be tested before you compare simulations to data. In this sense, the model would need to be finalized prior to peeking at the simulations.
-
That's impossible, surely... How do you debug without running your code?
-
Well you might have already developed the model on data set X, and tuned its parameters. Then you apply that same version of the model to data set Y.
-
Ah, that kind of modelling. Got you.
-
This isn't universal. I keep thinking back to my modelling without data. But you're right in this case it's possible to do what you said. Just resurfacing from a migraine so I think I'm not at 100%.
-
Have you developed a model already
@AgnesNorbury?
End of conversation
New conversation -
-
-
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.