Right. I see the similarity in that both are pretty good tools to lower/prevent overfitting. Would that be fair to say? I'm definitely interested in reading more; crossval is heavily under-taught and underutilized in my fields.
-
-
Replying to @Research_Tim @the_Sage_BB and
How does prereg prevent overfitting in a concrete way? Crossval does it in a very objective way, if used correctly.
1 reply 0 retweets 1 like -
Replying to @o_guest @the_Sage_BB and
Because, when used correctly, prereg prevents or at least reduces analytical choices that are data-dependent. You can't just run multiple analyses until you find one "that works". I don't think prereg prevents all causes of overfitting but it makes it a lot less likely.
1 reply 0 retweets 1 like -
Replying to @Research_Tim @the_Sage_BB and
That's not really enough (too much of an underspecified procedure) though for modeling. We need formal ways of doing it for a model. Which is why we do this: https://en.m.wikipedia.org/wiki/Training,_validation,_and_test_sets …
1 reply 0 retweets 2 likes -
Replying to @o_guest @Research_Tim and
Formal systems need formal evaluations.
1 reply 0 retweets 2 likes -
Replying to @o_guest @Research_Tim and
To be clear: All data gets a chance to be in all of the three sets.
2 replies 0 retweets 1 like -
Replying to @o_guest @the_Sage_BB and
Agreed! I don't want to argue that prereg can replace crossval, is better than it, or does exactly the same thing. There are some similarities in their utility though. Both are great tools given the right circumstances and should be used more often, when appropriate.
1 reply 0 retweets 1 like -
Replying to @Research_Tim @the_Sage_BB and
Modelling though is not the same as analysing data in a deep way. So prereg in the cases I can think of is not able to help with avoiding overfit.
1 reply 0 retweets 1 like -
-
Replying to @Research_Tim @the_Sage_BB and
I'm glad to see quite a few people like you who realise we're not trying to stop you overhauling the method in your (sub*)fields. We're all for open science. We just want to do it right in our (sub*)fields too.
1 reply 0 retweets 3 likes
That's a Kleene star, just in case. 
-
-
Replying to @o_guest @Research_Tim and
As soon as you want to test your model to real data then making sure it's independent data is important or overfitting becomes an issue. (cross- / leave-one-out) validation and such should help do what preregistration does for empirical studies. Agreed!
0 replies 0 retweets 0 likesThanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.