While preterm birth is still the leading cause of death among young children, we noticed a large number (24!) of studies reporting near-perfect results on a public dataset when estimating the risk of preterm birth for a patient. (2/6)
-
-
Prikaži ovu nit
-
At first, we were unable to reproduce their results until we noticed that a large number of these studies had one thing in common: they used over-sampling to mitigate the imbalance in the data (more term than preterm cases). (3/6)
Prikaži ovu nit -
After discovering this, we were able to reproduce their results, but only when making a fundamental methodological flaw: applying over-sampling before partitioning data into training and testing set. (4/6)
Prikaži ovu nit -
In this work, we highlight why applying over-sampling before data partitioning results in overly optimistic results and reproduce the results of all studies we suspected of making that mistake. Moreover, we study the impact of over-sampling, when applied correctly. (5/6)
Prikaži ovu nit -
This work has been a significant combined effort of a great number of people:
@isa_belle_idh@fongenae@svhoecke@thomeestr@fdeturck@svhoecke@lusterck and more.@uzgent@imec_int@IDLabResearch If you have any questions, please get in touch with us! (6/6)Prikaži ovu nit
Kraj razgovora
Novi razgovor -
-
-
Hmmm what tools are they using? And why are they not using imbalanced-learn? (I assume they might not be using python but I'm sure R has similar tools). There is good reasons to use well-tested tools.
-
Well another commonality they all had is that they didn't provide the code (I wonder why...). Nevertheless, even with imbalanced-learn you can easily make that mistake: X, y = http://smote.fit _sample(X, y) for train_ix, test_ix in KFold().split(X, y): ....
- Još 6 drugih odgovora
Novi razgovor -
-
-
This is metascience, and there is a lot of ground to cover. All research should be reproducible! Great work!!
-
Thanks Dieter! Indeed, ideally all research is reproducible, but this is often hard when using sensitive medical data. But in this case, the dataset was public. Policy in these cases should be: no corresponding code = desk reject
- Još 5 drugih odgovora
Novi razgovor -
-
-
People oversample test set?

-
Yes... But indirectly by oversampling the entire dataset and then sampling a test set from it. It is quite shocking indeed!
Kraj razgovora
Novi razgovor -
Čini se da učitavanje traje već neko vrijeme.
Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.