In summary: set an appropriate `class_weight`, and use very large batches so that each batch contains at least a few positive samples. Make sure to monitor precision and recall during training.
Show this thread
How to train and evaluate a deep learning model for very imbalanced classification (e.g. classification with 99.82% of negatives and 0.18% of positives)? Here's an example using the Kaggle credit card fraud dataset.https://colab.research.google.com/drive/1xL2jSdY-MGlN60gGuSH_L30P7kxxwUfM …
-
-
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
The main idea of dealing with a highly imbalanced class dataset is class weights as mentioned in the given notebook.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
I thought keras handles this (imbalance data) automatically
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Thanks for sharing. Curious why dividing both train and valiadation sets by the mean and std of train set?
-
To have mean=0 std=. It's exactly the same that sckit does with standard scaler with its default values.https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler …
End of conversation
New conversation -
-
-
How worries should you be about how volatile the validation loss or with such imbalanced data is the loss not the appropriate metric and false/positive etc is?
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
What’s the standard solution for dealing with class imbalance for multiclass-multilabel problems? Simply using the class_weight parameter does not work, if I remember correctly. Or am I mistaken?
-
You need imblearn library and over or undersample the class
- Show replies
New conversation -
-
-
Thanks for the share!
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Thanks for sharing. What are the different ways to calculate class weight?
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.