How to train your TensorFlow or PyTorch RecSys models 400x faster?
Use Merlin Dataloaders 😄
Two new examples just got merged! 🥳
✅ TensorFlow: github.com/NVIDIA-Merlin/
✅ PyTorch: github.com/NVIDIA-Merlin/
Conversation
Replying to
Unfortunately, the most up-to-date version of merlin-dataloaders requires Python > 3.7, and that is the version of Python available on Kaggle and colab.
You can still use it if you pin the version to merlin-dataloaders==0.0.2 though 🙂
1
2
Show replies
Replying to
You probably want to use DALI with images, video, and audio, and Merlin Dataloader with tabular data.
For instance, for loading images, DALI would likely be your best bet, but once you have image embeddings, Merlin Dataloader would be the way to go 🙂
1
4
Show replies
Replying to
You should see speed-ups at any size, in particular with larger batch sizes.
Generally, things don't only run faster, but you also have to write less code, so Merlin Dataloader will most likely be my goto solution for anything involving tabular data 🙂
2
Replying to
It seems like merlin-dataloader uses CPU when cudf isn't installed. I struggle to install cudf on my local machine so is it still faster than regular data loaders in that case?
1
2
Replying to
Yes, I believe so.
IIRC in the environment where I ran this I didn't have cudf installed
Quote Tweet
Merlin Dataloader is 119x faster than my own PyTorch Dataset + Dataloader combo!
This is revolutionary for tabular data 
Let's take a closer look at what is going on.
Let's take a closer look at what is going on.Show this thread
Replying to
Can this load from remote (eg S3) in the background? That’s something I appreciate from the WebDataset loaders