Jason Antic

Here's something I've found really useful in my DeOldify research: Keep a separate huge master set of images created from various sources (eg open images), then use Jupyter notebooks made specifically to generate training datasets from that master and output them elsewhere. 1/

In the notebooks I do a lot of filtering out of images from the master, depending on the task. You can filter out images for being grayscale when you're doing colorization, for example. Or filtering out for a min resolution, etc. I've found this helps data quality a lot. 2/

The bigger your master set is, the more aggressive filters you can apply and still get a big enough dataset at the end. This also gives you the freedom to use sloppy but good enough filtering techniques that have a lot of false positives such as blurry image detection. 3/

For the master you can use a huge and fairly slow/cheap drive as opposed to a fancy ssd/nvme. Save the speedy drive space for the resulting filtered and processed datasets. 4/

This sounds a lot like your own image data lake pattern. Raw, transformed, curated, published images. On cheap storage. How do you track the models, versions and what inputs / parameters affected the model? How are outcomes measured / what metrics would you review?

Model versions: Github! Inputs/Parameters/Outcomes: Well documented experiment Jupyter notebooks that I save in a separate backup drive to freeup space along with tensorboard outputs and model checkpoints. Point is I can reproduce with ease and not think too much about it.

Does it mean you add new code whenever you want to change parameters or you save stuff just when you find something interesting? Also, I guess you don't fix your seeds!?

I just try to make sure that experiments at any point of time can be 100% reproduced in terms of the functioning but not necessarily to the point where the exact data is used (fixed seeds). Benchmarks do use the same exact images though, so effectively the same outcome there.