Tom Brown

(2/4) Most ML people deal with silent errors and slow feedback loops via the "ratchet" approach: 1) Start with known working model 2) Record learning curves on small task (~1min to train) 3) Make a tiny code change 4) Inspect curves 5) Run full training after ~5 tiny changes

(4/4) Within the ratchet approach, I want more tools and best practices for making feedback loops shorter and for making errors louder. Below is a short list of development speed hacks that I have found useful.

ML dev speed hack #0 - Overfit a single batch - Before doing anything else, verify that your model can memorize the labels for a single batch and quickly bring the loss to zero - This is fast to run, and if the model can't do this, then you know it is broken

ML dev speed hack #1 - PyTorch over TF - Time to first step is faster b/c no static graph compilation - Easier to get loud errors via assertions within the code - Easier to drop into debugger and inspect tensors (TF2.0 may solve some of these problems but is still raw)

ML dev speed hack #2 - Assert tensor shapes - Wrong shapes due to silent broadcasting or reduction is an extreme hot spot for silent errors, asserting on shapes (in torch or TF) makes them loud - If you're ever tempted to write shapes in a comment, make an assert instead

ML dev speed hack #3 - Add ML test to CI - If more than one entrypoint or more than one person working on the codebase, then add a test that runs for N steps and then checks loss - If you only have one person and entrypoint then an ML test in CI is probably overkill

ML dev speed hack #4 - Use ipdb.set_trace() - It's hard to make an ML job take less than 10 seconds to start, which is too slow to maintain flow - Using the ipdb workflow lets you zero in on a bug and play with tensors with a fast feedback loop