@citnaj Some students are going to "present" a bit about your work this Wednesday for my gen-ed ML course! Question: What does "unshock" refer to, e.g. in https://github.com/jantic/DeOldify/blob/master/ColorizeTraining.ipynb …?
-
That's awesome! "unshock" has been a workaround for the sensitivity the model has when switching sizes in training. The activations change overly-dramatically, and if you're not careful you can bust your weights and get weird artifacts as a result.
Easing into this with lower lr helps (hence "unshock"). With my latest model I've replaced this with a smooth incline on the learning rate basically starting near 0 and going to target from there. Seems to work well.
