Everything about this tweet makes me happy. Ilya can’t help but be an artist in everything he does :)
Quote Tweet
I asked @ilyasut how to set neural network init. He accidentally replied with a poem:
You want to be on the edge of chaos
Too small, and the init will be too stable, with vanishing gradients
Too large, and you'll be unstable, due to exploding gradients
You want to be on the edge