Jason Antic

1/ Random thought: Deep Learning model development has seemed to me to be a lot like cooking. Typical example: Images looking a little strange here? Train on this other loss to tone it down a bit. How much? Until it's done!

2/ I'm not claiming this is typical. I really don't know. I'm in my own little world. But I can tell you this: From day one of getting into deep learning I viewed the endeavor as incredibly complex. Tackling incredibly complex things successfully requires different tools.

3/ Here's the thing about high complexity: It's very easy to fool yourself into reaching wrong conclusions about cause/effect. In fact that's probably the norm. Why is that? Well a good part of that is that our working memory simply isn't equipped to deal many moving parts.

4/ You've probably experienced the severe difficulty of trying to recall more than 7-8 digits that you've just seen. Now imagine that you're trying to reason about a system with millions of parameters, many more connections, and innumerable interactions with the outside world.

5/ What's the answer to getting this complexity under control (somewhat)? Well good abstractions- ideally "lossless"- go a long way. Turning many things into one vastly reduces cognitive load and increases your chances of success.

6/ This is why people who say "chemistry is just physics" or "biology is just chemistry" are not just being annoyingly reductionist, but are also quite frankly wrong. These abstracted levels of study are absolutely necessary if we are to make sense of the complexity.