Azhir Mahmood  🚀 🤖 🧬 🧠 🪐

Do you mean within a lifetime or across generations? Obviously true for the latter, less clear for the former.

within a lifetime. Would not be contrarian otherwise I think ;)

I agree that we don't know given current methods if this is true or not. I'm inclined to believe that having too many dimensions of plasticity can impede learning so there is perhaps some virtue in having some more rigid parameters, esp those that allow for homeostasis. >

but isn't the main thing we learn from DL that giving more parameters tends to always help?

Only if the training set is able to scale up too right? Given a fixed amount of experience, it's not clear that more parameters is always better.

you can view large networks as ensembles of smaller networks. So, weirdly, DL systems often have pretty surprisingly good performance at low N.

Does the increased number of parameters always help if the input data remains constant? Wouldn’t increased parameterisation eventually have marginal gains and thus eventually be energetically unfavourable?