Will Manidis

this was a joke but this is also true what deepmind here took very little compute compared to what is commonly being done with text and from what I can tell nothing new on the architecture side all these tools were around, it "just" took good engineeringhttps://twitter.com/alth0u/status/1333498820508807169 …

yup, repurpose self attention transformer to do graph learning and profit none of these are new methods

all of the low hanging fruit left in applied ML is safely hanging in the trees behind a shield of difficult engineering, data set gen, and tuning self supervision

haha maybe so if you prune down the space to "applied ML" but i think there are lots of theory advances left in Deep RL, novel architectures etc some of these will become clearer on the next compounding of compute power

i continue to believe that the interplay between cognitive neuroscience and deep learning will be one of the great flywheels of 21st century science and i'm not sure that we're close to the end