completely agree: remarkable ≠ semantically coherent. system has no idea that a unicorn has one horn, blithely asserts it has four, etc. simultaneously close and not close to Searle’s room, not at all close to genuine understanding.https://twitter.com/Grady_Booch/status/1096841495519232000 …
-
100% agree on the asymptote question. They give an encouraging indication w/r/t model size (see below), not data. But good news on the data is that since it's unsupervised, you just need to feed it more text. Going from 40GB --> 40 TB is just more web scraping/curating.pic.twitter.com/X0MjRlBsTt
I see a hint of diminishing returns there, actually (fewer percentage point increase for last doubling in model size). Will be interesting to see, certainly.