Conversation

Replying to
Roughly, as AI models are scaled up (data + compute + model params) they develop deeper, more general and abstract reasoning capabilities. The view among proponents of the scaling hypothesis is that these capabilities may (will) eventually include principles of logic.
1
Replying to
Ah, okay. Yeah, I've heard that argument, but I think it's kinda a leap of faith at this point. There's no clear reason to believe it other than that biological brains seem to be an existence proof if you squint enough.
1
2
Replying to
hmm I've tended to take it more seriously, given that it seems to explain the history of language modelling quite well. As model size, data, compute ↑, models went from writing words → phrases → sentences → paragraphs, to now being able to write blogs & do math (GPT+)
1
Replying to and
The math thing might be at the core of the question: GPT-3 can add small numbers (~3 digits) together pretty well but fails for larger ones So an interesting question (currently debated) is whether it's "learned math" or is just babbling really well
1
1
Replying to
Yes, but the key question for scaling is whether "it just learned math words as part of a language model" is any different from "it understands math words", or "it understands math". Pro-scalers would argue that the answer converges to "no" in the high-scale limit.
1
Replying to and
This is why I think disembodied AI is a dead end. You need to put the brain in a robot body and have it be in a feedback loop with the real world and develop a domain-entangled intelligence. Math is not word games. We *count things* with it as a starting point.
1
1
Show replies