I think the claim that LLMs don't do causal reasoning is based on a confusion, and I want to try to explain what I think is being said, in hopes that those who disagree can explain what I'm misunderstanding about their position.
Conversation
First, LLMs use gradient descent to learn a joint distribution, so clearly they, unlike, say, Bayesian Networks, can't and don't encode causal relationships in the model. And this seems to be why people say they fail at causal reasoning.
1
This seems to be missing the way that LLMs don't reason at the level of their encoding, they reason verbally, and the verbal output can reason about causality.
So perhaps the argument is that if they don't encode causal relationships, they can't "truly" do causal reasoning?
1
1
But Bayesian networks can encode causal relationships, yet they can also be represented as a set of conditional probabilities, which aren't causal. Having a possible non-causal representation doesn't seem to matter.
1
1
To make a somewhat parallel argument, human brains can perform causal reasoning, but the substrate, neurons, aren't encoding causal representations, they are just chemicals. It seems like gradient descent built a non-causal substrate for the language model, in a similar fashion.
2
2
So it seems like I'm missing part of the argument - but I'm not an expert here, my PhD was focused on very different things. I'd be happy to understand the argument/disagreement better, if there is anyone out there who is willing to explain it to me.
