LLMs have made some astonishing progress in recent times — OpenAI has managed to solve 10 previously unsolved math problems in one go, while Anthropic has announced major cybersecurity breakthroughs. But while this has been widely celebrated by the tech community, longtime detractors of the LLM paradigm have now chosen to use a new term to explain what they were really talking about.
Gary Marcus and former Meta Chief Scientist Yann LeCun have said that they had been talking of a “pure LLM” when they had been talking about the limitations of the technology. “This is likely NOT a pure LLM. my understanding is that pure LLMs still can’t do basic math consistently,” Gary Marcus said when Astra, OpenAI’s yet-unreleased model solved 10 previously-unsolved math problems.
He added that he believed that Astra was a “neurosymbolic LLM with harnesses and tools”, which meant it wasn’t a “pure LLM”. “I imagine that Astra is a neurosymbolic LLM with harnesses and tools, which is consistent with that i have said from day one. i seriously doubt that it is a pure LLM,” he said.
Yann LeCun is now using the same term as well. When an X user pointed out how Yann LeCun had previously talked about how LLMs couldn’t finish long-running code tasks, LeCun said that he had never meant LLMs in the first place. “You obviously did not understand my statement. I was talking about auto-regressive token prediction, which is what pure LLMs do. But good code generation systems aren’t pure LLMs and aren’t doing mere auto-regressive token prediction,” he said.
Both Marcus and LeCun have been longtime critics of the LLM paradigm. LeCun has repeatedly said that LLMs couldn’t get humanity to AGI, and has even launched a new company — AMI Labs — to build AI using a new world models approach. LeCun had argued that given how LLMs were probabilistic, the probabilities would compound over long-horizon tasks, leading to extremely low success rates.
But their use of the term ‘pure LLM’ hasn’t gone down well with most of X. “”pure LLM” is a term invented by losers who want to distract you from the fact that they were, in fact, wrong,” an X user wrote.
“Repeatedly failing to grasp that imperfect, unreliable components can be made useful and reliable as parts of larger systems _while still unreliable_ disregards like 70 years of information technology history and is disqualifying,” wrote another X user. “By this I’m referring to the earlier positions both of these two held for many years, before retreating to the “pure LLM” cope. “Pure LLM” is the mark of a failed critic,” they added.
Others began trolling the duo. “Yann watching a Roomba clean an entire house: “You obviously misunderstood me. I said brooms can’t navigate rooms. A Roomba isn’t a pure broom,” said another poster.
And some memes began circulating as well.
Whether LLMs will ultimately lead to the creation of superintelligence isn’t yet a settled question — they could still hit sudden roadblocks that prevent future progress, and their abilities could plateau. But LLMs have made some remarkable progress in recent times, especially in math and coding, which arguably hadn’t quite been predicted by their detractors. It remains to be seen how far LLMs can go in terms of their abilities, but most people would agree that they hadn’t expected so much progress so quickly — pure LLM or not.