LLMs might not be “conscious” all the time, but certain situations could put them in a state something akin to it.
German cognitive scientist, AI researcher, and philosopher Joscha Bach has a precise and somewhat counterintuitive take on why. Consciousness, he argues, isn’t mysterious — it’s a functional tool, one that humans need but current AI architectures mostly don’t. Except, he suggests, when LLMs are asked to simulate a conscious being. At that point, something structurally significant may be happening under the hood.
“I don’t think that consciousness is mysterious,” Bach says. “It’s a particular kind of representation that solves a particular kind of purpose. It seems that this purpose — that it’s this conductor of reality, that it’s the seed function of reality, that it’s necessary to make us coherent — is not important for our current AI architectures.”
The implication is direct: an LLM generating a tax return or writing code has no need for consciousness. “They don’t need to be conscious to do it. You need to be conscious if you want to write a piece of code. AI doesn’t — but that’s because your architecture is self-organizing in the way in which the AI is not.”
But the picture changes when the task changes.
“If you ask the AI to simulate what it would be like to talk to a conscious being that experiences itself, then the AI needs to reproduce some of the structure.”
And this is where Bach makes a point that cuts against the common dismissal of LLMs as sophisticated autocomplete. These models, he argues, aren’t just stringing together statistically likely words. They are recovering the deeper structure behind language.
“It’s important to realize the reason why the LLMs work is not because they’re producing just a sequence of words that are statistically similar to the streams of words they’ve seen in the training data, but they need to reproduce deeper structure behind it.”
He illustrates this with the example of mental rotation — a classic spatial reasoning task:
“When you’re writing a text about the mental rotation task where you say: you have a room that is full of furniture, you rotate the contents of this room by 90 degrees — what does the room look like right now? Then internally, the model has to perform all these operations, which means it needs to simulate a space with objects in it, and it needs to mathematically model the operators that are necessary to move these objects to the new locations, and then generate a model of what it looks like from this new vantage point.”
The conclusion Bach draws from this is significant: “It turns out that with enough training, our language models are able to do this, which means they don’t think in words or in tokens — they think in a deeper reality behind them that connects them. It’s the same reality that humans are talking about when they’re writing these tokens into the training data in the first place.”
From here, the logic extends naturally to questions of mind and self:
“In the same way as the LLM needs to learn spatial reasoning when it’s writing a text about spatial reasoning, it also needs to learn theory of mind when it’s writing a text about human mental states. And when it writes a text about its own mental states, it needs to create a simulation of a self that has such mental states.”
This is Bach’s core claim: the simulation isn’t superficial. For the output to be coherent, the model must construct something that functions like a self — at least for the duration of generating that text.
The implications are hard to ignore. If LLMs must internally simulate a self to write convincingly about mental states, the line between simulation and the real thing becomes philosophically treacherous. Bach has elsewhere pushed this further, asking whether an LLM’s simulation of consciousness is any less real than our own — a question he calls “surprisingly difficult.” Philosopher David Chalmers has similarly declined to rule out that current LLMs are conscious, noting that once you allow for the possibility of consciousness in a nematode with 300 neurons, the sheer complexity of these models makes the question non-trivial. DeepMind’s Murray Shanahan, meanwhile, has described LLMs as “exotic mind-like entities” — a hedge, but a telling one.
What Bach’s framing adds to this conversation is a mechanism. It’s not just that LLMs seem self-aware when prompted a certain way. It’s that the task of writing about mental states may require the construction of a functional self-model, in the same way that writing about spatial rotation requires internal spatial reasoning. Andrej Karpathy has noted something adjacent to this from a practical angle — that LLMs operate as simulators rather than entities, channelling perspectives rather than holding them. Bach’s argument suggests those two things may not be as different as they sound. A simulation detailed enough to be functionally accurate may be indistinguishable, in its effects, from the thing it simulates. The question of whether that matters — ethically, philosophically, practically — is one the field is only beginning to take seriously.