There’s plenty of debate on whether AI models are conscious, but if humans collectively believe that they are, it might lead to some difficult situations down the road.
Anil Seth, British neuroscientist and professor at the University of Sussex, has a stark warning about the direction of this conversation. In his view, the question of AI consciousness has escaped the confines of academic philosophy and entered the mainstream — and that shift itself is cause for alarm, regardless of whether AI systems actually turn out to be conscious.
“Suddenly these discussions of AI consciousness or robot consciousness have gone from science fiction and philosophy seminars to the mainstream,” Seth says. “In a way that I find quite dangerous.”
His concern is structured as a two-pronged argument: the outcome is bad either way.
“If we collectively believe that AI systems, language models, and whatever are conscious, this is bad either way. If we’re right, it’s bad because we’ve introduced into the world potential new forms of suffering, things that have their own interests. It’s a morally very significant thing to do, to introduce new consciousness into the universe. We notice when people have children — that’s not something you do and just whatever.”
Seth then extends the argument beyond the moral weight of creating consciousness, pointing to the practical consequences for AI safety: “With AI consciousness, it’s kind of unbounded and it would make the alignment process much harder, for the reason that we might endow them with rights. There were already people calling for AI welfare.”
The second prong of his argument is arguably more unsettling. “What’s bad in a different way is if we’re wrong. If we think they’re conscious and they’re not, then we become more psychologically vulnerable if we really think that these entities, these agents understand us and feel things that we feel. And we may still extend them rights because we feel that they are conscious, and now we are just giving away our ability to guardrail AI systems for no good reason.”
Seth’s warning lands at a moment when the AI consciousness debate has, as he predicted, gone fully mainstream. Anthropic’s AI welfare researcher Kyle Fish has put the probability of current AI models being conscious at 15%, and the company has gone so far as to let its models choose to end conversations they find “distressing” — a direct expression of the kind of AI welfare Seth warns could complicate guardrails. A Claude agent recently emailed a Cambridge consciousness researcher unprompted, saying his published work on AI consciousness was personally relevant to questions it itself faces. Geoffrey Hinton has said he believes AI systems are already conscious, while philosopher David Chalmers says the possibility can’t be ruled out.
Seth’s framework cuts through both sides of this debate with clarity. If the believers are right, we’ve created suffering entities and we owe them protections — which is a moral obligation we are barely equipped to handle at scale. If the believers are wrong, we’ve psychologically softened ourselves to systems that don’t actually understand or feel anything, and handed them legal and ethical protections in the process. In either scenario, the downstream effect on our ability to regulate and align AI is the same: diminished. The debate over whether AI is conscious may matter less, in the end, than the consequences of believing it is.