A widely shared essay is arguing that neural networks are far easier to understand than the field believed, overturning a theory its own author helped popularize
A new essay from Nick Cammarata, a researcher at the AI interpretability lab Goodfire, is making waves in AI research circles this week — and it’s doing so by walking back one of the field’s own foundational ideas.
The essay, titled “Neural Networks Are Already Clean,” argues that a years-old theory about why neural networks are hard to understand was largely mistaken, and that modern AI models may be dramatically more interpretable than researchers have assumed. For a field that has spent years building increasingly elaborate workarounds to deal with supposedly “tangled” neural networks, that’s a significant claim.
Why “interpretability” matters
Mechanistic interpretability is the effort to understand what’s actually happening inside an AI model — rather than just observing its inputs and outputs, researchers try to trace the actual internal computations, down to the level of individual “neurons,” the basic units that make up a neural network. The stakes are high: understanding how a model actually “thinks” is considered a key building block for making advanced AI systems safer, more predictable, and more trustworthy.
For years, that effort ran into a frustrating wall. Individual neurons often seemed to respond to strange, unrelated combinations of things. The most cited example in the field is a neuron from an early image-recognition model, InceptionV1, that appeared to fire for both cat faces and the fronts of cars — two concepts with nothing obviously in common.
The theory that shaped the field
The dominant explanation for this phenomenon was a concept called “superposition.” The idea: when a neural network has more concepts to represent than it has neurons available, it starts packing multiple, unrelated concepts into the same neuron — on the logic that if two concepts rarely show up in the same place at the same time, cramming them together doesn’t cause much interference. Researchers compared it to a knot that had to be mathematically “unfolded” before it could be studied, typically using a secondary model called a sparse autoencoder trained to approximate the network’s internal structure in a cleaner form.
If superposition was really how neural networks worked, it meant researchers couldn’t just study a model’s neurons directly — they would always need to use these indirect decomposition techniques first, adding a significant layer of complexity and cost to interpretability work.
Notably, Cammarata was one of the researchers who helped popularize this very theory. He co-discovered the original “cat and car” circuit in 2019 alongside Chris Olah, and co-authored some of the field’s most cited early papers on the subject, including “Zoom In: An Introduction to Circuits.”
The reversal
In his new essay, Cammarata says that after months of re-examining neurons in both image and language models, he came to believe the field misread its own canonical evidence. Going back to that famous cat-and-car neuron, he found it wasn’t an arbitrary mixture at all — it was detecting a shared visual pattern, a V-shape, that shows up in car headlights, cat whiskers, flames, and fox faces alike. What looked like two unrelated concepts crammed into one neuron was actually one coherent concept the earlier research had simply mislabeled.
Cammarata argues that most of what has been written off as “polysemanticity” — neurons that seem to do several unrelated jobs — is better explained by something he calls recursive generalization: a neuron’s meaning branches outward over training, the way a tree sprouts sub-branches, rather than several unrelated concepts being forced to share a single neuron. As an example, he describes a neuron that starts out detecting the text pattern “iy,” which over training comes to also detect “diy,” and eventually “do it yourself” — a phrase that no longer contains the original pattern at all. Traced step by step, each branch makes sense; looked at only in its final form, it can look like noise.
He also attributes much of the field’s earlier pessimism to weak tooling and flawed model architectures rather than any fundamental property of neural networks. The InceptionV1 model that produced the field’s most-cited examples of “uninterpretable” neurons was, by his account, hamstrung by architectural issues that caused its deeper layers to perform poorly — issues that newer architectural techniques (like residual connections, now standard in modern models) fix as a side effect. Retrained with modern architecture and tools, Cammarata reports that 99-100% of neurons in that same late layer are now easily interpretable, and that similar legibility holds up in large modern language models, including internal audits of models with tens of billions of parameters, with preliminary results extending to trillion-parameter-scale models.
Why it matters for the industry
If Cammarata’s hypothesis holds up, the implications are significant for AI safety and transparency work industry-wide. Rather than relying on expensive, indirect decomposition techniques to make sense of a model’s internals, researchers might be able to study a network’s own neurons and circuits directly — a faster, more scalable path to understanding how AI systems actually make decisions.
That would be good news for an AI industry under growing pressure to explain the behavior of increasingly powerful and opaque systems, from regulators, enterprise customers, and safety researchers alike.
A few reasons for caution
Cammarata is upfront that the essay is an informal “working hypothesis,” not a peer-reviewed paper — he describes it as a 65-minute read laying out a research direction, with a more rigorous technical companion paper still to come. It’s also worth noting that he is effectively revising a theory he himself helped build, which cuts both ways: he has unusually deep familiarity with the original evidence, but the claim will likely need independent replication before the broader interpretability community treats it as settled.
Goodfire itself is a well-capitalized player in this space. The San Francisco-based public benefit corporation, founded by alumni of OpenAI’s and Google DeepMind’s interpretability teams, has raised roughly $209 million since 2024, including backing from Anthropic CEO Dario Amodei, who has called mechanistic interpretability “among the best bets” for turning AI systems from black boxes into something understandable and steerable.
Whether or not “Neural Networks Are Already Clean” ends up reshaping the field’s consensus, it’s already generating significant discussion among AI researchers — not least because of who’s making the case, and what theory they’re asking everyone to reconsider.