Goodfire's neuron interpretability thesis faces a superposition objection

Goodfire researcher Nick Cammarata argues that researchers have been too quick to treat a neuron with several apparent meanings as evidence of an inscrutable neural network. In an October 15 essay, he revisits examples from earlier interpretability work, including his own, and proposes a different explanation for many of them. He calls the essay an informal account of a working hypothesis. A fuller technical paper on the language-model experiments is still forthcoming.

The big change

• What changed: Cammarata offers a way to study some model behavior through the network's own neurons and circuits. His reinterpretation of earlier examples and preliminary language-model audit challenge one explanation for why neurons seem to mix meanings; they do not establish that the whole network is understood.
• Why it matters: Researchers investigating a model's answer could use neuron-level traces without first training a separate model to extract features. That would make direct investigation more practical where the traces are reliable, but a readable trace can still omit important computation.
• What to watch: The promised technical paper needs to show how the language-model audit was done and how its explanations hold up, especially for weak activations. That evidence will help determine where studying native neurons is useful and where other methods remain necessary.

Why a neuron can appear to mean several things

Mechanistic interpretability tries to explain a model's behavior by examining the computations inside it. A neuron that responds to apparently unrelated inputs is called polysemantic. One influential explanation, superposition, is that a network packs more features into its available dimensions than a one-feature-per-neuron picture would allow. On that account, a single neuron may participate in several unrelated features, and researchers may need to separate those features before they can understand the circuit.

Cammarata challenges a narrower version of that account: the idea that apparently unrelated jobs are commonly packed into the same neurons because they do not interfere. His essay does not settle every use of “superposition,” including how information can be represented across the model's residual stream.

His clearest example comes from InceptionV1, an image classifier studied in early interpretability research. A neuron that appeared to respond to cat faces and car fronts became a familiar example of polysemanticity. Cammarata now argues that the shared signal is a pointed, V-shaped pattern. He points to its inputs, later connections, and response to pointed versus round forms. In another early circuit that seemed to hide a car feature among dog neurons, he argues that some of the apparent dog features reflect shared pose and misleading feature visualizations. These are Cammarata's reanalyses of specific circuits, including work he previously helped publish, rather than a general proof that unrelated features never share neurons.

He calls the broader alternative recursive generalization. As a neuron learns, its job can branch into nearby meanings, then branch again. One language-model example in the essay starts with the letter sequence “iy,” extends to “diy,” and then to “do it yourself.” The final phrase no longer contains the original letters, yet the proposed path of learning gives the apparent mixture a history that a researcher can investigate. The essay says the authors have watched such branching in training runs, while acknowledging they have not rigorously shown that every heavily polysemantic neuron forms this way.

What the language-model evidence shows so far

Cammarata reports preliminary audits of Qwen3-32B in which roughly 90% of neuron poles had one main job, often with related offshoots; about 7% had two related jobs; and roughly 2–4% remained unexplained. A pole is the positive or negative side of a neuron's response, so those figures should not be read as a completed census of whole neurons. The essay says the estimates change with thresholds for weak activations, circuit depth, and the judgment of what counts as one job. Cammarata also presents interactive neuron and circuit examples, including interventions that change a model's answer. They illustrate his investigative approach, but the essay is not a peer-reviewed, independently replicated measurement of interpretability across language models.

The distinction between a useful trace and a complete explanation matters. Neel Nanda, a Google DeepMind interpretability researcher, wrote in a public response that neuron-based attribution graphs could help answer practical questions about why a model behaved a certain way. He also expects such graphs to miss substantial computation, as other tracing approaches can. He does not regard the essay as substantial progress toward fully reverse-engineering models or solving interpretability.

Nanda's technical objection is more specific than a rejection of neurons as a research tool. He argues that superposition remains plausible, particularly for information represented across the residual stream and for computations involving many neurons. He suspects weak neuron activations in language models may carry meaningful information that the essay has not adequately accounted for. Cammarata argues in a separate section that weak activations can often be removed with limited cost and that some of their effects reflect fine distinctions. His numerical ablation example there is from a vision model. The disagreement over how far that evidence transfers to language models remains open.

For researchers and safety practitioners, the immediate proposition is methodological: studying the original network's neurons alongside their inputs, outputs and interventions may reveal more than a neuron label or an activation gallery alone. The promised companion paper is a chance to examine the language-model methods and test the limits of that proposition. Neither the essay nor Nanda's response demonstrates that these explanations make AI systems safer.

Sources & further reading

• Nick Cammarata, “Neural Networks Are Already Clean” (Goodfire, October 15, 2026). The author's informal research essay and working hypothesis supplies the circuit reanalyses, recursive-generalization account and preliminary Qwen3-32B estimates. Its own notes describe threshold sensitivity and a forthcoming technical companion paper.
• Neel Nanda's response to the essay (LessWrong discussion, accessed October 19, 2026). Nanda sees practical value in neuron-based investigation while disputing claims of major progress toward complete interpretability and arguing that superposition and weak activations remain important.