Nick Cammarata, researcher at Goodfire, argues that neural networks are simpler than previously thought, reinterpreting key examples like the cat-car neuron as shape detectors, not polysemantic units.
He challenges the long-held belief in superposition, suggesting that neurons do not arbitrarily pack unrelated concepts. Cammarata's work, detailed in an essay titled 'Neural Networks Are Already Clean,' reevaluates the foundational theories of mechanistic interpretability.
Cammarata's analysis suggests that the famous cat-car neuron in InceptionV1 is actually a V-shape detector, responding to pointed shapes like fox snouts and flames, while being activated negatively by round shapes like the letter O. This challenges the notion of polysemanticity, where neurons represent multiple unrelated concepts.
His reinterpretation extends to the 'Zoom In' superposition circuit, which he now sees as ordinary generalization rather than an example of superposition.
The essay also highlights the role of outdated architecture and weak analysis tools in complicating neural network interpretation. Cammarata suggests that better analysis methods could reveal more about how neurons respond.
He introduces the concept of 'recursive generalization,' where neurons evolve to respond to broader categories, such as moving from 'iy' to 'diy' and eventually to 'do it yourself.' This could offer insights into how networks process information.
In language models, Cammarata cites preliminary results showing that about 90% of neuron poles have one main function, with 7% having two related roles. Only 2 to 4% remain unexplained, according to Goodfire.
He argues that large-scale investigation, aided by AI agents, outperforms complex decomposition methods. This approach, rooted in direct neuron study, reflects a return to the field's original methods with modern tools.
Cammarata's findings could have significant implications for AI safety, as understanding neural networks better might help detect deception or dangerous behavior.
OpenAI's Roon praised the work, calling it a historic moment that could shift alignment from a speculative field to a more grounded science.
However, others like Neel Nanda remain skeptical, arguing that superposition is still a valid explanation for how models store information.
Source: thedecoder
