In standard transformers, most neurons respond to multiple unrelated concepts (polysemanticity), and single-concept neurons are rare exceptions rather than the norm.

Both neural network representations and psychotherapy outcomes resist decomposition into independent factors, suggesting a shared structural property of complex adaptive systems.

The number and quality of interpretable features extracted by sparse autoencoders increases predictably as model size grows, suggesting feature-level interpretability is not an artifact of small-scale experiments.

While SAE-based feature analysis identifies what a model represents, circuit-level analysis reveals how computations flow, providing causal evidence about model behavior that feature-level analysis alone cannot establish.

Models prefer polysemantic neurons even when monosemantic solutions are available, because polysemantic representations achieve lower cross-entropy loss.

Rather than decomposing models into interpretable parts, representation engineering identifies and manipulates high-level concepts directly in activation space, offering control without requiring full mechanistic understanding.

Sparse autoencoders successfully identify local structure but cannot guarantee a global coordinate system for the full representation space.