Notes on Toy Models of Superposition

Notes on Toy Models of Superposition (Elhage & others, 2022) and its follow-up Superposition, Memorization, and Double Descent (Henighan & others, 2023), from Anthropic’s Transformer Circuits thread.

The first paper constructs a setting in which the ground-truth features are known, then watches a neural network pack more of those features into its hidden layer than the layer has dimensions. The construction is small enough that polysemanticity can be explained exactly rather than inferred from examples that make a neuron activate.

Features are not coordinates

The paper treats a feature as a property of the input that the model represents. If the representation is linear, each feature corresponds to a direction in activation space. A neuron is one coordinate axis in that space. Those are only the same thing when a feature direction happens to line up with an axis.

Whether the axes are worth inspecting depends on the architecture. The residual stream has no privileged basis: rotating the representation and applying the inverse rotation to the next weight matrix leaves the model unchanged. An element-wise activation function breaks that rotational symmetry. The individual coordinates of a transformer MLP therefore form a privileged basis, which gives features some reason to align with neurons, though it does not force them to. The two sides of the next diagram (Fig. 1) isolate this difference: an arbitrary rotation preserves the computation on the left, whereas the coordinate-wise nonlinearity on the right makes the axes consequential.

A non-privileged basis where features can point in any direction and a privileged basis where a nonlinearity makes the coordinate axes special — **Figure 1:** A change of basis has no effect in a non-privileged representation such as the residual stream. Element-wise nonlinearities make the MLP basis privileged, so its coordinate directions can meaningfully be called neurons. Figure from the paper.

Superposition occurs when the model represents more features than there are dimensions available by assigning the features to non-orthogonal directions. High-dimensional spaces contain many directions that are nearly orthogonal, and sparse features rarely need to be active at the same time. Together, those facts make an overcomplete representation possible. The projection sketch (Fig. 2) pictures the hypothesis literally: start with a larger space containing one axis per feature, then compress it into the smaller activation space we observe.

A hypothetical disentangled network with one neuron per feature projected into a smaller observed network whose neurons are polysemantic — **Figure 2:** The superposition hypothesis pictures a smaller network as a lossy projection of a larger, sparse network with one direction per feature. Figure from the paper.

Viewed in the smaller network’s coordinates, the same projection produces the many-to-many pattern in the following diagram (Fig. 3). Several feature directions cross each coordinate axis, so the neurons appear polysemantic even when the features themselves are well-defined directions.

Several feature directions crossing a smaller set of neuron axes, causing each neuron to respond to multiple features — **Figure 3:** Polysemanticity is the coordinate-level appearance of features that do not align with individual neurons. Figure from the paper.

The toy autoencoder

The model starts with an input $x \in \mathbb{R}^n$ . Each coordinate $x_i$ is a synthetic feature. It is zero with probability $S_i$ and otherwise sampled uniformly from $[0,1]$ , so larger $S_i$ means a sparser feature. Each feature also has an importance $I_i$ , which controls how costly it is to reconstruct badly.

The model compresses $x$ into $m$ hidden dimensions, with $m<n$ :

h = Wx, \qquad \hat{x} = \operatorname{ReLU}(W^T h + b) = \operatorname{ReLU}(W^T W x + b).

It minimizes importance-weighted reconstruction error,

\mathcal{L} = \mathbb{E}_x \left[ \sum_i I_i (x_i-\hat{x}_i)^2 \right].

The tied weights make the representation easy to read. Column $W_i$ is the direction used to encode feature $i$ , while $W_i^T h$ reads that feature back out. The diagonal entries of $W^T W$ measure how strongly each feature represents itself. The off-diagonal entries are dot products between different feature directions.

A linear version of this model, with no final ReLU, learns a PCA-like solution: it gives separate dimensions to the $m$ most important features and drops the rest. The ReLU model behaves the same way when features are dense. Its behavior changes as they become sparse.

Sparsity changes what fits

Writing out one reconstructed feature shows what the model is trading off:

\hat{x}_i = \operatorname{ReLU} \left( \lVert W_i\rVert^2 x_i + \sum_{j\ne i}(W_i\cdot W_j)x_j + b_i \right).

The first term carries the feature itself. Every other represented feature can leak into its readout through an off-diagonal dot product. Orthogonal feature vectors avoid this interference, but there can be at most $m$ mutually orthogonal directions in an $m$ -dimensional space.

Sparsity lowers the cost of using non-orthogonal directions because interfering features seldom co-occur. The ReLU lowers it further. When only feature $j$ is active, negative interference into an inactive feature $i$ is clipped to zero. A negative bias can also filter small positive interference, at the cost of weakening small true activations. The interference diagram (Fig. 4) traces these terms through reconstruction: isolated sparse activations are usually recoverable, while simultaneous activations expose the ambiguity created by shared directions.

Interference between non-orthogonal feature directions during reconstruction, including ambiguity when several sparse features activate together — **Figure 4:** Non-orthogonal features leak into one another's readouts. Sparse activation and the ReLU make much of that leakage harmless, but simultaneous features can still produce ambiguous reconstructions. Figure from the paper.

The benefit of representing another feature eventually outweighs its expected interference cost. For two features competing for one dimension, the model has three basic choices: drop one feature, give it the dimension instead, or place the pair in opposite directions. The last choice represents both features but fails when they activate together. As sparsity and relative importance vary, the optimum switches sharply between these strategies. The regime chart (Fig. 5) records those switches as discontinuous changes in feature norms and geometry, rather than as a smooth rotation from one solution to another.

Learned weight matrices and per-feature norms across sparsity levels, moving from dropped features through antipodal pairs into denser superposition — **Figure 5:** Features pass through distinct regimes as sparsity increases: unrepresented, represented with dedicated capacity, or represented in superposition. Figure from the paper.

The paper calls these transitions phase changes in the loose sense of discontinuous changes in the learned solution. The toy model makes the tradeoff unusually clean: importance determines the benefit of keeping a feature, and sparsity determines how often its interference is paid.

Fractions of a dimension

With two hidden dimensions and five features of decreasing importance, a dense model keeps the two most important features on orthogonal axes. At higher sparsity it can store four features as two antipodal pairs. When the inputs are sparse enough, all five fit as the vertices of a pentagon. Those three solutions appear left to right in the figure (Fig. 6); the growing number of nonzero vectors is the visual evidence that sparsity lets more features fit.

Feature embeddings in a two-dimensional toy model at increasing sparsity: orthogonal axes, antipodal pairs, and a pentagon — **Figure 6:** A two-dimensional model first keeps two orthogonal features, then four features as antipodal pairs, and finally five features in a pentagon. More features are represented as the expected interference cost falls. Figure from the paper.

The paper measures how much capacity a feature occupies with its feature dimensionality:

D_i = \frac{\lVert W_i\rVert^2} {\sum_j (\hat{W}_i\cdot W_j)^2}, \qquad \hat{W}_i = \frac{W_i}{\lVert W_i\rVert}.

A feature with a dedicated direction has $D_i=1$ . Each member of an antipodal pair has $D_i=1/2$ . Five evenly spaced features in a two-dimensional pentagon have $D_i=2/5$ . In efficiently packed solutions, the feature dimensionalities approximately sum to the number of available dimensions.

Training repeatedly finds a small set of these fractional values. They correspond to regular geometric arrangements such as triangles, tetrahedra, pentagons, and square antiprisms. The connection comes from the loss: once feature norms are fixed, minimizing interference resembles a generalized Thomson problem in which points repel one another on a sphere. The authors are cautious about expecting these exact polytopes in real networks; their more durable point is that superposition can have organized geometry rather than being arbitrary entanglement.

The model also shows discrete changes during training. Individual features jump between fractional dimensionalities as the geometry rearranges, with small drops in loss at the same time. A smooth aggregate loss curve can therefore hide abrupt changes in what the network represents.

The paper tests whether superposition is limited to storage by training a related network to compute absolute value. A separate representation would use $\operatorname{ReLU}(x_i)+\operatorname{ReLU}(-x_i)$ for every feature. With sparse inputs, the trained model both represents and computes more features than it has neurons. This is a limited example, but it shows that a circuit can operate on superposed features without first assigning each one a dedicated neuron.

The same off-diagonal terms also create adversarial directions. In the toy model, adversarial vulnerability rises sharply when superposition forms because a small perturbation spread across many interfering coordinates can change a feature’s readout. The paper treats the connection as preliminary and does not claim that superposition is the main source of adversarial examples in real models.

Memorizing data points instead of features

Superposition, Memorization, and Double Descent keeps the reconstruction setup but trains on finite datasets. With very few examples, the clean geometry no longer appears in the columns of $W$ . It appears in the hidden representations $h_i=Wx_i$ of the individual training examples.

In that regime, the model treats each example as if it were a feature and packs the examples into regular polygons. This is a geometric form of memorization. It is cheaper to store the $T$ observed examples than to recover the much larger set of features that generated them.

As the dataset grows, the strategy stops working. The model moves through a disorganized middle regime and eventually represents the underlying features instead. Test loss peaks during that transition, producing data double descent: adding examples first makes generalization worse, then makes it better. The next figure (Fig. 7) puts the loss curve and representation geometry on the same horizontal axis, so the peak can be located at the handoff between example-based and feature-based solutions.

Test loss and representation geometry across dataset sizes, moving from memorized training examples through a transition to generalizing features — **Figure 7:** With little data, hidden representations of individual examples form regular polygons. With enough data, the feature directions form the clean geometry instead. Test loss peaks between the two regimes. Figure from the paper.

This experiment has no label noise because the input is also the reconstruction target. The irreducible error comes from compressing 10,000 possible features into a two-dimensional bottleneck. Weight decay and training duration affect the height of the test-loss bump, so the result is a mechanistic example of double descent rather than a general explanation of it. The parameter-norm plot (Fig. 8) supplies a second view of the transition: norms rise while the model packs in individual examples, then drop when it reorganizes around reusable features.

Parameter norms rising in the memorization regime, falling through the transition, and leveling off in the generalization regime — **Figure 8:** Parameter norms grow as the small-data solution packs in more training examples, then fall sharply as the model changes to a feature-based solution. Figure from the paper.

From toy models to real models

These papers establish that trained neural networks can use superposition when the data has sparse features and a nonlinear decoder. They also show exactly how polysemantic neurons arise in that setting. Both results are about synthetic reconstruction models with known features. The existence and form of superposition in language models still have to be established from their activations.

For interpretability, neurons may be the wrong unit of analysis. If a layer represents an overcomplete set of sparse feature directions, inspecting one coordinate at a time mixes those directions together. Towards Monosemanticity instead uses sparse autoencoders to recover a larger dictionary from the activations.

The 2025 note on interference weights (opens in a new tab) studies the spurious connections that appear when features and their weights are compressed into superposition. The July 2025 circuits update (opens in a new tab) rewrites parts of the original mathematical framework in terms of features extracted from superposed representations.

Further material

ARENA’s superposition and sparse-autoencoder exercises (opens in a new tab) — implement the toy model and reproduce several of its geometric solutions.

References

These are notes on the works marked ★ — cite the original rather than this page.

Elhage, N., & others. (2022). Toy Models of Superposition. Transformer Circuits Thread. https://transformer-circuits.pub/2022/toy_model/index.html ★
Henighan, T., & others. (2023). Superposition, Memorization, and Double Descent. Transformer Circuits Thread. https://transformer-circuits.pub/2023/toy-double-descent/index.html ★

Table of Contents

Features are not coordinates
The toy autoencoder
Sparsity changes what fits
Fractions of a dimension
Memorizing data points instead of features
From toy models to real models
Further material
References