Large language models (LLMs) are mostly trained to predict the next token in a sequence. Despite this simple training objective, it seems that models learn to use advanced mechanisms to process the input and reason about it.
For example, when model is asked to write a poem, it seems that instead of improvising token-by-token and then coming up with a rhyming word at the end of the line, it selects the final rhyming word already at the beginning of the line, then writes the line to fit with the final word it opted to use, and finishes with the previously selected word.
These findings come from Anthropic’s research Tracing the Thoughts of a Large Language Model (March 2025, blog post, paper). This post is an overview of this research.
Setup
The experiments were done on a Local Replacement Model, which is a more interpretable version of Haiku 3.5, Anthropic’s smallest model in production.
Background: Interpretability and polysemanticity
Let’s backtrack a bit. A neural network is usually seen as a black-box—we give it some input and it returns some output, but we do not know what happens inside. The goal of interpretability research is to find out what really happens inside the network.
One way this could be done is by inspecting neuron activations during network processing. For example, there could be neurons firing only on very specific concepts, such as ‘European capital’ or ‘malicious code’. This would help us understand how the model processes the input and also potentially steer the generations by amplifying or reducing neuron activations.
Unfortunately, this is not the case, and individual neurons usually fire on all kinds of unrelated concepts. If a single neuron fires on inputs related to cars, dogs, happiness, UTF-8 encoding, and other 20 things, it’s hard to interpret its high activation. This is known as polysemanticity. One hypothesis for why this happens is that there are many more concepts than the neurons, so the model needs to use a single neuron to represent more stuff.
Reducing polysemanticity
A standard method to reduce polysemanticity is Sparse Autoencoders (SAEs). An SAE is a component which takes an input, expands it to bigger dimension, and then shrinks it back to the original dimension—while being trained to maximize similarity of its output and the input. In addition, the training also penalizes large activations of the intermediate neurons.
These intermediate neurons are often more interpretable than the original ones. In Scaling monosemanticity, one such neuron was classified as “The Golden Gate Bridge”—it activated on concepts related to the bridge. If the value of the neuron was multiplied during the forward pass, the “Golden Gate Claude” would invariably respond with bridge-related answers.
This paper uses a more powerful version of Sparse-autoencoders called Cross-layer Transcoder (CLT).
Local replacement model
Taking the Haiku 3.5, adding and training the CLT layers creates a Replacement Model, which contains 30 million additional neurons or features (see SAE figure above).
Although the CLTs are trained to reconstruct the input, there is still some reconstruction error, leading to differences from the activations of the original model. To fix this, additional error nodes are added to each layer. Error node is a feature with the reconstruction error from the original model on a given prompt.
The attention patterns are also inherited from the forward pass of the prompt on the original model, creating a Local Replacement Model (LRM) (local because the error nodes and attention patterns are prompt-specific). As a result, the output of the LRM is equal to the output of the original model.
To recap:
Replacement model = Haiku 3.5 trained with trained CLT layers
Local replacement model = Replacement model evaluated on a specific prompt with error nodes and attention patterns added from the forward pass of that prompt on the Haiku 3.5.
Interpreting the LRM
After creating the LRM for a given prompt, it’s now possible to inspect its activations, which should be more interpretable than those of the original model. The activations are represented by an attribution graph, where each feature becomes a node in the graph.
To reduce the size of the graph, features without significant influence on the output are removed. To simplify this even further, the related features are manually grouped into supernodes. For example, feature ‘say European capital’ (activates on Stockholm, Madrid, …), and ‘say U.S. capital’ (activates on Austin, Sacramento, …) would be manually grouped into a ‘say a capital’ supernode.
Results
The paper then analyses model behavior in several contexts. This is presented in a neat interface, and I encourage you to skim through the website. Some highlights:
The aforementioned poem-planning example (§ 4)
Shared multi-lingual circuits (§ 5)
Some discovered features were multilingual, e.g., a feature that activates when describing large things in different languages. The model would use this same feature to say the opposite of “small” in multiple languages.
Lack of meta-cognition (§ 6)
The researchers track down a process by which the model answers a two-digit addition question. However, when asked to describe the process by which it arrived at an answer, the model provides an explanation that does not match the observed process.
Multi-step reasoning when asking a patient additional questions for medical diagnosis (§ 7)
The model first extracts the symptoms from the input (pregnancy, headache, …), picks the most likely (preeclampsia) and alternative diagnoses, thinks about other symptoms associated with preeclampsia (epigastric pain, visual deficits, …), and finally asks whether she experiences visual disturbances.
One has to keep in mind that this work was done on a single model, Claude Haiku 3.5. Moreover, the authors caveat that the results are claims only about the specific presented examples, without making a broader claim that the observed reasoning happens all the time in similar situations.
Notes
To me, this paper demonstrates that the intelligence arising in LLMs may be more general than I previously thought. The observed processes, such as planning or multi-step reasoning, are powerful building blocks of an intelligence that goes beyond simple memorization (of course, there is still a lot of memorization).
The examples in the paper seem remarkably similar to how a human mind would approach them. If I were writing a poem, I would probably mostly reason from the rhyming words and write the rest of the line to fit, just like Haiku did. When researching a potential diagnosis online, I too begin with my symptoms, find conditions that match them, then cross-check by looking for other symptoms associated with those conditions.
It’s fascinating that all this is possible in a forward pass of a transformer model trained on a next-token prediction.