SAEs on real models

From toy features to real activations

In week 7 we knew the true features because we generated the data. In a real model, we only have activations and behavior. An SAE gives a proposed dictionary of latents, then humans test whether those latents are coherent.

One vocabulary note ARENA is strict about this week: a feature is a property of the underlying data ("mentions a bridge"), while a latent is a direction the SAE learned. Keeping the words separate stops you from silently assuming every SAE direction corresponds to a real feature; whether it does is exactly the question under test.

The dictionaries are huge: thousands to millions of latents. Some are crisp, some are dead, some split one concept across several latents, and some are not interpretable. That uncertainty is part of the field, not a failure of the exercise.

Reading a feature dashboard

A Feature dashboard is a field microscope for one latent. Look for:

• top-activating text examples: what reliably turns the latent on;
• activation statistics: whether it is rare, common, or dead;
• logit effects: which output tokens are promoted or suppressed;
• manual tests: prompts you invent to check the story.

Before the session, open Neuronpedia and search for a model you recognize. Try to find one feature that looks semantic, one that looks syntactic, and one you cannot explain.

Running an SAE as an instrument

The code pattern is model plus SAE plus cache. The exact model names may change, but the mental model is stable: run text, collect SAE latent activations, inspect the strongest latents.

from sae_lens import SAE, HookedSAETransformer

model = HookedSAETransformer.from_pretrained("gpt2-small")
sae = SAE.from_pretrained(release="gpt2-small-res-jb", sae_id="blocks.7.hook_resid_pre")
_, cache = model.run_with_cache_with_saes("The Golden Gate Bridge", saes=[sae])
acts = cache["blocks.7.hook_resid_pre.hook_sae_acts_post"]
print(acts[0, -1].topk(5).indices)

The top indices are not answers by themselves. They are leads you investigate with dashboards and prompt tests.

Honest limitations

SAEs can have reconstruction error: the decoded activation is not exactly the original. Latents can be dead, split, or too broad. A feature can look meaningful because the examples are cherry-picked. The cure is the same evidence habit as week 9 will formalize: inspect, intervene, and track metrics.

Training your own SAE is stretch. It adds data pipelines and the two headline evaluation metrics: L0, the average number of latents active on a given token (lower means sparser), and loss recovered, how much of the model's original performance survives when you splice the SAE's reconstruction back into the forward pass. Useful, but not required to get the core idea.

Pair-session guide

Core work is ARENA 1.3.3 section 1: run pretrained SAEs, inspect features, and build dashboards or dashboard-like reports. Stretch work is deeper latent analysis and training/evaluation. If free Colab runs out of memory on larger models, switch to GPT-2-small variants rather than debugging CUDA for an hour.

What you should see

You should produce or inspect at least one feature dashboard and be able to state a hypothesis like "this latent fires on bridge names" or "this one tracks Python exception code." The best outcome is one genuine surprise where a latent looks far more human-readable than you expected.

Where to go next

• Anthropic's Towards Monosemanticity is ARENA's single top recommendation for this notebook: the first large-scale demonstration that SAE latents on a real (1-layer) language model are interpretable. Read the first half of "Problem Setup" and the deep dives on individual latents.
• Anthropic's Scaling Monosemanticity scales the recipe to Claude 3 Sonnet; it is the origin of the famous Golden Gate Bridge feature.
• DeepMind's Gemma Scope announcement introduces the open-source SAE suite the notebook's Gemma sections use.