Linear probes

The puzzle: does the model know more than it says?

Feed a language model the sentence "The city of Chicago is in Canada." The model may continue the text or agree with it under a leading prompt. Did any internal activation still register this is false?

A Llama layer stores several thousand unlabeled floating-point numbers per token, with no is_false field to read. If those activations represent truth in a readable form, a classifier could flag statements that conflict with the model's internal state. The deception-probing papers at the end of this week test that possibility.

Weeks 6 through 9 traced circuits to explain how a model computes a behavior. This week asks a narrower question: Does an activation contain a given piece of information in a linearly readable form? A linear probe answers that question with one learned direction.

A probe is boring on purpose

Start with a runnable toy. Suppose you collect activation vectors for true and false statements. If the model represents truth as a direction, the two groups shift toward opposite sides of activation space. A simple classifier averages each group, subtracts the false mean from the true mean, and uses that direction as a score.

# 01: a mass-mean probe on toy data
import numpy as np

rng = np.random.default_rng(0)
d = 8                                   # pretend d_model = 8
true_acts  = rng.normal(0, 1, (100, d)) + 3.0 * np.ones(d) / np.sqrt(d)
false_acts = rng.normal(0, 1, (100, d)) - 3.0 * np.ones(d) / np.sqrt(d)

direction = true_acts.mean(0) - false_acts.mean(0)   # μ_true − μ_false
scores = np.concatenate([true_acts, false_acts]) @ direction
labels = np.array([1]*100 + [0]*100)
acc = ((scores > 0).astype(int) == labels).mean()
print(round(float(acc), 2))
# 0.99  # the difference between cluster means separates this toy dataset

That difference-of-means direction defines a mass-mean probe, one of the two probe types in the notebook. The other is logistic regression: a learned vector w and bias b score an activation h as w·h + b. Gradient descent adjusts one straight decision boundary between the classes. The model contains no hidden layers or nonlinear feature extractor.

# no-run
# 02: logistic regression on activations
import torch

acts = torch.randn(32, 768)
probe = torch.nn.Linear(768, 1)
labels = torch.randint(0, 2, (32, 1)).float()
loss = torch.nn.functional.binary_cross_entropy_with_logits(probe(acts), labels)
print(loss.item())
# about 0.7  # random binary logits give loss near ln(2), or 0.693

Simplicity provides the experimental control. A deep classifier could build a truth detector from raw signals that the language model never organized for easy access. A linear probe reads only a weighted sum of the existing activation coordinates. High held-out accuracy therefore shows that truth-related information lies along a simple direction. A successful mass-mean probe makes the claim stronger because its direction comes directly from two class averages.

Keep one caveat visible: a probe establishes readability. A causal intervention must test whether the model uses that direction.

Truth as a direction

Each statement becomes one point in activation space: a high-dimensional vector extracted from a chosen layer and token position. If the model linearly represents truth, true and false statements form two clouds. A Truth direction points from the false cloud toward the true cloud. The week's experiments study that geometry.

In the notebook's real-model results, the trained logistic-regression direction and the mass-mean direction align closely. That alignment explains why the cheaper probe performs well there.

Extraction choices define what the probe can read. Pick the same hook point and token position for every statement. A common choice is the residual stream at the final token, where the activation has processed the whole claim. An earlier token lacks information from later words. Pooling across tokens asks a different question and can hide which position carries the signal. Record the model revision, tokenizer, prompt template, layer, hook point, and position alongside the activation array so a later comparison keeps incompatible representations separate.

Layer sweeps reuse the same train and test split at every depth. Train one probe per layer, then plot held-out accuracy across layers. A rise in the middle layers suggests when the representation becomes linearly accessible; a later decline may show that subsequent computation transforms or overwrites it. Use held-out predictive performance for the layer comparison. Each probe weight lives in a representation shaped by its layer's computation.

PCA (principal component analysis) offers a visual check before supervised training. It projects high-dimensional activations onto the two or three directions with the most variance, producing a lossy shadow of the point cloud. Visible separation between true and false statements shows that major geometric variation tracks the labels. The notebook sweeps PCA across layers: early layers show little separation, while middle layers produce distinct clouds.

PCA can also miss a useful truth direction. It preserves directions with the most overall variance, while a class-separating direction may carry less variance than topic, syntax, or statement length. Treat a separated PCA plot as visible evidence and an overlapping plot as inconclusive. Color the same projection by likely confounds, such as dataset source or template, to see whether the apparent truth clusters follow another label.

The week 7 connection is direct. A probe uses supervision to find one named feature direction. Week 8's sparse autoencoders find thousands of directions without labels and require interpretation afterward. A probe starts with a named concept and labeled examples, then searches for its direction.

The surprise: high accuracy is where the work starts

Probe accuracy can hide a familiar testing failure: a probe can succeed for the wrong reason.

Suppose false statements are shorter than true statements or come from one template, topic, or source. The probe can learn whichever property separates the classes: length, topic, punctuation, or a dataset fingerprint. Synthetic data makes this confound visible:

# 03: probe hygiene and the confound trap
import numpy as np

rng = np.random.default_rng(1)
n = 200
truth = rng.integers(0, 2, n)                    # the property we WANT to probe
noise = rng.normal(0, 1, (n, 6))                 # 6 dims carrying no signal at all
# in the TRAINING data, feature 0 ("statement length") happens to track truth exactly
X_train = np.column_stack([3.0 * truth + 0.1 * rng.normal(size=n), noise])
# in the PARAPHRASED test data, long statements are now the FALSE ones
X_test = np.column_stack([3.0 * (1 - truth) + 0.1 * rng.normal(size=n), noise])

w = X_train[truth == 1].mean(0) - X_train[truth == 0].mean(0)
train_acc = (((X_train @ w) > (X_train @ w).mean()) == truth).mean()
test_acc  = (((X_test @ w) > (X_test @ w).mean()) == truth).mean()
print(round(float(train_acc), 2), round(float(test_acc), 2))
# 1.0 0.0  # the shortcut succeeds in training and reverses on the changed test set

The probe scored 100% by reading the length column. A confound is a variable that correlates with the label while tracking another property. Real datasets rarely name it "column 0," so probe evaluation needs production-grade testing:

• split by dataset or template because a random-row split can share shortcuts across both sets;
• include simple baselines;
• test paraphrases and held-out topics;
• look for a probe that succeeds for the wrong reason.

The notebook spends time on datasets and pandas-like wrangling because most probe failures begin in experimental design. A random-row split can place the same template in train and test, letting the probe exploit that shared template. Grouped splits keep every example from one template, source, or topic on one side of the boundary.

Report more than aggregate accuracy. Break results down by dataset, template, class, and layer. Check whether one easy subset carries the mean and whether false positives cluster around a topic or punctuation pattern. Fit the same probe after shuffling labels; its held-out score should fall to chance. Compare against baselines built from token count or bag-of-words features. If a text-only baseline matches the activation probe, the dataset may expose a shortcut before the model's internal state adds useful evidence.

Keep threshold selection inside the training data. For logistic regression, fit the weights and any regularization strength on training and validation folds, then evaluate the locked classifier once on the held-out group. For a mass-mean probe, estimate both class means from training examples and choose the score threshold without consulting test labels. Report balanced accuracy when classes differ in size, since raw accuracy can reward a classifier that predicts the majority class. Confidence intervals across bootstrap samples or several grouped splits show whether a small score gap is stable.

From reading to intervening

A hygienic, generalizing probe establishes that information is present and readable. The model may still ignore that direction when choosing its next token. Week 9's causal standard therefore applies: co-variation with truth leaves the behavioral role open.

A good probe generates a hypothesis for intervention. Push the activation along the probe direction or use patching-style edits, then measure whether the model's output changes as predicted. Notebook section 3 connects the cheap readout with week 9's causal tests. Week 11 develops direction-based interventions into steering.

Design the intervention with controls. Normalize the probe direction, sweep several positive and negative scales, and compare the target metric with an unedited run. A truth direction should produce a signed prediction: movement toward the true-class side should affect truth-related outputs in one direction, while the opposite edit should reverse the effect. Apply matched random directions with the same norm to estimate how much any perturbation changes the model. Also track language-model loss or output quality, since a large edit can damage the activation broadly and create an apparent effect through general disruption.

Sections 4 and 5 apply probes to deception. They study models whose outputs conflict with internally readable truth signals and ask whether probes trained on simple contrasts transfer to scenarios where a model receives instructions to mislead. A linear detector will have false positives and distribution shifts. It still provides a concrete safety measurement with auditable failure modes.

Pair-session guide

Core work is ARENA 1.3.1 sections 1 and 2: setup and visualizing truth representations (extract activations, PCA, a layer sweep asking "where does truth live?"), then training and comparing probes across datasets. Stretch work is section 3 (causal interventions), section 4 (probing for deception), and section 5 (attention probes for high-stakes detection). Assign one person to be the "skeptic" for each result: their job is to name a possible confound before the group celebrates accuracy.

This notebook probes Llama models. Sections 1–3 load Llama-2-13B, which needs about 26 GB in bfloat16, and section 4 uses Llama-3.1-8B-Instruct. Plan for an A100-class GPU and a Hugging Face token with access to the gated Llama repositories. Verify the token and runtime before the session; the free Colab T4 lacks enough memory for the 13B model.

What you should see

You should see PCA plots where true and false statements separate in some mid-layer activations. Train a probe with high held-out accuracy, then create one engineered failure where a shortcut succeeds on one split and fails on a paraphrased or grouped split.

Where to go next

Week 11 changes the operation from reading model directions to writing along them. The notebook's core readings provide the supporting experiments:

• The Geometry of Truth by Marks & Tegmark is the paper sections 1-3 replicate: truth directions that generalize across datasets and are causally implicated in outputs.
• Detecting Strategic Deception Using Linear Probes by Goldowsky-Dill et al. (Apollo Research) is the paper behind section 4: probes trained on simple contrastive data generalizing to realistic deception scenarios.
• EleutherAI's attention probes post is the friendly version of the high-stakes-detection paper behind section 5; the paper itself is optional.