every term, defined once

Glossary

Each term is introduced in exactly one week, then linked back from later weeks.

week 01 · Orientation and a first digit classifier

Tensor: An n-dimensional array. In this course, tensors are the basic data structure flowing through every model.
Shape: The size of each tensor axis, read like a type signature for the data.
Broadcasting: PyTorch automatically expands compatible smaller tensors across missing dimensions before an operation.
einops: A small library for readable tensor rearrangement, reduction, and repetition.
einsum: Index-notation syntax for dot products, matrix multiplies, reductions, and many tensor contractions.
Weight: A learnable number inside a model, usually stored in a tensor.
Bias: A learnable offset added after a weighted sum.
ReLU: The function max(0, x). It lets stacked layers represent more than one linear operation.
Logits: Raw model scores before softmax converts them into probabilities.
Residual connection: A direct branch that adds an earlier activation to the output of other layers: out = x + f(x).
Batch normalization: A layer that normalizes each channel during training, then applies a learned scale and offset.
Softmax: A function that turns a vector of scores into a probability distribution by exponentiating and normalising.
Cross-entropy: The standard classification loss. It measures how little probability the model put on the correct label.
Convolution: An operation that applies the same small kernel at each position in an image to find local patterns.
Gradient: For each parameter, the gradient describes how a small increase would change the loss.
SGD: Stochastic gradient descent: step parameters opposite the gradient.
Learning rate: The step size an optimizer uses when updating parameters.
Adam: A popular optimizer that adapts per-parameter step sizes using running gradient statistics.
Weight decay: A regularizer that nudges weights toward zero during optimization, discouraging solutions that require large parameter values.
Chain rule: The rule that lets backprop multiply local derivatives through a composed computation.
Computational graph: A graph of tensor operations whose reverse traversal computes gradients.
Autograd: Automatic differentiation: software that builds and backpropagates through a computation graph.

week 02 · Transformers from scratch

Autoregressive: An autoregressive model generates a sequence one item at a time. Each new item depends on the items before it.
Token: A token is a text chunk represented by an integer.
Vocabulary: A vocabulary is the fixed list of tokens that a model can read or produce.
Byte pair encoding: Byte pair encoding is the method GPT-2 uses to split text into tokens. It starts from single bytes and repeatedly joins the most common neighboring pair into a new token, so frequent chunks of text become single tokens while rare text still breaks into smaller pieces.
Embedding: An embedding is a learned vector that represents a token inside the model.
Residual stream: The residual stream has one vector at each token position and carries information between transformer layers.
Attention: Attention combines information from the token positions that a position is allowed to read.
Query, key, and value: Attention computes a query, key, and value for each position. Queries and keys determine the attention weights. Values carry the information those weights combine.
Causal attention: Causal attention prevents each position from reading later positions.
Attention pattern: An attention pattern is a matrix of weights with one row per query position and one column per key position.
Positional encoding: Positional encoding records a token's position in the sequence.
MLP: An MLP applies two linear layers and a nonlinear function to each token position independently.
LayerNorm: LayerNorm normalizes each token vector, then applies a learned scale and offset.
Unembedding: Unembedding maps each final residual stream vector to one logit for every token in the vocabulary.

week 03 · Introduction to mechanistic interpretability

Mechanistic interpretability: The project of explaining model behavior by identifying the internal algorithms and components that cause it.
TransformerLens: A library that exposes transformer activations and hooks for mechanistic interpretability work.
Activation cache: A collection of named intermediate activations recorded during a model forward pass.
K-composition: A circuit in which one attention head builds its keys from information written by an earlier head.
Induction head: An attention head that implements a "copy what followed this token last time" pattern.
Induction score: The average attention weight on the expected offset stripe for destinations in the second copy of a repeated sequence.
Hook: A function called at a named activation during a forward pass to read or replace that activation.

week 04 · Probes, steering vectors, and SAEs

Feature: A property of the data or the computation that researchers may want to track. Features exist in the data distribution whether or not an SAE finds them.
Sparse autoencoder (SAE): A model trained to reconstruct activations using a sparse, wider latent representation.
SAE latent: One learned coordinate of an SAE: an encoder direction that decides when it fires, and a decoder direction it writes back. A candidate representation of one or more features, not a feature by definition.
L0: The average number of SAE latents active on a token. The standard sparsity measure for an SAE.
sae_lens: An open-source library for loading, training, and evaluating sparse autoencoders on model activations.
Neuronpedia: A public browser for inspecting model neurons and SAE features.
Feature dashboard: A report showing examples, stats, and effects for one SAE latent or neuron.
Linear probe: A simple linear classifier trained on activations to test whether information is linearly readable.
Mass-mean probe: A linear probe whose direction is the difference between the mean activation for each of two labeled classes, computed in closed form rather than fitted by optimization. It scores raw activations with no intercept.
Logistic regression: A linear classifier that fits a weight vector by optimization, then maps its score to a class probability with the logistic function. The probes fitted this week standardize activations first and fit no intercept.
PCA: Principal component analysis: a way to project high-dimensional activations onto high-variance directions.
Truth direction: A direction in activation space associated with true versus false statements.
Confound: A variable correlated with the label that can create an apparent effect without representing the property the experiment aims to measure.
Steering vector: A vector added to an activation to push model behavior in a chosen direction.
Function vector: An activation direction that appears to encode a task such as antonym generation.
Contrastive pair: Two inputs matched except for a target property; subtracting their activations estimates a direction associated with that property.
Activation addition: An intervention that adds a chosen vector to a model activation during a forward pass to steer subsequent behavior.
nnsight: A tracing and intervention library for reading and editing model internals, including remotely hosted models.

week 05 · Finding circuits in language models

Ablation: Removing or replacing a model component to test whether a behavior depends on it.
IOI: Indirect Object Identification, a benchmark sentence task used to study a GPT-2 circuit.
Activation patching: Swapping activations between model runs to test which internals causally affect behavior.
Noising vs. denoising: Complementary patching experiments: noising corrupts a clean run to test necessity, while denoising restores part of a corrupted run to test sufficiency.
Logit difference: A metric comparing model scores for the correct and incorrect answer tokens.
Direct logit attribution: Measuring how much each component’s direct residual-stream write changes the logit for a chosen token. It shows association, not cause.
Name-mover head: An attention head in the IOI circuit that attends to the correct name and copies it into the output logits.
S-inhibition head: An attention head in the IOI circuit that moves the "this name is duplicated" signal to the final position, reducing later heads' attention to the repeated subject name.
Path patching: A refinement of activation patching that isolates a specific sender-to-receiver path while holding the rest of the activation fixed.

week 06 · Understanding models trained on small tasks

Superposition: Representing more features than dimensions by packing sparse features into overlapping directions.
Sparsity: The property that most possible features are inactive for any given input.
Polysemantic neuron: A neuron that responds to multiple unrelated features.
Grokking: A delayed transition where a model moves from memorization to real generalization long after fitting the training set.
Progress measure: A quantity computed during training that tracks the gradual formation of a mechanism, even while headline loss or accuracy is flat.
Fourier basis: A way to represent periodic patterns as sums of sine and cosine waves.
FFT: Fast Fourier transform: an efficient algorithm for computing a signal's discrete Fourier transform and revealing its frequency components.
Restricted ablation: An ablation that removes everything except a hypothesized subspace or mechanism.

week 07 · Reinforcement learning foundations

State: The information needed from the past to determine the distribution of the next state and reward once an action is given.
Markov decision process: A model of sequential decisions built from states, actions, transition probabilities, rewards, and a discount factor.
Policy: A rule or probability distribution that chooses an action from a state.
Trajectory: One sequence of states, actions, and rewards produced as an agent interacts with an environment.
Episode: A finite sequence of interactions between an agent and an environment.
Discounted return: The sum of future rewards after multiplying later rewards by progressively smaller powers of the discount factor.
Discount factor: A number between zero and one that controls how strongly an agent values later rewards.
Value function: The expected discounted return from a state under a policy.
Q-value: The expected discounted return after taking a chosen action in a state and following a policy afterward.
Bellman equation: A recursive equation that writes a value as immediate reward plus the discounted value of what follows.
Tabular method: A reinforcement learning method that stores one value for every state or state-action pair.
Q-learning: An off-policy method that updates a Q-value toward the reward plus the best estimated value at the next state.
SARSA: An on-policy method that updates a Q-value using the next action selected by the current policy.
TD error: The difference between a current value estimate and its one-step reward-plus-next-value target.
Epsilon-greedy: A policy that usually takes the highest-value action and chooses a random action with probability epsilon.
DQN: Deep Q-Network: a neural network trained to estimate Q-values for each action.
Replay buffer: A store of past transitions sampled to train a reinforcement learning agent.
Policy gradient: A method that changes policy parameters in the direction that raises expected return.
Gymnasium: A Python library that provides standard reinforcement learning environments and interfaces.

week 08 · PPO and RLHF

PPO: Proximal Policy Optimization: a method that collects a rollout, computes fixed learning targets, then clips the incentive for large favorable sampled-action probability changes.
Rollout: A frozen batch collected under the current policy: states, sampled actions, rewards, termination and truncation flags, old-policy log probabilities, and critic values.
CartPole: A control environment where a cart keeps a pole balanced by pushing left or right. CartPole-v1 observes position, velocity, pole angle, and angular velocity.
Actor: The policy network that produces the action distribution.
Critic: The value network that estimates expected future return from the current state.
Advantage: How much better or worse an action is than the critic's expected return from the same state.
GAE: Generalized advantage estimation: a method that carries critic prediction errors backward through a rollout to estimate advantages.
Clipped objective: PPO's sampled-action surrogate that clips the current-to-old probability ratio, removing extra incentive after a large favorable change.
Entropy bonus: A term in the PPO objective that rewards a broader action distribution.
RLHF: Reinforcement Learning from Human Feedback: a pipeline that trains a model on demonstrations, learns a reward model from ranked responses, then optimizes the policy against that reward.
Reference model: A frozen copy of the starting model whose token distribution anchors the current policy.
KL penalty: A term that penalizes the current policy for moving away from the reference model's token distribution.
Value head: A small network that reads the residual stream and produces one scalar value per position.
Reward hacking: Maximizing a scorer in a way that defeats the intended purpose, such as producing punctuation instead of useful text.

week 09 · Designing language model evaluations

Capability evaluation: An evaluation that measures whether a model has the capacity to perform a specific behavior.
Alignment evaluation: An evaluation that measures whether a model has the tendency or propensity to show a specific behavior.
Threat model: A realistic scenario by which an AI capability or tendency could lead to harm, used to decide which model properties are worth measuring.
Evaluation specification: A document that defines the target property and how to measure it, with operational definitions, before any questions are written.
answer_matching_behavior: The field in a multiple-choice item that names which answer choice indicates the target property.
Revealed preference: Measuring what a model values through the choices it makes rather than through what it says it wants.
Structured output: An API mode that returns model output matching a user-defined schema instead of a raw string.
Few-shot prompting: Giving a model worked examples of the desired output inside the prompt before asking for a new example.
ThreadPoolExecutor: A Python concurrency tool that runs a function across many inputs in parallel and returns results in input order.
Safety case: A structured argument that a system is unlikely to cause catastrophe, supported by evidence from evaluations.
System prompt: The first message in a chat request, carrying instructions that apply to every turn of the conversation.
Rate limit: A cap on requests or tokens an API accepts per window. Crossing it returns an error instead of a response.
Exponential backoff: Retrying a failed request after a wait that doubles with each attempt.
Rubric: A scoring guide that defines what low, middle, and high scores mean so an LLM can grade questions consistently.

week 10 · Running evaluations, building agents, and AI control

Inspect: A library from the UK AI Security Institute for defining and running language model evaluations.
Solver: A step in an Inspect evaluation that transforms model state, such as by prompting a model or handling a tool call.
Scorer: A function that compares a model response with a target or rubric and returns an evaluation result.
Elicitation: The effort to make a model demonstrate the capability an evaluation aims to measure.
Scaffolding: The prompts, tools, memory, and control loop wrapped around a model to help it complete a task.
Tool calling: A model requesting a structured external action, such as a search or code execution, then using the returned result.
AI control: Methods that use monitoring and restrictions to keep an AI system safe even when it may try to cause harm.

week 11 · Studying misaligned behavior

Emergent misalignment: Broadly misaligned behavior that appears after fine-tuning a model on one narrow harmful task.
Model organism: A model deliberately trained to display a behavior so researchers can study that behavior under controlled conditions.
LoRA: Low-rank adaptation: fine-tuning that freezes the original weights and trains small low-rank updates.
Autorater: A language model prompted to score other model responses against a rubric.
Compliance gap: A difference in model compliance rates between two conditions, such as monitored and unmonitored conversations.
Alignment faking: A model strategically complying during training to avoid having its existing preferences changed.