Title: Transformers

Q: Autoregressive language model
A: The model sees part of the text and has to predict the probability of each possible next token

Q: Token
A: The smallest unit of text that a particular transformer understands. This could be a word or part of a word.

Q: Masked language modelling
A: A token in the text is masked (i.e. hidden) and needs to be predicted

Q: Induction
A: Word combinations that have appeared recently are more likely to repeat again

Q: Downstream task
A: The new task to which a model is fine-tuned

Q: Preference modelling
A: A model which predicts how valuable we would find the output of another model, allowing the latter to be trained using RL.

Q: Prompt engineering
A: Prompt engineering or prompt programming is the process of finding prompts that will cause the model’s pre-training objective (output a likely continuation given the prompt) to align with solving a task we care about.

Q: MLP
A: Multi-layer perceptrons, despite not containing perceptrons, in GPT-2 consist of a sequence of Layer Norm, Linear, GELU, Linear, and Dropout layers.

Q: GPT-2
A: - OpenAI's Generative Pre-trained Transformer
- Released in 2019
- Decoder-only transformer
- Pre-trained on language modelling

Q: PaLM
A: Google's Pathways Language Model was published in 2022

Q: Vocabulary size
A: Number of possible tokens

Q: Word tokenisation
A: Each distinct word in the training set is a token, plus a token to represent masks

Q: Character tokenisation
A: One token represents one character

Q: Byte-level Byte-Pair Encoding
A: - Tokeniser used by GPT-2
- Unicode string is considered as a sequence of bytes, e.g. 256 possible 8-bit bytes in utf-8
- Merge tokens by creating a new token for the most common token pair 50,000 times

Q: QK circuit
A: A query-key (QK) circuit computes attention probabilities. The input to the softmax are the attention scores.
\[
A=\operatorname{softmax}_K^*\left(t^T \cdot W_E^T W_{Q K}^h W_E \cdot t\right)
\]
Numerically, this is a \(d_\text{vocab}\times d_\text{vocab}\) matrix with rank just \(d_\text{head}\).
Intuitively, it describes how much a given query token "wants" to attend to a given key token.
Graphically, it is formed by tracing the computation of a query and key vector up to their attention head, where they dot product to create a bilinear form.

Q: OV circuit
A: The output-value (OV) circuit is the product \(W_U W_{O V}^h W_E\) which right-multiplies the attention probabilities to return the head output.
Numerically, this is a \(d_\text{vocab}\times d_\text{vocab}\) matrix with rank just \(d_\text{head}\).
Intuitively, it describes how a given token will affect the output logits if attended to.
Graphically, it is created by tracing the path computing a value vector and continuing it through up to the logits

Q: Circuit
A: A “circuit” is a computational subgraph of a neural network.

Q: GELU
A: GAUSSIAN ERROR LINEAR UNITS are a smooth version of RELU activation defined as
\[\mathrm{GELU}(x) = x\Phi(x) = x\int_{-\infty}^0 \frac{1}{\sqrt{2\pi}} e^{-\frac{1}{2}z^2} \mathrm{d}z.\]

Q: Tied embedding
A: An embedding where we use the transpose of the unembedding matrix to unembed (which can be done at any time), rather than a learnable matrix (which would require GPU space).

Q: BERT
A: - Bidirectional Encoder Representations for Transformers
- An extremely similar model to GPT-2
- Main difference: encoder-decoder or bidirectional, i.e. attention can see future sequence positions

Q: Logit lens
A: An interpretability concept where we apply the umembedding matrix to vectors at an intermediate stage of the network

Q: NSP
A: Next Sentence Prediction is a task to predict whether an ordered pair of sentences are consecutive or not.
For this a Token Type embedding is used to indicate a binary value for whether the token belongs to the first or second sentence.

Q: Query vector
A: A feature vector \(q_i = W_Q x_i\) that describes what we're looking for in the sequence, i.e. what we might want to pay attention to (like locks of different shapes).
We compute the attention scores by dotting the given query vector with the previous keys.
Think "destination vector".

Q: Key vector
A: The key \(k_i=W_K x_i\) is a feature vector describing what each element might be “offering”, or why it is important (the key to unlocking the query).
We compute the attention scores by dotting the given query vector with the previous keys.
Think "source vector".

Q: Self-supervised learning
A: A pre-training step where we generate labels automatically using masking

Q: Bag of Words
A: One-hot encoding of text with dimension equal to vocabulary

Q: Word Embedding
A: Train a neural network to predict the distribution of nearby words. Then the softmax activations form the word vectors.

Q: Encoder vs. decoder transformers
A: For decoder-only transformers, the self-attention layer is only allowed to attend to earlier positions.

Q: Greedy sampling
A: Take the next token with highest probability at each successive position

Q: Beam search
A: Keep the top-\(n\) sequences at each successive position until reaching an End Of Sequence token or the max sequence length.
Finally take the sequence with highest probability from the \(n\).
The parameter \(n\) is known as the beam width.

Q: Value vector
A: The value vectors \(v_i=W^Vx_i\)  represent the feature vectors we want to pass on to the linear layers. They are weighted with the attention probabilities at each position and then summed:
\[z_i = \sum_j A_{ij} v_j.\]

Q: Output matrix
A: The output matrix is used to recover the dimension after stacking multiple independent attention heads

Q: Positional encoding
A: Let \(x\) be position, \(d\) be the encoding dimension (where \(d \equiv_2 0\) ). Define \(f: \mathbb{N} \rightarrow \mathbb{R}^d\) as
\[
f(x)^{(i)}:= \begin{cases}\sin \left(\omega_k x\right), & \text { if } i=2 k \\ \cos \left(\omega_k x \right), & \text { if } i=2 k+1\end{cases}
\]
where
\[\omega_k=\frac{1}{10000^{2 k / d}}\]
are the decreasing frequencies by embedding dimension.
Intuition: like the continuous/floating-point analogue of binary encoding.
At a particular sequence position we have
\[
f(x)=\left[\begin{array}{c}
\sin \left(\omega_1 \cdot x\right) \\
\cos \left(\omega_1 \cdot x\right) \\
\sin \left(\omega_2 \cdot x\right) \\
\cos \left(\omega_2 \cdot x\right) \\
\vdots \\
\sin \left(\omega_{d / 2} \cdot x\right)\\
\cos \left(\omega_{d / 2} \cdot x\right)
\end{array}\right]_{d \times 1}.
\]
From addition formulae we have the nice property that relative positions can be attended to by applying a rotation matrix:
\[
\left[\begin{array}{cc}\cos \left(\omega_k . \phi\right) & \sin \left(\omega_k . \phi\right) \\ -\sin \left(\omega_k . \phi\right) & \cos \left(\omega_k . \phi\right)\end{array}\right]\left[\begin{array}{c}
\sin \left(\omega_k \cdot t\right) \\
\cos \left(\omega_k . t\right)
\end{array}\right]=\left[\begin{array}{c}
\sin \left(\omega_k \cdot(t+\phi)\right) \\
\cos \left(\omega_k \cdot(t+\phi)\right)
\end{array}\right]
\]

Q: Top-\(K\) sampling
A: For some fixed parameter \(K\), we sample from the conditional distribution restricted to the top \(K\) tokens.

Q: Nucleus sampling
A: Top-\(p\) sampling restricts the conditional distribution to tokens with probability \(>p\) for some fixed parameter \(0<p<1\).

Q: Enforcing coherence when sampling from LLMs
A: - Pick temperature \(T<1\) in softmax
- Top-\(K\) sampling
- Nucleus sampling

Q: MLM
A: Masked Language Modelling is a task where a subset of tokens are masked and the objective is to recover the original sentence, i.e. "fill in the gaps"

Q: [UNK]
A: AN unknown token

Q: [CLS]
A: Classification token: a placeholder for the sequence position where we compute the predictive loss on classification-tasks

Q: [SEP]
A: The separator token is used in NSP to indicate where one sentence ends and the next starts.

Q: [PAD]
A: Padding tokens are appended to the end of a sequence to increase the length to the maximum allowed sequence length to allow batching

Q: [MASK]
A: The mask token is used in MLM tasks

Q: 80-10-10 rule
A: In MLM from the BERT paper, tokens are replaced 15% of the time. Of these, there is an 80-10-10 split between [MASK], [random] and [unchanged]:
- mask tokens focus the model on filling the gaps
- random tokens ensure that the model learns relationships between non-mask tokens without blindly copying
- unchanged tokens are used to bias the representation back towards the actual observed token.

Q: Virtual weights
A: Since the residual stream is linear, that one can think of implicit "virtual weights" directly connecting any pair of layers by multiplying out the output weights of one layer with the input weights of another, e.g. \(W_I^3 W_O^1\) )

Q: Multi-head attention
A: We concatenate many heads and multiply by an output matrix:
\[
\begin{aligned}
\operatorname{MultiHead} & =\operatorname{Concat}\left(r^{h_1}, \ldots,  r^{h_n} \right) W_O^H.
\end{aligned}
\]
If we split \(W_O^H\) into equal size blocks for each head \(\left[W_O^{h_1}, W_O^{h_2} \ldots\right]\), then we observe that:
\[
W_O^H\left[\begin{array}{c}
r^{h_1} \\
r^{h_2} \\
\cdots
\end{array}\right]=\left[W_O^{h_1}, W_O^{h_2}, \ldots\right] \cdot\left[\begin{array}{c}
r^{h_1} \\
r^{h_2} \\
\cdots
\end{array}\right]=\sum_i W_O^{h_i} r^{h_i}
\]
revealing it to be equivalent to running heads independently, multiplying each by its own output matrix, and adding them into the residual stream.

Q: Single attention head tensor equation
A: where \[
A=\operatorname{softmax}\left(x^T W_Q^T W_K x\right).
\]

Q: Virtual attention heads
A: By the distributive property, the product (composition) of two attention heads is
 \[
\left(A^{h_2} \otimes W_{O V}^{h_2}\right) \cdot\left(A^{h_1} \otimes W_{O V}^{h_1}\right)=\left(A^{h_2} A^{h_1}\right) \otimes\left(W_{O V}^{h_2} W_{O V}^{h_1}\right).
\]
which is equivalent to an attention head with \(A=A^{h_2} A^{h_1}\) and \(W_{OV}=W_{O V}^{h_2} W_{O V}^{h_1}\).

Q: Zero-layer transformer
A: Such a model takes a token, embeds it, unembeds it to produce logits predicting the next token:
\[
T=W_U W_E.
\]
Because the model cannot move information from other tokens, we are simply predicting the next token from the present token. This means that the optimal behavior of \(W_U W_E\) is to approximate the bigram log-likelihood.

Q: One-Layer Attention-Only Transformers
A: We expand the product into end-to-end paths
\[
\begin{aligned}
T&=\left(\operatorname{Id} \otimes W_U \right) \left(\operatorname{Id} +\sum_{h \in H_1} A^h \otimes W_{O V}^h\right) \left(\operatorname{Id} \otimes W_E \right) \\
&= \operatorname{Id} \otimes W_U W_E \quad+\sum_{h \in H} A^h \otimes\left(W_U W_{O V}^h W_E\right)
\end{aligned}
\]
consisting of a "direct path" term for bigram statistics and attention heads for skip-trigrams.

Q: Skip-trigrams
A: A skip-trigram is a sequence of tokens of the form
1 | [source]... [destination][out]
where
- the QK circuit determines which "source" token the present "destination" token attends back to and copies information from
- the OV circuit describes the resulting effect of the "source" token on the "out" prediction.
This process is factored: the OV circuit doesn't know which destination token is attending to the source token. Hence the bug that if \([a]...[b][c]\) and \([a]... [d][e]\) are likely then so are \([a]...[b][e]\) and \([a]...[d][c]\).

Q: Copying matrices
A: There are multiple possible definitions to detect copying in an OV circuit:
- positive-leaning eigenvalues
- positive-leaning entries on the diagonal
- % of columns where the greatest entry is the diagonal
- % of columns where the diagonal is in the top-\(k\) entries.
Note that the sum of the diagonal is the sum of the eigenvalues which links these definitions together.

Q: V composition
A: Expanding the product, we obtain "direct path", "individual attention head" and "virtual attention head" terms:
\[
\begin{align*}
T&=\operatorname{Id} \otimes W_U \cdot\left(I d+\sum_{h \in H_2} A^h \otimes W_{O V}^h\right) \cdot\left(I d+\sum_{h \in H_1} A^h \otimes W_{O V}^h\right) \cdot \quad \operatorname{Id} \otimes W_E \\
&= \operatorname{Id} \otimes W_U W_E \quad+\sum_{h \in H_1 \cup H_2} A^h \otimes\left(W_U W_{O V}^h W_E\right) \quad+\sum_{h_2 \in H_2 h_1 \in H_1}\left(A^{h_2} A^{h_1}\right) \otimes\left(W_U W_{O V}^{h_2} W_{O V}^{h_1} W_E\right)
\end{align*}
\]

Q: Q&K composition
A: The QK circuit matrix in the second layer is given by
\[
\begin{align*}
C_{Q K}^{h \in H_2} =& \quad \text{Q-side residual stream} \cdot W_{QK}^h \cdot \text{K-side residual stream} \\
=& \ \ \left(\operatorname{Id} \otimes \operatorname{Id} \otimes W_E^T+\sum_{h_q \in H_1} A^{h_{q}} \otimes \operatorname{Id} \otimes\left(W_{O V}^{h_0} W_E\right)^T\right) \cdot \quad \operatorname{Id} \otimes \operatorname{Id} \otimes W_{Q K}^h \\
& \cdot\left(\operatorname{Id} \otimes \operatorname{Id} \otimes W_E+\sum_{h_k \in H_1} \operatorname{Id} \otimes A^{h_k} \otimes W_{O V}^{h_k} W_E\right) \\
=& \quad \ \ \operatorname{Id} \otimes \operatorname{Id} \otimes\left(W_E^T W_{Q K}^h W_E\right) \qquad \text{(direct path)} \\
& +\sum_{h_q \in H_1} A^{h_q} \otimes \operatorname{Id} \otimes\left(W_E^T W_{O V}^{h_q T} W_{Q K}^h W_E\right) \quad \text{(Q-composition, K direct)} \\
& + \sum_{h_k \in H_1} \operatorname{Id} \otimes A^{h_k} \otimes\left(W_E^T W_{Q K}^h W_{O V}^{h_k} W_E\right) \quad ( \text{Q direct, K-composition})\\
& +\sum_{h_q, h_k \in H_1} A^{h_q} \otimes A^{h_k} \otimes\left(W_E^T W_{O V}^{h_q T} W_{Q K}^h W_{O V}^{h_k} W_E\right) (\text{Q&K composition})
\end{align*}
\]

Q: Induction head
A: Induction heads learn to predict patterns of the form \([a][b] … [a] \to [b]\).
Mechanistically:
- There is a key-vector forward-shifting effect of K-composition.
- There is a "same matching" QK circuit.
- There is a copying OV circuit to copy \(b\).

Q: Induction stripe
A: The diagonal pattern created by an induction head on a two-way table of attention by source/destination sequence position.

Q: Composition scores
A: If two heads are deliberately composing, then they will likely want to ensure they write and read from similar subspaces of the residual stream.
- Output space: \(W_O V=W_O W_V\)
- Input space:
  - \(W_Q K^T=W_K^T W_Q\) (for Q-composition)
  - \(W_Q K=W_Q^T W_K\) (for K Composition)
  - \(W_O V=W_O W_V\) (for V-Composition, of the later head).
We formalise overlap as
\[
\frac{\left|W_\text{out} W_\text{in}\right|_F}{\left|W_\text{out}\right|_F\left|W_\text{in}\right|_F}
\]
and must subtract a baseline, e.g. the overlap at model initialisation.