Title: Mechanistic Interpretability

Q: Direction of max imbalance
A: Given a binary classification problem, the activation of the first class is
\[
\hat{y}_0=\operatorname{softmax}(\operatorname{logits}[0, 0]) = \operatorname{sigmoid}(\operatorname{logits}[0,0]-\operatorname{logits}[0,1])
\]
which is maximised in the \(\operatorname{logits}[0,0]-\operatorname{logits}[0,1]\) direction.
For any linear or approximately linear hidden layer \(\mathbf{x}^{(\ell)} = L\mathbf{x}^{(\ell - 1)}\) with direction of max imbalance \(\mathbf{v}^{(\ell)}\) we want to maximise
\[
\mathbf{v}^{(\ell)} \cdot \mathbf{x}^{(\ell)} = (\mathbf{v}^{(\ell)} \cdot L)\mathbf{x}^{(\ell - 1)}
\] so we pick \(\mathbf{v}^{(\ell - 1)} = \mathbf{v}^{(\ell)} \cdot L \).

Q: MLP contribution by neuron
A: Let \(\operatorname{MLP}(\vec{x})=\vec{d}+A f(B \vec{x}+\vec{c})\) for activation function \(f\), matrices \(A, B\) and let the number of neuron activations (or intermediate size) \(f(B \vec{x}+\vec{c})\) be \(n\). Then we can write
\[
\operatorname{MLP}(\vec{x})=\vec{d}+\sum_{i=0}^{n-1} f\left(B_{[i,:]]} \cdot \vec{x}+c_i\right) A_{[:, i]}
\]
prompting the following definitions:
- the \(i\) th row of \(B\) is the in-direction of neuron \(i\)
- the ith column of \(A\) is the out-direction of neuron \(i\).
Then we see that each neuron reads in-direction from \(\vec{x}\) and writes a scaled version of out-direction to the stream.

Q: Direct Logit attribution
A: A consequence of the residual stream is that the output logits are the sum of the contributions of each layer, and thus the sum of the results of each head.
This means we can decompose the output logits into a term coming from each head and directly do attribution like this.

Q: Ablation
A: An ablation is a simple causal intervention on a model - we pick some part of it and set it to zero.
This is a crude proxy for how much that part matters.