Title: Generative models

Q: Generator network
A: - models \(p(X, Y)\)
- generates negative samples for discriminator
- network input is noise, sample space is called latent dimension or latent space and is generally \(N(0, 1)\)

Q: Discriminator network
A: - models \(p(Y|X)\)
- called critic if outputs logits instead
- architecture is mirror of generator, with convolutions instead of transposed convolutions

Q: Minimax loss
A: - \(\mathbb{E}_x[\log(D(x))]+\mathbb{E}_z[\log(1−D(G(z)))]\)
- LHS represents the discriminators true positives
- RHS represents the discriminators true negatives
- $D$ maximises, $G$ minimises

Q: Wasserstein loss
A: - Critic \(D(x) - D(G(z))\)
- Generator \(D(G(z))\)
- weights must be clipped
- better convergence

Q: Transposed convolution
A: - Upsampling/decompressing version of convolutions
- Equivalent to a convolution with \(k-1-p\) padding of input, spacing of input equal to \(s\)
- \(o=is+k-1-2p\)
- AKA fractionally-strided convolutions

Q: Auto-encoder
A: - Learn a compression of high dimensional data
- Structure: \(x\) -> encoder -> \(z\) -> decoder -> \(x'\)
- Objective: \(\min d(x, x')\), e.g. L2 loss, known as reconstruction loss
- Add shocks to inputs, keep target fixed
- For images, the encoder is a convolution and the decoder is a transposed convolution

Q: Sparse autoencoder
A: - The average activation of a neuron is \(\hat{\rho}_j^{\ell}=\sum_{i=1}^n a_j^{\ell}\left(\mathbf{x}_i\right)\)
- For a given sparsity parameter, commonly \(\rho=0.05\), we add a KL penalty to the loss to enforce \(\hat{\rho}_j^{\ell} \approx \rho\)

Q: \(k\)-sparse auto-encoder
A: Only keeps the top \(k\) activations in the bottleneck layer

Q: Contractive auto-encoder
A: Penalises the Frobenius norm of the encoder activations, i.e. the sum of squares of the partial derivatives, to reduce sensitivity to input

Q: VAE
A: Variational Autoencoder replaces the deterministic encoder/decoder \(\mathbf{x}\mapsto \mathbf{z}\mapsto\mathbf{x}\) with
- The likelihood function \(p_\theta(\mathbf{x} | \mathbf{z})\) is the probabilistic decoder
- The estimated posterior \(q_\phi(\mathbf{z} | \mathbf{x})\) is the probabilistic encoder, used as integrating to get the true posterior is intractable

Q: ELBO
A: We want to maximise the log-likelihood or evidence:
\[
\begin{aligned}
p(x) & =\int_z p_\theta(x \mid z) p(z) \\
\log p(x) & \geq\mathbb{E}_{z \sim p(z)}\left[\log p_\theta(x \mid z)\right] \quad \text{(Jensen)}
\end{aligned}
\]
but this becomes intractable with dimensionality of \(z\). Instead we learn a model for the posterior
\[
\begin{aligned}
p(x) & =\int q_\phi(z \mid x) \frac{p_\theta(x \mid z) p(z)}{q_\phi(z \mid x)} \\
\log p(x) & \geq \mathbb{E}_{z \sim q_\phi(z \mid x)}\left[\log\frac{p_\theta(x \mid z) p(z)}{q_\phi(z \mid x)}\right] \\
&=: \text{ELBO}
\end{aligned}
\]
such that we are sampling from the much smaller space of likely latent vectors to decode to \(x\).
We then seek to maximise this Evidence Lower Bound, often in the alternate form
\[
\begin{align*}
\text{ELBO} &= \mathbb{E}_{\mathbf{z} \sim q_\phi(\mathbf{z} \mid \mathbf{x})}\left[\log p_\theta(\mathbf{x} \mid \mathbf{z}) - \log \frac{q_\phi(\mathbf{z} \mid \mathbf{x})}{p_\theta(\mathbf{z})}\right] \\
&=\mathbb{E}_{\mathbf{z} \sim q_\phi(\mathbf{z} \mid \mathbf{x})} \log p_\theta(\mathbf{x} \mid \mathbf{z})-D_{\mathrm{KL}}\left(q_\phi(\mathbf{z} \mid \mathbf{x}) \|p_\theta(\mathbf{z})\right) \\
&=\text{reconstruction loss} - \text{regularisation term}
\end{align*}
\]
We must use reverse KLD for the penalty term so that the true distribution determines the domain.

Q: KLD of probabilistic encoder
A: We can show that the \(D_\text{KL}\) of the posteriors is the magnitude of the error in the estimate:
\[
\begin{aligned}
& D_{\mathrm{KL}}\left(q_\phi(\mathbf{z} \mid \mathbf{x}) \| p_\theta(\mathbf{z} \mid \mathbf{x})\right) \\
=&\int q_\phi(\mathbf{z} \mid \mathbf{x}) \log \frac{q_\phi(\mathbf{z} \mid \mathbf{x})}{p_\theta(\mathbf{z} \mid \mathbf{x})} d \mathbf{z} \\
=&\int q_\phi(\mathbf{z} \mid \mathbf{x}) \log \frac{q_\phi(\mathbf{z} \mid \mathbf{x}) p_\theta(\mathbf{x})}{p_\theta(\mathbf{z}, \mathbf{x})} d \mathbf{z} \\
=&\int q_\phi(\mathbf{z} \mid \mathbf{x})\left(\log p_\theta(\mathbf{x})+\log \frac{q_\phi(\mathbf{z} \mid \mathbf{x})}{p_\theta(\mathbf{z}, \mathbf{x})}\right) d \mathbf{z} \\
=&\log p_\theta(\mathbf{x})+\int q_\phi(\mathbf{z} \mid \mathbf{x}) \log \frac{q_\phi(\mathbf{z} \mid \mathbf{x})}{p_\theta(\mathbf{z}, \mathbf{x})} d \mathbf{z} \\
=& \text{evidence} - \text{ELBO} .
\end{aligned}
\]

Q: Reparameterisation trick for VAEs
A: We remove the sampling from the loss function to allow backpropagation as follows:
\[
\begin{aligned}
& \mathbf{z} \sim q_\phi\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)=\mathcal{N}\left(\mathbf{z} ; \boldsymbol{\mu}^{(i)}, \boldsymbol{\sigma}^{2(i)} \boldsymbol{I}\right) \\
& \mathbf{z}=\boldsymbol{\mu}+\boldsymbol{\sigma} \odot \boldsymbol{\epsilon}, \text { where } \boldsymbol{\epsilon} \sim \mathcal{N}(0, \boldsymbol{I}).
\end{aligned}
\]
If \(\sigma=0\), then this reduces to an autoencoder so it is important to have the KL penalty to ensure \(\sigma \approx 1\).

Q: Hierarchical VAEs
A: Stacking VAEs gives an extra consistency term in the ELBO loss
\[
\begin{align*}
&\mathbb{E}_{q_\phi\left(z_1, z_2 \mid x\right)}\left[\log \frac{p_\theta\left(x, z_1, z_2\right)}{q_\phi\left(z_1, z_2 \mid x\right)}\right] \\
= \ &\underbrace{\mathbb{E}_{q_\phi\left(z_1 \mid x\right)}\left[\log p_\theta\left(x \mid z_1\right)\right]}_{\text {reconstruction loss }}-\underbrace{D_{\mathrm{KL}}\left(q_\phi\left(z_1 \mid x\right) \| p_\theta\left(z_1 \mid z_2\right)\right)}_{\text {consistency term }}-\underbrace{D_{\mathrm{KL}}\left(q_\phi\left(z_2 \mid z_1\right) \| p\left(z_2\right)\right)}_{\text {regularisation term }}.
\end{align*}
\]
In general:
\[
\begin{align*}
L(\theta) = & \mathbb{E}_q[-\log p_\theta\left(\mathbf{x}_0 \mid \mathbf{x}_1\right)] \\
+ &\sum_{t=2}^T D_{\mathrm{KL}}\left(q\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0\right) \| p_\theta\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t\right)\right) \\
+ & D_{\mathrm{KL}}\left(q\left(\mathbf{x}_T \mid \mathbf{x}_0\right) \| p\left(\mathbf{x}_T\right)\right) \\
= & \underbrace{L_0}_{\text {reconstruction loss }}+\underbrace{\sum_{t=2}^T L_t}_{\text{consistency}}+\underbrace{L_T}_{\text {regularisation term }}.
\end{align*}
\]

Q: Diffusion models: high-level summary
A: A hierarchical VAE with a pre-defined forward process \(q_\phi\) which is adding noise at each step, until \(\mathbf{x}_T\) is uniform noise so the final regularisation term approximates to 0.
Then we just reverse the 2nd law of thermodynamics to learn the backward process.

Q: Diffusion models: forward process
A: The forward process is given by
\[
\mathbf{x}_t=\sqrt{1-\beta_t} \mathbf{x}_{t-1}+\sqrt{\beta_t} \delta_t \text { where } \delta_t \sim \mathcal{N}(\mathbf{0}, \mathbf{I})
\]
where \(\left(\beta_t\right)\) are fixed parameters making up the variance schedule, usually increasing as we get closer to max entropy.
The product of these simplifies to (using bars)
\[
\mathbf{x}_t=\sqrt{\bar{\alpha}_t} \mathbf{x}_0+\sqrt{1-\bar{\alpha}_t} \epsilon_t, \quad \epsilon_t \sim \mathcal{N}(\mathbf{0}, \mathbf{I})
\]
where \(\alpha_t=1-\beta_t\), and \(\bar{\alpha}_t=\prod_{s=1}^t \alpha_s\).

Q: Diffusion models: loss function
A: We can simplify the consistency term in the loss function as (using tilde)
\[
\begin{gathered}
q\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0\right)=\mathcal{N}\left(\mathbf{x}_{t-1} ; \tilde{\boldsymbol{\mu}}\left(\mathbf{x}_t, \mathbf{x}_0\right), \tilde{\beta}_t \mathbf{I}\right) \\
\tilde{\beta}_t=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t} \cdot \beta_t \\
\tilde{\boldsymbol{\mu}}_t=\frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t-\frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}} \boldsymbol\epsilon_t\right)
\end{gathered}
\]
where \(\boldsymbol\epsilon_t\) is known given \(\mathbf{x}_t, \mathbf{x}_0\) are fixed.
By picking \(\boldsymbol{\Sigma}_\theta=\tilde{\beta}_t \mathbf{I}\) to match variances, this gives \(D_{\mathrm{KL}} \propto \left\|\tilde{\boldsymbol{\mu}}_t-\boldsymbol{\mu}_\theta\right\|^2\).
It helps to reparameterise using the noise \(\boldsymbol\epsilon_t\) for which we define a model \(\boldsymbol\epsilon_\theta\left(\mathbf{x}_t, t\right)\) as
\[
\begin{aligned}
\boldsymbol{\mu}_\theta & =\frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t-\frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}} \boldsymbol\epsilon_\theta\left(\mathbf{x}_t, t\right)\right) \\
& =\frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t-\frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}} \boldsymbol\epsilon_\theta\left(\sqrt{\bar{\alpha}_t} \mathbf{x}_0+\sqrt{1-\bar{\alpha}_t} \boldsymbol\epsilon_t, t\right)\right)
\end{aligned}
\]
and then the loss becomes
\[
L_t^{\text {simple }}=\left\|\boldsymbol\epsilon_t-\boldsymbol\epsilon_\theta\left(\sqrt{\bar{\alpha}_t} \mathbf{x}_0+\sqrt{1-\bar{\alpha}_t} \boldsymbol\epsilon_t, t\right)\right\|^2 .
\]

Q: Diffusion models: Gradient Descent
A: We sample $t$ uniformly and then perform a gradient descent step using loss function
\[
L_t^{\text {simple }}=\left\|\boldsymbol{\epsilon}_t-\boldsymbol{\epsilon}_\theta\left(\sqrt{\bar{\alpha}_t} \mathbf{x}_0+\sqrt{1-\bar{\alpha}_t} \boldsymbol{\epsilon}_t, t\right)\right\|^2.
\]

Q: CLIP
A: - Contrastive Language-Image Pre-Training is a model trained on text-image pairs
- Contrastive loss: the average of the row-wise/column-wise cross-entropy loss for the image-text cosine similarity matrix

Q: U-Net
A: CNN consisting of a contracting path and an expansive path, which gives it the u-shaped architecture.
Contracting path: repeat convolution, ReLU, max-pooling operation.
Expansive pathway: transposed convolutions and concatenations with high-resolution features from the contracting path

Q: Stable diffusion
A: - Text encoding is done using a frozen CLIP text encoder, i.e. fully pre-trained
- The vision transformer part of CLIP is not used
- U-Net downsamples to a smaller latent space and conditions on the text
- The image encoder/decoder is a VAE

Q: Classifier guidance
A: Unconditional diffusion models move in the direction of the score function \(\nabla_x\log p(x)\). In conditional problems, we have a label \(y\) so using Bayes this becomes
\[
\nabla_x \log p(x \mid y)=\nabla_x \log p(y \mid x)+\nabla_x \log p(x).
\]
We recover the unconditional term plus a conditioning term which is naturally a classifier, which must be trained separately.
We can also sample using temperature \(\gamma^{-1}\) where \(p_\gamma(x \mid y) \propto p(x) \cdot p(y \mid x)^\gamma\) which gives
\[
\nabla_x \log p_\gamma(x \mid y)=\nabla_x \log p(x)+\gamma \nabla_x \log p(y \mid x)
\] where \(\gamma\) is called the guidance scale.

Q: Classifier-free guidance
A: Substituting Bayes into the classifier guidance formula to remove the classifier term:
\[
\nabla_x \log p_\gamma(x \mid y)=(1-\gamma) \nabla_x \log p(x)+\gamma \nabla_x \log p(x \mid y) .
\]
Now we can train RHS functions simultaneously using conditioning dropout: replace \(y\) with a mask token some (usually 10-20%) of the time.

Q: GAN
A: Generative Adversarial Networks consist of two models: a generator and discriminator.
These models are trained together using minimax or Wasserstein loss to encourage the generator to produce examples which are difficult to discriminate.