Title: neuralnetworksanddeeplearning

Q: Perceptron
A: A perceptron is a neuron with binary activation function
\[a(\mathbf{x}, \mathbf{w}) = \mathbb{1}[\mathbf{x}\cdot\mathbf{w} > 0].\]

Q: Weighted input
A: The weighted input to a layer \(l\) is
\[\mathbf{z}^l = W^l \mathbf{a}^{l-1}+\mathbf{b}^l,\]i.e. the input to the activation function.

Q: Neuron error
A: We define the error \(\delta_j^l\) of neuron \(j\) in layer \(l\) by
\[\delta_j^l = \frac{\partial C}{\partial z_j^l}.\]

Q: Error in the output layer
A: The components of \(\delta^L\) are given by the chain rule as
\[\begin{align*} \delta_j^L &= \frac{\partial C}{\partial a_j^L} \sigma'(z_j^L) \\ \boldsymbol\delta^L &= \nabla_{a^L} C \odot \sigma'(\mathbf{z}^L). \end{align*}\]

Q: Induction equation for network error
A: To flow errors backwards through the network we have
\[\boldsymbol\delta^l = ((W^{l+1})^T\boldsymbol\delta^{l + 1} \odot \sigma'(\mathbf{z}^l).\]This derives from the chain rule using
\[\frac{\partial z_i^{l+1}}{\partial z_j^l} = w^{l + 1}_{ij} \sigma'(z_j^l).\]

Q: Derivative of cost wrt bias
A: From the chain rule, since the weighted input depends linearly on the bias we have
\[\begin{align*} \frac{\partial C}{\partial b_j^l} &= \delta_j^l. \end{align*}\]

Q: Derivative of cost wrt weight
A: From the chain rule, since the weighted input multiplies the weights by the activations in the previous layer:
\[\frac{\partial C}{\partial w^l_{jk}} = \delta^l_ja^{l-1}_k.\]

Q: Saturation
A: When a logistic neuron has activation close to 0 or 1, it will learn slowly.

Q: Backpropagation algorithm
A: We compute the gradient of the cost function as follows:
1. Feedforward: Starting from the input activations, propagate the activations forward theedrough the network
2. Output error: From the output error, compute the vector \(\boldsymbol\delta^L = \nabla_{a^L} C \odot \sigma'(\mathbf{z}^L)\).
3. Backpropagate the error: For each \(l=L-1, \ldots 2, 1\) feed the error backwards using \(\boldsymbol\delta^l = ((W^{l+1})^T\boldsymbol\delta^{l + 1} \odot \sigma'(z_l)\).
4. Output: The gradient of the cost function is given by \(\frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k\delta^l_j\) and \(\frac{\partial C}{\partial b_j^l} = \delta_j^l\).
You can then apply gradient descent to minimise the cost.

Q: Dropout
A: A regularisation technique for neural networks where we randomly and temporarily drop half of the hidden neurons at each step in gradient descent. Finally we halve the weights for all of the hidden neurons to compensate.

Q: Initiliasing neuron weights
A: To ensure that weighted inputs are sharply peaked at 0 and avoid saturation, we initialise a neuron with \(n\) inputs by sampling from \(N\left(0, \frac{1}{n}\right)\) such that the weighted inputs have standard deviation \(O(1)\).

Q: Stochastic gradient descent
A: Gradient descent is applied to random mini-batches to speed up learning

Q: Convolutional layer
A: Given a \(P\times P\) input image, we can feed these activations into an initial \(\frac{P-p+1}{s}\times \frac{P-p+1}{s}\) hidden layer where \(p<< P\) is the size of the overlapping local receptive fields and \(s\) is the stride length.
Each neuron in this first hidden layer has shared weights and biases based on the assumption of translation invariance of the feature map we are trying to learn.
Hence the output of neuron \((j, k)\) is given by the convolution
\[\begin{align*} a_{jk}^1 &= \sigma\left(b + \sum_{l=0}^{p-1} \sum_{m=0}^{p-1} w_{l,m} a_{js+l, ks+m} \right) \\ a^1 &= \sigma(b + w * a^0). \end{align*}\]

Q: Pooling layer
A: We condense the information from the convolution layers by aggregating the activations over small partitions of the image, reducing the dimensions of the weights.
Common aggregation functions are to take the maximum or the square root of the sum of squares (L2 pooling).

Q: Convolution neural network
A: Input neurons \(\to\) convolutional layer of multiple parallel feature maps \(\to\) pooling layer \(\to\)dense output layer.