Title: pytorch

Q: Tensor dimensions
A: 1. Batch size
2. Channel, e.g. 3 for RGB
3. Height
4. Width
5. Depth

Q: Padding
A: Number of pixels concatenated to an image before processing the kernel of a CNN.
e.g. zero-padding concatenates zeros to an axis

Q: Output dimension of CNN
A: For input dimension \(i\), kernel size \(k\), padding \(p\), stride \(s\), the output dimension is
\[o = \left\lfloor\frac{i - k + 2p}{s}\right\rfloor + 1.\]Example: half-padding or same-padding, ensures that \(o=i\) by setting \(k=2p+1\) and unit-strides.
Example: full-padding sets \(p=k-1\) and unit-strides such that \(o=i+k-1\).

Q: Convolution layer constructor
A: The constructor
1 | nn.Conv2d
takes args
1. in_channels (int): Number of channels in the input image
2. out_channels (int): Number of channels produced by the convolution
3. kernel_size (int or tuple): Size of the convolving kernel
4. stride (int or tuple, optional): Stride of the convolution. Default: 1
5. padding (int, tuple or str, optional): Padding added to all four sides of the input. Default: 0
6. padding_mode (string, optional): ``'zeros'``, ``'reflect'``, ``'replicate'`` or ``'circular'``. Default: ``'zeros'``

Q: Pooling layer constructor
A: The constructor
1 | nn.MaxPool2d
takes args
1. kernel_size: the size of the window to take a max over
2. stride: the stride of the window. Default value is :attr:`kernel_size`
3. padding: implicit zero padding to be added on both sides.

Q: Dense layer constructor
A: Constructor
1 | torch.nn.Linear(in_features, out_features, bias=True, device=None, dtype=None)
takes args
1. in_features – size of each input sample
2. out_features – size of each output sample
3. bias – If set to False, the layer will not learn an additive bias. Default: True.

Q: Net constructor
A: We construct a custom neural net as follows
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import torch.nn as nn
import torch.nn.functional as F
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(...
    def forward(self, x):
        x = F.relu(self.layer1(x))
        ...
        return x

Q: Stochastic gradient descent constructor
A: For a Classification Cross-Entropy loss and SGD with momentum:
1
2
3
4
import torch.optim as optim
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
This takes args
1. params (iterable): iterable of parameters to optimize or dicts defining parameter groups
2. lr (float): learning rate
3. momentum (float, optional): momentum factor (default: 0) for soft updates to gradient vector.

Q: Basic training loop
A: Looping over the data for each epoch:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
net = net.to(device=device)
for epoch in range(n_epochs):  # loop over the dataset multiple times
    for i, (inputs, labels) in enumerate(trainloader, 0):
        inputs = inputs.to(device=device)
        labels = labels.to(device=device)
        # zero the parameter gradients for hard updates
        optimizer.zero_grad()
        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

Q: Accuracy on test data
A: Performance on test data:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
correct = 0
total = 0
# since we're not training, we don't need to calculate the gradients for our outputs
with torch.no_grad():
    for data in testloader:
        images, labels = data
        # calculate outputs by running images through the network
        outputs = net(images)
        # the class with the highest activation is what we choose as prediction
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
print(f'Accuracy of the network on test images: {100 * correct // total} %')

Q: Concatenating mini-batch of images
A: To create a grid of images
1
2
3
4
5
6
torchvision.utils.make_grid(
	tensor: Union[torch.Tensor, List[torch.Tensor]],
	nrow: int = 8,
	padding: int = 2,
	**kwargs
) → torch.Tensor
which takes args
- tensor (Tensor or list) – 4D mini-batch Tensor of shape \(B \times C \times H \times W\) or a list of images all of the same size.
- nrow (int , optional) – Number of images displayed in each row of the grid. The final grid size is (B / nrow, nrow). Default: 8.
- padding (int , optional) – amount of padding. Default: 2.
This returns a tensor of dimensions \(C \times (H+2p) \times (B(W+p)+p)\).

Q: einops
A: - Use brackets to merge or de-merge dimensions
- Need an aggregation method to reduce
- Need to specify number of repeats
1
2
3
4
5
6
7
8
from einops import reduce, rearrange, repeat
# compose a new dimension of batch and width
rearrange(ims, 'b h w c -> h (b w) c')
# average over batch
reduce(ims, 'b h w c -> h w c', 'mean')
# repeat along a new axis
repeat(ims[0], 'h w c -> h new_axis w c', new_axis=5)
repeat(ims[0], 'h w c -> h 5 w c')

Q: as_strided
A: Create a view of an array where you specify the new shape and the number of bytes to stride in each axis.
N.B. use np.ndarray.itemsize to get number of bytes per item
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
from numpy.lib.stride_tricks import as_strided
def as_strided(x, shape=None, strides=None):
    """
    Parameters
    ----------
    x : ndarray
        Array to create a new.
    shape : sequence of int, optional
        The shape of the new array. Defaults to ``x.shape``.
    strides : sequence of int, optional
        The strides of the new array. Defaults to ``x.strides``.
    Returns
    -------
    view : ndarray
    """
    ...

Q: Fetch MNIST data
A: To download the data, convert PIL images to tensor and normalise:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))
])
trainset = datasets.MNIST(root="./data", train=True, transform=transform, download=True)
trainloader = DataLoader(trainset, batch_size=64, shuffle=True)

Q: Train and Eval Modes
A: In training mode, batch norm uses the batch mean and variance. Also: update a running_mean and running_var.
In eval mode, batch norm uses the running mean and variance.

Q: Basic training loop with wandb
A: Initialising the run, watching the model, logging the loss and saving the final model:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
def train(config: dict) -> None:
    start_time = time.time()
    wandb.init(project="project_name", config=config)
    wandb.watch(model, criterion=loss_fn, log="all", log_freq=10, log_graph=True)
    wandb.log({"elapsed": time.time() - start_time}, step=examples_seen)
    ...
    filename = f"{wandb.run.dir}/model_state_dict.pt"
    wandb.save(filename)

Q: Sweeps using wandb
A: To create a sweep and run twice in parallel:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
def train(config: dict = None) -> None:
    wandb.init()
    if config is None:
        config = wandb.config
    ...
sweep_config = {
    'method': 'random',
    'metric': {'name': 'test_accuracy', 'goal': 'maximize'},
    'parameters':
    {
        'batch_size': {'values': [64, 128, 256]},
        'epochs': {'min': 1, 'max': 3},
        'lr': {'max': 0.1, 'min': 0.0001, 'distribution': 'log_uniform_values'}
     }
}
sweep_id = wandb.sweep(sweep=sweep_config, project='w2d1_resnet')
wandb.agent(sweep_id=sweep_id, function=train, count=2)

Q: Multinomial sampling
A: 1 | torch.distributions.categorical.Categorical(probs=None, logits=None, validate_args=None)
Creates a categorical distribution parameterized by either probs or logits (but not both).
1
2
m = Categorical(torch.tensor([ 0.25, 0.25, 0.25, 0.25 ]))
m.sample()  # equal probability of 0, 1, 2, 3

Q: nn.Embedding
A: A simple lookup table that stores embeddings of a fixed dictionary and size. Parameters:
- num_embeddings (int) – size of the dictionary of embeddings
- embedding_dim (int) – the size of each embedding vector.
Variables:
- weight (Tensor) – the learnable weights of the module of shape (num_embeddings, embedding_dim) initialized from \(N(0,1)\).

Q: nn.utils.clip_grad_norm_
A: Clips gradient norm of an iterable of parameters.
The norm is computed over all gradients together, as if they were concatenated into a single vector. Gradients are modified in-place.
Parameters:
- parameters (Iterable[ Tensor ] or Tensor) – an iterable of Tensors or a single Tensor that will have gradients normalized
- max_norm (float or int) – max norm of the gradients
- norm_type (float or int) – type of the used p-norm. Can be 'inf' for infinity norm.

Q: Copying weights
A: 1 | self.load_state_dict(dict(other.named_parameters())

Q: Parameter groups
A: To use different hyperparameters for different subsets of parameters SGD takes a list of dictionaries
1
2
3
4
torch.optim.SGD([
    {'params': model.base.parameters()},
    {'params': model.classifier.parameters(), 'lr': 1e-3}
], lr=1e-2, momentum=0.9)

Q: Hooks
A: Hooks are used to record or modify hidden layer activations
1
2
3
hook(module, input, output) -> Union[None, modified_output: t.Tensor]
torch.nn.modules.module.register_module_forward_hook(hook) -> torch.utils.hooks.RemovableHandle
torch.utils.hooks.RemovableHandle.remove() # removes the hook

Q: Type checking
A: The following will throw an error if the input dimensions do not match:
1
2
3
4
from torchtyping import TensorType as TT
@typechecked
def function(tensor: TT["dim1", "dim2": dim2]):
    pass