Title: Hardware

Q: Data transfer cost
A: Time spent transferring data between CPU and GPU

Q: Network costs
A: Time spent transferring data from one node to another

Q: Memory Bandwidth cost
A: Time spent transferring data from:
- CUDA global memory to CUDA shared memory
- DRAM to SRAM.

Q: SRAM
A: - Static RAM
- Made up of flip-flops (transistor-based circuit to store bits)
- Faster and more expensive than DRAM
- used for specialised applications, e.g. the cache and internal registers of a CPU

Q: DRAM
A: - Dynamic RAM
- stores data using capacitors
- used for a computer's main memory
- responsible for "CUDA Out of Memory' errors
- located off-chip

Q: GDDR
A: Graphics Double Data Rate memory is a type of DRAM designed for high-bandwidth applications, e.g. GPUs. Double Data Rate can deliver two bits per tick.

Q: ALU
A: - arithmetic logic unit
- arithmetic/bitwise operations on integer binary numbers
- e.g. tensor cores

Q: SMs
A: - Streaming Multiprocessors
- processors with a low clock rate target and a small cache
- able to execute several thread blocks in parallel
- an array of these make up the CUDA architecture
- 1 SM = 8 CUDA cores = 32 threads
- located on-chip with the SRAM

Q: Memory-bound operation
A: Compute time is constrained by memory-bandwidth

Q: Operator fusion
A: - performing multiple operations in SRAM before moving back to DRAM to reduce bandwidth cost
- can write custom CUDA kernels to benefit from this

Q: Compute intensity
A: amount of compute for fixed number of memory accesses

Q: Runtime
A: time spent on actual computation, not memory transfers

Q: Overhead
A: - Time spent NOT on transfers or compute
- overhead-bound if runtime doesn’t scale with data size

Performance bottlenecks and solutions for GPUs

Q: Memory bandwidth calculation
A: - (Memory clock GHz x Pump Rate x Bus Width bits / 8) * GDDR type multiplier = Bandwidth in GB/s
- Pump rate is 2 for DDR
- Bus width is the number of parallel connections between the processor and memory
- Factor of 8 to convert bus width from bits to bytes
- GDDR type multiplier is 2 for GDDR3, 4 for GDDR5.
- e.g. GTX-Titan: 3 Gbps * 2 * 384 / 8 = 288 GB/s

Q: Data Parallelism
A: - copy the same model weights into multiple workers and assign a fraction of data to each worker
- Bulk synchronous parallels (BSP)
  - Workers sync data at the end of every minibatch.
  - It prevents model weights staleness
  - but each machine has to halt and wait for others to send gradients.
- Asynchronous parallel (ASP)
  - Every GPU worker processes the data asynchronously
  - no waiting or stalling
  - However, it can easily lead to stale weights
  - Even though it increases the computation time, it may not speed up training time to convergence.
- gradient accumulation in Distribution Data Parallel (DDP)
  - middle ground: synchronize gradients globally once every x iterations

Q: Model Parallelism
A: - aims to solve the case when the model weights cannot fit into a single node
- computation and model parameters are partitioned across multiple machines
- sequential dependency leads to large “bubbles” of idle time

Q: Pipeline Parallelism
A: - combines MP and DP
- The main idea is to split one minibatch into multiple microbatches and enable each stage worker to process one microbatch simultaneously
- Common implementations are GPipe and PipeDream

Q: Tensor Parallelism
A: One tensor operation is split across multiple nodes

Q: Mixture of Experts
A: - Structure
  - feed-forward networks as experts \(\{E_i\}_{i=1}^n\)
  - A trainable gating network G to learn a probability distribution over n experts so as to route the traffic to a few selected experts, e.g. G(x) = softmax(xW_g)
- Auxiliary loss
  - to avoid some weak learners from being used
  - sum of sigma^2/mu^2
- shrinking batch problem
  - every expert network only gets a fraction of training samples
  - try to use a batch size as large as possible