Loading
How a neural network computes and how it learns. The course starts with neurons, activations and backpropagation, then covers the losses and optimisers that train a network, the normalisation layers that keep training stable, and the recurrent and attention layers that handle sequences.
01
A layer is a linear map followed by a nonlinear activation. Without the activation, a deep network is only one linear map.
02
Backpropagation applies the chain rule from the loss back to every weight. Each layer multiplies the incoming gradient by its own local derivative.
03
Softmax turns logits into probabilities, and cross entropy scores them against the true class. Together they give a simple gradient: probabilities minus the one-hot target.
Practise it
04
Optimisers turn gradients into weight updates. Momentum smooths the direction, and Adam also scales each weight's step by its recent gradient size.
05
Normalisation layers rescale activations to mean zero and variance one, then apply a learnt scale and shift. Batch norm uses batch statistics and needs separate train and eval modes. Layer norm works on one sample at a time.
06
Clipping limits the size of a gradient step when gradients explode. Label smoothing stops a classifier from pushing its logits to extreme values.
07
A recurrent cell reads a sequence one step at a time and carries a hidden state forward. The LSTM adds gates and a cell state so that gradients survive over long sequences.
08
Attention lets each position take a weighted average of all the others, with weights from query and key dot products. Multi-head attention runs several of these in parallel on smaller projections.