Loading
The parts of a modern decoder-only language model and how it runs. The course covers the GPT block, causal attention, rotary position embeddings, RMSNorm and SwiGLU, the KV cache for fast generation, sampling methods, and LoRA fine-tuning.
01
A GPT model is token and position embeddings, a stack of attention and MLP blocks with residual connections, and a head that predicts the next token.
Practise it
02
Each position attends only to itself and earlier positions, so the model can train on every next token at once. Many heads run in parallel on slices of the vector.
03
RoPE rotates pairs of query and key features by an angle that grows with position. The dot product then depends only on the distance between two tokens.
Practise it
04
RMSNorm scales a vector by its root mean square and skips the mean. SwiGLU is a gated feed-forward layer. Llama and most recent models use both.
Practise it
05
When a model generates one token at a time, the keys and values of earlier tokens do not change. A cache keeps them, so each step only computes the new token.
Practise it
06
Greedy decoding takes the most likely token. Temperature, top-k and top-p sampling trade some likelihood for variety.
07
LoRA freezes the pretrained weights and learns a small low-rank update for each layer. It trains a tiny share of the parameters.
Practise it