Loading
How an optimiser moves the weights downhill. The course starts with momentum and adaptive step sizes, then covers learning rate schedules, line search and second-order methods such as Newton and L-BFGS, and ends with how torch.optim puts these ideas into code.
01
Momentum keeps a running average of past gradients, so steps grow along a consistent direction and the zig-zag across a narrow valley cancels out. Nesterov momentum takes the gradient at the point where the momentum will carry the weights.
Practise it
02
Adaptive methods divide each gradient entry by a running estimate of its size, so every weight gets its own step size. AdamW keeps weight decay out of that division.
03
The best learning rate changes during training. A short warmup keeps the first steps stable, and a cosine decay lets the weights settle at the end.
Practise it
04
A line search picks the step size along a search direction. Backtracking shrinks the step until the loss goes down enough, and the Wolfe conditions also require the slope to flatten.
05
Newton's method uses the curvature to choose both the direction and the size of a step. L-BFGS estimates that curvature from recent gradients, so it works when the Hessian is too large to store.
06
Every torch.optim optimiser keeps parameter groups with their own settings and a state dict for each parameter, and updates the weights in place in step(). Schedulers change the learning rate stored in each group.