The guide
Every model in this course is trained by an optimiser, and the optimiser decides how fast it trains and sometimes whether it trains at all. Plain gradient descent is the starting point. The rest is a set of fixes for the ways it fails.
Momentum fixes the zig-zag in narrow valleys by averaging gradients over time. RMSProp fixes the problem that one learning rate does not suit every parameter. Learning rate schedules fix the early steps, when the estimates are poor, and the late steps, when you want to settle.
The last two problems come from classical optimisation. Line search picks the step size by testing it, and Newton's method uses curvature to jump to the minimum. Both are too expensive for large networks, but they explain what the cheaper methods are approximating.