Loading
The calculus inside every training loop. The course covers derivatives and the chain rule, gradients and gradient descent, Jacobians and Hessians, how to check a derivative with finite differences, and how automatic differentiation computes all of it for you.
01
A derivative is the rate of change of a function at a point, the slope of its graph there. It tells you which way to move the input to make the output smaller.
02
The derivative of a composition is the product of the derivatives of its parts. Backpropagation is the chain rule applied from the loss back to every parameter.
03
The gradient collects the partial derivatives of a function of many inputs. It points uphill, so a small step against it makes the function smaller.
04
A function from vectors to vectors has a matrix of partial derivatives, the Jacobian. Backpropagation multiplies the incoming gradient by it without building it.
05
The Hessian holds the second derivatives and describes the curvature of a function. Its eigenvalues say how sharp a minimum is, and a Hessian vector product uses it without storing it.
06
A derivative can be estimated by evaluating the function at two nearby points. The estimate is slow and not exact, but it is the standard way to check a gradient you derived.
07
Autograd records each operation and runs the chain rule backwards through the record. A custom Function lets you write the backward rule for one operation yourself.