Numerical Gradient

Estimate the gradient of a scalar function using central differences.

df/dx_i ≈ (f(x + eps * e_i) - f(x - eps * e_i)) / (2 * eps)

Perturb one coordinate at a time, leaving the rest alone, and collect the results into an array with the same shape as x.

Use the central difference above, not the one-sided (f(x + eps) - f(x)) / eps. The central version has error of order eps^2 instead of eps, which is why it is what everyone uses for gradient checking.

This is how you catch a wrong backward pass: compare your analytic gradient against this, and if they disagree the analytic one is usually wrong.

Do not modify the caller's x.

Function signature

def numerical_gradient(f, x, eps=1e-5):

Example

Input

numerical_gradient(lambda v: np.sum(v ** 2), np.array([1.0, 2.0]))

Output

array([2., 4.])

Reasoning

The gradient of sum(v^2) is 2v. Central differences are exact for a quadratic, so the estimate is [2, 4] up to rounding.

Constraints and notes

• Use the central difference, not a one sided one.
• Return an array with the same shape as x.
• Do not change the caller's x.