RMSProp

Write one update step of the RMSProp optimiser.

RMSProp gives every parameter its own step size. It keeps a running average of the squared gradient for each parameter. A parameter whose gradients are large gets its step divided by a large number, and a parameter whose gradients are small gets a relatively bigger step. The result is that all parameters move at a similar rate.

sq_avg_new = rho * sq_avg + (1 - rho) * grad ** 2
params_new = params - lr * grad / (sqrt(sq_avg_new) + eps)

Return the new parameters and the new squared average.

Function signature

def rmsprop_update(params, grads, sq_avg, lr=0.01, rho=0.9, eps=1e-8):

Example

Input

params = np.array([1.0, 2.0])
grads  = np.array([0.1, -0.2])
sq_avg = np.zeros(2)
rmsprop_update(params, grads, sq_avg, lr=0.01)

Output

(array([0.96837722, 2.03162278]), array([0.001, 0.004]))

Reasoning

The squared average becomes 0.1 * [0.01, 0.04] = [0.001, 0.004]. Its square roots are [0.0316, 0.0632], so each gradient divided by its root is about +3.162 and -3.162. Both parameters move by the same amount, 0.0316, even though one gradient is twice the other. That equal step is the point of RMSProp.

Constraints and notes

• All arrays have the same shape.
• Add eps after the square root, not inside it.
• Return new arrays. Do not modify the inputs in place.