Chain Rule Through a Small Network

Differentiate a two step composition by hand.

The forward pass on a scalar input is

z = w1 * x + b1
h = tanh(z)
y = w2 * h + b2

Given a scalar grad_y = dL/dy, propagate it back to every parameter and to the input. The chain rule says multiply the incoming gradient by the local derivative at each step:

dL/dw2 = grad_y * h        dL/db2 = grad_y
dL/dh  = grad_y * w2
dL/dz  = dL/dh * (1 - tanh(z)^2)
dL/dw1 = dL/dz * x         dL/db1 = dL/dz        dL/dx = dL/dz * w1

The derivative of tanh is 1 - tanh(z)^2, which you can compute from h directly without touching z again. Saving the output rather than the input is exactly what real frameworks do for tanh and sigmoid.

Return a dict with keys "w1", "b1", "w2", "b2" and "x".

Function signature

def backward(x, w1, b1, w2, b2, grad_y):

Example

Input

backward(x=1.0, w1=1.0, b1=0.0, w2=1.0, b2=0.0, grad_y=1.0)

Output

{"w1": 0.41997, "b1": 0.41997, "w2": 0.76159, "b2": 1.0, "x": 0.41997}

Reasoning

z = 1, so h = tanh(1) = 0.76159, which is dL/dw2. dL/dz = 1 * 1 * (1 - 0.76159^2) = 0.41997. Because x = w1 = 1, the same value reaches w1, b1 and x.

Constraints and notes

• All inputs and outputs are scalars.
• Return a dict with the keys w1, b1, w2, b2 and x.