Recurrent Cell Forward Pass

Run a vanilla recurrent cell over a sequence.

At each timestep t:

h_t = tanh(x_t @ W_x + h_(t-1) @ W_h + b)

x has shape (seq_len, d_in), W_x is (d_in, d_hidden), W_h is (d_hidden, d_hidden), b is (d_hidden,), and h0 is (d_hidden,).

Return the stacked hidden states with shape (seq_len, d_hidden), so row t is h_t. h0 itself is not part of the output.

The same W_x, W_h and b are used at every step. That weight sharing is what lets one set of parameters process a sequence of any length, and it is also the source of the problem: the gradient through t steps carries W_h to the power t, so it explodes or vanishes exponentially. That is what gradient clipping patches and what LSTM's cell state actually fixes.

Function signature

def rnn_forward(x, h0, W_x, W_h, b):

Example

Input

x  = np.array([[1.0], [0.0]])      # 2 steps, d_in = 1
h0 = np.zeros(1)
W_x = np.array([[1.0]])
W_h = np.array([[1.0]])
b  = np.zeros(1)
rnn_forward(x, h0, W_x, W_h, b)

Output

array([[0.76159416],
       [0.64201499]])

Reasoning

Step 1: h = tanh(1 + 0) = 0.7616. Step 2: the input is 0, so h = tanh(0.7616) = 0.6420. The hidden state carries the first input forward.

Constraints and notes

• Use the same weights at every step.
• Return shape (seq_len, d_hidden), without h0.