Q-Learning Update

Apply one tabular Q-learning update.

Q[s, a] estimates the return from taking action a in state s and acting greedily after that. After the agent takes action a in state s, gets reward r and lands in s_next, Q-learning moves Q[s, a] part of the way towards a target:

target  = r                              if done
        = r + gamma * max_a' Q[s_next, a']   otherwise
Q[s, a] = Q[s, a] + alpha * (target - Q[s, a])

The max makes Q-learning off-policy. It learns about the greedy policy even while the agent explores with a different one. When the episode ends there is no next state, so the target is just the reward.

Return a new table. Do not change the input.

Function signature

def q_update(Q, s, a, r, s_next, done, alpha, gamma):

Example

Input

Q = np.array([[0.0, 0.0],
              [1.0, 2.0]])
q_update(Q, s=0, a=1, r=1.0, s_next=1, done=False, alpha=0.5, gamma=0.9)

Output

array([[0. , 1.4],
       [1. , 2. ]])

Reasoning

The best value in state 1 is 2.0, so the target is 1 + 0.9 * 2 = 2.8. Q[0, 1] moves half way from 0 towards 2.8, which is 1.4. No other entry changes.

Constraints and notes

• Q has shape (n_states, n_actions).
• Only the single entry Q[s, a] changes.