Loading
How an agent learns from reward. The course starts with Markov decision processes and discounted returns, then covers dynamic programming, temporal difference learning and Q-learning, exploration in bandits, policy gradients with advantages, and the deep Q-network.
01
An agent sees a state, takes an action, gets a reward and moves to a new state. Its goal is the discounted return, the sum of future rewards with each step worth a factor gamma less.
02
With a known model of the world, the Bellman equations give the value of a policy and the optimal value. Policy evaluation and value iteration apply them again and again until the values stop changing.
Practise it
03
Temporal difference methods learn from single transitions and use their own estimate of the next state as part of the target. Q-learning does this for action values with a max over the next actions, so it learns the greedy policy while it explores.
04
A multi-armed bandit is the simplest case of the trade-off between exploring and exploiting. UCB picks the arm with the best optimistic estimate, so arms that were tried less get a bonus.
Practise it
05
A policy network gives a probability for each action. REINFORCE raises the log probability of each action in proportion to the return after it, and a baseline cuts the variance of that estimate.
06
The advantage says how much better an action was than the state's usual value. Generalised advantage estimation mixes TD errors over many steps, with lambda to trade bias against variance.
Practise it
07
DQN replaces the Q-table with a neural network. A replay buffer and a slowly moving target network make the training stable, and Double DQN reduces the upward bias of the max.
Practise it