• Formulating a problem as an MDP
  • Identifying the states, actions, reward function, transition probabilities and discount factor, and then drawing the state transition diagram.
• Finite-horizon MDP: $\max_\pi \mathbb{E}\left[ \sum_{t=0}^{T} \gamma^t R_{t+1} \right], \quad \pi_t(s)$
  • An MDP restricted to a fixed, finite number of steps.
  • Every episode must reach an end state and stop, so the return can't blow up and gamma can be one.
  • Its optimal policy is time-dependent, or non-stationary, because the best action can depend on how many steps are left.
  • Q. How does a finite-horizon MDP differ from an infinite-horizon one?
    • A finite-horizon MDP has a fixed end, so gamma can be one and the best action can change with the remaining time. An infinite-horizon MDP runs forever, usually needs gamma below one to converge, and has a policy that doesn't depend on time.
• Bellman equation: $v(s) = \mathcal{R}_s + \gamma \sum_{s'} \mathcal{P}_{ss'} v(s')$
  • A state's value is the immediate reward plus the discounted future value.
  • The immediate reward $\mathcal{R}_s$ is the reward expected right away in state $s$.
  • The discounted future value $\gamma \sum_{s'} \mathcal{P}_{ss'} v(s')$ is the average value of the next states, weighted by how likely each one is and scaled down by $\gamma$.
  • It turns one long return into one step plus the rest, so the values of states can be computed from each other.
  • Q. What is the Bellman equation for the state-value function used for?
    • It calculates the expected return of a state under a given policy, as the immediate reward plus the discounted value of the next states. Every DP method is built on it.
  • Q. Can the Bellman equation be solved directly?: $v = \mathcal{R} + \gamma \mathcal{P} v \;\Rightarrow\; v = (I - \gamma \mathcal{P})^{-1} \mathcal{R}$
    • For a given policy it is a linear system, so small MDPs can be solved directly by matrix inversion. But that costs about the cube of the number of states, so large MDPs need iterative methods like DP, Monte Carlo or TD.
• Bellman expectation equation: $v_\pi(s) = \sum_a \pi(a \mid s) \left( \mathcal{R}_s^a + \gamma \sum_{s'} \mathcal{P}_{ss'}^a v_\pi(s') \right)$
  • The Bellman equation for an agent that follows a policy $\pi$, so actions come into it.
  • First the policy picks each action $a$ with probability $\pi(a \mid s)$.
  • That action gives the reward $\mathcal{R}_s^a$ and leads to each next state $s'$ with probability $\mathcal{P}_{ss'}^a$.
  • For each action, we add that immediate reward to the discounted future value of the next states, then average over all the actions the policy might pick.
  • Q. How are the state value and the action value related?: $v_\pi(s) = \sum_a \pi(a \mid s) \, q_\pi(s, a)$
    • A state's value is the policy-weighted average of its action values, so the actions the policy picks more often count more.
  • Q. What is the Bellman expectation equation for the action value?: $q_\pi(s, a) = \mathcal{R}_s^a + \gamma \sum_{s'} \mathcal{P}_{ss'}^a \sum_{a'} \pi(a' \mid s') \, q_\pi(s', a')$
    • The action is already fixed, so we take its reward, move to each next state, and average the next action values the policy would pick there.
• Optimal value function: $v_*(s) = \max_\pi v_\pi(s), \quad q_*(s, a) = \max_\pi q_\pi(s, a)$
  • The maximum value any policy can achieve from each state.
  • The optimal action-value function is the maximum value achievable after taking a given action in a given state.
• Optimal policy: $\pi_*(a \mid s) = \mathbb{1}\left[ a = \arg\max_{a'} q_*(s, a') \right]$
  • A policy whose value is greater than or equal to every other policy's value in all states.
  • Every MDP has at least one, there is always a deterministic one, and all optimal policies share the same optimal value function.
  • Q. If we know the optimal action-value function, how do we get the optimal policy?
    • In each state we pick the action with the highest optimal action value. That greedy policy is optimal.
• Bellman optimality equation: $v_*(s) = \max_a \left( \mathcal{R}_s^a + \gamma \sum_{s'} \mathcal{P}_{ss'}^a v_*(s') \right), \quad q_*(s, a) = \mathcal{R}_s^a + \gamma \sum_{s'} \mathcal{P}_{ss'}^a \max_{a'} q_*(s', a')$
  • The Bellman equation for optimal values, where a state's value is the best action's immediate reward plus the discounted expected optimal value of the next states.
  • Because of the max, it is non-linear, so it can't be solved by matrix inversion and must be solved iteratively.
  • The version for the optimal action value is the core of Q-learning and DQN.
  • Q. What is the difference between the Bellman expectation equation and the Bellman optimality equation?
    • The expectation equation averages over the actions of a given policy, so it is linear and used for evaluation. The optimality equation takes the max over actions, so it is non-linear and gives the optimal values.
• Dynamic programming
  • An optimization method for sequential problems that breaks a big problem into sub-problems and combines their solutions.
  • Dynamic means the sequential part, and programming means optimizing a policy.
  • In RL it solves a known MDP recursively with the Bellman equation.
  • Q. What are the strengths and limits of DP?
    • It is guaranteed to converge to an optimal policy in polynomial time, which is far faster than searching every possible policy. But it needs a model, and it becomes impractical for huge state spaces because of the curse of dimensionality.
  • Q. Can dynamic programming only be applied with perfect information about the environment?
    • No. DP needs a model of the environment, but that model can be stochastic, and it can be a learned, approximate model, as in model-based RL.
• Policy evaluation: $v_{k+1}(s) = \sum_a \pi(a \mid s) \left( \mathcal{R}_s^a + \gamma \sum_{s'} \mathcal{P}_{ss'}^a v_k(s') \right)$
  • Computing the value function of a given policy.
  • It is a prediction problem.
  • It repeatedly applies the Bellman expectation backup to every state until the largest change falls below a small threshold.
  • Q. How do values propagate during policy evaluation?
    • Value propagation is how values spread outwards from the states with terminal rewards, one step further with each sweep.
• Policy improvement: $\pi'(s) = \arg\max_a q_\pi(s, a) = \arg\max_a \left( \mathcal{R}_s^a + \gamma \sum_{s'} \mathcal{P}_{ss'}^a v_\pi(s') \right)$
  • Making a new policy by acting greedily with respect to the current policy's values.
  • The new policy is at least as good as the old one.
  • Q. How do you extract a greedy policy from the values?
    • Greedy policy extraction is getting a policy from the values by picking, in each state, the action that moves toward the highest-valued next state.
• Policy iteration: $\pi_0 \xrightarrow{E} v_{\pi_0} \xrightarrow{I} \pi_1 \xrightarrow{E} v_{\pi_1} \xrightarrow{I} \cdots \xrightarrow{I} \pi_*$
  • An algorithm that evaluates the current policy and then improves it by acting greedily on its values.
  • It repeats these two steps until the policy stops changing.
  • Q. Should value iteration and policy iteration give different policies?
    • No, both should give the same optimal policy. They differ in convergence speed and in the number of iterations.
  • Q. Which converges first, the policy or the values?
    • The policy. It becomes stable after a few iterations, while the values keep changing.
• Value iteration: $v_{k+1}(s) = \max_a \left( \mathcal{R}_s^a + \gamma \sum_{s'} \mathcal{P}_{ss'}^a v_k(s') \right)$
  • An algorithm that turns the Bellman optimality equation into an update rule.
  • It keeps updating the values using the best action in each state until the change is tiny, and then extracts the policy by taking the action with the best value.
  • Q. What is the difference between policy iteration and value iteration?
    • Policy iteration alternates full policy evaluation and greedy improvement until the policy stops changing. Value iteration skips the separate evaluation, keeps updating the values with the best action, and extracts the policy at the end.
• Modified policy iteration
  • Policy iteration that stops the evaluation after a few sweeps before improving.
  • With one sweep it becomes value iteration, and with many sweeps it becomes full policy iteration.
• Synchronous DP
  • DP that updates every state from the values of the previous sweep.
  • It keeps two arrays, an old one and a new one.
  • New values are computed only from the old ones, so it needs more memory and more iterations.
• In-place DP
  • DP that uses each new value as soon as it is computed.
  • It keeps one array and overwrites values straight away, in any order.
  • New values are used immediately, so it needs less memory and fewer iterations.
  • It still converges as long as every state keeps getting updated.
  • Q. What is the difference between synchronous and in-place DP?
    • Synchronous DP uses two arrays and only updates from the old values. In-place DP uses one array and uses the new values immediately, so it converges in fewer iterations. Both converge.
• Real-time DP
  • DP that only updates the states relevant to the agent's current trajectory while it acts.
  • It uses interaction-driven online updates and lazy evaluation, so it only computes values for the states it actually visits.
  • It suits large problems where sweeping the whole state space is impossible, and it gives a near-optimal policy for the relevant region.
• Approximate DP
  • DP that replaces the value table with a function approximator, such as a linear model or a neural network.
  • It is used to handle the curse of dimensionality.
  • Eligibility traces and experience replay help keep the approximation errors under control.
• Model-based and model-free
  • Model-based methods have or learn the MDP and plan with it by computation.
  • Model-free methods have no access to the dynamics or the reward function.
  • They learn directly from sampled experience.
• Prediction and control
  • Prediction is evaluating a given policy to tell how good it is.
  • Control is finding the best policy.
  • Q. What is the difference between Monte Carlo prediction and control?
    • Prediction values a fixed policy from episodes. Control finds the optimal policy using Q and epsilon-greedy improvement.
• Generalized policy iteration
  • The general loop of evaluation and improvement behind RL control methods.
  • We start with any policy, evaluate it, improve it, and repeat until we reach the optimal value function and the optimal policy.
  • Q. Why can't we use V for model-free control?
    • Greedy improvement over V needs a one-step lookahead with the transition matrix, which we don't have. With Q we just pick the best action, so it stays model-free.

• Monte Carlo methods
  • Estimating an unknown quantity by drawing many random samples and averaging the results.
  • The name comes from the Monte Carlo casino in Monaco, because the method relies on randomness.
  • It is used when an analytical solution is hard or impossible, and more samples give better estimates.
• Monte Carlo in RL
  • A model-free method that learns values by averaging the actual returns of complete episodes.
  • It uses no bootstrapping, and each episode can start anywhere but must end.
  • Q. What is the key shift from DP to Monte Carlo?
    • We drop the assumption that the model is known. We move from planning to learning from episodes.
  • Q. What is a complete episode, and why does Monte Carlo need one?
    • It is an episode that reaches a terminal state. Monte Carlo needs the actual return, and that is only known at the end.
  • Q. What are the limitations of Monte Carlo?
    • It needs complete episodes, so it only works for episodic tasks and gets feedback late. It also has high variance, so it needs many episodes, and it is impractical when data is expensive. The need for complete episodes is what motivates TD.
  • Q. Which method is designed for episodic tasks where the reward only comes at the end?
    • Monte Carlo. It waits until the end of the episode and learns from the actual return, so it doesn't need intermediate rewards.
• Monte Carlo prediction: $v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s] \approx \frac{1}{N(s)} \sum_{i=1}^{N(s)} G^{(i)}$
  • Estimating a policy's value as the empirical mean of the returns we actually observe, instead of the true expected return.
  • It uses no bootstrapping, only actual returns from complete episodes.
  • Q. Where is the policy in Monte Carlo prediction?
    • The policy generates the sample episodes, and Monte Carlo returns its value function.
  • Q. Why do we need many episodes?
    • So that every state is visited often enough to give a stable estimate.
• First-visit Monte Carlo
  • Monte Carlo prediction that only counts the first visit to a state in each episode when averaging the returns.
• Every-visit Monte Carlo
  • Monte Carlo prediction that counts every visit to a state in each episode.
  • Q. What is the difference between first-visit and every-visit Monte Carlo?
    • First-visit only averages the first visit to a state in each episode, so its samples are independent and identically distributed, and unbiased. Every-visit averages all the visits, so it uses more data and is simpler, but it is slightly biased with few samples. Both converge as the number of episodes grows.
• Incremental mean: $\mu_k = \mu_{k-1} + \frac{1}{k}(x_k - \mu_{k-1}), \quad V(S_t) \leftarrow V(S_t) + \alpha \left( G_t - V(S_t) \right)$
  • A way to update an average a little toward each new return, instead of storing all the returns.
  • With a step size of one over the count it matches the exact average.
  • With a constant step size it becomes a moving average that forgets old experience, which suits non-stationary problems.
  • Q. Why update the value incrementally instead of overwriting it?
    • If we overwrote it, the value would only reflect the latest episode.
• Epsilon-greedy policy improvement: $\pi(a \mid s) = \begin{cases} \frac{\epsilon}{m} + 1 - \epsilon & \text{if } a = \arg\max_{a'} Q(s, a') \\ \frac{\epsilon}{m} & \text{otherwise} \end{cases}$
  • Making the new policy epsilon-greedy with respect to Q.
  • It exploits the best action with probability one minus epsilon and acts randomly with probability epsilon.
  • The policy improvement theorem says the new policy is better than or equal to the old one.
  • Q. Why is greedy improvement on Q not enough?
    • It never explores, so it can get stuck on a sub-optimal action. Epsilon-greedy fixes that.
• Monte Carlo control
  • Finding the best policy from complete episodes, without a model.
  • It is GPI with Monte Carlo evaluation of Q and epsilon-greedy improvement after every episode.
  • It doesn't wait for a full evaluation before improving.
  • Improving after every episode is noisier, like SGD compared with batch, but it reaches the optimal Q and policy much faster.
• GLIE Monte Carlo control: $\lim_{k \to \infty} N_k(s, a) = \infty, \quad \lim_{k \to \infty} \pi_k(a \mid s) = \mathbb{1}\left[ a = \arg\max_{a'} Q_k(s, a') \right], \quad \epsilon_k = \frac{1}{k}$
  • Monte Carlo control that keeps exploring early on but becomes fully greedy in the end.
  • GLIE stands for Greedy in the Limit with Infinite Exploration.
  • Every state-action pair keeps being visited, and the policy becomes greedy in the limit.
  • In practice we update Q after every episode and decay epsilon, such as one over the episode number.
  • Q. Why use GLIE instead of a fixed epsilon?
    • Once we have a good policy we want to exploit it. A fixed epsilon keeps exploring forever, so GLIE decays it.
• Q. What is the difference between DP and Monte Carlo?
  • DP is model-based and bootstraps from estimated values, so it has lower variance. Monte Carlo is model-free and uses actual returns from complete episodes, so it has higher variance and needs episodic tasks.
• Temporal Difference learning
  • A model-free method that, unlike Monte Carlo, learns from incomplete episodes.
  • It takes one step and then immediately updates the previous state's value.
  • It combines Monte Carlo sampling with DP bootstrapping, so it is updating a guess toward a guess.
  • Q. Why does TD do better in Markov environments?
    • TD builds the underlying MDP step by step from one-step transitions. Monte Carlo just averages full returns, so it suits non-Markov environments.
• TD target and TD error: $V(S_t) \leftarrow V(S_t) + \alpha \big[ \underbrace{R_{t+1} + \gamma V(S_{t+1})}_{\text{TD target}} - V(S_t) \big], \quad \delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$
  • The TD target is the immediate reward plus the discounted value of the next state.
  • It uses the real reward for the first step and the current guess for the rest.
  • The TD error is the target minus the current estimate, and the value moves a step toward the target.
  • Q. Why can't the Monte Carlo update be used directly in TD?
    • Without a complete episode the full return is unknown. So TD replaces it with the TD target.
  • Q. What does the step size alpha control?
    • The step size alpha is how fast the estimate moves toward the target. A constant alpha gives more weight to recent experience.
• Bootstrapping and sampling
  • Bootstrapping means updating an estimate from another estimate.
  • TD and DP bootstrap.
  • Monte Carlo uses actual returns instead.
  • Sampling means learning from sampled experience instead of the full model.
  • Monte Carlo and TD sample.
  • DP doesn't sample, because the dynamics are known.
  • Q. How deep are the backups in TD, Monte Carlo and DP?
    • A backup is updating a state's value from the states that follow it. TD does shallow one-step backups on one sampled path, Monte Carlo does deep backups along a full sampled path to the end, and DP does full backups over every branch weighted by probability.

• TD lambda: $\text{TD}(0) \rightarrow \text{TD}(\lambda) \rightarrow \text{TD}(1) \approx \text{MC}$
  • A method that sits between one-step TD and Monte Carlo, where lambda controls how far into the future it learns from.
  • Lambda zero is TD zero, and lambda one approaches Monte Carlo.
• On-policy and off-policy
  • On-policy learning is learning about and improving the same policy that the agent follows.
  • Off-policy learning is following a behavior policy to explore while learning a different target policy, so it can even learn from logs or human demonstrations.
  • GLIE Monte Carlo control and SARSA are on-policy, while Q-learning and DQN are off-policy.
  • Q. Why is SARSA on-policy and Q-learning off-policy?
    • SARSA updates with the next action from the policy it follows. Q-learning updates with the max action, which is a greedy target policy, while a separate behavior policy explores.
• SARSA: $Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \left[ R_{t+1} + \gamma Q(S_{t+1}, A_{t+1}) - Q(S_t, A_t) \right]$
  • A TD method that learns the value of the policy it follows and improves that same policy.
  • It is on-policy TD control, TD evaluation plus epsilon-greedy policy improvement.
  • The name comes from state, action, reward, next state and next action.
  • The next action comes from the same epsilon-greedy policy that is being improved, and it doesn't need to finish evaluation before improving.
  • Q. Are SARSA and TD different methods?
    • No. SARSA is a control algorithm that uses TD to update the Q values.
  • Q. Can you explain the SARSA update in words?
    • The target is the immediate reward plus the discounted Q value of the next state and the next action actually taken. We move the current Q value toward that target by alpha times the error.
• Q-learning: $Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \left[ R_{t+1} + \gamma \max_a Q(S_{t+1}, a) - Q(S_t, A_t) \right]$
  • A TD method that learns the optimal action values, whatever policy it uses to explore.
  • It is off-policy TD control.
  • A behavior policy, such as epsilon-greedy, explores.
  • The update uses the maximum Q value at the next state, which is the greedy target policy, so it replaces the actual next action with the best possible one.
  • It directly approximates the Bellman optimality equation.
  • Q. Can you explain the Q-learning update in words?
    • The target is the immediate reward plus the discounted maximum Q value at the next state. We move the current Q value toward that target by alpha times the error.
  • Q. In Q-learning, what does the Q value of a state-action pair represent?
    • It is the expected cumulative reward of taking that action in that state and then following the optimal policy afterwards.
  • Q. Will Q-learning be accurate after a few episodes?
    • No. Q starts as a guess and needs very many episodes, often millions of steps for complex problems.
• Q. What is the difference between Monte Carlo and TD?
  • Monte Carlo waits for the end of the episode and learns from the actual return, so it only works for episodic tasks. TD learns after every step from the TD target, so it works for both episodic and continuing tasks. Monte Carlo has high variance and zero bias, while TD has low variance and some bias.
• Q. Why does Monte Carlo have high variance but zero bias, and TD the opposite?
  • The Monte Carlo return adds up many random steps, but it is an actual return, so it is unbiased. The TD target only has noise from one step, but it bootstraps from an estimate, so it is biased.
• Q. Is TD always faster than Monte Carlo?
  • There is no proof that either one is always faster, so it depends on the problem. In practice TD often learns faster, because it can update after every step instead of waiting for the episode to end.
• Q. In the AB example, why do Monte Carlo and TD disagree on the value of A?
  • There are two states. A always goes to B with reward 0, and B ends with reward 1 in six of eight episodes. Monte Carlo saw A once with a return of zero, so it gives zero. TD sees that A always leads to B with a reward of zero, so A gets the same value as B, which is 0.75.

• Q. What does each of SARSA and Q-learning end up learning?
  • SARSA learns the value of the policy it actually follows, including its exploration. Q-learning learns the value of the optimal policy, whatever the behavior policy does.
• Q. Why does Q-learning take the risky path in cliff walking, and when would you choose SARSA?
  • The max makes Q-learning learn the optimal path regardless of exploration, so it sometimes falls off while exploring. SARSA accounts for exploration and learns the safer path, so I would choose it when safety during learning matters.

• Function approximation and generalization: $\hat{v}(s, \mathbf{w}) \approx v_\pi(s), \quad \hat{q}(s, a, \mathbf{w}) \approx q_\pi(s, a)$
  • Function approximation is estimating an unknown function from the observations we have, here a function that gives values for states instead of a table.
  • Because it learns from the states the agent has seen, it also gives values for unseen states, which is generalization.
  • A weight vector parameterizes the function, and Monte Carlo and TD still update it.
  • Q. Are tabular Monte Carlo and TD values exact?
    • No, they are approximations too. Function approximation does the approximation better and generalizes.
  • Q. Is approximating the state-value function prediction or control?
    • It is prediction. It evaluates a given policy by approximating its value function. Control needs action values.
  • Q. What is the scaling problem that motivates function approximation?
    • The scaling problem is that tabular methods store one value per state or state-action pair, which doesn't work for huge or continuous spaces like Go, backgammon or Atari. Such spaces can never be visited often enough, and the table blows up the memory.
• Types of function approximators
  • Function approximators range from tables, through decision trees and nearest neighbor, to linear combinations of features and differentiable models like neural networks.
  • In RL they must be trainable on non-stationary data.
  • Q. What are the three approximator configurations?
    • The first takes a state and outputs its value. The second takes a state and an action and outputs one Q value. The third takes a state and outputs the Q values for all actions, and then we take the max.
  • Q. Which approximator configuration would you choose?
    • It depends on the problem. Outputting Q for all actions suits discrete actions and real-time settings like Atari, because one forward pass gives every action value.
• Gradient descent: $\mathbf{w} \leftarrow \mathbf{w} - \alpha \nabla_{\mathbf{w}} J(\mathbf{w})$
  • An iterative optimization method that moves the weights against the gradient of the loss to reduce the error between the prediction and the target.
  • The step size controls how far each update goes.
• Value function approximation with SGD: $\Delta \mathbf{w} = \alpha \left( v_\pi(S) - \hat{v}(S, \mathbf{w}) \right) \nabla_{\mathbf{w}} \hat{v}(S, \mathbf{w})$
  • Updating the weights after each sampled state, by the step size times the error times the gradient of the prediction.
  • On average this equals the full gradient update.
  • It suits TD's one-step updates, and it also works for Monte Carlo.
• Oracle target: $J(\mathbf{w}) = \mathbb{E}_\pi\left[ \left( v_\pi(S) - \hat{v}(S, \mathbf{w}) \right)^2 \right]$
  • The true value of the policy, assumed to be given at first.
  • That makes it supervised learning, which minimizes the mean squared error to that value.
  • It is only a starting assumption to set up the method.
• Feature vectors: $\mathbf{x}(S) = \left( x_1(S), \ldots, x_n(S) \right)^\top$
  • A list of numbers that describes a state.
  • We need it because RL has experience, not features.
• Linear value function approximation: $\hat{v}(S, \mathbf{w}) = \mathbf{x}(S)^\top \mathbf{w}, \quad \nabla_{\mathbf{w}} \hat{v}(S, \mathbf{w}) = \mathbf{x}(S), \quad \Delta \mathbf{w} = \alpha \left( v_\pi(S) - \hat{v}(S, \mathbf{w}) \right) \mathbf{x}(S)$
  • Approximating the value as a weighted sum of the features.
  • Its gradient is just the feature vector, so the update is step size times error times features.
  • With SGD it converges to an approximate solution, not the exact value function.
  • Q. How does linear approximation compare with neural networks?
    • Linear approximation is efficient, but it needs handcrafted features and scales less. Neural networks can handle nonlinear inputs like raw frames, but they are a black box.
• Monte Carlo and TD targets: $\text{MC}: \Delta \mathbf{w} = \alpha \left( G_t - \hat{v}(S_t, \mathbf{w}) \right) \nabla_{\mathbf{w}} \hat{v}(S_t, \mathbf{w}), \quad \text{TD}: \Delta \mathbf{w} = \alpha \left( R_{t+1} + \gamma \hat{v}(S_{t+1}, \mathbf{w}) - \hat{v}(S_t, \mathbf{w}) \right) \nabla_{\mathbf{w}} \hat{v}(S_t, \mathbf{w})$
  • Monte Carlo and TD targets are what replace the true value when we train the approximator.
  • We need them because the true value is never given.
  • The Monte Carlo return is an unbiased but noisy sample of the true value, and the TD target is a biased sample.
  • Each state paired with its target becomes a supervised training example.
  • Q. Why is the Monte Carlo target unbiased and the TD target biased?
    • The Monte Carlo return is an actual return. The TD target uses an estimate of the next state's value.
• Control with function approximation
  • Finding the best policy when the action values come from a function approximator instead of a table.
  • It is GPI where the Q network does the approximate evaluation and epsilon-greedy does the improvement at every step.
• Action-value approximation: $J(\mathbf{w}) = \mathbb{E}_\pi\left[ \left( q_\pi(S, A) - \hat{q}(S, A, \mathbf{w}) \right)^2 \right], \quad \hat{q}(S, A, \mathbf{w}) = \mathbf{x}(S, A)^\top \mathbf{w}$
  • Approximating Q instead of V, using features of the state and action together.
  • The target Q is replaced with the Monte Carlo return or the TD target.
• Deadly triad
  • Three ingredients that together can make value learning diverge.
  • They are function approximation, bootstrapping and off-policy learning.
  • Q. What is wrong with Q-learning plus a neural network?
    • It is not the max, which is the core of the algorithm. The problem is that the same weights give both the prediction and the target. That causes correlated samples and a moving target, so it can diverge.
  • Q. What is the deadly triad?
    • It is function approximation, bootstrapping and off-policy learning used together. Q-learning with a neural network has all three, which is why it can diverge without extra tricks.
• Batch methods
  • Methods that store experience as a dataset, sample from it repeatedly to fit the approximator, and then refresh it.
  • They look for the value function that best fits all the stored data.
  • Q. What problem do batch methods solve?
    • Sample inefficiency. Incremental updates use each sample once, while batch methods reuse stored experience.
  • Q. What is sample inefficiency?
    • Sample inefficiency means using each experience once and then throwing it away, which wastes data.
• Least squares prediction: $LS(\mathbf{w}) = \sum_{t=1}^{N} \left( v_t^\pi - \hat{v}(s_t, \mathbf{w}) \right)^2, \quad \mathbf{w}^\pi = \arg\min_{\mathbf{w}} LS(\mathbf{w})$
  • Finding the weights that minimize the sum of squared errors between the approximate values and the target values in the stored dataset.
  • Sampling from the dataset with SGD converges toward this least squares solution.
• Deep neural networks
  • Universal approximators built from layers of linear and non-linear transformations and trained end to end with backpropagation.
  • They generalize to unseen states, handle continuous spaces and raw pixels, but they are a black box.
• Convolutional neural network
  • A neural network for images.
  • Convolution layers extract features with learned filters, pooling layers shrink the size while keeping important information, and fully connected layers turn the features into the output.
• Deep reinforcement learning
  • Using deep neural networks to represent the value function, the policy or the model.
  • RL creates the target, and SGD trains the network to fit it.
• Non-stationary target
  • A target that moves every time the weights update.
• Experience replay
  • A technique that stores transitions in a buffer and samples them randomly.
  • Each transition is a state, action, reward and next state.
  • Random sampling breaks the correlation between consecutive samples, and experience gets reused.
  • Q. What limits the replay buffer?
    • Memory. A bigger buffer holds more experience over a longer horizon.
• Fixed Q-targets: $\text{target} = r + \gamma \max_{a'} Q(s', a'; \mathbf{w}^-)$
  • A technique that uses a separate target network, frozen for a while, to give the target.
  • The online network learns toward it, only the online weights are updated, and the weights are copied over periodically.
• DQN: $\mathcal{L}(\mathbf{w}) = \mathbb{E}_{(s, a, r, s') \sim \mathcal{D}}\left[ \left( r + \gamma \max_{a'} Q(s', a'; \mathbf{w}^-) - Q(s, a; \mathbf{w}) \right)^2 \right]$
  • A deep RL method that learns Q values straight from raw inputs, such as game frames.
  • It is Q-learning with a deep network, experience replay and fixed Q-targets.
  • On Atari it takes the last four frames as input and outputs Q for every joystick action, with the change in score as the reward.
  • The same architecture and hyperparameters worked across all games, and it reached human level on many of them.
  • Q. Which DQN technique fixes which problem?
    • Experience replay with random sampling fixes the correlated samples. Fixed Q-targets fix the non-stationary target.
  • Q. What is frame stacking, and why is it used?
    • Frame stacking is feeding several consecutive frames together as one input, because a single frame can't show motion.
• Q. Why might nonlinear TD diverge when Monte Carlo does not?
  • TD bootstraps from its own estimates one step ahead. Monte Carlo uses the actual return to the end of the episode.
• Q. Do these methods converge with function approximation?
  • Tabular methods converge. Linear approximation converges for prediction and only chatters near the optimum for control, because all values share one weight vector. Nonlinear TD and off-policy TD may diverge. Monte Carlo converges in more cases than TD. It is hard to prove in theory, but in practice it often works.
• Policy search
  • Learning the policy directly, instead of deriving it from V or Q.
  • If all we care about is the optimal behavior, the policy is what we should learn.
  • Q. Why learn the policy directly instead of a value function?
    • If all we need is the optimal behavior, the policy can be simpler than the values. It can also be stochastic, and it works in high-dimensional and continuous action spaces.
  • Q. Is policy search model-based?
    • No. It is still model-free, and it learns from sampled experience.
• Parameterized policy: $\pi_\theta(a \mid s) = \mathbb{P}[a \mid s, \theta]$
  • A function with its own parameters, often a neural network, that takes a state and outputs a probability for each action.
  • It is still model-free.
• Value-based, policy-based and actor-critic
  • Value-based methods learn V or Q, and the policy is implicit.
  • Policy-based methods only learn the policy.
  • Actor-critic methods learn both.
  • Q. What is the difference between value-based and policy-based methods?
    • Value-based methods derive a usually deterministic policy from V or Q. Policy-based methods learn the policy directly, have better convergence and can be stochastic, but they usually reach a local optimum and don't generalize well.
• Gradient-based and trust-region methods
  • Gradient-based methods update the policy by following the policy gradient directly.
  • REINFORCE, actor-critic and DDPG are gradient-based.
  • Trust-region methods also use the policy gradient, but restrict how big each update can be.
  • TRPO and PPO are trust-region methods.
  • Q. Why do we need trust-region methods at all?
    • A gradient can point in a really bad direction, so one unbounded update can give a really bad policy. A trust region keeps each update small, so the policy doesn't jump off the cliff.
  • Q. Are TRPO and PPO really gradient-free?
    • No. They are sometimes called non gradient based, but they still use the policy gradient. What changes is that the update is constrained or clipped.
• Deterministic and stochastic policies: $\pi(s) = a \quad \text{vs} \quad \pi(a \mid s) = \mathbb{P}[A_t = a \mid S_t = s]$
  • A deterministic policy maps each state to exactly one action.
  • A stochastic policy gives a probability for each action, so it is more general.
  • Sometimes the optimal policy is random, and value-based methods usually can't learn that because they derive one deterministic action.
  • Rock-Paper-Scissors is one example, because a deterministic policy is easily exploited.
  • Another is two states that look identical, where a deterministic policy picks the same, sometimes fatal, action in both.
• Policy objective: $J_1(\theta) = v_{\pi_\theta}(s_1), \quad J_{avV}(\theta) = \sum_s \mu_{\pi_\theta}(s) v_{\pi_\theta}(s), \quad J_{avR}(\theta) = \sum_s \mu_{\pi_\theta}(s) \sum_a \pi_\theta(a \mid s) \mathcal{R}_s^a$
  • A measure of how good the policy is.
  • For episodic tasks it is the expected return from the start state.
  • For continuing tasks it is the average value or the average reward per step, weighted by how often the agent visits each state.
• Gradient ascent: $\Delta \theta = \alpha \nabla_\theta J(\theta)$
  • Climbing the gradient of the objective.
  • We use it because RL maximizes reward instead of minimizing a loss.
• Gradient-free policy search
  • Optimizing the policy without gradients, using methods like hill climbing or genetic algorithms.
  • They exist but are less predictable.
• Score function: $\nabla_\theta \pi_\theta(s, a) = \pi_\theta(s, a) \frac{\nabla_\theta \pi_\theta(s, a)}{\pi_\theta(s, a)} = \pi_\theta(s, a) \nabla_\theta \log \pi_\theta(s, a)$
  • A way to estimate the policy gradient from sampled episodes.
  • It measures how sensitive the policy is to a change in its parameters, and tells us which direction makes an action more likely.
• Policy gradient theorem: $\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\left[ \nabla_\theta \log \pi_\theta(s, a) \, Q^{\pi_\theta}(s, a) \right]$
  • The result that tells us how to compute the gradient of the objective in a multi-step MDP.
  • The gradient is how the policy changes times how good the action is.
  • It replaces the instantaneous reward of a one-step problem with the action value Q.
  • It holds for the start value, average value and average reward objectives.
• REINFORCE: $\Delta \theta_t = \alpha \nabla_\theta \log \pi_\theta(s_t, a_t) \, G_t$
  • The Monte Carlo policy gradient method.
  • It plays a full episode, then makes the actions that led to a high return more likely and the actions that led to a low return less likely.
  • It uses the return as an unbiased sample of the action value, so it has low bias but high variance and is slow to converge.
  • Q. What kind of policy does REINFORCE use?
    • A stochastic policy. It outputs action probabilities and pushes up the probability of actions that led to higher returns.
• Actor-critic: $\delta = r + \gamma Q_{\mathbf{w}}(s', a') - Q_{\mathbf{w}}(s, a), \quad \theta \leftarrow \theta + \alpha \nabla_\theta \log \pi_\theta(s, a) Q_{\mathbf{w}}(s, a), \quad \mathbf{w} \leftarrow \mathbf{w} + \beta \delta \phi(s, a)$
  • A method where an actor learns the policy and a critic judges the actor's actions.
  • The actor holds the policy and acts.
  • The critic estimates the value function with Monte Carlo or TD and gives feedback through the TD error.
  • In GPI terms the critic evaluates and the actor improves.
  • Q. How does actor-critic improve on REINFORCE?
    • The critic's value estimate replaces the noisy full return, and that reduces the variance.
  • Q. What does the critic's TD error do?
    • It tells the actor whether its action was better or worse than expected. It is used to update both the critic and the actor.
• Advantage function: $A(s, a) = Q(s, a) - V(s)$
  • How much better an action is than the average action in that state.
  • It is Q minus V.
  • It is used in TRPO, PPO and other deep RL methods.
  • Q. Why use the advantage function in policy gradient methods?
    • Because it reduces the variance of the policy updates. It focuses the update on how much better an action is than average, not on the raw value.
• KL divergence: $D_{KL}(P \parallel Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)}$
  • Kullback-Leibler divergence, a measure of how much one probability distribution differs from another.
  • In TRPO it measures how far the new policy has moved from the old one.
• TRPO: $\max_\theta \hat{\mathbb{E}}_t\left[ r_t(\theta) \hat{A}_t \right] \quad \text{s.t.} \quad D_{KL}\left( \pi_{\theta_{old}} \parallel \pi_\theta \right) \le \delta$
  • A policy search method that only accepts an update if the new policy stays within a trust region, measured by the KL divergence from the old policy.
  • It is conservative and stable, but computationally expensive.
  • Q. How does a trust region differ from plain gradient descent?
    • Gradient descent picks the steepest direction and moves a fixed step. A trust region first defines how far it is safe to move and then finds the best point inside that region.
  • Q. Isn't a trust region just a smaller step size?
    • No. The step size controls the speed. The trust region limits how far the new policy can move away from the old one.
  • Q. When would you use TRPO or PPO?
    • For complicated, continuous state and action spaces, where learning the policy directly is the goal.

• PPO
  • A policy search method that clips the policy ratio to a small range around one, instead of computing a trust region.
  • It is simpler and much faster than TRPO.
  • Q. What is the difference between TRPO and PPO?
    • TRPO enforces a KL trust region at every iteration, which is stable but expensive. PPO clips the policy ratio instead, which is simpler and much faster.
  • Q. What is the main purpose of clipping in PPO?
    • Clipping prevents excessively large policy updates. It removes the benefit of pushing the policy ratio beyond the clip range, so the new policy stays close to the old one.
  • Q. Is the epsilon in PPO the same as in epsilon-greedy?
    • No. In PPO it sets the clipping range and is usually around 0.1 or 0.2. It is not an exploration rate.
• Clipped surrogate objective: $J^{CLIP}(\theta) = \hat{\mathbb{E}}_t\left[ \min\left( r_t(\theta) \hat{A}_t, \; \text{clip}\left( r_t(\theta), 1 - \epsilon, 1 + \epsilon \right) \hat{A}_t \right) \right]$
  • PPO's objective, which rewards better actions but stops rewarding the policy for moving too far from the old one.
  • It is the policy ratio times the advantage, with the ratio clipped.
  • If an action was better than expected, PPO increases its probability but gets no extra benefit beyond the clip range, and the same in the other direction.
  • Clipping therefore removes the incentive for very large updates.
  • The ratio itself can still leave the range, but clipping takes away the reward for pushing it further.
  • Q. What is the policy ratio?: $r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{old}}(a_t \mid s_t)}$
    • The policy ratio is the probability of an action under the new policy divided by its probability under the old policy. One means no change, above one means the action became more likely, and below one means less likely.

• Q. What are the problems with vanilla policy gradient?
  • Each update creates a new policy, so the data is non-stationary. The step size is also hard to choose. If it is too big, the next data comes from a bad policy and it may never recover. If it is too small, experience is used inefficiently.
• Q. Why not use experience replay in on-policy policy gradient?
  • The stored data comes from older, different policies, but the update needs data from the current policy.
• Model-based RL
  • An approach that learns the transition dynamics and the reward model from experience, and then plans with them using value iteration, policy iteration or tree search.
  • It combines learning and planning.
  • Q. What is the difference between model-free and model-based RL?
    • Model-free RL learns a value function or a policy directly from experience. Model-based RL learns a model and then plans with it.
  • Q. What are the advantages and the weakness of model-based RL?
    • Learning the model is an efficient supervised problem, and the model can also estimate its own uncertainty. But it is a two-step process, and the model error and the planning error compound.
  • Q. What is a model?
    • The model is the MDP itself, meaning the states, actions, transition probabilities, reward function and discount factor. Model-based RL tries to approximate the environment's underlying MDP.
• Model learning: $(S_t, A_t) \rightarrow (R_{t+1}, S_{t+1}), \quad \eta^* = \arg\min_\eta L(\eta)$
  • Fitting the model from experience as a supervised learning problem, with the state and action as input and the reward and next state as targets.
  • Predicting the reward is regression, and predicting the next state is density estimation.
  • Q. How do we get the experience to learn a model?
    • By interacting with the environment, collecting either complete Monte Carlo episodes or one-step TD transitions.
• Table lookup model: $\hat{\mathcal{P}}_{ss'}^a = \frac{N(s, a, s')}{N(s, a)}, \quad \hat{\mathcal{R}}_s^a = \frac{1}{N(s, a)} \sum_{t=1}^{T} \mathbb{1}\left[ S_t = s, A_t = a \right] R_{t+1}$
  • The simplest model, a table of transition probabilities and rewards estimated from experience.
  • It estimates the transition probabilities by counting how often each next state follows each state-action pair, and the rewards by averaging.
• Parameterized model: $S_{t+1} \sim \mathcal{P}_\eta(S_{t+1} \mid S_t, A_t), \quad R_{t+1} \sim \mathcal{R}_\eta(R_{t+1} \mid S_t, A_t)$
  • A learned approximation of the environment's MDP, with its own parameters.
  • It is made of a transition model and a reward model.
  • It assumes the state and action spaces are known, the Markov property holds, and the next state and the reward are conditionally independent, so they can be modeled separately.
• Sample-based planning
  • Using the learned model only as a simulator to generate experience.
  • Model-free methods, such as Q-learning, then learn from that experience, which reduces real interactions with the environment.
• Simple Monte Carlo search: $Q(s_t, a) = \frac{1}{K} \sum_{k=1}^{K} G_t^{(k)}, \quad a_t = \arg\max_a Q(s_t, a)$
  • A search that uses a model and a simulation policy to simulate many episodes from the current state for each action, and takes the action with the highest average return.
• Expectimax tree: $Q(s, a) = \sum_{s'} \mathcal{P}(s' \mid s, a) \left[ \mathcal{R}(s, a, s') + \gamma V(s') \right], \quad V(s) = \max_a Q(s, a), \quad O\left( (|\mathcal{S}| |\mathcal{A}|)^H \right)$
  • A full search tree from the current state where action nodes take the max and chance nodes take the expected value over next states.
  • It computes optimal Q values for the current state, but its size grows exponentially with depth.
  • Q. How does MCTS differ from an expectimax tree?
    • An expectimax tree expands every branch up to a depth and computes exact expectations, so it is exhaustive but exponential in depth. MCTS builds a partial tree, picks branches with UCB and estimates values from sampled returns, so it is approximate but its cost is controlled by the simulation budget.
• Monte Carlo Tree Search: $\text{UCB}(s, a) = \underbrace{\bar{V}_i}_{\text{exploitation}} + \underbrace{c \sqrt{\frac{\ln n(s)}{n(s, a)}}}_{\text{exploration}}$
  • A model-based search that puts the current state at the root and grows a partial search tree by simulating episodes with the model.
  • After the search, it takes the action with the highest value.
  • AlphaGo and AlphaZero use it.
  • Q. Is MCTS used in model-free RL?
    • No. MCTS needs a model to simulate episodes from the current state, so it is a model-based planning method.
  • Q. What are the four steps of MCTS?
    • Selection, also called tree traversal, with UCB, then expansion with a new child node, then a random rollout to the end, and finally backpropagation of the return to the root, updating visit counts and values along the path.
  • Q. In UCB, which part is exploitation and which is exploration?
    • The average value is exploitation, and the bonus term for less-visited actions is exploration. Calling the average value exploration is a common mistake.
  • Q. Why does MCTS build only a partial tree?
    • The full tree is too large, so it only expands where UCB points.
  • Q. What are the strengths and limits of MCTS?
    • It is a highly selective best-first search that evaluates states dynamically, uses samples to break the curse of dimensionality and works with a black-box model. But it needs a finite horizon and clear rules, and the tree has to fit in memory, so it suits games like Go but not every problem.
  • Q. What is a rollout?
    • A rollout is a random simulation from a newly expanded node until the episode ends or a depth limit is reached, giving a return for that node.

• Alignment problem
  • The gap between what an LLM is trained to do, predicting likely text, and what humans want, useful and safe answers.
  • Its output can be fluent but still incorrect, biased or harmful.
  • Traditional metrics like BLEU don't capture human preferences either.
• RLHF: $\text{Pre-training}: \max_\theta \sum_t \log P_\theta(x_t \mid x_{<t}), \quad \text{RLHF}: \max_\theta \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}\left[ R(x, y) \right]$
  • Fine-tuning a model with RL using a reward learned from human preferences.
  • Pre-training maximizes the likelihood of the next token, while RLHF maximizes the expected reward of human-preferred responses.
  • In RL terms the policy is the LLM, the state is the context, the action is generating tokens, and the reward comes from the reward model.
  • Q. Why not train an LLM with an automatic reward or with likelihood alone?
    • Most prompts have no automatic reward, and likelihood only predicts text. RLHF pushes the output toward what humans prefer.
  • Q. What are the four steps of the RLHF pipeline?
    • Pre-training, then supervised fine-tuning, then reward model training, and finally RL optimization with an algorithm like PPO.
  • Q. What are the benefits of RLHF?
    • Responses align better with human behavior and values, they are safer and more acceptable, and it handles tasks where the reward is hard to define. It also means less content moderation and higher user satisfaction.
  • Q. What are the challenges of RLHF?
    • Human labeling is costly, human bias gets carried into the reward model and the policy, the model can hack the reward, and it is hard to scale feedback to large models.

• Human feedback
  • Human judgment given as preferences between two outputs, rankings of several outputs, or evaluations of whether an answer is better or worse.
  • Q. Why collect preferences rather than numeric scores?
    • A numeric score might not represent the preference correctly. Choosing the better of two responses captures it directly.
• Pre-training
  • Training the model on a massive corpus with next-token prediction to learn language and general knowledge.
  • Its output can still be biased, irrelevant or unsafe.
• Supervised fine-tuning
  • Fine-tuning the model on high-quality instruction and response demonstrations so that it follows instructions.
  • But following instructions doesn't guarantee alignment with human preferences.
  • Q. What does SFT give that pre-training does not, and what is still missing?
    • SFT gives instruction following. It still lacks human preferences, which the reward model and the RL step add.
• Reward model
  • A model that scores responses the way humans would.
  • Humans compare or rank several responses to the same prompt, and the reward model is trained on those preferences, so it turns human judgment into a numeric reward function.
  • Q. Why is the reward model central to RLHF?
    • It is the only source of reward for the policy. If it is wrong, PPO optimizes the wrong behavior.
• RL fine-tuning with PPO
  • The step where PPO updates the LLM to get a higher score from the reward model.
  • The policy generates responses, the reward model scores them, and a KL penalty stops the policy drifting too far from the reference model.
  • Q. What is the KL penalty?: $\max_\theta \; \mathbb{E}\left[ R(x, y) \right] - \beta D_{KL}\left( \pi_\theta \parallel \pi_{ref} \right)$
    • The KL penalty is a term that penalizes the updated policy for moving far from the reference policy. A higher penalty gives more conservative updates and reduces over-optimization.
  • Q. Why does the RL fine-tuning step need a penalty?
    • Without it, the policy can change drastically and over-optimize the reward. The KL penalty keeps it close to the reference model, just as PPO and TRPO limit their updates.
• Reward hacking
  • A model exploiting the reward function to get a high score without producing genuinely better responses.
  • It is the over-optimization problem in RLHF.
• Full fine-tuning and PEFT
  • Full fine-tuning is fine-tuning that updates all the weights.
  • PEFT, parameter-efficient fine-tuning, is fine-tuning that freezes most of the weights and only trains a small set of parameters, such as LoRA, which is far cheaper.
• Hierarchical Reinforcement Learning
  • RL that breaks a complex task into smaller sub-tasks at different levels.
  • A high-level policy chooses which sub-task to do, and low-level policies carry it out.
• Options
  • A temporally extended action, like a sub-policy or skill.
  • It has a set of states where it can start, its own policy, and a condition for when it terminates.
  • Q. What concept does HRL introduce to simplify complex tasks?
    • Options. Each option is a reusable sub-policy, so the agent can plan over a few big steps instead of many small actions.

1. Title and name, current affiliation and position
2. Education, experience, and research interests
3. One paper from the master's, plus skills
4. One paper from my time as a researcher, covered in detail because it is close to the professor's work
5. Projects and papers from my time as a researcher
6. What I want to do in the PhD

• Model: Transition Probability, Reward Function, State, Action, Discount Factor
• Planning: use the learned models to evaluate possible actions, construct or iprove a value function or policy using Value/Policy Iteration.
• Underlying MDP: The env's actual state transitions and reward structure.
  • Transition Model $P(s'|s,a)$
  • Reward Model $R(s,a)$
  • Goal: Approximate the env's MDP and use if for planning.

Advantages of Model-Based RL​

• Efficient Model Learning: Supervised learning can efficiently learn transition and reward models from experience.
• Model Uncertainty: The learned model can estimate uncertainty in its prediction

Disadvantages of Model-Based RL​

• Two-Step Process: First learn the environment model, then use it to construct a value function or policy.
• Two Sources of Approximation Error:
  • Approximated Model Error: The learned model may inaccurately predict transitions or rewards.
  • Approximated Planning Error: Planning with the learned model may produce approximate or suboptimal results.
  • Compounding Error: Model and planning errors can accumulate, reducing the accuracy of the final policy.

MDP Model​

$\text{MDP} = \langle S, A, P, R \rangle_{\eta}$

• Assumption: State space $S$ and Action space $A$ are known and the env satisfies the Markov property.
• Model $\mu = \langle P_{\eta}, R_{\eta} \rangle$ is parameterized by $\eta$ and represents state-transitions and rewards.
  • $P_{\eta} \approx P$
  • $R_{\eta} \approx R$

$S_{t+1} \sim P_{\eta}(s_{t+1} | s_t, a_t)$ $R_{t+1} \sim R_{\eta}(r_{t+1} | s_t, a_t)$

$P(s_{t+1}, R_{t+1} | s_t, a_t) = P(s_{t+1} | s_t, a_t) P(R_{t+1} | s_t, a_t)$

• Assume that the next state and reward are conditionally independent given the current state and action (not a consequence of the Markov property).
• This allows transition and reward distributions to be modeled separately.

Model Learning​

• Goal: Approximate a model $\mu_\eta$ from experience $\{S_1, A_1, R_2, \ldots, S_T\}$.
• Supervised Learning: Use $(S_t, A_t)$ as input and $(R_{t+1}, S_{t+1})$ as targets.
• Loss Function: Measure prediction errors using a loss function (e.g., MSE).
• Empirical Loss: Average prediction error over observed training data.
• Use supervised learning framework, find parameters $\eta$ that minimize empirical loss (MSE).

Lookup Table Model​

• Underlying Model is presented as a lookup table.
• A model is an explicit $\text{MDP}, \hat{P}, \hat{R}$

$\hat{P}_{ss'}^a = \frac{1}{N(s,a)} \sum_{t=1}^T 1((S_t, A_t, S_{t+1}) = (s,a,s'))$

• $N(s,a)$ is the number of times State-Action pair has been visited.
• $1$: Indicator function that is 1 if the condition is true, 0 otherwise.
• $\hat{P}_{ss'}^a$ is the estimated probability of transitioning from state $s$ to state $s'$ under action $a$.

$\hat{P}(s'|s,a) = \frac{N(s,a,s')}{N(s,a)}$

• Estimated Transition Probability by counting the number of times the State-Action-Next State tuple has been observed.

• Calculate the average reward for the State-Action pair.

Single Action MRP Example​

Two States $\{A, B\}$ and no discount, 8 episodes of learning.

• A → B: 100% transition probability, reward = 0.
• B → Terminal: 75% chance of reward 1, 25% chance of reward 0.

• MC: Estimates values using observed complete returns.
  • $V(B) = 6/8 = 0.75$
  • $V(A) = 0$
• TD: Uses bootstrapping, updates a state's value using the estimated value of the next state.
  • $V(B) = 0.75$
  • $V(A) \leftarrow \gamma + \gamma V(B) = 0 + 0.75 = 0.75$
  • Requires a backup of $A$ after $V(B)$ has been learned.

Planning with a model​

• Given Model $\mu_\eta = \langle P_{\eta}, R_{\eta} \rangle$
• Solve $\text{MDP} \langle S, A, P_{\eta}, R_{\eta} \rangle$ by $\eta$.
• With Planning algorithm:
  • Value Iteration
  • Policy Iteration
  • Tree searching
  • etc.

Sample-based Planning​

• Generate a lot of samples or data then, use model free methods such as MC, SARSA, Q-learning, etc. to plan.
• $S_{t+1} \sim P_{\eta}(S_{t+1} | S_t, A_t)$
• $R_{t+1} \sim R_{\eta}(R_{t+1} | S_t, A_t)$
• Data Efficiency: Reduce real environment interactions by learning from simulated experiences.

Monte Carlo Search​

• Given a model $\mu_v$ and simulated policy $\pi$
• For each $a \in A$:

• Simulate $k$ episodes from current state $s_t: \{ s_t, a, R_{t+1}^k, \ldots, S_T^k \} \sim \mu_v, \pi$
• Evaluate actions by mean returns (MC evaluation)
• $G_t^{(k)}$: Return from the $k$-th simulation.
• Select current action with max value

$a_t = \arg\max_{a \in A} Q(s_t, a)$

e.g.

• $Q(s_t, \text{Left}) = \frac{3 + 5 + 4}{3} = 4$
• $Q(s_t, \text{Right}) = \frac{2 + 1 + 3}{3} = 2$
• Select action with max value: $\text{Left}$

Expectimax Tree​

• White Node: Choose action with max value.
• Black Node: Average state values of all possible actions.
• Compute Optimal $Q(s,a)$ values for the current state by construction an expectimax tree.

• $P_\nu(s'|s,a)$: Transition probability.
• $R_\nu(s,a,s')$: Expected Reward
• $V(s')$: Value of the next state.

Limitations of Expectimax Tree​

• Scaling Exponentially with Depth: Tree size grows exponentially with depth $H$.
• $|\mathcal S|$: Number of possible states.
• $|\mathcal A|$: Number of possible actions.
• $H$: Depth of the tree.

Monte Carlo Search Tree, MCTS​

• Given a model $\mu_v$
• Build a search tree rooted at the current state $s_t$, sample actions and next states.
• Iteratively Construct and update the tree by performing $K$ simulation episodes starting from the root node.
• Once search is finished, select the current action with maximum value in the search tree.

$a_t = \arg\max_{a \in A} Q(s_t, a)$

Tree Traversal​

• Use Upper Confidence Bound (UCB) to select which branches to explore.
• $c$: Exploration coefficient.
• $n(s)$: Number of visits to state $s$.
• $n(s,a)$: Number of visits to state-action pair $(s,a)$.
• Exploration Bonus: Encourages selecting less-visited actions.
• UCB: Balances exploitation of high-value actions and exploration of less-visited actions.
  • In bandit problems, $c$ is a constant to balance exploration and exploitation.

Node expansion​

• Add a new child node to the search tree for an unexplored action.

Rollout​

• Perform a random simulation from the expanded node until termination or a depth limit.
• Calculate the total return from the simulation.

Backpropagation​

• Propagate the simulation return back through the selected path to the root.
• Update visit counts and estimated values of the visited nodes.

Advantages of MCTS​

• Highly Selective Best-First Search
• Evaluates states dynamically, unlike DP.
• Use Samples to break curse of dimensionality.
• Works for "black box" model.
• Computationally efficient.

Reinforce Learning with Human Feedback, RLHF​

Why Human Feedback?​

• Some tasks do not have clear or automatically measurable rewards.
• Humans can provide:
  • Preferences: Choose between two outputs (A > B).
  • Rankings: Rank multiple outputs (1st, 2nd, 3rd).
  • Evaluations: Judge whether an answer is better or worse.
• Example: Evaluating which chatbot response is more polite.

Alignment Problem​

• LLMs are pretrained to predict text, not necessarily to generate useful responses.
• Generated text may be harmful, biased, or factually incorrect.
• Traditional evaluation metrics (e.g., BLEU) do not directly capture human preferences.
• Human feedback helps align model behavior with human preferences for helpfulness, truthfulness, and safety.
• Example: A model may generate plausible but impractical coding advice.

Core Idea of RLHF​

• Human Feedback: Use human evaluations and preferences to guide model training.
• Alignment: Align model behavior with human expectations and preferences.
• Optimization: Train models to maximize rewards reflecting human preferences rather than only next-token likelihood.
  • Goal of LLM's pretraining: $\max_\theta \sum_{t}\log P_\theta(x_t | x_{<t})$
    • Maximize the likelihood of the next token.
  • Goal of RLHF: $\max_\theta \mathbb{E}_{y \sim \pi_\theta(\cdot|x)} R(x, y)$
    • Maximize the expected reward of human-preferred responses.
• Application: Widely used in NLP and LLM alignment.

RLHF Training Pipeline​

Pre-training → SFT → Reward Model Training → RL Optimization

1. Pre-training: Train a base model on a large text corpus using next-token prediction.
2. Supervised Fine-Tuning (SFT): Fine-tune the base model on high-quality instruction-response examples.
3. Reward Model Training: Train a reward model using human-labelled preferences to predict response quality.
4. Reinforcement Learning: Optimize the model's policy to maximize rewards predicted by the reward model.

Step 1. Pre-training​

• Select a base model pretrained on a massive text corpus.
• Learns general knowledge and language patterns.
• Limitation: May produce biased, unsafe, or irrelevant responses.

Step 2. Supervised Fine-Tuning (SFT)​

• Goal: Train the base model to follow instructions and generate reasonable responses.
• Collect high-quality demonstration data (instruction, input, output).
• Fine-tune the base model on the curated dataset.
• Limitation: Instruction-following does not guarantee alignment with human preferences.

Step 3. Reward Model Training​

• Goal: Learn a reward function that reflects human preferences.
• Generate multiple responses for the same prompt.
• Humans compare or rank the responses.
• Train a reward model to assign scores based on human preferences.
• Example: A professional, polite response (B) is preferred over a blunt response (A): B > A.
• Outcome: Convert human judgments into numerical reward scores.

Step 4. RL with Human Feedback​

• Goal: Optimize the LLM policy using the trained reward model.
• Algorithm: PPO (example).
• Generate responses using the current policy.
• Score responses using the reward model from Step 3.
• Update the policy to maximize reward.
• Use a KL penalty to limit deviation from the reference policy and reduce over-optimization.
  • KL Penalty: Penalizes large deviations between the updated policy and the reference policy.
  • Higher KL Penalty → More conservative policy updates.

Chatgpt​

Benefits of RLHF​

• Alignment: Aligns model behavior with human values and preferences.
• Safety & Usefulness: Encourages safer, more useful responses with higher human acceptance.
• Ambiguous Goals: Handles tasks where rewards are difficult to define explicitly.

Challenges of RLHF​

• Cost: Requires extensive human annotations, making training expensive.
• Human Bias: Biases in human feedback can be transferred to the reward model and policy.
• Over-optimization: Models may exploit the reward function rather than produce genuinely better responses (Reward Hacking).
• Scalability: Collecting and applying human feedback becomes difficult as models and tasks scale.

Applications of RLHF​

• ChatGPT: Improvement in safety and helpfulness
• Claude: Constitutional AI for more nuanced alignment
• Gemini: Large-scale RLHF for multimodal capabilities
• Robotics: Learning task from human demonstrations
• Teaching robots via human feedback to cook
• Medical AI systems trained with doctor's feedback to align with human expert medical standards.

Instead of $V$ and $Q$ functions, we directly estimate the optimal policy.

$\pi^{\theta}(a|s) \approx P(a|s, \theta)$

• Policy was generated directly from the value function (e.g. $\epsilon$-greedy)
• Parameterize the policy directly.
  • Instead of $v(s,w)$ or $\hat{q}(s,a,w)$, we have $\pi^{\theta}(a|s)$
  • Given a particular state $s$, what actions should we take? (What is probability of that particular action?)
  • $\theta$ is a learnable parameter which could be a neural network.

Value-based and Policy-based Methods​

• Value-based: Learnt value function, Implicit policy (e.g. $\epsilon$-greedy)
• Policy-based: No value function, Learnt policy.
  • Good convergence properties
  • Effective in high-dimensional or continuous action spaces
    • if it is really continuous and large, it is not efficient as well.
  • Can learn stochastic policies.
  • Sometimes policies are relatively simple, values and models are complex.
  • Typically reaches local optima.
  • Obtained knowledge is specific and does not generalize well.
• Actor-Critic: Learnt value function, Learnt policy.

Policy Gradient Techniques​

• Trust-Region Based: Optimize the policy in a trust zone (closer circle).

Stochastic policies​

Rock-Paper-Scissors​

• A deterministic policy is easily exploited
  • Deterministic policy: If we follow a set of action or sequence of actions, we will definitely reach the goal and will always reach teh same location.
• A uniform random policy is optimal policy.

Grid World​

• Agent cannot differentiate between the grey states.
• $\phi(s,a)$ features describing state $s$ and action $a$.
• $\phi(s,E) = (\underbrace{1, 0, 1, 0}_{\text{NESW}}, \underbrace{0, 1, 0, 0}_{\text{NESW}})$ is the North and the South are walls and the agent will move to the East.
• If $\pi(\text{grey}) = W$, then when the agent is in the left grey state, it will move west, get stuck in a loop, and never reach the goal.

• Therefore, an optimal stochastic policy moves randomly $E$ or $W$ in grey states.
  • $\pi_{\theta}(\text{wall to N and S, move E}) = 0.5$
  • $\pi_{\theta}(\text{wall to N and S, move W}) = 0.5$
• The agent will reach the goal with high probability.
• Policy-based methods can learn Stochastic policies. 4563

Objective Function in Policy-based Methods​

Given a policy $\pi_{\theta}(a|s)$ with parameters $\theta$, the goal is to find the best $\theta$.

• In an episodic task (with an end state), $J_1(\theta)$ is for that particular start state and measures how much expected return we can get.
• $S_1$ is the start state.
• $J(\theta)$ is the measure of the quality of the policy or the objective function to optimize.

• In continuing environments (doesn't have an end state), consider the states and take the average value weighted by how often the agent visits each state.
• $\mu_{\pi\theta}(s)$ is how often the agent is visiting that particular state $s$.

• Use the average reward per time step.
• Small batches can be used to estimate this average reward during training.

Policy-based Methods​

• Policy-based RL is an optimization problem, find $\theta$ that maximize $J(\theta)$.
  • $J(\theta)$ is the policy objective function.
• Gradient-free methods: Hill Climbing, Genetic Algorithms, ...
• Gradient-based

$\Delta{\theta} = \alpha \nabla_{\theta} J(\theta)$

• Policy gradient method will search for a local minimum in $J(\theta)$ by ascending the gradient of the Policy.
  • why ascending? because we want to maximize the objective function's value/rewards.
  • $\nabla_{\theta} J(\theta) = \begin{pmatrix} \frac{\partial J(\theta)}{\theta_1} \\ \vdots \\ \frac{\partial J(\theta)}{\theta_n} \end{pmatrix}$
  • $\alpha$ is the step size.

• Assume the policy $\pi_{\theta}$ is differentiable.
• Compute an estimate of the policy gradient.
• Compute $\Delta{\theta} J(\theta)$ using Monte Carlo samples since we need to explore the environment, gather information, and then learn the policy.

Score Function​

• Assume the policy $\pi_{\theta}$ is differentiable.
• $\nabla_{\theta} \pi_{\theta}(s, a)$ is the gradient.

• Multiplying and dividing by $\pi_\theta(s,a)$ is called the Likelihood Ratio Trick.
  • Since $\frac{\pi_\theta(s,a)}{\pi_\theta(s,a)} = 1$, this does not change the original value.
    • $\nabla_\theta \pi_\theta(s,a)$ is Gradient of the policy
    • $\pi_\theta(s,a)$ is the policy.
    • $\frac{\nabla_\theta \pi_\theta(s,a)}{\pi_\theta(s,a)}$ measures how sensitive the policy is to changes in $\theta$, relative to the policy value itself.
  • From the derivative rule of the logarithm, $\frac{\nabla_\theta \pi_\theta(s,a)}{\pi_\theta(s,a)} = \nabla_\theta \log \pi_\theta(s,a)$.
• Score Function: $\nabla_\theta \log \pi_\theta(s,a)$.