RL 002

Sequential Decision Making

• A sequence is a series of events or actions that occur in a specific order over time.
• Goal: select actions that maximize total expected future reward.
• It may require balancing immediate and long-term rewards.
• It may be better to sacrifice short-term gains for long-term benefits.
• It may require strategic behavior to achieve high rewards.

Example of SDM

• Web Ads
  • Agent AdBot
    1. Choose and Ad $A_t$.
    2. Receives View time $O_t$
    3. Receives Click on Ad $R_t$
  • Environment
    1. Receives the Chosen Ad $A_t$.
    2. Provides the View time $O_{t+1}$.
    3. Provides the Click on Ad $R_{t+1}$.
• Robot picking trash
  • Agent Robot
    1. Moves arms to pickup trash $A_t$.
    2. Receives Camera image of the room $O_t$.
    3. Receives Reward if no trash on the floor $R_t$
  • Environment
    1. Receives robots action/arm movements $A_t$.
    2. Provides Camera Image of the room $O_{t+1}$.
    3. Provide reward of +1 $R_{t+1}$
• Robot making Pizza.
  • Agent Robot
    1. Makes an action from Action space $A_t$ (a range of possible actions, Arm Movements, Get Pizz from Oven, Place Pizza on Tray, Cut Pizza...).
    2. Receives Camera image of the kitchen $O_t$.
    3. Receives Reward if pizza is made correctly $R_t$
  • Environment
    1. Receives robots action/arm movements $A_t$.
    2. Provides Camera Image of the kitchen $O_{t+1}$.
    3. Provide reward appropriate to the action $R_{t+1}$.
  • Reward space
    • +10 Successfully made pizza.
    • +1 Get Pizza from oven.
    • +1 Place Pizza on tray.
    • +1 Get Packing box.
    • +1 Cut Pizza successfully.

Rewards

• a scalar signal that indicates how well the agent is doing at time step $t$. (agent's performance)
• Examples of rewards:
  • Robot play soccor:
    • $+ve$ reward for scoring a goal.
    • $-ve$ reward for kicking the ball out of bounds.
  • Chess
    • $+ve$ reward for winning the game.
  • Autonomous drone flying stunt:
    • $+ve$ reward for following the intended trajectory.
    • $-ve$ reward for crashing into an obstacle.