BCSE432E –
Reinforcement Learning
Reinforcement Learning (RL)
• RL is a general framework for
training an artificial intelligence
model to solve a certain task or goal.
• Reinforcement learning is a
framework to learn any task.
• In machine learning and optimal
control, RL is concerned with how an
intelligent agent should take actions
in a dynamic environment in order
to maximize a reward signal.
Hide and Seek Video from OpenAI
AlphaStar
Deepmind’s AlphaStar taking on a pro-gamer in the game StarCraft 2.
AlphaStar
• AlphaStar is an intriguing and unorthodox player – one with
the reflexes and speed of the best pros but strategies and a
style that are entirely its own.
• The way AlphaStar was trained, with agents competing
against each other in a league, has resulted in gameplay
that’s unimaginably unusual; it really makes you question
how much of StarCraft’s diverse possibilities pro players have
really explored.
Robotics
The ANYmal robot from the Robotics System Lab in Zürich learned to recover from a fall
Real world examples
• Goal-oriented Chatbot,
Trained to Negotiate
about Sales.
[Link]
-acl-2022/
Elements of RL
• A description of the RL framework is as follows: We have an
agent that tries to solve a task in a certain environment. The
concept of agent should be taken very broadly here, an
agent can be a robot, a chatbot, a virtual character, etc.
• At every timestep t, the agent needs to choose an action a.
After this action it might receive a reward r and we get a new
observation of its state s.
• The new state can be determined both by the action of the
agent and also by the environment the agent is operating in.
• The RL problem is trying to maximize the cumulative reward
the agent gets over time.
• We are now searching for the optimal behavior, the
best sequence of actions the monkey can take, to
maximize the cumulative number of bananas it will
get.
• The likely reason for this intersection of domains, is that
reinforcement learning is the study of a fundamental
problem.
• It is essentially the science of decision taking.
• We will be looking at it from the umbrella of computer
science and machine learning.
Field of Machine Learning
• Supervised learning
• Unsupervised learning
• Reinforcement Learning
The exploration-exploitation trade-off
The exploration-exploitation trade-off
• Whenever we train an RL-agent, the agent will need some time to
explore, it needs to take some actions that it hasn’t taken before, in
order to discover how to solve the problem.
• On the other hand, we can’t let the agent always take random
actions, because these random actions might lead to nothing.
• Sometimes we want the agent to leverage what it has already
learned to try and optimize further.
• This is the exploration-exploitation trade-off, we want an automated
way to strike a good balance between letting the agent explore and
taking actions for which it already knows what they will lead to.
The sparse-reward problem
Tic-Tac-Toe
• More accurate estimates of the probabilities of winning
• Back up the value of the state after each greedy move to
the state before the move, as suggested by the arrows
• Current value of the earlier state is updated to be closer
to the value of the later state.
• This can be done by moving the earlier state’s value a
fraction of the way toward the value of the later state.
• If we let St denote the state before the greedy move, and St+1
the state after the move, then the update to the estimated
value of St, denoted V (St), can be written as
• Where α is a small positive fraction called the step-size
parameter, which influences the rate of learning.
• This update rule is an example of a temporal-difference
learning method, so called because its changes are based on
a difference, V(St+1) − V(St), between estimates at two
successive times.
A good way to understand reinforcement learning is to consider some
of the examples and possible applications that have guided its
development
• A master chess player makes a move. The choice is informed both by
planning—anticipating possible replies and counterreplies—and by
immediate, intuitive judgments of the desirability of particular positions
and moves.
• An adaptive controller adjusts parameters of a petroleum refinery’s
operation in real time. The controller optimizes the yield/cost/quality
trade-o↵ on the basis of specified marginal costs without sticking strictly to
the set points originally suggested by engineers.
• A gazelle calf struggles to its feet minutes after being born. Half an hour
later it is running at 20 miles per hour.
• A mobile robot decides whether it should enter a new room in search of
more trash to collect or start trying to find its way back to its battery
recharging station. It makes its decision based on the current charge level
of its battery and how quickly and easily it has been able to find the
recharger in the past.
• Phil prepares breakfast by performing a sequence of goal-oriented
actions. He continuously observes the environment and makes decisions
at each step. This demonstrates intelligent, goal-directed behavior through
interaction with the environment.
In all of these examples the agent can use its experience to improve
its performance over time.
• The chess player refines the intuition he uses to evaluate
positions, thereby improving his play;
• The gazelle calf improves the efficiency with which it can run;
• Phil learns to streamline making his breakfast.
• The knowledge the agent brings to the task at the start—either
from previous experience with related tasks or built into it by
design or evolution—influences what is useful or easy to learn,
but interaction with the environment is essential for adjusting
behavior to exploit specific features of the task.
Elements of Reinforcement Learning
• The elements of Reinforcement Learning (RL) are the
fundamental components that define how an intelligent
agent learns by interacting with its environment. The four
main elements are:
1. Agent
• The learner or decision-maker. It observes the environment and
chooses actions.
• Example: A robot navigating a maze or a chess-playing program.
2. Environment
• Everything the agent interacts with. It responds to the agent's
actions by changing state and providing rewards.
• Example: The maze, chessboard, or a self-driving car's
surroundings.
3. State (S)
A representation of the current situation of the environment.
The agent uses the current state to decide what action to take.
Example:
Robot: Current location in the maze.
Chess: Current board configuration.
4. Action (A)
A choice made by the agent. Actions affect the environment and
may lead to new states.
Example: Move left, right, up, or down.
Accelerate, brake, or turn.
5. Reward (R)
• A numerical feedback signal from the environment.
Indicates how good or bad an action was.
• The objective is to maximize the cumulative reward.
Example:
+10 for reaching the goal.
-1 for hitting an obstacle.
6. Policy (π)
• The strategy the agent follows to choose actions.
• Maps states to actions.
• Can be: Deterministic: One action for each state.
• Stochastic: A probability distribution over actions.
7. Value Function (V)
• Estimates how good a state is.
• Measures the expected cumulative future reward starting
from a state.
8. Action-Value Function (Q)
• Estimates how good taking a particular action in a given state is.
9. Model (Optional)
Predicts how the environment behaves.
Given a state and action, it predicts:
• The next state.
• The reward.
Used in model-based RL. Some algorithms (called model-
free RL) do not require a model
The RL framework
• The RL framework involves an agent that tries to solve a task
in a certain environment.
• At every timestep t, the agent needs to choose an action a.
• After this action it might receive a reward r and we get a
new observation of its state s.
• The new state can be determined both by the action of the
agent and also by the environment the agent is operating in.
• In RL we want to maximize the future accumulated reward.
Interaction Cycle
• The cycle repeats:
• Observe the current state.
• Choose an action.
• Execute the action.
• Receive a reward.
• Observe the next state.
• Update the policy or value
estimates.
Limitations of Reinforcement Learning
• Large Data Requirement: RL often requires a vast number of
interactions with the environment before learning an effective
policy.
• High Computational Cost: Training RL models can be
computationally expensive and time-consuming.
• Exploration vs. Exploitation Trade-off: Balancing exploration of
new actions with exploitation of known good actions is
challenging.
• Sparse or Delayed Rewards: Learning becomes difficult when
rewards are infrequent or received long after actions are taken.
• Safety Concerns: Trial-and-error learning may be unsafe in real-
world applications such as healthcare or autonomous driving.
• Difficult Reward Design: Poorly designed reward functions
can lead to unintended or suboptimal behaviors.
• Slow Convergence: RL algorithms may require many
training episodes before converging to an optimal policy.
• Scalability Issues: Performance may degrade in
environments with very large or continuous state and
action spaces.
• Lack of Generalization: Policies learned in one environment
may not transfer well to different environments.
• Limited Interpretability: Many deep RL models are difficult
to understand and explain, making their decisions less
transparent.
Scope of Reinforcement Learning
• Reinforcement Learning (RL) is widely used in domains
where an agent learns optimal actions through interaction
with an environment.
• Robotics: Robot navigation, manipulation, path planning,
and autonomous control.
• Gaming: Mastering complex games such as chess, Go, Atari
games, and multiplayer strategy games.
• Autonomous Vehicles: Decision-making for self-driving cars,
lane keeping, obstacle avoidance, and route planning.
• Healthcare: Personalized treatment planning, drug discovery,
and medical resource optimization.
• Finance: Algorithmic trading, portfolio optimization, fraud
detection, and risk management.
• Recommendation Systems: Personalized recommendations
for movies, music, products, and online content.
• Industrial Automation: Process optimization, predictive
maintenance, and smart manufacturing.
• Telecommunications: Network routing, bandwidth
allocation, and traffic management. Smart Energy Systems:
Energy management, smart grids, and demand-response
optimization.
• Smart Energy Systems: Energy management, smart grids,
and demand-response optimization.
• Natural Language Processing: Dialogue systems, text
generation, and machine translation optimization.
Main Concepts of RL
• States
• Actions
• Reward functions
• Environment dynamics (Markov Decision Processes)
• Policies
• Trajectories and return
• Value function & Q-function
Game of Pong
• There are two players, represented
by the green and the red paddles.
• In this example, we will consider
the player who operates the green
paddle to be our agent.
• The goal of Pong is to bounce back the ball, past the other
players’ paddle.
• When either player manages to bounce the ball past the other
player, they get a point and the ball starts moving from the
centre of the field again. The player with the highest score wins.
States
• A state is usually a vector, matrix or other tensor.
• The state should describe the relevant information needed in
order to decide which action to take at timestep t. It will be
denoted as the state at timestep t as sₜ .
States
• In the case of Pong, a good description of the state can be a
vector (an array of numbers) containing the position of the
players’ paddle, the position of the ball and the angular
velocities of the ball.
• This would be enough for our agent to decide whether it’s
best to move the paddle up or down.
• Because from this information it can check its own position,
the position of the ball and where the ball is moving
towards.
Actions
• In Pong we want our agent to be able to move the paddle up
and down.
• Discrete Action Space
• Continuous Action Spaces
Reward function
• The reward function tells us after every action how much
reward the agent got for taking that action.
• The reward function takes a state Sₜ and an action Aₜ as
input at every timestep t, and outputs a reward Rₜ . The
reward Rₜ is a scalar (number) that indicates how well the
agent did.
• In Pong, we could opt for giving a reward of 1 every time the
agent manages to make the ball go past the other players’
paddle, 0 when nobody scores, and -1 when he misses the
ball causing the opponent to score.
• Formally, the reward function has the following signature:
• Where S represents the set of states, A the set of actions and
ℝ the set of real numbers.
Environment dynamics
• The environment dynamics answer the following question:
given a state and an action, what will be the next state?
• Typically, we don’t need to model these environment
dynamics ourselves.
• Our RL algorithm will need to figure them out on its own.
Even more so, for many problems the environment dynamics
are unknown.
• We will briefly talk about the idea of environment dynamics
nonetheless, because the only requirement for us to be able
to apply RL to a problem, is that we are in theory able to
define the problem as a Markov Decision Process (MDP).