Unit 4
By
P. Steffy Sherly,
Assistant professor/CSE/SOC/VTU
MODEL BASED RL
Model-Based Reinforcement
Learning (Model-Based RL) is a class
of RL methods where the agent
learns a model of the environment
and uses it for planning to make
better decisions.
Model BASED RL
Policy Optimization, Continuous Learning:
KEY COMPONENTS OF MBRL
Launch of new policy to a wider audience
MODEL FREE VS MODEL BASED
Model-Based RL is significantly more sample-efficient, making it ideal for scenarios
where real-world data is expensive (e.g., robotics, healthcare).
Model-Free RL is computationally simpler, especially at test time. In contrast,
Model-Based RL incurs extra cost for planning and model updates.
Final TRADE OFF TABLE
Criteria Model-Free RL Model-Based RL
Sample Efficiency ❌ Low ❌ High
Computational Complexity
❌ Lower ❌ Higher
(Training)
❌ Can be slow (planning
Test-Time Complexity ❌ Fast
needed)
❌ No (model error
Robustness to Model Error ❌ Yes
accumulates)
Exploration Quality ❌ Often heuristic ❌ Can plan uncertain regions
Games, large-scale learning Real-world robotics, data-
Use Case
with simulators limited settings
Bayesian Approaches to
Reinforcement Learning
representing uncertainty about how a system transitions between
states using a probability distribution.
Bayesian RL: Example Methods
Method Description
Augments the MDP state with a belief over
Bayes-Adaptive MDP (BAMDP) dynamics; planning is done in this expanded
space
Sample an MDP from the current posterior and
Posterior Sampling for RL (PSRL)
follow its optimal policy
Sample model parameters from posterior and act
Thompson Sampling
optimally under that sample
Maintain distributions over Q-values (e.g.,
Bayesian Q-Learning
Gaussian) instead of single estimates
Approximate Bayesian exploration via ensemble
Bootstrapped DQN
of Q-networks
Use Bayesian neural networks or MC dropout to
Bayesian Deep RL model uncertainty in deep policies or value
functions
What is data efficient learning?
Data efficient learning is a type of machine learning that can understand
complex domains without the need for large amounts of data.
Data efficient learning utilizes smaller data sets, with the aim of learning more
quickly, while still maintaining the same performance level, when compared to
other machine learning methods
Data efficient machine learning often utilizes the reinforcement learning
method. This means that these algorithms use a reward and punishment system
as part of their programming.
Reinforcement learning can be paired with a neural network Gaussian process to
improve outcomes. Gaussian processes are used to assign probabilities to events
and outcomes.
Data efficient learning can help businesses and organizations in many ways, including:
Quickly processing data: Rather than having to spend a lot of time analyzing large
amounts of data, data efficient processes can quickly assess small amounts of data and
return relevant results and predictions.
No need for big data: In traditional machine learning methods, a large amount of data is
required before accurate predictions can be made. This could mean that data just sits in
storage until enough has been gathered to use effectively. In comparison, efficient learning
can start analyzing data and returning predictions much more quickly.
Protect data privacy: Because only a small amount of data is required for these processes,
there is no need to store large amounts of data and unnecessarily risk a data breach.
The
number of samples directly related to data efficiency — fewer
samples to reach good performance means higher efficiency.
PAC-MDP -Probably Approximately Correct
Markov Decision Process
It is a formal framework in reinforcement learning (RL) that provides sample
complexity guarantees — essentially answering the question:
How many steps (or episodes) does an RL agent need before its policy is
near-optimal with high probability?
Algorithms like R-MAX and E³ are examples of PAC-MDP algorithms that explore
optimistically and achieve good sample complexity.
in PAC-MDP, sample complexity refers to the number of time steps where the agent's policy
is not ϵ\ϵ-optimal.
Typical bound:
(Explicit Explore or Exploit)
Upper Confidence Reinforcement
Learning
R-MAX Algorithm
it is used to estimate unknown quantities like rewards or transition probabilities based on
sampled interactions.
E³ (Explicit Explore or Exploit)
UCRL (Upper Confidence
Reinforcement Learning)
Empirical Empirical estimation
estimation refers to the process of estimating the reward
function and transition probabilities of an environment based on the
actual experiences collected by the agent . It plays a critical role in
model-based reinforcement learning algorithms such as R-MAX, E³,
and UCRL.
Learning with Off-line Data
Off-policy Evaluation:
Evaluation is done by 3 algorithms
LSTD - Least-Squares Temporal Difference Learning
LSPE - Least-Squares Policy Evaluation
Fitted Q-Iteration
LSPE — Least-Squares Policy
Evaluation
solving complex problems step by step by repeatedly improving an estimate based on
some update rule — until it converges to the correct or desired solution.
Fitted Q-Iteration (FQI)
Learning with incompletely specified
models
Reinforcement Learning (RL) and
Planning
Classical Conditioning and RL
a stimulus refers to any signal or input from the environment that the agent can observe and use to make decisions.
Classical conditioning mechanisms can be framed as value function
learning using RL tools like TD Learning.
Key concepts
Example: Classical Conditioning via
TD Learning
Example: Classical Conditioning via
TD Learning
Blocking and Higher-order
Conditioning
Higher-order Conditioning (Second-
order Conditioning)
The Rescorla - Wagner Model
Instrumental Conditioning
In Instrumental Conditioning, we use TD learning with action values (Q-learning or SARSA).
Key Algorithms Using Cognitive Maps
Formal Definition
Advantages
END OF UNIT 4