Reinforcement Learning

Module 6
01-04-2026
 Reinforcement Learning
Module 6

• Introduction to Deep Learning

• Deep-Q Learning
• Value-based Deep RL: Deep-Q network
• Policy-based Deep RL: REINFORCE
• Asynchronous methods for Deep RL:
• Advanced Actor Critic (A2C)
• Asynchronous Advanced Actor Critic (A3C)
• Model-based Deep RL

01-04-2026
 Introduction to Deep Learning

01-04-2026
Deep-Learning: Introduction
Deep Learning is a subfield of Machine Learning that uses multi-layer neural networks to approximate
complex functions.
• Mathematically:
𝑦=𝑓 𝑥 𝜃
where: 𝑥= input, 𝑦= output, 𝜃= parameters (weights & biases), 𝑓= neural network function
• In RL: 𝑥= state, 𝑦= value 𝑉 𝑠 ,Q-value 𝑄 𝑠 𝑎 ,or policy 𝜋 𝑎 ∣ 𝑠

• Earlier ML used, Linear regression, Logistic regression, Simple models. These fail when:
• Data is high-dimensional (images, audio)
• Patterns are complex and nonlinear
• Deep networks can approximate highly nonlinear functions.
• This is supported by the Universal Approximation Theorem:
• A neural network with at least one hidden layer can approximate any continuous function
(under mild conditions).
01-04-2026
Deep-Learning: Introduction
Neural networks are inspired by biological neurons. Artificial Neuron Model
𝑧 = 𝑤𝑇𝑥 + 𝑏
𝑎=𝜎 𝑧
where, 𝑤= weights, 𝑏= bias, 𝜎= activation function, 𝑎= output
A deep neural network has Input layer, One or more hidden layers, and Output layer
• Each layer performs:
𝑙
𝑎 =𝜎 𝑊 𝑙 𝑎 𝑙−1
+𝑏 𝑙

01-04-2026
Deep-Learning: Activation Functions (mostly
used)

01-04-2026
Deep-Learning: Loss Functions
Loss Function Mathematical Formulation Typical Use

Mean Squared Error (MSE) - Regression, value prediction

Mean Absolute Error (MAE) - Regression, value prediction

Binary Cross-Entropy - Binary classification

Categorical Cross-Entropy - Multi-class classification

Hinge Loss - Support Vector Machines (SVM)

01-04-2026 7
Deep-Learning: Gradient Descent
Gradient Descent is an optimization algorithm used to minimize a loss (cost) function by iteratively
updating model parameters in the direction of the negative gradient of the loss.
It is widely used in deep learning and deep reinforcement learning for training neural networks.

• Objective Function:
Suppose we have a model with parameters:
𝜃 = 𝜃1 𝜃2 … 𝜃𝑛
We define a loss function:
𝐿 𝜃
The goal is to find parameters that minimize the loss:
𝜃 ∗ = 𝑎𝑟𝑔 𝑚𝑖𝑛 𝐿 𝜃
𝜃
Example (squared error):
ො 2
𝐿 𝜃 = ቀ𝑦 − 𝑦)

01-04-2026
Deep-Learning: Gradient Descent
• Initialize Parameters
Choose an initial value for parameters:
𝜃0
Usually initialized randomly or with small values or small random numbers. Example
𝜃0 =0
• Compute the Gradient of the Loss Function
The gradient of the loss with respect to parameters tells us the direction of steepest increase of
the loss.

𝜕𝐿 𝜕𝐿 𝜕𝐿
∇𝜃 𝐿 𝜃 = …
𝜕𝜃1 𝜕𝜃2 𝜕𝜃𝑛
For a single parameter case:
𝑑𝐿
𝑑𝜃

01-04-2026
Deep-Learning: Gradient Descent
• Update the Parameters
Parameters are updated in the opposite direction of the gradient to reduce the loss.
Update rule:
𝜃𝑛𝑒𝑤 = 𝜃𝑜𝑙𝑑 − 𝛼∇𝜃 𝐿 𝜃
Where 𝛼= learning rate, ∇𝜃 𝐿 𝜃 =gradient

• Repeat Until Convergence

The steps of computing gradients and updating parameters are repeated until:
Loss stops decreasing, or
Gradient becomes very small
Mathematically:
∥ ∇𝜃 𝐿 𝜃 ∥≈ 0

01-04-2026
Deep-Learning: Gradient Descent
• Iterative Update Representation
At iteration 𝑘:
𝑘+1 𝑘 𝑘
𝜃 =𝜃 − 𝛼∇𝜃 𝐿 𝜃
This continues for:
𝑘 = 0,1,2, …
until convergence.

01-04-2026
Deep-Learning: Gradient Descent
• Example: Suppose: 𝐿 𝜃 = (𝜃 − 3)2
𝑑𝐿
Derivative: =2 𝜃−3
𝑑𝜃
Let: 𝛼 = 0.1
Initial value: 𝜃0 = 0
• Iteration 1
𝑑𝐿
Gradient: = 2 0 − 3 = −6
𝑑𝜃
Update: 𝜃1 = 0 − 0.1 −6 = 0.6

• Iteration 2
Gradient: 2 0.6 − 3 = −4.8
Update: 𝜃2 = 0.6 − 0.1 −4.8 = 1.08
The parameter gradually moves toward 3, which minimizes the loss.

01-04-2026
Deep-Learning: Backpropagation
Backpropagation: Backpropagation computes gradients using the chain rule.
Core idea:
𝑑𝐿 𝑑𝐿 𝑑𝑎 𝑑𝑧
= ⋅ ⋅
𝑑𝑊 𝑑𝑎 𝑑𝑧 𝑑𝑊

The following steps are performed to minimize the cost function

• Forward pass
• Loss computation
• Backward pass
• Weight update

01-04-2026
Deep-Learning: Learning Process

01-04-2026 Source: [Link]

Deep Learning: Types of DNNs
Types of NN: [Link]

Network Type Main Purpose Typical Applications

Feedforward NN General function approximation Regression, classification

CNN Spatial feature extraction Image recognition

RNN Sequential data processing Time-series, speech

LSTM Long-term sequence modeling NLP, translation

GRU Efficient sequence learning Text prediction

Autoencoder Feature learning Compression, anomaly detection

GAN Data generation Image synthesis

Transformer Attention-based sequence modeling NLP, language models

01-04-2026 15
Deep-Learning: Why Deep Learning for RL?
Earlier in RL:
𝑉 𝑠 = 𝑡𝑎𝑏𝑙𝑒
But in real-world problems:
• State space huge
• Continuous states
• Image-based input
• High dimensional features

Deep learning allows:

𝑉(𝑠; 𝑤)
𝑄(𝑠, 𝑎; 𝑤)
𝜋(𝑎|𝑠; 𝜃)
• This leads to Deep Reinforcement Learning.
01-04-2026
Deep-Learning: Limitation
Limitations of Deep Learning
• Requires large data
• High computational cost
• Hyperparameter sensitivity
• Overfitting
• Vanishing / Exploding gradients

01-04-2026
 Deep-Q Learning

01-04-2026
Deep-Q Learning
Recall tabular Q-learning:

This works only if:

• State space is small
• We can store Q-values in a table
Fails when:
• State is an image
• Continuous state space
• Millions of states
• Example:
Atari game → raw pixel input (84×84×4 frames), Number of possible states ≈ enormous
So instead of a Q-table, we approximate: 𝑄 𝑠 𝑎 𝑤 using a neural network.

01-04-2026
Deep-Q Learning: Core Concept
• Deep Q-Learning is an extension of the classical Q-learning algorithm that uses deep
neural networks to approximate the Q-function.
• In traditional Q-learning, the Q-values for each state-action pair are stored in a table.
However, when the state space becomes very large or continuous, maintaining such a
table becomes infeasible.
• Deep Q-Learning addresses this limitation by replacing the Q-table with a neural network
that estimates Q-values.
In Deep Q-Learning, the action-value function is represented as:
𝑄(𝑠, 𝑎, 𝑤)
Where, 𝑠= state, 𝑎= action, 𝑤= parameters (weights) of the neural network.

• The neural network takes the state as input and produces Q-values for possible actions
as output.
01-04-2026
Deep-Q Learning: Core Concept
Deep Q-learning approximates the Q-function using a neural network rather than a Q-table.
In classical Q-learning:
𝑄(𝑠, 𝑎)
In deep Q-learning:
𝑄(𝑠, 𝑎, 𝜃)
where 𝜃represents neural network parameters.
The update target is still derived from the Bellman equation:

The loss minimized is:

So deep Q-learning = Q-learning + neural network function approximation.

01-04-2026
Deep-Q Learning: Core Concept
• Bellman Equation in DQL
Deep Q-Learning is based on the Bellman optimality equation used in Q-learning:
𝑄∗ 𝑠 𝑎 = 𝔼 𝑟 + 𝛾𝑚𝑎𝑥 ′
𝑄∗ 𝑠 ′ 𝑎′
𝑎
The neural network is trained so that its Q-value predictions satisfy this relationship.

• Target Value
For each experience 𝑠 𝑎 𝑟 𝑠 ′ ,the target value is computed as:

′ ′
𝑦 = 𝑟 + 𝛾𝑚𝑎𝑥
′
𝑄 𝑠 𝑎 𝜃
𝑎
Where, 𝑟= reward received, 𝑠 ′ =next state, 𝛾= discount factor, 𝑎′ represents all possible
actions in the next state 𝑠 ′ , and 𝜃 represents the parameters (weights and biases) of the
neural network used to approximate the Q-function.

01-04-2026
Deep-Q Learning: Core Concept
• Loss Function
The neural network is trained by minimizing the difference between the predicted Q-value and
the target value.
𝐿 𝜃 = ቀ𝑦 − 𝑄 𝑠 𝑎 𝜃 )2
This is known as the Bellman error or temporal-difference error.
• Parameter Update
The network parameters are updated using gradient descent:
𝜃 ← 𝜃 − 𝛼∇𝜃 𝐿 𝜃
where 𝛼= learning rate.
• Exploration Strategy
Deep Q-Learning typically uses an ε-greedy policy to balance exploration and exploitation.
random action with probability 𝜖
𝜋 𝑎∣𝑠 =ቊ
arg 𝑚𝑎𝑥𝑎 𝑄 𝑠 𝑎 𝜃 with probability 1 − 𝜖
This ensures that the agent explores new actions while still exploiting learned knowledge.

01-04-2026
Deep-Q Learning: Learning Process
The learning process of Deep Q-Learning involves the following steps:
• Observe the current state 𝑠.

• Select an action using the ε-greedy policy.

• Execute the action and observe reward 𝑟 and next state 𝑠 ′ .

• Compute the target value using the Bellman equation.

• Calculate the loss between predicted and target Q-values.

• Update the neural network parameters using gradient descent.

• Repeat the process for many interactions with the environment.

01-04-2026
Deep-Q Learning: Advantage and Limitation
Advantages of Deep Q-Learning
• Handles large state spaces
• Enables generalization across similar states
• Works with high-dimensional inputs such as images
• Can be combined with other deep learning architectures

Limitations of Deep Q-Learning

• Training can be unstable
• Neural networks may diverge during learning
• Requires large amounts of data
• Sensitive to hyperparameters
These challenges motivated the development of Deep Q-Network (DQN), which introduces
stabilization techniques.

01-04-2026
Value-Based Deep RL: Deep Q-Network

01-04-2026
Value-Based Deep RL
Value-based Deep RL combines classical value-based methods (like Q-learning) with deep neural
networks to handle large or continuous state spaces.
• In traditional RL, We store values in a table that works only for small problems
• In deep RL, We approximate the value function using a neural network
𝑄(𝑠, 𝑎, 𝜃) ≈ 𝑄∗ (𝑠, 𝑎)
Where: 𝑠= state, 𝑎= action, 𝜃= neural network parameters (weights), Output = estimated action-value
Deep Q-Learning (DQL) is the foundation of value-based deep RL
Update Target:
′ ′
𝑦 = 𝑟 + 𝛾𝑚𝑎𝑥
′
𝑄(𝑠 , 𝑎 , 𝜃)
𝑎
Loss Function:
2
𝐿 𝜃 = 𝑦 − 𝑄(𝑠, 𝑎, 𝜃)
The network learns by minimizing this loss.

01-04-2026
Value-Based Deep RL: Deep Q-Network
DQN is a practical implementation of deep Q-learning. Key Components:
• Q-Function Approximation: Instead of using a table to store Q-values for each state-action pair, DQN uses a
neural network to approximate the Q-values. The input to the network is the state, and the output is a set of Q-
values for all possible actions.
• Experience Replay: To stabilize the training, DQN uses a memory buffer (replay buffer) to store experiences
(state, action, reward, next state). The network is trained on random mini-batches of experiences from this
buffer, breaking the correlation between consecutive experiences and improving sample efficiency.
• Store transitions:
(𝑠, 𝑎, 𝑟, 𝑠 ′ )
Train using random samples which breaks correlation
• Target Network: DQN introduces a second neural network, called the target network, which is used to calculate
the target Q-values. This target network is updated less frequently than the main network to prevent rapid
oscillations in learning.
• Maintain two networks; Online network: 𝑄(𝑠, 𝑎, 𝑤) and Target network: 𝑄(𝑠, 𝑎, 𝑤 − ).
• Update periodically: 𝑤 − ← 𝑤.
• This prevents moving target instability.

01-04-2026
Deep Q-Network: Architecture
DQN was introduced in Playing Atari with Deep Reinforcement Learning and extended in Human-level
control through deep reinforcement learning. It achieved human-level performance in Atari games using
only raw pixels.

DQN Architecture
• For image inputs (Atari):
• Convolutional layers
• Fully connected layers
• Output layer → Q-values for each action

• For low-dimensional states:

• Multi-Layer Perceptron (MLP)

01-04-2026
Deep Q-Network: Architecture

01-04-2026 Source: [Link]

Deep Q-Network: Mathematical Formulation
Step 1: Define Q-function:
𝑄(𝑠, 𝑎, 𝑤)
where, 𝑤= neural network weights
Step 2: Define Target:
′ ′ −
𝑦 = 𝑟 + 𝛾𝑚𝑎𝑥
′
𝑄(𝑠 , 𝑎 , 𝑤 )
𝑎
where, 𝑤 − =target network parameters
Step 3: Loss Function:
𝐿 𝑤 = 𝑦 − 𝑄(𝑠, 𝑎, 𝑤) 2 .
This is called Bellman error minimization.
Step 4: Gradient Update:
𝑤 ← 𝑤 − 𝛼∇𝑤 𝐿 𝑤
Equivalent to:
𝑤 ← 𝑤 + 𝛼𝛿∇𝑤 𝑄(𝑠, 𝑎, 𝑤)
where,
01-04-2026𝛿 = 𝑦 − 𝑄(𝑠, 𝑎, 𝑤)
Deep Q-Network: Algorithm
Initialization
• Initialize replay buffer:
𝒟=∅
• Initialize online network parameters:
𝑤
• Initialize target network parameters:
𝑤− = 𝑤

• For each time step 𝑡

1. Observe state
𝑠𝑡
2 .Select action (ε-greedy policy)

random action, with probability 𝜖

𝑎𝑡 = ቊ
arg 𝑚𝑎𝑥𝑎 𝑄 𝑠𝑡 𝑎 𝑤 , with probability 1 − 𝜖
01-04-2026
Deep Q-Network: Algorithm
3. Execute action and observe transition
𝑠𝑡 𝑎𝑡 𝑟𝑡+1 𝑠𝑡+1

4. Store transition in replay buffer

𝒟 ← 𝒟 ∪ 𝑠𝑡 𝑎𝑡 𝑟𝑡+1 𝑠𝑡+1

5. Sample mini-batch from replay buffer

൛ 𝑠𝑖 𝑎𝑖 𝑟𝑖+1 𝑠𝑖+1 }𝑁
𝑖=1
∼𝒟

6. Compute target values

𝑟𝑖+1 , if 𝑠𝑖+1 is terminal

𝑦𝑖 = ቊ
𝑟𝑖+1 + 𝛾𝑚𝑎𝑥𝑎′ 𝑄 𝑠𝑖+1 𝑎′ 𝑤 − , otherwise

01-04-2026
Deep Q-Network: Algorithm
7. Compute loss function
𝑁
1 2
𝐿 𝑤 = ෍ 𝑦𝑖 − 𝑄 𝑠𝑖 𝑎𝑖 𝑤
𝑁
𝑖=1

8. Update network parameters (Gradient Descent)

𝑤 ← 𝑤 − 𝛼∇𝑤 𝐿 𝑤

9. Update target network (periodically)

𝑤− ← 𝑤

01-04-2026
Deep Q-Network: Improvement
Overestimation Bias
In standard DQN:
𝑚𝑎𝑥
′
𝑄(𝑠 ′ , 𝑎′ )
𝑎
• Noise in Q-values causes overestimation.
• This leads to optimistic value predictions.

Double DQN (Improvement)

To reduce bias:
𝑦 = 𝑟 + 𝛾𝑄 𝑠 ′ arg 𝑚𝑎𝑥 𝑄 𝑠 ′ 𝑎 𝑤 𝑤 −
𝑎
Key idea:
• Online network selects action
• Target network evaluates action
• This significantly improves stability.

01-04-2026
Deep Q-Network (DQN) is a value-based deep reinforcement learning algorithm that combines Q-
learning with deep neural networks. In classical Q-learning, we maintain a table of Q-values for each
state–action pair. However, this becomes impossible when the state space is large or continuous,
such as when the state is an image. DQN solves this problem by approximating the Q-function using
a neural network, denoted as 𝑄 𝑠 𝑎 𝑤 ,where 𝑤represents the network parameters.
The key idea of DQN is to train the neural network so that its Q-value predictions satisfy the Bellman
optimality equation. The network is trained to minimize the squared difference between the predicted
Q-value and a target value defined as:
′ ′ −
𝑦 = 𝑟 + 𝛾𝑚𝑎𝑥′
𝑄 𝑠 𝑎 𝑤
𝑎
−
where 𝑤 are the parameters of a separate target network. The loss function used is the Bellman
error:
𝐿 = ቀ𝑦 − 𝑄 𝑠 𝑎 𝑤 )2
Two major innovations make DQN stable. First, experience replay stores transitions in a buffer and
samples mini-batches randomly, breaking correlations in data. Second, a target network is used to
generate stable targets, preventing divergence caused by rapidly changing Q-values. DQN is off-
policy because it learns the optimal policy independent of the exploration policy. However, it works
only for discrete action spaces and does not have theoretical convergence guarantees under
nonlinear function approximation.

01-04-2026
 Policy-Based Deep RL: REINFORCE

01-04-2026
Policy-Based Deep RL
Recall value-based methods (e.g., DQN):
𝜋 𝑠 = arg 𝑚𝑎𝑥 𝑄(𝑠, 𝑎)
𝑎
This works well when:
• Action space is discrete, small number of actions
Problems:
• Cannot handle continuous actions easily
• Argmax not differentiable
• Overestimation bias issues
Policy-Based Idea
Directly parameterize policy:
𝜋(𝑎|𝑠; 𝜃)
where:𝜃 = neural network parameters, output is probability distribution over actions
Goal:
𝑚𝑎𝑥 𝐽 𝜃
𝜃
01-04-2026
Policy-Based Deep RL: REINFORCE
REINFORCE is a policy-based reinforcement learning algorithm that directly learns the policy without
estimating value functions like Q-learning. It was introduced by Ronald J. Williams.
• REINFORCE is a Monte Carlo policy gradient method.
Instead of learning:
𝑄(𝑠, 𝑎)
it learns a parameterized policy:
𝜋(𝑎|𝑠; 𝜃)
where 𝜃 = parameters (weights of neural network), Output = probability of taking action 𝑎 in
state 𝑠.

• The goal is to maximize expected return:

𝐽 𝜃 = 𝔼 𝐺0
where 𝐺𝑡 =total return from time step 𝑡

01-04-2026
REINFORCE: Concept
If an action gives high reward → increase its probability
If an action gives low reward → decrease its probability
• This is done using:
∇𝑤 log 𝜋 𝑎|𝑠; 𝑤

REINFORCE Update Rule

𝑤 ← 𝑤 + 𝛼𝐺𝑡 ∇𝑤 log 𝜋 (𝐴𝑡 |𝑆𝑡 ; 𝑤)
where: 𝛼= learning rate, 𝐺𝑡 =return, ∇𝑤 log 𝜋 =gradient of log probability

Why Use Log?

Using log simplifies gradient:
∇𝑤 𝜋
∇𝑤 log 𝜋 =
𝜋
This makes computation easier and stable.
01-04-2026
REINFORCE: Algorithm
Initialize:
𝑤
For each episode:
1. Generate episode:
𝑆0 , 𝐴0 , 𝑅1 , 𝑆1 , 𝐴1 , … , 𝑆𝑇

2. For each time step 𝑡:

Compute return:
𝐺𝑡 = 𝑅𝑡+1 + 𝛾𝑅𝑡+2 + ⋯ + 𝛾 𝑇−𝑡−1 𝑅𝑇

3. Update parameters:
θ ← θ + 𝛼𝐺𝑡 ∇θ log 𝜋 𝐴𝑡 |𝑆𝑡 ; θ

01-04-2026
REINFORCE: Policy Network Structure
In policy-based methods like REINFORCE, instead of learning Q-values, we directly learn a policy
function:
𝜋 𝑎|𝑠; 𝜃
This is implemented using a neural network, called the policy network.

Policy network takes the-

• Input:
𝑠 state
• Output:
• For discrete action spaces:
Probability distribution over actions
𝜋 𝑎∣𝑠
• For continuous action spaces:
Parameters of a probability distribution (e.g., mean and variance of Gaussian)

01-04-2026
REINFORCE: Discrete Action Case (Softmax Policy)
When actions are discrete, the network outputs a vector:
𝑧 = 𝑓𝜃 𝑠
where, 𝑓𝜃 𝑠 =neural network output (logits), 𝑧𝑖 =score for action 𝑎𝑖
We convert these scores into probabilities using Softmax:
𝑒 𝑧𝑖
𝜋(𝑎𝑖|𝑠; 𝜃) =
σ 𝑗 𝑒 𝑧𝑗
Intuition:
• Larger 𝑧𝑖 →higher probability Example
• Smaller 𝑧𝑖 →lower probability Suppose:
𝑧 = [2,1,0]
Then:
• Softmax ensures:
𝑒2
• Probabilities sum to 1: σ𝑖 𝜋 𝑎𝑖 ∣ 𝑠 = 1 𝜋 𝑎1 = 2 ≈ 0.67
• All probabilities are positive: 𝜋 𝑎𝑖 ∣ 𝑠 ≥ 0 𝑒 + 𝑒1 + 𝑒 0
• Differentiable which is required for gradient-based learning
𝜋 𝑎2 ≈ 0.24, 𝜋 𝑎3 ≈ 0.09

01-04-2026
REINFORCE: Continuous Action Case (Gaussian Policy)
When actions are continuous (e.g., torque, speed), the policy outputs:
• Mean: 𝜇(𝑠; 𝜃)
• Variance: 𝜎 2 (𝑠; 𝜃)
• The action is sampled from a Gaussian distribution:
𝑎 ∼ 𝒩(𝜇 𝑠 𝜃 , 𝜎 2 𝑠 𝜃 )

• Policy Representation
𝜋(𝑎|𝑠; 𝜃) = 𝒩(𝑎|𝜇 𝑠 𝜃 , 𝜎 2 𝑠 𝜃 )
Gaussian is:
• Suitable for continuous actions
• Allows exploration via variance
• Differentiable; supports gradient learning

01-04-2026
REINFORCE: Variance Issue
REINFORCE update:
𝜃 ← 𝜃 + 𝛼𝐺𝑡 ∇𝜃 log 𝜋 𝐴𝑡 ∣ 𝑆𝑡
𝐺𝑡 =total return from time 𝑡
Issues:
• High variance
• Returns fluctuate a lot
• Same state-action → very different rewards
• Slow convergence
• Learning becomes unstable
• Noisy updates
• Gradient direction unreliable
• REINFORCE has high variance Because:
• Entire episode reward affects update
• Delayed rewards
• Environment randomness
01-04-2026
REINFORCE: Variance Reduction (Baseline)
To reduce variance, we subtract a baseline:
𝜃 ← 𝜃 + 𝛼 𝐺𝑡 − 𝑏 𝑆𝑡 𝛻𝜃 𝑙𝑜𝑔 𝜋 𝐴𝑡 ∣ 𝑆𝑡
Key Idea
• Instead of using raw return 𝐺𝑡 ,use:
Adjusted return = 𝐺𝑡 − 𝑏 𝑆𝑡
Why this helps?
• If action is better than average → positive update
• If action is worse than average → negative update
• Common Baseline: Value Function is the most common choice; 𝑏 𝑆𝑡 = 𝑉 𝜋 𝑆𝑡
So,
𝐺𝑡 − 𝑉 𝑆𝑡
Interpretation
𝐴𝜋 𝑆𝑡 𝐴𝑡 = 𝐺𝑡 − 𝑉 𝑆𝑡
This is called the Advantage Function.
01-04-2026
REINFORCE: Advantage Function
Advantage Function:
𝐴 𝑠, 𝑎 = 𝑄(𝑠, 𝑎) − 𝑉 𝑠
Interpretation:
How much better is action 𝑎compared to average action at state 𝑠?

• Now we need:
• Policy → Actor
• Value function 𝑉 𝑠 →Critic
So:
REINFORCE + baseline ⇒ Actor−Critic
01-04-2026
REINFORCE: Example
Suppose:
• Policy outputs probabilities:
𝜋 𝑎1 ∣ 𝑠 = 0.6, 𝜋 𝑎2 ∣ 𝑠 = 0.4
• Action 𝑎1 taken
• Return 𝐺𝑡 = 5
• Learning rate 𝛼 = 0.1
• Gradient of log probability (assume):
∇𝜃 log 𝜋 𝑎1 ∣ 𝑠 = 0.8
Update:
𝜃 ← 𝜃 + 0.1 5 0.8
𝜃 ← 𝜃 + 0.4
Intuition:
𝜃 increases; This increases log 𝜋 𝑎1 ∣ 𝑠
Probability of 𝑎1 increases.
01-04-2026
REINFORCE is a fundamental policy-based reinforcement learning algorithm introduced by
Ronald Williams. Unlike value-based methods such as DQN, REINFORCE directly optimizes
the policy without learning a Q-function. The policy is parameterized as 𝜋 𝑎 ∣ 𝑠𝜃 ,where
𝜃represents neural network parameters. The objective is to maximize the expected return:
𝐽 𝜃 = 𝔼 𝐺0
Using the policy gradient theorem and the log-derivative trick, the gradient of the objective can
be expressed as:
∇𝜃 𝐽 𝜃 = 𝔼 𝐺𝑡 ∇𝜃 log 𝜋 𝐴𝑡 ∣ 𝑆𝑡
This leads to the update rule:
𝜃 ← 𝜃 + 𝛼𝐺𝑡 ∇𝜃 log 𝜋 𝐴𝑡 ∣ 𝑆𝑡
Intuitively, REINFORCE increases the probability of actions that lead to high returns and
decreases the probability of actions that lead to low returns. Since it uses full episode returns,
it is an unbiased estimator of the policy gradient but suffers from high variance. This high
variance makes learning unstable and slow. To address this issue, a baseline such as a value
function can be subtracted from the return, which leads to actor–critic methods.

01-04-2026
Asynchronous Methods for Deep RL: Advantage
Actor-Critic (A2C) and Asynchronous Advantage
Actor-Critic(A3C)

01-04-2026
Asynchronous Methods for Deep RL
• The algorithm was introduced in Asynchronous Methods for Deep Reinforcement Learning
(DOI: 10.48550/arXiv.1602.01783). It showed that parallel actor-learners can stabilize training
without experience replay.
Recall DQN:
• Needs replay buffer
• Needs target network
• Works only for discrete actions Idea:
• Large memory requirement Instead of replay memory,
Run multiple agents in parallel
Problems:
Each interacts with its own copy of environment
• Replay memory expensive Gradients are asynchronously applied to shared
• Off-policy learning unstable in some settings parameters
• Not ideal for continuous control
This stabilizes learning via decorrelated experience.

01-04-2026
Asynchronous Methods for Deep RL
Actor–Critic has two components:

01-04-2026
Advantage Actor-Critic (A2C)
“Advantage” Actor–Critic:
Instead of using full return 𝐺𝑡 ,we use advantage function:
𝐴(𝑠, 𝑎) = 𝑄(𝑠, 𝑎) − 𝑉 𝑠
In practice:
𝐴𝑡 ≈ 𝛿𝑡
This reduces variance significantly.

Advantage Actor–Critic (A2C) is the synchronous version of A3C. It combines:

• Policy Gradient (Actor)
• Value Function Approximation (Critic)
• Advantage estimation (variance reduction)

• A2C is the deterministic, synchronous variant commonly used in modern deep RL libraries.

01-04-2026
Advantage Actor-Critic (A2C): Mathematical
Formulation
• We maintain, Policy network 𝜋 𝑎 ∣ 𝑠𝜃 , and Value network 𝑉 𝑠 𝑤 We maintain:
Step 1: Collect Trajectory (n-step)
For t to t+n:
𝑅𝑡 = 𝑟𝑡 + 𝛾𝑟𝑡+1 + ⋯ + 𝛾 𝑛 𝑉 𝑠𝑡+𝑛
Step 2: Compute Advantage
𝐴𝑡 = 𝑅𝑡 − 𝑉 𝑠𝑡
Step 3: Actor Loss
𝐿𝑎𝑐𝑡𝑜𝑟 = − log 𝜋 𝑎𝑡 ∣ 𝑠𝑡 𝐴𝑡
Step 4: Critic Loss
𝐿𝑐𝑟𝑖𝑡𝑖𝑐 = ቀ𝑅𝑡 − 𝑉 𝑠𝑡 )2
Step 5: Entropy Bonus (Exploration)
𝐿𝑒𝑛𝑡𝑟𝑜𝑝𝑦 = −𝛽𝐻 𝜋 𝑠𝑡
Encourages exploration.
Final Combined Loss
01-04-2026
𝐿 = 𝐿𝑎𝑐𝑡𝑜𝑟 + 𝑐1 𝐿𝑐𝑟𝑖𝑡𝑖𝑐 + 𝑐2 𝐿𝑒𝑛𝑡𝑟𝑜𝑝𝑦
Advantage Actor-Critic (A2C): Algorithm

Step 1: Initialize parameters 𝜃, 𝑤

Step 2: Loop:

Step 3: Run N environments in parallel

Step 4: Collect n-step rollouts

Step 5: Compute returns 𝑅𝑡

Step 6: Compute advantages 𝐴𝑡

Step 7: Compute loss

Step 8: Perform gradient update

Repeat
01-04-2026
Advantage Actor-Critic (A2C): Example
Given:
𝑟𝑡 = 2
𝛾 = 0.9
𝑉 𝑠𝑡 = 1.5
𝑉 𝑠𝑡+1 = 2.0
Compute TD target:
𝑅𝑡 = 𝑟𝑡 + 𝛾𝑉 𝑠𝑡+1 = 2 + 0.9 2.0 = 2 + 1.8 = 3.8
Advantage:
𝐴𝑡 = 3.8 − 1.5 = 2.3
If:
∇𝜃 log 𝜋 𝑎𝑡 ∣ 𝑠𝑡 = 0.5
Actor update:
𝜃 ← 𝜃 + 𝛼 2.3 0.5
If 𝛼 = 0.1:
𝜃 ← 𝜃 + 0.115
01-04-2026
Advantage Actor-Critic (A2C): Strengths and
Limitations
Strengths of A2C
• Works for continuous control
• More stable than pure policy gradient
• No replay buffer needed
• Simple implementation
• Parallelizable

Limitations
• Still sample inefficient
• Sensitive to hyperparameters
• Slower than PPO in many tasks

01-04-2026
 Advantage Actor–Critic (A2C) is a synchronous actor–critic algorithm that reduces the
variance of policy gradient methods by introducing a critic. In A2C, the actor represents the
policy 𝜋 𝑎 ∣ 𝑠𝜃 ,while the critic estimates the state value function 𝑉 𝑠 𝑤 .Instead of using
the full return 𝐺𝑡 ,A2C uses an advantage estimate:
𝐴𝑡 = 𝑅𝑡 − 𝑉 𝑠𝑡
where 𝑅𝑡 is typically an n-step bootstrapped return:
𝑅𝑡 = 𝑟𝑡 + 𝛾𝑟𝑡+1 + ⋯ + 𝛾 𝑛 𝑉 𝑠𝑡+𝑛
The actor is updated using:
𝜃 ← 𝜃 + 𝛼𝐴𝑡 ∇𝜃 log 𝜋 𝑎𝑡 ∣ 𝑠𝑡
and the critic is updated by minimizing the squared error:
𝐿𝑐𝑟𝑖𝑡𝑖𝑐 = ቀ𝑅𝑡 − 𝑉 𝑠𝑡 )2
By subtracting the value function, the algorithm measures how much better the chosen
action is compared to the average action at that state. This significantly reduces variance
while introducing small bias due to bootstrapping. A2C runs multiple environments in
parallel and performs synchronized gradient updates, making it more stable and GPU-
friendly compared to its asynchronous counterpart.

01-04-2026
Asynchronous Advantage Actor-Critic (A3C)
DQN required, Replay buffer and Target network. A3C replaces replay buffer with Multiple
parallel workers, Independent environments, Asynchronous gradient updates

A3C Architecture:

Each worker: Why Asynchronous helps?

• Copies global network • Reduces correlation

• Interacts with environment • Implicit decorrelation of data

• Computes gradients • Stabilizes updates

• Updates global parameters asynchronously • Eliminates replay buffer

01-04-2026
Asynchronous Advantage Actor-Critic (A3C):
Mathematical Formulation
n-step return
𝑅𝑡 = 𝑟𝑡 + 𝛾𝑟𝑡+1 + ⋯ + 𝛾 𝑛 𝑉 𝑠𝑡+𝑛
Advantage:
𝐴𝑡 = 𝑅𝑡 − 𝑉 𝑠𝑡
Actor Loss
𝐿𝑎𝑐𝑡𝑜𝑟 = − log 𝜋 𝑎𝑡 ∣ 𝑠𝑡 𝐴𝑡
Critic Loss
𝐿𝑐𝑟𝑖𝑡𝑖𝑐 = ቀ𝑅𝑡 − 𝑉 𝑠𝑡 )2
Entropy Regularization
𝐿𝑒𝑛𝑡𝑟𝑜𝑝𝑦 = −𝛽𝐻 𝜋 𝑠𝑡
Encourages exploration.
Total Loss
𝐿 = 𝐿𝑎𝑐𝑡𝑜𝑟 + 𝑐1 𝐿𝑐𝑟𝑖𝑡𝑖𝑐 + 𝑐2 𝐿𝑒𝑛𝑡𝑟𝑜𝑝𝑦

01-04-2026
A2C vs A3C Comparison

01-04-2026 Source: [Link]

A2C vs A3C Comparison

01-04-2026
 Asynchronous Advantage Actor–Critic (A3C) extends the actor–critic framework by
introducing parallel training through multiple worker agents. Each worker interacts with its
own environment instance and computes gradients locally. These gradients are then applied
asynchronously to a shared global network. This approach eliminates the need for
experience replay, as parallel environments naturally decorrelate data.

Like A2C, A3C uses advantage estimates based on n-step returns. The total loss includes
three components: actor loss, critic loss, and entropy regularization. Entropy regularization
encourages exploration by preventing the policy from becoming deterministic too quickly.

The asynchronous updates introduce stochasticity that helps exploration and stabilizes
learning without requiring a replay buffer or target network. However, asynchronous updates
may introduce gradient noise and require careful tuning. A3C demonstrated that deep
reinforcement learning could be trained efficiently on CPUs without specialized replay
memory mechanisms.

01-04-2026
 Model-Based Deep RL

01-04-2026
Model-Based Deep RL
In reinforcement learning, a “model” means:
𝑃(𝑠 ′ |𝑠, 𝑎) and 𝑅(𝑠, 𝑎)
That is:
• Transition dynamics
• Reward function
In classical RL, model-based methods include:
• Dynamic Programming
• Dyna
• Monte Carlo Tree Search
In Deep RL, we learn a neural network model of the environment.

01-04-2026
Model-Based Deep RL: Core Idea
Model-free methods require huge data, sample inefficient, and expensive in real-world
(robotics, healthcare). Whereas model-based methods learn environment model, plan
inside the model, and Improve sample efficiency.

Instead of directly learning policy:

• Learn model:
𝑠Ƹ𝑡+1 = 𝑓𝜙 𝑠𝑡 𝑎𝑡
𝑟𝑡Ƹ = 𝑔𝜙 𝑠𝑡 𝑎𝑡
Use model for:
• Planning
• Generating synthetic rollouts
• Policy optimization

01-04-2026
Model-Based Deep RL: Approaches
There are two main approaches:
1. Learn Model + Plan
• Learn neural network model:
෠ ′ , 𝑠|𝑎)
𝑃(𝑠
Then use:
• Model Predictive Control (MPC)
• Tree search
• Dynamic programming

2. Learn Model + Improve Policy

• Use model to generate imaginary rollouts
• Update policy using simulated data
• Similar to Dyna architecture

01-04-2026
Model-Based Deep RL: Example
1. World Models
A major contribution in model-based deep RL is: World Models
Idea:
• Learn compressed latent representation
• Learn transition model in latent space
• Train controller inside learned world
2. Example: MuZero: A breakthrough algorithm-
Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
Key idea:
• Learn implicit model
• Use Monte Carlo Tree Search
• Do not explicitly learn environment dynamics
MuZero combines:
• Value learning
01-04-2026
• Policy learning
• Planning
Model-Free vs Model-Based

01-04-2026
Model-Based Deep RL: Advantages and
Limitations
Advantages
• Higher sample efficiency
• Enables planning
• Works well in robotics
• Can simulate risky scenarios

Limitations
• Model bias (errors accumulate)
• Hard to learn accurate model
• Computationally expensive
• Long-horizon prediction unstable

01-04-2026