Deep Learning Models as Function Approximators
• Traditional reinforcement learning (RL) methods such as Q-learning use a lookup table (Q-
table) to store Q-values for each possible state–action pair 𝑠 𝑎 .
In many real-world problems (e.g., video games, robotics, or autonomous vehicles), the
number of possible states is extremely large or continuous, making it impossible to store or
learn Q-values in a simple table.
• To overcome this limitation, deep neural networks (DNNs) are employed as function
approximators.
These models can learn to approximate the true Q-function 𝑄 𝑠 𝑎 through experience and
generalize across unseen states.
• Deep learning allows reinforcement learning agents to scale to complex environments.
• The neural network acts as a universal approximator for the Q-function.
• CNNs process visual states, while fully connected layers map them to action-value estimates.
• The network parameters WWW are optimized iteratively to minimize prediction error.
• The resulting Q-network forms the basis of modern deep reinforcement learning algorithms.
Key features of Using CNNs in RL
Efficient State Encoding: CNNs automatically extract visual features from raw pixels.
Dimensionality Reduction: Reduces large pixel inputs into compact feature representations.
Generalization: Learns common game patterns and strategies.
End-to-End Learning: No need for handcrafted feature engineering.
Stability and Performance: CNN-based Q-networks outperform tabular Q-learning in high-dimensional
tasks.
Advantages of Deep Function Approximators
• Scalability
• Generalization ability
• Adaptability to complex environments
• Integration with advanced RL algorithms (DQN, Double DQN, Dueling DQN, A3C)
• Representation learning
• End-to-end learning
Limitations and Challenges
• Instability during training
• Sample inefficiency
• Overestimation bias
• High computational cost
• Hyperparameter sensitivity
Concept of Function Approximation
A function approximator is any model that estimates an unknown function from data.
In RL, we approximate the Q-function 𝑄 𝑠 𝑎 ,which represents the expected future reward of performing
action 𝑎in state 𝑠, and then following the optimal policy.
𝑄 𝑠 𝑎 ≈ 𝑄 𝑠 𝑎 𝑊
Here:
𝑄 =estimated Q-value produced by the neural network,
𝑊= parameters (weights and biases) of the neural network,
The goal of learning = adjust 𝑊such that 𝑄 𝑠 𝑎 𝑊 ≈true 𝑄 𝑠 𝑎 .
Role of Function Approximators
Estimate Value Functions in RL: In RL, we often need the value function 𝑉 𝑠 or action-value function
𝑄 𝑠 𝑎 .Exact computation is often impossible for large state [Link] approximators (like neural
networks, linear regression, or decision trees) estimate 𝑉 𝑠 or 𝑄 𝑠 𝑎 from samples.
Generalization:They allow generalization to unseen states.
Reduce Memory Requirements:Function approximators compress this information into a set of parameters
𝜃.
Enable Continuous and High-Dimensional Control :Approximators like neural networks handle continuous
inputs and outputs.
Method1 :The Bootstrapping Trick
To overcome this, Q-learning uses an idea called bootstrapping, where an improved estimate of the Q-value is
created using partial knowledge of the future.
This is achieved using the Bellman Equation — a recursive relationship for Q-values.
𝑄 𝑠𝑡 𝑎𝑡 = 𝑟𝑡 + 𝛾 max 𝑄 𝑠𝑡+1 𝑎
𝑎
Here:
𝑟𝑡 :immediate reward after taking action 𝑎𝑡
𝛾 :discount factor (0 < γ < 1)
max 𝑎 𝑄 𝑠𝑡+1 𝑎 :predicted future return from the best next action
If the episode ends after action 𝑎𝑡 ,we set 𝑄 𝑠𝑡+1 𝑎 = 0.
Method 2:Neural Network Representation
In deep Q-learning, a neural network F(Xₜ, W, a) is used to approximate Q-values:
𝐹 𝑋𝑡 𝑊 𝑎 = 𝑄 𝑠𝑡 𝑎
𝐹 𝑋𝑡 𝑊 𝑎 →predicted Q-value for performing action 𝑎in state 𝑠𝑡
𝑄 →predicted (estimated) value, indicated by a circumflex (ˆ)
𝑊→ learnable weight parameters updated during training
Learning 𝑊is crucial because it directly influences the accuracy of Q-value estimation and, therefore, the
policy performance.
Structure of the Q-Network
The deep neural network used in this context is known as a Q-Network (or Deep Q-Network — DQN). The
goal of Q-learning is to learn the weights (W) of a neural network so that it can correctly estimate the Q-
function, i.e.,
𝑄 𝑠𝑡 𝑎𝑡 = expected future discounted reward from taking action 𝑎𝑡 in state 𝑠𝑡
However, the main challenge is that Q-values cannot be directly observed — they depend on future rewards
that have not yet occurred.
It is a Convolutional Neural Network (CNN) that:
• Converts high-dimensional pixel input into meaningful feature representations.
• Produces one Q-value for each possible action.
• For example, in a game where possible actions are UP, DOWN, LEFT, and RIGHT,
the Q-network will have four output neurons corresponding to:
𝑄 𝑠𝑡 “𝑈𝑃”
𝑄 𝑠𝑡 “𝐷𝑂𝑊𝑁”
𝑄 𝑠𝑡 “𝐿𝐸𝐹𝑇”
𝑄 𝑠𝑡 “𝑅𝐼𝐺𝐻𝑇”
Using the Bellman equation, we can write:
𝐹 𝑋𝑡 𝑊 𝑎𝑡 = 𝑟𝑡 + 𝛾 𝑚𝑎𝑥 𝐹 𝑋𝑡+1 𝑊 𝑎
𝑎
This gives a target for training, even without observing real future rewards.
Loss Function for Q-Learning
To train the network, we define a loss function based on the difference between:
The predicted Q-value for the current state-action pair, and the target Q-value obtained from the Bellman
equation.
2
𝐿𝑡 = 𝑟𝑡 + 𝛾 𝑚𝑎𝑥 𝐹 𝑋𝑡+1 𝑊 𝑎 − 𝐹 𝑋𝑡 𝑊 𝑎𝑡
𝑎
Weight Update Rule
Using backpropagation, the weights are updated as:
𝜕𝐹 𝑋𝑡 𝑊 𝑎𝑡
𝑊 ←𝑊+𝛼 𝑟𝑡 + 𝛾 𝑚𝑎𝑥 𝐹 𝑋𝑡+1 𝑊 𝑎 − 𝐹 𝑋𝑡 𝑊 𝑎𝑡
𝑎 𝜕𝑊
Here:
𝛼: learning rate
𝛻𝑊 𝐹 𝑋𝑡 𝑊 𝑎𝑡 :gradient of network output with respect to weights
Initially, Q-values are random, but they gradually converge as training continues.
Working of Q-Network
Step 1. Input State 𝑋𝑡 (Prepare the Observation) Perform action 𝑎𝑡 in state 𝑠𝑡 ,observe reward 𝑟𝑡 and
next state 𝑋𝑡+1.
• The network receives a compact representation of the current environment, typically a stack of the last
𝑚frames (e.g., 𝑚 = 4in Atari).
• Each frame is preprocessed:
Converted to grayscale
Cropped
Downsampled
Normalized
• All processed frames are stacked together to form 𝑋𝑡 .
• Stacking frames provides short-term temporal information (such as velocity and direction)
that a single frame cannot capture.
• The input tensor shape is typically 𝑚 𝐻 𝑊 or 𝐻 𝑊 𝑚 depending on the framework.
Step 2. Convolutional Layers (Extract Spatial/Temporal Features)
• 𝑋𝑡 is passed through several convolutional layers.
• Convolutional operations use small kernels and local receptive fields to detect low-level
patterns (edges, corners).
• Deeper layers combine low-level patterns into higher-level features (objects, moving sprites,
game entities).
Convolutional layers may include:
1. Nonlinearities (e.g., ReLU activation)
2. Pooling/striding to reduce spatial resolution
3. Batch normalization for stability
The final result is a set of feature maps that represent the important visual structure of the
current state
Compute the target Q-value:
𝑄𝑡+1 = max 𝐹 𝑋𝑡+1 𝑊 𝑎
𝑎
(Set to 0 if episode ends.)
3)Fully Connected (Dense) Layers (Compose Features into Decision Features)
• The final convolutional feature maps are flattened (or global-pooled) before entering fully connected
layers.
• These dense layers integrate the visual features across the whole screen.
• The resulting high-level representations are useful for decision-making.
Architectures may incorporate advanced techniques such as:
• Dropout (for regularization)
• Dueling streams (value and advantage)
• Residual connections (for deeper networks)
Dense layers perform the mapping from vision features → decision features, which are later used to estimate
Q-values. Compute the predicted Q-value:
𝐹 𝑋𝑡 𝑊 𝑎𝑡
4)Output Layer (Produce Q-values for Every Action)
• The last layer is a linear layer with ∣ 𝐴 ∣outputs — one output per possible action.
• Each output 𝑄 𝑠𝑡 𝑎𝑖 represents the predicted Q-value for taking action 𝑎𝑖 in state 𝑠𝑡 .
• The output layer uses no activation function, since Q-values can take any real value (unbounded).
• In environments with illegal or invalid actions, outputs for those actions are masked before selecting an
action.
• Compute the loss:𝐿𝑡 = ቀ𝑟𝑡 + 𝛾𝑄 𝑡+1 − 𝐹 𝑋𝑡 𝑊 𝑎𝑡 )2
• Backpropagate the loss and update weights W.
5)Action Selection (Choose an Action Using Policy — Usually ϵ-Greedy)
• The agent converts the Q-value vector into an action.
• The ϵ-greedy policy is commonly used for balancing exploration and exploitation:
• With probability (1 − ϵ) → choose the greedy action arg max 𝑎 𝑄 𝑠𝑡 𝑎 .
• With probability (ϵ) → choose a random valid action (exploration).
• During training, ϵ is gradually reduced (annealed) — large at the start and smaller later — to encourage
more exploitation over time.
• During inference or testing, the agent typically uses a pure greedy policy (ϵ ≈ 0).
Key Improvements for Stability
(a) Experience Replay
Stores a replay buffer (past experiences as ⟨s, a, r, s′⟩ tuples).
Randomly samples mini-batches for training to break correlation between consecutive samples.
Old experiences are dropped as new ones are added.
(b) Target Network
Uses two networks:
Online network: updated every step
Target network: updated slowly or periodically
Prevents oscillation and improves stability.
(c) Prioritized Experience Replay
Assigns higher priority to experiences with larger learning potential (TD-error).
Helps the agent focus on more informative experiences.