REINFORCEMENT

LEARNING
Complete Study Guide
Theory | Formulas | Numericals | Exam Tips
1. What is Reinforcement Learning?
Reinforcement Learning (RL) is a type of machine learning where an agent learns by interacting
with an environment. It learns through trial and error, receiving rewards or penalties based on its
actions.
Key Comparison:
Aspect Supervised Learning Reinforcement Learning
Method Teach by example Teach by experience
Input Labeled dataset Environment interaction
Feedback Correct label given Reward/Penalty given
Goal Learn from examples Maximize cumulative reward

2. Core Terminologies (Must Know!)

2.1 Agent
The learner or decision-maker. It interacts with the environment and takes actions.
💡 Exam Tip: In Mario: Mario is the agent.

2.2 State (s)

The current situation or observation of the environment at a given time. It represents what the
agent sees.
💡 Exam Tip: In Mario: Mario's current position, enemy locations, coin positions = state.

2.3 Action (a)

The move the agent makes in a given state. The set of all possible actions is called the action
space.
💡 Exam Tip: In Mario: Move left, move right, jump = actions.

2.4 Policy (π - Pi)

The strategy or rule the agent uses to decide which action to take in a given state. A policy
maps states to actions.
Formally:
π(a | s) = P(A = a | S = s)
This is the probability of taking action 'a' when in state 's'.
• Deterministic Policy: One specific action for each state
• Stochastic Policy: A probability distribution over actions
 💡 Exam Tip: Policy is the brain of the agent — it is what we are trying to learn/optimize in RL!

2.5 Reward (R)

A scalar feedback signal the agent receives after taking an action. It tells the agent how good or
bad the action was.
• Positive reward = good action
• Negative reward = bad action (penalty)
The GOAL of RL: Maximize the total (cumulative) reward over time.
💡 Exam Tip: Reward design matters! If winning gives +10 and collecting a coin gives +100, the agent
will prefer coins over winning.

2.6 State Transition

How the environment moves from one state to the next based on the agent's action.
s' = f(s, a) [deterministic]
s' ~ P(s' | s, a) [stochastic/random]
In practice, we often do NOT know the state transition function — only the environment knows
it.
• Deterministic: Same action in same state always leads to same next state (e.g., Chess)
• Stochastic: Next state has randomness (e.g., enemy movement in Mario)

3. Rewards, Returns & Discounting

3.1 Return (Ut) — Cumulative Future Reward
The return at time t is the sum of all future rewards from time t to the end of the episode.
Ut = Rt + Rt+1 + Rt+2 + ... + Rn
Where n is the final time step (end of game/episode).

3.2 Discounted Return — The Important One!

Future rewards are less certain and less valuable than immediate rewards. We multiply future
rewards by a discount factor gamma (γ).
Ut = Rt + γ·Rt+1 + γ²·Rt+2 + ... + γ^(n-t)·Rn
Compact form:
Ut = Σ γ^(k) · R(t+k) [sum from k=0 to n-t]

Gamma (γ) Value Meaning Agent Behavior

γ=0 Only immediate reward matters Greedy, short-sighted
γ=1 All future rewards equally valued Far-sighted, risky
 0 < γ < 1 (e.g., 0.9) Future discounted gradually Balanced — most common!

💡 Exam Tip: γ = 0.9 is a common default. Current reward is NOT discounted (k=0 gives γ^0 = 1).

3.3 Solving a Discounted Return Numerical

EXAMPLE: γ = 0.9, rewards are R1=+1, R2=+1, R3=+1, R4=-1. Find U1.
Step-by-step:
1. Write the formula: U1 = R1 + γ·R2 + γ²·R3 + γ³·R4
2. Substitute values: U1 = 1 + (0.9)(1) + (0.9²)(1) + (0.9³)(-1)
3. Calculate powers: γ² = 0.81, γ³ = 0.729
4. U1 = 1 + 0.9 + 0.81 + (-0.729) = 1.981
ANSWER: U1 = 1.981
💡 Exam Tip: Always remember: the first reward (Rt) is multiplied by γ^0 = 1, second by γ^1, third by
γ^2, etc.

3.4 Randomness in Returns

Before the game is played (at time t), future rewards Rt, Rt+1, ... Rn are UNKNOWN random
variables → written in UPPERCASE (Ut).
After the game ends, all rewards are observed → written in lowercase (ut) — just a plain
number.
📝 Note: Uppercase = Random Variable (unknown). Lowercase = Observed value (known number).

4. Value Functions
4.1 Why Value Functions?
At time t, Ut is a random variable — we don't know its exact value. We need a way to evaluate
'how good is this situation?' reliably.
Solution: Take the EXPECTATION (average) of Ut. This removes randomness and gives a real
number.

4.2 Action-Value Function Q(s,a) — Q-Function

The Q-function tells us: 'How good is it to take action a in state s, following policy π?'
Qπ(s, a) = E[Ut | St = s, At = a]
Breaking it down:
• E[ ] = Expectation (average over all possible futures)
• Ut = discounted return from time t
• Given St = s (we are in state s) and At = a (we take action a)
 • π = the policy being followed after time t
Key properties:
• A BETTER policy → HIGHER Q values
• Q depends on both current state AND current action
• Q acts as a CRITIC — it evaluates how good the agent's performance is

4.3 State-Value Function V(s)

The V-function tells us: 'How good is it to be in state s, following policy π?'
Vπ(s) = E[Ut | St = s]
Difference from Q-function: V only depends on state, not on a specific action.
Function Depends On Question Answered
Q(s, a) State + Action How good is this (state, action) pair?
V(s) State only How good is this state?

4.4 Relationship: Q and V

Vπ(s) = Σ π(a|s) · Qπ(s, a) [sum over all actions a]
State value = weighted average of Q-values, where weights are the policy probabilities.
💡 Exam Tip: If you know Q, you can always compute V. But not vice versa!

5. Q-Learning
5.1 Core Idea
Q-Learning is an algorithm to find the OPTIMAL Q-function (Q*) without knowing the
environment model.
• It is model-free and off-policy
• It directly approximates the optimal Bellman equation
• Works by updating Q-values using observed (s, a, r, s') tuples

5.2 Q-Learning Update Rule (THE FORMULA)

Q(s, a) ← Q(s, a) + α · [r + γ · max Q(s', a') - Q(s, a)]
Understanding each part:
Symbol Meaning
Q(s, a) Current Q-value for (state, action) pair
α (alpha) Learning rate — how fast we update (0 < α ≤ 1)
 r Immediate reward received after taking action a
γ (gamma) Discount factor for future rewards
max Q(s', a') Best Q-value for the NEXT state s' (over all actions)
[r + γ·max Q - Q(s,a)] TD Error — the difference between target and current

5.3 Solving a Q-Learning Numerical

EXAMPLE: α = 0.5, γ = 0.9
Current state: s, Action taken: a, Reward: r = +1
Next state: s', max Q(s', a') = 2.0 (best Q in next state)
Current Q(s, a) = 1.0
Find updated Q(s, a):
Step 1: Find TD Target = r + γ · max Q(s', a')
TD Target = 1 + 0.9 × 2.0 = 1 + 1.8 = 2.8
Step 2: Find TD Error = TD Target - Q(s, a)
TD Error = 2.8 - 1.0 = 1.8
Step 3: Update Q-value
Q(s, a) = 1.0 + 0.5 × 1.8 = 1.0 + 0.9 = 1.9
ANSWER: Updated Q(s, a) = 1.9
💡 Exam Tip: The three steps are always: (1) Compute TD Target, (2) Compute TD Error, (3) Apply
Update.

5.4 Q-Table Example (Value Iteration)

In simple problems, we store Q-values in a table where rows = states, columns = actions.
Example Q-Table:
State / Action A1 A2 A3 A4
S1 +1 +2 -1 0
S2 +2 0 +1 -2
S3 -1 +1 0 -2
S4 -2 0 +1 +1

From this table: In state S1, the best action is A2 (Q=+2). In state S2, the best action is A1
(Q=+2).
📝 Note: Q-Tables only work for small state/action spaces. For large spaces, we use Neural Networks
→ Deep Q-Learning (DQN).
6. Policy in Depth
6.1 What is Policy?
Policy defines how the agent behaves — it maps states to actions (or probability distributions
over actions).
π : S → A (deterministic)
π(a|s) = P(A=a | S=s) (stochastic)

6.2 Stochastic Policy Example

In state s, the agent has 3 possible actions: Up, Left, Right.
The policy might output: π(Up|s) = 0.7, π(Left|s) = 0.2, π(Right|s) = 0.1
The agent samples from this distribution → mostly goes Up but sometimes explores other
directions.
💡 Exam Tip: The probabilities must sum to 1.0 for any given state s.

6.3 Optimal Policy (π*)

The policy that maximizes the expected cumulative reward. Finding it is the ultimate goal of RL.
π* = argmax_π Vπ(s) for all states s

7. Three Types of Reinforcement Learning

Aspect Model-Based Value-Based Policy-Based
What it learns Model of the world State/action values Policy function directly
Action selection Plan using model Choose highest value Sample from policy
action
Exploration Built into planning Must add separately Built-in (stochastic)
Example Dyna-Q, World Models Q-Learning, DQN REINFORCE, PPO, A3C
algorithms
Best for Known/learnable envs Discrete action spaces Continuous action
spaces

8. Environment Types
Dimension Type 1 Type 2
Observability Fully Observable (Chess — you see Partially Observable (Poker —
everything) hidden cards)
Agents Single Agent (Atari game) Multi-Agent (DeepTraffic — multiple
cars)
Dynamics Deterministic (Cart Pole) Stochastic (DeepTraffic)
Time Static (Chess — board waits) Dynamic (DeepTraffic — moves in
real-time)
Actions Discrete (Chess — finite moves) Continuous (Cart Pole — force
value)

9. Classic RL Examples
Cart-Pole Balancing
• Goal: Balance a pole on a moving cart
• State: Pole angle, angular speed, cart position, horizontal velocity
• Actions: Apply horizontal force left or right
• Reward: +1 for every time step the pole stays upright

Atari / Doom Game

• Goal: Maximize game score (or eliminate opponents)
• State: Raw pixel values from the screen
• Actions: Button combinations (Up, Down, Left, Right, Shoot, Jump...)
• Reward: +ve for scoring/killing, -ve for dying

Robotic Arm Grasping

• Goal: Pick up objects of different shapes
• State: Raw camera pixels
• Actions: Move arm in different directions, grasp/release
• Reward: +1 when successful pickup, 0 or -1 otherwise

Human Life (Philosophical)

• Goal: Survival? Happiness?
• State: All sensory input (sight, hearing, taste, smell, touch)
• Actions: Think, move
• Reward: Homeostasis (biological well-being)
10. Complete Formula Cheat Sheet
Formula Name Mathematical Form What It Means
Policy Function π(a|s) = P(A=a|S=s) Prob. of action a in state s
Simple Return Ut = Rt + Rt+1 + ... + Rn Sum of all future rewards
Discounted Return Ut = Σ γ^k · R(t+k) Weighted sum, future less valued
Action-Value (Q) Qπ(s,a) = E[Ut | St=s, At=a] Expected return from (s,a)
State-Value (V) Vπ(s) = E[Ut | St=s] Expected return from state s
Q-to-V Relation Vπ(s) = Σ π(a|s) · Qπ(s,a) V = weighted average of Q
Q-Learning Update Q(s,a) ← Q(s,a) + TD update rule
α[r+γ·maxQ(s',a')-Q(s,a)]
Optimal Policy π* = argmax Vπ(s) Policy that maximizes value

11. Step-by-Step Guide to Solving Numericals

Type 1: Discounted Return Calculation
Given: γ, sequence of rewards R1, R2, ... Rn. Find: Ut
Method:
5. Write formula: Ut = Rt + γ·R(t+1) + γ²·R(t+2) + ...
6. Compute each γ^k: γ¹, γ², γ³ ... (key step!)
7. Multiply each reward by its corresponding γ^k
8. Sum all terms

Type 2: Q-Learning Update

Given: α, γ, current Q(s,a), reward r, max Q(s',a'). Find: updated Q(s,a)
Method:
9. Compute TD Target = r + γ · max Q(s', a')
10. Compute TD Error = TD Target - Q(s, a)
11. New Q(s, a) = Old Q(s, a) + α × TD Error

Type 3: Best Action from Q-Table

Given: A Q-table with states and actions. Find: optimal action for a state
Method: For the given state, find the column with the MAXIMUM Q-value. That action is optimal
(greedy policy).
Type 4: V from Q (and vice versa)
Given: Q-values and policy probabilities. Find: V(s)
V(s) = π(a1|s)·Q(s,a1) + π(a2|s)·Q(s,a2) + ...
Simply multiply each Q-value by its policy probability and sum.

12. Practice Numericals with Solutions

Numerical 1: Discounted Return
Q: γ = 0.8, rewards: R1=+2, R2=+3, R3=-1, R4=+4. Find U1.
U1 = 2 + (0.8)(3) + (0.8²)(-1) + (0.8³)(4)
= 2 + 2.4 + (0.64)(-1) + (0.512)(4)
= 2 + 2.4 - 0.64 + 2.048
ANSWER: U1 = 5.808

Numerical 2: Q-Learning Update

Q: α = 0.3, γ = 0.95, Q(s,a) = 3.0, r = +2, max Q(s',a') = 5.0. Find new Q(s,a).
TD Target = 2 + 0.95 × 5.0 = 2 + 4.75 = 6.75
TD Error = 6.75 - 3.0 = 3.75
New Q(s,a) = 3.0 + 0.3 × 3.75 = 3.0 + 1.125
ANSWER: New Q(s,a) = 4.125

Numerical 3: V from Q
Q: In state s, π(left|s) = 0.4, π(right|s) = 0.6. Q(s,left) = 2.0, Q(s,right) = 5.0. Find V(s).
V(s) = 0.4 × 2.0 + 0.6 × 5.0
= 0.8 + 3.0
ANSWER: V(s) = 3.8

13. Important Concepts & Exam FAQs

Why Discount Future Rewards?
Three reasons:
• Uncertainty: Future is uncertain — a reward now is more reliable than a promise later
• Time value: Immediate rewards are practically more useful
 • Mathematical: Ensures infinite-horizon problems converge (sum doesn't blow up)

What is the Difference Between On-Policy and Off-Policy?

• On-Policy: Learn Q using the SAME policy being followed (e.g., SARSA)
• Off-Policy: Learn Q using a DIFFERENT policy than being followed (e.g., Q-Learning)
💡 Exam Tip: Q-Learning is off-policy because it uses max Q(s') regardless of what action is actually
taken next.

What is Exploration vs. Exploitation?

• Exploitation: Use current knowledge to maximize reward (greedy)
• Exploration: Try new actions to discover potentially better strategies
• Epsilon-Greedy: With probability ε explore randomly, else exploit
💡 Exam Tip: If you only exploit, you miss better options. If you only explore, you never use what you
learned.

What is a Model in RL?

The agent's internal representation of how the environment works:
• Transition model: P(s' | s, a) — predicts next state
• Reward model: R(s, a) — predicts reward
Model-free methods (Q-Learning) don't need a model. Model-based methods learn or use a
model.

Deep Reinforcement Learning

When the state/action space is too large for a Q-table, we use a Neural Network to approximate
Q(s,a). This is called Deep Q-Network (DQN).
• Input: State (e.g., raw pixels)
• Output: Q-value for each possible action
• Training: Minimize difference between predicted Q and TD target

14. Quick Summary — The Big Picture

Reinforcement Learning = Agent + Environment + Reward Signal
The agent observes a state, takes an action using its policy, receives a reward, moves to a new
state — and repeats. The goal is to learn a policy that maximizes cumulative discounted reward
over time.

The Learning Loop:

State → Policy → Action → Environment → Reward + New State →
(repeat)
The agent improves by updating its value function (Q/V) or policy directly based on received
rewards. Over time, it learns what actions lead to the best outcomes.

Good luck! You've got this. 🎯