Experiment 1
Title: A case study to understand Markov decision process (MDP) terminologies like states,
state space, actions, action space, trajectory, episode, reward, return, and policy. Implement a
random policy in a 5×5 Maize environment
Objective: Students will be able
• To understand MDP terminologies
• To analyze control tasks in terms of state space, action space, rewards and returns
• To implement random policy in a 5×5 Maize environment
Explanation/Stepwise Procedure/ Algorithm:
Explain Markov Decision Process
➢ A Markov Decision Process (MDP) is a mathematical framework used in
Reinforcement Learning (RL) to model decision-making where outcomes are partly
random and partly controlled by an agent.
An MDP is defined by the tuple: (S, A, P, R, γ)
Where, S is Set of states, A is Set of actions, P is Transition probability function, R is
Reward function, and γ (gamma) is Discount factor.
Working of an MDP
Step 1: Initialize Environment
• Define all possible states (S).
• Define available actions (A) for each state.
• Set reward structure R(s, a).
• Set discount factor γ (0 ≤ γ ≤ 1).
Step 2: Agent Observes Current State (Agent starts in an initial state 푆0).
Step 3: Select Action (Policy-Based)
• Agent selects an action using a policy (π).
• Policy: 휋(푎 ∣ 푠)→ Probability of taking action a in state s.
Step 4: State Transition - Environment transitions to next state 푆푡+1based on transition
probability: 푃(푆푡+1 ∣ 푆푡 , 퐴 푡 )
Step 5: Agent receives reward: 푅푡 = 푅( 푆푡 , 퐴 푡 )
Step 6: Update Value Using Bellman Equation:
푉(푠) = ∑ 휋(푎 ∣ 푠) ∑ 푃( 푠′ ∣ 푠, 푎)[푅(푠, 푎) + 훾푉(푠′)]
푎 푠′
Step 7: Repeat Until Terminal State that is, continue until goal or terminal condition is
reached.
The Algorithm works on the Markov Property that states, the next state depends only
on the current state and action, not on past history. 푃(푆푡+1 ∣ 푆푡 , 퐴 푡 )
Learning Outcomes (3 to 5): Write learnings from this experiment (what you understood, new
functions or libraries learned from this experiment)
➢ From this RL experiment on Markov Decision Process (MDP), I was able to learn the
following:-
i. The framework of Markov Decision Process (MDP) and how its decision-
making process is modeled using states, actions, rewards, transition
probabilities, and discount factor.
ii. The Bellman Expectation Equation and how its able to recursively define value
functions.
iii. The Policy and Value Functions and how they help us evaluate long-term
rewards.
iv. Markov Property being able to simplify sequential decision problems.
v. Iterative updates of policy and value functions help in computing their optimal
values.
Assignment Questions/Practice Problems:
1. Write the difference between
a. Trajectory and episode
Trajectory Episode
i) A sequence of states, actions, and rewards A complete interaction sequence from start
generated by an agent state to terminal state
ii) Can be partial (may not reach terminal
Always ends in a terminal state
state)
iii) Same as trajectory but terminates at
( S0, A0, R1, S1, A1, R2, ... )
goal/failure
iv) Used in step-wise learning Used in episodic tasks
b. Reward and return
Reward (Rt) Return (Gt)
i) Immediate feedback received after taking
Total accumulated future reward
an action
ii) Instantaneous Long-term cumulative
iii) Given by environment Gt = R{t+1} + 훾R{t+2} + 훾2 R{t+3} + ...
2. What is policy in RL?
➢ A policy in Reinforcement Learning is a strategy that defines how an agent selects
actions in different states. It is represented as: 휋(푎 ∣ 푠), which is the probability of
taking action a when in state s.
References: (Mention references/website links)
i) Richard S. Sutton and Andrew G. Barto - Reinforcement Learning: An Introduction
(2nd Edition) (Chapters 3 & 4: Markov Decision Processes and Dynamic
Programming)
ii) OpenAI Gym Documentation
iii) Geeks-for-Geeks – Markov Decision Process (MDP) in Reinforcement Learning
Code:
3/2/26, 8:13 PM RL_Lab1.ipynb - Colab
Installing Gymnasium
!pip install gymnasium
Requirement already satisfied: gymnasium in /usr/local/lib/python3.12/dist-packages (1.2.3)
Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium) (2.0.2)
Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium) (3.1.2)
Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium) (4.15.0)
Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.12/dist-packages (from gymnasium) (0.0.
Import Gymnasium
import gymnasium as gym
Create an Environment
env = [Link]("CartPole-v1")
Reset environment to start a new episode
observation, info = [Link]()
Choose an action: 0 = push cart leꢀ, 1 = push cart right
action = env.action_space.sample() #Random action for now
The agent performs an action in an environment
observation, reward, terminated, truncated, info = [Link](action)
Action and Observation Spaces
#Discrete action space (button presses)
env = [Link]('CartPole-v1')
print(f"Action space: {env.action_space}") #Discrete{2} - left or right
print(f"Sample Action: {env.action_space.sample()}") #0 or 1
#Box observation space (continuous values)
print(f"Observation space: {env.observation_space}") #Box with 4 values
#Box([-4.8, -inf, -0.418, -inf], [4.8, inf, 0.418, inf])
print(f"Sample Observation: {env.observation_space.sample()}") #Random valid observation
Action space: Discrete(2)
Sample Action: 1
Observation space: Box([-4.8 -inf -0.41887903 -inf], [4.8 inf 0.41887903 inf], (4,
Sample Observation: [-1.2479628 0.8965794 0.04168286 -0.52253014]
First Intro: Program to understand RL Loop
import [Link] as plt
from [Link] import clear_output
import time
#Create the CartPole environment
env = [Link]("CartPole-v1", render_mode="rgb_array")
#Reset the environment to start a new episode
[Link] 1/3
3/2/26, 8:13 PM RL_Lab1.ipynb - Colab
observation, info = [Link]()
#Flag to track whether episode has ended
episode_over = False
total_reward = 0
while not episode_over:
action = env.action_space.sample()
observation, reward, terminated, truncated, info = [Link](action)
total_reward += reward
episode_over = terminated or truncated
#Rendering and visualization
frame = [Link]()
clear_output(wait
[Link](frame) = True)
[Link]("off")
[Link]()
[Link](0.09)
#Cleanup after episode completion
[Link]()
print(f"Episode finished! Total Reward: {total_reward}")
Episode finished! Total Reward: 33.0
Frozen Lake Example
import numpy as np
env = [Link]("FrozenLake-v1", is_slippery=True, success_rate=1.0/3.0, reward_schedule=(1, 0, 0),
render_mode="rgb_array")
while total_reward <= 0:
obs, info = [Link]()
terminated = truncated = False
total_reward = 0
action_probabilities = (0.25, 0.25, 0.25, 0.25)
while not terminated or truncated:
action = [Link](range(4), p = action_probabilities)
observation, reward, terminated, truncated, info = [Link](action)
total_reward += reward
episode_over = terminated or truncated
#Rendering and visualization
frame = [Link]()
clear_output(wait = True)
[Link](frame)
[Link]("off")
[Link]()
[Link](0.5)
[Link]()
print("Episode finished! Total reward: ", total_reward)
[Link] 2/3
3/2/26, 8:13 PM RL_Lab1.ipynb - Colab
Episode finished! Total reward: 1
Start coding or generate with AI.
[Link] 3/3