REINFORCEMENT LEARNING
RL PROFILE
• Reinforcement Learning (RL) is a branch of machine learning that
focuses on learning from trial and error, rewards and penalties, and
interaction with dynamic environments. It has many applications in
robotics, gaming, finance, and more.
Choose your projects wisely
• The first step to creating a RL portfolio is to select the projects that
demonstrate your knowledge, creativity, and problem-solving
abilities. You can choose from existing RL benchmarks, such as
OpenAI Gym, Atari games, or MuJoCo, or you can create your own
custom environments and tasks. The key is to choose projects that are
relevant to your goals, interests, and target audience. For example, if
you want to work in robotics, you can show how you trained a
simulated robot to perform tasks like grasping, walking, or
manipulation. If you want to work in gaming, you can show how you
developed an agent that can play complex games like StarCraft or
Dota 2.
Explain your methods and results
• The second step to creating a RL portfolio is to explain your methods
and results clearly and concisely. You should leverage a combination
of text, code, images, videos, and interactive demos to showcase your
work. Your portfolio should include the motivation and objective of
your project, the RL algorithm and framework you used, the
environment and agent specifications, the hyperparameters and
evaluation metrics, the challenges and limitations you faced, the
results and insights you obtained, as well as future work and
improvements you suggest.
• Dive into computational specifics, such as the reward function, policy
updates, and exploration-exploitation trade-offs.
• Use advanced terminology and explain how the model is optimized
using approaches like Q-learning or Monte Carlo methods.
• Reinforcement Learning is an aspect of Machine learning where an
agent learns to behave in an environment, by performing certain
actions and observing the rewards/results which it gets from those
actions.
• With the advancements in Robotics Arm Manipulation, Google Deep
Mind beating a professional Alpha Go Player, and recently the
OpenAI team beating a professional DOTA player, the field of
reinforcement learning has exploded in recent years.
• ROBOT END EFFECTOR / ARM MANIPULATOR
RL 1:Agenda
• Reinforcement learning is and its nitty-gritty like rewards, tasks, etc
• 3 categorizations of reinforcement learning
Baby starts walking and falls due to some obstacle
Baby starts walking and successfully reaches the in between and gets bruised. Negative Reward
couch:Positive Reward +n
Reward Maximization :
The RL agent basically works on
a hypothesis of reward
maximization. That’s why
reinforcement learning should
have best possible action in
order to maximize the reward.
Tasks and their types in reinforcement learning
• Continuous tasks: RL agent that does automated Forex/Stock trading.
In this case, the agent has to learn how to choose the best actions and
simultaneously interact with the environment. There is no starting
point and end state.
• Episodic task: In this case, we have a starting point and an ending
point called the terminal state. This creates an episode: a list of
States (S), Actions (A), and Rewards (R).
• Exploration and exploitation trade-off: Reinforcement learning is an
important exploration and exploitation trade-off concept. Exploration is about
finding more information about an environment, whereas exploitation is exploiting
already-known information to maximize the rewards.
• To build an optimal policy, the RL agent is faced with the dilemma of whether to
explore new states at the same time as maximizing its reward. This is known as
Exploration versus Exploitation trade-off
• Atari games have been used as a test bed for reinforcement
learning algorithms.
• The policy is the strategy that dictates the actions the agent takes as a
function of the agent’s state as well as the environment.
• three types of RL implementations:
• Policy-based RL uses a policy or deterministic strategy that maximizes
cumulative reward
• Value-based RL tries to maximize an arbitrary value function
• Model-based RL creates a virtual model for a certain environment
and the agent learns to perform within those constraints
Approaches to Reinforcement Learning
• Policy-based approach: we have a policy that we need to optimize.
The policy defines how the agent behaves:
We learn a policy function which helps us
in mapping each state to the best action.
Getting deep into policies, we further
divide policies into two types:
•Deterministic: a policy at a given state(s) will
always return the same action(a). It means, it
is pre-mapped as S=(s) A=(a).
•Stochastic: It gives a distribution of
probability over different actions. i.e
Stochastic Policy p( A = a | S = s )
Value-Based Approach
• In value-based RL, the goal of the agent is to optimize the value
function V(s) which is defined as a function that tells us the
maximum expected future reward the agent shall get at each
state.
• The value of each state is the total amount of the reward an RL
agent can expect to collect over the future, from a particular
state.
Atari Games
• Atari, Inc. is an American video gaming company based in New York
City, and a subsidiary of the Atari SA holding company. It is the main
entity serving the commercial Atari brand globally since 2003.
• Check any three RL based Atari games .
• The aim is not to look for immediate reward but to optimize for
maximum cumulative reward over the length of training. Time is also
important – the reward agent doesn’t just rely on the current state,
but on the entire history of states.
• The environment in a reinforcement learning algorithm is commonly
expressed as a Markov decision process (MDP)
• Does RL need data?
• Applications of RL
• DeepMind
• self-driving cars
• [Link]
Reinforcement Learning
TOOL 1 :MATLAB
TOOL-2: Using Python
EXAMPLES
EXAMPLES
an AI agent is present within a maze environment, and its goal is to find a reward. The agent interacts with the
environment by performing some actions, and based on those actions, the state of the agent gets changed,
and it also receives a reward or penalty as feedback.
• Representation Learning is a process in machine learning where
algorithms extract meaningful patterns from raw data to create
representations that are easier to understand and process. These
representations can be designed for interpretability, reveal hidden
features, or be used for transfer learning.
• Good representations of data (e.g., text, images) are critical for solving
many tasks (e.g., search or recommendations)
• DNN models are the tool of choice in realizing such systems, because
they excel at learning semantically meaningful representations.
• What representation learning is, and why it’s valuable
• Methods for representation learning
• Learned Representations: how to evaluate good representations and
when to use which methods
Examples
• A master chess player makes a move. The choice is informed both by
planning—anticipating possible replies and counterreplies—and by
immediate, intuitive judgments of the desirability of particular
positions and moves.
• An adaptive controller adjusts parameters of a petroleum refinery’s
operation in real time. The controller optimizes the yield/cost/quality
trade-off on the basis of specified marginal costs without sticking
strictly to the set points originally suggested by engineers.
• A gazelle calf struggles to its feet minutes after being born. Half an
hour later it is running at 20 miles per hour.
• A mobile robot decides whether it should enter a new room in search
of more trash to collect or start trying to find its way back to its battery
recharging station. It makes its decision based on the current charge
level of its battery and how quickly and easily it has been able to find
the recharger in the past.
• Advanced controls: Controlling nonlinear systems is a challenging
problem that is often addressed by linearizing the system at different
operating points. Reinforcement learning can be applied directly to the
nonlinear system.
• in all these examples the effects of actions cannot be fully predicted;
thus the agent must monitor its environment frequently and react
appropriately.
• Automated driving: Making driving decisions based on camera input
is an area where reinforcement learning is suitable considering the
success of deep neural networks in image applications.
• Robotics: Reinforcement learning can help with applications like
robotic grasping, such as teaching a robotic arm how to manipulate a
variety of objects for pick-and-place applications (39:38). Other
robotics applications include human-robot and robot-robot
collaboration.
• Scheduling: Scheduling problems appear in many scenarios including
traffic light control and coordinating resources on the factory floor
towards some objective. Reinforcement learning is a good alternative
to evolutionary methods to solve these combinatorial optimization
problems.
• Calibration: Applications that involve manual calibration of
parameters, such as electronic control unit (ECU) calibration,
may be good candidates for reinforcement learning.
Reinforcement Learning Workflow
• Create the environment: The environment can be either a simulation
model, or a real physical system, but simulated environments are
usually a good first step since they are safer and allow
experimentation.
• Define the reward: Next, specify the reward signal that the agent uses
to measure its performance against the task goals and how this signal
is calculated from the environment. Reward shaping can be tricky and
may require a few iterations to get it right.
• Create the agent : [ Consists policy+ RL ALGO]
• Choose a way to represent the policy (such as using neural networks
or look-up tables).
• Select the appropriate training algorithm. Different representations are
often tied to specific categories of training algorithms. But in general,
most modern reinforcement learning algorithms rely on neural
networks as they are good candidates for large state/action spaces and
complex problems.
Train and validate the agent
• Set up training options (like stopping criteria) and train the agent to
tune the policy. Make sure to validate the trained policy after training
ends.
• If necessary, revisit design choices like the reward signal and policy
architecture and train again.
• Reinforcement learning is generally known to be sample inefficient;
training can take anywhere from minutes to days depending on the
application.
• For complex applications, parallelizing training on multiple CPUs,
GPUs, and computer clusters will speed things up
Deploy the policy
• Training an agent using reinforcement learning is an iterative process.
Decisions and results in later stages can require you to return to an
earlier stage in the learning workflow. For example, if the training
process does not converge to an optimal policy within a reasonable
amount of time, you may have to update any of the following before
retraining the agent:
• Training settings
• Reinforcement learning algorithm configuration
• Policy representation
• Reward signal definition
• Action and observation signals
• Environment dynamics
Elements of Reinforcement Learning
• Beyond the agent and the environment, one can identify four main
sub-elements of a reinforcement learning system:
• a policy,
• a reward signal,
• a value function, and,
• optionally, a model of the environment.
Create model-free and
model-based reinforcement
learning agents using
popular algorithms such as
DQN, PPO, and SAC.
Alternatively, develop your
own custom algorithms with
provided templates. Use RL
Agent block to bring your
agents into Simulink
Activity 1 : 7/2/2024 // week 2
1. Pathfinding in Grid World using Q-Learning
2. [Link]
minutes/creating-an-automated-meeting-minutes-generator-with-
whisper-and-gpt-4 // Minutes of meeting generator
3. Image generation using ChatGpt
• Reinforcement Learning Agents :
• The goal of reinforcement learning is to train an agent to complete a task within an
uncertain environment. At each time interval, the agent receives observations and
a reward from the environment and sends an action to the environment.
• The reward is an immediate measure of how successful the previous action (taken
from the previous state) was with respect to completing the task goal.
• The agent and the environment interact at each of a sequence of discrete time steps
Reinforcement Learning Environments
• the agent and the environment interact at each of a sequence of discrete time steps:
[Link] a given time step t, the environment is in a state S(t), which results in the
observation O(t).
[Link] on O(t) and its internal policy function, the agent calculates an action A(t).
[Link] on both the state S(t) and the action A(t), and according to its internal
dynamics, the environment updates its state to S(t+1), which results in the next
observation O(t+1).
[Link] on S(t), A(t), and S(t+1), the environment also calculates a scalar
reward R(t+1). The reward is an immediate measure of how good the action A(t)
is.
[Link] the next time step t+1 the agent receives the observation O(t+1) and the
reward R(t+1).
[Link] on the history of observations and rewards received, the learning algorithm
updates the agent policy parameters in an attempt to improve the policy function.
The parameter update may occur at each step or after a subsequence of steps.
[Link] on O(t+1) and on its policy function, the agent calculates the next
action A(t+1), and the process is repeated.
Different types of environment
• Markov Decision Process (MDP) Environments:
• MDP environments are environments in which state and observation
belong to finite spaces, and state transitions are in general governed by
stochastic rules.
• Grid world environments are a special case of MDP environments.
Here, the state represents a position in a two-dimensional grid, while
the action represents a move from the current position to the next,
which an agent might attempt.
• Grid world environments are often used in many introductory
reinforcement learning examples.
• three types of MDP environment [MATLAB]
• Predefined grid world environments :
• Custom Environments
• You can create different types of custom environments
• Neural Network Environments
• Neural network environments can be used within model-based
reinforcement learning agents, such as Model-Based Policy
Optimization (MBPO) Agents.
Exercise
• Create a Custom Environment Using Step and Reset Functions
• Discrète Action Space Cart-Pole MATLAB Environment
• Implement the above problem using a Python environment
WEEK 3
• Origin and history of Reinforcement Learning research. Its
connections with other related fields and with different branches of
machine learning.
• Reinforcement Learning Problem - Elements of Reinforcement
Learning- Limitations and Scope-Examples-
• Extended Example: Tic-Tac-Toe. Are ‘representation learning’ and
‘reinforcement learning’ the same terms? - Future Trends- Deep
reinforcement learning.
History of RL : 100 yrs ago : intersection of machine learning and control theory
[Link]
Deterministic systems : ROBOT , Autonomous
vehicle
Reinforcement Learning in Games and Entertainment
• For example, in the task of food delivery, the rider is the agent, and
he/she needs to navigate the streets (actions) to reach the goal of
reaching the customer’s house while also making sure he/she is on the
correct route (state). The biggest reward comes when the rider reaches
the customer and delivers the food, and penalties in between can come
in the form of taking a wrong turn, caught in a traffic jam, etc. that
prevents the rider from completing the task.
• game Mario Brothers
• Automated game testing:
• There are almost infinite patterns of Actions that the Agent can take.
Walking, running, shooting, changing weapons type, reloading,
interacting with objects and items, and the list goes on.
• Conventional game testing normally includes tens of hours of
gameplay by beta testers to make sure the game works fine, not just
free from coding bugs, but also specifications bugs such as a ledge that
is too high for the player to jump on, etc. By automating gameplay,
testers just need to set a certain time to let the AI complete the game,
and if the AI cannot complete the game, then only testers will be
required to check the logs for why it couldn’t complete.
Automated game tuning :For example, in a fighting game developers normally want to create various
characters with different skills and abilities but overall evenly matched