Ethiopian Defence University, College of
Engineering
CT-6713: Machine Learning in
Cybersecurity
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 1
Reinforcement Learning Algorithms
• Q-Learning
• SARSA
• Monte Carlo
• Deep Q network
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 2 2
Reinforcement Learning Algorithms
• It is about learning the optimal behavior in an environment
to obtain maximum reward
• This optimal behavior is learned through interactions with
the environment
• Observations of how it responds, similar to children
exploring the world around them
• Learning the actions that help them achieve a goal
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 3 3
Reinforcement Learning Algorithms
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 4 4
Example : Q - Learning
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 5 5
Contd…
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 6 6
Contd…
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 7 7
Contd…
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 8 8
Contd
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 9 9
Contd
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 10 10
Contd
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 11 11
Contd
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 12 12
Contd
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 13 13
Contd
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 14 14
Contd…
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 15 15
Contd…
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 16 16
State Action Reward State Action(SARSA) algorithm
• State-action-reward-state-action (SARSA) is an on-policy
algorithm
• Designed to teach a machine learning model a new Markov
decision process policy in order to solve reinforcement
learning challenges
• It’s an algorithm where, in the current state (S), an action
(A) is taken and the agent gets a reward (R), and ends up in
the next state (S1), and takes action (A1) in S1
• Therefore, the tuple (S, A, R, S1, A1) stands for the
acronym SARSA.
• It’s called an on-policy algorithm because it updates the
policy based on actions taken
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 17 17
How Does the SARSA Algorithm Works?
• The SARSA algorithm works by carrying out actions based
on rewards received from previous actions
• To do this, SARSA stores a table of state (S)-action (A)
estimate pairs for each Q-value
• This table is known as a Q-table, while the state-action
pairs are denoted as Q(S, A).
• The SARSA process starts by initializing Q(S, A) to
arbitrary values
• In this step, the initial current state (S) is set, and the initial
action (A) is selected by using an epsilon-greedy algorithm
policy based on current Q-values
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 18 18
Contd…
• An epsilon-greedy policy balances the use of exploitation
and exploration methods in the learning process to select
the action with the highest estimated reward.
• Exploitation involves using already known, estimated
values to get more previously earned rewards in the
learning process
• Exploration involves attempting to find new knowledge on
actions, which may result in:
• Short-term,
• Sub-optimal actions during learning
• But may yield long-term benefits to find the best
possible action and reward
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 19 19
Contd…
• From here, the selected action is taken, and the reward (R)
and next state (S1) are observed
• Q(S, A) is then updated, and the next action (A1) is selected
based on the updated Q-values
• Action-value estimates of a state are also updated for each
current action-state pair present, which estimates the value
of receiving a reward for taking a given action
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 20 20
Q – Learning Vs SARSA Algorithms
Capt. Mehari K (Ph.D) Ethiopian University, Engineering College 21 21