Course Introduction

21ARE402 Introduction to Deep Reinforcement Learning

Course Details and Objectives
Course Details
Name of the Course Introduction to Deep Reinforcement Learning
Course Code 21ARE402
Course Type Theory
Course Credit 3-0-0-3 (3 credits)

Course objectives
Implement and use backpropagation algorithms to train deep neural networks
Apply regularization techniques to training deep neural networks
Apply optimization techniques to training deep neural networks
Course Outcomes and CO/PO Mapping
Course Outcomes
[Link]. COs Description
1 CO1 Understand and implement deep learning networks and algorithms
2 CO2 Understand implement reinforcement learning algorithms.
3 CO3 Understand and implement deep reinforcement learning algorithms
4 CO4 Apply deep reinforcement learning in automation and robotics

CO/PO Mapping
CO/ PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 PO10 PO11 PO12 PSO1 PSO2 PSO3
PO
CO1 3 2 2 2 3 1 2 3
CO2 3 2 2 2 3 1 2 3
CO3 3 2 2 2 3 1 2 3
CO4 3 2 2 2 3 1 2 3
Syllabus
• Unit 1 : 15 hours
• Deep Learning: Artificial Neurons - the Building Blocks of Deep
Learning, Feed-Forward Deep Neural Networks (DNN),
Architectural Considerations in Deep Learning: Activation
Functions in Deep Learning, Loss Functions in Deep Learning,
Optimizers in Deep Learning: Gradient Descent and Error Back-
Propagation, Stochastic Gradient Descent and Adaptive Learning
Rate, Hyper-Parameter Selection, Regularization; Convolutional
Neural Networks: Convolutional Layer, Pooling Layer, Flattened and
Fully Connected Layers; Recurrent Neural Networks, LSTM, Deep
learning examples.
Syllabus
• Unit 2: 15 hours
• Reinforcement Learning: Agents, environments, State and action,
Reward, Reinforcement learning as a Markov Decision Process
(MDP), Value Functions & Bellman Equations, Prediction and
Control by Dynamic Programing, Monte Carlo Methods for Model
Free Prediction and Control, Temporal difference learning,
Function Approximation Methods, Policy Gradients., Applications
in industrial automation and Robotics
• Unit 3: hours
• Deep Reinforcement Learning Algorithms: Policy-based
Algorithms, Value-based Algorithms, Model-based Algorithms,
Combined Methods, On-policy and Off-policy Algorithms, Deep
Reinforcement Learning for the Control of Robotic Manipulation.
Reference / Text Books
1. Ian Goodfellow, Yoshua Bengio and Aeron Courville, Deep Learning, MIT
Press, First Edition, 2016.
2. Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An
Introduction, 2nd Edition, The MIT Press, 2018
3. Hao Dong, Zihan Ding, and Shanghang Zhang, Deep Reinforcement
Learning: Fundamentals, Research and Applications, Springer, 2020
4. Laura Graesser and Wah Loon Keng, Foundations of Deep Reinforcement
Learning: Theory and Practice in Python, Addison-Wesley, 2020
5. Sudharsan Ravichandiran, Hands-On Reinforcement Learning with
Python: Master reinforcement and deep reinforcement learning using
OpenAI Gym and TensorFlow, 2nd Edition, 2020.
Evaluation Pattern
Component Weightage Remarks
Continuos Infernal Déamination (CIE) Evaluation Components
Mid Term 30
Continuous Assessment CA1 7.5 Implementation of deep learning in
robotics and automation
Continuous Assessment CA2 7.5 Implementation of reinforcement
learning in robotics and automation
Continuous Assessment CA3 7.5 Implementation of deep reinforcement
learning in robotics and automation
Continuous Assessment CA4 7.5 Quiz/Viva
Semester End Examination (SEE) Components
End Semester Examination 40 --
The Definition and Subsets of AI
Official definitions of AI ranges from
“…systems or machines that mimic
human intelligence to perform tasks and
can iteratively improve themselves based
on the information they collect” to "the
ability of a digital computer...to perform
tasks commonly associated with (human)
characteristics, such as the ability to
reason, discover meaning, generalize, or
learn from past experience".

The theme in most common definitions is

AI 'mimics' human behaviour.
 Relationship: AI, ML, DL, RL, DRL
Artificial Intelligence
Machine Learning
Deep Learning

Deep Reinforcement Learning

Reinforcement Learning

Relationship of artificial intelligent, machine learning, deep learning,

reinforcement learning, and deep reinforcement learning
The Evolution of AI, ML, and DL
Machine Learning
Unsupervised Learning

Clustering can be defined as “A way

of grouping the data points into
different clusters, consisting of
similar data points. The objects with
the possible similarities remain in a
group that has less or no similarities
with another group."
 Supervised Learning

Known data
Model
Known responses
In supervised learning, models are trained using Training
labelled dataset, where the model learns about
each type of data. Once the training process is Model
completed, the model is tested on the basis of Predicted responses
test data (a subset of the training set), and then New data
Testing
it predicts the output.
Machine Learning Vs. Deep Learning

The Deep Learning algorithm doesn't need a software engineer to identify

features but is capable of automatic feature engineering through its neural
network.
Deep Learning Vs. Machine Learning

Factors Deep Learning Machine Learning

Data Requires large data Can train on
requirement lesser data
Accuracy Provides high Gives lesser
accuracy accuracy
Training time Takes longer to Takes less time to
train train
Hardware Requires GPU to Trains on CPU
dependency train properly Deep is a technical term. It refers to the
number of layers in a neural network. A
Hyperparameter Can be tuned in Limited tuning shallow network has one so-called hidden
Tuning various different capabilities layer, and a deep network has more than one.
ways Multiple hidden layers allow deep neural
networks to learn features of the data
Important AI-related terms and definitions
Term Description
Machine Process by which an algorithm encodes statistical regularities from a database of examples into
learning (ML) parameter weights for future predictions
Deep learning Multilayered complex ML platform comprised of numerous computational layers able to make
(DL) accurate predictions
Supervised Training an ML algorithm using previously labeled training examples, consisting of inputs and
learning desired outputs provided by an expert
Unsupervised When an ML algorithm discovers hidden patterns or data groupings without the need for
learning human intervention
Reinforcement Learning strategies towards acting optimally in certain situations with respect to a given
learning criterion; such an algorithm obtains feedback on its performance by comparison with this
criterion through reward values during training
Model A trained ML algorithm that can make predictions from unseen data
Training Feeding an ML algorithm with examples from a training dataset towards deriving useful
parameters for future predictions
Important AI-related terms and definitions
Term Description
Features Components of a dataset describing the characteristics of the studied observations
Decision tree Nonparametric supervised learning method visualized as a graph representing the choices and
their outcomes in the form of a tree; each tree consists of nodes (attributes in the group to be
classified) and branches (values that a node can take)
Random forest Ensemble classification technique that uses “parallel ensembling”, fitting several decision tree
classifiers in parallel on dataset subsamples
Naïve Bayes Classification technique assuming independence among predictors (i.e., assumes that the
(NB) presence of a feature in the class is unrelated to the presence of any other feature)
Logistic Algorithm using a logistic function to estimate probabilities that can overfit high-dimensional
regression datasets, being suitable for datasets that can be linearly separated
K-nearest “Instance-based learning” or a non-generalizing learning algorithm that does not focus on
neighbors constructing a general internal model but, rather, stores all instances corresponding to the
(KNN) training data in an n-dimensional space and classifies new data points based on similarity
measures
 Important AI-related terms and definitions
Term Description
Support vector Supervised learning model that can efficiently perform linear and nonlinear classifications,
machine (SVM) implicitly mapping their inputs into high-dimensional feature spaces
Boosting Family of algorithms converting weak learners (i.e., classifiers) to strong learners (i.e.,
classifiers that are arbitrarily well-correlated with the true classification) towards decreasing
the bias and variance
Artificial neural An ML technique that processes information in an architecture comprising many layers
network (ANN) (“neurons”), each inter-neuronal connection extracting the desired parameters incrementally
from the training data
Deep neural network A DL architecture with multiple layers between the input and output layers
(DNN)
Convolutional neural A class of DNN displaying connectivity patterns similar to the connectivity patterns and image
network (CNN) processing in the visual cortex

Busnatu Ș, Niculescu AG, Bolocan A, Petrescu GED, Păduraru DN, Năstasă I, Lupușoru M, Geantă M, Andronic O,
Grumezescu AM, Martins H. Clinical Applications of Artificial Intelligence-An Updated Overview. J Clin Med. 2022 Apr
18;11(8):2265. doi: 10.3390/jcm11082265. PMID: 35456357; PMCID: PMC9031863.
 What is reinforcement learning?
• “Reinforcement learning, a type of machine learning, in which agents
take actions in an environment aimed at maximizing their cumulative
rewards” – NVIDIA
• “Reinforcement learning (RL) is based on rewarding desired behaviors or
punishing undesired ones. Instead of one input producing one output,
the algorithm produces a variety of outputs and is trained to select the
right one based on certain variables” – Gartner
• “It is a type of machine learning technique where a computer agent
learns to perform a task through repeated trial and error interactions
with a dynamic environment. This learning approach enables the agent
to make a series of decisions that maximize a reward metric for the task
without human intervention and without being explicitly programmed to
achieve the task” – Mathworks
 Explanation to Reinforcement Learning

How does reinforcement

learning work?

You can see a dog and a master. Let’s imagine

you are training your dog to get the stick. Each
time the dog gets a stick successfully, you
offered him a feast (a bone let’s say). Eventually,
the dog understands the pattern, that whenever
the master throws a stick, it should get it as
early as it can to gain a reward (a bone) from a
master in a lesser time.
Reinforcement Learning in ML
Terminologies used in Reinforcement
Learning
• Agent: is the sole decision-maker and learner
• Environment: a physical world where an agent learns and decides the actions to
be performed
• Action: a list of action which an agent can perform
• State: the current situation of the agent in the environment
• Reward: For each selected action by agent, the environment gives a reward. It’s
usually a scalar value and nothing but feedback from the environment
• Policy: the agent prepares strategy(decision-making) to map situations to actions.
• Value Function: The value of state shows up the reward achieved starting from
the state until the policy is executed
• Model: Every RL agent doesn’t use a model of its environment. The agent’s view
maps state-action pairs probability distributions over the states
Deep Reinforcement Learning
Deep reinforcement learning is a
subfield of machine learning that
combines reinforcement learning
and deep learning. RL considers
the problem of a computational
agent learning to make decisions
by trial and error. Deep RL
incorporates deep learning into
the solution, allowing agents to
make decisions from
unstructured input data without
manual engineering of the state
space.
 Application in Robotics
• Object Recognition: Object recognition is a crucial task in robotics, and it is
essential for autonomous navigation and manipulation (e.g., DL techniques
like CNN).
• Motion Planning: Motion planning is a key task in robotics that involves
finding a collision-free path for a robot to move from one point to another
(e.g., Reinforcement Learning).
• Control: Control is another important task in robotics, and it involves
regulating the movement of robots (e.g., Deep Reinforcement Learning).
• Localization: Localization is the process of determining the position of a robot
in its environment (e.g., ML techniques such as SVM and Random Forests).
• Object Detection: Object detection is the process of detecting and localizing
objects in an image (e.g., DL techniques like Faster R-CNN and YOLO).
Advantages of AI, ML and DL applications in
advanced robotics
• Automation: AI, ML, and DL can automate many repetitive and mundane tasks in
robotics, freeing up human resources to focus on more complex tasks.
• Enhanced accuracy: These technologies can improve the accuracy and precision
of robotic systems, reducing errors and improving overall performance.
• Adaptability: AI-powered robots can adapt to changing environments and tasks,
making them highly versatile and useful in a range of industries and applications.
• Predictive Maintenance: Machine learning algorithms can help robots to predict
when maintenance or repairs are required, leading to reduced downtime and
cost savings.
• Improved Decision Making: AI and ML algorithms can analyze large amounts of
data and make informed decisions based on that data, allowing robots to make
better decisions and take appropriate actions.
Advantages of AI, ML and DL applications in
advanced robotics
• Improved efficiency: By optimizing processes and reducing waste, AI, ML, and DL
can improve the overall efficiency of robotics systems, resulting in cost savings
and increased productivity.
• Better decision-making: AI, ML, and DL can enable robots to make better
decisions based on data analysis and pattern recognition, leading to improved
performance and outcomes.
• Adaptability: These technologies can enable robots to adapt to changing
environments and situations, making them more versatile and capable of
handling a wider range of tasks.
• Increased safety: By automating hazardous or dangerous tasks, AI, ML, and DL
can improve safety in the workplace, reducing the risk of accidents and injuries.
• Cost Reduction: The implementation of AI and ML applications in advanced
robotics can significantly reduce costs associated with labor and maintenance.
• Improved Decision-making: By using AI and ML algorithms, robots can make
informed decisions based on data analysis, resulting in better overall
performance.