Deep Learning and

Reinforcement
Learning-BAI701
Module-1
DEEP LEARNING 1
 Syllabus
Introduction to Deep Learning
– Introduction,
– Shallow Learning,
– Deep Learning,
– Why to use Deep Learning,
– How Deep Learning Works,
– Deep Learning Challenges,.
– How Learning Differs from Pure Optimization,
– Challenges in Neural Network Optimization.
• Textbook 1: Ch 1.1 – 1.6, Textbook 2: 8.1,8.2

DEEP LEARNING 2
 Introduction to Deep Learning
• Machine learning systems, with shallow or deep
architectures, have ability to learn and improve
with experience.
• The process of machine learning begins with the
raw data which is used for extracting useful
information that helps in decision-making.
• The primary aim is to allow a machine to learn
useful information just like humans do.
• At abstract level, machine learning can be carried
out using following approaches:
– Supervised learning
– Unsupervised learning
– Semi-supervised learning

DEEP LEARNING 3
 Introduction to Deep Learning
• Supervised learning adapts a system such that for a given input
data it produces a target output. The learning data is made up of
tuples (attributes, label) where “attributes” represent the input
data and “label” represents the target output. The goal here is to
adapt the system so that for a new input the system can predict the
target output. Supervised learning can use both continuous and
discrete types of input data.
• Unsupervised learning involves data that comprises of input vectors
without any target output. There are different objectives in
unsupervised learning, such as clustering, density estimation, and
visualization. The goal of clustering is to discover groups of similar
data items on the basis of measured or perceived similarities
between the data items. The purpose of density estimation is to
determine the distribution of the data within the input space. In
visualization, the data is projected down from a high-dimensional
space to two or three dimensions to view the similar data items.
DEEP LEARNING 4
 Introduction to Deep Learning
Semi-supervised learning first uses unlabeled data to
learn a feature representation of the input data and then
uses the learned feature representation to solve the
supervised task. The training dataset can be divided into
two parts: the data samples with corresponding labels
and the data samples where the labels are not known.
Semi-supervised learning can involve not providing with
an explicit form of error at each time but only a
generalized reinforcement is received giving indication of
how the system should change its behavior, and this is
sometimes referred to as reinforcement learning.
Reinforcement learning has been successful in
applications as diverse as autonomous helicopter flight,
robot legged locomotion, cell-phone network routing,
marketing strategy selection, factory control and
efficient webpage indexing.

DEEP LEARNING 5
 Shallow Learning
• Shallow architectures are well understood and
perform good on many common machine learning
problems, and they are still used in a vast majority of
today’s machine learning applications.
• However, there has been an increased interest in
deep architectures recently, in the hope to find
means to solve more complex real-world problems
(e.g., image analysis or natural language
understanding) for which shallow architectures are
unable to learn models adequately.

DEEP LEARNING 6
 Deep Learning
• Definition and Characteristics: Deep learning is a
subset of machine learning and artificial intelligence
(AI) that uses deep networks with multiple hidden
layers to learn features at various abstraction levels.
• Unlike traditional machine learning, which requires
manual feature extraction, deep learning processes
raw data (e.g., pixel values) directly and
automatically learns features during training through
hierarchical layers.
• Hierarchical learning: Lower layers detect basic
features (e.g., edges), while higher layers build on
these to form more complex concepts (e.g., object
parts).
DEEP LEARNING 7
Fig. 1.1 a Conventional machine learning using
hand-designed feature extraction algorithms
b deep learning approach using hierarchy of
representations that are learnt automatically

DEEP LEARNING 8
 Deep Learning(Cont..)
• The word “deep” refers to learning successive
layers of increasingly meaningful representations
of input data.
• The number of layers used to model the data
determines the depth of the model. Current deep
learning often involves learning tens or even
hundreds of successive layers of representation
from the training data automatically.
• The conventional approaches to machine learning
often focus on learning only one or two layers of
representations of data; such approaches are
often categorized as shallow learning.

DEEP LEARNING 9
Deep learning and machine learning are subfields of Artificial Intelligence (AI).
Figure 1.2 illustrates the relationship between AI, machine learning, and deep
learning.
DEEP LEARNING 10
• In deep learning, the successive layers of representations
may be learned via sub models, which are structured in
the form of layers stacked on the top of each other.
• As deep learning network has typically more layers and
parameters, it has the potential to represent more
complex inputs.
• Although deep learning has been around since 1980s, it
was relatively un popular for several years as the
computational infrastructure (both hardware and
software) was not adequate and the available datasets
were quite small.
• With the decline in the popularity of the conventional
neural networks, it was only recently that deep networks
made a big reappearance by achieving spectacular results
in speech recognition and computer vision tasks.
DEEP LEARNING 11
 Deep learning(Cont..)
• Some of the aspects that helped in the
evolution of deep networks are listed below:
– Improved computational resources for processing
massive amounts of data an training much larger
models.
– Automatic feature extraction.

DEEP LEARNING 12
 Deep Learning(Cont…)
• The term artificial neural networks has a reference to
neuroscience but deep learning networks are not models
of the brain; however, deep learning models are
formulated by only drawing inspiration from the
understanding of biological brain.
• Not all the components of deep models are inspired by
neuroscience; some of them come from empirical
exploration, theory, and intuition.
• The neural activity in our brains is far more complex than
might be suggested by simply studying artificial neurons.
The learning mechanisms used by deep learning models
are in no way comparable to the human brain, but can be
described as a mathematical framework for learning
representations from data.

DEEP LEARNING 13
Fig. 1.3 A deep learning network for digit classification

DEEP LEARNING 14
Fig. 1.4 Representations learnt by a deep network for digit
classification during the first pass. Network structural
changes can be incorporated that result in desired
representations at various layers

DEEP LEARNING 15
• The deep network uses several layers to transform
the input image (here a digit) in order to recognize
what the digit is.
• Each layer performs some transformations on the
input that it receives from the previous layers.
• The deep network transforms the digit image into
representations that tend to capture a higher level of
abstraction.
• Each hidden layer transforms the input image into a
representation that is increasingly different from the
original image and increasingly informative about the
final result.
• The representations learnt help to distinguish
between different concepts which in turn help to
find out similarities between it.
DEEP LEARNING 16
• Deep network can be thought of as a multistage
distillation information operation, where layers use
multiple filters on the information to obtain an
increasingly transformed form of information (i.e.,
the information useful with regard to some task).
• In summary, a deep learning network constructs
features at multiple levels, with higher features
constructed as functions of lower ones. It is a
fast-growing field that circumvents the problem of
feature extraction which is used as a prelude by
conventional machine learning approaches. Deep
learning is capable of learning the appropriate
features by itself, requiring little steering by the
user.

DEEP LEARNING 17
 Why to Use Deep Learning?
• Deep learning has revolutionized machine
learning by eliminating the need for manual
feature engineering, which traditionally required
collaboration between domain experts and ML
practitioners. In contrast to conventional
algorithms that rely on handcrafted features,
deep learning automatically learns hierarchical
representations directly from data using neural
networks. This ability makes deep learning
particularly powerful for handling
high-dimensional data like images, speech, and
text.
DEEP LEARNING 18
 Why to Use Deep Learning?(Cont…)

• Its rise to prominence around 2007 brought

significant advances in perceptual tasks such
as image recognition, speech recognition, and
natural language processing. Deep learning
models have achieved near-human or even
superhuman performance in several domains
including self-driving cars, digital assistants
(e.g., Siri, Alexa), game playing (e.g., Go,
Chess), and real-time translation.
DEEP LEARNING 19
 Why to Use Deep Learning?(Cont…)
• The strength of deep networks lies in their
ability to approximate complex non-linear
functions, their use of large datasets, and the
availability of computational power (e.g.,
GPUs). Regularization techniques further help
these large models generalize well on unseen
data. Furthermore, deep learning models are
largely domain-agnostic, requiring minimal
domain-specific tuning, thus offering
scalability across diverse applications.
DEEP LEARNING 20
 Why to Use Deep Learning?(Cont…)
• Today, tech giants like Google, Microsoft,
Facebook, and Amazon rely heavily on deep
learning for tasks ranging from personalized
recommendations to automated customer
support. Its ability to learn from raw data with
minimal human intervention makes deep
learning not only a practical solution for
current AI challenges but also a promising
direction for the future of artificial
intelligence.
DEEP LEARNING 21
Summary Why Use Deep Learning – Key Points
1. Automated Feature Learning-Deep learning learns
features directly from raw data, removing the need for
manual feature engineering.
2. Handles High-Dimensional Data-Effective in processing
complex data types like images, speech, and text.
3. Superior Accuracy-Outperforms traditional machine
learning models in tasks like image classification and speech
recognition.
4. Domain-Agnostic Models-Requires minimal
domain-specific adjustments—models can generalize across
different applications.
5. Breakthrough in AI Tasks- Achieved near-human or
superhuman performance in: Image and speech recognition,
Natural language processing, Game playing (Go, Chess, Shogi)
DEEP LEARNING 22
6. Scalability with Big Data & Compute
Power-Leverages vast data and computational
resources (e.g., GPUs) for better performance.
7. Used by Tech Giants-Widely adopted by Google,
Amazon, Microsoft, Facebook, Baidu for search, ads,
translation, and virtual assistants.
8. Minimal Human Intervention- Requires less
hand-engineering, enabling faster deployment and
experimentation.
9. Real-World Impact-Applied in self-driving cars,
healthcare, finance, customer service (e.g., chatbots),
and more.
10. Future Potential -Expected to aid in software
development, scientific discovery, and combined AI
systems with significant technological impact.

DEEP LEARNING 23
 How Deep Learning Works?
• Deep Networks Structure
– Deep learning models map inputs to targets using
multiple layered transformations
– Each layer performs a transformation determined by
its weights (numerical parameters).
– These transformations are learned from training data
• Learning in Deep Networks
– Learning = Finding optimal weight values across all
layers.
– The goal: input examples → correctly mapped to their
target outputs.
– Complexity arises as weights are interdependent
(change in one affects others).

DEEP LEARNING 24
 How Deep Learning Works?(cont.)
• Role of Loss Function (Objective Function)
– Measures difference between predicted output and actual
target
– Provides a quantitative feedback of how well the network is
performing.
– Training objective: Minimize this error using the function.
• Training Process with Back propagation
– Starts with random weights→ output is initially inaccurate →
high loss.
– For each training example:
• Calculate loss.
• Adjust weights to reduce loss.
– Done using optimizer algorithm, mainly back propagation.
– Repeated over multiple iterations until loss is minimized

DEEP LEARNING 25
How Deep Learning Works?(cont.)
• Network Learning Completion
– Network is considered trained when predicted
outputs ≈ true targets
– This state signifies that the model has effectively
learned the data patterns

DEEP LEARNING 26
 Deep Learning Challenges
Deep learning networks have brought their own set
of problems and challenges which outweighed the
benefits of deep architectures for several decades.
• Training these architectures for general use was
impractically slow. With limited computational
power, deep learning networks were already
overtaken by other approaches such as kernel
methods. With the significant growth in
computational power (particularly in GPUs and
distributed computing) and access to large labeled
datasets paved the way for its return.

DEEP LEARNING 27
 Deep Learning Challenges(Cont.)
• However, despite the remarkable advances in this area,
training deep models with a huge number of free
parameters is an intricate and ill-posed optimization
problem. Many research works have been dedicated to
creating efficient training methods for deep architectures.
The strategies reported in the literature that deal with the
difficulties of training deep networks include developing
better optimizers, using well-designed initialization
strategies, using activation functions based on local
competition and using skip connections between layers
with the aim to improve the flow of information.
However, deep network training still faces problems
which are caused by the stacking of several nonlinear
transformations and need to be addressed.
DEEP LEARNING 28
 Deep Learning Challenges(Cont.)
• Moreover, deep learning involves using large
amounts of data to learn progressively. While large
amounts of data are available in many applications,
however, in some areas copious amount of data are
rarely available. More flexible models are required to
achieve an enhanced learning ability when only a
limited amount of data is available.
• Deep learning networks are very good at solving one
problem; however, using deep networks to solve a
very similar problem requires retraining and
reassessment. Although there are many
advancements in this aspect, more work is required
in developing deep learning models which can
perform multitasks without the need of reworking on
the whole architecture.
DEEP LEARNING 29
 How Learning Differs from Pure
Optimization
Optimization algorithms used for training of deep models
differ from traditional optimization algorithms in several ways.
• Machine learning usually acts indirectly. In most machine
learning scenarios, we care about some performance
measure P , that is defined with respect to the test set and
may also be intractable. We therefore optimize P only
indirectly. We reduce a different cost function J(θ) in the
hope that doing so will improve P . This is in contrast to
pure optimization, where minimizing J is a goal in and of
itself. Optimization algorithms for training deep models also
typically include some specialization on the specific
structure of machine learning objective functions.

DEEP LEARNING 30
• Typically, the cost function can be written as an
average over the training set, such as

(8.1)
• where L is the per-example loss function, f (x; θ) is
the predicted output when the input is x,
ˆpdata is the empirical distribution. In the
supervised learning case, y is the target output.
Throughout this chapter, we develop the
unregularized supervised case, where the
arguments to L are f(x; θ) and y. However, it is
trivial to extend this development, for example, to
include θ or x as arguments, or to exclude y as
arguments, in order to develop various forms of
regularization or unsupervised learning

DEEP LEARNING 31
• Equation 8.1 defines an objective function
with respect to the training set. We would
usually prefer to minimize the corresponding
objective function where the expectation is
taken across the data generating distribution
pdata rather than just over the finite training
set:

DEEP LEARNING 32
 How Learning Differs from Pure
Optimization(Cont..)
• Empirical Risk Minimization
• Surrogate Loss Functions and Early Stopping
• Batch and Minibatch Algorithms

DEEP LEARNING 33