LY, SEM.

-VII,
DEEP LEARNING
( Course Code- AIC701)
Module 1 : Introduction to Deep
Learning

D. K. Bathe

Module 1 - DL/AIDS/KJSIT/23-24
Module 1:Introduction to Deep
Learning:
1.1: Biological neuron, Mc-Culloch Pitts
Neuron, Perceptron, Perceptron Learning,
Delta learning, Multilayer Perceptron: Linearly
separable, linearly non-separable classes

1.2: Deep Networks: Fundamentals, Brief

History, Three Classes of Deep Learning Basic
Terminologies of Deep Learning

Module 1 - DL/AIDS/KJSIT/25-26
What is the Perceptron model in Machine
Learning?
● Perceptron is Machine Learning algorithm for supervised learning of
various binary classification tasks. Further, Perceptron is also understood
as an Artificial Neuron or neural network unit that helps to detect certain
input data computations in business intelligence.
● Perceptron model is also treated as one of the best and simplest types of
Artificial Neural networks. However, it is a supervised learning algorithm of
binary classifiers. Hence, we can consider it as a single-layer neural
network with four main parameters, i.e., input values, weights and Bias,
net sum, and an activation function.
● In machine learning, the perceptron (or McCulloch–Pitts neuron) is an
algorithm for supervised learning of binary classifiers. A binary classifier is
a function which can decide whether or not an input, represented by a
vector of numbers, belongs to some specific class. It is a type of linear
classifier, i.e. a classification algorithm that makes its predictions based on
a linear predictor function combining a set of weights with the feature
vector.

Module 1 - DL/AIDS/KJSIT/23-24
What is Binary classifier in Machine Learning?
● In Machine Learning, binary classifiers are defined as the
function that helps in deciding whether input data can be
represented as vectors of numbers and belongs to some
specific class.
● Binary classifiers can be considered as linear classifiers. In
simple words, we can understand it as a classification algorithm
that can predict linear predictor function in terms of weight and
feature vectors.

Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
 ○ Input Nodes or Input Layer:

This is the primary component of Perceptron which accepts the initial data into the system for further
processing. Each input node contains a real numerical value.

○ Wight and Bias:

Weight parameter represents the strength of the connection between units. This is another most important
parameter of Perceptron components. Weight is directly proportional to the strength of the associated
input neuron in deciding the output. Further, Bias can be considered as the line of intercept in a linear
equation.

○ Activation Function:

These are the final and important components that help to determine whether the neuron will fire or not.
Activation Function can be considered primarily as a step function.

Types of Activation functions:

○ Sign function
○ Step function, and
○ Sigmoid function

Module 1 - DL/AIDS/KJSIT/23-24
How does Perceptron work?
In Machine Learning, Perceptron is considered as a single-layer neural network that consists of
four main parameters named input values (Input nodes), weights and Bias, net sum, and an
activation function. The perceptron model begins with the multiplication of all input values and
their weights, then adds these values together to create the weighted sum. Then this weighted
sum is applied to the activation function 'f' to obtain the desired output. This activation function
is also known as the step function and is represented by 'f'.

Module 1 - DL/AIDS/KJSIT/23-24
This step function or Activation function plays a vital role in ensuring that output is mapped
between required values (0,1) or (-1,1). It is important to note that the weight of input is
indicative of the strength of a node. Similarly, an input's bias value gives the ability to shift the
activation function curve up or down.

Perceptron model works in two important steps as follows:

Step-1

In the first step first, multiply all input values with corresponding weight values and then add
them to determine the weighted sum. Mathematically, we can calculate the weighted sum as
follows:

∑wi*xi = x1*w1 + x2*w2 +…wn*xn

Add a special term called bias 'b' to this weighted sum to improve the model's performance.

∑wi*xi + b

Step-2

In the second step, an activation function is applied with the above-mentioned weighted sum,
which gives us output either in binary form or a continuous value as follows:

Y = f(∑wi*xi + b)

Module 1 - DL/AIDS/KJSIT/23-24
Single layer Perceptron

Multi-layer Perceptron

Image source- [Link]

• Like a single-layer perceptron model, a multi-layer perceptron model
also has the same model structure but has a greater number of hidden
layers.
• The multi-layer perceptron model is also known as the Backpropagation
algorithm, which executes in two stages as follows:
• Forward Stage: Activation functions start from the input layer in the forward
stage and terminate on the output layer.
• Backward Stage: In the backward stage, weight and bias values are modified as
per the model's requirement. In this stage, the error between actual output
and demanded originated backward on the output layer and ended on the
input layer.
• Hence, a multi-layered perceptron model has considered as multiple
ANN having various layers in which activation function does not remain
linear, similar to a single layer perceptron model. Instead of linear,
activation function can be executed as sigmoid, TanH, ReLU, etc., for
deployment.
• A multi-layer perceptron model has greater processing power and can
process linear and non-linear patterns. Further, it can also implement
logic gates such as AND, OR, XOR, NAND, NOT, XNOR, NOR.
• Image source- [Link]
• Advantages of Multi-Layer Perceptron:
• A multi-layered perceptron model can be used to solve
complex non-linear problems.
• It works well with both small and large input data.
• It helps us to obtain quick predictions after the training.
• It helps to obtain the same accuracy ratio with large as well
as small data.

• Disadvantages of Multi-Layer Perceptron:

• In Multi-layer perceptron, computations are difficult and
time-consuming.
• In multi-layer Perceptron, it is difficult to predict how much
the dependent variable affects each independent variable.
• The model functioning depends on the quality of the
training.
Learning rule enhances the Artificial Neural Network’s performance by applying this rule over the
network. Thus learning rule updates the weights and bias levels of a network when certain conditions
are met in the training process. it is a crucial part of the development of the Neural Network.

1. Hebbian Learning Rule

Donald Hebb developed it in 1949 as an unsupervised learning algorithm in the neural network. We can
use it to improve the weights of nodes of a network. The following phenomenon occurs when
● If two neighbor neurons are operating in the same phase at the same period of time, then the
weight between these neurons should increase.
● For neurons operating in the opposite phase, the weight between them should decrease.
● If there is no signal correlation, the weight does not change, the sign of the weight between two
nodes depends on the sign of the input between those nodes
● When inputs of both the nodes are either positive or negative, it results in a strong positive
weight.
● If the input of one node is positive and negative for the other, a strong negative weight is
present.

Mathematical Formulation:
δw=αxiy
where δw=change in weight,α is the learning [Link] the input vector,y the output.
2. Perceptron Learning Rule
It was introduced by Rosenblatt. It is an error-correcting rule of
a single-layer feedforward network. it is supervised in nature
and calculates the error between the desired and actual output
and if the output is present then only adjustments of weight are
done.
Computed as follows:

Assume (x1,x2,x3……………………….xn) –>set of input vectors and

(w1,w2,w3…………………..wn) –>set of weights

y=actual output; wo=initial weight; wnew=new weight; δw=change in weight; α=learning rate

actual output(y)=wixi

learning signal(ej)=ti-y (difference between desired and actual output)

δw=αxiej

wnew=wo+δw
Now, the output can be calculated on the basis of the input
and the activation function applied over the net input and
can be expressed as:
y=1, if net input>=θ
y=0, if net input<θ
3. Delta Learning Rule
It was developed by Bernard Widrow and Marcian
Hoff and It depends on supervised learning and has
a continuous activation function. It is also known as
the Least Mean Square method and it minimizes
error over all the training patterns.

It is based on a gradient descent approach which

continues forever. It states that the modification in
the weight of a node is equal to the product of the
error and the input where the error is the difference
between desired and actual output.
Assume (x1,x2,x3……………………….xn) –>set of input vectors

and (w1,w2,w3…………………..wn) –>set of weights

y=actual output, wo=initial weight, wnew=new weight. δw=change in weight

Error= ti-y; Learning signal(ej)=(ti-y)y’

y=f(net input)= ∫wixi

δw=αxiej=αxi(ti-y)y’

wnew=wo+δw

The updating of weights can only be done if there is a difference between

the target and actual output(i.e., error) present:
case I: when t=y
then there is no change in weight

case II: else

wnew=wo+δw
Types of Perceptron Models
Based on the layers, Perceptron models are divided into two types. These are as follows:

1. Single-layer Perceptron Model

2. Multi-layer Perceptron model

Single Layer Perceptron Model:

This is one of the easiest Artificial neural networks (ANN) types. A single-layered perceptron model
consists feed-forward network and also includes a threshold transfer function inside the model. The
main objective of the single-layer perceptron model is to analyze the linearly separable objects with
binary outcomes.

In a single layer perceptron model, its algorithms do not contain recorded data, so it begins with
inconstantly allocated input for weight parameters. Further, it sums up all inputs (weight). After adding
all inputs, if the total sum of all inputs is more than a pre-determined value, the model gets activated and
shows the output value as +1.

If the outcome is same as pre-determined or threshold value, then the performance of this model is
stated as satisfied, and weight demand does not change. However, this model consists of a few
discrepancies triggered when multiple weight inputs values are fed into the model. Hence, to find desired
output and minimize errors, some changes should be necessary for the weights input.

"Single-layer perceptron can learn only linearly separable patterns."

Module 1 - DL/AIDS/KJSIT/23-24
Multi-Layered Perceptron Model:

Like a single-layer perceptron model, a multi-layer perceptron model also has the same model structure
but has a greater number of hidden layers.

The multi-layer perceptron model is also known as the Backpropagation algorithm, which executes in
two stages as follows:

○ Forward Stage: Activation functions start from the input layer in the forward stage and terminate
on the output layer.
○ Backward Stage: In the backward stage, weight and bias values are modified as per the model's
requirement. In this stage, the error between actual output and demanded originated backward
on the output layer and ended on the input layer.

Hence, a multi-layered perceptron model has considered as multiple artificial neural networks having
various layers in which activation function does not remain linear, similar to a single layer perceptron
model. Instead of linear, activation function can be executed as sigmoid, TanH, ReLU, etc., for
deployment.

A multi-layer perceptron model has greater processing power and can process linear and non-linear
patterns. Further, it can also implement logic gates such as AND, OR, XOR, NAND, NOT, XNOR, NOR.

Module 1 - DL/AIDS/KJSIT/23-24
Advantages of Multi-Layer Perceptron:

○ A multi-layered perceptron model can be used to solve complex non-linear

problems.
○ It works well with both small and large input data.
○ It helps us to obtain quick predictions after the training.
○ It helps to obtain the same accuracy ratio with large as well as small data.

Disadvantages of Multi-Layer Perceptron:

○ In Multi-layer perceptron, computations are difficult and time-consuming.

○ In multi-layer Perceptron, it is difficult to predict how much the dependent variable
affects each independent variable.
○ The model functioning depends on the quality of the training.

Module 1 - DL/AIDS/KJSIT/23-24
Perceptron Function
Perceptron function ''f(x)'' can be achieved as output by multiplying the input 'x' with the learned weight coefficient 'w'.

Mathematically, we can express it as follows:

f(x)=1; if w.x+b>0

otherwise, f(x)=0

○ 'w' represents real-valued weights vector

○ 'b' represents the bias
○ 'x' represents a vector of input x values.

Characteristics of Perceptron
The perceptron model has the following characteristics.

1. Perceptron is a machine learning algorithm for supervised learning of binary classifiers.

2. In Perceptron, the weight coefficient is automatically learned.
3. Initially, weights are multiplied with input features, and the decision is made whether the neuron is fired or not.
4. The activation function applies a step rule to check whether the weight function is greater than zero.
5. The linear decision boundary is drawn, enabling the distinction between the two linearly separable classes +1
and -1.
6. If the added sum of all input values is more than the threshold value, it must have an output signal; otherwise,
no output will be shown.

Module 1 - DL/AIDS/KJSIT/23-24
Future of Perceptron
● The future of the Perceptron model is much bright and significant as it
helps to interpret data by building intuitive patterns and applying them
in the future.
● Machine learning is a rapidly growing technology of Artificial
Intelligence that is continuously evolving and in the developing phase;
● hence the future of perceptron technology will continue to support and
facilitate analytical behavior in machines that will, in turn, add to the
efficiency of computers.
● The perceptron model is continuously becoming more advanced and
working efficiently on complex problems with the help of artificial
neurons.

Module 1 - DL/AIDS/KJSIT/23-24
Introduction to Deep
Learning

Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
1. Deep learning is a type of machine learning in which a model learns to
perform classification tasks directly from images, text, or sound.

2. Deep learning is usually implemented using a neural network

architecture. The term “deep” refers to the number of layers in the
network – the more layers, the deeper the network.

3. Traditional neural networks contain only 2 or 3 layers, while deep

networks can have hundreds.

4. Self-driving cars, Automation, Robots taking over jobs.

5. These are just a few of the topics that the media covers when discussing
artificial intelligence (AI). Of course, the story is far more complex, and the
burgeoning fields of AI, machine learning, and deep learning can have a
profound impact on our business, career, and our life on a daily basis.

Module 1 - DL/AIDS/KJSIT/23-24
Deep Learning is a subfield of machine learning that is concerned with
algorithms inspired by the brain’s structure and functions known as artificial
neural networks.
A computer model can be taught using Deep Learning to run classification
actions using pictures, texts or sounds as input?
Three technology enablers make this degree of accuracy possible:
1. Easy access to massive sets of labelled data: Data sets such as
ImageNet and PASCAL VoC are freely available and are useful for training
on many different types of objects.
2. Increased computing power: High-performance GPUs accelerate the
training of the massive amounts of data needed for deep learning,
reducing training time from weeks to hours.
3. Pretrained models built by experts: Models such as AlexNet can be
retrained to perform new recognition tasks using a technique called
transfer learning. While AlexNet was trained on 1.3 million
high-resolution images to recognize 1000 different objects, accurate
transfer learning can be achieved with much smaller datasets.

Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Background and Definations
● D1:“A class of machine learning techniques that
exploit many layers of non-linear information
processing for supervised or unsupervised feature
extraction and transformation, and for pattern
analysis and classification”
● D2:“A sub-field within machine learning that is
based on algorithms for learning multiple levels of
representation in order to model complex
relationships among data. Higher-level features and
concepts are thus defined in terms of lower-level
ones, and such a hierarchy of features is called a
deep architecture. Most of these models are based
on unsupervised learning of representations.”

Module 1 - DL/AIDS/KJSIT/23-24
● D3:: “A sub-field of machine learning that is based on
learning several levels of representations, corresponding to
a hierarchy of features or factors or concepts, where
higher-level concepts are defined from lower-level ones,
and the same lowerlevel concepts can help to define many
higher-level concepts.
● D4:“Deep learning is a set of algorithms in machine
learning that attempt to learn in multiple levels,
corresponding to different levels of abstraction. It typically
uses artificial neural networks. The levels in these learned
statistical models correspond to distinct levels of concepts,
where higher-level concepts are defined from lower-level
ones, and the same lowerlevel concepts can help to define
many higher-level concepts.”

Module 1 - DL/AIDS/KJSIT/23-24
● D5:: “Deep Learning is a new area of Machine
Learning research, which has been introduced
with the objective of moving Machine
Learning closer to one of its original goals:
Artificial Intelligence. Deep Learning is about
learning multiple levels of representation and
abstraction that help to make sense of data
such as images, sound, and text.”

Module 1 - DL/AIDS/KJSIT/23-24
Two Key Aspects:

[Link] consisting of multiple layers or

stages of nonlinear information processing;

[Link] for supervised or unsupervised

learning of feature representation at
successively higher, more abstract layers.

Module 1 - DL/AIDS/KJSIT/23-24
ML VS DL

Module 1 - DL/AIDS/KJSIT/23-24
Why do we need Deep Learning

? Machine Learning can not deal with crucial

problems like Image Recognition, Natural
Language Processing
? Dimensionality
? Feature Extraction

Module 1 - DL/AIDS/KJSIT/23-24
Why Now?

Module 1 - DL/AIDS/KJSIT/23-24
Deep Learning Process
Deep learning is a class of machine learning
algorithms that uses multiple layers to progressively
extract higher-level features from the raw input.
The process is articulated in 5 steps:
● Step 1: Understand the Problem.
● Step 2: Identify Data
● Step 3: Select Deep Learning ALgorithms
● Step 4: Training the Model
● Step 5: Test the Model

Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Deep Learning Applications
● Self driving car
● News Aggregation and Fraud News Detection
● Natural Language Processing (NLP)
● Virtual Assistants
● Entertainment
● Fraud Detection
● Personalisations
● Colorization of Black and White Images
● Adding Sounds To Silent Movies
● Language Translation
● Image Captioning

Module 1 - DL/AIDS/KJSIT/23-24
Limitations of Deep
learning
● Learn through observations
● Issue of biases
● learning rate
● hardware requirements
● Deep learning requires large amounts of data.

Module 1 - DL/AIDS/KJSIT/23-24
Neural Network-Biological
Neuron

Image Source: Wikipedia

Module 1 - DL/AIDS/KJSIT/23-24
 Analogy

Input Output

Financial
Institutiona Family
Conditio
l support Support
n

X1 X2 X3
1 0

1 0 1 0 1 0

(1,0,0)=?
=X1W1+X2W2+X3W
Priority Weight(W) Threshold(T)=5 3
(6,2,1) =6
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Data

Module 1 - DL/AIDS/KJSIT/23-24
Data

Module 1 - DL/AIDS/KJSIT/23-24
AND GATE

Module 1 - DL/AIDS/KJSIT/23-24
OR GATE

Module 1 - DL/AIDS/KJSIT/23-24
Basic deep learning
terminologies
● Neuron
● Weights
● Bias
● Activation function
● Neural network
● Input/output/Hidden layer
● MLP
● Forward Propagation
● Cost Function

Module 1 - DL/AIDS/KJSIT/23-24
Basic deep learning
terminologies
● Gradient Decent
● Learning Rate
● Backpropogation
● Batches
● Epochs
● Dropout
● Batch normalization

Module 1 - DL/AIDS/KJSIT/23-24
Historical Context of Deep
Learning
● Shallow-structured architectures
● Human information processing mechanisms
(e.g., vision and audition), however, suggest
the need of deep architectures
● Artificial neural network research:
new-generation neural networks

Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Three Classes of Deep
Learning Networks
❑ Deep networks for unsupervised or
generative learning

❑ Deep networks for supervised learning

❑ Hybrid deep networks

Module 1 - DL/AIDS/KJSIT/23-24
Simple examples on NN

Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
Module 1 - DL/AIDS/KJSIT/23-24
THANK
YOU

Module 1 - DL/AIDS/KJSIT/23-24