+

CS 522 – Selected Topics in CS

Lecture 07 – Artificial Neural Network (ANN)

March 20, 2022

 2

+ Outline
 What is ANN?
 Components of ANN
 How it works?
 Activation Functions
 Types of ANN
 Feedforward NN
 Back-Propagation in NN

 Role of Activation Function in Back-Propagation

 Basics for creating Deep NN model

March 20, 2022

 3

+ What is Neural Network?

 Neural Networks as a mathematical function that maps a given input to a
desired output.
 Neural Networks consist of the following components
 An input layer, x
 An arbitrary amount of hidden layers
 An output layer, ŷ
 A set of weights and biases between each layer, W and b
 A choice of activation function for each hidden layer, σ.

March 20, 2022

 4

+ What is Neuron?

March 20, 2022

 5

+ From Biological to Artificial Neurons

Behavior of an artificial neural network to any particular input depends upon:
 structure of each node (activation function)
 structure of the network (architecture)
 weights on each of the connections

.... these must be learned !

 6

+ McCulloch-Pitts Neuron (1st ver.)

 7

+ McCulloch-Pitts Neuron (1st ver.)

A visual example for OR and AND operations with McCulloch Pitts
Neuron is shown in Figure

Since the inputs from the events in McCulloch Pitts Neuron can only be Boolean values (0 or 1),
its capabilities were minimal.
 8

+ Linear Threshold Unit (LTU) Neuron (2nd ver.)

 Linear
Threshold Unit (LTU) was introduced in 1957 which
address the problem of McCulloch Pitts Neuron.
 In
an LTU, weights are assigned to each event, and these weights
can be negative or positive.
 The outcome of each even is still given a Boolean value (0 or 1), but
then is multiplied with the assigned weight.
 9

+ Linear Threshold Unit (LTU) Neuron (2nd ver.)

 Problem:
 The weight are fixed
 The output is still binary (0 or 1)
 It can’t be used to learn complex pattern available in the data.
 10

+ Perceptron (3rd ver.)

 Perceptron is a binary classification algorithm for supervised learning and
consists of a layer of LTUs.
 In a perceptron, LTUs use the same event outputs as input.
 The perceptron algorithm can adjust the weights to correct the behavior of
the trained neural network.
 Inaddition, a bias term may be added to increase the accuracy performance
of the network.

f(x) = ⟨w, x⟩ + b
 Ithelps to classify the given input data; the output value is f(x) calculated as

 where w is a vector of weights and ⟨·, ·⟩ denotes the dot product. We use the dot
product as we are computing a weighted sum. The sign of f(x) is used to classify x
as either a positive or a negative instance..
 11

+ Perceptron (3rd ver.)

 12

+ Perceptron (3rd ver.)

March 20, 2022

 13

+ How A N N works?
Wa
X1

Wb Y
X2 ∑ f

Wc
X3

Summati
Input units weights computati
on
Connection on
functio
(dendrit (synap n (axo
e) se) n)

March 20, 2022

 14

+ How A N N works?
 The artificial neuron receives one or more inputs and sums them to produce an
output.

 The sums of each node are weighted, and the sum is passed through a non-linear
function known as an activation function or transfer function.

 The activation functions usually have a sigmoid

shape, but they may also take the form of other non-
linear functions, piecewise linear functions, or step
functions.

 “Weighted sum” of its input, adds a bias and then

decides by activation function that whether it should

be “fired” or not March 20, 2022
 15

+ Activity - Example Z = w1.x1 + w2.x2 + w3.x3 + x4.w4

𝑊 . 𝑋 +0
+ 1.w0

𝑇
X1 X2 X3 X4 Y W = {w1, w2, w3,

𝑊
w4}
X = {x1, x2, x3,
x4}

• Inputs are taking row by

row
• W’s are randomly
• Based
assigned values the inpu
on proces t
summation s are
perform

• and the activation

decided
function whether the
neuron should take
1 + W0
action or not
 16
+ Activation Functions

The placement of the activation function

March 20, 2022

 17
+ Perceptron (3rd ver.)
 Single-Layer Perceptron:
 Consist of one layer of perceptron
 There is one layer for outputs along with a single input layer that
receives the inputs.

March 20, 2022

 18
+ Perceptron (3rd ver.)
Multilayer Perceptron (MLP):
 When hidden layers are added to a single-layer perceptron.
 An MLP is considered as a type of deep neural network

 The artificial neural networks

we build for everyday problems
are examples of MLP

March 20, 2022

 19

+ What is Neural Network?

Architecture of a Neural Network

Input layer is typically excluded when counting the number of layers in a Neural Network
March 20, 2022
 20

+
What is
Input Layer: Neural Network?
• The nodes of the input layer are passive. (i.e., not modify the data),
• They receive value on their input and duplicate it to their multiple outputs.
• All of the input variables are represented as input nodes; each value from the
input layer is duplicated and sent to all of the hidden nodes in first layer.
Hidden Layer(s):
• All of the input variables that came from the input layer are combined across
one or more nodes (summation / activation) perform in the hidden layer.
• This creates new features from the input data and then pass these features to
a new hidden layer and so on to get into the output layer
March 20, 2022
 21

+
What is Neural Network?
Output Layer
• Here we finally use an activation function that maps to the desired
output format

Hidden Layers
• Hidden neural network layers are set up in many different ways.
• In some cases, weighted inputs are randomly assigned.
• In other cases, they are fine-tuned and calibrated through a
process called backpropagation
March 20, 2022
 +
Activation Functions

March 20, 2022 22

 24

+ Activation function
Activation function decides, whether a neuron should be activated or
not by calculating weighted sum and further adding bias with it.

There are three types of activation functions:

 Binary step function
 Linear activation function
 Non-linear activation function

March 20, 2022

 25

+ Activation function
Binary Step Function:
• It is a threshold-based activation function.
• It check the output of neuron,
• if the neuron output is above the threshold,
• the activation neuron will send the same signal to the next layer
• otherwise, will keep inactive the neuron. (or stop to participate…)

Problem with Binary Step Function:

 We can’t use it in multiclassification problems. It’s just say YES or NO
 It’s
only a trigger function and can’t change any of the data that comes from the
previous layer, so we need a more complex activation function.

March 20, 2022

 26

+ Activation function
Linear Activation Function:
• It uses the function A = w ∗ x.
• It takes the inputs, multiplied by the weights for each neuron, and
creates an output signal proportional to the input.
• In one sense, a linear function is better than a step
function because it allows multiple outputs, not just YES and N O .

Problem with Binary Step Function:

 It’s a constant function, so we can’t use the backpropagation technique.
 It has limited power and ability to handle the complexity varying
of parameters of input data

March 20, 2022

 27

+ Activation function
Non-Linear Activation Function:
• Many neural network models nonlinear activation
use functions.
• They allow the model to create complex mappings between
the network’s inputs and outputs, which are essential for
learning and modeling complex data such as images, video,
audio, and data sets that are nonlinear or have high
dimensionality.
• The types of nonlinear activation functions that are used today
in neural network models are
• sigmoid/logistic • leaky ReLU
• tanh/hyperbolic tangent • softmax
• ReLU (rectified linear unit) • swish March 20, 2022
 28

+ Sigmoid Logistic Activation Function

It is a function which is plotted as ‘S’ shaped graph.

 Nature: Non-linear.

 Value Range: 0 to 1

 Uses: Usually used in output layer of a binary

classification, where result is either 0 or 1, as value for
sigmoid function lies between 0 and 1 only so, result
can be predicted easily to be 1 if value is greater
than 0.5 and 0 otherwise.

March 20, 2022

 29

+ Tanh Function
The activation that works almost always better than sigmoid function is Tanh
function also knows as Tangent Hyperbolic function.
 Value Range: -1 to +1

 Nature: Non-linear
 Uses: Usually used in hidden layers a neural
of
network as its values lies between -1 to 1.
 The advantage is that the negative inputs will be
mapped strongly negative, and the zero inputs will
be mapped near zero in the tanh graph.

 Both tanh and logistic sigmoid activation functions

are used in feed-forward nets.
March 20, 2022
 30

+ Rectified linear unit (ReLU)

 It is the most widely used activation function. Implemented in hidden layers of
Neural network.

 Value Range :- [0, inf)

 Nature : Non-Linear,
 Uses :- ReLU is less computationally expensive
than tanh and sigmoid because it involves
simpler mathematical operations.

 ReLU learns much faster than sigmoid

and
Tanh function.
It is used in almost all the convolutional neural networks or
deep learning.
March 20, 2022
 31

+Leaky ReLU
ReLU activation function is not taking care of the negative values and turn
them into zero in the graph, which in turn making the neuron to not
participate in the learning process of the neural network.

 Leaky ReLU is an attempt to solve the dying ReLU problem

 The leak helps to increase the range of the

ReLU function.

March 20, 2022

 32

+ Softmax Function
 The softmax function is same as sigmoid function

 Nature: Non-Linear

 Uses: Usually used to handle multi-classification problems.

 Output:

 It calculate the probabilities of each target class and divided by all the possible
target classes.
 The class with high probability is the target class.

March 20, 2022

 33

+ Swish Activation Function

 Google created this activation function called swish; it performs
better than ReLU with a similar level of computational efficiency.

March 20, 2022

 34

+ Choosing the Right Activation Function

 The basic rule of thumb is if you really don’t know what activation function
to use, then simply use ReLU as it is a general activation function and is
used in most cases these days.

 We can use Swish, Tanh as well in replace of ReLU.

 If your output is for binary classification then, sigmoid function is

very
natural choice for output layer.

 The softmax function is a more generalized logistic activation function

which is used for multiclass classification.

March 20, 2022

 +
Feedforward N N

March 20, 2022 35

 36

+ Feedforward Neural Network

 All nodes are fully connected
 Feed-forward NNs allow signals to travel one way only; from input to
output.
 There is no feedback (loops) i.e., the output of any layer does not affect
that same layer.
 Activation flows from input layer to output, without back loops
 They are extensively used in pattern recognition.

 In most cases this type of networks is trained using

Backpropagation method.

March 20, 2022

 37

+ Feedforward Neural Network

Shallow Feedforward Neural Network (Single-layer Perceptron)

 This is the simplest feedforward neural Network and does not contain any
hidden layer

 It only consists of a single layer of output nodes.

 We do not include the input layer, the reason

for that is because at the input layer no
computations is done, the inputs are fed
directly to the outputs via a series of weights.

March 20, 2022

 38

+ Feedforward Neural Network

Deep-Feedforward Neural Network (MLP - Multi-layer Perceptron)
 This class of networks consists of multiple layers of computational units, usually
interconnected in a feed-forward way.

 Each neuron in one layer has directed connections to the neurons of the
subsequent layer.
 In many applications the units of these networks
apply a sigmoid function as an activation
function.

 MLP are very more useful and one good reason is

that, they are able to learn non-linear
representations (most of the cases the data
presented to us is not linearly separable). March 20, 2022
 39

+ How are the weights initialized?

In general, initial weights are randomly chosen, with
typical values between -1.0 and 1.0 or -0.5 and 0.5.

 There are two types of NNs. The first type is known as

 Fixed Networks (feedforward NN) – where the weights
are fixed

 Adaptive Networks (Backward Propagation NN) –

where the weights are changed to reduce prediction error.

Network activation Forward Step

Error propagation Backward Step

March 20, 2022
 +
Back Propagation

March 20, 2022 40

 41

+
Backward propagation
 The goal with back propagation algorithm is to update each weight in the
network so that the actual output is closer to the target output, thereby
minimizing the error for each output neuron and the network as a whole.

 In Backward Propagation, we try to sequentially

update the weights, first by making a forward
pass on the network, after which we first update
the weights of the last layer, using the label and
last layer outputs, then subsequently use this
information recursively on the layer just before
and proceed.

March 20, 2022

 42

+ Backward propagation
 Partial derivatives, chain rules, and linear algebra are the main tools required to
deal with backpropagation
 Forbackpropagation to work, two basic assumptions are made regarding
the error function:
1. Total error can be written as a summation o f individual errors o f
training samples/minibatch,
2. Error can be written as a function of outputs of the network.

 Backpropagation consists of two parts:

 Forward pass, wherein we initialize the weights and make a
feedforward network to store all the values
 Backward pass, which is performed to have the stored values and update

the weights

March 20, 2022

 43

+
Bw-Prop: Training a Single Perceptron
 Let’s have dataset X1 X2 X3 X4 y

1 2 5 6 1

3 4 5 5 8

 If we want to apply multiple regression on this dataset.

Q1. How many columns (features) , we have =

March 20, 2022

 44

+ Bw-Prop: Training a Single Perceptron

 What Linear Regression model will do?
 It will identify a hyperplane, which pass through these points/data (4D hyper-plan)
and find the min-lost.

 What loss function we use in LR model?

 The plan with min-lost

 Mean-Square-Loss and we
are applying this to the LR-model
Single Perceptron Model
March 20, 2022
 45

+
Bw-Prop: Training a Single Perceptron
 Single Perceptron Model

March 20, 2022

 46

+ Bw-Prop: Training a Single Perceptron

What we can do within the LR model using Back-Propagation?

 Answer: we need to identify best values for w1, w2.

 But how to identify them
 STEPS
1. Initially we select some random values for weights w1, w2
2. After substituting values, we find and LOSS
3. Apply Stochastic Gradient Descent to update the weights

March 20, 2022

 47

+ Bw-Prop: Training a Single Perceptron

Single Perceptron Model

 For the Backward Propagation, we need to updates the

weights.

 First, we need to find the partial derivative of

LOSS function by applying CHAIN-RULE.

 Now calculate the new value for the weights

using stochastic gradient descent algorithm

March 20, 2022

 48

+
Bw-Prop: Training a Single Perceptron
 How long we need to update these values using SGD?
 Ideally, updates W’s till convergence (means no more updates in weights (w1, w2
and w0)

 How to process data by Backward Propagation?

 The model take 1 s t set of values for x1, x2 and update the weights
 Take the 2 n d set of values for x1, x2 and update the weights.
 ….
 This process continue until all records are processed and we called it 1 s t ep o c h …
followed by same process for 2 n d epoch and so on.

March 20, 2022

 49

+ Bw-Prop: Training a Single Perceptron

 What is Backward Propagation?
 We are taken input (record by record)
 pass through the model,
 perform activation function,
 calculate the and then LOSS function
and
 using SGD to update the weights.
 This is called Backward Propagation.

March 20, 2022

 50

+ Bw-Prop: Deep Model

March 20, 2022

 62

+ Applications of A N N
 Signal processing
 Pattern recognition, e.g. handwritten characters or face identification.
 Speech recognition
 Human Emotion Detection
 Sales forecasting
 Industrial process control
 Risk management
 Medical diagnosis
 Texture analysis

March 20, 2022

 63
+

Pros of A N N Cons of ANN

 It can model non-linear systems
 Long training Time
 The ability to learn allows the network to
adapt to changes in the surrounding  Hard to interpret
environment

 It can provide a confidence level to a given  Hard to implement: trail and error for
solution
choosing number of nodes, adjusting
 Can learn more complicated class weights
boundaries
 Can handle large number of features
 Have noise tolerance

March 20, 2022

 64
+

End of Lecture - 07

March 20, 2022