UNIT-IV

MACHINE LEARNING

Dr Soumya Ranjan Mishra

 Introduction to Neural Networks and McCulloch-Pitts Neuron

Introduction to Neural Networks

• Neural networks are computational models inspired by the structure and functioning of the
human brain.

• They consist of interconnected processing units called neurons, which work together to
solve complex problems.

• Neural networks are widely used in pattern recognition, machine learning, and
artificial intelligence applications.
Biological Inspiration

• The human brain consists of billions of neurons connected by synapses.

• Each neuron receives input signals, processes them, and transmits an output signal.

• Neural networks mimic this biological process in a simplified mathematical form.

Dendrite: Receives signals from other neurons

Soma: Processes the information

Axon: Transmits the output of this neuron

Synapse: Point of connection to other neurons

Our sense organs interact with the outer world and send the visual and sound
information to the neurons.

Let's say you are watching Friends. Now the information your brain receives is taken in
by the “laugh or not” set of neurons that will help you make a decision on whether to
laugh or not.
In reality, it is not just a couple of neurons which would do the decision making.

There is a massively parallel interconnected network of 10¹¹ neurons (100 billion) in our
brain and their connections are not as simple as I showed you above. It might look
something like this:
Artificial Neural Networks (ANNs)

An artificial neural network consists of:

Input Layer: Receives input signals (features).

Hidden Layers: Processes information using weighted connections.

Output Layer: Produces the final result.

Each connection in the network has an associated weight, which determines the strength of
the connection. The neuron applies an activation function (such as step function, sigmoid,
ReLU) to decide the output.
McCulloch-Pitts Neuron

The first computational model of a neuron was proposed by Warren MuCulloch

(neuroscientist) and Walter Pitts (logician) in 1943.

It may be divided into 2 parts.

The first part, g takes an input (ahem dendrite ahem),

performs an aggregation and based on the aggregated
value the second part, f makes a decision.
Lets suppose that I want to predict my own decision, whether to watch a random football
game or not on TV.

The inputs are all boolean i.e., {0,1} and my output variable is also boolean {0: Will watch
it, 1: Won’t watch it}.

So, x1 could be is PremierLeagueOn (I like Premier League more)

x2 could be isIt AFriendlyGame (I tend to care less about the friendlies)

x3 could be is NotHome (Can’t watch it when I’m running errands. Can I?)

x4 could be is ManUnitedPlaying (I am a big ManUnited fan) and so on.

These inputs can either be excitatory or inhibitory.

Inhibitory inputs are those that have maximum effect on the decision making
irrespective of other inputs i.e., if x3 is 1 (not home) then my output will always be 0
i.e., the neuron will never fire, so x3 is an inhibitory input.

Excitatory inputs are NOT the ones that will make the neuron fire on their own but
they might fire it when combined together. Formally, this is what is going on:
Structure of M-P Neuron

An M-P neuron consists of:

Inputs: x1,x2,…,xn

Weights: w1,w2,…,wn (Each input has an associated weight)

Summation Function: Computes weighted sum S

Threshold (θ): A fixed value that determines neuron activation

Activation Function: Step function

Mathematical Model
Example of M-P Neuron
Least Mean Squares (LMS) Algorithm in Neural Networks

The Least Mean Squares (LMS) algorithm is a supervised learning algorithm used for
training single-layer neural networks (also known as adaptive linear neurons, or ADALINE).

It is a type of gradient descent algorithm that minimizes the mean squared error between
the actual and predicted output.
Mathematical Formulation

The LMS algorithm updates the weight vector w to minimize the Mean Squared Error
(MSE):

Initialization:

Initialize weights w randomly or set them to small values.

Set a learning rate η (small positive value).

Compute Net Input:

This is based on gradient descent to minimize the error.

Repeat: Iterate until convergence (i.e., error is minimized below a threshold or after a
fixed number of iterations).
Repeat for remaining training samples

After multiple iterations, the weights stabilize, and the neural network converges to
a solution that minimizes the error.
Perceptron Model

• The perceptron is one of the simplest types of artificial neural networks used for binary
classification.

• It is a type of linear classifier that updates its weights based on the errors made during
training.

• The perceptron is inspired by biological neurons in the human brain, which receive
inputs, process them, and produce an output.

• It attempts to mimic the way neurons fire when the total input exceeds a certain
threshold.
Structure of Perceptron
A perceptron consists of:

Inputs (x1,x2,...,xn): Features of the dataset.

Weights (w1,w2,...,wn): Determines the importance of each input.

Bias (b): Helps shift the decision boundary.

Summation Function: Computes the weighted sum of inputs.

Activation Function: Applies a threshold to produce the final output

Multilayer Perceptron (MLP) and Hidden Layer Representation

A Multilayer Perceptron (MLP) is a type of artificial neural network (ANN) that consists of
multiple layers of neurons, enabling it to learn complex patterns and perform classification
and regression tasks.

MLP consists of three types of layers:

Input Layer: Receives the input features.

Hidden Layer(s): Performs computations and feature extraction.

Output Layer: Produces the final prediction or classification.

Each neuron in a layer is connected to all neurons in the next layer, making MLP a fully
connected network.
Multilayer Perceptron (MLP) and Hidden Layer Representation
Role of Hidden Layers in MLP

• The hidden layers help in learning hierarchical representations of data.

• Each neuron in a hidden layer applies a weighted sum followed by an activation

function to introduce non-linearity.

• More hidden layers allow the network to learn complex and abstract patterns, enabling
deep learning capabilities.

For a given input X = [x₁, x₂, ..., xₙ], the output of a hidden neuron is computed as:
Activation Functions in MLP

Sigmoid: Used in binary classification, but suffers from vanishing gradient issues.

Tanh: Similar to sigmoid but ranges from -1 to 1.

ReLU (Rectified Linear Unit): Popular for deep networks due to its efficiency.

Softmax: Used in the output layer for multi-class classification.

Backpropagation and Learning in MLP

• MLP is trained using backpropagation, which adjusts weights using gradient descent.

• The loss function (e.g., Mean Squared Error) helps in computing errors.

• The learning rate controls the step size for weight updates.
Advantages of MLP

✅ Can learn non-linear functions.

✅ Effective for both classification and regression problems.

✅ Works well with structured data.

Limitations of MLP

❌ Requires large amounts of data for training.

❌ Prone to overfitting if not regularized.

❌ Computationally expensive for deep networks.

Back propagation Algo
There are two units in the Input Layer, two units in the Hidden Layer and two units in the Output Layer.

The w1,w2,w2,…,w8 represent the respective weights. b1 and b2 are the biases for Hidden Layer and
Output Layer

we’ll be passing two inputs i1 and

i2, and perform a forward pass to
compute total error

Then a backward pass to

distribute the error inside the
network and update wei ghts
accordingly.
• Computation of weighted sum

• S qu a sh i n g o f t he we i ghte d su m us i n g a n
activation function.

• The result from the activation function becomes

an input to the next layer (until the next layer is
an Output Layer).

• In this example, we’ll be using the Sigmoid

function (Logistic function) as the activation
function. The Sigmoid function basically takes an
input and squashes the value between 0 and +1.
Neural Network Design and Forward Propagation Calculation

Design a feedforward neural network with:

Input Layer: 2 neurons (x₁ = 0.05, x₂ = 0.10)

Hidden Layer: 2 neurons (h₁, h₂)
Output Layer: 2 neurons (o₁, o₂)

Given Weights and Biases:

Weights from Input to Hidden Layer:

w₁ = 0.15, w₂ = 0.20 (Connected to h₁)
w₃ = 0.25, w₄ = 0.30 (Connected to h₂)

Bias for Hidden Layer: b₁ = 0.35

Weights from Hidden to Output Layer:

w₅ = 0.40, w₆ = 0.45 (Connected to o₁)
w₇ = 0.50, w₈ = 0.55 (Connected to o₂)

Bias for Output Layer: b₂ = 0.60

Tasks:

Calculate the net input and activation for each hidden neuron (h₁ and h₂) using the

sigmoid activation function:

Compute the net input and activation for the output neurons (o₁ and o₂) using the same
activation function.

Determine the final output values for o₁ and o₂ after forward propagation.
3.1 Compute Gradients for Output Layer Weights
Exploding Gradient Problem and Vanishing Gradient
Problem
Introduction to Convolutional Neural Networks (CNN)
What is a CNN?
A Convolutional Neural Network (CNN) is a deep learning architecture designed to process structured grid data,
such as images.

CNNs are widely used in image classification, object detection, and facial recognition due to their ability to
automatically extract meaningful features from images.
Components of CNN

Convolution Layer: Extracts features from the input image using filters (kernels).

Activation Function (ReLU): Introduces non-linearity to help learn complex patterns.

Pooling Layer (Max/Average Pooling): Reduces dimensionality while preserving

important features.

Fully Connected Layer: Makes final predictions using extracted features.

What is a Pooling Layer?

Similar to the Convolutional Layer, the Pooling layer is responsible for reducing the spatial
size of the Convolved Feature. This is to decrease the computational power required to
process the data by reducing the dimensions. There are two types of pooling average pooling
and max pooling.
ReLU (Rectified Linear Unit) Activation Function in CNN
The ReLU (Rectified Linear Unit) activation function is one of the most commonly used activation functions
in Convolutional Neural Networks (CNNs).

It introduces non-linearity into the model, helping CNNs learn complex patterns in images.