MODULE-4

Introduction to Probability based Learning

Definition:

Probability-based learning combines prior knowledge or prior probabilities with observed

data to make predictions about future events.

Role of Probability Theory:

Uses probability theory to model randomness, uncertainty, and noise in data.

Purpose:

It helps in predicting outcomes and learning from large datasets by using Bayes' rule to infer
unknown quantities.

Randomness vs. Determinism:

Probabilistic Model: Involves randomness and uses probability distributions to find

solutions.
Deterministic Model: Does not involve randomness and will yield the same result with
identical initial conditions.

Bayesian Learning:

A type of probabilistic learning that uses subjective probabilities, i.e., probabilities based on
individual belief and interpretation about the outcome, which can change over time.
Involves the inference of model parameters using subjective probabilities.

Bayesian Algorithms:

Naive Bayes Learning: A simple probabilistic classifier based on Bayes' theorem with strong
independence assumptions.
Bayesian Belief Network (BBN): A graphical model that represents a set of variables and
their conditional dependencies

Bayes' Rule: Forms the foundation of probabilistic learning and Bayesian learning
algorithms for inferring useful information.
Fundamentals of Bayes Theorem

Naive Bayes Model relies on Bayes theorem that works on the principle of three kinds of
probabilities called prior probability, likelihood probability, and posterior probability.

Prior Probability
It is the general probability of an uncertain event before an observation is seen or some
evidence is collected.
It is the initial probability that is believed before any new information is collected.

Likelihood Probability

Likelihood probability is the relative probability of the observation occurring for each class
or the sampling density for the evidence given the hypothesis.

It is stated as P (Evidence | Hypothesis), which denotes the likeliness of the occurrence of the
evidence given the parameters.

Posterior Probability

It is the updated or revised probability of an event taking into account the observations from the
training data.

P (Hypothesis | Evidence) is the posterior distribution representing the belief about the
hypothesis, given the evidence from the training data.

Therefore, Posterior probability = prior probability + new evidence.

Classification Using Bayes Model

Naive Bayes Classification is based on Bayes’ theorem, which calculates the posterior
probability using prior probabilities.
Bayes’ theorem determines the probability of a hypothesis (h) given evidence (E):

Posterior Probability is proportional to Prior Probability × Likelihood Probability.

 Bayes’ theorem helps calculate posterior probabilities for multiple hypotheses to select
the one with the highest probability, known as Maximum A Posteriori (MAP)
Hypothesis.

MAP Hypothesis (hMAP):

Maximum Likelihood (ML) Hypothesis (hML):

Bayes’ theorem ensures correctness by defining relationships between events within a sample
space, enabling reliable probabilistic predictions.
Naïve Bayes Algorithm
 Chapter 10
Artificial Neural Networks
The term "Artificial neural network" refers to a biologically inspired sub-field of artificial intelligence
modelled after the brain.
An Artificial neural network is usually a computational network based on biological neural networks
that construct the structure of the human brain.
Similar to a human brain has neurons interconnected to each other, artificial neural networks also have
neurons that are linked to each other in various layers of the networks. These neurons are known as
nodes.

The biological neuron consists of main four parts:

• dendrites: nerve fibres carrying electrical signals to the cell .
• cell body: computes a non-linear function of its inputs
• axon: single long fiber that carries the electrical signal from the cell body to other neurons
• synapse: the point of contact between the axon of one cell and the dendrite of another,
regulating a chemical connection whose strength affects the input to the cell.
Dendrites are tree like networks made of nerve fiber connected to the cell body.
An Axon is a single, long connection extending from the cell body and carrying signals from the
neuron. The end of axon splits into fine strands. It is found that each strand terminated into small
bulb like organs called as synapse. It is through synapse that the neuron introduces its signals to
other nearby neurons. The receiving ends of these synapses on the nearby neurons can be found
both on the dendrites and on the cell body. There are approximately 104 synapses per neuron in the
human body. Electric impulse is passed between synapse and dendrites. It is a chemical process
which results in increase/decrease in the electric potential inside the body of the receiving cell. If
the electric potential reaches a thresh hold value, receiving cell fires & pulse / action potential of
fixed strength and duration is send through the axon to synaptic junction of the cell. After that, cell
has to wait for a period called refractory period.

Difference between biological and Artificial Neuron

ARTIFICIAL NEURONS:
Artificial neurons are like biological neurons that are linked to each other in various layers of the
networks. These neurons are known as nodes.
A node or a neuron can receive one or more input information and process it. artificial neurons are
connected by connection links to another neuron. Each connection link is associated with a synaptic
weight. The structure of a single neuron is shown below:
Fig: McCulloch-Pitts Neuron Mathematical model.

Simple Model of an ANN

The first mathematical model of a biological neuron was designed by McCulloch-Pitts in 1943.
It includes 2 steps:
1. It receives weighted inputs from other neurons.
2. It operates with a threshold function or activation function.

Basically, a neuron takes an input signal (dendrite), processes it like the CPU (soma), passes
the output through a cable like structure to other connected neurons (axon to synapse to
other neuron’s dendrite).

OR
 Working:
The received input are computed as a weighted sum which is given to the activation function
and if the sum exceeds the threshold value the neuron gets [Link] neuron is the basic
processing unit that receives a set of inputs x1,x2,x3,….xn and their associated weights
w1,w2,w3,….wn. The summation function computes the weighted sum of the inputs
received by the neuron.
Sum=∑xiwi

Activation functions:
• To make work more efficient and for exact output, some force or activation is given. Like
that, activation function is applied over the net input to calculate the output of an ANN.
Information processing of processing element has two major parts: input and output. An
integration function (f) is associated with input of processing element.

• Several activation functions are there.

 Artificial Neural Network Structure
• Artificial Neural Networks Computational models inspired by the human brain: – Massively
parallel, distributed system, made up of simple processing units (neurons) – Synaptic
connection strengths among neurons are used to store the acquired knowledge.

• Knowledge is acquired by the network from its environment through a learning process.

• The Neural Network is constructed from 3 type of layers:

• Input layer — initial data for the neural network.
• Hidden layers — intermediate layer between input and output layer and place where all the
computation is done.

• Output layer — produce the result for given inputs.

 PERCEPTRON AND LEARNING THEORY
• The perceptron is also a simplified model of a biological neuron.
• The perceptron is an algorithm for supervised learning of binary classifiers. It is a type of
linear classifier, i.e. a classification algorithm that makes all of its predictions based on a
linear predictor function combining a set of weights with the feature vector.
• One type of ANN system is based on a unit called a perceptron.

OR

• The perceptron can represent all boolean primitive functions AND, OR, NAND , NOR.
• Some boolean functions can not be represented .
– E.g. the XOR function
Major components of a perceptron
• Input
• Weight
• Bias
• Weighted summation
• Step/activation function
• output
WORKING:
• Feed the features of the model that is required to be trained as input in the first layer. All
weights and inputs will be multiplied – the multiplied result of each weight and input will be
added [Link] Bias value will be added to shift the output function .This value will be
presented to the activation function (the type of activation function will depend on the need)
The value received after the last step is the output value.
The activation function is a binary step function which outputs a value 1, if f(x) is above the
threshold value Θ and a 0 if f(x) is below the threshold value Θ. Then the output of a neuron
is:
TYPES OF ANN
1. Feed Forward Neural Network
2. Fully connected Neural Network
3. Multilayer Perceptron
4. Feedback Neural Network

Feed Forward Neural Network:

Feed-Forward Neural Network is a single layer perceptron. A sequence of inputs enters the layer and are
multiplied by the weights in this model. The weighted input values are then summed together to form a total.
If the sum of the values is more than a predetermined threshold, which is normally set at zero, the output
value is usually 1, and if the sum is less than the threshold, the output value is usually -1.
The single-layer perceptron is a popular feed-forward neural network model that is frequently used for
classification.
The model may or may not contain hidden layer and there is no backpropagation.
Based on the number of hidden layers they are further classified into single-layered and multilayered feed
forward network.

Fully connected Neural Network:

 A fully connected neural network consists of a series of fully connected layers that connect
every neuron in one layer to every neuron in the other layer.

 The major advantage of fully connected networks is that they are “structure agnostic” i.e. there
are no special assumptions needed to be made about the input.
Multilayer Perceptron:
A multi-layer perceptron has one input layer and for each input, there is one neuron (or node), it has
one output layer with a single node for each output and it can have any number of hidden layers and
each hidden layer can have any number of nodes.
The information flows in both directions.
The weight adjustment training is done via backpropagation.
Every node in the multi-layer perception uses a sigmoid activation function. The sigmoid activation
function takes real values as input and converts them to numbers between 0 and 1 using the sigmoid
formula.

Feedback Neural Network:

Feedback networks also known as recurrent neural network or interactive neural network are
the deep learning models in which information flows in backward direction.
It allows feedback loops in the network. Feedback networks are dynamic in nature, powerful and
can get much complicated at some stage of execution
Neuronal connections can be made in any way.
RNNs may process input sequences of different lengths by using their internal state, which can
represent a form of memory.
They can therefore be used for applications like speech recognition or handwriting recognition.
Popular Applications of Artificial Neural Networks

Advantages and Disadvantages of ANN

Limitations of ANN
Challenges of ANN