MACHINE LEARNING

Artificial Neural Networks

INTRODUCTION
The human nervous system is an intricate network composed of billions of neurons that
coordinate perception, movement, and cognitive functions. It enables individuals to sense,
interpret, store, and respond to stimuli from their surroundings. Structurally, the nervous system
is divided into the Central Nervous System (CNS) which includes the brain and spinal cord
and the Peripheral Nervous System (PNS), consisting of all neurons located outside the CNS.
Neurons are broadly classified into three types:

I. Sensory neurons: - which transmit information to the CNS

II. Motor neurons: - which convey signals from the CNS to various body parts
III. Interneurons: - which facilitate communication between neurons within the CNS.

The fundamental role of a neuron is to receive, process, and transmit information throughout
the body.

BIOLOGICAL NEURONS
A typical biological neuron consists of four main components: dendrites, soma (cell body),
axon, and synapse. The dendrites receive input signals from other neurons and transmit
them to the soma, where the information is processed. Each neuron can connect to thousands
of other neurons through its axon, allowing the transmission of processed signals. When the
received input exceeds a certain threshold, the neuron becomes activated and transmits
signals to other neurons through synapses in the form of electrical impulses, known as
spikes. A neuron may receive signals from one or multiple neurons, and collectively, these
interconnected neurons form complex networks that process information and produce
responses. The basic structure of a biological neuron is illustrated in Figure 10.1.

1|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

ARTIFICIAL NEURONS
Artificial neurons, also referred to as nodes, are modelled after biological neurons. Each node
can receive one or more input signals and process them to produce an output. These nodes are
connected through links, and each connection is associated with a synaptic weight that
represents the strength or importance of the signal transmitted between neurons. The structure
of a single neuron is shown in Figure 10.2.

Figure 10.2: An Artificial Neuron

Simple Model of an Artificial Neuron

The first mathematical model of a biological neuron was proposed by Warren McCulloch and
Walter Pitts in 1943. This model operates in two main steps:
1. It receives inputs from other neurons, each associated with a weight.
2. It processes these inputs using a threshold (activation) function.
The inputs are combined as a weighted sum and passed to the activation function; if this sum
exceeds a specified threshold, the neuron is activated (fires). The mathematical model of a
neuron is shown in Figure 10.3.
The neuron is the basic processing unit that receives a set of input 𝑥1 , 𝑥2 , … . 𝑥𝑛 and their
associated weights 𝑤1 , 𝑤2 , … 𝑤𝑛 The Summation function Net-sum' Eq. (10.1) computes the
weighted sum of the inputs received by the neuron.
𝑛

𝑁𝑒𝑡 − 𝑠𝑢𝑚 = ∑ 𝑥𝑖 𝑤𝑖
𝑖=1

The activation function is a binary step function that produces an output of 1 when the net input
exceeds the threshold value (∅ᵢ), and 0 when it falls below the threshold. Thus, the activation
function is applied to the net sum to determine whether the neuron is activated, as represented
in Eq. (10.2).

𝑓(𝑥) = 𝐴𝑐𝑡𝑖𝑣𝑎𝑡𝑒 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 (𝑁𝑒𝑡 − 𝑠𝑢𝑚)

2|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

Then, output of a neuron

1 𝑖𝑓 𝑓(𝑥) ≥ ∅
𝑌={
0 𝑖𝑓 𝑓(𝑥) < ∅

Figure 10.3: McCulloch & Pitts Neuron Mathematical Model

The McCulloch–Pitts neuron model can represent only simple Boolean functions, where inputs
and outputs are binary. For instance, an AND function produces an output only when all inputs
are 1, while an OR function produces an output if at least one input is 1. In this model, both the
weights and threshold values are fixed, limiting its flexibility

Artificial Neural Network Structure

An Artificial Neural Network (ANN) is inspired by the human brain and exhibits intelligent
behaviour through a network structure represented as a directed graph. It consists of
interconnected neuron nodes arranged in layers: input, hidden, and output layers.

Figure 10.4: Artificial Neural Network Structure

3|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

The input layer receives input values (𝑥1 , 𝑥2 , … , 𝑥𝑛 ) and forwards them to the hidden layer
through weighted connections (synaptic weights). Each neuron processes these inputs by
computing a weighted sum, adding a bias, and passing the result through an activation
function. If the computed value exceeds a certain threshold, the neuron becomes activated.
The activation function introduces non-linearity by transforming the weighted sum into an
output value. Finally, the output layer produces the result, typically as a single output value.

Activation Functions
Activation functions are mathematical functions used in neural networks to convert input
signals into output signals. They determine whether a neuron should activate based on the
received inputs. These functions also normalize the output, typically within ranges such as (0,
1) or (−1, 1).

Activation functions can be either linear or non-linear. Linear functions are generally applied
in simple models like binary perceptrons, where data is classified into two groups. In
contrast, non-linear functions are continuous and are essential for handling complex, high-
dimensional data such as audio, images, and video. Common activation functions used in
ANNs include both linear and non-linear types.

1. Identity Function or Linear Function

𝑓(𝑥) = 𝑥 ∀𝑥

The value of 𝑓(𝑥) increases linearly or proportionally with the value of x. This function is
useful when we do not want to apply any threshold. The output would be just the weighted sum
of input values. The output value ranges between − ∞ 𝑎𝑛𝑑 + ∞

2. Binary Step Function

1 𝑖𝑓 𝑓(𝑥) ≥ ∅
𝑓(𝑥) = {
0 𝑖𝑓 𝑓(𝑥) < ∅

The output value is binary, i.e., 0 or 1 based on the threshold value ∅. If value of 𝑓(𝑥) is greater
than or equal to ∅, it outputs 1 or else it outputs 0.

3. Bipolar Step Function

1 𝑖𝑓 𝑓(𝑥) ≥ ∅
𝑓(𝑥) = {
−1 𝑖𝑓 𝑓(𝑥) < ∅

The output value is bipolar, i.e., +1 or -1 based on the threshold value ∅. If value of 𝑓(𝑥) is
greater than or equal to ∅, it outputs +1 or else it outputs -1.

4. Sigmoidal Function or Logistic Function

1
𝜎𝑥 =
1 + 𝑒𝑥

4|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

It is a commonly used non-linear activation function that produces an S-shaped curve, with
output values ranging between 0 and 1. However, it suffers from the vanishing gradient
problem, where extremely low or high input values result in minimal changes in output,
limiting learning effectiveness.

5. Bipolar Sigmoid Function

1 − 𝑒𝑥
𝜎𝑥 =
1 + 𝑒𝑥

It outputs values between -1 and +1.

6. Ramp Functions
1 𝑖𝑓 𝑥 > 1
𝑓(𝑥) = { 𝑥 𝑖𝑓 0 ≤ 𝑥 ≤ 1
0 𝑖𝑓 𝑥 < 0

It is a linear function whose upper and lower limits are fixed.

7. Tanh - Hyperbolic Tangent Function

The Tanh function is a non-linear activation function and a scaled form of the sigmoid function.
It produces output values in the range of −1 to 1 and, like the sigmoid function, it is affected
by the vanishing gradient problem.
2
tan ℎ(𝑥) = −1
1 + 𝑒 −2𝑥

8. ReLu - Rectified Linear Unit Function

This activation function is widely used in deep learning models, especially in hidden layers, as
it helps reduce the vanishing gradient problem. It outputs 0 for negative input values and
behaves like a linear function for positive inputs.
𝑥 𝑖𝑓 𝑥 ≥ 0
f(𝑥) = max(0, 𝑥) { 0 𝑖𝑓 𝑥 < 0

9. SoftMax Function

This is a non-linear activation function used in the output layer for multi-class classification.
It computes the probability of each class, with outputs ranging between 0 and 1. The probability
for a class is obtained by dividing the exponential of its input value by the sum of exponentials
of all input values.
𝑒 𝑥𝑖
𝑠(𝑥𝑖 ) = 𝑤ℎ𝑒𝑟𝑒 𝑖 = 0, … 𝑘
∑𝑘𝑗=0 𝑒 𝑥𝑖

5|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

PERCEPTRON AND LEARNING THEORY

The first neural network model, known as the Perceptron, was developed by Frank Rosenblatt
in 1958 as a linear binary classifier for supervised learning. It extends the Warren
McCulloch–Walter Pitts neuron model by incorporating the Hebbian Learning principle for
adjusting weights.
The perceptron introduces variable weights and a bias input, allowing neurons to learn
patterns from data. Rosenblatt also proposed a supervised learning algorithm through which
the perceptron automatically updates its weights and thresholds based on training examples to
achieve accurate classification. The perceptron model (illustrated in Figure 10.5) operates
through the following four steps.
1. Inputs from other neurons
2. Weights and bias
3. Net sum
4. Activation function

Figure 10.5: Perceptron Model

Thus, the modified neuron model receives a set of 𝑖𝑛𝑝𝑢𝑡𝑠 𝑥1 , 𝑥2 . . . , 𝑥𝑛 , their associated
𝑤𝑒𝑖𝑔ℎ𝑡𝑠 𝑤1 , 𝑤2 . . . , 𝑤𝑛 and a bias. The summation function 'Net-sum' Eq. (10.13) computes the
weighted sum of the inputs received by the neuron.
𝑛

𝑁𝑒𝑡 − 𝑠𝑢𝑚 = ∑ 𝑥𝑖 𝑤𝑖
𝑖=1

After computing the Net-sum', bias value is added to it and inserted in the activation function
as shown below:

𝑓(𝑥) = 𝐴𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 (𝑁𝑒𝑡 − 𝑠𝑢𝑚 + 𝑏𝑖𝑎𝑠)

The activation function is a binary step function which outputs a value 1 if f(x) is above the
threshold value ∅, and 𝑎 0 𝑖𝑓 𝑓(𝑥) is below the threshold value ∅. Then, output of a neuron:
1 𝑖𝑓 𝑓(𝑥) ≥ ∅
𝑌={
0 𝑖𝑓 𝑓(𝑥) < ∅

Before learning how a neural network works, let us learn about how a perceptron model works.
6|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

Algorithm: Perceptron Algorithm

Set initial 𝑤𝑒𝑖𝑔ℎ𝑡𝑠 𝑤₁, 𝑤2 , . . . , 𝑤𝑛 and bias ∅ to a random value in the range [−0.5, 0.5]. For
each Epoch,

1. Compute the weighted sum by multiplying the inputs with the weights and add the
products.
2. Apply the activation function on the weighted sum:
𝑌 = 𝑆𝑡𝑒𝑝 ((𝑥1 𝑤1 + 𝑥 2 𝑤2 ) − ∅)
3. If the sum is above the threshold value, output the value as positive else output the
value as negative.
4. Calculate the error by subtracting the estimated output 𝑌𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 from the desired
𝑜𝑢𝑡𝑝𝑢𝑡 𝑌𝑑𝑒𝑠𝑖𝑟𝑒𝑑 .
𝑒𝑟𝑟𝑜𝑟 𝑒(𝑡) = 𝑌𝑑𝑒𝑠𝑖𝑟𝑒𝑑 − 𝑌𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑
[𝐼𝑓 𝑒𝑟𝑟𝑜𝑟 𝑒(𝑡) 𝑖𝑠 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒, 𝑖𝑛𝑐𝑟𝑒𝑎𝑠𝑒 𝑡ℎ𝑒 𝑝𝑒𝑟𝑐𝑒𝑝𝑡𝑟𝑜𝑛 𝑜𝑢𝑡𝑝𝑢𝑡 𝑌 𝑎𝑛𝑑 𝑖𝑓 𝑖𝑡 𝑖𝑠 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒,
𝑑𝑒𝑐𝑟𝑒𝑎𝑠𝑒 𝑡ℎ𝑒 𝑝𝑒𝑟𝑐𝑒𝑝𝑡𝑟𝑜𝑛 𝑜𝑢𝑡𝑝𝑢𝑡 𝑌. ]
5. Update the weights if there is an error:
∆𝑤 = ∝× 𝑒(𝑡) × 𝑥𝑖
𝑤𝑖 = 𝑤𝑖 + ∆𝑤𝑖

𝑤ℎ𝑒𝑟𝑒, 𝑥𝑖 is the input value, 𝑒(𝑡) is the error at step t, ∝ is the learning rate and ∆𝑤𝑖 , is the
difference in weight that has to be added to 𝑤𝑖 .

Example 10.1: Consider a perceptron to represent the Boolean function AND with the initial
weights 𝑤1 = 0.3, 𝑤2 = −0.2, learning rate ∝ = 0.2 and bias ∅ = 0.4 as shown in Figure
10.6. The activation function used here is the Step 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑓(𝑥) which gives the output value
as binary, i.e., 0 𝑜𝑟 1. If value of 𝑓(𝑥) is greater than or equal to 0, it outputs 1 or else it outputs
0. Design a perceptron that performs the Boolean function AND and update the weights until
the Boolean function gives the desired output.

Figure 10.6: Perceptron for Boolean Function AND

7|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

Solution:
Desired output for Boolean function AND is shown in Table 10.1.
Table 10.1: AND Truth Table

𝒙𝟏 𝒙𝟐 𝒀𝒅𝒆𝒔
0 0 0
0 1 0
1 0 0
1 1 1
For each Epoch, weighted sum is calculated and the activation function is applied to compute
the estimated output 𝑌𝑒𝑠𝑡 . Then, 𝑌𝑒𝑠𝑡 is compared with 𝑌𝑑𝑒𝑠 to find the error. If there is an error,
the weights are updated. Tables 10.2 to 10.5 show how the weights are updated in the four
Epochs.

Table 10.2: Epoch 1

𝑬𝒑𝒐𝒄𝒉 𝒙𝟏 𝒙𝟐 𝒀𝒅𝒆𝒔 𝒀𝒆𝒔𝒕 𝒆𝒓𝒓𝒐𝒓 𝒘𝟏 𝒘𝟐 𝑺𝒕𝒂𝒕𝒖𝒔

1 0 0 0 Step ((0 x 0.3+0 x-0.2) 0 0.3 -0.2 No
-0.4) =0 change
0 1 0 𝑆𝑡ep ((0 x 0.3 +1 x- 0 0.3 -0.2 No
0.2)-0.4) = 0 change
1 0 0 𝑆tep((1 x 0.3+0 x-0.2)- 0 0.3 -0.2 No
0.4)=0 change
1 1 1 Step ((1 x 0.3+1 x-0.2) 1 0.5 0 change
-0.4) = 0

For input (1, 1) the weights are updated as follows:

∆𝑤1 = ∝ × 𝑒(𝑡) × 𝑥1 , = 0.2 × 1 × 1 = 0.2

𝑤₁ = 𝑤₁ + ∆𝑤₁ = 0.3 + ∆𝑤₁ = 0.3 + 0.2 = 0.5

∆𝑤2 = ∝ × 𝑒(𝑡) × 𝑥2 = 0.2 𝑥1 𝑥 1 = 0.2

𝑤2 = 𝑤2 + ∆𝑤2 = −0.2 + ∆𝑤₂ = −0.2 + 0.2 = 0

Table 10.3: Epoch 2

𝑬𝒑𝒐𝒄𝒉 𝒙𝟏 𝒙𝟐 𝒀𝒅𝒆𝒔 𝒀𝒆𝒔𝒕 𝒆𝒓𝒓𝒐𝒓 𝒘𝟏 𝒘𝟐 𝑺𝒕𝒂𝒕𝒖𝒔

2 0 0 0 Step ((0 x 0.5+ 0 x 0) - 0 0.5 0 No
0.4)=0 change
0 1 0 𝑆𝑡ep ((0 x 0.5+1 x 0)- 0 0.5 0 No
0.4) = 0 change
1 0 0 𝑆tep((1 x 0.5+0 x 0) - -1 0.3 0 change
0.4) = 1
1 1 1 Step((1 x 0.3 +1 x 0) - 1 0.5 0.2 change
0.4) = 0

8|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

For input (1, 0) the weights are updated as follows:

∆𝑤1 = ∝ × 𝑒(𝑡) × 𝑥1 , = 0.2 × − 1 × 1 = −0.2

𝑤1 = 𝑤1 + ∆𝑤1 = 0.5 + ∆𝑤1 = 0.5 − 0.2 = 0.3

∆𝑤2 = ∝ × 𝑒(𝑡) × 𝑥2 = 0.2 × −1 × 0 = 0

𝑤2 = 𝑤2 + ∆𝑤2 = 0 + ∆𝑤 2 = 0 + 0.2 = 0.2

For input (1, 1), the weights are updated as follows:

∆𝑤1 = ∝ × 𝑒(𝑡) × 𝑥1 , = 0.2 × 1 × 1 = 0.2

𝑤1 = 𝑤1 + ∆𝑤1 = 0.3 + ∆𝑤1 = 0.5 + 0.2 = 0.5

∆𝑤2 = ∝ × 𝑒(𝑡) × 𝑥2 = 0.2 × 1 × 1 = 0.2

𝑤2 = 𝑤2 + ∆𝑤2 = 0 + ∆𝑤2 = 0 + 0.2 = 0.2

Table 10.4: Epoch 3

𝑬𝒑𝒐𝒄𝒉 𝒙𝟏 𝒙𝟐 𝒀𝒅𝒆𝒔 𝒀𝒆𝒔𝒕 𝒆𝒓𝒓𝒐𝒓 𝒘𝟏 𝒘𝟐 𝑺𝒕𝒂𝒕𝒖𝒔

3 0 0 0 Step ((0 x 0.5 + 0 x 0.2) -0.4)=0 0 0.5 0.2 No change

0 1 0 Step ((0x 0.5+1 x 0.2)−0.4) =0 0 0.5 0.2 No change
1 0 0 Step ((1 x 0.5+0 x 0.2) −0.4) =1 -1 0.3 0.2 Change
1 1 1 Step ((1 x 0.3 + 1 x 0.2) −0.4) 0 0.3 0.2 No change
=1
For input (1, 0) the weights are updated as follows:
∆𝑤1 = ∝ × 𝑒(𝑡) × 𝑥1 , = 0.2 × − 1 × 1 = −0.2

𝑤1 = 𝑤1 + ∆𝑤1 = 0.5 + ∆𝑤1 = 0.5 − 0.2 = 0.3

∆𝑤2 = ∝ × 𝑒(𝑡) × 𝑥2 = 0.2 × −1 × 0 = 0

𝑤2 = 𝑤2 + ∆𝑤2 = 0 + ∆𝑤2 = 0 + 0.2 = 0.2

Table 10.5: Epoch 4

𝑬𝒑𝒐𝒄𝒉 𝒙𝟏 𝒙𝟐 𝒀𝒅𝒆𝒔 𝒀𝒆𝒔𝒕 𝒆𝒓𝒓𝒐𝒓 𝒘𝟏 𝒘𝟐 𝑺𝒕𝒂𝒕𝒖𝒔

4 0 0 0 Step ((0 x 0.3 + 0 x 0.2) -0.4) =0 0 0.3 0.2 No change
0 1 0 Step ((0 x 0.3 +1 x0.2) -0.4) =0 0 0.3 0.2 No change
1 0 0 Step ((1 x 0.3 +0 x 0.2) -0.4) =0 0 0.3 0.2 No change
1 1 1 Step ((1 x 0.3 +1 x 0.2) -0.4) = 0 0.3 0.2 No change
1
It is observed that with 4 Epochs, the perceptron learns and the weights are updated to 0.3 and
0.2 with which the perceptron gives the desired output of a Boolean AND function.

9|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

XOR Problem
A perceptron model can solve all Boolean functions which are linearly separable. However, the
XOR problem was identified in 1969 by Minsky and Papert. An XOR function returns a 1, if
the two inputs are not equal and a 0 if they are equal. Following is the truth table of an XOR
function shown in Table 10.6.

Table 10.6: XOR Truth Table

𝒙𝟏 𝒙𝟐 𝒀
0 0 1
0 1 0
1 0 0
1 1 1

Since the XOR problem is not linearly separable, a single-layer perceptron fails to classify it,
leading to the development of the Multi-Layer Perceptron (MLP). Early MLPs were limited
due to the absence of an effective learning algorithm. In 1974, Paul Werbos introduced the
concept of backpropagation for a three-layer network, and in 1986, David Rumelhart and James
McClelland generalized it for multi-layer networks.

This advancement led to the success of Artificial Neural Networks and Deep Neural Networks
in solving complex problems. The MLP, in particular, can handle non-linearly separable
problems effectively.

Delta Learning Rule and Gradient Descent

Learning in neural networks is achieved by adjusting the weights to minimize the difference
between the desired output and the predicted output. This difference, known as the error or cost
function, is typically continuous and differentiable.

This learning approach, called the delta rule (also known as the Widrow-Hoff Rule), is a form
of backpropagation used for training. The training error is commonly defined as half of the
squared difference between the target output and the actual output.
1 (𝑂 − 𝑂𝐸𝑠𝑡𝑜𝑚𝑎𝑡𝑒𝑑 )2
𝑇𝑟𝑎𝑖𝑛𝑖𝑛𝑔 𝐸𝑟𝑟𝑜𝑟 = ∑ 𝐷𝑒𝑠𝑖𝑟𝑒𝑑
2
𝑑𝑒𝑡

𝑤ℎ𝑒𝑟𝑒, 𝑇 is the training dataset, 𝑂𝐷𝑒𝑠𝑖𝑟𝑒𝑑 and 𝑂Estimated are the desired target output and
estimated actual output, respectively, for a training instance d.

Gradient descent is an optimization technique used to minimize the cost function by iteratively
moving toward a local minimum in the direction opposite to the gradient. The step size at each
iteration is controlled by the learning rate and the slope of the gradient.

10 | P a g e
Dept. of CSE, RNSIT
 MACHINE LEARNING

This method forms the basis of the backpropagation algorithm used in Multi-Layer Perceptrons
(MLPs). Before exploring MLPs in detail, it is important to understand the various types of
neural networks, which differ in structure, activation functions, and learning mechanisms.

TYPES OF ARTIFICIAL NEURAL NETWORKS

Artificial Neural Networks (ANNs) consist of multiple interconnected neurons arranged in
layers. Their design may vary depending on the network architecture, activation functions,
and learning rules used. A typical ANN is composed of three types of layers: an input layer,
one or more hidden layers, and an output layer. In general, an ANN contains one input
layer, one output layer, and zero or more hidden layers, where the hidden layers help in
learning complex patterns from the input data.

Feed Forward Neural Network

This is the simplest form of a neural network, where neurons are arranged in layers and
information flows only in the forward direction. It may include a hidden layer or may consist
only of input and output layers, and it does not involve backpropagation. Based on the number
of hidden layers, it is classified as either a single-layer or multi-layer feedforward network.
These networks are easy to design, fast in operation, and simple to maintain; however, they
have limited capability in handling complex problems. They are mainly used for basic
classification tasks and simple image processing applications.

The model of a Feed Forward Neural Network is shown in Figure 10.7.

Figure 10.7: Model of a Feed Forward Neural Network

Fully Connected Neural Network

Fully connected neural networks are those in which every neuron in one layer is connected to
every neuron in the subsequent layer through weighted connections. The model of a fully
connected neural network is shown in Figure 10.8.

11 | P a g e
Dept. of CSE, RNSIT
 MACHINE LEARNING

Figure 10.8: Model of a Fully Connected Neural Network

Multi-Layer Perceptron (MLP)

This type of ANN consists of multiple layers, including one input layer, one output layer, and
one or more hidden layers. Each neuron in a layer is fully connected to all neurons in the next
layer. Information flows forward, where inputs are multiplied by weights, passed through
activation functions, and propagated to subsequent layers. If the output is incorrect, the error is
propagated backward to adjust weights and biases, enabling the network to learn from training
data.

Figure 10.9: Model of a Multi-Layer Perceptron

12 | P a g e
Dept. of CSE, RNSIT
 MACHINE LEARNING

These networks are commonly used in deep learning for tasks such as classification, speech
recognition, medical diagnosis, and forecasting. However, they are more complex and
computationally slower than simpler models. The MLP model is illustrated in Figure 10.9.

Feedback Neural Network

Feedback neural networks contain feedback connections that enable information to flow in
both forward and backward directions. The output of the network can be fed back to neurons
in the same layer or to neurons in previous layers, making the network more dynamic and
adaptive during training. The structure of a feedback neural network is illustrated in Figure
10.10.

Figure 10.10: Model of a Feedback Neural Network

LEARNING IN A MULTI-LAYER PERCEPTRON

A Multi-Layer Perceptron (MLP) is a type of feedforward neural network that consists of
multiple layers of neurons. In this network, each neuron in one layer is fully connected to the
neurons in the next layer. Typically, an MLP includes three types of layers: an input layer,
one or more hidden layers, and an output layer.

𝑥1 : (1, 0, 1, 0) → 𝑈𝑛𝑖𝑡 1

𝑥₂; (1, 0, 0, 0) → 𝑈𝑛𝑖𝑡 1

𝑥3 : (1, 1, 1,1) → 𝑈𝑛𝑖𝑡2

𝑥4 : (0, 1, 1, 0) → 𝑈𝑛𝑖𝑡 2
This process is continued for many epochs until the feature map does not change.

13 | P a g e
Dept. of CSE, RNSIT
 MACHINE LEARNING

POPULAR APPLICATIONS OF ARTIFICIAL NEURAL

NETWORKS
Artificial Neural Network (ANN) learning mechanisms are widely used to model complex,
non-linear processes. They are capable of handling noisy and incomplete data, making them
effective for pattern recognition and prediction tasks across various domains. Key applications
include:

1. Real-time applications: Face recognition, emotion detection, self-driving cars,

navigation, routing, and target tracking.

2. Business applications: Stock trading, sales forecasting, customer behavior analysis,

and market research.

3. Banking and finance: Credit and loan prediction, fraud detection, risk assessment,
currency forecasting, and real-estate valuation.

4. Education: Adaptive learning systems and student performance analysis.

5. Healthcare: Medical diagnosis, image analysis, pattern recognition, and drug

discovery.

6. Engineering applications: Robotics, aerospace, electronics, manufacturing,

communication systems, chemical analysis, and food research.

ADVANTAGES AND DISADVANTAGES OF ANN

Advantages of ANN

1. Capable of solving complex problems involving non-linear relationships.

2. Able to learn and recognize intricate patterns, similar to human problem-solving.

3. Supports parallel processing, enabling faster predictions.

4. Handles incomplete, noisy, or insufficient data effectively.

5. Scales well to large datasets and often outperforms traditional learning methods.

Limitations of ANN

1. Requires high computational power and parallel processing, especially for large
networks and multiple training epochs.

2. Functions as a “black box,” making it difficult to interpret internal operations and

learned representations.

3. Model development is complex and time-consuming.

4. Requires large amounts of data and performs poorly on small datasets.
5. Computationally more expensive compared to traditional machine learning techniques.

14 | P a g e
Dept. of CSE, RNSIT
 MACHINE LEARNING

CHALLENGES OF ARTIFICIAL NEURAL NETWORKS

Challenges in Modelling Real-Time Applications with ANNs
1. Training complexity: Training a neural network is challenging due to issues like
overfitting and underfitting, especially when the training data is inadequate or not
representative. Models trained on simulated data may not generalize well to real-world
scenarios. Additionally, large volumes of data are required to build robust models for
real-time applications.

2. Parameter optimization: Determining optimal weights and biases is difficult, making

it challenging to achieve an efficient and accurate model.

15 | P a g e
Dept. of CSE, RNSIT
 MACHINE LEARNING

Bayesian Learning
Introduction to Probability-Based Learning
Probability-based learning is an important practical learning approach that combines prior
knowledge (prior probabilities) with observed data. It uses principles of probability theory to
model uncertainty, randomness, and noise, enabling the prediction of future events and inference
of unknown values. This approach is widely used for handling large datasets and applies Bayes’
rule to learn from data and make predictions.
In a probabilistic model, randomness plays a key role, resulting in outcomes that follow a
probability distribution. In contrast, a deterministic model does not involve randomness; it
produces the same output for the same input every time, leading to a single definite outcome.
Bayesian Learning
Bayesian learning differs from general probabilistic learning by incorporating subjective
probabilities, which are based on an individual’s belief or interpretation of an event. These
probabilities are not fixed and can be updated over time as new information becomes available.
Bayesian learning uses prior probabilities along with observed data and applies Bayes’ rule to
estimate the parameters of a model. Two widely used algorithms in this approach are Naïve Bayes
and Bayesian Belief Networks (BBN). These methods utilize prior knowledge and update it
systematically to derive meaningful insights from data.
Fundamentals of Bayes Theorem
The Naïve Bayes model is built on Bayes’ theorem, which is based on three fundamental types of
probabilities:
✓ Prior probability
✓ Likelihood probability
✓ Posterior probability
Prior Probability
Prior probability represents the probability of an event before any evidence or observation is taken
into account. It reflects the initial belief about an event based on existing knowledge.
Likelihood Probability
Likelihood probability refers to the probability of observing the given data under a specific
hypothesis. It is expressed as:
𝑃(𝐸𝑣𝑖𝑑𝑒𝑛𝑐𝑒 ∣ 𝐻𝑦𝑝𝑜𝑡ℎ𝑒𝑠𝑖𝑠)
This indicates how likely the observed evidence is when the hypothesis is assumed to be true.

1|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

Posterior Probability
Posterior probability is the revised probability of a hypothesis after considering new evidence or
data. It is expressed as:
𝑃(𝐻𝑦𝑝𝑜𝑡ℎ𝑒𝑠𝑖𝑠 ∣ 𝐸𝑣𝑖𝑑𝑒𝑛𝑐𝑒)
It represents the updated belief about the hypothesis after incorporating the observed data.
Bayes’ theorem provides a systematic way to update prior beliefs using new evidence. Thus, the
posterior probability is obtained by combining prior knowledge with the likelihood of the observed
data.
Classification Using Bayes Model
Naïve Bayes classification is based on Bayes' Theorem, which provides a way to compute the
probability of a class given observed data. It determines the most likely hypothesis by combining
prior knowledge with new evidence.
The posterior probability 𝑃( ℎ ∣ 𝐸 ) is computed using the prior probability 𝑃(ℎ), the likelihood
𝑃( 𝐸 ∣ ℎ ),and the marginal probability 𝑃(𝐸) It is expressed as:
𝑃( 𝐸 ∣ ℎ )𝑃(ℎ)
𝑃( ℎ ∣ 𝐸 ) = 8.1
𝑃(𝐸)
P(h) represents the prior probability of hypothesis h, which is the initial belief about the
hypothesis before considering any training data or evidence.
𝑃(𝐸) denotes the marginal probability of the evidence E, calculated without knowing which
hypothesis is true.

𝑃(𝐸 ∣ ℎ) is the likelihood, i.e., the probability of observing evidence E assuming that hypothesis
h is true.
𝑃(ℎ ∣ 𝐸)is the posterior probability, which indicates how likely the hypothesis h is after
observing the evidence E.
From Bayes’ theorem, it can be understood that:
Posterior Probability ∝ Prior Probability × Likelihood
Bayes’ theorem enables the computation of posterior probabilities for multiple hypotheses, and
the hypothesis with the highest probability is selected. This selection process is known as the
Maximum A Posteriori (MAP) hypothesis.

2|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

Maximum A Posteriori (MAP) Hypothesis

Given a set of possible hypotheses, the one with the highest posterior probability is selected as
the most probable hypothesis. This is known as the Maximum A Posteriori (MAP) hypothesis,
denoted as ℎ𝑀𝐴𝑃
Using Bayes’ theorem, it is defined as:
ℎ𝑀𝐴𝑃 = arg ℎ 𝑚𝑎𝑥 𝑃( ℎ ∣ 𝐸 ) = arg ℎ 𝑚𝑎𝑥 𝑃(𝐸|ℎ)𝑃(ℎ) 8.2
Thus, the MAP hypothesis considers both the prior probability and the likelihood to determine
the most suitable hypothesis.
Maximum Likelihood (ML) Hypothesis
When all hypotheses are assumed to be equally likely, the prior probabilities become constant. In
this case, only the likelihood is considered.
The hypothesis that maximizes the likelihood is called the Maximum Likelihood (ML)
hypothesis, denoted as ℎ𝑀𝐿 :
arg ℎ 𝑚𝑎𝑥 𝑃( 𝐸 ∣ ℎ ) 8.3
Thus, ML focuses only on how well the hypothesis explains the observed data, ignoring prior
beliefs.
MAP → Considers both prior probability and likelihood
ML → Considers only likelihood (assumes equal priors)
Correctness of Bayes Theorem
To verify Bayes’ theorem, consider two events AAA and BBB in a sample space SSS. By
evaluating their probabilities and conditional relationships, Bayes’ theorem establishes a
consistent method to compute posterior probabilities using prior knowledge and observed
evidence.
ATF TTF TTF
BFTTFTFTF
5
𝑃(𝐴) =
8
4
𝑃(𝐵) =
8
2
𝑃(𝐴|𝐵) =
4
2
𝑃(𝐵|𝐴) =
5
𝑃(𝐵|𝐴)𝑃(𝐴) 2
𝑃(𝐴|𝐵) = ==
𝑃(𝐵) 4

3|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

𝑃(𝐴|𝐵)𝑃(𝐵) 2
𝑃(𝐵|𝐴) = ==
𝑃(𝐴) 5
Example: Application of Bayes Theorem
Consider a boy who has a volleyball tournament the next day, but he is feeling sick today.
Normally, there is a 40% chance that he falls sick. The boy is highly interested in volleyball, so
there is a 90% probability that he participates in the tournament. Additionally, there is a 20%
chance that he becomes sick given that he participates.
Given Data
𝑃(𝑃𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔) = 0.9
𝑃( 𝑆𝑖𝑐𝑘 ∣ 𝑃𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔 ) = 0.2
𝑃(𝑆𝑖𝑐𝑘) = 0.4
Required: Find the probability that the boy participates in the tournament given that he is sick:
𝑃( 𝑃𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔 ∣ 𝑆𝑖𝑐𝑘 )
Solution: Using Bayes’ theorem:
𝑃( 𝑆𝑖𝑐𝑘 ∣ 𝑃𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔 )
𝑃( 𝑃𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔 ∣ 𝑆𝑖𝑐𝑘 ) = 𝑃(𝑃𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔) ×
𝑃(𝑆𝑖𝑐𝑘)
0.9 × 0.2
= = 0.45
0.4
Final Answer: The probability that the boy will participate in the tournament given that he is sick
is 0.45 (or 45%).
Naïve Bayes Algorithm
Naïve Bayes is a supervised classification algorithm used for both binary and multi-class
problems. It is based on Bayes’ theorem and consists of a family of classifiers that follow a
common probabilistic approach.
The algorithm assumes that all features are independent of each other and contribute equally to
the outcome. Although, in reality, features may depend on the target class, they are treated as
independent during computation. Each feature independently contributes to the probability
calculation, which simplifies the classification process.
Naïve Bayes is particularly effective for large datasets and is known for its simplicity, speed, and
efficiency, making it one of the most widely used classification techniques.
Applications
Some common applications of Naïve Bayes include:
✓ Text classification
✓ Recommendation systems

4|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

✓ Face recognition

Algorithm: Naïve Bayes

1. Calculate the prior probabilities for each target class.
2. Determine the frequency matrix and compute the likelihood probabilities for all
features.
3. Apply Bayes’ theorem to compute the probability of each hypothesis.
4. Use the Maximum A Posteriori (MAP) rule to assign the test instance to the class with
the highest probability.

Example 8.2

Assess a student’s performance using the Naïve Bayes algorithm based on the dataset provided
in Table 8.1. The objective is to predict whether a student will receive a job offer in the final year.

Practical Communication
[Link] CGPA Interactiveness Job Offer
Knowledge Skills
1 ≥9 Yes Very Good Good Yes
2 ≥8 No Good Moderate Yes
3 ≥9 No Average Poor No
4 <8 No Average Good No
5 ≥8 Yes Good Moderate Yes
6 ≥9 Yes Good Moderate Yes
7 <8 Yes Good Poor No
8 ≥9 No Very Good Good Yes
9 ≥8 Yes Good Good Yes
9 ≥8 Yes Average Good Yes

Solution

The training dataset T consists of 10 instances with attributes such as CGPA, Interactiveness,
Practical Knowledge, and Communication Skills, as shown in Table 8.1. The target variable is
Job Offer, which has two possible classes: Yes or No.

Step 1: Compute Prior Probability

The target feature Job Offer is a binary classification with two classes:
Job Offer = Yes
Job Offer = No
From the dataset:

5|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

✓ Number of instances with Job Offer = Yes = 7

✓ Number of instances with Job Offer = No = 3

The prior probability is calculated as:

𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 𝑖𝑛 𝑎 𝑐𝑙𝑎𝑠𝑠

𝑃𝑟𝑖𝑜𝑟 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 =
𝑇𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠

Prior Probabilities
7
𝑃(𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠) =
10
3
𝑃(𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) =
10
Thus, the prior probabilities for the target variable are obtained based on the frequency of each
class in the dataset.
Table 8.2: Frequency Matrix and Prior Probability of Job Offer

Job Offer Classes No. of Instances Probability Value

Yes 7 P (Job Offer = Yes) = 7/10
No 3 P (Job Offer = No) = 3/10
Step 2: Compute Frequency matrix and Likelihood Probability for each of the feature.
Step 2(a): Feature - CGPA

Table 8.3 shows the frequency matrix for the feature CGPA.

Table 8.3: Frequency Matrix of CGPA

CGPA Job Offer = Yes Job Offer = No

≥9 3 1
≥8 4 0
<8 0 2
Total 7 3
Table 8.4 shows how the likelihood probability is calculated for CGPA using conditional
probability.

Table 8.4: Likelihood Probability of CGPA

CGPA Job Offer = Yes Job Offer = No

≥9 P (CGPA ≥ 9| Job Offer= Yes) = 3/7) P (CGPA ≥ 9| Job Offer= No) = 1/3)
≥8 P (CGPA ≥ 8| Job Offer= Yes) = 4/7) P (CGPA ≥ 8| Job Offer= No) = 0/3)
<8 P (CGPA < 8 | Job Offer= Yes) = 0/7) P (CGPA < 8 | Job Offer= No) = 2/3)
As explained earlier the Likelihood probability is stated as the sampling density for the evidence
given the hypothesis. It is denoted as P (Evidence | Hypothesis), which says how likely is the
occurrence of the evidence given the parameters.

It is calculated as the number of instances of each attribute value and for a given class value divided
by the number of instances with that class value.
6|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

For example P (CGPA 29 | Job Offer = Yes) denotes the number of instances with ′CGPA ≥
9 and ′Job Offer = Yes′ divided by the total number of instances with 'Job Offer = Yes'. From
the Table 8.3

Frequency Matrix of CGPA, number of instances with 'CGPA 29' and 'Job Offer = Yes' is 3. The
total number of instances with 'Job Offer = Yes' is 7. Hence, P (CGPA ≥9 | Job Offer = Yes) =3/7.
Similarly, the Likelihood probability is calculated for all attribute values of feature CGPA.

Step 2(b): Feature – Interactiveness

Table 8.5: Frequency Matrix of Interactiveness

Interactiveness Job Offer = Yes Job Offer = No

YES 5 1
NO 2 2
TOTAL 7 3
Table 8.6 shows how the likelihood probability is calculated for Interactiveness using conditional
probability.

Table 8.6: Likelihood Probability of Interactiveness

Interactiveness 𝐏(𝐉𝐨𝐛 𝐎𝐟𝐟𝐞𝐫 = 𝐘𝐞𝐬) 𝐏(𝐉𝐨𝐛 𝐎𝐟𝐟𝐞𝐫 = 𝐍𝐨)

YES P (Interactiveness = Yes | Job Offer = Yes) P (Interactiveness = Yes |

=5/7 Job Offer =No) = 1/3
NO P (Interactiveness = No | Job Offer = Yes) P (Interactiveness = No |
=2/7 Job Offer = No) =2/3
Step 2(c): Feature - Practical Knowledge

Table 8.7 shows the frequency matrix for the feature Practical Knowledge.
Table 8.7: Frequency Matrix of Practical Knowledge

Practical Knowledge Job Offer = Yes Job Offer = No

very Good 2 0
Average 1 2
Good 4 1
Total 7 3
Table 8.8 shows how the likelihood probability is calculated for Practical Knowledge using
conditional probability.

Table 8.8: Likelihood Probability of Practical Knowledge

Practical 𝑷(𝐉𝐨𝐛 𝐎𝐟𝐟𝐞𝐫 = 𝐘𝐞𝐬) 𝑷(𝐉𝐨𝐛 𝐎𝐟𝐟𝐞𝐫 = 𝐍𝐨)

Knowledge
very Good P (Practical Knowledge = Very P (Practical Knowledge = Very Good |
Good | Job Offer = Yes) =2/7 Job Offer = No) = 0/3

7|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

Average P (Practical Knowledge = | Job Offer = Yes) = 1/7 P (Practical

Average | Job Offer = Yes) = 1/7 Knowledge = Average | Job Offer =
No) = 2/3
Good P (Practical Knowledge = Good | P (Practical Knowledge = Good | Job
Job Offer = Yes) = 4/7 Offer = No) = 1/3
Step 2(d): Feature - Communication Skills

Table 8.9 shows the frequency matrix for the feature Communication Skills.

Table 8.9: Frequency Matrix of Communication Skills

Communication Skills Job Offer = Yes Job Offer = No

Good 4 1
Moderate 3 0
Poor 0 2
Total 7 3
Table 8.10: Likelihood Probability of Communication Skills

Communication 𝑷(𝐉𝐨𝐛 𝐎𝐟𝐟𝐞𝐫 = 𝐘𝐞𝐬) 𝑷(𝐉𝐨𝐛 𝐎𝐟𝐟𝐞𝐫 = 𝐍𝐨)

Skills
Good P (Communication Skills = Good P (Communication Skills = Good | Job
| Job Offer = Yes) = 4/7 Offer = No) = 1/3
Moderate P (Communication Skills = P (Communication Skills = Moderate |
Moderate | Job Offer = Yes) = Job Offer = No) = 0/3
3/7
Poor P (Communication Skills = Poor P (Communication Skills = Poor | Job
| Job Offer = Yes) = 0/7 Offer = No) = 2/3

Step 3: Use Bayes theorem Eq. (8.1) to calculate the probability of all hypotheses.

Given the test 𝑑𝑎𝑡𝑎 = (𝐶𝐺𝑃𝐴 ≥ 9, 𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠, 𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙 𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒 =

𝐴𝑣𝑒𝑟𝑎𝑔𝑒, 𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑆𝑘𝑖𝑙𝑙𝑠 = 𝐺𝑜𝑜𝑑), apply the Bayes theorem to classify whether the
given student gets a Job offer or not.

𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠 | 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎) = (𝑃(𝐶𝐺𝑃𝐴 ≥ 9 |𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 𝑌𝑒𝑠) 𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠
= 𝑌𝑒𝑠 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠) 𝑃 (𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙 𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒
= 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠) 𝑃 (𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑆𝑘𝑖𝑙𝑙𝑠
= 𝐺𝑜𝑜𝑑 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠) 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠)))/(𝑃 (𝑇𝑒𝑠𝑡 𝐷𝑎𝑡𝑎))

We can ignore P (Test Data) in the denominator since it is common for all cases to be considered.
Hence,

8|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠 | 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎) = (𝑃(𝐶𝐺𝑃𝐴 ≥ 9 𝑙𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠) 𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠
= 𝑌𝑒𝑠 |𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠) 𝑃 (𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙 𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒 = 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑌𝑒𝑠) 𝑃 (𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑆𝑘𝑖𝑙𝑙𝑠 = 𝐺𝑜𝑜𝑑 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠) 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑌𝑒𝑠)
3 5 1 4 7
= × × × ×
7 7 7 7 10
= 𝟎. 𝟎𝟏𝟕𝟓

Similarly, for the other case ′𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜′ ,

We compute the probability,

𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜! 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎) = (𝑃(𝐶𝐺𝑃𝐴 29 |𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠
= 𝑌𝑒𝑠 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙 𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒 = 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑁𝑜) 𝑃 (𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑆𝑘𝑖𝑙𝑙𝑠 = 𝐺𝑜𝑜𝑑 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑁𝑜))/(𝑃(𝑇𝑒𝑠𝑡 𝐷𝑎𝑡𝑎)).

𝑃 (𝐶𝐺𝑃𝐴 29 |𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟

= 𝑁𝑜) 𝑃 (𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙 𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒 = 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑁𝑜) 𝑃 (𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑆𝑘𝑖𝑙𝑙𝑠 = 𝐺𝑜𝑜𝑑 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑁𝑜)
1 1 2 1 3
× × × ×
3 3 3 3 10
𝟎. 𝟎𝟎𝟕𝟒

Step 4: Use Maximum A Posteriori (MAP) 𝑯𝒚𝒑𝒐𝒕𝒉𝒆𝒔𝒊𝒔, 𝑴𝑨𝑷 𝑬𝒒. (𝟖. 𝟐) to classify the test
object to the hypothesis with the highest probability.

Since
𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠 | 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎)
ℎ𝑎𝑠 𝑡ℎ𝑒 ℎ𝑖𝑔ℎ𝑒𝑠𝑡 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑣𝑎𝑙𝑢𝑒, 𝑡ℎ𝑒 𝑡𝑒𝑠𝑡 𝑑𝑎𝑡𝑎 𝑖𝑠 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠
′𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠′.
Zero Probability Error
In Example 8.1, consider the test data to be (CGPA ≥ 8, Interactiveness = Yes,

Practical knowledge = Average, Communication Skills = Good)

When computing the posterior probability,

𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠 | 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎) = (𝑃(𝐶𝐺𝑃𝐴 ≥ 8 |𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠)

𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠)
𝑃 (𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙 𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒 = 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠)
𝑃 (𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑆𝑘𝑖𝑙𝑙𝑠 = 𝐺𝑜𝑜𝑑 𝐼 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠)

9|Page
Dept. of CSE, RNSIT
 MACHINE LEARNING

𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠)))/(𝑃(𝑇𝑒𝑠𝑡 𝐷𝑎𝑡𝑎))

𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠 | 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎) = (𝑃(𝐶𝐺𝑃𝐴 ≥ 8 |𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 𝑌𝑒𝑠)

𝑃(𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠)
𝑃 (𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙 𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒 = 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠)
𝑃 (𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑆𝑘𝑖𝑙𝑙𝑠 = 𝐺𝑜𝑜𝑑 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠)
𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠)

4 5 1 4 7
= × × × ×
7 7 7 7 10

𝟎. 𝟎𝟐𝟑𝟑
Similarly, for the other case ′𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜,
When we compute the probability:
𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜| 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎) = (𝑃(𝐶𝐺𝑃𝐴 ≥ 8 |𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠
= 𝑌𝑒𝑠 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙 𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒
= 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑆𝑘𝑖𝑙𝑙𝑠
= 𝐺𝑜𝑜𝑑 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜))/(𝑃(𝑇𝑒𝑠𝑡 𝐷𝑎𝑡𝑎))

(𝐶𝐺𝑃𝐴 ≥ 8 |𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 =

𝑁𝑜) 𝑃 (𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙 𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒 = 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 =No)
𝑃 (𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑆𝑘𝑖𝑙𝑙𝑠 = 𝐺𝑜𝑜𝑑 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁o)
0 1 2 1 3
= × × × ×
3 3 3 3 10
=𝟎

Since the probability value is zero, the model fails to predict, and this is called as Zero Probability
error. This problem arises because there are no instances in the given Table 8.1 for the attribute
value CGPA ≥8 and Job Offer = No and hence the probability value of this case is zero. This zero-
probability error can be solved by applying a smoothing technique called Laplace correction
which means given 1000 data instances in the training dataset, if there are zero instances for a
particular value of a feature we can add 1 instance for each attribute value pair of that feature
which will not make much difference for 1000 data instances and the overall probability does not
become zero. Now, let us scale the values given in Table 8.1 for 1000 data instances.
The scaled values without Laplace correction are shown in Table 8.11.

Table 8.11: Scaled Values to 1000 without Laplace Correction

CGPA Job Offer = Yes Job Offer = No

≥9 P (CGPA ≥ 9| Job Offer= Yes) = P (CGPA ≥ 9| Job Offer= No) = 100/300)
300/700)
≥8 P (CGPA ≥ 8| Job Offer= Yes) = P (CGPA ≥ 8| Job Offer= No) = 0/300)
400/700)
<8 P (CGPA < 8 | Job Offer= Yes) = P (CGPA < 8 | Job Offer= No) = 200/300)
0/700)

10 | P a g e
Dept. of CSE, RNSIT
 MACHINE LEARNING

Now, add 1 instance for each CGPA-value pair for 'Job Offer = No'. Then,
𝑃 (𝐶𝐺𝑃𝐴 ≥ 9 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) = 101/303 = 0.333
𝑃 (𝐶𝐺𝑃𝐴 ≥ 8 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) = 1/303 = 0.0033
𝑃 (𝐶𝐺𝑃𝐴 < 8 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) = 201/300 = 0.6634
With scaled values to 1003 data instances, we get
𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠 | 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎) = (𝑃(𝐶𝐺𝑃𝐴 ≥ 8 |𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑌𝑒𝑠) 𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑌𝑒𝑠) 𝑃 (𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙 𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒 = 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑌𝑒𝑠) 𝑃 (𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑆𝑘𝑖𝑙𝑙𝑠 = 𝐺𝑜𝑜𝑑 𝑖 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑌𝑒𝑠) 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠)
400 500 100 400 700
= × × × ×
700 700 700 700 1003
𝟎. 𝟎𝟐𝟑𝟐𝟓

𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜 | 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎) = 𝑃(𝐶𝐺𝑃𝐴 28 |𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠

= 𝑌𝑒𝑠 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙 𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒
= 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑆𝑘𝑖𝑙𝑙𝑠
= 𝐺𝑜𝑜𝑑 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜)
1 100 200 100 303
× × × ×
303 300 300 300 1003

𝟎. 𝟎𝟎𝟎𝟎𝟕𝟑𝟖𝟓
Thus, using Laplace Correction, Zero Probability error can be solved with Naïve Bayes classifier.
Brute Force Bayes Algorithm
Applying Bayes theorem, Brute Force Bayes algorithm relies on the idea of concept learning
wherein given a hypothesis space H for the training dataset T, the algorithm computes the
posterior probabilities for all the hypothesis hi ∈ H. Then, Maximum A Posteriori (MAP)
Hypothesis, hMAP is used to output the hypothesis with maximum posterior probability. The
algorithm is quite expensive since it requires computations for all the hypotheses. Although
computing posterior probabilities is inefficient, this idea is applied in various other algorithms
which is also quite interesting.

Bayes Optimal Classifier

Bayes optimal classifier is a probabilistic model, which in fact, uses the Bayes theorem to find the
most probable classification for a new instance given the training data by combining the
predictions of all posterior hypotheses, This is different from Maximum A Posteriori (MAP)
𝐻𝑦𝑝𝑜𝑡ℎ𝑒𝑠𝑖𝑠, ℎ𝑀𝐴𝑃 which chooses the maximum probable hypothesis or the most probable
hypothesis. Here, a new instance can be classified to a possible 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑣𝑎𝑙𝑢𝑒 𝐶𝑖 . by the
following Eq. (8.4).

= max ∑ 𝑃(𝐶𝑖 |ℎ𝑖 )𝑃(ℎ𝑖 | 𝑇) 8.4

ci
ℎ𝑖 ∈𝐻

11 | P a g e
Dept. of CSE, RNSIT
 MACHINE LEARNING

Example 8.3 Given the hypothesis space with 4 ℎ𝑦𝑝𝑜𝑡ℎ𝑒𝑠𝑖𝑠 ℎ1 , ℎ2 ℎ3 𝑎𝑛𝑑 ℎ4 . Determine if the
patient is diagnosed as COVID positive or COVID negative using Bayes Optimal classifier.
Solution: From the 𝑡𝑟𝑎𝑖𝑛𝑖𝑛𝑔 𝑑𝑎𝑡𝑎𝑠𝑒𝑡 𝑇, the posterior probabilities of the four different
hypotheses for a new instance are given in Table 8.12.
Table 8.12: Posterior Probability Values
P (h. T) P (COVID Positive |𝒉) P (COVID Negative |𝒉)
0.3 0 1
0.1 1 0
0.2 1 0
0.1 1 0

ℎ𝑀𝐴𝑃 𝑐ℎ𝑜𝑜𝑠𝑒𝑠 ℎ, which has the maximum probability 𝑣𝑎𝑙𝑢𝑒 0.3 as the solution and gives the
result that the patient is COVID negative. But Bayes Optimal classifier combines the predictions
of ℎ2 , ℎ3 𝑎𝑛𝑑 ℎ4 𝑤ℎ𝑖𝑐ℎ 𝑖𝑠 0.4 and gives the result that the patient is COVID positive.

∑ 𝑃(𝐶𝑂𝑉𝐼𝐷 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒 |ℎ)𝑃(ℎ₁|𝑇) = 0.3 × 1 = 0.3

ℎ𝑖 ∈𝐻

∑ 𝑃(𝐶𝑂𝑉𝐼𝐷 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 |ℎ)𝑃(ℎ₁|𝑇) = 0.1 × 1 + 0.2 × 1 + 0.1 × 1 = 0.4

ℎ𝑖 ∈𝐻
Therefore
max (𝐶𝑜𝑣𝑖𝑑 𝑃𝑜𝑠𝑖𝑡𝑣𝑒 , 𝐶𝑜𝑣𝑖𝑑 𝑁𝑒𝑔𝑎𝑡𝑣𝑖𝑒)
𝑐𝑖𝑑𝑖

𝑚𝑎𝑥 𝐶𝑖𝑑𝑖(𝐶𝑜𝑣𝑖𝑑 𝑃𝑜𝑠𝑖𝑡𝑣𝑒 ,𝐶𝑜𝑣𝑖𝑑 𝑁𝑒𝑔𝑎𝑡𝑣𝑖𝑒) ∑ℎ𝑖 ∈𝐻 𝑃(𝐶𝑖 |ℎ𝑖 )𝑃(ℎ𝑖 | 𝑇) = 𝐶𝑜𝑣𝑖𝑑 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒
Thus, this algorithm, diagnoses the new instance to be COVID positive.

Gibbs Algorithm
The main drawback of Bayes optimal classifier is that it computes the posterior probability for all
hypotheses in the hypothesis space and then combines the predictions to classify a new instance.
Gibbs algorithm is a sampling technique which randomly selects a hypothesis from the hypothesis
space according to the posterior probability distribution and classifies a new instance. It is found
that the prediction error occurs twice with the Gibbs algorithm when compared to Bayes Optimal
classifier.

NAÏVE BAYES ALGORITHM FOR CONTINUOUS ATTRIBUTES

There are two ways to predict with Naive Bayes algorithm for continuous attributes:
1. Discretize continuous feature to discrete feature.
2. Apply Normal or Gaussian distribution for continuous feature.

12 | P a g e
Dept. of CSE, RNSIT
 MACHINE LEARNING

Gaussian Naive Bayes Algorithm

In Gaussian Naive Bayes, the values of continuous features are assumed to be sampled from a
Gaussian distribution.
Example 8.4: Assess a student's performance using Naïve Bayes algorithm for the continuous
attribute. Predict whether a student gets a job offer or not in his final year of the course.
𝑇ℎ𝑒 𝑡𝑟𝑎𝑖𝑛𝑖𝑛𝑔 𝑑𝑎𝑡𝑎𝑠𝑒𝑡 𝑇 consists of 10 𝑑𝑎𝑡𝑎 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 with attributes such as
′𝐶𝐺𝑃𝐴′ 𝑎𝑛𝑑 ′𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠′ as shown in Table 8.13.
The target variable is Job Offer which is classified as Yes or No for a candidate student.
Table 8.13: Training Dataset with Continuous Attribute

[Link] CGPA Interactiveness Job Offer

1 9.5 Yes Yes
2 8.2 No Yes
3 9.3 No No
4 7.6 No No
5 8.4 Yes Yes
6 9.1 Yes Yes
7 7.5 Yes No
8 9.6 No Yes
9 8.6 Yes Yes
10 8.3 Yes Yes

Solution: Step 1: Compute the prior probability for the target feature ′𝑱𝒐𝒃 𝑶𝒇𝒇𝒆𝒓′ .

Prior probabilities of both the classes are calculated using the same formula (refer to Table 8.14).
Table 8.14: Prior Probability of Target Class

Job Offer Classes No. of Instances Probability Value

Yes 7 P (Job Offer = Yes) = 7/10
No 3 P (Job Offer = No) = 3/10

Step 2: Computation of Frequency Matrix and Likelihood Probabilities

In this step, the frequency matrix is constructed, and the likelihood probability for each feature
is calculated. For categorical features, likelihood probabilities are computed based on the
frequency of feature values within each class. For continuous attributes, likelihood probabilities
are determined using the Gaussian (Normal) Distribution. In the given dataset, CGPA is a
continuous feature; hence, the Gaussian distribution is applied to estimate its likelihood
probability. The likelihood for a continuous feature is calculated using the Gaussian distribution
formula, as given below:

13 | P a g e
Dept. of CSE, RNSIT
 MACHINE LEARNING

𝑃(𝑋𝑖 = 𝑥𝑘 |𝐶𝑖 ) = 𝑔( 𝑥𝑘 , 𝜇𝑖𝑗 , 𝜎𝑖𝑗 ) 8.5

where,
𝑋𝑖 is the 𝑖 𝑡ℎ continuous attribute in the given dataset and 𝑥𝑘 is a value of the attribute. 𝐶𝑖 denotes
the 𝐽𝑡ℎ class of the target feature. 𝜇𝑖𝑗 denotes the mean of the values of that continuous attribute
𝑋𝑖 , with respect to the 𝑐𝑙𝑎𝑠𝑠 𝑗 of the target feature. 𝜎𝑖𝑗 denotes the standard deviation of the
values of that continuous attribute 𝑋𝑖 ,with respect to the 𝑐𝑙𝑎𝑠𝑠 𝑗 of the target feature.
Hence, the normal distribution formula is given as:
2
1 (𝑥𝑘 − 𝜇𝑖𝑗 )
𝑃(𝑋, = 𝑥𝑘 |𝐶𝑖 ) = 𝑒^ 8.6
𝜎𝑖𝑗 √2𝜋 2 𝜎𝑖𝑗2
Step 2(a): Consider the feature CGPA
In this example CGPA is a continuous attribute,
To calculate the likelihood probability for this continuous attribute, first compute the mean and
standard deviation for CGPA with respect to the target 𝑐𝑙𝑎𝑠𝑠 ′𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟′.
Here, 𝑋, = 𝐶𝐺РА
𝐶 = ′𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠′
Mean and Standard Deviation for 𝑐𝑙𝑎𝑠𝑠 ′𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠′ are given as:
𝜇𝑖𝑗 = 𝜇𝐶𝐺𝑃𝐴−𝑌𝐸𝑆 =8.814286
𝜎𝑖𝑗 = 𝜎𝐶𝐺𝑃𝐴−𝑌𝐸𝑆 =0.58146

Mean and Standard Deviation for 𝑐𝑙𝑎𝑠𝑠 ′𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜′ are given as:
𝜇𝑖𝑗 = 𝜇𝐶𝐺𝑃𝐴−𝑁𝑜 =8.133333
𝜎𝑖𝑗 = 𝜎𝐶𝐺𝑃𝐴−𝑁𝑜 =1.011599

Once Mean and Standard Deviation are computed, the likelihood probability for any test value
using Gaussian distribution formula can be calculated.
Step 2(b): Consider the feature Interactiveness
Interactiveness is a discrete feature whose probability is calculated as earlier. Table 8.15 shows
the frequency matrix for the feature Interactiveness.
Table 8.15: Frequency Matrix of Interactiveness

Interactiveness Job Offer = Yes Job Offer = No

YES 5 1
NO 2 2
TOTAL 7 3

Table 8.16 shows how the likelihood probability is calculated for Interactiveness using conditional
probability.

14 | P a g e
Dept. of CSE, RNSIT
 MACHINE LEARNING

Table 8.16: Likelihood Probability of Interactiveness

Interactiveness 𝐏(𝐉𝐨𝐛 𝐎𝐟𝐟𝐞𝐫 = 𝐘𝐞𝐬) 𝐏(𝐉𝐨𝐛 𝐎𝐟𝐟𝐞𝐫 = 𝐍𝐨)

YES P (Interactiveness = Yes | P (Interactiveness = Yes |
Job Offer = Yes) =5/7 Job Offer =No) = 1/3
NO P (Interactiveness = No | P (Interactiveness = No |
Job Offer = Yes) =2/7 Job Offer = No) =2/3

Step 3: Use Bayes theorem to calculate the probability of all hypotheses.

Consider the test data to be (𝐶𝐺𝑃𝐴 = 8.5, 𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠).
For the hypothesis ′𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠′: 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠 | 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎) = (𝑃(𝐶𝐺𝑃𝐴 =
8.5 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠) 𝑥 𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 =
𝑌𝑒𝑠) 𝑥 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠)
To compute 𝑃 (𝐶𝐺𝑃𝐴 = 8.5 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠) use Gaussian distribution formula:
𝑃 (𝑋𝑖 = 𝑥|𝐶𝑖 = 𝑔(𝑥𝑖 , 𝜇𝑖𝑗 𝜎𝑖𝑗 )

Hence:
𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠 | 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎) = (𝑃(𝐶𝐺𝑃𝐴 = 8.5 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑌𝑒𝑠) 𝑥 𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 |𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠) 𝑥 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑌𝑒𝑠)
5 7
0.594 × ×
7 10
0.297

Similarly, for the hypothesis ′𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜′:

𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜 | 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎) = 𝑃 (𝐶𝐺𝑃𝐴 = 8.5 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 =
𝑁𝑜) 𝑥 𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑥 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜)

15 | P a g e
Dept. of CSE, RNSIT
 MACHINE LEARNING

𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) = 1/3

= 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) = 0.369

Hence,
𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜 | 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎) = 𝑃 (𝐶𝐺𝑃𝐴 = 8.5 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑁𝑜) 𝑃 (𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 | 𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑁𝑜) 𝑥 𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟
= 𝑁𝑜) = 0.369 𝑥 1/3 𝑥 3/10 = 0.0369
Step 4: Use Maximum A Posteriori (MAP) Hypothesis, MAP to classify the test object to the
hypothesis with the highest probability. Since
𝑃 (𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠 | 𝑇𝑒𝑠𝑡 𝑑𝑎𝑡𝑎)
has the highest probability 𝑣𝑎𝑙𝑢𝑒 𝑜𝑓 0.297,the test data is classified as ′𝐽𝑜𝑏 𝑂𝑓𝑓𝑒𝑟 = 𝑌𝑒𝑠′.

Other Popular Types of Naïve Bayes Classifiers

Several widely used variants of the Naïve Bayes classifier are described below:
1. Bernoulli Naïve Bayes Classifier
The Bernoulli Naïve Bayes classifier is designed for discrete binary features. In this
approach, each feature is represented as a Boolean variable, taking values such as yes/no
or 0/1. It is especially effective in text classification tasks, where features indicate whether
a particular word is present or absent in a document.
2. Multinomial Naïve Bayes Classifier
The Multinomial Naïve Bayes classifier extends the Bernoulli model to handle categorical
or integer-valued features. It is particularly suitable for text-based applications, where
features represent the frequency of words occurring in a document rather than just their
presence or absence.
3. Multi-class Naïve Bayes Classifier
This variant is applicable to problems involving more than two classes. It enables the
model to assign a given test instance to one of several possible target classes, making it
suitable for multi-class classification scenarios.

16 | P a g e
Dept. of CSE, RNSIT