Machine learning BCS602

Module - 04
Bayesian Learning
Bayesian Learning is a method of learning that uses probability to make predictions and
improves its results by updating its knowledge when new data is received.
Or
It is a method that learns from data by updating probabilities when new information is added.
4.1 INTRODUCTION TO PROBABILITY-BASED LEARNING
Probability-based learning is a method that combines prior knowledge (prior probability) with
observed data to make predictions.
• It is one of the most important practical learning methods used in machine learning to
handle real-world data.
• It combines previously known information (prior probability) with new data (observed
data) to improve prediction accuracy.
• It uses the concept of probability theory to model randomness, uncertainty, and noise
present in the data.
• It helps in predicting future events by analyzing patterns and probabilities from the
given dataset.
• This method is widely used for modelling large datasets and extracting useful
information from them.
• It uses Bayes’ rule to infer unknown values, update beliefs, and learn from data step by
step.
Probabilistic Model:
• In a probabilistic model, randomness is considered, and the output is expressed in the form
of probability distribution instead of a single value.
• It is more suitable for real-world problems where uncertainty and incomplete data are present.
Deterministic Model:
• In a deterministic model, there is no randomness, and the same input always produces the
same output.
• It gives a single fixed result and does not handle uncertainty.
4.2 Fundamentals of Bayes Theorem
• Naïve Bayes model is based on Bayes theorem
• It works using three types of probabilities:
– Prior probability
– Likelihood probability
– Posterior probability
1. Prior Probability
• Prior probability is the initial probability of an event before any evidence is observed
• It represents the belief about an event before collecting new data
• It is based on past knowledge or experience
2. Likelihood Probability
• Likelihood probability is the probability of observing the data given a hypothesis
• It is written as:
P(Evidence | Hypothesis)

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

• It shows how likely the evidence is when the hypothesis is true

3. Posterior Probability
• Posterior probability is the updated probability after considering new evidence
• It is written as:
P(Hypothesis | Evidence)
• It represents the final belief after learning from data
Important Relation:
• Posterior probability is calculated using prior probability and new evidence
Posterior Probability = Prior Probability + New Evidence (conceptual idea)
4.3 Classification Using Bayes Model(Bayes Theorem)
Classification is the process of finding which class an object belongs to.
The Bayes model helps to choose the best class by using probability.
• Naïve Bayes classification works based on Bayes theorem
• Bayes rule is used to calculate the posterior probability
• It helps to select the most probable hypothesis (class) from the given data
Important Terms:
Hypothesis (h):
– The class we want to predict
– Example: Pass / Fail
Evidence (E):
– The given data or input
– Example: Marks, Attendance
Bayes Formula:
𝑃(𝐸 ∣ ℎ) 𝑃(ℎ)
𝑃(ℎ ∣ 𝐸) =
𝑃(𝐸)

Terms
1. Prior Probability – P(h)
• It is the probability of hypothesis before seeing the data
• It is also called the initial guess
2. Likelihood Probability – P(E|h)
• It is the probability of evidence given the hypothesis
• It shows how well the data matches the class
3. Evidence – P(E)
• It is the probability of evidence from the dataset
• It is constant for all classes
4. Posterior Probability – P(h|E)
• It is the final probability after observing the data
• It is used to make the final decision
How Classification Works:
Step 1: Take all possible classes
– Example: Pass, Fail
Step 2: Calculate probability for each class using formula

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

Step 3: Compare the values

Step 4: Choose the class with highest probability.
Simple Example
Classes: Pass / Fail
Input: Good marks, High attendance
Calculate probability for both classes
If Pass probability is higher → Student is Pass

Decision Criteria in Bayes Classification

• In classification, we need a method to choose the best hypothesis (class)
• MAP and ML are decision criteria (methods) used for selecting the best hypothesis
1. Maximum A Posteriori (MAP) Hypothesis
MAP hypothesis is the hypothesis that has the maximum posterior probability It is also called
the most probable hypothesis
• Given a set of candidate hypotheses, we calculate posterior probability for each
• The hypothesis with the highest posterior value is selected
• It uses Bayes theorem for calculation
Formula:
ℎ𝑀𝐴𝑃 = arg⁡ max⁡ 𝑃(ℎ ∣ 𝐸)
𝑃(𝐸 ∣ ℎ) 𝑃(ℎ)
= arg⁡ max⁡
𝑃(𝐸)
• Since 𝑃(𝐸)is constant, we can write:
ℎ𝑀𝐴𝑃 = arg⁡ max⁡ 𝑃(𝐸 ∣ ℎ) 𝑃(ℎ)
2. Maximum Likelihood (ML) Hypothesis
ML hypothesis is the hypothesis that has the maximum likelihood probability
• It is used when all hypotheses are equally probable
• Only likelihood 𝑃(𝐸 ∣ ℎ)is considered
• Prior probability is ignored
Formula:
ℎ𝑀𝐿 = arg⁡ max⁡ 𝑃(𝐸 ∣ ℎ)
• ML selects hypothesis based only on evidence, not prior knowledge
Correctness of Bayes Theorem
Given:
• Consider two events A and B
• 𝑃(𝐴) = 5/8
• 𝑃(𝐵) = 4/8
Conditional Probabilities:
• 𝑃(𝐴 ∣ 𝐵) = 2/4
• 𝑃(𝐵 ∣ 𝐴) = 2/5
Verification:
• Using Bayes theorem:
𝑃(𝐵 ∣ 𝐴) 𝑃(𝐴) 2
𝑃(𝐴 ∣ 𝐵) = =
𝑃(𝐵) 4

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

𝑃(𝐴 ∣ 𝐵) 𝑃(𝐵) 2
𝑃(𝐵 ∣ 𝐴) = =
𝑃(𝐴) 5

Given:
• P(Participating) = 0.9
• P(Sick | Participating) = 0.2
• P(Sick) = 0.4
We need to find:
• P(Participating | Sick)
Step 1: Write Bayes Formula
𝑃(𝑆𝑖𝑐𝑘 ∣ 𝑃𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔) × 𝑃(𝑃𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔)
𝑃(𝑃𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔 ∣ 𝑆𝑖𝑐𝑘) =
𝑃(𝑆𝑖𝑐𝑘)
Step 2: Substitute Values
0.2 × 0.9
=
0.4
Step 3: Calculate
0.18
= = 0.45
0.4
Probability = 0.45 (or 45%)

4.3.1 Naïve Bayes Algorithm

Naïve Bayes is a supervised classification algorithm used for binary and multi-class problems
It works based on Bayes theorem
• It assumes that all features are independent of each other
• Each feature contributes independently to the final classification
• All features are given equal importance (weightage)
• It is a simple and efficient algorithm used for classification tasks
• It works well with large datasets
• Even though features may be dependent in real life, the algorithm assumes independence
(this is why it is called “Naïve”)
• During classification, each feature adds its own probability to the final result
Advantages:

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

• Simple and easy to implement

• Works fast even for large datasets
• Requires less training data
• Gives good performance in many real-world problems
Applications:
• Text classification (spam detection)
• Recommendation systems
• Face recognition

Solution
The training dataset T consists of 10 data instances with attributes such as CGPA,
Interactiveness, Practical Knowledge, and Communication Skills, as shown in Table 8.1.
The target variable is Job Offer, which is classified as Yes or No for a candidate student.
Step 1: Compute Prior Probability
The target feature Job Offer has two classes:
• Yes
• No
Hence, it is a binary classification problem.

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

Given a student instance, we need to classify whether:

Job Offer = Yes or Job Offer = No
Frequency Observation
From the training dataset:
• Number of instances with Job Offer = Yes = 7
• Number of instances with Job Offer = No = 3
Calculation of Prior Probability
The prior probability is calculated as:
Number of instances of a class
Prior Probability =
Total number of instances

Final Prior Probabilities

7
𝑃(Job Offer = Yes) =
10
3
𝑃(Job Offer = No) =
10

Step 2(a): Likelihood Probability for CGPA

Table 8.4 shows how the likelihood probability is calculated for CGPA using conditional
probability.
Likelihood Probability Table (CGPA)
CGPA P(Job Offer = Yes) P(Job Offer = No)

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

≥9 P(CGPA ≥9 Yes) = 3/7

≥8 P(CGPA ≥8 Yes) = 4/7
<8 P(CGPA <8 Yes) = 0/7
• Likelihood probability is defined as the probability of evidence given the hypothesis
• It is denoted as:
𝑃(Evidence ∣ Hypothesis)
• It tells how likely the evidence is when the hypothesis is true

How it is Calculated
Number of instances with that feature and class
𝑃(Feature value ∣ Class) =
Total number of instances of that class
Example Calculation
P(CGPA ≥9 | Job Offer = Yes)
• Number of students with:
CGPA ≥9 AND Job = Yes = 3
• Total students with Job = Yes = 7
𝑃(𝐶𝐺𝑃𝐴 ≥ 9 ∣ 𝑌𝑒𝑠) = 3/7
P(CGPA ≥9 | Job Offer = No)
• Count = 1
• Total Job = No = 3
𝑃(𝐶𝐺𝑃𝐴 ≥ 9 ∣ 𝑁𝑜) = 1/3
P(CGPA ≥8 | Job Offer = No)
• Count = 0
𝑃(𝐶𝐺𝑃𝐴 ≥ 8 ∣ 𝑁𝑜) = 0/3 = 0
P(CGPA <8 | Job Offer = No)
• Count = 2
𝑃(𝐶𝐺𝑃𝐴 < 8 ∣ 𝑁𝑜) = 2/3

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

Step 2(b): Likelihood Probability of Interactiveness

Formula Used
Count of (Feature + Class)
𝑃(Feature ∣ Class) =
Total count of Class
Step 1: From Dataset
We already know:
• Job Offer = Yes → 7 students
• Job Offer = No → 3 students
Step 2: Count Values
Interactiveness = YES
From dataset:
• YES & Job = Yes → 5 students
• YES & Job = No → 1 student
Interactiveness = NO
• NO & Job = Yes → 2 students
• NO & Job = No → 2 students
Step 3: Calculate Probabilities
For Job Offer = YES
1. P(Interactiveness = Yes | Yes)
5
=
7
2. P(Interactiveness = No | Yes)
2
=
7
For Job Offer = NO
3. P(Interactiveness = Yes | No)
1
=
3
4. P(Interactiveness = No | No)
2
=
3

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

Step 2(c): Practical Knowledge

We are finding:
𝑃(Practical Knowledge ∣ Job Offer)
This is called Likelihood Probability
“How likely is a student’s practical knowledge given Job = Yes/No?”
Step 1: Extract Required Data
From dataset, take only:
Practical Knowledge + Job Offer
Row Practical Knowledge Job
1 Very Good Yes
2 Good Yes
3 Average No
4 Average No
5 Good Yes
6 Good Yes
7 Good No
8 Very Good Yes
9 Good Yes
10 Average Yes
Step 2: Count Values (Frequency Matrix)
We count how many times each value appears with Yes/No.
Very Good
• With Job = Yes → Row 1, 8 → 2
• With Job = No → 0
So:
| Very Good | 2 | 0 |
Average
• With Job = Yes → Row 10 → 1
• With Job = No → Row 3, 4 → 2
So:
| Average | 1 | 2 |
Good
• With Job = Yes → Row 2,5,6,9 → 4
• With Job = No → Row 7 → 1
So:
| Good | 4 | 1 |
Step 3: Final Frequency Table
Practical Knowledge Job = Yes Job = No
Very Good 2 0
Average 1 2
Good 4 1
Total 7 3
Step 4: Apply Formula

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

Count of (Feature + Class)

𝑃(Feature ∣ Class) =
Total of Class
Step 5: Calculate Each Probability
For Job Offer = YES (Total = 7)
1. Very Good
2
𝑃(Very Good ∣ 𝑌𝑒𝑠) =
7
Out of 7 students who got job, 2 have very good knowledge
2. Average
1
𝑃(Average ∣ 𝑌𝑒𝑠) =
7
Only 1 student with average knowledge got job
3. Good
4
𝑃(Good ∣ 𝑌𝑒𝑠) =
7
Most students with good knowledge got job
For Job Offer = NO (Total = 3)
4. Very Good
0
𝑃(Very Good ∣ 𝑁𝑜) = = 0
3
No student with very good knowledge failed
5. Average
2
𝑃(Average ∣ 𝑁𝑜) =
3
Most students who failed had average knowledge
6. Good
1
𝑃(Good ∣ 𝑁𝑜) =
3
Few students with good knowledge failed
Step 6: Final Likelihood Table
Practical Knowledge P(Job Offer = Yes) P(Job Offer = No)
Very Good 2/7 0
Average 1/7 2/3
Good 4/7 1/3

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

Step 2(d): Communication Skills

Likelihood probability:
𝑃(Communication Skill ∣ Job Offer)

“How likely is a student’s communication skill given Job = Yes/No?”

Take Data from Dataset
Focus only on:
Communication Skills + Job Offer

Row Communication Job

1 Good Yes
2 Moderate Yes
3 Poor No
4 Good No
5 Moderate Yes
6 Moderate Yes
7 Poor No
8 Good Yes
9 Good Yes
10 Good Yes
Count (Frequency Matrix)
Good
• Job = Yes → rows 1,8,9,10 → 4
• Job = No → row 4 → 1
Moderate
• Job = Yes → rows 2,5,6 → 3
• Job = No → 0
Poor
• Job = Yes → 0
• Job = No → rows 3,7 → 2
Total
• Job = Yes → 7
• Job = No → 3
Apply Formula
Count
𝑃(Feature ∣ Class) =
Total of Class
Calculate Probabilities

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

For Job Offer = YES (Total = 7)

• P(Good | Yes) = 4/7
• P(Moderate | Yes) = 3/7
• P(Poor | Yes) = 0/7 = 0

For Job Offer = NO (Total = 3)

• P(Good | No) = 1/3
• P(Moderate | No) = 0/3 = 0
• P(Poor | No) = 2/3
Final Table
Communication P(Job Offer = Yes) P(Job Offer = No)
Good 4/7 1/3
Moderate 3/7 0
Poor 0 2/3

Step 3: Apply Bayes Theorem

Given Test Data
• CGPA ≥ 9
• Interactiveness = Yes
• Practical Knowledge = Average
• Communication Skills = Good
We must find:
𝑃(Job Offer = Yes | Test Data)
𝑃(Job Offer = No | Test Data)
Bayes Formula Used
𝑃(𝐷𝑎𝑡𝑎 ∣ 𝐶𝑙𝑎𝑠𝑠) × 𝑃(𝐶𝑙𝑎𝑠𝑠)
𝑃(𝐶𝑙𝑎𝑠𝑠 ∣ 𝐷𝑎𝑡𝑎) =
𝑃(𝐷𝑎𝑡𝑎)

Since P(Data) is same for both → we ignore it

Case 1: Job Offer = YES
Formula Expansion
𝑃(𝑌𝑒𝑠 ∣ 𝐷𝑎𝑡𝑎) = 𝑃(𝐶𝐺𝑃𝐴 ≥ 9 ∣ 𝑌𝑒𝑠) × 𝑃(𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 ∣ 𝑌𝑒𝑠) × 𝑃(𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙
= 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 ∣ 𝑌𝑒𝑠) × 𝑃(𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 = 𝐺𝑜𝑜𝑑 ∣ 𝑌𝑒𝑠) × 𝑃(𝑌𝑒𝑠)
Substitute Values
Feature Value
CGPA ≥9 3/7
Interactiveness = Yes 5/7
Practical = Average 1/7

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

Communication = Good 4/7

Prior P(Yes) 7/10
Calculation
= 3/7 × 5/7 × 1/7 × 4/7 × 7/10
Step-by-step multiply
3×5×1×4×7
=
7 × 7 × 7 × 7 × 10
420
=
24010
≈∗ 0.0175
Case 2: Job Offer = NO
Formula Expansion
𝑃(𝑁𝑜 ∣ 𝐷𝑎𝑡𝑎) = 𝑃(𝐶𝐺𝑃𝐴 ≥ 9 ∣ 𝑁𝑜) × 𝑃(𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 ∣ 𝑁𝑜) × 𝑃(𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙
= 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 ∣ 𝑁𝑜) × 𝑃(𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 = 𝐺𝑜𝑜𝑑 ∣ 𝑁𝑜) × 𝑃(𝑁𝑜)
Substitute Values
Feature Value
CGPA ≥9 1/3
Interactiveness = Yes 1/3
Practical = Average 2/3
Communication = Good 1/3
Prior P(No) 3/10
Calculation
= 1/3 × 1/3 × 2/3 × 1/3 × 3/10
Step-by-step multiply
1×1×2×1×3
=
3 × 3 × 3 × 3 × 10
6
=
810
≈ 0.0074
Step 4: Compare Results
Class Probability
Yes 0.0175
No 0.0074
Final Decision (MAP Rule) : Choose maximum probability
Since:
0.0175 > 0.0074
Final Answer
Job Offer = YES

8.3.2 Brute Force Bayes Algorithm

Brute Force Bayes algorithm is a method that
calculates probability for all hypotheses and selects the best one.

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

8.3.3 Bayes Optimal Classifier

Bayes Optimal classifier is a method that
uses all hypotheses together to make the final decision.
Difference
• MAP → picks one best hypothesis
• Bayes Optimal → uses all hypotheses
Formula:
Class = arg⁡ max⁡ ∑ 𝑃( 𝐶𝑖 ∣ ℎ) 𝑃(ℎ ∣ 𝑇)
𝐶𝑖
ℎ∈𝐻

We have 4 hypotheses:
Posterior Probability Table
Hypothesis P(h | T) P(COVID Positive | h) P(COVID Negative | h)
h1 0.3 0 1
h2 0.1 1 0
h3 0.2 1 0
h4 0.1 1 0
Step 1: MAP Method
Choose hypothesis with maximum P(h|T)
Hypothesis Value
h1 0.3 (MAX)
So:
MAP → choose h1
From table:
• h1 predicts → COVID Negative
MAP Result
COVID Negative
Step 2: Bayes Optimal Classifier
Formula:
∑𝑃(𝐶𝑖 ∣ ℎ) × 𝑃(ℎ ∣ 𝑇)
For COVID Negative
= 0.3 × 1 = 0.3
(only h1 gives negative)
For COVID Positive
= (0.1 × 1) + (0.2 × 1) + (0.1 × 1)
= 0.1 + 0.2 + 0.1 = 0.4

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

Step 3: Compare
Class Value
Negative 0.3
Positive 0.4

Final Answer
COVID Positive (because 0.4 > 0.3)

4.3.4 Gibbs Algorithm

The main drawback of Bayes optimal classifier is that it computes the posterior probability for
all hypotheses in the hypothesis space and then combines the predictions to classify a new
instance.
Gibbs algorithm is a sampling technique which randomly selects a hypothesis from the
hypothesis space according to the posterior probability distribution and classifies a new
instance. It is found that the prediction error occurs twice with the Gibbs algorithm when
compared to Bayes Optimal classifier.

4.4 Naïve Bayes Algorithm for Continuous Attributes.

There are two ways to predict with Naive Bayes algorithm for continuous attributes:
1. Discretize continuous feature to discrete feature.
2. Apply Normal or Gaussian distribution for continuous feature.
Gaussian Naive Bayes Algorithm
In Gaussian Naive Bayes, the values of continuous features are assumed to be sampled from a
Gaussian distribution.

Example 4.4: Assess a student’s performance using Naïve Bayes algorithm for the continuous
attribute. Predict whether a student gets a job offer or not in his final year of the course. The
training dataset T consists of 10 data instances with attributes such as ‘CGPA’ and
‘Interactiveness’ as shown in Table 8.13. The target variable is Job Offer which is classified as
Yes or No for a candidate student.

Solution:
Step 1: Compute the prior probability for the target feature ‘Job Offer’.
Prior probabilities of both the classes are calculated using the same formula.

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

Table: Prior Probability of Target Class

Step 2: Compute Frequency Matrix and Likelihood Probability for each Feature
Likelihood probabilities for a continuous attribute are obtained from Gaussian (Normal)
Distribution.
In the given dataset, CGPA is a continuous attribute. Hence, we apply Gaussian distribution to
calculate the likelihood probability.
Gaussian distribution for each feature is calculated using the formula:
𝑃(𝑋𝑖 = 𝑥𝑘 ∣ 𝐶𝑗 ) = 𝑔(𝑥𝑘 , 𝜇𝑖𝑗 , 𝜎𝑖𝑗 )
Where:
• 𝑋𝑖 → iᵗʰ continuous attribute
• 𝑥𝑘 → value of the attribute
• 𝐶𝑗 → jᵗʰ class of the target feature
• 𝜇𝑖𝑗 → mean of attribute 𝑋𝑖 for class 𝐶𝑗
• 𝜎𝑖𝑗 → standard deviation of attribute 𝑋𝑖 for class 𝐶𝑗

Normal Distribution Formula:

2
(𝑥𝑘 −𝜇𝑖𝑗 )
1 − 2
2𝜎𝑖𝑗
𝑃(𝑋𝑖 = 𝑥𝑘 ∣ 𝐶𝑗 ) = 𝑒
𝜎𝑖𝑗 √2𝜋

Step 2(a): Consider the feature CGPA

In this example CGPA is a continuous attribute, to calculate the likelihood probability for this
continuous attribute, first compute the mean and standard deviation for CGPA with respect to
the target class ‘Job Offer’.
Here, Xᵢ = CGPA
Cⱼ = ‘Job Offer = Yes’
Mean and Standard Deviation for class ‘Job Offer = Yes’ are given as:
μᵢⱼ = μ_CGPA–YES = 8.814286
σᵢⱼ = σ_CGPA–YES = 0.58146

NOTE
Standard Deviation
∑(𝑥𝑖 − 𝜇)2
𝑠=√
𝑛−1

Mean and Standard Deviation for class ‘Job Offer = No’ are given as:
Cⱼ = ‘Job Offer = No’
μᵢⱼ = μ_CGPA–NO = 8.133333
σᵢⱼ = σ_CGPA–NO = 1.011599

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

Once Mean and Standard Deviation are computed, the likelihood probability for any
test value using Gaussian distribution formula can be calculated.

Step 2(b): Consider the feature Interactiveness

Interactiveness is a discrete feature whose probability is calculated as earlier.
below Table shows the frequency matrix for the feature Interactiveness.
Table: Frequency Matrix of Interactiveness

Table 8.16 shows how the likelihood probability is calculated for Interactiveness using
conditional probability.
Table: Likelihood Probability of Interactiveness
Interactiveness P(Job Offer = Yes) P(Job Offer = No)
YES P(Interactiveness = Yes | Job Offer P(Interactiveness = Yes | Job Offer
= Yes) = 5/7 = No) = 1/3
NO P(Interactiveness = No | Job Offer P(Interactiveness = No | Job Offer
= Yes) = 2/7 = No) = 2/3
Step 3: Use Bayes theorem to calculate the probability of all hypotheses.
Consider the test data to be (CGPA = 8.5, Interactiveness = Yes).
• 𝑥𝑘 = 8.5(Test CGPA)
• 𝜇CGPA-YES = 8.814(Mean)
• 𝜎CGPA-YES = 0.581(Standard Deviation)

For the hypothesis

P(Job Offer = Yes) = P(CGPA = 8.5 | Job Offer = Yes) × P(Interactiveness = Yes | Job Offer
= Yes) × P(Job Offer = Yes)

To compute 𝑃(CGPA = 8.5 ∣ Job Offer = Yes)use Gaussian distribution formula:

𝑃(𝑋𝑖 = 𝑥𝑘 ∣ 𝐶𝑗 ) = 𝑔(𝑥𝑘 , 𝜇𝑖𝑗 , 𝜎𝑖𝑗 )
2
(𝑥𝑘 −𝜇𝑖𝑗 )
1 − 2
2𝜎𝑖𝑗
𝑃(𝑋CGPA = 8.5 ∣ 𝐶Job Offer = Yes ) = 𝑒
𝜎𝑖𝑗 √2𝜋
2
(8.5−𝜇CGPA-YES )
1 − 2
2𝜎CGPA-YES
𝑃(𝑋CGPA = 8.5 ∣ 𝐶Job Offer = Yes ) = 𝑒
𝜎CGPA-YES √2𝜋
𝑃(CGPA = 8.5 ∣ Job Offer = Yes) = 𝑔(𝑥𝑘 = 8.5, 𝜇𝑖𝑗 = 8.814, 𝜎𝑖𝑗 = 0.581)
2
(8.5−8.814)
1 −
= 𝑒 2×(0.581)2 = 𝟎. 𝟓𝟗𝟒
0.581√2𝜋
7
𝑃(Job Offer = Yes) =
10

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

𝑃(Job Offer = Yes | Test data)

= 𝑃(CGPA = 8.5 | Job Offer = Yes)
× 𝑃(Interactiveness = Yes | Job Offer = Yes) × 𝑃(Job Offer = Yes)
Or
P(Yes | data)=P(CGPA | Yes)×P(Interactiveness | Yes)×P(Yes)
𝟓 𝟕
= 𝟎. 𝟓𝟗𝟒 × × = 𝟎. 𝟐𝟗𝟕
𝟕 𝟏𝟎

Similarly, for the hypothesis ‘Job Offer = No’:

𝑃(Job Offer = No | Test data)
= 𝑃(CGPA = 8.5 | Job Offer = No) × 𝑃(Interactiveness = Yes | Job Offer = No)
× 𝑃(Job Offer = No)
𝑃(CGPA = 8.5 | Job Offer = No) = 𝑔(𝑥𝑘 = 8.5, 𝜇𝑖𝑗 = 8.133, 𝜎𝑖𝑗 = 1.0116)
2
(8.5−𝜇CGPA-NO )
1 − 2
2𝜎CGPA-NO
𝑃(𝑋CGPA = 8.5 ∣ 𝐶Job Offer = No ) = 𝑒
𝜎CGPA-NO √2𝜋
2
(8.5−8.133)
1 −
= 𝑒 2×(1.0116)2 = 0.369
1.0116√2𝜋
1
𝑃(Interactiveness = Yes | Job Offer = No) =
3
3
𝑃(Job Offer = No) =
10

𝑃(Job Offer = No | Test data) =

𝟏 𝟑
𝟎. 𝟑𝟔𝟗 × × = 𝟎. 𝟎𝟑𝟔𝟗
𝟑 𝟏𝟎

Class Probability
Job Offer = Yes 0.297
Job Offer = No 0.0369

Step 4: Use Maximum A Posteriori (MAP) Hypothesis, ℎ𝑀𝐴𝑃 to classify the test object to the
hypothesis with the highest probability.
Since 𝑃(Job Offer = Yes | Test data)has the highest probability value of 0.297, the test data is
classified as ‘Job Offer = Yes’.

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

CHAPTER 9
Artificial Neural Networks
[Link]
Artificial Neural Network (ANN) is a computational model inspired by the human brain,
consisting of interconnected processing units (neurons) that work together to learn patterns and
make decisions from data.

Introduction to Human Nervous System

The human nervous system consists of billions of neurons that act as processing units. These
neurons help humans to perceive, understand, and respond to the environment.
It enables us to:
• See, hear, and smell
• Understand our surroundings
• Learn, remember, and recognize patterns
Thus, the nervous system acts as a natural information processing system.

Structure of Nervous System

The human nervous system is divided into two main parts:
1. Central Nervous System (CNS)
The Central Nervous System consists of the brain and spinal cord. It is responsible for
processing information and controlling all activities of the body.
2. Peripheral Nervous System (PNS)
The Peripheral Nervous System includes all neurons outside the CNS. It connects different
parts of the body to the CNS and helps in communication.

Types of Neurons
Neurons are classified into three types:
1. Sensory Neurons
These neurons carry information from different parts of the body to the CNS.
2. Motor Neurons
These neurons transmit commands from the CNS to various body parts.
3. Interneurons
These neurons are present in the CNS and connect one neuron to another. They help in
processing and decision-making.

Working of a Neuron
The basic functioning of a neuron involves three steps:
1. Receiving information
2. Processing information
3. Transmitting information to another neuron or body part

Relation to Artificial Neural Networks

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

• Artificial Neural Networks (ANN) are inspired by the human brain structure and
functioning.
• Neurons in the brain are represented as nodes (neurons) in ANN.
• Connections (synapses) between neurons are represented as weights in ANN.
• Signals in the brain are represented as input data in ANN.
• Strength of connections in the brain is similar to weight values in ANN.
• Learning in the brain is similar to weight adjustment (training) in ANN.
• Processing of information in the brain is simulated as mathematical computation in
ANN.
• ANN processes data through input layer, hidden layer, and output layer.
• ANN can perform tasks like classification, prediction, and pattern recognition.

[Link] Neurons
We study biological neurons in Machine Learning to understand and design Artificial Neural
Networks inspired by the human brain for learning and decision-making.
• A biological neuron is the basic unit of the nervous system.
• It has four main parts:
o Dendrites
o Soma (cell body)
o Axon
o Synapse
• Dendrites receive signals from other neurons.
• Soma (cell body) processes the information.
• Axon carries signals away from the cell body.
• Synapse connects one neuron to another.
• A neuron fires only when input exceeds a threshold value.
• Signals are transmitted as electrical impulses (spikes).
• A neuron can connect to thousands of other neurons.
• Neurons form a network to process information and give response.

Dendrites receive input signals, the cell body (soma) processes them, and the axon transmits
the output to another neuron through the synapse.

[Link] Neurons
Artificial neurons are similar to biological neurons and are also called nodes. A neuron (node)
can receive one or more input signals and process them.

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

Artificial neurons are connected to each other through connection links. Each connection link
has an associated value called a synaptic weight, which determines the strength of the
connection.
These neurons work together to process information, similar to how neurons function in the
human brain.

3.1 Simple Model of an Artificial Neuron

The first mathematical model of a neuron was developed by McCulloch and Pitts in 1943.
This model explains how a neuron works.
It includes two main steps:
1. The neuron receives weighted inputs from other neurons.
2. It uses a threshold (activation) function to decide the output.
The inputs received by the neuron are combined as a weighted sum, which is then given to the
activation function. If this sum is greater than the threshold value, the neuron becomes active
(fires) The mathematical model of neuron is shown in Figure 10.3.

The neuron is a basic processing unit that takes inputs 𝑥1 , 𝑥2 , 𝑥3 , … , 𝑥𝑛 and their corresponding
weights 𝑤1 , 𝑤2 , 𝑤3 , … , 𝑤𝑛 .
The weighted sum of inputs is calculated using Net-sum Eq.(10.1):
Net-sum = ∑𝑥𝑖 𝑤𝑖

The activation function used here is a binary step function. It gives output:
• 1, if Net-sum is greater than or equal to threshold value 𝜃
• 0, if Net-sum is less than threshold value 𝜃
So, the activation function is applied to Net-sum as shown in Eq. (10.2):
𝑓(𝑥) = Activation function (Net-sum)

The final output of the neuron is:

1 if 𝑓(𝑥) ≥ 𝜃
𝑌={
0 if 𝑓(𝑥) < 𝜃

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

The McCulloch & Pitts neuron model can represent only a few Boolean functions. A Boolean
function works with binary inputs (0 or 1) and gives a binary output.
For example, in an AND function, the neuron gives output (fires) only when all inputs are 1.
In an OR function, the neuron gives output even if at least one input is 1.
Also, in this model, the weights and threshold values are fixed.

3.2 Artificial Neural Network Structure

Artificial Neural Network (ANN) is similar to the human brain and shows some level of
intelligence. It has a network structure that can be represented as a directed graph with neurons
(nodes) and connections (edges).
The nodes are arranged in layers, and they process information in parallel. The network usually
has three layers:
• Input layer
• Hidden layer
• Output layer
The input layer receives input data 𝑥1 , 𝑥2 , … , 𝑥𝑛 and sends it to the hidden layer.
The connections between layers have weights, called synaptic weights.
Each neuron performs calculations using the input data. If the weighted sum of inputs is greater
than the threshold, the neuron becomes active (fires).
Each neuron uses an activation function to decide its output.
The neuron calculates the sum of input values multiplied by their weights and may add a bias.
Then, the activation function converts this value into a non-linear output.
Finally, the output layer gives the result as a single value.

3.3 Activation Functions

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

Activation functions are mathematical functions used in each neuron of a neural network.
They convert the input signals into output signals.
The activation function decides whether a neuron should fire (activate) or not based on the
input it receives.
These functions also help to keep the output values in a specific range, such as:
• Between 0 and 1
• Or between –1 and +1
Activation functions can be of two types:
• Linear functions
• Non-linear functions
Linear functions are useful when input values can be divided into two groups. They are
commonly used in binary classification problems.
Non-linear functions are continuous functions that map input values into a range like (0, 1) or
(–1, 1). These are useful for handling complex data such as images, audio, and video, and
for learning high-dimensional data.
Below are some of the activation functions used in ANNs:
1. Identity Function or Linear Function
𝑓(𝑥) = 𝑥∀𝑥(10.4)

The value of 𝑓(𝑥)increases linearly or proportionally with the value of 𝑥. This function is
useful when we do not want to apply any threshold. The output would be just the weighted
sum of input values. The output value ranges between −∞and +∞.

2. Binary Step Function

1 if 𝑓(𝑥) ≥ 𝜃
𝑓(𝑥) = { (10.5)
0 if 𝑓(𝑥) < 𝜃

The output value is binary, i.e., 0 or 1 based on the threshold value 𝜃. If value of 𝑓(𝑥)is
greater than or equal to 𝜃, it outputs 1 or else it outputs 0.

3. Bipolar Step Function

1 if 𝑓(𝑥) ≥ 𝜃
𝑓(𝑥) = { (10.6)
−1 if 𝑓(𝑥) < 𝜃

The output value is bipolar, i.e., +1 or −1 based on the threshold value 𝜃. If value of 𝑓(𝑥)is
greater than or equal to 𝜃, it outputs +1 or else it outputs −1.

4. Sigmoidal Function or Logistic Function

1
𝜎(𝑥) = (10.7)
1 + 𝑒 −𝑥

It is a widely used non-linear activation function which produces an S-shaped curve and the
output values are in the range of 0 and 1. It has a vanishing gradient problem, i.e., no change
in the prediction for very low input values and very high input values.

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

5. Bipolar Sigmoid Function

1 − 𝑒 −𝑥
𝜎(𝑥) = (10.8)
1 + 𝑒 −𝑥

It outputs values between −1 and +1.

6. Ramp Functions
1 if 𝑥 > 1
𝑓(𝑥) = {𝑥 if 0 ≤ 𝑥 ≤ 1 (10.9)
0 if 𝑥 < 0

It is a linear function whose upper and lower limits are fixed.

7. Tanh – Hyperbolic Tangent Function

The Tanh function is a scaled version of the sigmoid function which is also non-linear. It also
suffers from the vanishing gradient problem. The output values range between −1 and 1.
2
tanh⁡(𝑥) = − 1(10.10)
1 + 𝑒 −2𝑥

8. ReLU – Rectified Linear Unit Function

This activation function is a typical function generally used in deep learning neural network
models in the hidden layers. It avoids or reduces the vanishing gradient problem. This
function outputs a value of 0 for negative input values and works like a linear function if the
input values are positive.
𝑥 if 𝑥 ≥ 0
𝑟(𝑥) = max⁡(0, 𝑥) = { (10.11)
0 if 𝑥 < 0

9. Softmax Function
This is a non-linear function used in the output layer that can handle multiple classes. It
calculates the probability of each target class which ranges between 0 and 1. The probability
of the input belonging to a particular class is computed by dividing the exponential of the
given input value by the sum of the exponential values of all the inputs.
𝑒 𝑥𝑖
𝑠(𝑥𝑖 ) = 𝑘 ,where 𝑖 = 0 … 𝑘(10.12)
∑𝑗=0 𝑒 𝑥𝑗
[Link] and Learning Theory
The first neural network model called Perceptron was developed by Frank Rosenblatt in
1958. It is a linear binary classifier used in supervised learning.
He improved the McCulloch & Pitts neuron model by combining it with the Hebbian
learning rule for adjusting weights.
He introduced:
• Variable weights (weights can change)

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

• An extra input called bias

He suggested that artificial neurons can learn weights and threshold values from data
automatically.
He developed a supervised learning algorithm that helps the neuron learn correct weights
from training data.
The perceptron model (shown in Figure 10.5) consists of 4 steps:
1. Inputs from other neurons
2. Weights and bias
3. Net sum
4. Activation function

Thus, the modified neuron model takes a set of inputs 𝑥1 , 𝑥2 , … , 𝑥𝑛 , their corresponding
weights 𝑤1 , 𝑤2 , … , 𝑤𝑛 , and a bias. The summation function Net-sum calculates the weighted
sum of the inputs received by the neuron.
Net-sum = ∑𝑥𝑖 𝑤𝑖 (10.13)

After calculating the Net-sum, the bias value is added to it and the result is passed to the
activation function as shown below:
𝑓(𝑥) = Activation function (Net-sum + bias)(10.14)

The activation function is a binary step function. It gives output:

• 1, if 𝑓(𝑥) ≥ 𝜃
• 0, if 𝑓(𝑥) < 𝜃
Then, the final output of the neuron is:
1 if 𝑓(𝑥) ≥ 𝜃
𝑌={ (10.15)
0 if 𝑓(𝑥) < 𝜃

Before learning how a neural network works, let us understand how a perceptron model
works.

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

Example 1: Consider a perceptron to represent the Boolean function AND with the initial
weights 𝑤1 = 0.3, 𝑤2 = −0.2, learning rate 𝛼 = 0.2and bias 𝜃 = 0.4as shown in Figure 10.6.
The activation function used here is the Step function 𝑓(𝑥)which gives the output value as
binary, i.e., 0 or 1. If value of 𝑓(𝑥)is greater than or equal to 0, it outputs 1 or else it outputs 0.
Design a perceptron that performs the Boolean function AND and update the weights until the
Boolean function gives the desired output.

Solution: Desired output for Boolean function AND is shown in Table 10.1.
Table 10.1: AND Truth Table
x₁ x₂ Y_des
0 0 0
0 1 0
1 0 0
1 1 1

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

For each Epoch, weighted sum is calculated and the activation function is applied to compute
the estimated output 𝑌𝑒𝑠𝑡 . Then, 𝑌𝑒𝑠𝑡 is compared with 𝑌𝑑𝑒𝑠 to find the error. If there is an error,
the weights are updated.
W1 and w2 formula : Wnew=wold+α×Error×x

Tables 10.2 to 10.5 show how the weights are updated in the four Epochs.
Table 10.2: Epoch 1

Epoch x₁ x₂ Y_des Y_est (Net=(x1w1+x2w2 Error=Ydes w₁ w₂ Status

)−θ) −Yest
1 0 0 0 Step((0 × 0.3 + 0 × -0.2) 0 0.3 -0.2 No
− 0.4) = 0 change
0 1 0 Step((0 × 0.3 + 1 × -0.2) 0 0.3 -0.2 No
− 0.4) = 0 change
1 0 0 Step((1 × 0.3 + 0 × -0.2) 0 0.3 -0.2 No
− 0.4) = 0 change
1 1 1 Step((1 × 0.3 + 1 × -0.2) 1 0.5 0 Change
− 0.4) = 0

For input (1, 1) the weights are updated as follows:

Formula Δ𝑤 = 𝛼 × 𝑒(𝑡) × 𝑥
Δ𝑤1 = 0.2 × 1 × 1 = 0.2
𝑤1 = 0.3 + 0.2 = 0.5
Δ𝑤2 = 0.2 × 1 × 1 = 0.2
𝑤2 = −0.2 + 0.2 = 0

Final Updated Weights

• 𝑤1 = 0.5
• 𝑤2 = 0

Table 10.2: Epoch 2

Epoch x₁ x₂ Y_des Y_est Error w₁ w₂ Status

2 0 0 0 Step((0 × 0.5 + 0 × 0) − 0.4) = 0 0 0.5 0 No change
0 1 0 Step((0 × 0.5 + 1 × 0) − 0.4) = 0 0 0.5 0 No change
1 0 0 Step((1 × 0.5 + 0 × 0) − 0.4) = 1 -1 0.3 0 Change
1 1 1 Step((1 × 0.3 + 1 × 0) − 0.4) = 0 1 0.5 0.2 Change

For input (1, 0), the weights are updated as follows:

Δ𝑤1 = 𝛼 × 𝑒(𝑡) × 𝑥1 = 0.2 × (−1) × 1 = −0.2
𝑤1 = 𝑤1 + Δ𝑤1 = 0.5 + (−0.2) = 0.3
Δ𝑤2 = 𝛼 × 𝑒(𝑡) × 𝑥2 = 0.2 × (−1) × 0 = 0
𝑤2 = 𝑤2 + Δ𝑤2 = 0 + 0 = 0

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

For input (1, 1), the weights are updated as follows:

Δ𝑤1 = 𝛼 × 𝑒(𝑡) × 𝑥1 = 0.2 × 1 × 1 = 0.2
𝑤1 = 𝑤1 + Δ𝑤1 = 0.3 + 0.2 = 0.5
Δ𝑤2 = 𝛼 × 𝑒(𝑡) × 𝑥2 = 0.2 × 1 × 1 = 0.2
𝑤2 = 𝑤2 + Δ𝑤2 = 0 + 0.2 = 0.2
Final Updated Weights
• 𝑤1 = 0.5
• 𝑤2 = 0.2
Table 10.4: Epoch 3
Epoch x₁ x₂ Y_des Y_est Error w₁ w₂ Status
3 0 0 0 Step((0 × 0.5 + 0 × 0.2) − 0.4) = 0 0 0.5 0.2 No change
0 1 0 Step((0 × 0.5 + 1 × 0.2) − 0.4) = 0 0 0.5 0.2 No change
1 0 0 Step((1 × 0.5 + 0 × 0.2) − 0.4) = 1 -1 0.3 0.2 Change
1 1 1 Step((1 × 0.3 + 1 × 0.2) − 0.4) = 1 0 0.3 0.2 No change
For input (1, 0), the weights are updated as follows:
Δ𝑤1 = 𝛼 × 𝑒(𝑡) × 𝑥1 = 0.2 × (−1) × 1 = −0.2
𝑤1 = 𝑤1 + Δ𝑤1 = 0.5 + (−0.2) = 0.3
Δ𝑤2 = 𝛼 × 𝑒(𝑡) × 𝑥2 = 0.2 × (−1) × 0 = 0
𝑤2 = 𝑤2 + Δ𝑤2 = 0.2 + 0 = 0.2
Final Updated Weights
• 𝑤1 = 0.3
• 𝑤2 = 0.2
Table 10.5: Epoch 4
Epoch x₁ x₂ Y_des Y_est Error w₁ w₂ Status
4 0 0 0 Step((0 × 0.3 + 0 × 0.2) − 0.4) = 0 0 0.3 0.2 No change
0 1 0 Step((0 × 0.3 + 1 × 0.2) − 0.4) = 0 0 0.3 0.2 No change
1 0 0 Step((1 × 0.3 + 0 × 0.2) − 0.4) = 0 0 0.3 0.2 No change
1 1 1 Step((1 × 0.3 + 1 × 0.2) − 0.4) = 1 0 0.3 0.2 No change

It is observed that with 4 Epochs, the perceptron learns and the weights are updated to 0.3
and 0.2, with which the perceptron gives the desired output of a Boolean AND function.
Final Weights:
𝑤1 = 0.3, 𝑤2 = 0.2

Final Output:
Perceptron correctly implements AND function

4.1 XOR Problem

A perceptron model can solve only those Boolean functions which are linearly separable.
However, the XOR problem was identified in 1969 by Minsky and Papert.

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

An XOR function gives output 1 when the two inputs are different, and 0 when they are the
same. The truth table of XOR function is shown below.

Table 10.6: XOR Truth Table

x₁ x₂ Y
0 0 0
0 1 1
1 0 1
1 1 0

Since XOR is not linearly separable, a single-layer perceptron cannot classify it. To solve this
problem, a multi-layer perceptron (MLP) was introduced.
Initially, the MLP did not achieve much success due to lack of proper training methods. Later,
in 1974, Werbos introduced the backpropagation algorithm for training multi-layer networks,
and in 1986, a general backpropagation algorithm for multi-layer perceptron networks was
introduced by Rumelhart and McClelland.
After this, ANN and deep neural networks became successful in solving complex problems.
The MLP can solve non-linearly separable problems like XOR.

4.2 Delta Learning Rule and Gradient Descent

In neural networks, learning is done by adjusting the weights to reduce the difference between
the desired output and the predicted output.
This difference is called error or cost function. Since this function is continuous, it can be
minimized.
This learning method is called the Delta rule (Widrow-Hoff rule / Adaline rule) and is used
to train the network.
Training Error Formula
1
Training Error = ∑(𝑂𝑑𝑒𝑠𝑖𝑟𝑒𝑑 − 𝑂𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 )2
2

• 𝑂𝑑𝑒𝑠𝑖𝑟𝑒𝑑 → actual (target) output

• 𝑂𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 → predicted output
Gradient Descent
Gradient descent is an optimization method used to minimize the error.
• It reduces error by moving towards the minimum value
• It moves in the negative direction of the gradient
• The step size depends on the learning rate
Gradient descent is the base of backpropagation algorithm used in Multi-Layer Perceptron
(MLP).

[Link] of Artificial Neural Networks

Artificial Neural Networks (ANNs) consist of multiple neurons arranged in layers.
There are different types of ANNs based on:
• Network structure

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

• Activation functions used

• Learning rules
In an ANN, there are three main layers:
• Input layer
• Hidden layer
• Output layer
A general ANN consists of:
• One input layer
• One output layer
• Zero or more hidden layers
5.1 Feed Forward Neural Network
This is the simplest neural network where neurons are arranged in layers and the information
flows only in one direction (forward). This model may or may not have a hidden layer, and
there is no backpropagation.
Based on the number of hidden layers, it is divided into:
• Single-layer feedforward network
• Multi-layer feedforward network
These networks are simple to design and easy to maintain. They are used for simple
classification and basic image processing.

5.2 Fully Connected Neural Network

In this network, each neuron in one layer is connected to all neurons in the next layer.
This means every neuron shares information with all neurons in the next layer, making it a
fully connected structure.

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

5.3 Multi-Layer Perceptron (MLP)

This type of ANN has multiple layers:
• One input layer
• One output layer
• One or more hidden layers
Each neuron in a layer is connected to all neurons in the next layer.
Information flows in two directions:
• Forward direction → input is processed
• Backward direction → error is sent back (backpropagation)
If the output is wrong, the error is sent back to adjust weights and bias to improve results.
MLP is widely used in deep learning for:
• Classification
• Speech recognition
• Medical diagnosis
These networks are powerful but more complex and slower.

5.4 Feedback Neural Network

In this network, there are feedback connections between neurons.
• Information flows in both directions
• Output can be sent back as input
This makes the network more dynamic during training.
Connections may exist:
• Between neurons in the same layer
• Or between different layers

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

[Link] APPLICATIONS OF ARTIFICIAL NEURAL NETWORKS

ANN learning mechanisms are used in many complex applications that involve modelling of
non-linear processes. ANN is a useful model that can handle even noisy and incomplete data.
They are used to model complex patterns, recognize patterns and solve prediction problems
like humans in many areas such as:
1. Real-time applications: Face recognition, emotion detection, self-driving cars,
navigation systems, routing systems, target tracking, vehicle scheduling, etc.
2. Business applications: Stock trading, sales forecasting, customer behaviour
modelling, market research and analysis, etc.
3. Banking and Finance: Credit and loan forecasting, fraud and risk evaluation,
currency price prediction, real-estate appraisal, etc.
4. Education: Adaptive learning software, student performance modelling, etc.
5. Healthcare: Medical diagnosis or mapping symptoms to a medical case, image
interpretation and pattern recognition, drug discovery, etc.
6. Other Engineering Applications: Robotics, aerospace, electronics, manufacturing,
communications, chemical analysis, food research, etc.
[Link] AND DISADVANTAGES OF ANN
Advantages of ANN
1. ANN can solve complex problems that involve non-linear processes.
2. ANN can learn and recognize complex patterns and solve problems similar to
humans.
3. ANN has parallel processing ability and can give results faster.
4. It can work even with less knowledge and can handle incomplete and noisy data.
5. ANN works well with large datasets and performs better than many other learning
methods.

[Link] of ANN
1. ANN requires powerful processors because training needs many iterations (epochs).
Each neuron needs computation, which becomes difficult for large networks with
large data.

Prof. Mamatha N B Dept. of CSE(AIML)

Machine learning BCS602

2. ANN works like a “black box”, so it is very difficult to understand what happens
inside the hidden layers. Also, it is hard to understand how data is represented at each
layer.
3. Building ANN models is complex and takes more time.
4. ANN usually needs a large amount of data and does not perform well with small
datasets.
5. ANN requires more computational resources compared to traditional machine
learning methods.

[Link] OF ARTIFICIAL NEURAL NETWORKS

The major challenges in real-time ANN applications are:
1. Training a neural network is difficult. Problems like overfitting and underfitting can
occur if the data is not proper. It is also hard to apply the model to real-world data if it
is trained on simulated data. Also, ANN needs a large amount of training data to work
well in real-time.
2. Finding the correct weights and bias values is difficult, and it is hard to get the best
(optimal) model.

Prof. Mamatha N B Dept. of CSE(AIML)