Machine learning BCS602

Module - 04
Bayesian Learning
Bayesian Learning is a method of learning that uses probability to make predictions and
improves its results by updating its knowledge when new data is received.
Or
It is a method that learns from data by updating probabilities when new information is added.
4.1 .INTRODUCTION TO PROBABILITY-BASED LEARNING
Probability-based learning is a method that combines prior knowledge (prior probability) with
observed data to make predictions.
• It is one of the most important practical learning methods used in machine learning to
handle real-world data.
• It combines previously known information (prior probability) with new data (observed
data) to improve prediction accuracy.
• It uses the concept of probability theory to model randomness, uncertainty, and noise
present in the data.
• It helps in predicting future events by analyzing patterns and probabilities from the
given dataset.
• This method is widely used for modelling large datasets and extracting useful
information from them.
• It uses Bayes’ rule to infer unknown values, update beliefs, and learn from data step by
step.
Probabilistic Model:
• In a probabilistic model, randomness is considered, and the output is expressed in the form
of probability distribution instead of a single value.
• It is more suitable for real-world problems where uncertainty and incomplete data are present.
Deterministic Model:
• In a deterministic model, there is no randomness, and the same input always produces the
same output.
• It gives a single fixed result and does not handle uncertainty.
4.2 Fundamentals of Bayes Theorem
• Naïve Bayes model is based on Bayes theorem
• It works using three types of probabilities:
– Prior probability
– Likelihood probability
– Posterior probability
1. Prior Probability
• Prior probability is the initial probability of an event before any evidence is observed
• It represents the belief about an event before collecting new data
• It is based on past knowledge or experience
2. Likelihood Probability
• Likelihood probability is the probability of observing the data given a hypothesis
• It is written as:
P(Evidence | Hypothesis)
• It shows how likely the evidence is when the hypothesis is true
3. Posterior Probability
• Posterior probability is the updated probability after considering new evidence

Prof. Mamtha N B Dept. of CSE(AIML)

Machine learning BCS602

• It is written as:
P(Hypothesis | Evidence)
• It represents the final belief after learning from data
Important Relation:
• Posterior probability is calculated using prior probability and new evidence
Posterior Probability = Prior Probability + New Evidence (conceptual idea)
4.3 Classification Using Bayes Model(Bayes Theorem)
Classification is the process of finding which class an object belongs to.
The Bayes model helps to choose the best class by using probability.
• Naïve Bayes classification works based on Bayes theorem
• Bayes rule is used to calculate the posterior probability
• It helps to select the most probable hypothesis (class) from the given data
Important Terms:
Hypothesis (h):
– The class we want to predict
– Example: Pass / Fail
Evidence (E):
– The given data or input
– Example: Marks, Attendance
Bayes Formula:
𝑃(𝐸 ∣ ℎ) 𝑃(ℎ)
𝑃(ℎ ∣ 𝐸) =
𝑃(𝐸)

Terms
1. Prior Probability – P(h)
• It is the probability of hypothesis before seeing the data
• It is also called the initial guess
2. Likelihood Probability – P(E|h)
• It is the probability of evidence given the hypothesis
• It shows how well the data matches the class
3. Evidence – P(E)
• It is the probability of evidence from the dataset
• It is constant for all classes
4. Posterior Probability – P(h|E)
• It is the final probability after observing the data
• It is used to make the final decision
How Classification Works:
Step 1: Take all possible classes
– Example: Pass, Fail
Step 2: Calculate probability for each class using formula
Step 3: Compare the values
Step 4: Choose the class with highest probability.
Simple Example
Classes: Pass / Fail
Input: Good marks, High attendance
Calculate probability for both classes

Prof. Mamtha N B Dept. of CSE(AIML)

Machine learning BCS602

If Pass probability is higher → Student is Pass

Decision Criteria in Bayes Classification

• In classification, we need a method to choose the best hypothesis (class)
• MAP and ML are decision criteria (methods) used for selecting the best hypothesis
1. Maximum A Posteriori (MAP) Hypothesis
MAP hypothesis is the hypothesis that has the maximum posterior probability It is also called
the most probable hypothesis
• Given a set of candidate hypotheses, we calculate posterior probability for each
• The hypothesis with the highest posterior value is selected
• It uses Bayes theorem for calculation
Formula:
ℎ𝑀𝐴𝑃 = arg⁡ max⁡ 𝑃(ℎ ∣ 𝐸)
𝑃(𝐸 ∣ ℎ) 𝑃(ℎ)
= arg⁡ max⁡
𝑃(𝐸)
• Since 𝑃(𝐸)is constant, we can write:
ℎ𝑀𝐴𝑃 = arg⁡ max⁡ 𝑃(𝐸 ∣ ℎ) 𝑃(ℎ)
2. Maximum Likelihood (ML) Hypothesis
ML hypothesis is the hypothesis that has the maximum likelihood probability
• It is used when all hypotheses are equally probable
• Only likelihood 𝑃(𝐸 ∣ ℎ)is considered
• Prior probability is ignored
Formula:
ℎ𝑀𝐿 = arg⁡ max⁡ 𝑃(𝐸 ∣ ℎ)
• ML selects hypothesis based only on evidence, not prior knowledge
Correctness of Bayes Theorem
Given:
• Consider two events A and B
• 𝑃(𝐴) = 5/8
• 𝑃(𝐵) = 4/8
Conditional Probabilities:
• 𝑃(𝐴 ∣ 𝐵) = 2/4
• 𝑃(𝐵 ∣ 𝐴) = 2/5
Verification:
• Using Bayes theorem:
𝑃(𝐵 ∣ 𝐴) 𝑃(𝐴) 2
𝑃(𝐴 ∣ 𝐵) = =
𝑃(𝐵) 4
𝑃(𝐴 ∣ 𝐵) 𝑃(𝐵) 2
𝑃(𝐵 ∣ 𝐴) = =
𝑃(𝐴) 5

Prof. Mamtha N B Dept. of CSE(AIML)

Machine learning BCS602

Given:
• P(Participating) = 0.9
• P(Sick | Participating) = 0.2
• P(Sick) = 0.4
We need to find:
• P(Participating | Sick)
Step 1: Write Bayes Formula
𝑃(𝑆𝑖𝑐𝑘 ∣ 𝑃𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔) × 𝑃(𝑃𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔)
𝑃(𝑃𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔 ∣ 𝑆𝑖𝑐𝑘) =
𝑃(𝑆𝑖𝑐𝑘)
Step 2: Substitute Values
0.2 × 0.9
=
0.4
Step 3: Calculate
0.18
= = 0.45
0.4
Probability = 0.45 (or 45%)

4.3.1 Naïve Bayes Algorithm

Naïve Bayes is a supervised classification algorithm used for binary and multi-class problems
It works based on Bayes theorem
• It assumes that all features are independent of each other
• Each feature contributes independently to the final classification
• All features are given equal importance (weightage)
• It is a simple and efficient algorithm used for classification tasks
• It works well with large datasets
• Even though features may be dependent in real life, the algorithm assumes independence
(this is why it is called “Naïve”)
• During classification, each feature adds its own probability to the final result
Advantages:
• Simple and easy to implement
• Works fast even for large datasets
• Requires less training data
• Gives good performance in many real-world problems

Prof. Mamtha N B Dept. of CSE(AIML)

Machine learning BCS602

Applications:
• Text classification (spam detection)
• Recommendation systems
• Face recognition

Solution
The training dataset T consists of 10 data instances with attributes such as CGPA,
Interactiveness, Practical Knowledge, and Communication Skills, as shown in Table 8.1.
The target variable is Job Offer, which is classified as Yes or No for a candidate student.
Step 1: Compute Prior Probability
The target feature Job Offer has two classes:
• Yes
• No
Hence, it is a binary classification problem.
Given a student instance, we need to classify whether:
Job Offer = Yes or Job Offer = No
Frequency Observation
From the training dataset:
• Number of instances with Job Offer = Yes = 7
• Number of instances with Job Offer = No = 3

Prof. Mamtha N B Dept. of CSE(AIML)

Machine learning BCS602

Calculation of Prior Probability

The prior probability is calculated as:
Number of instances of a class
Prior Probability =
Total number of instances

Final Prior Probabilities

7
𝑃(Job Offer = Yes) =
10
3
𝑃(Job Offer = No) =
10

Step 2(a): Likelihood Probability for CGPA

Table 8.4 shows how the likelihood probability is calculated for CGPA using conditional
probability.
Likelihood Probability Table (CGPA)
CGPA P(Job Offer = Yes) P(Job Offer = No)
≥9 P(CGPA ≥9 Yes) = 3/7
≥8 P(CGPA ≥8 Yes) = 4/7
<8 P(CGPA <8 Yes) = 0/7
• Likelihood probability is defined as the probability of evidence given the hypothesis
• It is denoted as:
𝑃(Evidence ∣ Hypothesis)
• It tells how likely the evidence is when the hypothesis is true

Prof. Mamtha N B Dept. of CSE(AIML)

Machine learning BCS602

How it is Calculated
Number of instances with that feature and class
𝑃(Feature value ∣ Class) =
Total number of instances of that class
Example Calculation
P(CGPA ≥9 | Job Offer = Yes)
• Number of students with:
CGPA ≥9 AND Job = Yes = 3
• Total students with Job = Yes = 7
𝑃(𝐶𝐺𝑃𝐴 ≥ 9 ∣ 𝑌𝑒𝑠) = 3/7
P(CGPA ≥9 | Job Offer = No)
• Count = 1
• Total Job = No = 3
𝑃(𝐶𝐺𝑃𝐴 ≥ 9 ∣ 𝑁𝑜) = 1/3
P(CGPA ≥8 | Job Offer = No)
• Count = 0
𝑃(𝐶𝐺𝑃𝐴 ≥ 8 ∣ 𝑁𝑜) = 0/3 = 0
P(CGPA <8 | Job Offer = No)
• Count = 2
𝑃(𝐶𝐺𝑃𝐴 < 8 ∣ 𝑁𝑜) = 2/3

Step 2(b): Likelihood Probability of Interactiveness

Formula Used
Count of (Feature + Class)
𝑃(Feature ∣ Class) =
Total count of Class
Step 1: From Dataset
We already know:
• Job Offer = Yes → 7 students
• Job Offer = No → 3 students
Step 2: Count Values
Interactiveness = YES

Prof. Mamtha N B Dept. of CSE(AIML)

Machine learning BCS602

From dataset:
• YES & Job = Yes → 5 students
• YES & Job = No → 1 student
Interactiveness = NO
• NO & Job = Yes → 2 students
• NO & Job = No → 2 students
Step 3: Calculate Probabilities
For Job Offer = YES
1. P(Interactiveness = Yes | Yes)
5
=
7
2. P(Interactiveness = No | Yes)
2
=
7
For Job Offer = NO
3. P(Interactiveness = Yes | No)
1
=
3
4. P(Interactiveness = No | No)
2
=
3

Step 2(c): Practical Knowledge

We are finding:
𝑃(Practical Knowledge ∣ Job Offer)
This is called Likelihood Probability
“How likely is a student’s practical knowledge given Job = Yes/No?”
Step 1: Extract Required Data
From dataset, take only:
Practical Knowledge + Job Offer
Row Practical Knowledge Job
1 Very Good Yes
2 Good Yes

Prof. Mamtha N B Dept. of CSE(AIML)

Machine learning BCS602

3 Average No
4 Average No
5 Good Yes
6 Good Yes
7 Good No
8 Very Good Yes
9 Good Yes
10 Average Yes
Step 2: Count Values (Frequency Matrix)
We count how many times each value appears with Yes/No.
Very Good
• With Job = Yes → Row 1, 8 → 2
• With Job = No → 0
So:
| Very Good | 2 | 0 |
Average
• With Job = Yes → Row 10 → 1
• With Job = No → Row 3, 4 → 2
So:
| Average | 1 | 2 |
Good
• With Job = Yes → Row 2,5,6,9 → 4
• With Job = No → Row 7 → 1
So:
| Good | 4 | 1 |
Step 3: Final Frequency Table
Practical Knowledge Job = Yes Job = No
Very Good 2 0
Average 1 2
Good 4 1
Total 7 3
Step 4: Apply Formula
Count of (Feature + Class)
𝑃(Feature ∣ Class) =
Total of Class
Step 5: Calculate Each Probability
For Job Offer = YES (Total = 7)
1. Very Good
2
𝑃(Very Good ∣ 𝑌𝑒𝑠) =
7
Out of 7 students who got job, 2 have very good knowledge
2. Average
1
𝑃(Average ∣ 𝑌𝑒𝑠) =
7
Only 1 student with average knowledge got job
3. Good

Prof. Mamtha N B Dept. of CSE(AIML)

Machine learning BCS602

4
𝑃(Good ∣ 𝑌𝑒𝑠) =
7
Most students with good knowledge got job
For Job Offer = NO (Total = 3)
4. Very Good
0
𝑃(Very Good ∣ 𝑁𝑜) = =0
3
No student with very good knowledge failed
5. Average
2
𝑃(Average ∣ 𝑁𝑜) =
3
Most students who failed had average knowledge
6. Good
1
𝑃(Good ∣ 𝑁𝑜) =
3
Few students with good knowledge failed
Step 6: Final Likelihood Table
Practical Knowledge P(Job Offer = Yes) P(Job Offer = No)
Very Good 2/7 0
Average 1/7 2/3
Good 4/7 1/3

Step 2(d): Communication Skills

Likelihood probability:
𝑃(Communication Skill ∣ Job Offer)

“How likely is a student’s communication skill given Job = Yes/No?”

Take Data from Dataset
Focus only on:
Communication Skills + Job Offer

Prof. Mamtha N B Dept. of CSE(AIML)

Machine learning BCS602

Row Communication Job

1 Good Yes
2 Moderate Yes
3 Poor No
4 Good No
5 Moderate Yes
6 Moderate Yes
7 Poor No
8 Good Yes
9 Good Yes
10 Good Yes
Count (Frequency Matrix)
Good
• Job = Yes → rows 1,8,9,10 → 4
• Job = No → row 4 → 1
Moderate
• Job = Yes → rows 2,5,6 → 3
• Job = No → 0
Poor
• Job = Yes → 0
• Job = No → rows 3,7 → 2
Total
• Job = Yes → 7
• Job = No → 3
Apply Formula
Count
𝑃(Feature ∣ Class) =
Total of Class
Calculate Probabilities
For Job Offer = YES (Total = 7)
• P(Good | Yes) = 4/7
• P(Moderate | Yes) = 3/7
• P(Poor | Yes) = 0/7 = 0

For Job Offer = NO (Total = 3)

• P(Good | No) = 1/3
• P(Moderate | No) = 0/3 = 0
• P(Poor | No) = 2/3
Final Table
Communication P(Job Offer = Yes) P(Job Offer = No)
Good 4/7 1/3
Moderate 3/7 0
Poor 0 2/3

Prof. Mamtha N B Dept. of CSE(AIML)

Machine learning BCS602

Step 3: Apply Bayes Theorem

Given Test Data
• CGPA ≥ 9
• Interactiveness = Yes
• Practical Knowledge = Average
• Communication Skills = Good
We must find:
𝑃(Job Offer = Yes | Test Data)
𝑃(Job Offer = No | Test Data)
Bayes Formula Used
𝑃(𝐷𝑎𝑡𝑎 ∣ 𝐶𝑙𝑎𝑠𝑠) × 𝑃(𝐶𝑙𝑎𝑠𝑠)
𝑃(𝐶𝑙𝑎𝑠𝑠 ∣ 𝐷𝑎𝑡𝑎) =
𝑃(𝐷𝑎𝑡𝑎)

Since P(Data) is same for both → we ignore it

Case 1: Job Offer = YES
Formula Expansion
𝑃(𝑌𝑒𝑠 ∣ 𝐷𝑎𝑡𝑎) = 𝑃(𝐶𝐺𝑃𝐴 ≥ 9 ∣ 𝑌𝑒𝑠) × 𝑃(𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 ∣ 𝑌𝑒𝑠) × 𝑃(𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙
= 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 ∣ 𝑌𝑒𝑠) × 𝑃(𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 = 𝐺𝑜𝑜𝑑 ∣ 𝑌𝑒𝑠) × 𝑃(𝑌𝑒𝑠)
Substitute Values
Feature Value
CGPA ≥9 3/7
Interactiveness = Yes 5/7
Practical = Average 1/7
Communication = Good 4/7
Prior P(Yes) 7/10
Calculation
= 3/7 × 5/7 × 1/7 × 4/7 × 7/10
Step-by-step multiply
3×5×1×4×7
=
7 × 7 × 7 × 7 × 10
420
=
24010
≈∗ 0.0175
Case 2: Job Offer = NO
Formula Expansion
𝑃(𝑁𝑜 ∣ 𝐷𝑎𝑡𝑎) = 𝑃(𝐶𝐺𝑃𝐴 ≥ 9 ∣ 𝑁𝑜) × 𝑃(𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 = 𝑌𝑒𝑠 ∣ 𝑁𝑜) × 𝑃(𝑃𝑟𝑎𝑐𝑡𝑖𝑐𝑎𝑙
= 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 ∣ 𝑁𝑜) × 𝑃(𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑖𝑜𝑛 = 𝐺𝑜𝑜𝑑 ∣ 𝑁𝑜) × 𝑃(𝑁𝑜)
Substitute Values
Feature Value
CGPA ≥9 1/3
Interactiveness = Yes 1/3

Prof. Mamtha N B Dept. of CSE(AIML)

Machine learning BCS602

Practical = Average 2/3

Communication = Good 1/3
Prior P(No) 3/10
Calculation
= 1/3 × 1/3 × 2/3 × 1/3 × 3/10
Step-by-step multiply
1×1×2×1×3
=
3 × 3 × 3 × 3 × 10
6
=
810
≈ 0.0074
Step 4: Compare Results
Class Probability
Yes 0.0175
No 0.0074
Final Decision (MAP Rule) : Choose maximum probability
Since:
0.0175 > 0.0074
Final Answer
Job Offer = YES

8.3.2 Brute Force Bayes Algorithm

Brute Force Bayes algorithm is a method that
calculates probability for all hypotheses and selects the best one.
8.3.3 Bayes Optimal Classifier
Bayes Optimal classifier is a method that
uses all hypotheses together to make the final decision.
Difference
• MAP → picks one best hypothesis
• Bayes Optimal → uses all hypotheses
Formula:
Class = arg⁡ max⁡ ∑ 𝑃( 𝐶𝑖 ∣ ℎ) 𝑃(ℎ ∣ 𝑇)
𝐶𝑖
ℎ∈𝐻

We have 4 hypotheses:
Posterior Probability Table
Hypothesis P(h | T) P(COVID Positive | h) P(COVID Negative | h)
h1 0.3 0 1

Prof. Mamtha N B Dept. of CSE(AIML)

Machine learning BCS602

h2 0.1 1 0
h3 0.2 1 0
h4 0.1 1 0
Step 1: MAP Method
Choose hypothesis with maximum P(h|T)
Hypothesis Value
h1 0.3 (MAX)
So:
MAP → choose h1
From table:
• h1 predicts → COVID Negative
MAP Result
COVID Negative
Step 2: Bayes Optimal Classifier
Formula:
∑𝑃(𝐶𝑖 ∣ ℎ) × 𝑃(ℎ ∣ 𝑇)
For COVID Negative
= 0.3 × 1 = 0.3
(only h1 gives negative)
For COVID Positive
= (0.1 × 1) + (0.2 × 1) + (0.1 × 1)
= 0.1 + 0.2 + 0.1 = 0.4
Step 3: Compare
Class Value
Negative 0.3
Positive 0.4

Final Answer
COVID Positive (because 0.4 > 0.3)

4.3.4 Gibbs Algorithm

The main drawback of Bayes optimal classifier is that it computes the posterior probability for
all hypotheses in the hypothesis space and then combines the predictions to classify a new
instance.
Gibbs algorithm is a sampling technique which randomly selects a hypothesis from the
hypothesis space according to the posterior probability distribution and classifies a new
instance. It is found that the prediction error occurs twice with the Gibbs algorithm when
compared to Bayes Optimal classifier.

Prof. Mamtha N B Dept. of CSE(AIML)