ARTIFICIAL

INTELLIGENCE AND
MACHINE LEARNING
MODULE 4 – BAYESIAN LEARNING
BAYESIAN LEARNING - TOPICS TO
BE COVERED
• Introduction
• Baye’s theorem
• Baye’s theorem and Concept Learning
• Maximum Likelihood and Least-Squared Error Hypotheses
• Maximum Likelihood for Predicting
• Minimum Description Length Principle
• Bayes Optimum Classifier
• Gibbs Algorithm
• Naïve Baye’s Classifier
• Bayesian Belief Networks
• EM (Estimating Means) Algorithm
INTRODUCTION
• Bayesian Learning is a fundamental concept in
Machine Learning that uses probability to make
decisions under uncertainty.
• It is based on updating beliefs as new data
becomes available.
Core Idea
• Instead of learning a single “best model,”
Bayesian learning
 Maintains a probability distribution over
models (hypotheses)
 Updates this distribution using data
How Bayesian Learning Works
How Bayesian Learning Works
• Start with a prior belief about the model
• Observe data
• Compute likelihood
• Update belief using Bayes’ theorem → get
posterior
• Repeat as more data arrives
Learning = continuous updating of beliefs
Posterior = Prior × Likelihood
Bayes’ Theorem (Foundation)
𝑃 𝐷∣𝐻 𝑃 𝐻
•𝑃 𝐻∣𝐷 =
𝑃 𝐷
• Where:
• P(H) → Prior (initial belief about hypothesis)
• P(D | H) → Likelihood (how well hypothesis
explains data)
• P(H | D) → Posterior (updated belief after
seeing data)
• P(D) → Evidence (normalizing constant)
 Example (Intuition)

• Step-by-Step Learning Process

• Step 1: Start with Prior
• You assume something about the model before
seeing data.
• Example:
“A student usually scores around 70%”
• Step 2: Observe Data
• You collect new evidence.
• Example:
Student scores: 85%, 90%, 88%
Example (Intuition)

• Step 3: Update Belief

• Use Bayes’ theorem to update your belief.
• Now you believe:
“Student is likely a high performer”
• Step 4: Repeat
• As more data comes, belief keeps improving.
SOLVED EXAMPLES IN BAYES
LEARNING
• Let’s work on a simple NLP problem with Bayes
Theorem. By using NLP, I can detect spam e-
mails in my inbox.
• Assume that the word ‘offer’ occurs in 80% of
the spam messages in my account.
• Also, let’s assume ‘offer’ occurs in 10% of my
desired e-mails.
• If 30% of the received e-mails are considered
as a scam, and I will receive a new message
which contains ‘offer’, what is the probability
that it is spam?
• Now, I assume that I received 100 e-mails.
• The percentage of spam in the whole e-mail is
30%. So, I have 30 spam e-mails and 70 desired
e-mails in 100 e-mails.
• The percentage of the word ‘offer’ that occurs
in spam e-mails is 80%. It means 80% of 30 e-
mail and it makes 24.
• Now, I know that 30 e-mails of 100 are spam and
24 of them contain ‘offer’ where 6 of them not
contains ‘offer’.
• The percentage of the word ‘offer’ that occurs
in the desired e-mails is 10%. It means 7 of
them (10% of 70 desired e-mails) contain the
word ‘offer’ and 63 of them not.
• The question was what is the probability of
spam where the mail contains the word ‘offer’:
• We need to find the total number of mails which
contains ‘offer’ ;
• 24 +7 = 31 mail contain the word ‘offer’
• Find the probability of spam if the mail
contains ‘offer’ ;
• In 31 mails 24 contains ‘offer’ means 77.4% =
0.774 (probability)
• NOTE: In this example, I choose the percentages
which give integers after calculation. As a
general approach, you can think that we have
100 units at the beginning so if the results
are not an integer, it will not create a
problem. Such that, we cannot say 15.3 e-mails
• Solution with Bayes’ Equation:
• A = Spam
• B = Contains the word ‘offer’
• P( contains offer|spam) = 0.8 (given in the
question)
• P(spam) = 0.3 (given in the question)
• Now we will find the probability of e-mail with
the word ‘offer’. We can compute that by adding
‘offer’ in spam and desired e-mails. Such that;
• P(contains offer) = 0.30.8 + 0.70.1 = 0.31

• As it is seen in both ways the results are the

same. In the first part, solved the same
question with a simple chart and for the second
part, solved the same question with Bayes’
theorem.
• Problem 2:
• Solve one more example from a popular topic as
Covid-19.
• As you know, Covid-19 tests are common
nowadays, but some results of tests are not
true.
• Let’s assume; a diagnostic test has 99%
accuracy and 60% of all people have Covid-19.
• If a patient tests positive, what is the
probability that they actually have the
disease?
• The total units which have positive results=
59.4 + 0.4 = 59.8
• 59.4 units (true positive) is 59.8 units means
99.3% = 0.993 probability
• With Bayes’;

• P(positive|covid19) = 0.99
• P(covid19) = 0.6
• P(positive) = 0.60.99+0.40.01=0.598
• In machine learning, a Version Space is the
subset of all possible hypotheses (the
hypothesis space) that are strictly consistent
with every training example observed so far.
1. Key Components & Representation
• Because listing every consistent hypothesis is
often impractical, version spaces are typically
represented by two boundaries:
• Specific Boundary: The set of the most specific
hypotheses consistent with the data. These
hypotheses only cover the positive examples
seen and no more.
• General Boundary: The set of the most general
hypotheses consistent with the data. These
cover all positive examples and as much of the
remaining space as possible without including
any negative examples.
 Brute Force
• The term "Brute Force" is used as a metaphor in
computer science to describe an approach that
relies on sheer computational power rather than
cleverness or mathematical insight.
• In the context of the List-Then-Eliminate
algorithm, it is called brute force because it
takes the most literal, unoptimized path to
finding the version space.
Problems solving: Bayesian
Learning
• To illustrate Bayes rule, consider a medical diagnosis
problem in which there are two alternative hypotheses:
(1) that the patient has a particular form of cancer,
and (2) that the patient does not.
The available data is from a particular laboratory
Test with two possible outcomes:
⨁(positive) and ⊖ (negative)
• We have prior knowledge that over the entire population of
people only .008 have this disease.
• Further more, the lab test is only an imperfect indicator of
the disease.
• The test returns a correct positive result in only 98% of
the cases in which the disease is actually present
• And a correct negative result in only 97% of the cases in
which the disease is not present.
• In other cases, the test returns the opposite result.
Problems solving : Bayesian
Learning
• The above situation can be summarized by the
following probabilities:

• Suppose we now observe a new patient for whom

the lab test returns a positive result. Should
we diagnose the patient as having cancer or
not?
 𝑃 𝐷∣𝐻 𝑃 𝐻
W.K.T. 𝑃 𝐻∣𝐷 =
𝑃 𝐷
MINIMUM DESCRIPTION LENGTH (MDL)
MINIMUM DESCRIPTION LENGTH
Minimum Description Length (MDL)
Principle
• MDL gives a way to choose between competing
models.
• The best model is the one that describes the
data in the simplest, most efficient way.
• Occam's razor, a popular Inductive bias:"choose
the shortest explanation for the observed
data.“
• We consider a Bayesian perspective on this
issue and a closely related principle called
the Minimum Description Length (MDL) principle.
Minimum Description Length (MDL)
Principle
• The Minimum Description Length principle is
motivated by interpreting the definition of
hmap in the light of basic concepts from
information theory.

• which can be equivalently expressed in terms of

maximizing the log2

• or alternatively minimizing the negative of

this quantity
Minimum Description Length (MDL)
Principle
• To explain this, let us introduce a basic result from information
theory:

• Consider the problem of designing a code to transmit messages

drawn at random, where the probability of encountering message i
is Pi.

• We are interested here in the most compact code, that is, we are
interested in the code that minimizes the expected number of bits
we must transmit in order to encode a message drawn at random.

• To minimize the expected code length = assign shorter codes =

messages that are more probable.

• Shannon and Weaver (1949) showed that the optimal code (i.e. the
code that minimizes the expected message length)assigns—log2 pi
bits*to encode message i.

• We will refer to number of bits required to encode message i using

code C as the description length of message i with respect to C,
which we denote by Lc(i).
Minimum Description Length (MDL)
Principle
• The Minimum Description Length (MDL) principle
recommends choosing the hypothesis that
minimizes the sum of these two description
lengths.
• Of course to apply this principle in practice
we must choose specific encodings or
representations appropriate for the given
learning task.
• Assuming we use the codes C1 and C2 to
represent the hypothesis and the data given the
hypothesis, we can state the MDL principle as
Minimum Description Length principle: Choose
hMDL where
Why MDL uses log base 2?
In MDL, we measure code length in bits,
and:
•1 bit = log₂
•So we must use log base 2
BAYESIAN BELIEF NETWORK
• Bayesian Belief Network (BBN) = Bayesian Network =
Probabilistic Graphical Model

• What it is:
• A BBN is a directed acyclic graph (DAG) where:
• Nodes = random variables (e.g., Rain, Traffic, Accident)
• Edges (arrows) = probabilistic dependencies between variables
• Each node has a conditional probability table (CPT) that
quantifies how it depends on its parent nodes.

• Key idea
• It combines:
• Graph theory (structure of dependencies)
• Probability theory (uncertainty and inference)
• So instead of listing a huge joint probability table, a
BBN factorizes it into smaller, manageable pieces.
Expectation-Maximization – EM
Algorithm
• The EM algorithm has been used to train
Bayesian belief networks as well as radial
basis function networks.
• The EM algorithm is also the basis for many
unsupervised clustering algorithms.
• It is the basis for the widely used Baum-Welch
forward-backward algorithm for learning
Partially Observable Markov Models.
Expectation-Maximization – EM
Algorithm
• The Expectation-Maximization (EM) algorithm is
an iterative optimization technique used to
estimate unknown parameters in probabilistic
models, particularly when the data is incomplete,
noisy or contains hidden (latent) variables.
• It works in two steps:
• E-step (Expectation Step):
• Using the current parameter estimates, the algorithm
calculates the expected values of the missing or hidden
variables.
• Essentially, it assigns probabilities or
"responsibilities" to different hidden outcomes given the
observed data.
• M-step (Maximization Step):
• With these updated expectations from the E-step, the
algorithm then re-estimates the model parameters by
maximizing the expected log-likelihood.
• This improves how well the model explains the observed
Expectation-Maximization – EM
Algorithm
Expectation-Maximization – EM
Algorithm
These two steps are repeated until convergence,
which typically means that:
The parameter values stop changing
significantly or
The log-likelihood improves only by a
negligible amount.
By iteratively repeating these steps the EM
algorithm seeks to maximize the likelihood of
the observed data.
Expectation-Maximization – EM
Algorithm
• Key Concepts – Most commonly used key terms
• Latent Variables: Variables that are not
directly observed but are inferred from the
data. They represent hidden structure (e.g.,
cluster assignments in Gaussian Mixture
Models).
• Likelihood: The probability of the observed
data given a set of model parameters. EM aims
to find parameter values that maximize this
likelihood.
• Log-Likelihood: The natural logarithm of the
likelihood function. It simplifies calculations
(turning products into sums) and is numerically
more stable when dealing with very small
Expectation-Maximization – EM
Algorithm
• Maximum Likelihood Estimation (MLE): A
statistical approach to estimating parameters
by choosing the values that maximize the
likelihood of observing the given data. EM
extends MLE to cases with hidden or missing
variables.
• Posterior Probability: In Bayesian inference,
this represents the probability of parameters
(or latent variables) given the observed data
and prior knowledge. In EM, posterior
probabilities are used in the E-step to
estimate the "responsibility" of each hidden
variable.
• Convergence: The stopping criterion for the
iterative process. EM is said to converge when
updates to parameters or improvements in log-
• Need for EM:
• GMMs are used for clustering data that cannot
be described by a single Gaussian, such as
multi-modal data.
• Because calculating the MLE for GMMs involves
log-sum terms, a closed-form solution is not
available, requiring an iterative method like
EM.
• Primary Applications:
• The Gaussian Mixture Model (GMM) and the EM
algorithm are widely used for tasks involving
unsupervised learning, probabilistic
clustering, and density estimation.
• Image Segmentation, Speech and Speaker
Recognition, Anomaly Detection, Bioinformatics,
Missing Data Imputation, Customer Segmentation.
• Instances generated by a mixture of two Normal
Distributions with identical variance σ.
• The instances are shown by points along the x
axis.
• If the means of the Normal distributions are
unknown, the EM algorithm can be used to search
for their maximum likelihood estimates.
• An isocontour is a curve representing a
constant scalar value across a 2D surface or
data set, where every point along the line
shares the same value.
• Used to represent boundaries, these lines
visualize density estimates, terrain elevation,
temperature, or pressure.
• Common method for extracting lines from data
grids.
• Definition: Lines of equal value, often used in
scientific visualization to connect points of
equal magnitude.
• Applications: Common in topological maps
(elevation), weather reports
(pressure/temperature), and medical image