Machine
Learning
BOE073
UNIT-3
Aashna Kushwaha
[Link]
Evaluating Hypothesis
• A hypothesis is an assumption about some event. A hypothesis is a
statement that can be tested to determine whether it is true or false.
• In machine learning, a hypothesis is an idea or explanation about
something that can be tested by an experiment.
• The evaluation of machine learning models and algorithms is called
hypothesis testing, and the evaluation of the accuracy of a hypothesis is
called evaluating a hypothesis.
[Link]
Estimating Hypothesis Accuracy
TRUE ERROR (ERRORD(H)) SAMPLE ERROR (ERRORS(H))
[Link]
TRUE ERROR
“The true error of a hypothesis(h) is the probability that (h) will misclassify an
instance drawn at random”. It is given as :
Errortrue(h)=Px~D[h(x)≠f(x)]
where,
Errortrue(h) = Error in hypothesis (h) w.r.t data (D)
P = Probability
f(x) = Target Function (real output)
h(x) = Hypothesis Function (model)
[Link]
SAMPLE ERROR
“The sample error is the fraction of examples in a sample dataset that are
misclassified by the hypothesis”.
𝟏 𝑵
Errorsample(h)= σ𝒊=𝟏 𝜹 [ h(xi),f(xi)]
𝑵
where, where:
N: number of examples in the sample
𝜹: del operator represents a “change”
1, if ℎ 𝑥𝑖 ≠ 𝑓 𝑥𝑖
𝐿 ℎ 𝑥𝑖 𝑓 𝑥𝑖 =ቊ
0, otherwise
[Link]
BASICS OF SAMPLE THEORY
“Sampling is the process of getting information from a large dataset”.
Characteristics of a sample
• A good sample should give a correct representation of the entire population.
• A sample must be obtained by a probability process.
• Sample size should be small
Types of Sampling
• Simple Random Sampling
• Stratified Random Sampling
• Multi-Stage Sampling
• Systematic Sampling Method
• Cluster Sampling
• Non- probability Sampling
[Link]
BASICS OF SAMPLE THEORY
Random Variable
“A random variable is a variable whose possible values are determined by the outcomes
of a random experiment”.
There are two main types:
Discrete Random Variable → takes specific values (e.g., number of heads in coin tosses)
Continuous Random Variable → takes any real value within an interval (e.g., height,
weight)
Example:
Let 𝑋= number of heads when flipping 2 coins.
Possible values: 𝑋 = 0 1 2
[Link]
BASICS OF SAMPLE THEORY
Probability Distribution
A probability distribution describes how probabilities are assigned to possible values of a
random variable.
∑P(X=xi)=1(discrete)
∞
න 𝑓 𝑥 𝑑𝑥 = 1 (continuous)
−∞
Example:
For a fair die:
P(X=x) = 1/6, x=1,2,3,4,5,6
[Link]
BASICS OF SAMPLE THEORY
Sample Mean
The sample mean is the average value of the observations in a sample.
Formula:
𝑛
ˉ 1
𝑋 = 𝑋𝑖
𝑛
𝑖=1
Example:
If the sample heights (in cm) are 160, 165, 170,
then:
ˉ 160 + 165 + 170
𝑋= = 165
3
[Link]
BASICS OF SAMPLE THEORY
Sample Variance
The sample variance measures the spread or dispersion of the data points around the
mean.
Formula:
(use 𝑛 − 1for an unbiased estimate of population variance)
Example:
If 𝑋 = 246 ,
ቆ2 − 4)2 + ൬4 − 4)2 + ቀ6 − 4)2
ˉ 8
𝑋 = 4, 𝑠 2 = = =4
2 2
[Link]
BASICS OF SAMPLE THEORY
Standard Deviation
The standard deviation is the square root of variance — it represents how much data
values deviate (on average) from the mean.
Formula:
Example:
If 𝑠 2 = 4 ,then 𝑠 = 2.
[Link]
BASICS OF SAMPLE THEORY
Estimator
An estimator is a statistic used to estimate a population parameter.
Good estimators:
Unbiased
Efficient
Consistent
Example:
Setting a y target (age.) = 50
[Link]
BASICS OF SAMPLE THEORY
Confidence Level and Confidence Interval
A confidence interval (CI) gives a range where we expect the true population parameter
to lie with a certain confidence level (e.g., 95%).
Formula:
where 𝑧𝛼/2 =critical z-value (e.g., 1.96 for 95%)
[Link]
BASICS OF SAMPLE THEORY
Central Limit Theorem
In probability and statistics, the central limit theorem states that “If we consider a larger
random sample from a population, then its probability will always be normally distributed”.
Formula:
𝝁 = 𝝁𝒙
[Link]
Comparing learning algorithm
When comparing algorithms, we want to evaluate how well they learn patterns from
training data and generalize to unseen data.
5. Example
Suppose we want to predict if a student passes or fails based on study hours.
We train:
•Logistic Regression
•Decision Tree
•Random Forest
.
Model Accuracy Precision Recall
Logistic Regression 0.84 0.81 0.79
Decision Tree 0.88 0.85 0.83
Random Forest 0.91 0.89 0.87
Random Forest performs best — it’s chosen as the final model
[Link]
Bayes Theorem
Bayes’ Theorem is a fundamental concept in probability theory and machine learning, used to
update the probability of a hypothesis as new evidence is observed.
Bayes' Theorem is a mathematical formula used to determine the conditional probability of an
event based on prior knowledge and new evidence.
Where, A = Hypothesis
B = Given data
P(A/B) = Posterior Probability
P(B/A) = Likehood Probability
P(A) = Prior probability
P(B) = Marginal Probability
[Link]
Bayes Theorem Example
Example: To calculate the probability of ‘Fire’ when “Smoke” is given with data as:
Given, P(Fire) = Prior Probability = 0.3
P(Smoke/Fire) = 0.5
P(Smoke) = 0.7
Solution:
To calculate ,
P(Fire|Smoke) = P(Smoke|Fire) x P(Fire)
P(Smoke)
Put numericals values
P(Fire|Smoke) = (0.5) x (0.3)
(0.7)
= 0.15/0.7 = 0.214
P(Fire|Smoke) = 0.214 ans
[Link]
Maximum a Posterior(MAP) Hypothesis
and the Maximum Likelihood (ML) Hypothesis
The maximum probable hypothesis is called the Maximum A Posterior (MAP) hypothesis. It is
denoted by (hmap )
Choose the hypothesis that is most probable after seeing the data, taking into account both:
• how well it fits the data (likelihood)
• how likely it was before seeing the data (prior)
hMAP = Arg max P(D|h) . P(h)
P(D)
Choose the hypothesis that makes the observed data most probable, ignoring any prior beliefs.
• You only care about how well the hypothesis fits the data.
• You don’t use any prior information about which hypotheses were more likely before seeing the data.
hML = Arg max P(D|h)
[Link]
Bayes Optimal Classifier
The Bayes Optimal Classifier (BOC) is the
best possible classifier you can have — it
gives the lowest possible error rate (called
the Bayes error).
In simple terms:
The Bayes Optimal Classifier predicts the
class that has the highest posterior
probability given the data.
[Link]
Naïve Bayes Classifier
The Naïve Bayes algorithm is a supervised learning algorithm. It is based on bayes
theorem. It is used for solving classification problems in machine learning.
The Naïve Bayes algorithm consists of two words “Naïve” and “Bayes”. It can be described
as:
• Naïve: Naïve means “untrained” or “without experience” . This algorithm called naïve
bayes because it assumes that the occurrence of a certain feature is independent of
other features.
• Bayes: It defined on Bayes theorem. It is given as:
[Link]
Weather Condition Dataset
0 Outlook Play
0 RAINY YES
1 SUNNY YES
2 OVERCAST YES
3 OVERCAST YES
4 SUNNY NO
5 RAINY YES
6 SUNNY YES
7 OVERCAST YES
8 RAINY NO
9 SUNNY NO
10 SUNNY YES
11 RAINY NO
12 OVERCAST YES
13 OVERCAST YES
[Link]
Weather Condition Dataset
Step1: Make a frequency table from the given data set are:
Weather Yes No
OVERCAST 5 0
RAINY 2 2
SUNNY 3 2
TOTAL 10 4
Step2: Make Likelihood Table are:
WEATHER NO YES LIKELIHOOD
OVERCAST 0 5 5/14 = 0.35
RAINY 2 2 4/14 = 0.29
SUNNY 2 3 5/14 = 0.35
ALL 4/14 = 0.29 10/14 = 0.71
[Link]
Weather Condition Dataset
Step 3: Apply Bayes’ Theorem as:
Put values , P(YES/SUNNY) = P(SUNNY/YES) X P(YES)
P(SUNNY)
From likelihood tables , we get,
P(Sunny/Yes) = 3/10 = 0.3
P(Sunny) = 0.35
P(yes) = 0.71
P(Yes/Sunny) = 0.3 x 0.71 = 0.60
0.35
P(Yes/Sunny) = 0.60
[Link]
Weather Condition Dataset
Similarly,
P(No/Sunny) = P (Sunny/No) x P(No)
P(Sunny)
P(Sunny/No) = 2/4 = 0.5
P(No) = 0.29
P(Sunny) = 0.35
P(No/Sunny) = 0.5 x 0.29
0.35
P(No/Sunny) = 0.41
As the P(yes/Sunny) > P(No/Sunny) i.e 0.60 > 0.41
Therfore, we can say that on sunny day , the player will go for play.
[Link]
Bayesian Belief Networks
A Bayesian belief network is a probabilistic graphical model. It represents a
set of variables and their conditional dependencies using a directed acyclic
graph (DAG).
It combines:
Graph theory (to show dependencies visually)
Bayes’ theorem (to calculate probabilities)
[Link]
EM Algorithm
The expectation–maximization (EM) algorithm is used to find the unknown (unseen)
variables from observed variables of the sample space.
• Latent variables: The unseen variables are called latent variables. The EM algorithm
is used to find the latent variables of a dataset.
• Maximum Likelihood Estimation: In other words, we can also say that the EM
algorithm is used to find or estimate the maximum likelihood of latent variables.
Steps:
Expectation step : By using the observed available data of
the data set, we estimate or find the values of the missing
data. After this we get complete data with no missing
values.
Maximization step: Now, we use complete data to upload
the parameters. Repeat step 2 and 1 until we converge the
solution.
[Link]
Application of EM Algorithm
• It is used to find the missing data (latent data) of a data set.
• It is used for parameter estimation of the Hidden Markov Model.
• It is used to calculate the Gaussian density of a function.
• It is used in Natural Language Processing, computer vision etc.
[Link]