Machine Learning

Dr. Irfan Yousuf

Department of Computer Science (New Campus)
UET, Lahore
(Week 3; February 23 - 27, 2026)
Outline
• Bayes Decision Theory
Naïve Bayes
Bayes Decision Theory
Bayes Decision Theory: Example
• Suppose we want to classify among the object-1 and object-2
with the given set of features X = [x1, x2, …, xn].

• Classification problem of Apple and Peach by color.

Bayes Decision Theory: Example
• Suppose we want to classify among the object-1 and object-2
with the given set of features X = [x1, x2, …, xn].

• Classification problem of Apple and Peach by color.

• Let's define a (probabilistic) variable w that describes the state of

nature.
• w = w1 (for object-1, Apple)
• w = w2 (for object-2, Peach)
Bayes Decision Theory: Example
• Based on Priors only
• Generally, we assume that there is some prior value P(w1)
that the next object is object-1 and P(w2) that the next object
is object-2. If we have no other object as in this problem,
then the sum of their prior is 1 i.e., the priors are exhaustive.

• The a priori or prior probability reflects our knowledge of

how likely we expect a certain state of nature before we can
observe the said state of nature.
Bayes Decision Theory: Example
• Based on Priors only

• A decision rule prescribes what action to take based on

observed input.
• What is a reasonable Decision Rule if
• the only available information is the prior, and
• the cost of any incorrect classification is equal?
• Decide w1 if P(w1) > P(w2); otherwise decide w2.
• What can we say about this decision rule?
Bayes Decision Theory: Example
• Based on Priors only

• If given; P(Apple) > P(Peach)

• If you do not have a chance to see the fruit,
• every time decide/predict as apple!
Bayes Decision Theory: Example
• If you are able to observe the color of the fruit,
• Question : P(apple|color)=? , P(peach|color)= ?
• Intuitively, choose the class with higher conditional probability.
• How to find these probabilities?
• Try using Bayes Rule :
• P(apple|color) = p(color|apple)*P(apple) / p(color)
• P(peach|color) = p(color|peach)*P(peach) / p(color)
Bayes Decision Theory: Example
• The class-conditional probability density function is the
probability density function for x, our feature, given that the
state of nature is w:
Bayes Decision Theory: Example
• The class-conditional probability density function is the
probability density function for x, our feature, given that the
state of nature is w:
Bayes Decision Rules: Analogy
Bayes Decision Rules: Analogy
Bayes Decision Rules: Analogy

An analogy is the morning commute, where you decide

whether to drive or bike based on the day's weather (a state of
nature) and the varying costs (traffic, cost of gas, effort) for
each option.

States of Nature (Uncertainties): These are the unknown

factors influencing your decision.
Example: The weather for the day—will it rain or be sunny?

Actions/Decisions: The choices you can make in response to

the uncertain states.
Example: Ride your bike, take the car, or use public transit.
Bayes Decision Rule: Analogy

Loss Function: The cost or loss associated with taking a

specific action when a particular state of nature occurs.

Example:
Rain: A car might be a low-loss option (quick commute),
while biking is a high-loss (wet and uncomfortable) one.
Sun: Biking might be the low-loss choice (pleasant, no gas
cost), while taking the car could be higher loss (traffic, fuel
cost).
Bayes Decision Rule: Analogy

Prior Probabilities: Your initial beliefs about how likely each

state of nature is, based on past experience or other
knowledge.

Example: Your prior belief that it's more likely to be sunny

than rainy in the morning.
Bayes Decision Rule: Analogy

New Information/Features: New observations that help

update your beliefs about the states of nature.

Example: Looking out the window and seeing dark clouds

before leaving for work
Bayes Decision Rule: Analogy

Posterior Probabilities: How your beliefs about the states of

nature are updated after considering the new information,
using Bayes' Theorem.

Example: If you see dark clouds, your posterior probability of

rain increases, making the rainy-day scenario more likely than
before.
Bayes Decision Rule: Analogy

The Decision: You choose the action that minimizes your

expected loss.

Example: If your posterior probability of rain is very high

after seeing the dark clouds, the expected loss of biking
becomes too high, leading you to choose the car.
Bayes Decision Rules and Loss Function
• What does it cost if you make the wrong decision?
• Suppose you decide y = 1, but the state is really y = −1,
Bayes Decision Rules and Loss Function
Confusion Matrix
Bayes Decision Rules and Loss Function
Bayes Decision Rules and Loss Function
Confusion Matrix
TPR denotes the sensitivity or true positive rate.
TNR denotes the specificity or true negative rate.
PPV (Positive Predicted Values) and NPV (Negative Predicted
Values) denote the positive and negative predictive values,
respectively.
Sensitivity
Sensitivity
• The term sensitivity is a statistical index of diagnostic
accuracy.
• It is also called the true positive rate, the recall, or probability
of detection.
• It has been defined as the ability of a test to identify correctly
all those who have the disease, which is “true-positive”.
• A 90 percent sensitivity means that 90 percent of the diseased
people screened by the test will give a “true-positive” result
and the remaining 10 percent a “false-negative” result.
• Thus, a highly sensitive test rarely overlooks an actual
positive.
Specificity
Specificity
• It is defined as the ability of a test to identify correctly those
who do not have the disease, that is, “true-negatives”.
• It is also called as the true negative rate.
• A 90 percent specificity means that 90 percent of the non-
diseased persons will give a “true-negative” result, 10 percent
of non-diseased people screened by the test will be wrongly
classified as “diseased” when they are not.
• Thus, a highly specific test rarely registers a positive
classification for anything that is not the target of testing.
Sensitivity vs. Specificity
• In medical tests, sensitivity is the extent to which actual
positives are not overlooked (so false negatives are few), and
specificity is the extent to which actual negatives are
classified as such (so false positives are few).
• Although a screening test ideally is both highly sensitive and
highly specific, we need to strike a balance between these
characteristics, because most tests cannot do both.
• If we want to increase sensitivity and to include all true
positives, we are obliged to increase the number of false
positives, which means decreasing specificity.
• Reducing the strictness of the criteria for a positive test can
increase sensitivity, but by doing this the test’s specificity is
reduced.
Sensitivity vs. Specificity
• A COVID test checks whether a person has the virus or not.
• There are only two real situations:
• The person HAS COVID
• The person DOES NOT have COVID
• The test can either:
• Say POSITIVE
• Say NEGATIVE
Sensitivity vs. Specificity
• Sensitivity means:
• Out of all the people who actually have COVID,
• how many did the test correctly identify as positive?
• Sensitivity tells us how good the test is at detecting sick
people.

• If sensitivity is high: Very few sick people are missed.

• If sensitivity is low: Some sick people may get a negative
result and spread the virus
Sensitivity vs. Specificity
• Suppose 100 people actually have COVID.
• The test correctly identifies 90 of them as positive.
• It misses 10 people (they test negative but are actually sick).
• Sensitivity = 90 / 100 = 90%
• This means the test detects 90% of infected people.
Sensitivity vs. Specificity
• Specificity means:
• Out of all the people who do NOT have COVID,
• how many did the test correctly identify as negative?
• Specificity tells us how good the test is at identifying healthy
people.

• If specificity is high: Very few healthy people are wrongly

told they are positive.
• If specificity is low: Many healthy people may panic
unnecessarily.
Sensitivity vs. Specificity
• Suppose 200 people do NOT have COVID.
• The test correctly says 180 are negative.
• But 20 people are wrongly told they are positive.
• Specificity = 180 / 200 = 90%
• This means the test correctly clears 90% of healthy people.
Sensitivity vs. Specificity
• Sensitivity
= How good the test is at finding sick people
= "Don’t miss real cases“

• Specificity
= How good the test is at avoiding false alarms
= "Don’t scare healthy people“

• High Sensi vity → Prevents spread (fewer missed cases)

High Specificity → Prevents panic (fewer false posi ves)
Sensitivity vs. Specificity
• Sensitivity of a test determines its ability to rule-out disease,
whereas the specificity of a test determines its ability to rule-
in disease?????
Sensitivity vs. Specificity
• Sensitivity is the probability of a positive test within a group
of patients who have the disease.
• Specificity is the probability of a negative test within a group
of patients who don't have the disease.

• Thus, sensitivity & specificity predicts whether the test will

be positive, given that the patient either has or doesn't have
the disease.
Bayes Decision Rules and Loss Function
Moving the Threshold: Right

Increases Specificity
Moving the Threshold: Left

Increases Sensitivity
Loglikelihood Ratio
• Sensitivity/specificity are more test-centered, whereas
likelihood ratios are more patient-centered.
Loglikelihood Ratio
Sensitivity vs. Specificity
ROC Curve (Receiver Operating Characteristics)
• A ROC curve is a graphical representation showing how the
sensitivity and specificity of a test vary in relation to one
another. To construct a ROC curve, samples known to be
positive or negative are measured using the test.

• The TPR (sensitivity) is plotted against the FPR (1 -

specificity) for given cut-off values to give a plot similar to
the one below. Ideally a point around the shoulder of the
curve is picked which both limits false positives whilst
maximizing true positives.
ROC Curve
ROC Curve
• A test that gave a ROC curve such as the yellow line would
be no better than random guessing, pale blue is good, but a
test represented by the dark blue line would be excellent.

• It would make cutoff determination relatively simple and

yield a high true positive rate at very low false positives rate
– sensitive and specific.
Multiclass Classification
Multiclass Classification
Bayes Decision Theory: Summary
• Bayes Decision Rule:
• If P(w1|x) > P(w2|x) we would decide that the object belongs
to class w1, otherwise class w2.

• Probability of Error
Bayes Decision Theory: Summary
• The evidence is unimportant as far as the decision is
concerned.
• It is just a scale factor that states how frequently we will
measure the feature with value x; it assures P(w1|x)+ P(w2|x)
= 1.
Bayes Decision Theory: Summary
Bayes Decision Theory: Summary
• Decision making relies on both the priors and the likelihoods
and Bayes Decision Rule combines them to achieve the
minimum probability of error.
Summary
• Bayes Decision Theory