Bayesian Decision Theory
Introduction to Bayesian Decision Theory
– Statistical approach for classification under
uncertainty.
– Quantifies tradeoffs between decisions using
probabilities and costs.
• Assumptions:
– Decision problem is probabilistic.
– All relevant probabilities are known (for now).
• Applications:
– Medical diagnosis, spam filtering, object
recognition.
• Scenario:
– Classify fish as Sea Bass (ω₁) or Salmon (ω₂).
– Observing fish on a conveyor belt (random
sequence).
• States of Nature (ω):
ω = ω₁ (Sea Bass) or ω = ω₂ (Salmon).
• Challenge: Predict the next fish type with
minimal error.
Prior Probabilities
•Definition:
• P(ω₁) = Probability the next fish is Sea Bass.
• P(ω₂) = Probability the next fish is Salmon.
•Properties:
• P(ω₁) + P(ω₂) = 1 (only two classes).
• Priors must exhibit exclusivity
•Example:
• If P(ω₁) = 0.6, P(ω₂) = 0.4 → More likely Sea Bass.
Decision Rule with Only Priors
• Rule:
Decide ω1 if P(ω1) > P(ω2)
Otherwise decide ω2
• Limitations:
Always picks the same class (no data used).
• Error Probability:
P(error) = min[P(ω₁), P(ω₂)].
Incorporating Evidence (Feature Measurement)
• Feature (x):
e.g., lightness of fish (continuous variable).
• Class-Conditional Density:
– p(x|ω₁) = Probability density of lightness for Sea
Bass.
– p(x|ω₂) = Probability density of lightness for
Salmon.
• Visualization:
– Graph of p(x|ω₁) vs. p(x|ω₂) (like Fig. 2.1 in text).
Bayes' Formula
• Posterior Probability:
• Components:
– Likelihood (p(x|ω_j)): How likely the feature is for class
ω_j.
– Prior (P(ω_j)): Initial belief about class ω_j.
– Evidence (p(x)): Normalizing factor.
• Informal Version:
Posterior=Likelihood×Prior/Evidence
• Finding probability of hypothesis
P(A|B)-Probability of A on B
A-hypothesis, B-Evidence/data
Likelihood: Probability of evidence given that
hypothesis is true(hypothesis considered is true
based on evidence)
Prior: Probability of hypothesis before considering
evidence
Decision Rule with Posteriors
• Optimal Rule:
– Decide ω₁ if P(ω₁|x) > P(ω₂|x); otherwise ω₂.
• Equivalent Form (Eliminating p(x)):
– Decide ω₁ if p(x|ω₁)P(ω₁) > p(x|ω₂)P(ω₂).
• Special Cases:
– If p(x|ω₁) = p(x|ω₂), decision depends only on priors.
– If P(ω₁) = P(ω₂), decision depends only on likelihoods.
• In general, both influence classification.
• Bayes rule ensures minimum probability of error.
Probability of Error
• Conditional Error:
P(error∣x)=min[P(ω1∣x),P(ω2∣x)]
• Total Error:
P(error)=∫ P(error∣x)p(x) dx
• Bayes Rule Minimizes P(error):
– No other rule can do better under these
assumptions.
Special Cases and Insights
• - If p(x | ω1) = p(x | ω2): Decision based on
priors
• - If P(ω1) = P(ω2): Decision based on
likelihoods
• In general, both influence classification.
• Bayes rule ensures minimum probability of
error.
• You’ve planned an elegant outdoor wedding. Normally,
the chance of strong winds on any given day is 10%.
• You wake up to a very cloudy sky, and it makes you
nervous. According to historical weather patterns:
• 70% of days that have strong winds also start out very
cloudy.
• However, only 20% of all days are very cloudy in
general.
• ❓Based on this cloudy morning, what is the updated
probability that strong winds will occur today?
• Should you start considering backup plans for your
wedding?