ITCS 6156/8156 Spring 2024
Machine Learning
Generative Models for
Classification
Instructor: Hongfei Xue
Email: [Link]@[Link]
Class Meeting: Mon & Wed, 4:00 PM – 5:15 PM, Denny 109
Some content in the slides is based on Dr. Raquel Urtasun’s lecture
Classification
elephant
bear
2
Discriminative Classifiers
elephant
bear
3
Discriminative Classifiers
elephant
bear
4
Generative Classifiers
elephant
bear
5
Generative vs Discriminative
6
Generative vs Discriminative
• Discriminative Classifiers:
• Often have better accuracy (Probability estimates for posteriors P (Y | X)
often more accurate. Generative models make often unrealistic
assumptions on the P(x|y))
• Can handle feature preprocessing (E.g. via nonlinear feature maps 𝜙 :
𝑅 ! → 𝑅 " with D >> d (explicitly) or kernel trick (implicitly). Modelling
distributions of preprocessed features too hard in generative models.)
• Generative Classifiers:
• Easy (not expensive) to fit. (”Training” often amounts to averaging /
counting things. No optimization algorithms necessary.)
• Can handle missing feature data more easily.
• Builds model for different classes separately. (Good if new classes are
added to the problem.)
7
• Handle unlabeled training data better.
Probability Formulas
8
Image from: [Link]
Bayes Classifier
9
A Discrete Example
• For a genetic defect test we have the following information:
• 1% of people have a certain genetic defect;
• For people who have genetic defects, 90% of tests show
positive results;
• For people who have no genetic defects, 90% tests show
negative results.
• If a person gets a positive test result, what is the probability
he/she actually have the genetic defect?
• Solution:
𝑃 𝑇=1𝐺=1 𝑃 𝐺=1
𝑃 𝐺=1𝑇=1 =
𝑃(𝑇 = 1)
𝑃 𝑇=1𝐺=1 𝑃 𝐺=1
=
𝑃 𝑇 =1 𝐺 =1 𝑃 𝐺 =1 +𝑃 𝑇 =1 𝐺 =0 𝑃 𝐺 =0
0.9 ∗ 0.01
= = 8.33%
0.9 ∗ 0.01 + 0.1 ∗ 0.99
10
Classification: Diabetes Example
Green: No Diabetes
Red: Diabetes
11
Gaussian Bayes Classifier
12
MLE for Gaussians
13
MLE for Gaussians
14
Computing the Mean
15
Computing the Variance
16
MLE of a Gaussian
17
Posterior Probability
18
Diabetes Example
19
Diabetes Example
20
Bayes Classifier
21
Multiple-dimensional Inputs
22
Gaussian Bayes Classifier
23
Multivariate Data
24
Multivariate Parameters
25
Multivariate Gaussian Distribution
26
Simplifying the Model
27
Naive Bayes
28
Naive Bayes Classifier
29
A Discrete Example
x(1): Humidity (level 1, 2, 3);
x(2): Wind intensity (Low, Mediate, Strong);
y: Rain (1) or not (-1).
• Will it rain when humidity is at level 2 and wind intensity is strong?
𝑃 𝑦 = 1|𝐻 = 2, 𝑊 = 𝑆 𝑃 𝑦 = 0|𝐻 = 2, 𝑊 = 𝑆
𝑃 𝑦 = 1 𝑃 𝐻 = 2, 𝑊 = 𝑆|𝑦 = 1 𝑃 𝑦 = 0 𝑃 𝐻 = 2, 𝑊 = 𝑆|𝑦 = 0
= =
𝑃(𝐻 = 2, 𝑊 = 𝑠) 𝑃(𝐻 = 2, 𝑊 = 𝑠)
𝑃 𝑦 = 1 𝑃 𝐻 = 2|𝑦 = 1 𝑃 𝑊 = 𝑆|𝑦 = 1 𝑃 𝑦 = 0 𝑃 𝐻 = 2|𝑦 = 0 𝑃 𝑊 = 𝑆|𝑦 = 0
= =
𝑃(𝐻 = 2, 𝑊 = 𝑠) 𝑃(𝐻 = 2, 𝑊 = 𝑠)
10 4 1 2 5 1 3 3
∗ ∗ ∗ ∗
= 15 10 10 = 75 = 15 5 5 = 75
𝑃(𝐻 = 2, 𝑊 = 𝑠) 𝑃(𝐻 = 2, 𝑊 = 𝑠) 𝑃(𝐻 = 2, 𝑊 = 𝑠) 𝑃(𝐻 = 2, 𝑊 = 𝑠)
• It will probably not rain. 30
Questions?