Machine Learning

Bayesian Learning

Lecturer: Duc Dung Nguyen, PhD.

Contact: nddung@[Link]

Faculty of Computer Science and Engineering

Hochiminh city University of Technology
 Contents

1. Linear Prediction

2. Bayesian Learning

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 1 / 30

Linear Prediction
 Linear Prediction

Linear supervised learning

• Many real processes can be approximated with linear models

• Linear regression often appears as a module of larger systems
• Linear problems can be solved analytically
• Linear prediction provides an introduction to many of the core concepts in machine
learning.

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 2 / 30

 Linear Prediction

Energy demand prediction

Wind speed People inside building Energy requirement

100 2 5
50 42 25
45 31 22
60 35 18

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 3 / 30

 Linear Prediction

Teen Birth Rate and Poverty Level Data

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 4 / 30

 Linear Prediction

Lung Function in 6 to 10 Year Old Children

mean

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 5 / 30

 Linear Prediction

Lung Function in 6 to 10 Year Old Children

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 6 / 30

 Linear Prediction

• In general the linear model is expressed as follows

d
X
ŷi = xij θj
j=1

• In matrix form
ŷ = Xθ

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 7 / 30

 Linear Prediction

• We can use optimization approach

J(θ) = (y − ŷ)> (y − ŷ)

• Least squares estimates

• Probabilistic approach

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 8 / 30

Bayesian Learning
 Bayesian Learning

• It involves direct manipulation of probabilities in order to find correct hypotheses.

• The quantities of interest are governed by probability distributions.
• Optimal decisions can be made by reasoning about those probabilities.

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 9 / 30

 Bayesian Learning

• Bayesian learning algorithms are among the most practical approaches to certain type of
learning problems
• Provide a useful perspective for understanding many learning algorithms that do not
explicitly manipulate probabilities.

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 10 / 30

 Features of Bayesian Learning

• Each training example can incrementally decrease or increase the estimated probability
that a hypothesis is correct.
• Prior knowledge can be combined with observed data to determine the final probability
of a hypothesis
• Hypotheses with probabilities can be accommodated
• New instances can be classified by combining multiple hypotheses weighted by the
probabilities.

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 11 / 30

 Bayes Theorem

P (D|h)P (h)
P (h|D) = (1)
P (D)

• P (h): prior probability of hypothesis h

• P (D): prior probability of training data D
• P (h|D): probability that h holds given D
• P (D|h): probability that D is observed given h

P(A and B)= P(A|B)P(B) = P(B|A)P(A)

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 12 / 30

 Bayes Theorem
P(D|B)
P(D|A)

find a hypothesis h that can

• Maximum A-posteriori hypothesis (MAP): make P(h|D) max
hM AP = arg max P (h|D) = arg max P (D|h)P (h) (2)
h∈H h∈H

P (h) is not a uniform distribution over H. need P(D)(likelihood), but we

dont know that (of the dataset)
P (D|h)P (h)
P (h|D) = (3)
P (D)

likelihood: distribution of data over some class of data

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 13 / 30

 Bayes Theorem

dont know distribution of likelihood

• Maximum Likelihood hypothesis (ML):

hM L = arg max P (h|D) = arg max P (D|h) (4)

h∈H h∈H

If P (h) is a uniform distribution over H.

P (D|h)P (h)
P (h|D) = (5)
P (D)

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 14 / 30

 Bayes Theorem

• 0.008 of the population have cancer

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 15 / 30

 Bayes Theorem

• 0.008 of the population have cancer

• Only 98% patients are correctly classified as positive

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 15 / 30

 Bayes Theorem

• 0.008 of the population have cancer

• Only 98% patients are correctly classified as positive
• Only 97% non-patiants are correctly classified as negative

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 15 / 30

 Bayes Theorem

p(cancer)=0.008
• 0.008 of the population have cancer
• Only 98% patients are correctly classified as positive p(+|c)=0.98
• Only 97% non-patiants are correctly classified as negative p(-|-c)
• Would a person with a positive result have cancer or not?

P (cancer|⊕) >< P (¬cancer|⊕)

c = argmax p(c|+) = argmax p(+|c)p(c)/p(+)

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 15 / 30

 Bayes Theorem

• Maximum A-posteriori hypothesis (MAP):

hM AP = arg max P (h|⊕)

h∈(cancer,¬cancer)
(6)
= arg max P (⊕|h)P (h)
h∈(cancer,¬cancer)

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 16 / 30

 Bayes Theorem

• P (cancer) = .008 → P (¬cancer) = .992

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 17 / 30

 Bayes Theorem

• P (cancer) = .008 → P (¬cancer) = .992

• P (⊕|cancer) = .98

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 17 / 30

 Bayes Theorem

• P (cancer) = .008 → P (¬cancer) = .992

• P (⊕|cancer) = .98
• P ( |¬cancer) = .97 → P (⊕|¬cancer) = .03

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 17 / 30

 Bayes Theorem

• P (cancer) = .008 → P (¬cancer) = .992

• P (⊕|cancer) = .98
• P ( |¬cancer) = .97 → P (⊕|¬cancer) = .03
• P (cancer|⊕) ≈ P (⊕|cancer)p(cancer) = .0078

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 17 / 30

 Bayes Theorem

• P (cancer) = .008 → P (¬cancer) = .992

• P (⊕|cancer) = .98
• P ( |¬cancer) = .97 → P (⊕|¬cancer) = .03
• P (cancer|⊕) ≈ P (⊕|cancer)p(cancer) = .0078
• P (¬cancer|⊕) ≈ P (⊕|¬cancer)P (¬cancer) = .0298

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 17 / 30

 Bayes Theorem

• Maximum A-posteriori hypothesis (MAP):

hM AP = arg max P (h|⊕)

h∈(cancer,¬cancer)

= arg max P (⊕|h)P (h) (7)

h∈(cancer,¬cancer)

= ¬cancer

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 18 / 30

 Bayes Optimal Classifier

• What is the most probable hypothesis given the training data?

• What is the most probable classification of a new instance given the training data?

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 19 / 30

 Bayes Optimal Classifier

• Hypothesis space = {h1 , h2 , h3 }

• Posterior probabilities = {.4, .3, .3} (h1 is hM AP )
• New instance x is classified positive by h1 and negative by h2 and h3

What is the most probable classification of x?

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 20 / 30

 Bayes Optimal Classifier

• The most probable classification of a new instance is obtained by combining the

predictions of all hypotheses weighted by their posterior probabilities:
X
arg max P (c|D) = arg max P (c|h).P (h|D) (8)
c∈C c∈C
h∈H

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 21 / 30

 Naive Bayes Classifier

good for large data size with no correlation between attribute

Example Sky AirTemp Humidity Wind Water Forecast EnjoySport

1 Sunny Warm Normal Strong Warm Same Yes
2 Sunny Warm High Strong Warm Same Yes
3 Rainy Cold High Strong Warm Change No
4 Sunny Warm High Strong Cool Change Yes
5 Cloudy Warm High Weak Cool Same Yes
6 Sunny Cold High Weak Cool Same No
7 Sunny Warm Normal Strong Warm Same ?
8 Sunny Warm Low Strong Cool Same cannot
? predict

likelihood=0
Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 22 / 30
 Naive Bayes Classifier

• Each instance x is described by a conjunction of attribute values < a1 , a2 , ..., an >

• The target function f (x) can take on any value from a finite set C
• It is to assign the most probable target value to a new instance

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 23 / 30

 Naive Bayes Classifier

CM AP = arg max P (c|a1 , a2 , ..., an )

c∈C
(9)
= arg max P (a1 , a2 , ..., an |c)P (c)
c∈C

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 24 / 30

 Naive Bayes Classifier

CM AP = arg max P (c|a1 , a2 , ..., an )

c∈C

= arg max P (a1 , a2 , ..., an |c)P (c)

c∈C
(10)
Y
CN B = arg max P (ai |c)P (c)
c∈C i=1,n

assuming that a1 , a2 , ..., an are independent given c

all attribute have no correlation

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 25 / 30

 Naive Bayes Classifier

Example Sky AirTemp Humidity Wind Water Forecast EnjoySport

1 Sunny Warm Normal Strong Warm Same Yes
2 Sunny Warm High Strong Warm Same Yes
3 Rainy Cold High Strong Warm Change No
4 Sunny Warm High Strong Cool Change Yes
5 Cloudy Warm High Weak Cool Same Yes
6 Sunny Cold High Weak Cool Same No
7 Sunny Warm Normal Strong Warm Same ?
8 Sunny Warm Low Strong Cool Same ?

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 26 / 30

 Naive Bayes Classifier

Estimating probabilities:

• Probability: the fraction of times the event is observed to occur over the total number of
opportunities nc /n
• What if the fraction is too small, or even zero?

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 27 / 30

 Naive Bayes Classifier

smoothing technique
Estimating probabilities:
nc + mp
(11)
n+m

• n: total number of training examples of a particular class.

• nc : number of training examples having a particular attribute value in that class.
• m: equivalent sample size
• p: prior estimate of the probability (equals 1/k where k is the number of possible values of
the attribute)

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 28 / 30

 Naive Bayes Classifier

Learning to classify text:

Y
CN B = arg max P (ai = wk |c).P (c)
c∈C i=1,n

find hypothesis c to maximize the Prob

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 29 / 30

 Naive Bayes Classifier

Learning to classify text:

Y
CN B = arg max P (ai = wk |c).P (c)
c∈C i=1,n
Y (12)
= arg max P (wk |c).P (c)
c∈C i=1,n

assuming that all words have equal chance occurring in every position

Lecturer: Duc Dung Nguyen, PhD. Contact: nddung@[Link] Machine Learning 30 / 30