The Bayes Theorem

Examples
Agha Ali Raza

CS535/EE514 – Machine Learning

Some Warmup: Monty Hall Problem
(Khan Academy: [Link]

• Three closed doors!

• Behind one is a great prize, while behind others are

goats!
Some Warmup: Monty Hall Problem
(Khan Academy: [Link]

• You choose a door!

• The host does not open your door

but one of the other doors.

• He does not open the door with a car behind it!

• Now you are offered a chance to swap your choice.
• Would you swap? And would it make any difference?
Some Warmup: Monty Hall Problem
(Khan Academy: [Link]

Initially,
– P(Goat) = 2/3 = 66%
– P(Car) = 1/3 = 33%

Now, if you don’t Switch:

– P(win) = 33%
– P(Fail) = 66%

If you switch
– If you initially chose a goat (66% chance) and now switch, you
will surely win as there is a car behind the remaining door!
– If you initially chose a car (33% chance) and now switch, you
will surely loose as there is a goat behind the remaining door!
– You should always swap!
 Result: If
Result: If
Behind door Behind door Behind door switching to
staying at
1 2 3 the door
door #1
offered
Goat Goat Car Wins goat Wins car
Goat Car Goat Wins goat Wins car
Car Goat Goat Wins car Wins goat

A player who stays with the

initial choice wins in only one
out of three of these equally
likely possibilities, while a player
who switches wins in two out of
three.
Monty Hall Problem: A variation
(Khan Academy: [Link]

What if, instead of switching, you randomly choose

again?
In the new universe you have two doors: Goat and
Car
– A new random choice can yield each with 50%
chance:
P(Goat) = 0.5
P(Car) = 0.5
To help convince yourselves, imagine these were not
doors but boxes. One has a diamond, while two
have coals.
– Initially you pick a box and put it in your pocket
– The host opens one other box
– Switch means you swap the box in your pocket
with the other closed box
– Re-choose means you put your box and the
closed one in a bag, shuffle and pick one out.
– The old universe had two possibilities, the new
one has two
Examples
In the upcoming examples we will use the following
forms of conditional probability and Bayes Theorem
to reach our conclusions:
|ℎ∩𝐷|
• 𝑃(ℎ|𝐷) = h D
|𝐷|
𝑃(ℎ∩𝐷)
• 𝑃(ℎ|𝐷) =
𝑃(𝐷)
𝑃(𝐷|ℎ) 𝑃(ℎ)
• 𝑃(ℎ|𝐷) =
𝑃(𝐷)
Bayes: Example 2
([Link]

1% of women at age forty who participate in routine screening have

breast cancer. 80% of women with breast cancer will get positive
mammograms. 9.6% of women without breast cancer will also get
positive mammograms.

A woman in this age group had a positive mammography in a routine

screening. What is the probability that she actually has breast cancer?
Bayes: Example 2
([Link]

The women with cancer, A:

The women with positive mammograms, B:

If we restrict our universe to B , only a small percentage actually have

cancer.
Bayes: Example 2 – Using counts
([Link]

Assume we have a population of 1,000 women

1% of them (10 women) have cancer. Of these, 80% (8
women) also test positive
990 women do not have cancer. Of these, 9.6% (95 women)
test positive
Of the 103 (8+95) women that test positive, only 8 have
cancer.
Therefore, probability of cancer given a positive result:
|𝑐𝑎𝑛𝑐𝑒𝑟 ∩ +| 8
𝑃 𝑐𝑎𝑛𝑐𝑒𝑟 + = = = 0.076%
|+| 103
Bayes: Example 2 – Using hMAP
([Link]

We can also do the same using MAP hypothesis:

𝑷 𝒄𝒂𝒏𝒄𝒆𝒓 = 𝟎. 𝟎𝟏 𝑷(¬𝒄𝒂𝒏𝒄𝒆𝒓) = 𝟎. 𝟗𝟗
𝑷 + 𝒄𝒂𝒏𝒄𝒆𝒓 = 𝟎. 𝟖 𝑷(−|𝒄𝒂𝒏𝒄𝒆𝒓) = 𝟎. 𝟐
𝑷 + ¬𝒄𝒂𝒏𝒄𝒆𝒓 = 𝟎. 𝟎𝟗𝟔 𝑷(−|¬𝒄𝒂𝒏𝒄𝒆𝒓) = 𝟎. 𝟗𝟎𝟒

𝑷(𝒄𝒂𝒏𝒄𝒆𝒓|+) = 𝑷(+|𝒄𝒂𝒏𝒄𝒆𝒓). 𝑷(𝒄𝒂𝒏𝒄𝒆𝒓) = (0.8) (0.01) = 0.008

𝑷(¬𝒄𝒂𝒏𝒄𝒆𝒓|+) = 𝑷(+|¬𝒄𝒂𝒏𝒄𝒆𝒓). 𝑷(¬𝒄𝒂𝒏𝒄𝒆𝒓) = (0.096) (0.99) = 0.095

Therefore, ℎ𝑀𝐴𝑃 = ¬𝑐𝑎𝑛𝑐𝑒𝑟.

We can also retrieve the actual 𝑷(𝒄𝒂𝒏𝒄𝒆𝒓|+) by normalizing using 𝑃(+).

𝑷 + 𝒄𝒂𝒏𝒄𝒆𝒓 . 𝑷 𝒄𝒂𝒏𝒄𝒆𝒓
𝑷 𝒄𝒂𝒏𝒄𝒆𝒓 + =
𝑷 +
𝑷 + 𝒄𝒂𝒏𝒄𝒆𝒓 . 𝑷 𝒄𝒂𝒏𝒄𝒆𝒓
=
𝑷 + 𝒄𝒂𝒏𝒄𝒆𝒓 . 𝑷 𝒄𝒂𝒏𝒄𝒆𝒓 + 𝑷 + ¬𝒄𝒂𝒏𝒄𝒆𝒓 . 𝑷 ¬𝒄𝒂𝒏𝒄𝒆𝒓
0.008
= = 𝟎. 𝟎𝟕𝟔
0.008+0.095

Although HMAP is ¬𝑐𝑎𝑛𝑐𝑒𝑟, Bayes theorem has updated our belief about P(cancer) from 0.01 to
0.076 using data, i.e. the positive text result.
Bayes: Example 3
([Link]

• Suppose you are a nurse screening a set of students for a sickness called Diseasitis.
You know, from past population studies, that around 20% of the students will have
Diseasitis at this time of year.

• You are testing for Diseasitis using a color-changing tongue depressor, which usually
turns black if the student has Diseasitis. Among patients with Diseasitis, 90% turn the
tongue depressor black. However, the tongue depressor is not perfect, and also turns
black 30% of the time for healthy students.

• One of your students comes into the office, takes the test, and turns the tongue
depressor black. What is the probability that they have Diseasitis?
Bayes: Example 3 – Using counts
([Link]

First, we imagine a population of 100 students, of

whom 20 have Diseasitis and 80 don't.
Bayes: Example 3 – Using counts
([Link]

• 90% of sick students turn their tongue depressor black, and 30% of
healthy students turn the tongue depressor black.
• We see black tongue depressors on 90% * 20 = 18 sick students,
and 30% * 80 = 24 healthy students.
Bayes: Example 3 – Using counts
([Link]

What's the probability that a student with a black tongue depressor has Diseasitis?
There are 18 sick students with black tongue depressors.
18 + 24 = 42 students in total turned their tongue depressors black.

|𝑑𝑖𝑠 ∩ +| 18
A patient with a black tongue depressor has 𝑃 𝑑𝑖𝑠 + = = = 𝟎. 𝟒𝟐𝟖
|+| 42
probability of being sick.
Bayes: Example 3 – Using hMAP
([Link]
We can also do the same using MAP hypothesis:
𝑷 𝒅𝒊𝒔 = 𝟎. 𝟐 𝑷(¬𝒅𝒊𝒔) = 𝟎. 𝟖
𝑷 + 𝒅𝒊𝒔 = 𝟎. 𝟗 𝑷(−|𝒅𝒊𝒔) = 𝟎. 𝟏
𝑷 + ¬𝒅𝒊𝒔 = 𝟎. 𝟑 𝑷(−|¬𝒅𝒊𝒔) = 𝟎. 𝟕

𝑷(𝒅𝒊𝒔|+) = 𝑷(+|𝒅𝒊𝒔). 𝑷(𝒅𝒊𝒔) = (0.9) (0.2) = 0.18

𝑷(¬𝒅𝒊𝒔|+) = 𝑷(+|¬𝒅𝒊𝒔). 𝑷(¬𝒅𝒊𝒔) = (0.3) (0.8) = 0.24

Therefore, ℎ𝑀𝐴𝑃 = ¬𝑑𝑖𝑠.

We can also retrieve the actual 𝑷(𝒅𝒊𝒔|+) by normalizing using 𝑃(+).

𝑷 + 𝒅𝒊𝒔 . 𝑷 𝒅𝒊𝒔
𝑷 𝒅𝒊𝒔 + =
𝑷 +
𝑷 + 𝒅𝒊𝒔 . 𝑷 𝒅𝒊𝒔
=
𝑷 + 𝒅𝒊𝒔 . 𝑷 𝒅𝒊𝒔 + 𝑷 + ¬𝒅𝒊𝒔 . 𝑷 ¬𝒅𝒊𝒔
0.18
= = 0.428
0.18+0.24

Although HMAP is ¬𝑑𝑖𝑠, Bayes theorem has updated our belief about P(dis) from 0.2 to 0.428 using
data, i.e. the positive test result.
Bayes: Example 3
([Link]

Many find this answer counter-intuitive at first:

– The test correctly detects Diseasitis 90% of the time!

– If the test comes back positive, why is it still less than 50% likely that
the patient has Diseasitis?

– Well, the test also incorrectly "detects" Diseasitis 30% of the time
in a healthy patient, and we start out with lots more healthy patients
than sick patients.

The test provides some evidence of the patient being sick. It goes from
20% before the test, to 43% after it.
We need to perform further tests, maybe more expensive ones.
Bayes - Intuition
([Link]

P is updated in the face of new evidence i.e. Probability mass P that is inconsistent
with evidence is eliminated
With a single, individual patient, before observing any evidence, there are four
possible worlds we could be in:

As the patient gets a positive result, the worlds with negative results get eliminated
from further consideration

So Bayes' rule requires us to update our beliefs in the face of new evidence: Throw
away the probability mass that is inconsistent with the evidence.
Bayes: Example 4
(Machine Learning, Tom Mitchell)
Bayes: Example 4 – Using hMAP
We can also do the same using MAP hypothesis:
𝑷 𝒄𝒂𝒏𝒄𝒆𝒓 = 𝟎. 𝟎𝟎𝟖 𝑷(¬𝒄𝒂𝒏𝒄𝒆𝒓) = 𝟎. 𝟗𝟗2
𝑷 + 𝒄𝒂𝒏𝒄𝒆𝒓 = 𝟎. 𝟗𝟖 𝑷(−|𝒄𝒂𝒏𝒄𝒆𝒓) = 𝟎. 𝟎𝟐
𝑷 + ¬𝒄𝒂𝒏𝒄𝒆𝒓 = 𝟎. 𝟎𝟑 𝑷(−|¬𝒄𝒂𝒏𝒄𝒆𝒓) = 𝟎. 𝟗𝟕

𝑷(𝒄𝒂𝒏𝒄𝒆𝒓|+) = 𝑷(+|𝒄𝒂𝒏𝒄𝒆𝒓). 𝑷(𝒄𝒂𝒏𝒄𝒆𝒓) = (0.98) (0.008) = 0.0078

𝑷(¬𝒄𝒂𝒏𝒄𝒆𝒓|+) = 𝑷(+|¬𝒄𝒂𝒏𝒄𝒆𝒓). 𝑷(¬𝒄𝒂𝒏𝒄𝒆𝒓) = (0.03) (0.992) = 0.0298

Therefore, ℎ𝑀𝐴𝑃 = ¬𝑐𝑎𝑛𝑐𝑒𝑟.

We can also retrieve the actual 𝑷(𝒄𝒂𝒏𝒄𝒆𝒓|+) by normalizing using 𝑃(+).

𝑷 + 𝒄𝒂𝒏𝒄𝒆𝒓 . 𝑷 𝒄𝒂𝒏𝒄𝒆𝒓
𝑷 𝒄𝒂𝒏𝒄𝒆𝒓 + =
𝑷 +
𝑷 + 𝒄𝒂𝒏𝒄𝒆𝒓 . 𝑷 𝒄𝒂𝒏𝒄𝒆𝒓
=
𝑷 + 𝒄𝒂𝒏𝒄𝒆𝒓 . 𝑷 𝒄𝒂𝒏𝒄𝒆𝒓 + 𝑷 + ¬𝒄𝒂𝒏𝒄𝒆𝒓 . 𝑷 ¬𝒄𝒂𝒏𝒄𝒆𝒓
0.0078
= = 𝟎. 𝟐𝟏
0.0078+0.0298

Although HMAP is ¬𝑐𝑎𝑛𝑐𝑒𝑟, Bayes theorem has updated our belief about P(cancer) from 0.008 to
0.208 using data, i.e. the positive test result.
Bayes: Example 4 – Using counts

Assuming a population of 10,000:

• 10,000 * 0.008 = 80 have cancer

• Of these, 80 * 0.98 = 78.4 test positive

• And, 10,000 * 0.992 = 9,920 do not have cancer

• Of these, 9,920 * 0.03 = 297.6 test positive

|𝐶𝑎𝑛𝑐𝑒𝑟 ∩ +| 78.4
• So, P(Cancer|+) = = = 0.208
|+| 78.4+297.6
 Now an example with words!
10% of all tokens in a corpus is the word “theater”. 80% of times it is
preceded by the word “movie”. 25% of other words are also preceded
by “movie”. We see the word “movie” is a sentence. What is the
probability that it will be followed by the word “theater”?

Movie _______ (theater?)

Assume we have 1,000 tokens in our corpus.

10% of them (100 tokens) are the word “theater”. Of these, 80% (80
“theater” tokens) are preceded by the word “movie”.
900 words in the corpus are words other then “theater”. Of these, 25%
(225 tokens) are preceded by the word “movie”.
Of the 305(80+225) words that are preceded by the word “movie”, 80
are “theater”
“𝒎𝒐𝒗𝒊𝒆 𝒕𝒉𝒆𝒂𝒕𝒆𝒓” 𝟖𝟎
𝑷(“𝒕𝒉𝒆𝒂𝒕𝒆𝒓”| “𝒎𝒐𝒗𝒊𝒆”) = = = 𝟎. 𝟐𝟔𝟐
𝒎𝒐𝒗𝒊𝒆 𝟑𝟎𝟓

But the story continues…

hMAP to the rescue!
Of all the words, that ever occurred after “movie”, here is a sorted break down:
- theater: 80
- tickets: 70
- night: 40
- food: 25
- plans: 10
- …
Therefore, of all the continuations of the word “movie”, the word “theater” is the most
probable one, hMAP = “theater”:
“𝒎𝒐𝒗𝒊𝒆 𝒕𝒉𝒆𝒂𝒕𝒆𝒓” 𝟖𝟎
𝑷(“𝒕𝒉𝒆𝒂𝒕𝒆𝒓”| “𝒎𝒐𝒗𝒊𝒆”) = = = 𝟎. 𝟐𝟔𝟐
𝒎𝒐𝒗𝒊𝒆 𝟑𝟎𝟓

“𝒎𝒐𝒗𝒊𝒆 𝒕𝒊𝒄𝒌𝒆𝒕𝒔” 𝟕𝟎
𝑷(“𝒕𝒊𝒄𝒌𝒆𝒕𝒔”| “𝒎𝒐𝒗𝒊𝒆”) = = = 𝟎. 𝟐𝟐𝟗
𝒎𝒐𝒗𝒊𝒆 𝟑𝟎𝟓

Models that assign probabilities to sequences of words are called Language Models

Language modeling can be viewed as classification: each word can be thought of as a

class, and so predicting the next word is classifying the context-so-far into a class
for each next word.
|𝒄𝒐𝒏𝒕𝒆𝒙𝒕 𝒘|
𝒘𝑴𝑨𝑷 = 𝒂𝒓𝒈𝒎𝒂𝒙𝒘 ∈ 𝑽 𝑷(𝒘|𝒄𝒐𝒏𝒕𝒆𝒙𝒕) = 𝒂𝒓𝒈𝒎𝒂𝒙𝒘 ∈ 𝑽
|𝒄𝒐𝒏𝒕𝒆𝒙𝒕|
Next: Types of Models
 For more details please visit

[Link]

Thank you!
41