What
does independence look
like?
Independence
S
Independence Definition 1:
P (AB) = P (A)P (B)
AB |AB| |A| |B|
= ⇥
A |S| |S| |S|
Independence Definition 2:
P (A|B) = P (A)
B
|AB| |A|
=
|B| |S|
Independence?
S
Independence Definition 1:
P (AB) = P (A)P (B)
0
|AB| |A| |B|
= ⇥
A |S| |S| |S|
B
Independence?
S
Independence Definition 2:
?
P (A|B) = P (A)
A |AB| ? |A|
=
|B| |S|
1 2
AB 6=
B 2 16
When we introduced conditions
Identities of probability remained
the same
But sometimes independence /
dependence relationships change
Current goals:
1) Recognize conditional
independence / dependence.
2) Get Intuition
Future goal:
Use conditional independence /
dependence in Machine Learning
Friday Night Fever
• Population of 10,000 people.
§ Of those, 300 have Malaria (event M) and 200 have
Bacterial Infection (event B). 6 people have both.
§ Have Fever if and only if you have Malaria or Bacteria.
§ Are M and B independent?
• Solution:
§ P(M) = 300 / 10,000 = 0.03
§ P(B) = 200 / 10,000 = 0.02
§ P(MB) = 6 / 10,000 = 0.0006
§ P(M)P(B) = 0.0006
§ P(M)P(B) = P(MB)
§ Independent
Friday Night Fever
• Population of 10,000 people.
§ Of those, 300 have Malaria (event M) and 200 have
Bacterial Infection (event B). 6 people have both.
§ Have Fever if and only if you have Malaria or Bacteria.
§ Are M and B independent given F?
• Solution:
§ Total people with Fever = 200+300 – 6 = 494
§ P(M|F) = 300 / 494 = 0.61
§ P(B|F) = 200 / 494 = 0.40
§ P(MB|F) = 6 / 494 = 0.012
§ P(M|F)P(B|F) = 0.224
§ P(M|F)P(B|F) ≠ P(MB|F)
§ Conditionally dependent
Conditional Dependence
10000 people
=
Conditional Dependence
P(M) = 0.03
300 have malaria
Conditional Dependence
P(B) = 0.02
200 have bacteria
Conditional Dependence
P(BM) = 0.006
6 have both
Conditional Dependence
If we condition
on B, the same
ratio of people
have malaria
P(M|B) = 6/200 = 0.03
P(M) = 300/10000 = 0.03
P(M) = P(M|B)
That’s the math
definition of
independence
Conditional Dependence
P(BM) = 0.006
6 have both
Conditional Dependence
If we condition
on M, the same
ratio of people
have bacteria
P(B|M) = 6/300 = 0.02
P(B) = 200/10000 = 0.02
P(B|M) = P(B)
Conditional Dependence
Malaria (M) Bacteria (B)
P(F|M) = 1 P(F|B) = 1
Fever (F)
*This is a “causal” diagram. It helps explain why things are independent
Conditional Dependence
Conditioned on Fever
If we condition on F,
we are left with only
the people who have
malaria and bacteria
Conditioned on Fever
P(B|F) = 200/494 = 0.40
P(M|F) = 300/494 = 0.61
Conditioned on Fever
Conditioned on Fever
P(B|F) = 200/494 = 0.40 Conditioned on Fever + Malaria
P(M|F) = 300/494 = 0.61
Conditioned on Fever
Test shows
Malaria
P(B|MF) = 6/300 = 0.02
P(B|F) ≠ P(B|MF)
That’s the math definition
of conditional dependence
Conditioned on Fever
P(B|F) = 200/494 = 0.40 Conditioned on Fever + Malaria
P(M|F) = 300/494 = 0.61
Conditioned on Fever
Test shows
Malaria
P(B|MF) = 6/300 = 0.02
P(B|F) != P(B|MF)
That’s the math definition
Test shows of conditional dependence
Conditioned on Bacteria
Fever + Bacteria
If we condition on F, the
events bacteria and malaria
become dependent
P(M|BF) = 6/200 = 0.03
P(M|F) ≠ P(M|BF)
Conditional Dependence
Malaria (M) Bacteria (B)
P(F|M) = 1 P(F|B) = 1
Fever (F)
*This is a “causal” diagram. It helps explain why things are independent
Parents With a Common Child
Say two independent parents have a common child:
When conditioned on the child they are no longer independent
Conditional Independence
House Voting
435 House Members
Are M and T independent?
• P(M) = 57/435 = 0.13
57 vote for gay • P(T) = 49/435 = 0.11
marriage (M) • P(M)P(T) = 0.014
• P(MT) = 45/435 = 0.10
• P(MT) ≠ P(M)P(T)
49 vote for • M and T are dependent
graduated tax (T)
House Voting
435 House Members
57 vote for gay
marriage (M)
57 House
Democrats
(D) 49 vote for
graduated tax (T)
Conditioned on Democrat
57 House Democrats (D)
Are M and T independent
given D?
53 vote for • P(M|D) = 53/57 = 0.93
gay • P(T|D) = 42/57 = 0.74
marriage • P(M|D)P(T|D) = 0.69
(M|D) • P(MT|D) = 39/57 = 0.68
• P(MT|D) ≅ P(M|D)P(T|D)
• M and T are conditionally
independent
42 vote for
graduated tax (T|D)
Conditional Dependence
Democrat (D)
Tax (T) Marriage (M)
*This is a “causal” diagram. It helps explain why things are independent
Netflix and Learn
K1
Like foreign emotional comedies
K )
E 4|
1
P (
) =
E 3, K 1
, E 2 ,
| E 1
P (E 4
E1 E2 E3 E4
Assume E1, E2, E3 and E4 are conditionally independent given K1
Children with a Common Parent
Siblings are dependent on one another. But in the presence of a common parent,
Become independent
End Review
Remember Learning to Code?
name
int a = 5;
double b = 4.2;
bit c = 1;
choice d = medium;
z 2 {high, medium, low}
Random Variable
• A Random Variable is a real-valued function
defined on a sample space
• Example:
§ 3 fair coins are flipped.
§ Y = number of “heads” on 3 coins
§ Y is a random variable
§ P(Y = 0) = 1/8 (T, T, T)
§ P(Y = 1) = 3/8 (H, T, T), (T, H, T), (T, T, H)
§ P(Y = 2) = 3/8 (H, H, T), (H, T, H), (T, H, H)
§ P(Y = 3) = 1/8 (H, H, H)
§ P(Y ≥ 4) = 0
Binary Random Variable
• A binary random variable is a random variable
with 2 possible outcomes (e.g., coin flip)
§ Now consider n coin flips, each which independently
come up heads with probability p
§ Y = number of “heads” on n flips
⎛ n ⎞ k
§ ⎜⎜ ⎟⎟ p (1 − p) n −k
P(Y = k) = , where k = 0, 1, 2, ..., n
⎝ k ⎠
n
⎛ n ⎞ k
§ So, ∑ ⎜⎜ ⎟⎟ p (1 − p) n −k = 1
k =0 ⎝ k ⎠
⎛ n ⎞ k
n
n−k n n
§ Proof: ∑ ⎜⎜ ⎟⎟ p (1 − p) = ( p + (1 − p)) = 1 = 1
k = 0 ⎝ k ⎠
Simple Game
• Urn has 11 balls (3 blue, 3 red, 5 black)
§ 3 balls drawn. +$1 for blue, -$1 for red, $0 for black
§ Y = total winnings
⎡⎛ 5 ⎞ ⎛ 3 ⎞⎛ 3 ⎞⎛ 5 ⎞⎤ ⎛11⎞ 55
§ P(Y = 0) = ⎢⎜⎜ ⎟⎟ + ⎜⎜ ⎟⎟⎜⎜ ⎟⎟⎜⎜ ⎟⎟⎥ ⎜⎜ ⎟⎟ =
⎣⎝ 3 ⎠ ⎝ 1 ⎠⎝ 1 ⎠⎝ 1 ⎠⎦ ⎝ 3 ⎠ 165
⎡⎛ 3 ⎞⎛ 5 ⎞ ⎛ 3 ⎞⎛ 3 ⎞⎤ ⎛11⎞ 39
§ P(Y = 1) = ⎢⎜⎜ ⎟⎟⎜⎜ ⎟⎟ + ⎜⎜ ⎟⎟⎜⎜ ⎟⎟⎥ ⎜⎜ ⎟⎟ = = P(Y = -1)
⎣⎝ 1 ⎠⎝ 2 ⎠ ⎝ 2 ⎠⎝ 1 ⎠⎦ ⎝ 3 ⎠ 165
⎛ 3 ⎞⎛ 5 ⎞ ⎛11⎞ 15
§ P(Y = 2) = ⎜⎜ ⎟⎟⎜⎜ ⎟⎟ ⎜⎜ ⎟⎟ = = P(Y = -2)
⎝ 2 ⎠⎝ 1 ⎠ ⎝ 3 ⎠ 165
⎛ 3 ⎞ ⎛11⎞ 1
§ P(Y = 3) = ⎜⎜ ⎟⎟ ⎜⎜ ⎟⎟ = = P(Y = -3)
⎝ 3 ⎠ ⎝ 3 ⎠ 165
Probability Mass Function
• A random variable X is discrete if it has countably
many values (e.g., x1, x2, x3, ...)
• Probability Mass Function (PMF) of a discrete
random variable is:
p(a) = P( X = a)
∞
• Since ∑ p ( xi ) = 1, it follows that:
i =1
⎧ p( xi ) ≥ 0 for i = 1, 2, ...
P( X = a) = ⎨
⎩ p( x) = 0 otherwise
where X can assume values x1, x2, x3, ...
PMF For a Single 6 Sided Dice
1/6
p(x)
X = outcome of roll
PMF for the sum of two dice
6/36
5/36
4/36
p(x) 3/36
2/36
1/36
X = total rolled
Cumulative Distribution Function
• For a random variable X, the Cumulative
Distribution Function (CDF) is defined as:
F (a) = P( X ≤ a) where − ∞ < a < ∞
• The CDF of a discrete random variable is:
F (a) = P( X ≤ a) = ∑ p( x)
all x ≤ a
CDF for a 6 sided dice
5/6
4/6
3/6
F(x)
2/6
1/6
X = outcome of roll
Expected Value
• The Expected Values for a discrete random
variable X is defined as:
E[ X ] = ∑ x p( x)
x: p ( x ) >0
• Note: sum over all values of x that have p(x) > 0.
• Expected value also called: Mean, Expectation,
Weighted Average, Center of Mass, 1st Moment
Expected Value
• Roll a 6-Sided Die. X is outcome of roll
p(1) = p(2) = p(3) = p(4) = p(5) = p(6) = 1/6
§
⎛ 1 ⎞ ⎛ 1 ⎞ ⎛ 1 ⎞ ⎛ 1 ⎞ ⎛ 1 ⎞ ⎛ 1 ⎞ 7
• E[X] = 1⎜ ⎟ + 2⎜ ⎟ + 3⎜ ⎟ + 4⎜ ⎟ + 5⎜ ⎟ + 6⎜ ⎟ =
⎝ 6 ⎠ ⎝ 6 ⎠ ⎝ 6 ⎠ ⎝ 6 ⎠ ⎝ 6 ⎠ ⎝ 6 ⎠ 2
• Y is random variable
§ P(Y = 1) = 1/3, P(Y = 2) = 1/6, P(Y = 3) = 1/2
• E[Y] = 1 (1/3) + 2 (1/6) + 3 (1/2) = 13/6
Indicator Variable
• A variable I is called an indicator variable for
event A if
⎧1 if A occurs
I = ⎨ c
⎩0 if A occurs
• What is E[I ]?
§ p(I =1) = P(A), p(I = 0) = 1 – P(A)
§ E[I ] = 1 P(A) + 0 (1 – P(A)) = P(A)
We’ll use this property frequently!
Lying with Statistics
“There are three kinds of lies:
lies, damned lies, and statistics”
– Mark Twain
• School has 3 classes with 5, 10 and 150 students
• Randomly choose a class with equal probability
• X = size of chosen class
• What is E[X]?
§ E[X] = 5 (1/3) + 10 (1/3) + 150 (1/3)
= 165/3 = 55
Lying with Statistics
“There are three kinds of lies:
lies, damned lies, and statistics”
– Mark Twain
• School has 3 classes with 5, 10 and 150 students
• Randomly choose a student with equal probability
• Y = size of class that student is in
• What is E[Y]?
§ E[Y] = 5 (5/165) + 10 (10/165) + 150 (150/165)
= 22635/165 ≈ 137
• Note: E[Y] is students’ perception of class size
§ But E[X] is what is usually reported by schools!
Expectation of a Function
• Let Y = g(X), where g is real-valued function
E[ g ( X )] = E[Y ] = ∑ y j p( y j )
j
= ∑ yj ∑ p( x ) i
j i:g ( x ) = y
i j
=∑ ∑ y j p ( xi )
j i:g ( x ) = y
i j
=∑ ∑ g ( x ) p( x )
i i
j i:g ( x ) = y
i j
= ∑ g ( xi ) p( xi )
i
Properties of Expectation
• Linearity:
E[aX + b] = aE[ X ] + b
§ Consider X = 6-sided die roll, Y = 2X – 1.
§ E[X] = 3.5 E[Y] = 6
• N-th Moment of X:
E[ X n ] = n
∑ p ( x)
x
x : p ( x ) >0
§ We’ll see the 2nd moment soon...