MACHINE LEARNING (CS
403/603)
Basics of
Probability Dr.
Puneet Gupta
Random
• A r.v. 𝑋 denotes possible outcomes of an event
Variable
• Can be discrete (i.e., finite many possible outcomes, like, coin-toss or
dice roll) or continuous (like, height)
X-axis denotes 𝑋
𝑝(𝑋)
Y-axis denotes
Random For a continuous r.v. 𝑋, a probability
𝑝(𝑋
For a discrete r.v. 𝑋 = 𝑥)Here,
or 𝑝(𝑥)
Variable
interval 𝑋 ∈(𝑥,𝑥+𝛿𝑥)
• we define of prob. within an
is meaningless
• 𝑝(𝑋) is called the
• 𝑝(𝑥)𝛿𝑥 is the prob. that 𝑋
(PMF) of r.v. 𝑋 ∈(𝑥,𝑥+𝛿𝑥) as 𝛿𝑥→0
probability mass function
• 𝑝(𝑥) or 𝑝(𝑋 = 𝑥) is • 𝑝(𝑥) is the probability density at
𝑋= 𝑥
PMF at 𝑥
the value of the
𝑝𝑥 ≥
𝑝𝑥 ≥ 0
0 𝑝𝑥 ≤
𝑝𝑥 𝑝 𝑥≤ = න 𝑝1 𝑥 𝑑𝑥 =
11 1
Joint Probability Distribution
Joint prob. dist. 𝑝(𝑋, 𝑌) models probability of co-occurrence of two
𝑋, 𝑌
r.v.
• For discrete r.v., the joint PMF 𝑝(𝑋, 𝑌 ) is like a table (that sums to
1)
• For two continuous r.v.’s 𝑋 and 𝑌, we have joint PDF
𝑝(𝑋, 𝑌)
Marginal Probability Distribution,
a.k.a, Sum Rule
Consider two r.v.’s X and Y (discrete or continuous), marginal probability
For discrete r.v.’s, 𝑝(𝑋)= ∑𝑦 (𝑝(𝑋,𝑌=𝑦) and 𝑝(𝑌)=∑𝑥 (𝑝(𝑋=𝑥,𝑌)
is PMF/PDF of one r.v. by fixing another r.v.
For discrete r.v., it is the sum
of the PMF table along the
rows/columns
For continuous r.v.’s,
Conditional Probability Distribution
• For two r.v.’s 𝑋 and 𝑌 (discrete or continuous), the conditional
PMF/PDF 𝑝(𝑋|𝑌) is the prob. dist. of one r.v. 𝑋, fixing other
r.v. 𝑌
• 𝑝(𝑋|𝑌 = 𝑦) or 𝑝(𝑌 |𝑋 = 𝑥) like taking a slice of the joint dist.
𝑝(𝑋, 𝑌 ), with slight modification.
Basic rules of probability will follow.
What is sum over all possible values?
How to extract the slice?
Important results in Probability
• Sum Rule: Gives the marginal probability distribution from joint probability distribution
• Product Rule: 𝑝(𝑋, 𝑌) = 𝑝(𝑌 |𝑋)𝑝(𝑋) = 𝑝(𝑋|𝑌)𝑝(𝑌)
• Bayes’ rule: Gives conditional probability distribution (can derive it from product rule)
• Chain Rule: 𝑝(𝑋_1, 𝑋_2, . . . , 𝑋_𝑁) = 𝑝(𝑋_1)𝑝(𝑋_2 |𝑋_1). . . 𝑝(𝑋_𝑁 |
𝑋_1, . . . , 𝑋_(𝑁−1))
Independence
𝑋 and 𝑌 are marginal independent, (𝑋 𝖬 𝑌), when one does not tell
anything about another.
Two r.v.’s 𝑋 and 𝑌 can be conditional independent when they may not
value of another r.v. 𝑍 is known.
be marginally independent on its own but becomes independent when the