Introduction to
Probability
Probability
• Everywhere
• Weather
• Airlines
• Stock Market
• Everyday Advertisements
Different Types of Probabilities
• Classical
• Empirical
• Subjective
Classical Probability
• Games – cards and dice
• Eg. 4 cards – 3 black and 1 red (ace of heart), what is the probability that when we
shuffles these cards and pull out one card which is the ace of heart?
• Assumes all events to be equally likely.
• Classical Probability = Total number of favorable outcomes divided by the total
number of outcomes
• Not much useful in daily decision making
Empirical Probability
• Based on relative frequency
• For prediction, we look at the past data
• Eg. Pharmaceutical companies
• Launch a new drug
• Help the patients with particular condition
• Side Effects
• Based on the collected data, we assess the probability of healing power and
side effects of that particular drug
• Stock Market
• No past data??
Subjective Probability
• Utterly new situation
• Apple iPad
• Expected probability based on an opinion and belief
Language
• Random Experiment
• A process leading to an uncertain outcome
• Basic Outcome
• What happens in a random experiment?
• Eg. Rolling a dice
• This is the outcome of a random experiment.
• Sample Space
• Collection of basic outcomes
• Event
• An event is a subset of basic outcomes
• Eg. We will get 2 on rolling a dice
Probability Definition
• Chance/likelihood that an uncertain event will occur.
• Number between 0 and 1
• Measure for how likely an outcome of a random experiment is.
• For an event A, this is usually denoted by P (A), 0 ≤ 𝑃(𝐴) ≤ 1
Definitions
• Classical Probability
!"."$ %&%'%()* +( ,"-. %/%()
• Prob. of an event =
0")1& (-'2%. "$ 21*+3 "-)3"'%*
• Empirical Probability
4"5 "$)%( 6+6 1( %/%() "33-. +( 1 *%.+%* "$ ).+1&*
• Prob. of an event =
0")1& (-'2%. "$ ).+1&*
• Subjective Probability
• We don’t have data
• We use something called gut feeling and the previous experience to predict the probability.
The Axioms
If E is an event and is a subset of S, the sample space, then the following are true:
1. 𝑃(𝑆) = 1 The probability of the sample space is 1.
2. 0 ≤ P(E) ≤ 1 The probability of an event E is between 0 and 1 inclusive.
3. For two disjoint events A and B (disjoint means that A⋂ B = ∅ ), P (A ⋃ B) = P(A)+ P(B).
Example
• Random Experiment – we rolled a dice
• Possible outcomes – 1,2,3,4,5,6
• Event A (even number appears on the dice) – 2,4,6
• Event B – 1,5
• P(A) and P(B)?
• P(A U B) ?
Additional Rules
4. Complement Rule
𝑃 𝐴̅ = 1 − 𝑃(𝐴)
5. The general addition rule
𝑃(𝐴 𝑈 𝐵) = 𝑃(𝐴) + 𝑃(𝐵) – 𝑃(𝐴 ∩ 𝐵)
6. If A and B are mutually exclusive then
P(A U B) = P(A) + P(B).
• Example:
• A = {2,4,6} and B = {1,2,3,4}
Standard Definitions
• Union
• The union of two events A and B is the set that contains all the events that are either in A or in
B or in both sets.
• Notation: A⋃ B.
• Intersection
• The intersection of two events A and B is the set that contains all the events that are both in A
and in B.
• Notation: A⋂ B.
• Complement
• The complement of an event A with respect to an event B
• Refers to all elements that are in B, but not in A. This is usually denoted by B ∩ 𝐴̅ or B|A.
Cont.
• Mutually Exclusive
• Two events A and B are said to be mutually exclusive if they have no basic outcomes in
common. This is usually denoted by B⋂ A = ∅ .
• Collectively Exhaustive
• Events A1, A2, ..., An are said to be collectively exhaustive events if A1 ⋃ A2 ...⋃ An = S, i.e.
the events in union completely cover the sample space S.
Agenda for Today
• Concept of Independence
• Subjective Probability
• Empirical Probability
• Conditional Probability
Independence
• Sum of two dice
• Random Experiment: Sum of 2 dice
• Possible outcomes: 2,3,4…,12
• Sample Space, S: S= {2,3,4…,12}
Independence Sum Counts Prob.
2 1 1/36
Sum 1 2 3 4 5 6 3 2 2/36
1 2 3 4 5 6 7 4 3 3/36
2 3 4 5 6 7 8 5 4
6 5
3 4 5 6 7 8 9
7 6 6/36
4 5 6 7 8 9 10 8 5
5 6 7 8 9 10 11 9 4
6 7 8 9 10 11 12 10 3
11 2
12 1 1/36
Example
• Let E be the event of tossing two dice such that the sum of the face is even. Let F
be the event that the sum of the faces is greater than 9.
• What is the probability of the sum of the faces being even and greater than 9.
• What we are looking for is P(E ∩ F). The only rolls that correspond to that event
are (6,4), (4,6), (5,5), (6,6).
• Thus P(E ∩ F)=4/36=1/9.
• What is the probability of the sum being even or greater than 9.
• Want to compute P(E U F).
• USE AN AXIOM P(E U F)=P(E)+P(F)-P(E ∩ F)=1/2+1/6-1/9=5/9
WHAT IS YOUR SUBJECTIVE PROBABILITY?
• Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student,
she was deeply concerned with issues of discrimination and social justice, and also participated in
anti-nuclear demonstrations.
• Which of the following eight options is more likely for Linda to be?
(1) Linda is a teacher in elementary school.
(2) Linda works in a bookstore and takes Yoga classes.
(3) Linda is active in the feminist movement.
(4) Linda is a psychiatric social worker.
(5) Linda is a member of the League of Women Voters.
(6) Linda is a bank teller.
(7) Linda is an insurance salesperson.
(8) Linda is a bank teller and active in a feminist movement.
Empirical Probability
• Historical data
• Estimate probability based on observations
• Example: Leading digits: 54,571
• How often do you think is the first/leading digit is 1 or 2 or 3 or….9
Benford’s Law
• Frequency of a digit “d” being the first place
!
• 𝑃 𝑑 = 𝑙𝑜𝑔!" 1 + , 𝑑 ∈ 1,2,3,4,5,6,7,8,9
#
d 1 2 3 4 5 6 7 8 9
P(d) .301 .176 .125 .097 .079 .067 .058 .051 .046
• This law holds for relative frequency probabilities in a variety of datasets
• Not applicable to assigned data
Conditional Probability
• Population dataset from India
• You came to visit India
• Q1: What is the probability that the first person you meet is younger than 20
years?
• Q2: This persons mother tongue is Hindi.
• Person residing in India:
• P(0-19) = 20.2%
• P(Hindi) = 22.4%
• When you talked to this person, he tells you: Actually, I am residing in the state of
Punjab
• P(0-19) = 32.4%
• P(Hindi) = 3.2%
Cont.
• Before the occurrence of Punjab:
• P(Hindi) = 22.4%
• After the occurrence of Punjab:
• P(Hindi) = 3.2%
• Change in Probability
• The probability of event A occurring is P(A) -> Information update, which states that event B
occurred
• What is the probability of event A occurring given that event B occurred?
• This new probability is called the conditional probability: P(A|B)
• P(Hindi) = 22.4%
• P(Hindi|Punjab) = 3.2%
Conditional Probability: Venn diagram
A B
!! " ! #"
!! " #" =
!! #"
S
Independence and Conditional Probability
• Events A and B are dependent
• Conditional probability is different from the original probability
• Otherwise, it will be same
• P(A|B) = P(A)
• P(B|A) = P(B)
Multiplication Rule
• General multiplication rule:
! "∩$
• 𝑃 𝐴𝐵 =
! $
! "∩$
• 𝑃(𝐵|𝐴) =
!(")
• 𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐴 𝐵 𝑃(𝐵) or 𝑃 𝐵 𝐴 𝑃(𝐴)
• Specialized multiplication rule:
• Independent:
• 𝑃 𝐵 𝐴 = 𝑃(𝐵) and 𝑃 𝐴 𝐵 = 𝑃(𝐴)
• 𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐴 𝑃(𝐵)
• Example:
• P(Punjab) = 17.6% and P(0-19) = 19.6%
• What is the proportion of Indians both live in Punjab and are 0-19 years?
Multiplication Rule
• Multiply probabilities all the times
• P(Punjab) = 0.176 and P(0-19|Punjab) = 0.196
• 𝑃(𝑃𝑢𝑛𝑗𝑎𝑏 ∩ 0 − 19 𝑦𝑒𝑎𝑟𝑠 𝑜𝑙𝑑)
• Apply the multiplication rule:
• P(Punjab)P(0-19 years|Punjab) = 0.176* 0.196 = 3.45%
• Rolling three fair dice: What is the probability of three ones in a row?
Multiplication Rule
• Assuming independence : multiplication rules simplified
• Eg. – We have a machine in the assembly line and that machine
carries heavy load and as a result it breaks down on average on one
out of 10 days
• P(good) = 0.9 and P(bad) = 0.1
• What is the probability that machine works fine for 2 days in a row?
• P(working well on first day)P(working well on 2nd day|worked well on
1st day)
Cont.
• 𝑤4 = Event that machine works well on day i
• 𝑓4 = Event that machine fails on day I
• P(𝑤5 ∩ 𝑤6 ) = P(𝑤5 )P(𝑤6 |𝑤5 )
• If we assumes that the events are independent, then P(𝑤6 |𝑤5 ) =
P(𝑤6 )
• P(𝑤5 ∩ 𝑤6 ) = 0.9*0.9 =0.81
Cont.
Probability Tables
• Limited Events
• To handle large number of probabilities
• Example: Overnight stays by foreign visitors in India
Africa America Europe Asia Totals
Hotel 0.010 0.065 0.064 0.461 0.600
Other 0.002 0.011 0.007 0.381 0.400
Totals 0.012 0.075 0.071 0.842 1.00
Cont.
B1 B2 Bk
A1 𝐴1 ∩ 𝐵1 𝐴1 ∩ 𝐵2 𝐴1 ∩ 𝐵𝑘
A2 𝐴2 ∩ 𝐵1 𝐴2 ∩ 𝐵2 𝐴2 ∩ 𝐵𝑘
Am 𝐴𝑚 ∩ 𝐵1 𝐴𝑚 ∩ 𝐵2 𝐴𝑚 ∩ 𝐵𝑘
Some Basic Definitions
• Marginal Probability
• The marginal probability is the unconditional probability on an event. This probability is not
conditioned on any other event.
• Notation: The marginal probabilities of two arbitrary events A and B is denoted by
respectively P(A) and P(B).
• Joint Probability
• The joint probability is the probability of simultaneous events. This is the probability of the
intersection of two or more events.
• Notation: The joint probability of two arbitrary events A and B is denoted by P(A⋂ B).
Cont.
• Conditional Probability
• The conditional probability is the probability of an event given that some other event has
occurred. The information from the event that has occurred can influence the probability of the
original event.
• Notation: For two events A and B, the conditional probability of A given B is denoted by P(A|
B) and the conditional probability of B given A is denoted by P(B|A).
• General Rule
• In general, for two arbitrary events A and B, P(A|B) ≠ P(B|A).
Bayes Rule
• Example:
• P(A) = Alarm sounds
• P(B) = Burglar in the house
• P(A), P(B), P(A|B), P(B|A)?
Example
• Manufacturing company – 3 suppliers (S, T and U)
• Good – 90% and 10% bad parts
• 60% good parts came from S, 25% good parts came from T and 15% good parts
came from U
• 40% bad – S, 30% - T, and 30% - U
• Which supplier delivers the highest proportion of good parts?
• P(good|S), P(good|T) and P(good|U)?
Cont.
• P(S|good) = 0.6, P(S|bad) = 0.4
• P(T|good) = 0.4, P(T|bad) = 0.3
• P(U|good) = 0.15, P(U|bad) = 0.3
S T U
Good 0.9 x 0.6 0.9 x 0.4 0.9 x 0.15 0.9
Bad 0.4 x 0.1 0.3 x 0.1 0.3 x 0.1 0.1
0.58 0.255 0.165 1
7(9""6∩;) 0.9 x 0.6
• P(good|S) = = = 0.93
7(;) =.>?
• P(good|T) = 0.882
• P(good|U) = 0.818
Formal Definition
• General multiplication rule:
• 𝑃 𝐴∩𝐵 =𝑃 𝐴 𝐵 𝑃 𝐵
• 𝑃(𝐴 ∩ 𝐵) = 𝑃 𝐵 𝐴 𝑃(𝐴)
• 𝑃 𝐴 𝐵 𝑃 𝐵 = 𝑃 𝐵 𝐴 𝑃(𝐴)
+(-|/)+ /
•𝑃 𝐵𝐴 = +(-)
+(/|-)+ -
•𝑃 𝐴𝐵 =
+(/)
Extended Bayes Rule
• P(B) – don’t know S
• 𝑃 𝐵 = 𝑃 𝐴 ∩ 𝐵 + 𝑃 𝐴̅ ∩ 𝐵
• 𝑃 𝐵 = 𝑃 𝐵 𝐴 𝑃 𝐴 + 𝑃 𝐵 𝐴̅ 𝑃 𝐴̅ A
=(>|@)= @ B
•𝑃 𝐴𝐵 = =(>) 𝑃(𝐴 ∩ 𝐵)
+(/|-)+ -
• 𝑃 𝐴 𝐵 = +(/|-)+ ̅
- 1+(/|-)+ -̅
• 𝑃 𝐴 = 𝑃 𝐴 ∩ 𝐵 + 𝑃 𝐴 ∩ 𝐵)
• 𝑃 𝐴 = 𝑃 𝐴 𝐵 𝑃 𝐵 + 𝑃 𝐴 𝐵) 𝑃 𝐵)
=(@|>)= > =(@|>)= >
•𝑃 𝐵𝐴 = =(@)
= =(@|>) B
> A=(@|>)= >B
Extended Bayes’ Rule
• 𝑃 𝐴 = 𝑃 𝐴 𝐵 𝑃 𝐵 + 𝑃 𝐴 𝐵) 𝑃 𝐵)
• This relation is analogous to Shannon’s Theorem in switching theory
and can be generalized w.r.t. the event space 𝑆′ = {𝐵5 , 𝐵6 , … , 𝐵C }:
• 𝑃 𝐴 = ∑C4D5 𝑃 𝐴 𝐵4 𝑃(𝐵4 )
• This relation is also known as the theorem of total probability or the
rule of elimination.
• Here the events 𝐵4 ’s are mutually exclusive and collectively exhaustive
Cont.
P(𝐵! )
𝐵! P(𝐴|𝐵! )
𝐴 ∩ 𝐵!
𝐵" 𝐴 ∩ 𝐵"
A
.. ..
𝐵# 𝐴 ∩ 𝐵#
Cont.
• In practice, a situation often arises in which the event A is known to have
occurred, but it is not known directly which mutually exclusive and
collectively exhaustive events 𝐵!, 𝐵3, … , 𝐵4 has occurred.
• In this situation, if we are interested in evaluating 𝑃 𝐵5 𝐴 i.e. the conditional
probability that one of these events, 𝐵5 occurs, given that A occurs.
+(-|/' )+ /' +(-|/' )+ /'
• 𝑃 𝐵5 𝐴 = ∑+ = +(-|/
()* +(-|/( )+(/( ) *) /* 1+(-|/, )+ /, 1⋯1+(-|/+ )+(/+ )
• 𝑃 𝐵5 𝐴 - Posteriori probability
Practice Problems
• Company A supplies 40% of the computers sold and is late 5% of the
time. Company B supplies 30% of the computers sold and is late 3% of
the time. Company C supplies another 30% and is late 2.5% of the
time. A computer arrives late - what is the probability that it came
from Company A?
Example 2
• In Orange County, 51% of the adults are males. One adult is randomly
selected for a survey involving credit card usage. It is later learned
that the selected survey subject was smoking a cigar. Also, 9.5% of
males smoke cigars, whereas 1.7% of females smoke cigars (based on
data from the Substance Abuse and Mental Health Services
Administration). Use this additional information to find the
probability that the selected subject is a male.