Probability Fundamentals for IT
Probability Fundamentals for IT
Information Technology
Introduction to Probability
• Elements of Probability
• Introduction
• Experiments & Outcomes
• Sample Spaces
• Events
• Algebra of Events
• Probability
• Combinatorial Theory
𝑃 =?
March 19, 2024 3
Language of Probability
• A sample space of an experiment, denoted by S or Ω, is the set of all possible outcomes of the
experiment.
• Examples of Sample space and Outcomes:
o Rolling a die:
• Sample space: {1, 2, 3, 4, 5, 6}. Each number is an outcome.
o Determining the gender of a newborn:
• Sample space: S = {girl, boy}.
o Two-child scenario:
• An incorrect approach may define the sample space as {two girls, two boys, one girl
and one boy}, but these are not equally likely.
• The correct approach assumes each child’s gender is independent (boy or girl),
yielding 4 equally likely outcomes: {boy-boy, girl-girl, boy-girl, girl-boy}.
March 19, 2024 6
Events
• The union of two events A and B is the event that occurs if at least one of
A or B occurs.
• Mathematical definition:
• Example:
• Let A = {1, 2, 3}, B = {3, 4, 5}
• Then A ∪ B = {1, 2, 3, 4, 5}
• The intersection of A and B is the event that occurs only if both A and
B occur simultaneously.
• Mathematical definition:
• Example:
• A = {2, 4, 6}, B = {4, 5, 6}
• A ∩ B = {4, 6}
• The complement of an event A is the event that A does not occur — that
is, all outcomes in the sample space except those in A.
• Mathematical definition:
• Example:
• If S = {1, 2, 3, 4, 5, 6}, A = {1, 3, 5}
• Then Aᶜ = {2, 4, 6}
• Example:
• A = {1, 2, 3}, B = {2, 4}
• A – B = {1, 3}
• The symmetric difference includes all outcomes that are in A or B but not
in both.
• Mathematical definition:
• Example:
• A = {1, 2, 3}, B = {3, 4, 5}
• A Δ B = {1, 2, 4, 5}
• Example:
• A = {1, 2}, B = {3, 4}
⇒ A and B are mutually exclusive.
• Let’s consider an example where our experiment rolls two fair 6-sided dices: one
red (D1) and one green (D2).
• Since the dices are distinguishable by color, the order of the outcomes matters.
• We can represent each possible result in the sample space as an ordered pair (D1,
D2), where D1, D2 ∈ {1, 2, 3, 4, 5, 6} are the values rolled by the red and green
dice. 1 2 3 4 5 6
• For example: 1 (1, 2) (1, 2) (1, 3) (1, 4) (1, 5) (1, 6)
2 (2,1) (2, 2) (2, 3) (2, 4) (2, 5) (2, 6)
(D1, D2) = (2, 3) = ( )
3 (3, 1) (3, 2) (3, 3) (3, 4) (3, 5) (3, 6)
4 (4, 1) (4, 2) (4, 3) (4, 4) (4, 5) (4, 6)
5 (5, 1) (5, 2) (5, 3) (5, 4) (5, 5) (5, 6)
6 (6, 1) (6, 2) (6, 3) (6, 4) (6, 5) (6, 6)
March 19, 2024 17
Example 1.1 (cont.)
• The sample space Ω is the set of all possible ordered pairs of values that could be rolled
by the die (|Ω| = 6 · 6 = 36 by the product rule). Let’s consider some events:
1. A = {(1, 1), (1, 2), (1, 3), (1, 4), (1, 5), (1, 6)}, the event that the red die is a 1.
2. B = {(1, 5), (2, 4), (3, 3), (4, 2), (5, 1)}, the event that the sum of the two rolls is 6.
3. C = {(1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6)}, the event that the value of both die is
equal.
1 2 3 4 5 6
4. Are A and B mutually exclusive? 1 (1, 1)A,C (1, 2)A (1, 3)A (1, 4)A (1, 5)A,B (1, 6)A
5. Are B and C mutually exclusive? 2 (2,1) (2, 2)C (2, 3) (2, 4)B (2, 5) (2, 6)
3 (3, 1) (3, 2) (3, 3)B,C (3, 4) (3, 5) (3, 6)
4 (4, 1) (4, 2)B (4, 3) (4, 4)C (4, 5) (4, 6)
5 (5, 1)B (5, 2) (5, 3) (5, 4) (5, 5)C (5, 6)
6 (6, 1) (6, 2) (6, 3) (6, 4) (6, 5) (6, 6)C
March 19, 2024 18
Algebra of Events
• The Algebra of Events refers to the set of operations and rules that define how events
in probability can be combined and related to each other.
• It provides a mathematical framework to manipulate events—sets of outcomes in a
sample space—similar to how algebra manipulates numbers.
• The operations follow a set of logical and mathematical laws, ensuring that the
results are both logically sound and mathematically valid. This framework allows us
to:
o Express complex and straightforward scenarios with precision.
o Systematically compute probabilities for combined or modified events.
o Apply consistent reasoning even in situations involving multiple overlapping
conditions.
March 19, 2024 19
Fundamental Laws of Event Algebra
• Commutative Laws
• Associative Laws
• Distributive Laws
• De Morgan’s Laws
• Elements of Probability
• Probability
• Frequentist & Bayesian Interpretation
• Axioms of Probability
• Corollaries of the Probability Axioms
• Equally Likely Outcomes
• Combinatorial Theory
• In the 20th century, one widely accepted way to define probability emerged
through the frequentist interpretation.
• Suppose we perform n independent trials of an experiment, and let n(E) represent
the number of trials in which the outcome belongs to the event E. The
probability of E is then defined as:
View on Randomness is an inherent property of the process being Randomness partly reflects uncertainty in our
Randomness studied. knowledge about the process.
Data Requires a large number of identical and repeatable Can work with limited data, using prior information
Requirement trials to estimate probabilities accurately. to compensate for scarcity.
Role of Prior Does not incorporate prior beliefs; probability is Explicitly incorporates prior beliefs or historical data
Knowledge determined purely from observed frequencies. into probability assessments.
Typical Quality control, gambling odds, physical sciences, Machine learning, decision-making under uncertainty,
Applications classical statistical inference. medical diagnostics, adaptive systems.
Objective, based solely on observed data; no subjective Flexible, can update beliefs as new evidence is
Strengths
bias from prior assumptions. collected; effective with small datasets.
Not practical for rare events or small datasets; ignores Can be subjective; results depend on the choice of
Limitations
prior information. prior.
March 19, 2024 26
Axioms of Probability
• Let Ω be a sample space (the set of all possible outcomes of a random experiment), and
let ℱ be a σ-algebra of subsets of Ω. A probability measure is a function:
• More generally, for any countable collection of pairwise mutually exclusive events E1,
E2, E3, …:
1. Complementation: since
This follows from the fact that the sample space contains all possible outcomes.
2. Monotonicity: if then
3. Inclusion–Exclusion Principle:
If events E and F share common outcomes, adding P(E) and P(F) counts those shared
outcomes twice. To correct this, we subtract P(E∩F) once so that the intersection is
counted only once, avoiding overcounting.
• In probability theory, a common and simple case occurs when all outcomes in a sample
space are equally likely — that is, each outcome has the same probability of occurring.
• This situation often arises in fair random experiments: tossing a fair coin, rolling a fair
die, or drawing a card from a well-shuffled deck.
• Definition:
• Let the sample space be , where n is the total number of possible
outcomes.
• If each outcome is equally likely, then:
• Since the total probability of the sample space is 1:
• If every outcome in the sample space Ω is equally likely, then for any event E ⊆
Ω containing ∣E ∣ outcomes, we can represent it as:
• Since each outcome has the same probability , the probability of E is:
• Elements of Probability
• Probability
• Combinatorial Theory
• Counting Methods
• Key Sampling Definitions
• The Multiplication Principle
• Permutations
• Combinations
• The real challenge in many problems lies in finding ∣E∣ and ∣S∣ efficiently—
especially when the sample space is large and listing every possibility is
impractical.
• Counting methods are systematic mathematical tools that allow us to determine
the number of possible arrangements, selections, or groupings without the need to
enumerate each one explicitly.
• In counting problems, the way we select objects significantly impacts the total number of
possible outcomes. The following definitions are essential:
o Sampling with replacement: means that after each selection, the chosen item is
returned to the original set, allowing it to be picked again with probability 1/n.
o Sampling without replacement: means that once an item is selected, it is removed
from the set, reducing the number of possible selections by one each time.
o Distinguishable objects: changing the order of the same objects results in a different
outcome. For example, the sequence (red, blue) is different from (blue, red).
o Indistinguishable objects: changing the order does not result in a new outcome. For
example, if objects are identical (two red balls), swapping them doesn’t produce a new
arrangement.
• Definition:
• A permutation is an arrangement of a set of objects in a specific order.
• The order matters — changing the order of elements creates a different
permutation.
• Permutations are used when we want to know how many different ordered
arrangements can be formed from a set of items.
• Types of Permutations:
• Permutations without repetition.
• Permutations with repetition.
• Permutations with identical items.
March 19, 2024 38
Permutations Without Repetition
• Permutations with repetition (also called permutations with replacement) occur when
we arrange a set of objects where repetition is allowed and the order matters.
• Each position in the arrangement can be filled by any element from the set.
• The same element may appear multiple times in different positions.
• Changing the order of elements produces a different arrangement, even if some
elements are repeated.
• If we have n possible distinct objects and want to form an arrangement of
length r with repetition allowed, then the total number of possible arrangements is:
• Interpretation:
• The first character has 10 possible choices (0–9).
• The second character also has 10 choices, and so on for the third and fourth.
• Permutations with identical items (also called permutations of a multiset) arise when
we arrange a set of n objects where some objects are indistinguishable from each
other:
• If all objects were different, the number of arrangements would be n!.
• When certain objects are identical, swapping them does not create a new
arrangement, so we must divide by the number of ways the identical items can be
permuted among themselves.
• Suppose we have n total objects, divided into k groups of identical items:
• n1 items of type 1 (identical to each other);
• …
• nk items of type k.
• The number of distinct permutations is:
• Problem: Determine the number of distinct permutations of the letters in the word
‘LEVEL’.
• Solution:
• The word LEVEL has 5 letters in total, n = 5.
• In “LEVEL”, L appears 2 times, E appears 2 times, V appears 1 time. So, we have:
• Definition:
• A combination is a selection of objects from a set where the order of selection
does not matter and no object is selected more than once.
• Combinations are used when the arrangement sequence is irrelevant — the
same objects in a different order are considered the same selection.
• Formula:
• If we have n distinct objects and want to choose r of them, the number of
possible combinations is:
• Computational shortcuts:
• Instead of computing C(n, k) directly by the formula, we can simplify the
fraction.
• Special cases:
• C (n, 0)=1 — choosing nothing is one way.
• C (n, 1)=n — choosing one item has n possibilities.
• C (n, n)=1 — choosing all items is one way.
• Problem: In a lottery, 6 distinct numbers are chosen from {1, 2, …, 49}. How many
different tickets are possible?
• Solution:
• Each ticket is a set of 6 different numbers. Order does not matter. No repetition (you
cannot pick the same number twice). So this is a combination problem:
• Interpretation:
• If order mattered, there would be 49 ⋅ 48 ⋅ 47 ⋅ 46 ⋅ 45 ⋅ 44 ordered ways (the
multiplication rule).
• But each unordered 6-number ticket is counted 6! times among those ordered lists (all
permutations of the same 6 numbers). Therefore divide by 6!.
• Problem: A bridge hand is made by selecting 13 cards from a standard 52-card deck,
where the order of cards does not matter. The 52 cards are all distinct.
Calculate the probability of drawing a 13-card hand containing only black cards (spades
♠ or clubs ♣).
Solution
• A bridge hand is formed by selecting any 13 cards from a standard 52-card
deck, regardless of order. Since all 52 cards are distinct, the number of different possible
bridge hands is given by the combination:
• Since there are 13 cards in each of the 4 suits (♥,♦,♣, ♠), The number of hands that
consist entirely of black cards (no ♥ or ♦) is:
• If the deck is well-shuffled and each of the 635 billion possible hands is equally likely, then
the probability P(A) of drawing a hand made up entirely of spades or clubs is:
Image from
Internet.
March 19, 2024 50
Example 1.11: Three of a Kind (cont.)
Solution
• All 5-card hands are equally likely; the total number of hands is:
• Count hands that are exactly “three of a kind”:
1. Choose the rank for the triple:
2. Choose its suits:
3. Choose the two kicker ranks (distinct from the triple and from each other):
4. Choose suits for each kicker:
Þ The number of “three of a kind” is:
• Therefore:
1. Provide a specific example for each of the nine types of probability events listed on
page 9. These include: simple event, compound event, sure event, impossible event,
mutually exclusive events, exhaustive events, equally likely events, independent events,
and dependent events.
2. Finding the Probability of the Intersection of Events from a Venn Diagram:
3. From a group of 6 men and 4 women, 5 people are randomly selected. Find the
probability of the following events:
a. Selecting at least one man.
b. Selecting 5 women.
4. Consider the experiment of tossing two fair coins simultaneously. Each coin has
two possible outcomes: Head (H) or Tail (T):
a. List the sample space of all possible outcomes.
b. Determine the probability of the event “at least one coin shows Head”.
7. A box contains 10 candies, of which 3 are red. Three candies are randomly
selected from the box. Find the probability of selecting at least one red candy.
8. In a standard poker game using a 52-card deck, a straight is a 5-card hand in
which the cards have consecutive ranks (e.g., 7-8-9-10-J), regardless of suit.
The Ace (A) can be used as either the lowest card (A-2-3-4-5) or the highest
card (10-J-Q-K-A). What is the probability of being dealt a straight in 5 cards?
a. Counting all straights, including straight flushes (all of the same suit).
b. Counting only straights that are not straight flushes.
[1] Michael Baron. (2014). Probability and Statistics for Computer Scientists, Second Edition (2nd.
ed.). Chapman & Hall/CRC.
[2] Molina Peralta, I. and García-Portugués, E. (2024). A First Course on Statistical Inference.
Version 2.4.1. ISBN 978-84-09-29680-4. Available at [Link]
[3] Sheldon M. Ross. (2009). Introduction to probability and statistics for engineers and scientists (4th.
ed.). Elsevier.
[4] Marco Taboga. (2021). Lectures on probability theory and mathematical statistics. Kindle Direct
Publishing. Online appendix.
[5] Hossein Pishro-Nik. (2014). Introduction to probability, statistics, and random processes. available
at [Link] Kappa Research LLC.
[6]