Conditional Probability: From First Principles to

Advanced Problem Solving

PART 1 — FOUNDATIONS
1.1 The Core Idea
Conditional probability answers one question: "Given that I already know something happened,
how does that change the odds of something else?"
Information changes probability. If you roll a fair die, P(result = 6) = 1/6. But if someone tells you
"the result is even," the world of possibilities shrinks from {1,2,3,4,5,6} to {2,4,6}, and now
P(result = 6 | even) = 1/3. Same die, same event, different probability — because you conditioned
on new information.

1.2 The Formal Definition

For events A and B, with P(B) > 0:
P(A | B) = P(A ∩ B) / P(B)
Read as "probability of A given B."
Why this formula, intuitively: Once you know B has happened, your entire sample space
collapses down to just B. Within that shrunken world, the only way A can also be true is if A and B
both happened (A ∩ B). So you're asking: "out of everything in B, what fraction also lies in A?"
That's exactly P(A∩B)/P(B).
Visual (Venn diagram) intuition: Imagine B as a circle. A∩B is the overlapping sliver.
Conditioning on B means B becomes your whole universe — so the answer is just "what fraction of
B's area is covered by A."

1.3 The Multiplication Rule

Rearranging the definition:
P(A ∩ B) = P(A | B) · P(B) = P(B | A) · P(A)
This is enormously useful for building up probabilities of joint events step-by-step (sequential
experiments, tree diagrams).
Chain rule (n events): P(A₁ ∩ A₂ ∩ ... ∩ Aₙ) = P(A₁) · P(A₂|A₁) · P(A₃|A₁∩A₂) · ... · P(Aₙ |
A₁∩...∩Aₙ₋₁)

1.4 Independence
A and B are independent if knowing B tells you nothing new about A:
P(A | B) = P(A) ⟺ P(A ∩ B) = P(A)·P(B)
Common trap: independent ≠ mutually exclusive. In fact, if A and B are mutually exclusive (and
both have positive probability), they are automatically dependent — because if B happens, A
definitely didn't, so P(A|B) = 0 ≠ P(A).

1.5 Law of Total Probability

If B₁, B₂, ..., Bₙ partition the sample space (mutually exclusive, collectively exhaustive):
P(A) = Σ P(A | Bᵢ) · P(Bᵢ)
This lets you compute an "unconditional" probability by breaking reality into cases, solving each
case (which is often easier), and weighting by how likely each case is.

1.6 Bayes' Theorem

The single most important reversal tool in probability:
P(B | A) = [P(A | B) · P(B)] / P(A)
Combined with total probability:
P(Bᵢ | A) = [P(A | Bᵢ) · P(Bᵢ)] / Σⱼ P(A | Bⱼ) · P(Bⱼ)
Vocabulary:
• P(Bᵢ) = prior (belief before evidence)
• P(A | Bᵢ) = likelihood (how well hypothesis Bᵢ explains evidence A)
• P(Bᵢ | A) = posterior (updated belief after evidence)
• P(A) = evidence / normalizing constant

PART 2 — GOOD-THINKING SOLVED PROBLEMS (Building

Intuition)
Problem 2.1 — The Two-Child Problem (classic, deceptively subtle)
A family has two children. At least one is a girl. What's the probability both are girls?
Naive wrong answer: 1/2 (people assume the "other" child is a coin flip).
Correct approach: List the sample space of two children by birth order, each equally likely: {BB,
BG, GB, GG}
"At least one girl" eliminates BB. Remaining: {BG, GB, GG} — three equally likely outcomes.
P(both girls | at least one girl) = P(GG ∩ at least one girl) / P(at least one girl) = (1/4) / (3/4) = 1/3
Why intuition fails: People implicitly picture "we already know which child is the girl" — but the
problem doesn't say which child, only that at least one is. That's a strictly weaker condition, and it
keeps BG and GB both alive as distinct possibilities.
The variant that trips people up further: "At least one is a girl born on a Tuesday." Surprisingly,
this changes the answer (to 13/27!) because specifying which girl (via a near-identifying detail)
shifts weight back toward 1/2-ish reasoning. This is a famous result showing how sensitive
conditional probability is to exactly what you condition on.

Problem 2.2 — Testing for a Rare Disease (the canonical Bayes

problem)
A disease affects 1% of a population. A test is 99% accurate for both sick and healthy people (99%
true positive rate, 99% true negative rate — i.e., 1% false positive rate). You test positive. What's
the probability you actually have the disease?
Setup with Bayes:
• P(Disease) = 0.01, P(No Disease) = 0.99
• P(Positive | Disease) = 0.99
• P(Positive | No Disease) = 0.01 (false positive rate)
P(Disease | Positive) = [P(Positive|Disease)·P(Disease)] / [P(Positive|Disease)·P(Disease) +
P(Positive|No Disease)·P(No Disease)]
= (0.99 × 0.01) / (0.99 × 0.01 + 0.01 × 0.99) = 0.0099 / (0.0099 + 0.0099) = 0.0099 / 0.0198 = 0.5
(50%)
The lesson (base rate neglect): Even with a "99% accurate" test, if the disease is rare, roughly half
of positive results are false alarms. The test's accuracy is overwhelmed by how few sick people
there are to begin with. This single insight is the difference between naive and trained probabilistic
reasoning — it shows up in medicine, spam filters, security screening, and legal reasoning (the
"prosecutor's fallacy").
Natural-frequency version (easier intuition, no formula): Imagine 10,000 people. 100 are sick
(1%). Of those, 99 test positive. Of the 9,900 healthy, 1% = 99 false-positive. Total positives = 99 +
99 = 198, of which only 99 are truly sick → 99/198 = 50%. This "population of 10,000" trick makes
Bayes' theorem almost unnecessary to memorize — it is Bayes' theorem, just concretized.

Problem 2.3 — Three Cards (a clean deduction exercise)

You have 3 cards: one is red on both sides, one is white on both sides, one is red on one side and
white on the other. You pick a random card and see a random face — it's red. What's the
probability the other side is also red?
Wrong intuitive answer: 1/2 (since it's either the red/red card or the red/white card).
Correct: count faces, not cards. There are 6 faces total: R1,R2 (red/red card), W1,W2
(white/white card), R3,W3 (red/white card). "You see red" means you're equally likely to be
looking at R1, R2, or R3 (3 equally likely faces).
Of these, R1 and R2 belong to the red/red card (other side is red); only R3 belongs to the mixed
card (other side is white).
P(other side red | seeing red) = 2/3
The lesson: When outcomes aren't naturally equally likely at the "card" level, drop down to the
truly equally-likely elementary outcomes (faces, not cards). This is the single most common source
of conditional probability errors — conditioning at the wrong "grain size."

PART 3 — TOUGHER / LENGTHIER SOLVED PROBLEMS

Problem 3.1 — Monty Hall (with full rigorous justification)
Three doors: one has a car, two have goats. You pick door 1. The host, who knows what's behind
each door, opens a different door revealing a goat. Should you switch?
Full solution via Bayes:
Let C = door with car (uniform over {1,2,3}). You pick door 1. Host opens door 2 or 3 (never your
door, never the car door).
We want P(C=1 | host opens door 3), compared to P(C=2 | host opens door 3).
Case C=1 (car behind your door): host can open either door 2 or 3, each with probability 1/2 (he has
a free choice). So P(opens 3 | C=1) = 1/2.
Case C=2: host must open door 3 (can't open door 2 — car's there; can't open door 1 — you picked
it). P(opens 3 | C=2) = 1.
Case C=3: host must open door 2 (never his own car door). P(opens 3 | C=3) = 0.
By Bayes: P(C=1 | opens 3) = [1/2 × 1/3] / [1/2×1/3 + 1×1/3 + 0×1/3] = (1/6) / (1/6 + 1/3) =
(1/6)/(1/2) = 1/3
P(C=2 | opens 3) = (1×1/3) / (1/2) = (1/3)/(1/2) = 2/3
Switching doubles your odds: 1/3 (stay) vs 2/3 (switch).
Why intuition fails so badly here: People treat the two remaining doors as symmetric after the
reveal. But the host's action isn't random — it's informed and constrained. He was forced not to
open the car door, and that constraint injects information asymmetrically: door 2 "absorbed" all the
probability mass of every scenario where the car could have been behind door 2 or 3, funneled
entirely onto the door he didn't open.
Fast intuitive check: Imagine 100 doors, you pick 1, host opens 98 goat-doors, leaving one other
door standing. Your original door: 1/100. That untouched door: 99/100 (it "absorbed" the near-
certainty that the car was among the other 99). Switching is obviously correct at this scale, and the
3-door case is the same logic, just compressed.

Problem 3.2 — Sequential Bayesian Updating (Multiple Pieces of

Evidence)
You have two coins: a fair coin (P(H)=0.5) and a biased coin (P(H)=0.8). You pick one at random
and flip it 3 times, getting H, H, T. What's the probability you picked the biased coin?
Setup:
 • Prior: P(Fair) = P(Biased) = 0.5
• Likelihood of HHT under Fair: 0.5×0.5×0.5 = 0.125
• Likelihood of HHT under Biased: 0.8×0.8×0.2 = 0.128
By Bayes: P(Biased | HHT) = (0.128 × 0.5) / (0.128×0.5 + 0.125×0.5) = 0.064 / (0.064+0.0625) =
0.064/0.1265 ≈ 0.506
Key insight — sequential updating is multiplicative and order-independent: You could instead
update after each flip one at a time (posterior after flip 1 becomes the prior for flip 2, etc.), and
you'd get the exact same final answer, regardless of the order H,H,T vs H,T,H vs T,H,H. This is
because Bayesian updating with independent observations is just accumulating a product of
likelihood ratios — order doesn't matter, only the final tally of evidence (2 heads, 1 tail) matters.
Generalizable formula (odds form of Bayes — often faster for multi-evidence problems):
Posterior odds = Prior odds × Likelihood ratio (for each independent piece of evidence, multiply in
another likelihood ratio)
Here: Prior odds (Biased:Fair) = 1:1. Likelihood ratio per H = 0.8/0.5 = 1.6. Likelihood ratio per T
= 0.2/0.5 = 0.4. Posterior odds = 1 × 1.6 × 1.6 × 0.4 = 1.024 : 1 → P(Biased) = 1.024/2.024 ≈ 0.506
✓ (matches, and was much faster to compute).

Problem 3.3 — A Lengthy Multi-Stage Tree Problem

An electronics factory has 3 machines: A (40% of output), B (35% of output), C (25% of output).
Defect rates: A = 2%, B = 3%, C = 5%. A product is picked at random and found defective.
Additionally, defective products from Machine B are caught by a secondary inspection with 90%
accuracy (90% chance a truly-B-defective item is correctly flagged as defective; assume A and C
items are directly reported without this secondary step). Given the item was reported defective, find
P(it came from B).
(This is deliberately messy to force careful modeling — a hallmark of "real" applied problems.)
Step 1 — Raw joint probabilities of (machine, truly defective):
• P(A, defective) = 0.40 × 0.02 = 0.008
• P(B, defective) = 0.35 × 0.03 = 0.0105
• P(C, defective) = 0.25 × 0.05 = 0.0125
Step 2 — Apply the reporting mechanism. A and C defects are reported directly (100% of the
time, per the problem's assumption). B defects pass through inspection with 90% detection:
• P(A, reported defective) = 0.008 × 1 = 0.008
• P(B, reported defective) = 0.0105 × 0.90 = 0.00945
• P(C, reported defective) = 0.0125 × 1 = 0.0125
Step 3 — Total probability of "reported defective": P(reported defective) = 0.008 + 0.00945 +
0.0125 = 0.02995
Step 4 — Bayes: P(B | reported defective) = 0.00945 / 0.02995 ≈ 0.3155 (31.6%)
Why this problem is a good exercise: It forces you to distinguish true conditional structure (defect
rates) from a secondary filtering process (inspection), and to correctly multiply through a two-stage
chain before applying Bayes. Many real-world problems (medical screening pipelines, fraud
detection with human review layers, multi-sensor systems) have exactly this "detection-rate on top
of base-rate" structure.

PART 4 — UNIQUE / COUNTERINTUITIVE THINKING

PROBLEMS
Problem 4.1 — Simpson's Paradox (conditional probability reversing
under aggregation)
Two treatments for a condition are each tested. In small stones: Treatment A cures 93% (81/87),
Treatment B cures 87% (234/270). In large stones: Treatment A cures 73% (192/263), Treatment B
cures 69% (55/80). So A beats B in BOTH subgroups. But overall: A cures (81+192)/(87+263) =
273/350 = 78%, while B cures (234+55)/(270+80) = 289/350 = 82.6%. Overall, B beats A!
Resolution: This isn't a contradiction — it's a warning about conditioning on the wrong variable,
or failing to condition on a confounder. Treatment A was disproportionately used on the harder
cases (large stones, lower cure rates for everyone), while B was used mostly on easy cases. The
aggregate numbers conflate treatment effectiveness with case difficulty. The correct causal
comparison is the stratified (within-subgroup) one — the overall comparison is confounded by an
unequal case mix.
General lesson: P(A wins | small stones) and P(A wins | large stones) can each favor A while P(A
wins) overall favors B, whenever the "weighting" of subgroups differs between the two things being
compared. This is why controlling for confounders is essential in causal reasoning, and why
marginal/aggregate statistics can actively mislead.

Problem 4.2 — The Boy Born on Tuesday (extreme sensitivity to

conditioning detail)
Mr. Smith has two children. One is a boy born on a Tuesday. What's the probability both children
are boys?
Naively you might expect 1/3 (as in Problem 2.1's basic version), or possibly assume the Tuesday
detail is irrelevant and it should still be 1/3. Actually it's neither — it's 13/27.
Why: Enumerate (gender, day-of-week) for both children: 14 possibilities per child (7 days × 2
genders), so 196 equally likely combos for two children. We condition on "at least one child is a
boy born on Tuesday."
Using complementary counting: P(no child is a Tuesday-boy) = (13/14)² = 169/196. So P(at least
one Tuesday-boy) = 27/196.
Among these 27, count how many have both children as boys. Using inclusion-exclusion on "boy-
born-Tuesday" being child 1 or child 2: this works out to 13 favorable outcomes (both boys, at least
one born Tuesday) out of 27.
P(both boys | at least one Tuesday-boy) = 13/27 ≈ 0.481
The deep lesson: As you add more specific identifying information about which individual satisfies
the condition, the answer drifts from 1/3 toward 1/2. In the limit of a "maximally specific" identifier
(e.g., "the boy born on March 3rd at 2:14pm"), the answer converges to exactly 1/2, because the
specific detail effectively "tags" one particular child, collapsing the problem back into "given this
specific child is a boy, is the other a boy?" — an independent coin flip. This is a genuinely subtle
and famous result (popularized by Gary Foshee) precisely because it shows conditional probability
isn't just about what you're told, but how specifically you're told it.

Problem 4.3 — The Prosecutor's Fallacy (confusing P(A|B) with P(B|A))

A crime is committed. A rare genetic/forensic marker found at the scene matches the defendant.
This marker occurs in 1 in 1,000,000 people. The prosecutor argues: "There's only a 1-in-a-million
chance this evidence would match an innocent person, so there's a 999,999-in-a-million chance the
defendant is guilty."
Why this is a fallacy: The prosecutor has confused P(match | innocent) = 1/1,000,000 with
P(innocent | match), which is what actually matters. These are NOT the same quantity (this is
exactly Bayes' theorem's warning).
Suppose the city has 8 million people, so roughly 8 innocent people would be expected to match the
marker purely by chance (8,000,000 × 1/1,000,000 = 8), in addition to the true culprit. Given a
match, the suspect is one of roughly 9 people who'd match (8 innocent + 1 guilty) — so P(guilty |
match) ≈ 1/9, drastically lower than "999,999 in a million"!
Bayes' theorem, explicitly: P(guilty | match) = P(match|guilty)·P(guilty) / [P(match|
guilty)·P(guilty) + P(match|innocent)·P(innocent)]
This requires a prior P(guilty) — some baseline probability the defendant did it before considering
the marker (e.g., based on being in the suspect pool at all) — which the forensic statistic alone can
never supply. Ignoring the prior (or equivalently, ignoring the size of the population that could have
matched by chance) is precisely the fallacy, and it has caused real, documented wrongful
convictions.

Problem 4.4 — Conditional Probability Can Be Non-Transitive (a

genuine "unique thinking" trap)
Claim: if P(A|B) > P(A) and P(B|C) > P(B), does P(A|C) > P(A)?
This is FALSE in general — a countable construction shows conditioning "effects" don't chain
transitively the way people assume. This matters in fields like epidemiology and machine learning
feature selection, where "B is a good predictor of A, and C is a good predictor of B" does not
guarantee "C is a good predictor of A" (especially if B is a collider or the relationships involve
suppression effects). This is closely related to why correlation is not transitive and why causal-
graph reasoning (not naive probability chaining) is needed for complex systems.

PART 5 — ADVANCED / GRADUATE-LEVEL EXTENSIONS

5.1 Conditional Probability with Continuous Random Variables
For continuous X, Y with joint density f(x,y), the conditional density is:
f(x | y) = f(x,y) / f_Y(y), valid wherever f_Y(y) > 0
This is the continuous analogue of P(A|B) = P(A∩B)/P(B) — but note we condition on an event of
probability zero (X=x exactly has zero probability for continuous variables), which is only
rigorously justified via measure theory (the Radon-Nikodym derivative, or heuristically, taking a
limit of conditioning on y ∈ [y, y+ε] as ε→0).
Example: If (X,Y) is uniform on the unit disk, then conditional on Y=y, X is uniform on the
interval [−√(1−y²), √(1−y²)]. The conditional distribution's shape itself changes depending on the
conditioning value — this "conditioning reshapes the support" phenomenon has no analogue in
simple discrete examples and trips up many students moving from discrete to continuous
probability.

5.2 Conditional Expectation

E[X | Y=y] = ∫x · f(x|y) dx (continuous) or Σx · P(X=x|Y=y) (discrete)
Tower Property (Law of Total Expectation): E[X] = E[ E[X|Y] ]
This is one of the most powerful tools in probability theory — E[X|Y] is itself a random variable (a
function of Y), and taking its expectation recovers E[X]. Used constantly in:
• Computing expectations via conditioning on a convenient auxiliary variable (e.g., number of
customers, first-step analysis in random walks)
• Martingale theory (E[Xₙ₊₁ | X₁,...,Xₙ] = Xₙ defines a martingale)
Conditional Variance formula (Law of Total Variance): Var(X) = E[Var(X|Y)] + Var(E[X|Y])
This decomposes total variance into "average within-group variance" plus "variance of group
means" — the probabilistic backbone of ANOVA (analysis of variance) in statistics.

5.3 Markov Property as Conditional Independence

A stochastic process has the Markov property if:
P(Xₙ₊₁ = x | X₁=x₁,...,Xₙ=xₙ) = P(Xₙ₊₁ = x | Xₙ=xₙ)
In words: the future, conditioned on the present, is independent of the past. This single conditional-
independence statement is the foundation of Markov chains, hidden Markov models, and
reinforcement learning's Markov Decision Processes.

5.4 Conditional Independence (the general concept)

A and B are conditionally independent given C if:
P(A ∩ B | C) = P(A|C) · P(B|C)
Crucially: A, B independent does NOT imply A, B conditionally independent given C, and vice
versa. This is the entire basis of Simpson's Paradox above, and of graphical models (Bayesian
networks), where edges encode exactly these conditional (in)dependence structures. A classic case:
two independent fair coins A, B; let C = "A and B show the same face." A and B are marginally
independent, but given C, knowing A tells you everything about B (they're now perfectly
correlated). Conditioning can create dependence between variables that were originally independent
— this is called "explaining away" or the collider effect in causal inference (e.g., two independent
causes of an effect become dependent once you condition on the effect being observed).

5.5 Bayesian Networks (brief orientation)

A Bayesian network encodes a joint distribution as a product of conditionals along a directed
acyclic graph:
P(X₁,...,Xₙ) = ∏ᵢ P(Xᵢ | Parents(Xᵢ))
Every piece of machinery above (chain rule, conditional independence, Bayes updating) is the
mathematical substrate underneath this. Modern applications: spam filters (naive Bayes), medical
diagnosis expert systems, and the belief-propagation algorithms underlying much of probabilistic
machine learning.

PART 6 — PRACTICE QUESTIONS (Basic → Extreme)

Tier 1 — Basic
1. A fair die is rolled. Find P(result > 4 | result is even).
2. A card is drawn from a standard deck. Find P(it's a King | it's a face card).
3. Two fair coins are flipped. Find P(both heads | at least one head).
4. In a class, 60% study math, 40% study physics, 25% study both. Find P(studies physics |
studies math).
5. If P(A) = 0.3, P(B) = 0.4, and A, B are independent, find P(A | B) and P(A ∩ B).
Tier 2 — Intermediate 6. A box has 5 red and 3 blue balls. Two balls are drawn without
replacement. Find P(second is red | first is red). 7. A factory has two machines; Machine 1 makes
70% of items with a 4% defect rate, Machine 2 makes 30% with a 7% defect rate. An item is
defective — find P(it came from Machine 2). 8. In a family with 3 children, given that at least two
are girls, find P(all three are girls). 9. A spam filter flags 95% of actual spam and incorrectly flags
2% of legitimate emails as spam. If 30% of all incoming email is spam, and an email is flagged,
find P(it's actually spam). 10. Events A, B, C are pairwise independent. Does that guarantee
P(A∩B∩C) = P(A)P(B)P(C)? Justify with reasoning (not just yes/no).
Tier 3 — Advanced 11. Rework the "Two-Child Problem" (2.1) but with three children,
conditioning on "at least one is a girl." Find P(all three are girls | at least one girl). 12. A jar has an
unknown number of coins, either all fair or all double-headed, with prior 50/50. You flip a
randomly chosen coin from the jar 5 times and get 5 heads. Find the posterior probability the coin is
double-headed... but suppose you flip it a 6th time and get a tail — what does this imply, and what
does it reveal about the model's assumptions? 13. Derive the Law of Total Variance formula Var(X)
= E[Var(X|Y)] + Var(E[X|Y]) starting only from the definition of variance and the Tower
Property. 14. Construct your own explicit numerical example demonstrating Simpson's Paradox
with two different categories (not medical treatments — try it with, e.g., batting averages across two
halves of a season, or admission rates across two departments). 15. In a Bayesian network with
three variables A → B → C (a simple chain), prove that A and C are conditionally independent
given B, directly from the factorization P(A,B,C) = P(A)P(B|A)P(C|B).
Tier 4 — Extreme / Research-flavored 16. Prove or disprove: if P(A|B) > P(A) and P(B|C) >
P(B), then necessarily P(A|C) > P(A). If false, construct an explicit counterexample with concrete
numbers. 17. (Generalized "Tuesday Boy") Suppose instead of "born on Tuesday" (1-in-7
identifying detail), the identifying detail has probability p of applying to any given child
(independent of gender), e.g. "at least one boy has [some trait with base rate p]." Derive a general
formula for P(both boys | at least one boy has the trait) as a function of p, and verify it gives 1/3 as
p→1 (a boy always has the trait — no distinguishing power) and 1/2 as p→0 (the trait is maximally
identifying). 18. In the "double-headed coin in a jar" setup of Q12, formally set this up as a
Bayesian updating problem with an explicit likelihood ratio per flip, and determine after how many
consecutive heads the posterior probability of "double-headed" first exceeds 0.9999, assuming a fair
coin has P(H)=0.5. 19. Explain, using the concept of a collider in a causal/Bayesian-network sense,
why conditioning on a common effect of two independent causes can make those causes appear
(spuriously) negatively correlated. Construct a concrete numerical joint distribution over 3 binary
variables demonstrating this "explaining away" effect precisely. 20. A rare disease has prevalence
that itself is uncertain (modeled as a Beta distribution over the true prevalence rate, rather than a
fixed known number). Given a positive test result on one patient, describe (conceptually, and if
you're comfortable, with the integral set up) how the posterior probability of disease should be
computed by integrating over the uncertain prevalence parameter, rather than treating it as a fixed
known constant. What does this reveal about a key hidden assumption in the "simple" version of the
disease-testing problem (2.2)?

Notes on how to use this guide

• Work Tier 1–2 problems algebraically first; Tier 3–4 problems reward drawing things out
(trees, Venn diagrams, or explicit enumeration) before reaching for formulas.
• If you want, I can walk through full worked solutions to any specific practice question, or
build a version of this focused specifically on how conditional probability underlies signal
processing / noise / detection theory (relevant if this connects to your circuits work — e.g.,
probability of bit errors given noise conditions in a communication channel).