Conditional Probability and Independence
Guy Lebanon January 6, 2006
In the following we assume that A, B are events in the sample space and P is a probability on . Denition 1. Two events A, B are dened to be independent if P (A B) = P (A)P (B). The intuitive meaning here is that the fact whether the event A occurred or not does not alter the likelihood of the event B occurring (and vice verse). Recall that if two events are disjoint A B = , they cant occur simultaneously. Therefore if two events are disjoint (or mutually exclusive) the occurrence of one precludes the occurrence of the other. Hence two events with positive probabilities cant be both disjoint and independent! Denition 2. The conditional probability of A given B is dened as (assuming that P (B) > 0), P (A|B) = P (A B) . P (B)
Note that in this case P (A B) = P (A|B)P (B) = P (B|A)P (A). The intuitive meaning here is the probability of A occurring assuming that the event B occurred. Note that if A, B are independent, P (A|B) = P (A) as it should if you put the intuitive meaning of independence and conditional probability together. Theorem 1. If A, B are independent, then so are Ac , B and so are A, B c and so are Ac , B c . Proof. For example, P (Ac B) = P (B \ A) = P (B) P (A B) = P (B) P (A)P (B) = (1 P (A))P (B) = P (Ac )P (B)
Theorem 2 (Bayes). P (A|B) = Proof. P (A|B)P (B) = P (A B) = P (B A) = P (B|A)P (A) P (B|A)P (A) . P (B)
Example: Consider a voting patterns of a group of 100 Americans. They are classied in the following contingency table according to their party and whether they live in a city or a small town. City 30 20 50 Small Town 15 35 50
Democrats Republicans
45 55 100
We dene A as the event that a person selected at random lives in the city, and B as the event that a person selected at random is a democrat. We then have the following quantities (under the classical interpretation) that are called the joint probabilities P (A B) = 0.3, P (Ac B) = 0.15, P (A B c ) = 0.2, P (Ac B c ) = 0.35. The following probabilities are called the marginal probabilities: P (A) = 0.5, P (B) = 0.45. We also have the conditional probabilities P (A|B) = 2/3, P (A|B c ) = 4/11, P (B|A) = 0.6, P (B|Ac ) = 0.3. Clearly A, B are not independent (since P (A|B) = P (A)). The probability that a randomly drawn person is a democrat is 0.45. If we know that we only consider people that live in the country, we condition on Ac and the probability decreases to 0.3. Using conditional probability we can use additional measurements to get a more accurate estimation of the likelihood of events. Theorem 3 (General Multiplication Rule). Generalizing the result P (A B) = P (A|B)P (B) we have that (by induction) P (A1 An ) = P (An |A1 An1 )P (A1 An1 ) = = P (A1 )P (A2 |A1 )P (A3 |A2 A1 ) P (An |A1 An1 ). Theorem 4 (The Law of Total Probability). Suppose that {A1 , . . . , An } are sets such that A1 An = , Ai Aj = for all i = j (such sets are called a partition of ). Then for every event B we have
n
P (B) =
i=1
P (Ai )P (B|Ai ).
Proof. By the third axiom of probability,
n n
P (B) = P ((B A1 ) (B An )) =
i=1
P (Ai B) =
i=1
P (Ai )P (B|Ai ).
Denition 3. Let {Ai }i be a nite or innite sequence of events. If every pair Ai , Aj , i = j is independent, we say that the events {Ai }i are pairwise independent. If for every nite subset of events Ai1 , . . . , Aik , we have P (Ai1 . . . Aik ) = P (Ai1 ) P (Aik ) we say that the events are independent. Pairwise independence is a strictly weaker condition that independence.