EPS 120

Everything you always needed to know about probability and Bayesian inference

. . . but were afraid to ask.

Kirchner

1. Probability. Let lowercase letters such as a, b, etc. denote either statements (which might be true or false) or events (which might occur or not). We use P(a) to denote the probability that a will occur (if a is an event) or the probability that a is true (if a is a statement). The probability of any event (or statement) a lies between 0 and 1 if a is uncertain. P(a)=1 if a is certainly true, and P(a)=0 if it is certainly false. In symbolic notation,

P(a|b) =

P(a and b) P(b)

This can be rewritten in more intuitively obvious form as P(a and b) = P(a|b) P(b) In other words, the probability that both a and b are true is simply the probability that b is true, times the probability that if b is true, a is also true. The strongest possible conditioning occurs when b implies a, whence P(a|b)=1 and P(a and b)=P(b). Conversely, when a and b are independent, P(a|b)=P(a), whence P(a and b)=P(a)P(b), as in (2) above. 8. Because P(a and b) equals P(b and a), then from (7), P(a|b) P(b) = P(b|a) P(a) Note that although P(a and b) and P(a or b) both commute (that is, the order of a and b can be reversed without changing P), this is not true for P(a|b). That is, P(a|b) is not equal to P(b|a), except in the special case where P(a)=P(b). Here's a trivial example. If someone is named "Jean", then the probability is high that this person is female (except in France), but if someone is female, the probability is not high that she is named "Jean". In such a simple example, the fallacy is easy to see. But the fact remains that some of the most common, and insidious, errors in probabilistic reasoning consist of mistaking P(a|b) for P(b|a). A simple example: say a hypothesis h strongly suggests that an observable event e should occur (i.e. P(e|h) is high). It does not follow that observing the event e provides strong evidence for h, (i.e. P(h|e) need not be high) because e might be probable whether or not h is true. 9. If a1, a2, a3, ...an are mutually exclusive (that is, no two can be true simultaneously) and jointly exhaustive (that is, at least one must be true), then P(a 1) + P(a2 ) + P(a 3) +...+ P(an) = 1 (A set of statements that are mutually exclusive and jointly exhaustive is sometimes termed a "partition", meaning something that partitions the realm of the possible into discrete propositions a1, a2, etc.) 10. If a1, a2, a3, ...an are mutually exclusive and jointly exhaustive, then for any proposition b, P(b) = P(b and a 1) + P(b and a2 ) +...+ P(b and an) since if b occurs at all it must occur with one of the a's (since one of the a's must occur). This is sometimes termed the "theorem of total probability".

0 P(a) 1

2. If a and b are independent (that is, the truth or falsity of one does not affect the probability of the other), then P(a and b) = P(a) P(b) and P(a or b) = P(a) + P(b) - P(a) P(b)

3. If a and b are mutually exclusive (that is, if one is true then the other must be false), then P(a and b) = 0 and P(a or b) = P(a) + P(b)

4. If we use ~a to denote the negation of a (that is, ~a is true when a is false, and vice versa), then the probability of ~a is P(~a) = 1 P(a) (since a and ~a are mutually exclusive, and therefore P(a or ~a)=P(a)+P(~a)=1). 5. If a and b are equivalent (that is, if one is true if and only if the other is true, which we denote a<=>b), then they have the same probability: if a<=>b then P(a) = P(b) The converse is not true; statements with the same probability need not be equivalent. 6. If a implies b (denoted a=>b), then the probability of b is at least as great as the probability of a: if a=>b then P(b) P(a) (since b occurs every time a occurs, and b might also occur at some other times as well). When b occurs if and only if a occurs, P(b)=P(a) as in (5) above; otherwise P(b)>P(a). Note that the fact that a implies b does not exclude the possibility that other things may also imply b. 7. Conditional probability. If a depends on b (that is, the probability of a is affected by whether b is true or false), we use P(a|b) to indicate the probability of a, given b (that is, the probability of a when b is true). This is also referred to as the probability of a conditioned on b, and can be expressed as

11. In the same vein, if a1, a2, a3, ...an are mutually exclusive and jointly exhaustive, then for any proposition b, P(b) = P(b|a1 ) P(a1 ) + P(b|a2 ) P(a 2) +...+ P(b|an) P(an) This is the basis for "fault tree analysis" and other forms of risk analysis that quantify the likelihood of an outcome (e.g., a nuclear reactor meltdown) that can occur through several different causal pathways. 12. If an hypothesis h implies some observable evidence e, and if h is possible, and if e is not inevitable, (that is, h=>e, P(h)>0, and P(e)<1), then P(h|e), the probability that the hypothesis is true, given that the evidence is observed, is greater than P(h), the probability that h is true in the absence of any information about whether or not e is observed, P(h|e) > P(h) In other words, if h predicts e, then the occurence of e raises the likelihood that h is true. Although h and e could be any two statements, and need not be an hypothesis and evidence for it, they are used in this example because it is such an important part of intuitive scientific inference. Where does (12) come from? Directly from (8), as follows: if P(h|e)P(e)=P(e|h)P(h), and P(e|h)=1 (because h implies e), then P(h|e)=P(h)/P(e), which must be greater than P(h) because P(e)<1. 13. If an hypothesis h indicates that some observable evidence e is probable, with a probability of P(e|h), then the probability that the hypothesis is true if e is indeed observed, P(h|e), depends on P(e|h), P(h), and P(e), as follows:
P(h|e) = P(e|h) P(h) P(e)

This is simply a generalization of (13), derived by substituting (11) for P(e) in (13). The variables are likewise generalized from (13): P(hi) (or P(hk))is the likelihood that hi (or hk) is true, before one knows whether or not the observable evidence e is true, P(e|hi) (or P(e|hk)) is the probability that e will be observed if hi (or hk)is true, and P(hk|e) is the likelihood that hk is true, if e is indeed observed. This is known as the second form of Bayes's Theorem, but was actually proposed by Laplace. This shows that observing e can provide strong evidence for hk (that is, P(hk|e) can be large), but only if e is unlikely under any of the other hypotheses (that is, P(e|hi) is small for hihk). 15. Where just one hypothesis is being tested, (14) can be reduced to only two alternatives: the hypothesis h, and all other possibilities (which imply that h is false, indicated by ~h). Then (14) becomes (after several terms are rearranged) the third form of Bayes' Theorem: P(h|e) = P(h) + P(h) P(h) = P(e|~h) P(e|~h) P(~h) P(h) + [1P(h)] P(e|h) P(e|h)

where P(h) is the "prior" likelihood that h is true (that is, the likelihood in the absence of any evidence whether e is true or not), P(h|e) is the "posterior" likelihood that h is true (that is, after e has indeed been observed), and P(e|h) and P(e|~h) are the probabilities that e would be observed if the hypothesis were true and if it were false. The expression above shows that the posterior likelihood, P(h|e), depends on only two factors: the prior assessment of likelihood, P(h), and the likelihood ratio P(e|~h)/P(e|h). Note that if e is likely if h is true, but very unlikely if h is false, the posterior likelihood P(h|e) can approach 1 after e is observed, regardless of what the prior likelihood P(h) is. That means that the evidence e could even convince a skeptic (someone who believed P(h) was low) that h is indeed true, but it can only do so if e would be sufficiently unlikely if h were false. Conversely, if e is likely when h is false, but very unlikely when h is true, the posterior likelihood P(h|e) can approach 0 after e is observed, regardless of whether P(h) was assumed to be high or low. That means that the evidence e could even convince a fanatical believer (someone who believed P(h) was high) that h is indeed false, but it can only do so if e would be very unlikely if h were true.

where, as in (12), P(h) is the probability that h is true in the absence of any information about whether or not e is observed, and P(e) is the probability that e would be observed whether or not h was true. This is Bayes' Theorem, which forms the basis for Bayesian inference, in conjunction with the two variants given below. It can be derived directly by rearranging the terms in (8). Bayes' Theorem quantifies how strongly experimental observations confirm or undermine a hypothesis. If h greatly increases the probability of e (that is, P(e|h) is large compared to P(e)), then observing e greatly increases the likelihood that h is true (that is, P(h|e) will be much larger that P(h). Conversely, if h strongly suggests that e should not occur (that is, P(e|h) is small), then observing e provides strong disconfirmation for h (that is, P(h|e) will be small). 14. If several alternative hypotheses, h1, h2, ... hn are mutually exclusive and jointly exhaustive, then the probability that any one of them, say hk, is true is: P(h k|e) =

i= 1

P(e|hi) P(hi)

P(e|hk) P(h k)

Adapted in part from Chapter 2 of C. Howson and P. Urbach, Scientific Reasoning: The Bayesian Approach, Open Court Publishing, La Salle, Illinois, 1993.

BayesianinferenceI: JohnDoeandtheAIDStest
Assume:theHIVtesthasafalsepositiverateof5% andafalsenegativerateof0% Datapoint:JohnDoetestspositive Question:whatistheprobabilitythatJohnDoehasHIV? Wronganswer:95% Rightanswerviaclassicalstatisticalinference: Dontaskthatquestion!(Factsarenotprobabilistic!) RightanswerfromBayesianstatistics: ItdependsontherateofHIVinthegroupJohnDoecamefrom! Illustration: AssumeHIVrateinthisgroupis1%. Test10000individuals. Results: 100truepositives(1%of10000) 495falsepositives(5%of9900) ChanceJohnDoehasHIVis: =(100truepositives)/(100true+495falsepositives) =100/59517%(not95%!)

Illustration2: WhatiftheHIVrateinJohnDoesgroupwere0.1%? Test10000individuals. Results: 10truepositives(0.1%of10000) 499.5falsepositives(5%of9990) ChanceJohnDoehasHIVis: =(10truepositives)/(10true+499.5falsepositives) =10/509.52%(not95%!) Importantlessons: Testresultsalonearemeaningless. Everythingdependsonlikelihoodthatpositiveresults aretruepositivesratherthanfalsepositives. Thislikelihoodisunknowablewithoutknowingthe backgroundriskofHIVinthepopulation! Illustration3: RepeattheHIVtestonthe509.5positives (assumetheseresultsareindependentofpreviousones) Results: 10truepositives(allofthem) 24.975falsepositives(5%of499.5) ChanceJohnDoehasHIVis: =(10truepositives)/(10true+24.975falsepositives) =10/3530%(not95%!)

Illustration4: RepeattheHIVtestagainonthe35positives (assumetheseresultsareindependentofpreviousones) Results: 10truepositives(allofthem) 1.25falsepositives(5%of25) ChanceJohnDoehasHIVis: =(10truepositives)/(10true+1.25falsepositives) =10/11.2590%(not95%!) Illustration5: RepeattheHIVtestyetagainonthe11.25positives (assumetheseresultsareindependentofpreviousones) Results: 10truepositives(allofthem) 0.0625falsepositives(5%of1.25) ChanceJohnDoehasHIVis: =(10truepositives)/(10true+0.0625falsepositives) =10/10.062599.4%(not95%!) Importantlesson: Bygeneratingenoughinformationfromtesting,wecan reachaclearresult(almost)regardlessofthebackgroundriskin thepopulation.

BayesianinferenceII: TheReverendThomasBayes,FRS, andBayesTheorem

...whichisntatheorem,andwasntderivedbyBayes... [Link],AnEssayTowardsSolvingaProblemintheDoctrineof Chances,PhilosophicalTransactionsoftheRoyalSocietyofLondon, 53,370418,1763. (Posestheproblembutsolvesitonlyforonespecialcase). PierreSimon,MarquisDeLaplace,TheorieAnalytiquedes Probabilites,1812. (Statesageneralsolutionbutdoesntproveit). Problem:Givensomesetofobservations,whatistheprobability thatagivenmodelorhypothesisiscorrect? Define: H ahypothesis d someobservabledata P(d|H) probabilityofobservingdifHistrue P(d|~H) probabilityofobservingdifHisfalse P(H|d) confidencethatHistrueifdisobserved note! P(d|H)P(H|d)!

Question: WhatistheprobabilitythatHistrue,anddisobserved? note P(Handd)=P(d|H)P(H) and P(Handd)=P(H|d)P(d) so P(d|H)P(H)=P(H|d)P(d) SO! P(d | H ) P( H ) P( H | d) = P(d) P(d | H ) P( H ) = P(d | H ) P( H ) + P(d |~ H ) P(~ H )
= chanceofdoccurringbecauseH istrue chanceofdoccurringwhetherornotH istrue

FormultiplehypothesesH1,H2,H3...: P(d | H i ) P( H i ) P( H i | d) = P(d) P(d | H i ) P( H i ) = P(d | H k ) P( H k )

k

chanceofdoccurringbecauseH i istrue chanceofdoccurringforanyreason

ThisisBayesTheorem,ormorehumblyanupdatingrule whichusesdatadtoupdateourpriorconfidenceP(H),yielding ourposteriorconfidenceP(d|H).

P( H | d) =

BayesianinferenceIII:Howsciencereallyworks
Twopossibilities: H sometheoryistrue ~H thattheoryisfalse Twooutcomesofatestofatheory: d dataareconsistentwiththetheory ~d dataareinconsistentwiththetheory Classicalstatisticscantellus: P(~d|~H)=1=power (likelihoodofrejectingatheorythatisfalse) P(d|~H)==falsenegativerate(TypeIIerrorrate) (riskofacceptingatheorythatisactuallyfalse) P(~d|H)==falsepositiverate(TypeIerrorrate) (riskofrejectingatheorythatsactuallytrue) P(d|H)=1=statisticalsignificance (likelihoodthattheorywillbeacceptedifitstrue) (ThesemanticsaboveareconventionalifHisthenullhypothesis) Butwhatwereallywanttoknowis: P(H|d) (Ifthedatasupportthetheory,whatarethe chancesthatthetheoryisactuallytrue?)

P(d | H ) P( H ) P(d) P(d | H ) P( H ) = P(d | H ) P( H ) + P(d |~ H ) P(~ H ) = chanceofsupportiveevidencebecausetheoryistrue chanceofsupportiveevidencewhetherornottheoryistrue

Soposttestconfidenceinthetheorycanbehigh,butonlyifthere wasalowriskthatthetheorycouldspuriouslypassthetest Inthelanguageofpowerandsignificance: (1 ) P( H ) P( H | d) = (1 ) P( H ) + (1 P( H )) 1 = 1 + P( H ) 1 = 1 1+ P( H ) + 1 1 Soifpowerishighenough(islowenough),thetestshould convinceanondogmaticskepticthatthetheoryisvalid (nondogmaticmeansP(H)>0)

P( H |~ d) =

P(~ d | H ) P( H ) P(~ d) P(~ d | H ) P( H ) = P(~ d | H ) P( H ) + P(~ d |~ H ) P(~ H ) =

10

BayesianinferenceIV:Hotspotdetection
Twopossibilities: H ahotspotexists ~H hotspotdoesnotexist Twooutcomesofanattempttodetectthehotspot: d hotspotdetected ~d hotspotnotdetected Classicalstatisticscantellus: P(d|H)=1=power (probabilitythathotspotwillbedetectedifitsthere) P(~d|H)==falsenegativerate(TypeIIerrorrate) (probabilityofmissinghotspoteventhoughitsthere) P(d|~H)==falsepositiverate(TypeIerrorrate)(=0?) (probabilityoffalselydetectingahotspotthatsnotthere) P(~d|~H)=1=statisticalsignificance(=1) (likelihoodthatiftheresnohotspot,nonewillbedetected) Butwhatwereallywanttoknowis: P(H|~d) (i.e.,theriskthattheresahotspotthere, eventhoughwevefailedtodetectit)

chanceofnondetectioneventhoughhotspotisthere chanceofnondetectionwhetherornothotspotisthere

AssumingP(~d|~H)=1(wewontfalselydetectanonexistent hotspot), P(~ d | H ) P( H ) P( H |~ d) = P(~ d | H ) P( H ) + P(~ H ) P( H ) = P( H ) + 1 P( H ) Ifissmall(highpowertodetecthotspotsiftheyexist) P( H ) P( H |~ d) 1 P( H ) i.e.,ifhotspotdetectionisreliableenough,failuretodetecta hotspotshouldbeconvincingtoanondogmaticskeptic thatis,ifissmallenough,posttestconfidencethatahotspot existsP(H|~d)canbesmall ...eventhoughpretestconfidenceP(H)washigh ...aslongasP(H)<1(nondogmaticprior)