6/12/2015 BayesianStatistics
BayesianStatistics
Intheclassical,orfrequentistapproachtoStatisticsweconsideraparameterafixedalthough
unknownquantity.ArandomsampleX1,..,Xnisdrawnfromapopulationindexedbyand,
basedontheobservedvaluesinthesample,[Link]
theBayesianapproachisconsideredaquantitywhosevariationcanbedescribedbya
probabilitydistribution(calledapriordistribution),whichisasubjectivedistribution
[Link]
thentakenfromapopulationindexedbyandthepriordistributionisupdatedwiththisnew
[Link]
usingBayes'formula,hencethenameBayesianStatistics.
Let'sdenotethepriordistributionby()andthesamplingdistributionbyf(x|),thenthejoint
pdf(pmf)ofXandisgivenby
f(x,)=f(x|)()
themarginalofthedistributionofXis
m(x)=f(x|)()d
andfinallytheposteriordistributionistheconditionaldistributionofgiventhesamplexand
isgivenby
(|x)=f(x|)()/m(x)
Wecanwritethisalsointermsofthethelikelihoodfunction:
(|x1,..,xn)=L(|x1,..,xn)()/m(x)
Example:Youwanttoseewhetheritisreallytruethatcoinscomeupheadsandtailswith
probability1/2.Youtakeacoinfromyourpocketandflipit10times.Itcomesupheads3
[Link]
probabilityofheads,pandfind =0.3.
ABayesiananalysiswouldproceedasfollows:letX1,..,XnbeiidBer(p).ThenY=X1+..+Xn
isBin(n,p).[Link],soithasvalueson[0,1].
Onedistributionon[0,1]weknowistheBeta,sowewilluseaBeta(,)asourprior.
Remember,thisisaperfectlysubjectivechoice,[Link]
distributiononYandpisgivenby
[Link] 1/6
6/12/2015 BayesianStatistics
whichisknownasthebetabinomialdistribution.
Notethatthat(Y,p)isarandomvectorwhereonecomponentiscontinuous(p)andtheotheris
discrete(Y).[Link].
Theposteriordestributionofpgivenyisthen
Ofcoursewestillneedto"extract"someinformationabouttheparameterpfromtheposterior
[Link],theposteriordistribution
isfixed(eventhoughitmaynotbeeasyorevenpossibletofinditanalytically)buthowwe
proceednowiscomp[[Link]
panaturalestimatoristhemeanoftheposteriordistribution,givenhereby
B=(y+)/(++n)
Thiscanbewrittenas
andweseethattheposteriormeanisalinearcombinationofthepriormeanandthesample
[Link] 2/6
6/12/2015 BayesianStatistics
mean.
Howaboutourproblemwiththe3headsinthe10flips?Well,wehavetocompletelyspecify
thepriordistribution,[Link].
Forexample,ifwefeelstronglythatthiscoinisjustlikeanyothercoinandthereforereally
shouldbeafaircoinweshouldchoosethemsothatthepriorputsalmostallitsweightat
around1/[Link]==100wegetE[p]=0.5andV[p]=[Link] B=
(3+100)/(100+100+10)=[Link]
strongpriortheactualsamplealmostdoesnotmatter,Forexamplefory=0wewouldhave
found B=0.476andfory=10itwouldbe B=0.524.
Maybewehaveneverevenheardtheword"coin"andhavenoideawhatonelookslike,let
alonewhatprobabilityof"heads"[Link]==1,thatistheuniform
distribution,[Link]
p.(Thisiscalledanuninformativeprior).Nowwefind B=(3+1)/(1+1+10)=0.3,whichis
justthesamplemeanagain.
inbayescoinwithwhich==1westudytheeffectofthesamplesizeontheestimateofp.
inbayescoinwithwhich==2westudytheeffectofalpha=[Link]
alphameansapriormoreconcentratedaround1/2.
ExamplesayX~Bin(n,p),[Link],[Link]
n=1,2,..andsoapriorisanysequencea1,a2,..withai0andai=1
Ifwewanttofindanestimatefornwecanuseforexamplethemode,thatisthenwhichhas
thelargestposteriorprobability.
Herearesomespecificexamples:sayweobservex=217andweknowp=[Link]
a)weknowonlythatn750,sowechooseai=1/750if1i750,0otherwise,
[Link] 3/6
6/12/2015 BayesianStatistics
[Link].n(217,0.37,rep(1/750,750))
b)weknownismostlikely500withastandarddeviationof50,
[Link].n(217,0.37,dnorm(1:750,500,50))
c)weknowthatn750andthatnisamultipleof50,
[Link].n(217,0.37,ifelse(c(1:750)%%50,0,1))
d)weknowthiswasoneofthefourexperimentswedid,withn=510,525,550or575
a=rep(0,750)
a[c(510,525,550,575)]=1
[Link].n(217,0.37,a)
TheBigQuestion:BayesianorFrequentist?
ShouldyoubeaBayesian?
[Link],itanswerstherightquestion,
P(Hypothesis|Data).Thereareothersaswell:
DecisionTheory
[Link]
usefulandimportantone:
Shouldyoubuyanewcar,orkeeptheoldoneforanotheryear?
Shouldyouinvestyourmoneyintothestockmarketorbuyfixedinterestbonds?
Shouldthegovermentlowerthetaxesorinsteadusethetaxesfordirectinvestments?
Indecisiontheoryonestartsoutbychoosingalossfunction,thatisafunctionthatassignsa
value(maybeintermsofmoney)toeverypossibleactionandeverypossibleoutcome.
ExampleYouareofferedthefollowinggame:youcaneithertake$10(let'scallthisactiona),
oryoucanflipacoin(actionb).Ifthecoincomesupheadsyouwin$50,ifitcomesupheads
youloose$[Link]:takethe$10orflipthecoin,andthree
possibleoutcomes,youwin$10,$50orloose$[Link]
obviousansweristhisone:
L(a)=10,L(b,"heads")=50,L(b,"tails")=10
[Link]
tabis$[Link]$8inyourpocket(andnocreditcardetc.)Now
ifyouwinorloose$10itdoesn'tmatther,eitherwayyoucan'tpayyourbill,anditwillbevery
[Link]$50,[Link]
[Link] 4/6
6/12/2015 BayesianStatistics
functionmightbe:
L(a)=0,L(b,"heads")=1000,L(b,"tails")=0
[Link]:let'scarryoutan
experiment,anddependingontheoutcomeoftheexperimentwechoseanaction.
Shouldyouinvestyourmoneyintothestockmarketorbuyfixedinterestbonds?
Let'sdothis:[Link],weinvestinthestock
market,otherwiseinbonds.
Indecisiontheoryadecisionruleiscalledinadmissibleifthereisanotherrulethatisbetterno
[Link]
inadmissiblerule.
Sowhat'stheconnectiontoBayesianStatistics?FirstthereareBayesiandecisionrules,which
combinepriorknowledgewiththeoutcomeoftheexperiment.
basedonthemovementoftheDowJonesinthelastyear,Ihaveacertainprobabilitythatit
willgoupoverthenextyear.
Nowthereisafamoustheorem(thecompleteclasstheorem)thatsaysthatalladmissiblerules
areBayesiandecisionrulesforsomeprior.
Optimality
Obviouslywhenwedosomethingitwouldbenicetodoitinanoptimal(best)[Link]
thatinBayesianstatisticsitisoftenpossibletoshowthatacertainmethodisbest,betterorat
leastasgoodasanyother.
WhytobeaFrequentist
Becauseyouhatepriors
orbettertosayyoudon'[Link]
entirelypossiblethattwoScientistswhohavethesamedataavailableandusethesamemethod
foranalysiscometodifferentconclusions,becausetheyhavedifferentpriors.
BecauseFrequentistmethodswork
FormostofthehistoryofStatistics,thatisfromabout1900toabout1960,therewas
(essentially)[Link](andsince)manymethodshavebeen
[Link]
methodswhentherightpriorisused,butnotall!
[Link] 5/6
6/12/2015 BayesianStatistics
Exampleoneofthemostusefulmodernmethods,calledtheBootstrap,isapurelyFrequentist
methodwithnoBayesiantheory.(ActuallythereissomethingcalledtheBayesianbootstrap,
butitisnotthesameastheclassicalbootstrap)
[Link],though,violatesthe
[Link],most
Bayesiansstudytheresidualsanyway.
Simplicity
Evenfortheeasiestproblems("estimatethemeanGPAofstudentsattheColegio")aBayesian
analysisalwaysseemstobecomplicated(chooseapriorandalossfunction,calculatethe
posterior,extracttheestimatefromtheposterior,trytodoallofthisoptimally)Frequentist
solutionsareoftenquickandeasy.
So?BeBoth!
[Link] 6/6
