An Introdu tion to Probabilisti Graphi al Models

Mi hael I. Jordan

University of California, Berkeley

June 30, 2003

2
Chapter 5

Statisti al Con epts

It is useful to attempt to distinguish the a tivities of the probability theorist and the statisti ian.
Our perspe tive in the previous hapters has been mainly that of the former|we have built graphi-
al models involving sets of random variables and shown how to ompute the probabilities of ertain
events asso iated with these random variables. Given a parti ular hoi e of graphi al model, on-
sisting of a graph and a set of lo al onditional probabilities or potentials, we have seen how to
infer the probabilities of various events of interest, su h as the marginal or onditional probability
that a parti ular random variable takes on a parti ular value.
Statisti s is in a ertain sense the inverse of probability theory. In a statisti al setting the
random variables in our domain have been observed and are therefore no longer unknown, rather
it is the model that is unknown. We wish to infer the model from the data rather than the data
from the model.
The problem of \inferring the model from the data" is a deep one, raising fundamental questions
regarding the nature of knowledge, reasoning, learning, and s ienti inquiry. In statisti s, the
study of these fundamental questions has often ome down to a distin tion between two major
s hools of thought|the Bayesian and the frequentist. In the following se tion we brie y outline
the key distin tions between these two s hools. It is worth noting that our dis ussion here will be
in omplete and that we will be returning to these distin tions at various jun tures in the book as
our development of graphi al models begins to bring the distin tions into learer relief. But an
equally important point to make is that many of the problems|parti ularly the omputational
problems|fa ed in these frameworks are losely related, even identi al. A great deal of important
work an be done within the graphi al models formalism that is equally useful to Bayesian and
frequentist statisti s.
Beyond our dis ussion of foundational issues, we will also introdu e several lasses of statisti al
problems in this hapter, in parti ular the ore problems of density estimation, regression and
lassi ation. As in earlier hapters our goal is to present enough in the way of on rete details to
make the dis ussion understandable, but to emphasize broad themes that will serve as landmarks
for our more detailed presentation in later hapters.
3
4 CHAPTER 5. STATISTICAL CONCEPTS

5.1 Bayesian and frequentist statisti s

Bayesian statisti s is in essen e an attempt to deny any fundamental distin tion between probability
theory and statisti s. Probability theory itself provides the apability for inverting relationships
between un ertain quantities|this is the essen e of Bayes rule |and Bayesian statisti s represents
an attempt to treat all statisti al inferen e as probabilisti inferen e.
Let us onsider a problem in whi h we have already de ided upon the model stru ture for
a given problem domain|for example, we have hosen a parti ular graphi al model in luding a
parti ular pattern of onne tivity|but we have not yet hosen the values of the model parameters |
the numeri al values of the lo al onditional probabilities or potentials. We wish to hoose these
parameter values on the basis of observed data. (In general we might also want to hoose the model
stru ture on the basis of observed data, but let us postpone that problem|see Se tion 5.3).
For every hoi e of parameter values we obtain a di erent numeri al spe i ation for the joint
distribution of the random variables X . We will hen eforth write this probability distribution
as p(x j ) to re e t this dependen e. Putting on our hats as probability theorists, we view the
model p(x j ) as a onditional probability distribution; intuitively it is an assignment of probability
mass to unknown values of X , given a xed value of . Thus,  is known and X is unknown. As
statisti ians, however, we view X as known|we have observed its realization x|and  as unknown.
We thus in some sense need to invert the relationship between x and . The Bayesian point of view
implements this notion of \inversion" using Bayes rule:
p(x j )p()
p( j x) = : (5.1)
p(x)
The assumptions allowing us to write this equation are noteworthy. First, in order to interpret
the left-hand side of the equation we must view  as a random variable. This is hara teristi of
the Bayesian approa h|all unknown quantities are treated as random variables. Se ond, we view
the data x as a quantity to be onditioned on|our inferen e is onditional on the event fX = xg.
Third, in order to al ulate p( j x) we see (from the right-hand side of Eq. (5.1)) that we must have
in hand the probability distribution p()|the prior probability of the parameters. Given that we
are viewing  as a random variable, it is formally reasonable to assign a (marginal) probability to
it, but one needs to think about what su h a prior probability means in terms of the problem we
are studying. Finally, note that Bayes rule yields a distribution over |the posterior probability
of  given x, not a single estimate of . If we wish to obtain a single value, we must (and will)
invoke additional prin iples, but it is worth noting at the outset that the Bayesian approa h tends
to resist ollapsing distributions to points.
The frequentist approa h wishes to avoid the use of prior probabilities in statisti s, and thus
avoids the use of Bayes rule for the purpose of assigning probabilities to parameters. The goal of
frequentist methodology is to develop an \obje tive" statisti al theory, in whi h two statisti ians
employing the methodology must ne essarily draw the same on lusions from a parti ular set of
data.
Consider in parti ular a oin-tossing experiment, where X 2 f0; 1g is a binary variable represent-
ing the out ome of the oin toss, and  2 (0; 1) is a real-valued parameter denoting the probability
of heads. Thus the model is the Bernoulli distribution, p(x j ) = x(1 )1 x. Approa hing the
5.1. BAYESIAN AND FREQUENTIST STATISTICS 5

problem from a Bayesian perspe tive requires us to assign a prior probability to  before observing
the out ome of the oin toss. Two di erent Bayesian statisti ians may assign di erent priors to
 and thus obtain di erent on lusions from the experiment. The frequentist statisti ian wishes
to avoid su h \subje tivity." From another point of view, a frequentist may laim that  is a
xed property of the oin, and that it makes no sense to assign probability to it. A Bayesian may
agree with the former statement, but would argue that p() need not represent anything about
the physi s of the situation, but rather represents the statisti ian's un ertainty about the value of
. Tossing the oin redu es the statisti ian's un ertainty, and hanges the prior probability into
the posterior probability p( j x). Bayesian statisti s views the posterior probability and the prior
probability alike as (possibly) subje tive.
There are situations in whi h frequentist statisti s and Bayesian statisti s agree that parameters
an be endowed with probability distributions. Suppose that we onsider a fa tory that makes
oins in bat hes, where ea h bat h is hara terized by a smelting pro ess that a e ts the fairness
of the resulting oins. A oin from a given bat h has a di erent probability of heads than a oin
from a di erent bat h, and ranging over bat hes we obtain a distribution on the probability of
heads . A frequentist is in general happy to assign prior probabilities to parameters, as long as
those probabilities refer to obje tive frequen ies of observing values of the parameters in repeated
experiments.
From the point of view of frequentist statisti s, there is no single preferred methodology for
inverting the relationship between parameters and data. Rather, the basi idea is to onsider
various estimators of , where an estimator is some fun tion of the observed data x (we will dis uss
a parti ular example below). One establishes various general riteria for evaluating the quality of
various estimators, and hooses the estimator that is \best" a ording to these riteria. (Examples
of su h riteria in lude the bias and varian e of estimators; these riteria will be dis ussed in
Chapter 26). An important feature of this evaluation pro ess is that it generally requires that
the data x be viewed as the result of a random experiment that an be repeated and in whi h
other possible values of x ould have been obtained. This is of ourse onsistent with the general
frequentist philosophy, in whi h probabilities orrespond to obje tive frequen ies.
There is one parti ular estimator that is widely used in frequentist statisti s, namely the maxi-
mum likelihood estimator. This estimator is popular for a number of reasons, in parti ular be ause
it often yields \natural estimators" (e.g., sample proportions and sample means) in simple settings
and also be ause of its favorable asymptoti properties.
To understand the maximum likelihood estimator, we must understand the notion of \likeli-
hood" from whi h it derives. Re all that the probability model p(x j ) has the intuitive inter-
pretation of assigning probability to X for ea h xed value of . In the Bayesian approa h this
intuition is formalized by treating p(x j ) as a onditional probability distribution. In the frequen-
tist approa h, however, su h a formal interpretation is suspe t, be ause it suggests that  is a
random variable that an be onditioned on. The frequentist instead treats the model p(x j ) as a
family of probability distributions indexed by , with no impli ation that we are onditioning on
.1 Moreover, to implement a notion of \inversion" between x and , we simply hange our point
1
To a knowledge this interpretation, frequentist treatments often adopt the notation p (x) in pla e of p(x j ).
We will sti k with p(x j ), hoping that the frequentist-minded reader will forgive us this abuse of notation. It will
6 CHAPTER 5. STATISTICAL CONCEPTS

Figure 5.1: A univariate density estimation problem. (See Se tion 5.2.1 for a dis ussion of density
estimation). The data fx1 ; x2 ; : : : ; xN g are given as X's along the abs issa. The parameter ve tor
 is the mean  and varian e 2 of a Gaussian density. Two andidate densities, involving di erent
values of , are shown in the gure. Density A assigns higher probability to the observed data than
density B , and thus would be preferred a ording to the prin iple of maximum likelihood.

of view|we treat p(x j ) as a fun tion of  for xed x. When interpreted in this way, p(x j ) is
referred to as the likelihood fun tion and it provides the basis for maximum likelihood estimation.
As suggested in Figure 5.1, the likelihood fun tion an be used to evaluate parti ular hoi es
of . In parti ular, if for a given value of  we nd that the observed value of x is assigned low
probability, then this is perhaps a poor hoi e of . A value of  that assigns higher probability to
x is preferred. Ranging over all possible hoi es of , we pi k that value of  that assigns maximal
probability to x, and treat this value as an estimate of the true :
^ML = argmax p(x j ): (5.2)
Thus the maximum likelihood estimate is that value of  that maximizes the likelihood fun tion.
Regardless of whether one agrees that this justi ation of the maximum likelihood estimate is
a natural one, it is ertainly true that we have an estimator|a fun tion of x|and we an evaluate
the properties of this estimator under various frequentist riteria. It turns out that maximum
likelihood is a good estimator under a variety of measures of quality, parti ularly in settings of large
sample sizes when asymptoti analyses are meaningful (indeed, maximum likelihood estimates an
be shown to be \optimal" in su h settings). In other settings, parti ularly in ases of small sample
sizes, maximum likelihood plays an important role as the starting point for the development of
more omplex estimators.
simplify our presentation throughout the rest of the book, liberating us from having to make distin tions between
Bayesian and frequentist interpretations where none are needed or implied.
5.1. BAYESIAN AND FREQUENTIST STATISTICS 7

Another appealing feature of likelihood-based estimation is that it provides a link between

Bayesian methods and frequentist methods. In parti ular, note that the distribution p(x j ) ap-
pears in our basi Bayesian equation Eq. (5.1). Note moreover that Bayesian statisti ians refer to
this probability as a \likelihood" as do frequentist statisti ians, even though the interpretation is
di erent. Symboli ally, we an interpret Eq. (5.1) as follows:
posterior / likelihood  prior; (5.3)
where we see that in the Bayesian approa h the likelihood an be viewed as a data-dependent
operator that transforms between the prior probability and the posterior probability. At a bare
minimum, Bayesian approa hes and likelihood-based frequentist approa hes have in ommon the
need to al ulate the likelihood for various values of . This is not a trivial fa t|indeed a major
fo us of this book is the set of omplex statisti al models in whi h the omputation of the likelihood
is itself a daunting omputational task. In working out e e tive omputational pro edures to deal
with su h models we are ontributing to both Bayesian and frequentist statisti s.
Let us explore this onne tion between Bayesian and frequentist approa hes a bit further. Sup-
pose in parti ular that we for e the Bayesian to hoose a parti ular value of ; that is, to ollapse
the posterior distribution p( j x) to a point estimate. Various possibilities present themselves; in
parti ular one ould hoose the mean of the posterior distribution or perhaps the mode. The mean
of the posterior is often referred to as a Bayes estimate :
Z
^Bayes =  p( j x)d; (5.4)

and it is possible and worthwhile to study the frequentist properties of Bayes estimates. The mode
of the posterior is often referred to as the maximum a posteriori (MAP) estimate:
^MAP = argmax p( j x) (5.5)
= argmax p(x j )p(); (5.6)
where in the se ond equation we have utilized the fa t that the fa tor p(x) in the denominator of
Bayes rule is independent of . In a setting in whi h the prior probability is taken to be uniform on
, the MAP estimate redu es to the maximum likelihood estimate. When the prior is not taken to
be uniform, one an still view Eq. (5.6) as the maximization of a penalized likelihood. To see this,
note that one generally works with logarithms when maximizing over probability distributions (the
fa t that the logarithm is a monotoni fun tion implies that it does not alter the optimizing value).
Thus one has:
^MAP = argmax flog p(x j ) + log p()g ; (5.7)
as an alternative expression for the MAP estimate. Here the \penalty" is the additive term log p().
Penalized log likelihoods are widely used in frequentist statisti s to improve on maximum likelihood
estimates in small sample settings (as we will see in Chapter 26).
It is important to emphasize, however, that MAP estimation involves a rather un-Bayesian use of
the Bayesian formalism, and it would be wrong to understand the distin tion between Bayesian and
frequentist statisti s as merely a matter of how to interpret a penalized log likelihood. To larify,
8 CHAPTER 5. STATISTICAL CONCEPTS

X Xnew

Figure 5.2: A graphi al representation of the problem of predi tion from a Bayesian point of view.

let us onsider a somewhat broader problem in whi h the di eren e between MAP estimation and
a fuller Bayesian approa h is more salient. Let us onsider the problem of predi tion, where we are
not interested in the value of  per se, but are interested in using a model based on  to predi t
future values of the random variable X . Let us suppose in parti ular that we have two random
variables, X and Xnew , whi h are hara terized by the same distribution, and that we wish to use
an observation of X to make a predi tion regarding likely values of Xnew . For simpli ity, let us
assume that X and Xnew are independent; more pre isely, we assume that they are onditionally
independent given . We write:
Z
p(xnew j x) = p(xnew ;  j x)d (5.8)
Z
= p(xnew j ; x)p( j x)d (5.9)
Z
= p(xnew j )p( j x)d: (5.10)

From the latter equation we see that the Bayesian predi tion is based on ombining the predi tions
a ross all values of , with the posterior distribution serving as a \weighting fun tion." That is,
interpreting the onditional probability p(xnew j ) as the predi tion of Xnew given , we weight this
predi tion by the posterior probability p( j x), and integrate over all su h weighted predi tions.
Note in parti ular that this al ulation requires the entire posterior probability, not merely its value
at a single point.
Within a frequentist approa h, we are not allowed to treat  as a random variable, and thus
we do not attribute meaning to the integral in Eq. (5.10). Rather, we would onsider various
\estimates" of xnew ; a natural hoi e might be the \plug-in estimate" p(xnew j ^ML ). Here we see
that the di eren e between the frequentist approa h and the Bayesian approa h has be ome more
signi ant; in the latter ase we have to perform an integral in order to obtain a predi tion. We
an relate the two approa hes if we approximate the posterior distribution by ollapsing it to a
delta fun tion at ^MAP , in whi h ase the integral in Eq. (5.10) redu es to the plug-in estimate
5.2. STATISTICAL PROBLEMS 9

p(xnew j ^MAP ). But in general this ollapse would not satisfy the Bayesian (who views the integral
as providing a better predi tor than any predi tor based on a point estimate) nor the frequentist
(who wants to be free to onsider a wider lass of estimates than the plug-in estimate).
As a nal note, onsider the graphi al model shown in Figure 5.2. This model aptures the
Bayesian point of view on the predi tion problem that we have just dis ussed. The parameter
 is depi ted as a node in the model; this is of ourse onsistent with the Bayesian approa h of
treating parameters as random variables. Moreover, the onditional independen e of X and Xnew
given  is re e ted as a Markov property in the graph. Finally, as we invite the reader to verify
in Exer ise ??, applying the elimination algorithm to the graph yields exa tly the al ulation in
Eq. (5.10). This is a re e tion of a general fa t|graphi al models provide a ni e way to visualize
and organize Bayesian al ulations. We will return to this point in later hapters. But let us
emphasize here that this linkage, appealing as it is, does not re e t any spe ial aÆnity between
graphi al models and Bayesian methods, but rather is a re e tion of the more general link between
Bayesian methods and probabilisti inferen e.

5.2 Statisti al problems

Let us now des end from the somewhat ethereal onsiderations of statisti al foundations to a rather
more on rete onsideration of problems in statisti al estimation. In this se tion we will dis uss
three major lasses of statisti al problems|density estimation, regression, and lassi ation. Not
all statisti al problems fall into one of these three lasses, nor is it always possible to unambiguously
hara terize a given problem in terms of these lasses, but there are ertain ore aspe ts of these
three problem ategories that are worth isolating and studying in a puri ed form.
We have two main goals in this se tion. The rst is to introdu e the graphi al approa h to
representing statisti al modeling problems, in parti ular emphasizing how the graphi al represen-
tation helps makes modeling assumptions expli it. Se ond, we wish to begin to work with spe i
probability distributions, in parti ular the Gaussian and multinomial distributions. We will use this
introdu tory se tion to illustrate some of the al ulations that arise when using these distributions.

5.2.1 Density estimation

Suppose that we have in hand a set of observations on a random variable X |in general a ve tor-
valued random variable|and we wish to use these observations to indu e a probability density
(probability mass fun tion for dis rete variables) for X . This problem|whi h we refer to generi ally
as the problem of density estimation|is a very general statisti al problem. Obtaining a model
of the density of X allows us to assess whether a parti ular observation of X is \typi al," an
assessment that is required in many pra ti al problems in luding fault dete tion, outlier dete tion
and lustering. Density estimation also underlies many dimensionality redu tion algorithms, where
a joint density is proje ted onto a subspa e or manifold, hopefully redu ing the dimensionality of a
data set while retaining its salient features. A related appli ation is ompression, where Shannon's
fundamental relationship between ode length and the negative logarithm of the density an be used
to design a sour e ode. Finally, noting that a joint density on X an be used to infer onditional
10 CHAPTER 5. STATISTICAL CONCEPTS

densities among omponents of X , we an also use density estimates to solve problems in predi tion.
To delimit the s ope of the problem somewhat, note that in regression and lassi ation the fo us
is on the relationship between a pair of variables, X and Y . That is, regression and lassi ation
problems di er from density estimation in that their fo us is on a onditional density, p(y j x), with
the marginal p(x) and the orresponding joint density of less interest, and perhaps not modeled at
all. We develop methods that are spe i to onditional densities in Se tions 5.2.2 and 5.2.3.
Density estimation arises in many ways in the setting of graphi al models. In parti ular we
may be interested in inferring the density of a parentless node in a dire ted graphi al model, or
the density of a set of nodes in a larger model (in whi h ase the density of interest is a marginal
density), or the joint density of all of the nodes of our model.
Let us begin with an example. Our example will be one of the most lassi al of all statisti al
problems|that of estimating the mean and varian e of a univariate Gaussian distribution.

Univariate Gaussian density estimation

Let us assume that X is a univariate random variable with a Gaussian distribution, that is:
 
1 1
p(x j ) = exp (x )2 ; (5.11)
(22 )1=2 22
where  and 2 are the mean and varian e, respe tively, and  , (; 2 ).2 We wish to estimate 
based on observations of X . Here we are assuming that we know the parametri form of the density
of X , and what is unknown are the numeri al values of the parameters ( f. Figure 5.1). Plugging
estimates of the parameters ba k into Eq. (5.11) provides an estimate of the density fun tion.
Clearly a single observation of X provides no information about the varian e and relatively poor
information about the mean. Thus we need to onsider multiple observations. What do we mean
by \multiple observations"? Let us interpret this to mean that we have a set of random variables,
fX1 ; X2 ; : : : ; XN g, and that these random variables are identi ally distributed. Thus ea h of the
variables Xn is hara terized by a Gaussian distribution p(xn j ), with the same  for ea h Xn .
In graphi al model terms, we have a model with N nodes, one for ea h random variable. Whi h
graphi al model should we use? What onne tivity pattern should we use? Let us suppose that
the variables are not only identi ally distributed but that they are also independent. Thus we have
the graphi al model shown in Figure 5.3. It should be emphasized that these assumptions are by
no means ne essary; they are simply one possible set of assumptions, orresponding to a parti ular
hoi e of graphi al model. (We will be seeing signi antly more omplex graphi al models on N
Gaussian nodes; see, e.g., the Kalman lter in Chapter 15).
The nodes in Figure 5.3 are shaded, re e ting the fa t that they are observed data. In general,
\data" are designated by the shading of nodes in our models. In the ontext of the Bayesian
approa h to estimation, this use of shading is the same onvention as we used in Chapter 2|in the
Bayesian approa h we ondition on the data in order to ompute probabilities for the parameters.
In the ontext of frequentist approa hes, where we no longer view ourselves as onditioning on the
2
We will often denote this density as N (; 2 ).
5.2. STATISTICAL PROBLEMS 11

X1 X2 X3 XN
Figure 5.3: A graphi al model representing the density estimation problem under an IID sampling
model. The assumption that the data are sampled independently is re e ted by the absen e of
links between the nodes. Ea h node is hara terized by the same density.

data, we simply treat shading as a diagrammati onvention to indi ate whi h nodes orrespond to
the observed data.
Letting X refer to the set of random variables (X1 ; X2 ; : : : ; XN ), and letting x refer to the obser-
vations (x1 ; x2 ; : : : ; xN ), we write the joint probability p(x j ) as the produ t of lo al probabilities,
one for ea h node in Figure 5.3:
N  
Y 1 1
p(x j ) = exp (x )2 (5.12)
n=1 (22 )1=2 22 n
( N )
1 1 X
= exp (x )2 ; (5.13)
(22 )N=2 22 n=1 n
or alternatively, given that this parti ular graph an be interpreted as either a dire ted graph or
an undire ted graph, we an view this joint probability as a produ t of potential fun tions on the
liques of the graph (whi h are singleton nodes in this ase).
Let us pro eed to al ulating parameter estimates. In parti ular let us al ulate the maximum
likelihood estimates of  and 2 . To do so we must maximize the likelihood p(x j ) with respe t to
. We nd it more onvenient to maximize the logarithm of the likelihood, whi h, given that the
logarithm is a monotoni fun tion, will not hange the results. Thus, let us de ne the log likelihood,
denoted l(; x), as:
l(; x) = log p(x j ); (5.14)
where we have reordered the variables on the left-hand side to emphasize that  is to be viewed
as the variable and x is to be viewed as a xed onstant. We now take the derivative of the log
likelihood with respe t to :
N !
l(; x)  N N 1 X
= log(2) log 2 (x )2 (5.15)
  2 2 22 n=1 n
N
1 X
= (x ): (5.16)
2 n=1 n
12 CHAPTER 5. STATISTICAL CONCEPTS

Setting equal to zero and solving, we obtain:

N
1X
^ML = x : (5.17)
N n=1 n
Thus we see that the maximum likelihood estimate of the mean of a Gaussian distribution is the
sample mean.
Similarly let us take the derivative of the log likelihood with respe t to 2 :
N !
l(; x)  N N 1 X
= log(2) log 2 (x )2 (5.18)
 2 2 2 2 22 n=1 n
N
N 1 X
= + (x )2 : (5.19)
22 24 n=1 n
Setting equal to zero and solving, we obtain:
N
2 = 1X
^ML (x ^ML )2 ; (5.20)
N n=1 n
and we see that the maximum likelihood estimate of the varian e is the sample varian e. (Note
that we are nding the joint estimates of  and 2 by setting both partial derivatives equal to zero
2 ).
and solving simultaneously; this explains the presen e of ^ML in the equation for ^ML

Bayesian univariate Gaussian density estimation

In the Bayesian approa h to density estimation the goal is to form a posterior density p( j x).
Let us onsider a simple version of this problem in whi h we take the varian e 2 to be a known
onstant and restri t our attention to the mean . Thus we wish to obtain the posterior density
p( j x), based on the prior density p() and the Gaussian likelihood p(x j ).
What prior distribution should we take for ? This is a modeling de ision, as was the de ision
to utilize a Gaussian for the probability of the data x in the rst pla e. As we will see, it is
mathemati ally onvenient to take p() to also be a Gaussian distribution. We will make this
assumption in this se tion, but let us emphasize at the outset that mathemati al onvenien e
should not, and need not, di tate all of our modeling de isions. Indeed, a major thrust of this book
is the development of methods for treating omplex models, pushing ba k the frontier of what is
\mathemati ally onvenient" and, in the Bayesian setting, permitting a wide and expressive range
of prior distributions.
If we take p() to be a Gaussian distribution, then we fa e another problem: what should we
take as the mean and varian e of this distribution? To be onsistent with the general Bayesian
philosophy, we should treat these parameters as random variables and endow them with a prior
distribution. This is indeed the approa h of hierar hi al Bayesian modeling, where we endow
parameters with distributions hara terized by \hyperparameters," whi h themselves an in turn
5.2. STATISTICAL PROBLEMS 13

X1 X2 X3 XN
Figure 5.4: The graphi al model for the Bayesian density estimation problem.

be endowed with distributions. While an in nite regress looms, in pra ti e it is rare to take the
hierar hi al Bayesian approa h to more than two or three levels, largely be ause there of diminishing
returns|additional levels make little di eren e to the marginal probability of the data and thus to
the expressiveness of our model.
Let us take the mean of p() to be a xed onstant 0 and take the varian e to be a xed
onstant  2 , while re ognizing that in general we might endow these parameters with distributions.
The graphi al model hara terizing our problem is shown in Figure 5.4. The graph has been
augmented with a node for the unknown mean . Note that there is a single su h node and that
its hildren are the data fXn g. Thus this graph provides more information than the graph of
Figure 5.3; in parti ular the independen e assumption is elaborated|the data are assumed to be
onditionally independent given the parameters.
The likelihood is identi al in form to the frequentist likelihood in Eq. (5.13). To obtain the
posterior we therefore need only multiply by the prior:
 
1 1
p() = exp ( 0 )2 (5.21)
(2 2 )1=2 2 2
to obtain the joint probability:
( )  
N
1 1 X 1 1
p(x; ) = exp (x )2 exp ( 0 )2 ; (5.22)
(22 )N=2 22 n=1 n (2 2 )1=2 2 2
whi h when normalized yields the posterior p( j x). Multiplying the two exponentials together
yields an exponent whi h is quadrati in the variable ; thus, normalization involves \ ompleting
the square." Appendix A presents the algebra (and in Chapter 13 we present a general matrix-
based approa h to ompleting the square|an operation that rops up often when working with
Gaussian random variables). The result takes the following form:
 
1 1
p( j x) = exp ( ~)2 ; (5.23)
(2~ 2 )1=2 2~2
14 CHAPTER 5. STATISTICAL CONCEPTS

where
N=2 1= 2
~ = x
 + ; (5.24)
N=2 + 1= 2 N=2 + 1= 2 0
where x is the sample mean, and where

N 1
 1
~ 2 = 2 + 2 : (5.25)
 
We see that the posterior probability is a Gaussian, with mean ~ and varian e ~ 2 .
Both the posterior varian e and the posterior mean have an intuitive interpretation. Note rst
that 2 =N is the varian e of a sum of N independent random variables with varian e 2 , thus 2 =N
is the varian e asso iated with the data. Eq. (5.25) says that we add the inverse of this varian e
to the inverse of the prior varian e to obtain the inverse of the posterior varian e. Thus, inverse
varian es add. From Eq. (5.24) we see that the posterior mean is obtained as a linear ombination
of the sample mean and the prior mean. The weights in this ombination an be interpreted as the
fra tion of the posterior varian e a ounted for by the varian e from the data term and the prior
varian e respe tively. These weights sum to one; thus, the ombination in Eq. (5.24) is a onvex
ombination.
As the number of data points N be omes large, the weight asso iated with x goes to one and
the weight asso iated with 0 approa hes zero. Thus in the limit of large data sets, the Bayes
estimate of  approa hes the maximum likelihood estimate of .

Plates
Let us take a qui k detour to dis uss a notational devi e that we will nd useful. Graphi al models
representing independent, identi ally distributed (IID) sampling have a repetitive stru ture that an
be aptured with a formal devi e known as a plate. Plates allow repeated motifs to be represented
in a simple way. In parti ular, the simple IID model shown in Figure 5.5(a) an be represented
more su in tly using the plate shown in Figure 5.5(b).
For the Bayesian model in Figure 5.6(a) we obtain the representation in Figure 5.6(b). Note that
the parameter  appears outside the plate; this aptures the fa t that there is a single parameter
value that is shared among the distributions for ea h of the Xn .
Formally, a plate is simply a graphi al model \ma ro." That is, to interpret Figure 5.5(b) or
Figure 5.6(b) we opy the graphi al obje t in the plate N times, where the number N is re orded
in the lower right-hand orner of the box, and apply the usual graphi al model semanti s to the
result.

Density estimation for dis rete data

Let us now onsider the ase in whi h the variables Xn are dis rete variables, ea h taking on one of
a nite number of possible values. We wish to study the density estimation problem in this setting,
re alling that \probability density" means \probability mass fun tion" in the dis rete ase.
As before, we will make the assumption that the data are IID, thus the modeling problem is
represented by the plate shown in Figure 5.5(b). Ea h of the variables Xn an take on one of M
5.2. STATISTICAL PROBLEMS 15

X1 X2 X3 XN Xn
N

(a) (b)
Figure 5.5: Repeated graphi al motifs an be represented using plates. The IID sampling model
for density estimation shown in (a) is represented using a plate in (b). The plate is interpreted by
opying the graphi al obje t within the box N times; thus the graph in (b) is a shorthand for the
graph in (a).

µ
µ

Xn
X1 X2 X3 XN N

(a) (b)
Figure 5.6: The Bayesian density estimation model shown in (a) is represented using a plate in (b).
Again, the graph in (b) is to be interpreted as a shorthand for the graph in (a).
16 CHAPTER 5. STATISTICAL CONCEPTS

values. To represent this set of M values we will nd it onvenient to use a ve tor representation.
In parti ular, let the range of Xn be the set of binary M - omponent ve tors with one omponent
equal to one and the other omponents equal to zero. Thus for a variable Xn taking on three values,
we have: 82 3 2 3 2 39
< 1 0 0 =
Xn 2 4 0 5 ; 4 1 5 ; 4 0 5 : (5.26)
:
0 0 1 ;
We use supers ripts to refer to the omponents of these ve tors, thus Xnk refers to the kth omponent
of thePvariable Xn . We have Xnk = 1 if and only if the variable Xn takes on its kth value. Note
that k Xnk = 1 by de nition.
Using this representation, we an write the probability distribution for Xn in a onvenient
general form. In parti ular, letting k represent the probability that Xn takes on its kth value, i.e.,
k , p(xkn = 1), we have:
p(xn j ) = 1xn 2xn


M
xM
1 2
n
: (5.27)
This is the multinomial probability distribution, Mult(1; ), with parameter ve tor  = (1 ; 2 ; : : : M ).
To al ulate the probability of the observation x, we take the produ t over the individual multino-
mial probabilities:
N
Y
p(x j ) =


MxMn
x1n x2n
1 2 (5.28)
n=1
PN 1 PN 2 PN M
= 1 n=1 xn 2 n=1 xn


M n=1 xn ; (5.29)
P
where the exponent Nn=1 xkn is the ount of the number of times the kth value of the multinomial
variable is observed a ross the N observations.
To al ulate the maximum likelihood estimates of the multinomial parameters we take the
logarithm of Eq. (5.29) to obtain the log likelihood:
N X
X M
l(; x) = xkn log k ; (5.30)
n=1 k=1

and it is this expression that we must maximize with respe t to .

This is a onstrained optimization problem for whi h we use Lagrange multipliers. Thus we
form the Lagrangian:
N X
X M M
X
~l(; x) = xkn log k + (1 k ); (5.31)
n=1 k=1 k=1
take derivatives with respe t to k :
PN
 ~l(; x) k
n=1 xn
=  (5.32)
k k
5.2. STATISTICAL PROBLEMS 17

and set equal to zero: PN k

n=1 xn = : (5.33)
^k;ML
Multiplying through by ^k;ML and summing over k yields:
M X
X N
 = xkn (5.34)
k=1 n=1
N X
X M
= xkn (5.35)
n=1 k=1
= N: (5.36)
Finally, substituting Eq. (5.36) ba k into Eq. (5.33) we obtain:
N
1X
^k;ML = xk : (5.37)
N n=1 n
P
Noting again that Nn=1 xkn is the ount of the number of times that the kth value is observed, we
see that the maximum likelihood estimate of k is a sample proportion.

Bayesian density estimation for dis rete data

In this se tion we dis uss a Bayesian approa h to density estimation for dis rete data. As in the
Gaussian setting, we spe ify a prior using a parameterized distribution and show how to ompute
the orresponding posterior.
An appealing feature of the solution to the Gaussian problem was that the prior and the posterior
have the same distribution|both are Gaussian distributions. Among other virtues, this implies
that Eq. (5.24) and Eq. (5.25) an be used re ursively |the posterior based on earlier observations
an serve as the prior for additional observations. At ea h step the posterior distribution remains
in the Gaussian family.
To a hieve a similar losure property in the dis rete problem we must nd a prior distribution
whi h when multiplied by the multinomial distribution yields a posterior distribution in the same
family. Clearly, this an be a hieved by a prior distribution of the form:
p() = C ( )1 1 1 2 2 1


MM 1 ; (5.38)
P
for i i = 1, where = ( 1 ; : : : ; M ) are hyperparameters and C ( ) is a normalizing onstant.3
This distribution, known as the Diri hlet distribution, has the same fun tional form as the multino-
mial, but the i are random variables in the Diri hlet distribution and parameters in the multinomial
distribution. The onstant C ( ) is obtained via a bit of al ulus (see Appendix B):
PM
( i=1 i ) ;
C( ) = QM (5.39)
i=1 ( i )
3
The negative one in the exponent is a onvention; we ould rede ne the i to remove it.
18 CHAPTER 5. STATISTICAL CONCEPTS

where (
) is the gamma fun tion. In the rest of this se tion we will not bother with al ulating the
normalization; on e we have a distribution in the Diri hlet form we an substitute into Eq. (5.39)
to nd the normalization fa tor.
We now al ulate the posterior probability:
PN 1 PN 2 PN M 1
p( j x) / 1 n=1 xn 2 n=1 xn


M n=1 xn 1 1 12 2 1


MM (5.40)
PN 1 PN 2 PN M
= 1 n=1 xn + 1 1 2 n=1 xn + 2 1


M n=1 xn + M 1 : (5.41)
P
N xk + . We see that to update the prior into a
This is a Diri hlet density, with parameters
PN n=1 n k
posterior we simply add the ount n=1 xkn to the prior parameter k .
It is worthwhile to onsider the spe ial ase of the multinomial distribution when M = 2. In
this setting, Xn is best treated as a binary variable rather than a ve tor; thus: xn 2 f0; 1g. The
multinomial distribution redu es to:
p(xn j ) = xn (1 )1 xn ; (5.42)
the Bernoulli distribution. The parameter  en odes the probability that Xn takes the value one.
In the ase M = 2, the Diri hlet distribution spe ializes to the beta distribution :
p() = C ( ) 1 1 (1 ) 2 1 ; (5.43)
where = ( 1 ; 2 ) is the hyperparameter. The beta distribution has its support on the interval
[0; 1℄. Plots of the beta distribution are shown in Figure 5.7 for various values of 1 and 2 . Note
that the uniform distribution is the spe ial ase of the beta distribution
P
when 1 = 1 and 2 = 1.
As the number of data points N be omes large, the sums Nn=1 xkn dominate the prior terms k
in the posterior probability. In this limit, the posterior approa hes the log likelihood in Eq. (5.30)
and the Bayes estimate of  approa hes the maximum likelihood estimate of .

Mixture models
It is important to re ognize that the Gaussian and multinomial densities are by no means the
universally best hoi es of density model. Suppose, for example, if the data are ontinuous data
restri ted to the half-in nite interval [0; 1). The Gaussian, whi h assigns density to the entire real
line, is unnatural here, and densities su h as the gamma or lognormal, whose support is [0; 1), may
be preferred. Similarly, the multinomial distribution treats dis rete data as an unordered, nite
set of values. In problems involving ordered sets, and/or in nite ranges, probability distributions
su h as the Poisson or geometri may be more appropriate. Maximum likelihood and Bayesian
estimates are available for these distributions, and indeed there is a general family known as the
exponential family|whi h in ludes all of the distributions listed above and many more|in whi h
expli it formulas an be obtained. (We will dis uss the exponential family in Chapter 8).
This larger family of distributions is still, however, restri tive. Consider the probability density
shown in Figure 5.8. This density is bimodal and we are unable to represent it within the family
of Gaussian, gamma or lognormal densities. Given a data set fxn g sampled from this density, we
5.2. STATISTICAL PROBLEMS 19

(.5,.5)
5
(1,1)
(2,2)
4 (10,30)

0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Figure 5.7: The beta( 1 ; 2 ) distribution for various values of the parameters 1 and 2 .

an naively t a Gaussian density, but the likelihood that we a hieve will in general be signi antly
smaller than the likelihood of the data under the true density, and the resulting density estimate
will bear little relationship to the truth.
Multimodal densities often re e t the presen e of subpopulations or lusters in the population
from whi h we are sampling. Thus, for example, we would expe t the density of heights of trees
in a forest to be multimodal, re e ting the di erent distributions of heights of di erent spe ies.
It may be that for a parti ular spe ies the heights are unimodal and reasonably well modeled by
a simple density, su h as a density in the exponential family. If so, this suggests a \divide-and-
onquer" strategy in whi h the overall density estimation is broken down into a set of smaller density
estimation problems that we know how to handle. Let us pro eed to develop su h a strategy.
Let fk (x j k ) be the density for the kth subpopulation, where k is a parameter ve tor. We
de ne a mixture density for a random variable X by taking the onvex sum over the omponent
densities fk (x j k ):
K
X
p(x j ) = k fk (x j k ); (5.44)
k=1
where the k are nonnegative onstants that sum to one:
K
X
k = 1: (5.45)
k=1
The densities fk (x j k ) are referred to in this setting as mixture omponents and the parameters k
20 CHAPTER 5. STATISTICAL CONCEPTS

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Figure 5.8: A bimodal probability density.

are referred to as mixing proportions. The parameter ve tor  is the olle tion of all of the param-
eters, in luding the mixing proportions:  , ( 1 ; : : : ; K ; 1 ; : : : ; K ). That the fun tion p(x j )
that we have de ned is in fa t a density follows from the onstraint that the mixing proportions
sum to one.
The example shown in Figure 5.8 is a mixture density with K = 2:
p(x j ) = 1 N (x j 1 ; 12 ) + 2 N (x j 2 ; 22 ); (5.46)
where the mixture omponents are Gaussian distributions with means k and varian es k2 . Gaus-
sian mixtures are a popular form of mixture model, parti ular in multivariate settings (see Chap-
ter 10).
It is illuminating to express the mixture density in Eq. (5.44) in a way that makes expli it its
interpretation in terms of subpopulations. Let us do this using the ma hinery of graphi al models.
As shown in Figure 5.9, we introdu e a multinomial random variable Z into our model. We also
introdu e an edge from Z to X . Following the re ipe from Chapter 2 we endow this graph with a
joint probability distribution by assigning a marginal probability to Z and a onditional probability
to X . Let k be the probability that Z takes on its kth value; thus, k , p(z k = 1). Moreover,
onditional on Z taking on its kth value, let the onditional probability of X , p(x j z k = 1), be
given by fk (x j k ). The joint probability is therefore given by:
p(x; z k = 1 j ) = p(x j z k = 1; )p(z k = 1 j ) (5.47)
= k fk (x j k ); (5.48)
5.2. STATISTICAL PROBLEMS 21

X
Figure 5.9: A mixture model represented as a graphi al model. The latent variable Z is a multi-
nomial node taking on one of K values.

where  , ( 1 ; : : : ; K ; 1 ; : : : ; K ). To obtain the marginal probability of X we sum over k:

K
X
p(x j ) = p(x; z k = 1 j ) (5.49)
k=1
K
X
= k fk (x j k ); (5.50)
k=1
whi h is the mixture model in Eq. (5.44).
This model gives us our rst opportunity to invoke our dis ussion of probabilisti inferen e
from Chapter 3. In parti ular, given an observation x, we an use Bayes rule to invert the arrow
in Figure 5.9 and al ulate the onditional probability of Z :
p(x j z k = 1; k )p(z k = 1)
p(z k = 1 j x; ) = P (5.51)
j p(x j z = 1; j )p(z = 1)
j j
f (x j k )
= Pk k : (5.52)
j j fj (x j j )
This al ulation allows us to use the mixture model to lassify or ategorize the observation x into
one of the subpopulations or lusters that we assume to underly the model. In parti ular we might
lassify x into the lass k that maximizes p(z k = 1 j x; ).
Let us turn to the problem of estimating the parameters of the mixture model from data.
We again assume for simpli ity a sampling model in whi h we have N IID observations fxn ; n =
1; : : : ; N g, while again noting that we will move beyond the IID setting in later hapters. The
IID assumption orresponds to repli ating our basi graphi al model N times, yielding the plate
shown in Figure 5.10. Note again that the variables Zn are unshaded|they are unobserved or
latent variables. We have introdu ed them into our model in order to make expli it the stru tural
assumptions that lie behind the mixture density that we are using, but we need not assume that
these variables are observed.
22 CHAPTER 5. STATISTICAL CONCEPTS

Zn

Xn
N

Figure 5.10: The mixture model under an IID sampling assumption.

The log likelihood is given by taking the logarithm of the joint probability asso iated with the
model, whi h in the IID ase be omes a sum of log probabilities. Again letting x = (x1 ; : : : ; xN ),
we have:
N
X K
X
l(; x) = log k fk (xn j k ): (5.53)
n=1 k=1
To obtain maximum likelihood estimates we take derivatives with respe t to  and set to zero. The
resulting equations are, however, nonlinear and do not admit a losed-form solution; solving these
equations requires iterative methods. While any of a variety of numeri al methods an be used, there
is a parti ular iterative method|the Expe tation-Maximization (EM) algorithm |that is natural
not only for mixture models but also for more general graphi al models. The EM algorithm involves
an alternating pair of steps, the E step and the M step. The E step involves running an inferen e
algorithm|for example the elimination algorithm that we dis ussed in Chapter 3|to essentially
\ ll in" the values of the unobserved nodes given the observed nodes. In the ase of mixture models,
this redu es to the invo ation of Bayes rule in Eq. (5.52). The M step treats the \ lled-in" graph
as if all of the lled-in values had been observed, and updates the parameters to obtain improved
values. In the mixture model setting this essentially redu es to nding separate density estimates
for the separate subpopulations. We will present the EM algorithm formally in Chapter 11, and
present its appli ation to mixture models in Chapter 10.

Nonparametri density estimation

In many ases data may ome from a omplex me hanism about whi h we have little or no prior
knowledge. The density underlying the data may not fall into one of the \standard" forms. The
density may be multimodal, but we may have no reason to suspe t underlying subpopulations and
may have no reason to attribute any parti ular meaning to the modes. When we nd ourselves in
5.2. STATISTICAL PROBLEMS 23
0.4

0.35

0.3

0.25

0.2

0.15

0.1

0.05

0
−3 −2 −1 0 1 2 3
x

Figure 5.11: An example of kernel density estimation. The kernel fun tions are Gaussians entered
at the data points xn (shown as rosses on the abs issa). Ea h Gaussian has a standard deviation
 = 0:35. The Gaussians have been s aled by dividing by the number of data points (N = 8). The
density estimate (shown as a dotted urve) is the sum of these s aled kernels.

su h a situation|by no means un ommon|what do we do?

Nonparametri density estimation provides a general lass of methods for dealing with su h
knowledge-poor ases. In this se tion we introdu e this approa h via a simple, intuitive nonpara-
metri method known as a kernel density estimator. We return to a fuller dis ussion of nonpara-
metri methods in Chapter 25.
The basi intuition behind kernel density estimation is that ea h data point xn provides eviden e
for non-zero probability density at that point. A simple way to harness this intuition is to pla e
an \atom" of mass at that point (see Figure 5.11). Moreover, making the assumption that the
underlying probability density is smooth, we let the atoms have a non-zero \width." Superimposing
N su h atoms, one per data point, we obtain a density estimate.
More formally, let k(x; xn ; ) be a kernel fun tion |a nonnegative fun tion integrating to one
(with respe t to x). The argument xn determines the lo ation of the kernel fun tion; kernels
are generally symmetri about xn. The parameter  is a general \smoothing" parameter that
determines the width of the kernel fun tions and thus the smoothness of the resulting density
estimate. Superimposing N su h kernel fun tions, and dividing by N , we obtain a probability
density:
N
1X
p^(x) = k(x; xn ; ): (5.54)
N n=1
This density is the kernel density estimate of the underlying density p(x).
A variety of di erent kernel fun tions are used in pra ti e. Simple (e.g., pie ewise polynomial)
24 CHAPTER 5. STATISTICAL CONCEPTS

fun tions are often preferred, partly for omputational reasons ( al ulating the density at a given
point x requires N fun tion evaluations). Gaussian fun tions are sometimes used, in whi h ase xn
plays the role of the mean and  plays the role of the standard deviation.
While the kernel fun tion is often hosen a priori, the value of  is generally hosen based on the
data. This is a nontrivial estimation problem for whi h lassi al estimation methods are often of
little help. In parti ular, it is important to understand that maximum likelihood is not appropriate
for solving this problem. Suppose that we interpret the density in Eq. (5.54) as a likelihood fun tion,
with  as the parameter. For most reasonable kernels, this \likelihood" in reases monotoni ally
as  goes to zero, be ause the kernel assigns more probability density to the points xn for smaller
values of . Indeed, in the limit of  = 0, the kernel generally approa hes a delta fun tion, giving
in nite likelihood to the data. A sum of delta fun tions is obviously a poor density estimate.
We will dis uss methods for hoosing smoothing parameters in Chapter 25. As we will see, most
pra ti al methods involve some form of ross-validation, in whi h a fra tion of the data are held
out and used to evaluate various hoi es of . Both overly small and overly large values of  will
tend to assign small probability density to the held-out data, and this provides a rational approa h
to hoosing .
The problem here is a general one, motivating a distin tion between parametri models and
nonparametri models and suggesting the need for distin t methods for their estimation. Under-
standing the distin tion requires us to onsider how a given model would hange if the number of
data points N were to in rease. For parametri models the basi stru ture of the model remains
xed as N in reases. In parti ular, for the Gaussian estimation problem treated in Se tion 5.2.1,
the lass of densities that are possible ts to the data remains the same whatever the value of
N ; for ea h N we obtain a Gaussian density with estimated parameters ^ and ^ 2 . In reasing the
number of data points in reases the pre ision of these estimates, but it does not in rease the lass
of densities that we are onsidering. In the nonparametri ase, on the other hand, the lass of
densities in reases as N in reases. In parti ular, with N + 1 data points it is possible to obtain
densities with N + 1 modes; this is not possible with N data points.
An alternative perspe tive is to view the lo ations of the kernels as \parameters"; the number of
su h \parameters" in reases with the number of data points. In e e t, we an view nonparametri
models as parametri , but with an unbounded, data-dependent, number of parameters. Indeed, in
an alternative language that is often used, parametri models are referred to as \ nite-dimensional
models," and nonparametri models are referred to as \in nite-dimensional models."
It is worthwhile to ompare the kernel density estimator in Eq. (5.54) to the mixture model
in Eq. (5.44). Consider in parti ular the ase in whi h Gaussian mixture omponents are used in
Eq. (5.44) and Gaussian kernel fun tions are used in Eq. (5.54). In this ase the kernel estimator
an be viewed as a mixture model in whi h the means are xed to the data point lo ations, the
varian es are set to 2 , and the mixing proportions are set to 1=N . In what sense are the two
di erent approa hes to density estimation really di erent?
Again, the key di eren e between the two approa hes is revealed when we let the number of
data points N grow. The mixture model is generally viewed as a parametri model, in whi h ase
the number of mixture omponents, K , does not in rease as the number of data points grows.
This is onsistent with our interpretation of a mixture model in terms of a set of K underly-
5.2. STATISTICAL PROBLEMS 25

ing subpopulations|if we believe that these subpopulations exist, then we do not vary K as N
in reases. In the kernel estimation approa h, on the other hand, we have no ommitment to under-
lying subpopulations, and we a ord no spe ial treatment to the number of kernels. As the number
of data points grows, we allow the number of kernels to grow. Moreover we generally expe t that
 will shrink as N grows to allow an in reasingly lose t to the details of the true density.
There are several aveats to this dis ussion. First, in the mixture model setting, we may not
know the number K of mixture omponents in pra ti e and we may wish to estimate K from the
data. This is a model sele tion problem (see Se tion 5.3). Solutions to model sele tion problems
generally involve allowing K to in rease as the number of data points in reases, based on the
fa t that more data points are generally needed to provide more ompelling eviden e for multiple
modes. Se ond, mixture models an also be used nonparametri ally. In parti ular, a mixture sieve
is a mixture model in whi h the number of omponents is allowed to grow with the number of data
points. This di ers from kernel density estimation in that the lo ation of the mixture omponents
are treated as free parameters rather than being xed at the data points; moreover, ea h mixture
omponent generally has its own (free) s ale parameter. Also, the growth rate of the number of
\parameters" in mixture sieves is slower than that of kernel density estimation (e.g., log N vs.
N ). As this dis ussion begins to suggest, however, it be omes diÆ ult to enfor e a lear boundary
between parametri and nonparametri methods. A given approa h an be treated in one way or
the other, depending on a modeler's goals and assumptions.
There is a general tradeo between exibility and statisti al eÆ ien y that is relevant to this
dis ussion. If the underlying \true" density is a Gaussian, then we probably want to estimate this
density using a parametri approa h, we an also use a kernel density estimate. The latter estimate
will eventually onverge to the true density, but it may require very many data points. A parametri
estimator will onverge more rapidly. Of ourse, if the true density is not a Gaussian, then the
parametri estimate would still onverge, but to the wrong density, whereas the nonparametri
estimate would eventually onverge to the true density. In sum, if we are willing to make more
assumptions then we get faster onvergen e, but with the possibility of poor performan e if reality
does not mat h our assumptions. Nonparametri estimators allow us to get away with fewer
assumptions, while requiring more data points for omparable levels of performan e.
There is also a general point to be made with respe t to the representation of densities in
graphi al models. As suggested in Figure 5.12, there are two ways to represent a multi-modal
density as a graphi al model. As shown in Figure 5.12(a), we an allow the lass of densities
p(x) at node X to in lude multi-modal densities, su h as mixtures or kernel density estimates.
Alternatively, we an use the \stru tured" model depi ted in Figure 5.12(b), where we obtain a
mixture distribution for Xn by marginalizing over the latent variable Zn . Although it may seem
natural to reserve the latter representation for parametri modeling, in parti ular for the setting
in whi h we attribute a \meaning" to the latent variable, su h a step is in general unwarranted.
The mixture sieve exempli es a situation in whi h we may wish to use graphi al ma hinery to
represent the stru ture of a nonparametri model expli itly. In general, the hoi es of how to
use and how to interpret graphi al stru ture are modeling de isions. While we may wish to use
graphi al representations to express domain-spe i stru tural knowledge, we may also be guided
by other fa tors, in luding mathemati al onvenien e and the availability of omputational tools.
26 CHAPTER 5. STATISTICAL CONCEPTS

Zn

Xn
N
Xn
N

(a) (b)
Figure 5.12: Two ways to represent a multi-modal density within the graphi al model formalism.
(a) The lo al probability model at ea h node is a mixture or a kernel density estimate. (b) A latent
variable is used to represent mixture omponents expli itly; marginalizing over the latent variable
yields a mixture model for the observable Xn .

There is nothing inappropriate about letting su h fa tors be a guide, but in doing so we must be
autious about any interpretation or meaning that we atta h to the model.

Summary of density estimation

Our goal in this se tion has not been to provide a full treatment of density estimation; indeed
we have only s rat hed the surfa e of what is an extensive literature in statisti s. We do hope,
however, to have introdu ed a few key ideas|the al ulation of maximum likelihood and Bayesian
parameter estimates for Gaussian and multinomial densities, the use of mixture models to obtain a
ri her lass of density models, and the distin tion between parametri and nonparametri density
estimation. Ea h of these ideas will be pi ked up and pursued in numerous ontexts throughout
the book.

5.2.2 Regression
In a regression model the goal is to model the dependen e of a response or output variable Y
on a ovariate or input variable X . We apture this dependen e via a onditional probability
distribution p(y j x). In graphi al model terms, we have a two-node model in whi h X is the parent
and Y is the hild (see Figure 5.13).
One way to treat regression problems is to estimate the joint density of X and Y and to al ulate
5.2. STATISTICAL PROBLEMS 27

Y
Figure 5.13: A regression model.

Xn

Yn
N

Figure 5.14: The IID regression model represented graphi ally.

the onditional p(y j x) from the estimated joint. This approa h for es us to model X , however,
whi h may not be desired. Indeed, in many appli ations of regression, X is high-dimensional and
hard to model. Moreover, the observations of X are often xed by experimental design or another
form of non-random pro ess, and it is problemati to treat them via a simple sampling model,
su h as the IID model. In summary, it is ne essary to develop methods appropriate to onditional
densities.
Our dis ussion here will be brief, with a fo us on basi representational issues.
We assume that we have a set of pairs of observed data, f(xn ; yn ); n = 1; : : : ; N g, where xn is
an observation of the input variable and yn is a orresponding observation of the output variable.
We again assume an independent, identi al distributed (IID) sampling model for simpli ity. The
graphi al representation of the IID regression model is shown as a plate in Figure 5.14.
Let us now onsider some of the possible hoi es for the onditional probability model p(yn j xn ).
28 CHAPTER 5. STATISTICAL CONCEPTS

y
x
x x
x x
x x x
x
x x x x
x x
x x
x x x
x
x
x

x
Figure 5.15: The linear regression model expresses the response variable Y in terms of the ondi-
tional mean fun tion|the line in the gure|and input-independent random variation around the
onditional mean.

As in the ase of density estimation, we have a wide spe trum of possibilities, in luding parametri
models, mixture models, and nonparametri models. We will dis uss these models in detail in
Chapters 6, 10, and 25, respe tively, but let us sket h some of the possibilities here.
A linear regression model expresses Yn as the sum of (1) a purely deterministi omponent that
depends parametri ally on xn , and (2) a purely random omponent that is fun tionally independent
of xn:
Yn = T xn + n; (5.55)
where is a parameter ve tor and n is a random variable having zero mean. Taking the onditional
expe tation of both sides of this equation yields E [Yn j xn ℄ = T xn . Thus the linear regression model
expresses Yn in terms of input-independent random variation n around the onditional mean T xn
(see Figure 5.15). The hoi e of the distribution of n , whi h ompletes the spe i ation of the
model, is analogous to the hoi e of a density model in density estimation, and depends on the
nature of Yn . \Linear regression" generally refers to the ase in whi h Yn is real-valued and the
distribution is taken to be N (0; 2 ). (In Chapter 8 we will be dis ussing \generalized linear models,"
whi h are regression models that are appropriate for other types of response variables). In the linear
regression ase, we have:
 
1 1 T x )2
P (yn j xn ; ) = exp (y ; (5.56)
(22 )1=2 22 n n

where for simpli ity we have taken yn to be univariate. The parameter ve tor  in ludes , whi h
determines the onditional mean, and 2 , whi h is the varian e of n and determines the s ale of
5.2. STATISTICAL PROBLEMS 29

the variation around the onditional mean.

Linear regression is in fa t broader than it may appear at rst sight, in that the fun tion Tx
n
need only be linear in and in parti ular may be nonlinear in xn . Thus the model:
Yn = T (x ) +  ; (5.57)
n n

where (
) is a ve tor-valued fun tion of xn , is a linear regression model. This model is a parametri
model in that (
) is xed and our freedom in modeling the data omes only from the nite set of
parameters .
The problem of estimating the parameters of regression models is in prin iple no di erent from
the orresponding estimation problem for density estimation. In the maximum likelihood approa h,
we form the log likelihood:
N
X
l(; x) = log p(yn j xn ; ); (5.58)
n=1
take derivatives with respe t to , set to zero and (attempt to) solve. We will dis uss the issues
that arise in arrying out this al ulation in later hapters.

Conditional mixture models

Mixture models provide a way to move beyond the stri tures of linear regression modeling. We an
onsider both a broader lass of onditional mean fun tions as well as a broader lass of density
models for n . Consider in parti ular the graphi al model shown in Figure 5.16(a). We have
introdu ed a multinomial latent variable Zn that depends on the input Xn ; moreover, the response
Yn depends on both Xn and Zn . This graph orresponds to the following probabilisti model:
K
X
p(yn j xn ; ) = p(znk = 1 j xn ; )p(yn j znk = 1; xn ; ); (5.59)
k=1

a onditional mixture model. Ea h mixture omponent p(yn j znk = 1; xn ) orresponds to a di erent

regression model, one for ea h value of k. The mixing proportions p(znk = 1 j xn ) \swit h" among the
regression models as a fun tion of xn . Thus, as suggested in Figure 5.16(a), the mixing proportions
an be used to pi k out regions of the input spa e where di erent regression fun tions are used. We
an parameterize both the mixing proportions and the regression models and estimate both sets of
parameters from data. This is a \divide-and- onquer" methodology in the regression domain. (We
provide a fuller des ription of this model in Chapter 10).
The example in Figure 5.16(a) utilizes mixing proportions that are sharp, nearly binary fun -
tions of Xn , but it is also of interest to onsider models in whi h these fun tions are smoother,
allowing overlap in the omponent regression fun tions. Indeed, in the limiting ase we obtain the
model shown in Figure 5.16(b) in whi h the latent variable Zn is independent of Xn . Here the
presen e of the latent variable serves only to indu e multimodality in the onditional distribution
p(yn j xn ). Mu h as in the ase of density estimation, su h a regression model may arise from a set
of subpopulations, ea h hara terized by a di erent \ onditional mean."
30 CHAPTER 5. STATISTICAL CONCEPTS

Xn
y
x
x x
x
x
x
(a) Zn x x x
x
x
x
x
zn = 0 x x zn = 1
x x
x
x x x
x
x x
Yn x
N x x x
x

Xn
y
x
x x
x
x x x
x x
x
x
x x
(b) Zn x
x

x
x
x x x
x x x x
x x x
Yn x
N x x

Figure 5.16: Two variants of onditional regression model. In (a), the latent variable Zn is depen-
dent on Xn . This orresponds to breaking up the input spa e into (partially overlapping) regions
labeled by the values of Zn . An example with binary Zn is shown in the gure on the right, where
the dashed line labeled by zn = 1 is the probability p(zn = 1 j xn ), and the dashed line labeled by
zn = 0 is the probability p(zn = 0 j xn ). The two lines are the onditional means of the regres-
sions, p(yn j zn ; xn ), for the two values of zn , with the leftmost line orresponding to zn = 0 and
the rightmost line orresponding to zn = 1. In (b), the latent variable Zn is independent of Xn .
This orresponds to total overlap of the regions orresponding to the values of Zn and yields an
input-independent mixture density for ea h value of xn .
5.2. STATISTICAL PROBLEMS 31

τ2 α

µ β
Xn

θ σ2

Yn
N

Figure 5.17: A Bayesian linear regression model. The parameter ve tor  is endowed with a
Gaussian prior, N (;  2 ). The varian e 2 is endowed with an inverse gamma prior, IG( ; ).

The parameters of onditional mixture models an be estimated using the EM algorithm, as

dis ussed in Chapter 10. Indeed, EM an be used quite generi ally for latent variable models su h
as those in Figure 5.16.

Nonparametri regression
Let us brie y onsider the nonparametri approa h to regression. While it is possible to use
nonparametri methods to expand the repertoire of probability models for n, a more ommon
usage of nonparametri ideas involves allowing a wider lass of onditional mean fun tions. The
basi idea is to break up the input spa e into (possibly overlapping) regions, with one su h region
for ea h data point. Let us give an example from the lass of methods known as kernel regression.
As in kernel density estimation, let k(x; xn ; ) be a kernel fun tion entered around the data point
xn . Denoting the onditional mean fun tion as f (x), we form an estimate as follows:
PN
f^(x) = n=1 k (x; xn ; )yn
PN (5.60)
m=1 k (x; xm ; )
That is, we estimate the onditional mean at x as the onvex sum of the observed values yn , where
the weights in the sum are given by the normalized values of the kernel fun tions, one for ea h xn ,
evaluated at x. Given that kernel fun tions are generally hosen to be \lo al," having most of their
support near xn , we see that the kernel regression estimate at x is a lo al average of the values yn
in the neighborhood of x.
We an on e again forge a link between the mixture model approa h and the nonparametri
kernel regression approa h. As we ask the reader to verify in Exer ise ??, taking the onditional
32 CHAPTER 5. STATISTICAL CONCEPTS

1 d
Xn X n2 Xn

Yn
N

Figure 5.18: A graphi al representation of the regression model in whi h the omponents of the
input ve tor are treated as expli it nodes.

mean of Eq. (5.59) yields a weighted sum of onditional mean fun tions, one for ea h omponent
k, where the weights are the mixing proportions p(znk = 1 j xn ). The kernel regression estimate
in Eq. (5.60)P
an be viewed as an instan e of this model, if we treat the normalized kernels
k(x; xn ; )= Nm=1 k(x; xm ; ) as mixing proportions, and the values yn as ( onstant) onditional
means. The same omments apply to this redu tion as to the analogous redu tion in the ase
of density estimation. In parti ular, as N in reases, the number of omponents K in a para-
metri onditional mixture model generally remain xed, whereas the number of kernels in the
kernel regression model grow. We an, however, onsider onditional mixture sieves, and obtain a
nonparametri variant of a mixture model.

Bayesian approa hes to regression

All of the models that we have onsidered in this se tion an be treated via Bayesian methods,
where we endow the parameters (or entire onditional mean fun tions) with prior distributions.
We then invoke Bayes rule to al ulate posterior distributions. Figure 5.17 illustrates one su h
Bayesian regression model.

Remarks
Let us make one nal remark regarding the graphi al representation of regression models. Note
that in this se tion we have treated the input variables Xn as single nodes, not availing ourselves
of the opportunity to represent the omponents of these ve tor-valued variables as separate nodes
(see Figure 5.18). This is onsistent with our treatment of Xn as xed variables to be onditioned
on; representing the omponents as separate nodes would imply marginal independen e between
the omponents, an assumption that we may or may not wish to make. It is important to note,
5.2. STATISTICAL PROBLEMS 33

Q Q

X X

(a) (b)
Figure 5.19: (a) The generative approa h to lassi ation represented as a graphi al model. Fitting
the model requires estimating the marginal probability p(q) and the onditional probability p(x j q).
(b) The dis riminative approa h to lassi ation represented as a graphi al model. Fitting the
model requires estimating the onditional probability p(q j x).

however, that regression methods are agnosti regarding modeling assumptions about the ondi-
tioning variables. Regression methods form an estimate of p(y j x) and this onditional density an
be omposed with an estimate of p(x) to obtain an estimate of the joint. This allows us to use
regression models as omponents of larger models. In parti ular, in the ontext of a graphi al model
in whi h a node A has multiple parents B1 ; B2 ; : : : ; Bk , we are free to use regression methods to
represent p(A j B1 ; B2 ; : : : ; Bk ), regardless of the modeling assumptions made regarding the nodes
Bi . Indeed ea h of the Bi may themselves be modeled in terms of regressions on variables further
\upstream."

5.2.3 Classi ation

Classi ation problems are related to regression problems in that they involve pairs of variables.
The distinguishing feature of lassi ation problems is that the response variable ranges over a
nite set, a seemingly minor issue that has important impli ations.
In lassi ation we often refer to the ovariate X as a feature ve tor, and the orresponding
dis rete response, whi h we denote by Q, as a lass label. We typi ally view the feature ve tors as
des riptions of obje ts, and the goal is to label the obje ts, i.e., to lassify the obje ts into one of
a nite set of ategories.
There are two basi approa hes to lassi ation problems, whi h an be interpreted graphi ally
in terms of the dire tion of the edge between X and Q. The rst approa h, whi h we will refer to
as generative, is based on the graphi al model shown in Figure 5.19(a), in whi h there is an arrow
from Q to X . This approa h is losely related to density estimation|for ea h value of the dis rete
34 CHAPTER 5. STATISTICAL CONCEPTS

Qn Qn

Xn Xn
N N

(a) (b)
Figure 5.20: The IID lassi ation models for the (a) generative approa h and (b) dis riminative
approa h.

variable Q we have a density, p(x j q), whi h we refer to as a lass- onditional density. We also
require the marginal probability p(q), whi h we refer to as the prior probability of the lass Q (it
is the probability of the lass before a feature ve tor X is observed). This marginal probability
is required if we are going to be able to \invert the arrow" and ompute p(q j x)|the posterior
probability of lass Q.
The se ond approa h to lassi ation, whi h we refer to as dis riminative, is losely related to
regression. Here we represent the relationship between the feature ve tors and the labels in terms
of an arrow from X to Q (see Figure 5.19(b)). That is, we represent the relationship in terms of
the onditional probability p(q j x). When lassifying an obje t we simply plug the orresponding
feature ve tor x into the onditional probability and al ulate p(q j x). Performing this al ulation,
whi h tells us whi h lass label has the highest probability, makes no referen e to the marginal
probability p(x) and, as in regression, we may wish to abstain from in orporating su h a marginal
into the model.
As in regression, we have a set of data pairs f(xn ; qn ) : n = 1; : : : ; N g, assumed IID for simpli ity.
The representations of the lassi ation problem as plates are shown in Figure 5.20.
On e again we postpone a general presentation of parti ular representations for the onditional
probabilities in lassi ation problems until later hapters. But let us brie y dis uss a anoni al
example that will illustrate some typi al representational hoi es, as well as illustrate some of
the relationships between the generative and the dis riminative approa hes to lassi ation. This
example and several others will be developed in onsiderably greater detail in later hapters.
We spe ialize to two lasses. Let us hoose Gaussian lass- onditional densities with equal
ovarian e matri es for the two lasses. An example of these densities (where we have assumed equal
5.2. STATISTICAL PROBLEMS 35

x2 x2
x x

x xx x x xx x
o o
x o x
o x x x x
o o
o o o o o o
o x o x
µ1 µ1
o o
x 1 x1
µ0 µ0

(a) (b)
Figure 5.21: (a) Contour plots and samples from two Gaussian lass- onditional densities for two-
dimensional feature ve tors xn = (x1n ; x2n ). The Gaussians have means 0 and 1 for lass qn = 0
and qn = 1, respe tively, and equal ovarian e matri es. (b) The solid lines are the ontours of
the posterior probability, p(qn = 1 j xn ). In the dire tion orthogonal to the linear ontours, the
posterior probability is a monotoni ally in reasing fun tion given by (Eq. (5.61)). This fun tion is
sket hed at the top of the gure.

lass priors) is shown in Figure 5.21(a). We use Bayes rule to ompute the posterior probability
that a given feature ve tor xn belongs to lass qn = 1. Intuitively, we expe t to obtain a ramp-like
fun tion whi h is zero in the vi inity of the lass qn = 0, in reases to one-half in the region between
the two lasses, and approa hes one in the vi inity of the lass qn = 1. This posterior probability
fun tion is shown in Figure 5.21(b), where indeed we see the ramp-like shape.
Analyti ally, as we show in Chapter 7, for Gaussian lass- onditional densities the ramp-like
posterior probability turns out to be the logisti fun tion :
1
p(qn = 1 j xn ) = ; (5.61)
1 + e T xn
where  is a parameter ve tor that depends on the parti ular hoi es of means and ovarian es for
the lass- onditional densities, as well as the lass priors. The inner produ t between  and xn is
a proje tion operation that is responsible for the linear ontours that we see in Figure 5.21(b).
Given these parametri forms for the lass- onditional densities (the Gaussian densities) and
the posterior probability (the logisti fun tion), we must spe ify how to estimate the parameters
based on the data. It is here that the generative and dis riminative approa hes begin to diverge.
From the generative point of view, the problem is that of estimating the means and ovarian es of
the Gaussian lass- onditional densities, as well as the lass priors. These are density estimation
36 CHAPTER 5. STATISTICAL CONCEPTS

x2 x2
x x
x x
x x x x x x
x x
o x x x
o x x x x x x x
x x
x x
o oo o x xx x x
x x
o x x
oo o xx x
o x
o x
o x

x1 x1
(a) (b)

Figure 5.22: (a) A lassi ation problem with the lass qn = 0 labeled with a \0" and the lass
qn = 1 labeled with a \x". (b) The same feature ve tors xn as in (a), but with the labels erased.

problems, and the ma hinery of Se tion 5.2.1 is invoked to solve them. With these density estimates
in hand, we derive an estimate of  and thereby al ulate an estimate of the posterior probability.
Essentially, the goal is to model the lasses, without any dire t attempt to dis riminate between
the lasses.
In the dis riminative approa h, on the other hand, the logisti fun tion is the entral obje t of
analysis. Indeed, in Chapter 7, we des ribe a regression-like method for estimating  dire tly from
data, without making referen e to the means and ovarian es of an underlying generative model.
Intuitively, this method an be viewed as an attempt to orient and position the ramp-like posterior
probability in Figure 5.21(b) so as to assign a posterior probability that is near zero to the points
xn having label qn = 0, and a posterior probability near one to the points xn having label qn = 1.
Essentially, the goal is to dis riminate between the lasses, without any dire t attempt to model
the lasses.
More generally, in a dis riminative approa h to lassi ation we are not restri ted to the lo-
gisti fun tion, or to any other fun tion that is derived from a generative model. Rather we an
hoose fun tions whose ontours appear to provide a natural hara terization of boundaries between
lasses. On the other hand, it may not always be apparent how to hoose su h fun tions, and in
su h ases we may prefer to take advantage of the generative approa h, in whi h the boundaries
arise impli itly via Bayes rule. In general, both the dis riminative and the generative approa hes
are important tools to have in a modeling toolbox.

Mixture models revisited

Suppose we onsider a lassi ation problem in whi h none of the lass labels are present. Is this
a sensible problem to pose? What an one possibly learn from unlabeled data, parti ularly data
that are ompletely unlabeled?
5.2. STATISTICAL PROBLEMS 37

Q1 Q2 Q3 Q4 Q5 Q6

X1 X2 X3 X4 X5 X6

Figure 5.23: A model for partially labeled data in whi h the feature ve tors x2 ; x5 and x6 are
labeled and the other feature ve tors are unlabeled.

Consider Figure 5.22(a), where we have depi ted a typi al lassi ation problem with two
lasses. Now onsider Figure 5.22(b), where we have retained the feature ve tors xn , but erased the
labels qn. As this latter plot makes lear, although the labels are missing, there is still substantial
statisti al stru ture in the problem. Rather than solving a lassi ation problem, we an solve
a lustering problem, making expli it the fa t that the data appear to fall into two lusters and
assigning feature ve tors to lusters.
In fa t we have already solved this problem. The mixture model approa h to density estimation
dis ussed in Se tion 5.2.1 treats the density in terms of a set of underlying \subpopulations" labeled
by a latent variable Z . The inferential al ulation p(zn j xn ) given in Eq. (5.52) expli itly al ulates
the probability that the feature ve tor xn belongs to ea h of the subpopulations.
The relationship between lassi ation and mixture models is also lari ed by omparing the
\generative" graphi al model in Figure 5.19(b) and the mixture model in Figure 5.9(a). These
are the same graphi al model|the only di eren e is the shading, orresponding to the assumption
that the labels Qn are observed in lassi ation whereas the latent variables Zn are unobserved
in mixture modeling. In the setting of unlabeled data the generative lassi ation model be omes
identi al to a mixture model.
In a more general setting we may have a \partially labeled" ase in whi h the labels Qn are
observed for some data points and unobserved for other data points. This situation is represented
graphi ally in Figure 5.23. We will be able to treat the problem of estimation in this ase using the
EM algorithm; indeed this \partially labeled" ase requires no additional ma hinery beyond that
already required for the mixture model.
It is ommon to refer to lassi ation and regression models as \supervised learning" models and
to refer to density estimation models as \unsupervised learning" models. In the omparison between
mixture models and lassi ation models just dis ussed, the distin tion refers to the observation
of the labels Qn ; one says that the labels in lassi ation are provided by a \supervisor." While
this terminology an be useful in making broad distin tions between models, it is our view that
the terminology does not re e t a fundamental underlying distin tion and we will tend to avoid its
use in this book. It is our feeling that many models are neither entirely \supervised" nor entirely
38 CHAPTER 5. STATISTICAL CONCEPTS

\unsupervised," and invoking the distin tion often for es us to group together methods that have
little in ommon as well as to separate methods that are losely related. We feel that a better way
to understand relationships between models is to make them expli it as graphs. Models an then
be ompared in terms of graphi al features su h as whi h variables are onsidered latent and whi h
observed, the dire tionalities of ar s that are used to represent onditional relationships, and the
presen e or absen e of parti ular stru tural motifs.

Remarks
We have already indi ated a relationship between mixture models and lassi ation, but there are
other roles for mixture models in the lassi ation setting. In parti ular, we an use mixtures
as lass- onditional densities in the generative approa h to lassi ation, just as we used mixture
models in the density estimation setting to extend the range of models that we onsidered. Also,
in the ontext of the dis riminative approa h to lassi ation, we an use onditional mixtures to
represent the posterior probability p(q j x), breaking this fun tion into overlapping pie es, mu h as
we did with the onditional mean in the ase of regression.
Similarly, nonparametri methods have many roles to play in lassi ation models. We an
either extend the generative approa h to allow nonparametri estimates of the lass- onditional
densities, or extend the dis riminative approa h to allow nonparametri estimates of the posterior
probability.
Finally, there are on e again Bayesian approa hes in all of these ases. From a graphi al point
of view, these Bayesian approa hes essentially involve making the parameters expli it as nodes, and
using hyperparameters to express prior probability distributions on these nodes.

5.3 Model sele tion and model averaging

Thus far we have assumed that a spe i model has been hosen in advan e and we have fo used on
representing the model graphi ally and estimating its parameters. In some ases this assumption is
reasonable|the model is determined by the problem and there is no need to onsider data-driven
approa hes to hoosing the model. More ommonly, however, we wish to use the data to make
informed hoi es regarding the model. We present a brief dis ussion of this problem|known as the
model sele tion problem|in this se tion, anti ipating our more detailed presentation in Chapter 26.
We onsider a lass M of possible models, letting m 2 M denote a spe i model in this family.
We also augment our earlier notation to in lude expli it referen e to the model; thus, p(x j ; m)
refers to the probability model for the random variable X , given a spe i model and a spe i
hoi e of parameters for that model.4 Also, in the Bayesian approa h, p( j m) refers to the prior
probability that we atta h to the parameters , and p( j x; m) refers to the orresponding posterior.
We wish to develop methods for hoosing m based on the data x.
Let us begin with the Bayesian approa h. Re all that unknowns are treated as random variables
in the Bayesian approa h; thus we introdu e a random variable M to denote the model. The range
4
For simpli ity we use the same notation  to represent the parameters in ea h of the models; in general we ould
allow the parameterization to vary with m.
5.3. MODEL SELECTION AND MODEL AVERAGING 39

of M is M, and m denotes a realization of M . The goal of Bayesian analysis is to al ulate the

posterior probability of M , onditioning on the data x:
p(x j m)p(m)
p(m j x) = : (5.62)
p(x)
Note two important features of this equation. First, as in the ase of parameter estimation, we
require a prior probability; in parti ular, we need to spe ify the prior probability p(m) of the model
m. Se ond, note the absen e of expli it mention of the parameter . The probabilities needed for
Bayesian model sele tion are marginal probabilities.
Let us onsider this latter issue in more detail. The al ulation of the posterior probability
in Eq. (5.62) requires the probability p(x j m), a onditional probability that is referred to as the
marginal likelihood. We ompute the marginal likelihood from the likelihood by integrating over
the parameters:
Z
p(x j m) = p(x;  j m)d (5.63)
Z
= p(x j ; m)p( j m)d; (5.64)
where the prior probability p( j m) plays the role of a weighting fun tion. Multiplying the marginal
likelihood by the prior probability p(m) yields the desired posterior p(m j x), up to the normalization
fa tor p(x).
If we wish to use the posterior to sele t a model, then we must ollapse the posterior to a point.
As in the ase of parameter estimation, various possibilities present themselves; in parti ular, a
popular approa h is to pi k the model that maximizes the posterior probability. An advantage of
this approa h is that it obviates the need to al ulate the normalization onstant p(x).
More generally, however, the Bayesian approa h aims to use the entire posterior. To illustrate
the use of the model posterior, let us again onsider the problem of predi tion. Taking Xnew to be
onditionally independent of X , given  and m, we have:
Z Z
p(xnew j x) = p(xnew ; ; m j x)ddm (5.65)
Z Z
= p(xnew j ; m)p(; m j x)ddm (5.66)
Z Z
= p(xnew j ; m)p( j x; m)p(m j x)ddm: (5.67)
From this latter equation, we see that a full Bayesian approa h to predi tion requires two posterior
probabilities: the model posterior p(m j x) from Eq. (5.62) and the parameter posterior p( j x; m)
from Eq. (5.1). These posteriors an be viewed as \weights" for the predi tion p(xnew j ; m); the
total predi tion an be viewed as a \weighted predi tion." This approa h to predi tion is referred
to as model averaging.
It should be a knowledged that it is a rare ir umstan e in whi h the integrals in Eq. (5.64)
and Eq. (5.67) an be done exa tly, and Bayesian model averaging and model sele tion generally
involve making approximations. We will dis uss some of these approximations in Chapter 26.
40 CHAPTER 5. STATISTICAL CONCEPTS

Frequentist approa hes to model sele tion avoid the use of prior probabilities and Bayes rule.
Rather, one onsiders various model sele tion pro edures, and evaluates these pro edures in terms of
various frequentist riteria. For example, one ould onsider a s enario in whi h the true probability
density is assumed to lie within the lass M, and ask that a model sele tion pro edure pi k the
true model with high frequen y. Alternatively, one ould ask that the pro edure sele t the \best"
model in M, where \best" is de ned in terms of a measure su h as the Kullba k-Leibler divergen e
between a model and the true probability density.
It is important to understand that maximum likelihood itself annot be used as a model sele tion
pro edure. Augmenting a model with additional parameters annot de rease the likelihood, and
thus maximum likelihood will prefer more omplex models. More omplex models may of ourse
be better than simpler models, if they provide a ess to probability densities that are signi antly
loser to the true density, but at some point there are diminishing returns and more omplex
models prin ipally provide a ess to additional poor models. The fa t that we have to estimate
parameters implies that with some probability we will sele t one of the poor models. Thus the
\varian e" introdu ed by the parameter estimation pro ess an lead to poorer performan e with a
more omplex model. Maximum likelihood is unable to address this \over tting" phenomenon.
One approa h to frequentist model sele tion is to \ orre t" maximum likelihood to a ount
for the varian e due to parameter estimation. The AIC method to be dis ussed in Chapter 26
exempli es this approa h. An alternative approa h, also dis ussed in Chapter 26, is the ross-
validation idea, in whi h the data are partitioned in subsets, with one subset used to t parameters
for various models, and another subset used to evaluate the resulting models.

5.4 Appendix A

In this se tion we al ulate the posterior density of  in the univariate Gaussian density estimation
problem. Re all that the joint probability of x and  is given by:
( N )
1 1 X 1 1
p(x; ) = exp (x )2 expf ( 0 )2 g; (5.68)
(22 )N=2 22 n=1 n 2
(2 ) 1 = 2 2 2
and the problem is to normalize this joint probability.
Let us fo us on the terms involving , treating all other terms as \ onstants" and dropping
them throughout. We have:
( )
N
1 X
1 2

p(x; ) / exp x2 2xn  + 2  20  + 20 (5.69)
22 n=1 n 2 2
( N   )
1X 1 2 1 2
2
  2
= exp x 2xn  +  + 2 2 0 + 0 (5.70)
2 n=1 2 n  N N N
(   )
N x
1X 1 1  
= exp 2 + 2 2 2 2 + 02  + C
n
(5.71)
2 n=1  N  N
5.5. HISTORICAL REMARKS AND BIBLIOGRAPHY 41
     
1 N 1 2 N x 
/ exp +  2 2 + 20  (5.72)
2 2  2  
(  "   1  #)
1 N 1 N 1 N x
  0
/ exp + 2 2 2 + 2 +  (5.73)
2 2  2   2  2
 
1  2 
= exp  2~ ; (5.74)
2~2
where   1
N 1
~ 2 = 2 + 2 (5.75)
 
and
N=2 1= 2
~ = x + ; (5.76)
N=2 + 1= 2 N=2 + 1= 2 0
This identi es the posterior as a Gaussian distribution with mean ~ and varian e ~ 2 .

5.5 Histori al remarks and bibliography