UNIT-4

THE NORMAL DISTRIBUTION AND ITS GEOMETRIC INTERPRETATIONS:

Normal Distribution is an important concept in statistics and the backbone of

Machine Learning.

As discovered by Carl Friedrich Gauss, Normal Distribution/Gaussian

Distribution is a continuous probability distribution. It has a bell-shaped curve that
is symmetrical from the mean point to both halves of the curve.

Mathematical Definition:

A continuous random variable “x” is said to follow a normal distribution with

parameter μ(mean) and σ(standard deviation), if it’s probability density function is
given by,

1
We can draw a connection between probabilistic and geometric models by
considering probability distributions defined over Euclidean spaces.
The most common such distributions are normal distributions, also called
Gaussians; here recalls the most important facts concerning univariate and
multivariate normal distributions.

We start by considering the univariate, two-class case. Suppose the values of x ∈ R

follow a mixture model: i.e., each class has its own probability distribution (a
component of the mixture model).
We will assume a Gaussian mixture model, which means that the components of
the mixture are both Gaussians. We thus have

The univariate normal or Gaussian distribution has the following probability

density function:

The distribution has two parameters:

1) μ, which is the mean or expected value, as well as the median (i.e., the point
where the area under the density function is split in half ) and the mode (i.e.,
the point where the density function reaches its maximum); and
2
 2) σ, which is the standard deviation and determines the width of the bell-
shaped curve.
z = (x − μ)/σ is the z-score associated with x; it measures the number of standard
deviations between x and the mean (it has itself mean 0 and standard deviation 1). It
follows that P(x|μ,σ) = 1 σP(z|0, 1), where P(z|0, 1) denotes the standard normal
distribution.
Note: In other words, any normal distribution can be obtained from the standard
normal distribution by scaling the x-axis with a factor σ, scaling the y-axis with a
factor 1/σ (so the area under the curve remains 1), and translating the origin over μ

Figure 9.2. If positive examples are drawn from a Gaussian with mean and standard
deviation 1 and negatives from a Gaussian with mean and standard deviation 2,
then the two distributions cross at x = ±1.85. This means that the maximum-
likelihood region for positives is the closed interval [−1.85, 1.85], and hence the
negative region is non-contiguous.

3
If the features are uncorrelated and have the same variance,
maximumlikelihood classification leads to the basic linear classifier,
whose decision boundary is orthogonal to the line connecting the means. (

As long as the per-class covariance matrices are identical, the Bayes-optimal

decision boundary is linear – if we were to decorrelate the features by rotation and
scaling, we would again obtain the basic linear classifier.

4
Unequal covariance matrices lead to hyperbolic decision boundaries, which means
that one of the decision regions is non-contiguous.
Non-contiguous decision regions can also occur in higher-dimensional spaces.
Notice the circles and ellipses in Figure 9.3, which provide a visual summary of the

obtain the interval [μ1 ⊕−σ1 ⊕,μ1 ⊕+σ1 ⊕] – i.e., one standard deviation around
covariance matrix. By projecting the shape for the positive class down to x-axis we

the mean – and similar for the negative class and the y-axis.

Three cases can be distinguished:

(i) both x and y standard deviations are equal and the correlation coefficient is
zero, in which case the shape is a circle;
(ii) the standard deviations are different and the correlation coefficient is zero,
which means the shape is an ellipse parallel to the axis with the largest
standard deviation;
(iii) the correlation coefficient is non-zero: the orientation of the ellipse gives the
sign of the correlation coefficient, and its width varies with the magnitude of
the correlation coefficient.

Note: for uncorrelated, unit-variance Gaussian features, the basic linear

classifier is Bayes-optimal

the standard normal distribution translates Euclidean distances into probabilities:

5
PROBABILISTIC MODELS FOR CATEGORICAL DATA:

 A Bernoulli trial is an experiment that results in two

outcomes: success and failure.
 One example of a Bernoulli trial is the coin tossing experiment, which results
in heads or tails.
 In a Bernoulli trial we define the probability of success and probability of
failure as follows:

 The Binomial Distribution arises when counting the number of successes S

in n independent Bernoulli trials with the same parameter θ. It is described by

 The categorical distribution generalises the Bernoulli distribution to k ≥ 2

outcomes. The parameter of the distribution is a k-vector

 the multinomial distribution tabulates the outcomes of n independent and

identically distributed (i.i.d.) categorical trials. That is, X = X1,...,Xk is a
k-vector of integer counts, and

6
Using A Naive Bayes Model For Classification :
 Assume that we have chosen one of the possible distributions to model our data X.
 In a classification context, we furthermore assume that the distribution depends on the
class, so that P(X|Y = spam) and P(X|Y = ham) are different distributions.
 The more different these two distributions are, the more useful the features X are for
classification.
 Thus, for a specific e-mail x we calculate both P(X = x|Y = spam) and P(X = x|Y = ham),
and apply one of several possible decision rules:

Working of Naïve Bayes' Classifier:

Working of Naïve Bayes' Classifier can be understood with the help of the below
example:

Suppose we have a dataset of weather conditions and corresponding target

variable "Play". So using this dataset we need to decide that whether we should play
or not on a particular day according to the weather conditions. So to solve this
problem, we need to follow the below steps:

1. Convert the given dataset into frequency tables.

2. Generate Likelihood table by finding the probabilities of given features.
3. Now, use Bayes theorem to calculate the posterior probability.

Example 9.4 (Prediction using a naive Bayes model):

7
Suppose our vocabulary contains three words a, b and c, and we use a multivariate Bernoulli
model for our e-mails, with parameters θ⊕ = (0.5, 0.67, 0.33) θ = (0.67,0.33,0.33)

This means, for example, that the presence of b is twice as likely in spam (+), compared with
ham.

The e-mail to be classified contains words a and b but not c, and hence is described by the bit
vector x = (1, 1, 0).

We obtain likelihoods

P(x|⊕) = 0.5·0.67·(1−0.33) = 0.222 P(x|) = 0.67·0.33·(1−0.33) = 0.148

The ML classification of x is thus spam.

In the case of two classes it is often convenient to work with likelihood ratios and odds.

The likelihood ratio can be calculated as:

This means that the MAP classification of x is also spam if the prior odds are more than 2/3, but
ham if they are less than that.

Advantages of Naïve Bayes Classifier:

o Naïve Bayes is one of the fast and easy ML algorithms to predict a class of
datasets.
o It can be used for Binary as well as Multi-class Classifications.
o It performs well in Multi-class predictions as compared to the other Algorithms.
o It is the most popular choice for text classification problems.

Applications of Naïve Bayes Classifier:

o It is used for Credit Scoring.

8
 o It is used in medical data classification.
o It can be used in real-time predictions because Naïve Bayes Classifier is an
eager learner.
o It is used in Text classification such as Spam filtering and Sentiment
analysis.

Types of Naïve Bayes Model:

There are three types of Naive Bayes Model, which are given below:

o Gaussian: The Gaussian model assumes that features follow a normal

distribution. This means if predictors take continuous values instead of
discrete, then the model assumes that these values are sampled from the
Gaussian distribution.
o Multinomial: The Multinomial Naïve Bayes classifier is used when the data is
multinomial distributed. It is primarily used for document classification
problems, it means a particular document belongs to which category such as
Sports, Politics, education, etc.
The classifier uses the frequency of words for the predictors.
o Bernoulli: The Bernoulli classifier works similar to the Multinomial classifier,
but the predictor variables are the independent Booleans variables. Such as if a
particular word is present or not in a document. This model is also famous for
document classification tasks.

DISCRIMINATIVE LEARNING
 Machine learning models can be classified into two types of models
– Discriminative and Generative models.
 Discriminative models, also referred to as conditional models, are a class of logistical models used
for classification or regression.
 Typical discriminative models include logistic regression (LR), conditional random fields (CRFs)
(specified over an undirected graph), decision trees, and many others.
9
  In simple words, a discriminative model makes predictions on the unseen data
based on conditional probability and can be used either for classification or
regression problem statements.
 On the contrary, a generative model focuses on the distribution of a dataset to
return a probability for a given example.
 Naive Bayes models are generative: after training they can be used to generate
data.
 The most commonly used discriminative models: logistic regression.
 In generative models the decision boundary is a by-product of modelling the
distributions of each class, logistic regression models the decision boundary
directly.

For example, if the classes are overlapping then logistic regression will tend to
locate the decision boundary in an area where classes are maximally overlapping,
regardless of the ‘shapes’ of the samples of each class. This results in decision
boundaries that are noticeably different from those learned by generative classifiers

The logistic regression model is simply given by

Assuming the class labels are y = 1 for positives and y = 0 for negatives, this defines
a Bernoulli distribution for each training example:

It is important to note that the parameters of these Bernoulli distributions are linked
through w and t, and consequently there is one parameter for every feature
dimension, rather than for every training instance.

10
This is called conditional likelihood to stress that it gives us the conditional
probability P(yi |xi) rather than P(xi) as in a generative model.

the logarithm of the likelihood function is easier to work with:

PROBABILISTIC MODELS WITH HIDDEN VARIABLES:

1) Expectation-Maximisation (EM)

2) Gaussian Mixture Models

Gaussian Mixture Models :

However, suppose that we have the same data, but without target labels.

This requires unsupervised learning, Suppose that the different classes each come from their
own Gaussian distribution.

This is known as multi-modal data, since there is one distribution (mode) for each different
class.

We can’t fit one Gaussian to the data, because it doesn’t look Gaussian overall.

There is, however, something we can do. If we know how many classes there are in the data,
then we can try to estimate the parameters for that many Gaussians, all at once.
11
If we don’t know, then we can try different numbers and see which one works best.

It is perfectly possible to use any other probability distribution instead of a Gaussian, but
Gaussians are by far the most common choice.

Then the output for any particular datapoint that is input to the algorithm will be the sum of the
values expected by all of the M Gaussians:

The figure also gives you some idea of how to use the mixture model once it has been created.
The probability that input xi belongs to class m can be written as (where a hat on a variable (ˆ·)
means that we are estimating the value of that variable):

12
The problem is how to choose the weights αm. The common approach is to aim for the
maximum likelihood solution. In fact, it is common to compute the log likelihood and then to
maximise that; it is guaranteed to be negative, since probabilities are all less than 1, and the
logarithm spreads out the values, making the optimisation more effective.

The algorithm that is used is an example of a very general one known as the expectation
maximisation (or more compactly, EM) algorithm.

The Expectation-Maximisation (EM) Algorithm:

The basic idea of the EM algorithm is that sometimes it is easier to add extra variables that are
not actually known (called hidden or latent variables) and then to maximise the function over
those variables.

This might seem to be making a problem much more complicated than it needs to be, but it
turns out for many problems that it makes finding the solution significantly easier.

In order to see how it works, we will consider the simplest interesting case of the Gaussian
mixture model: a combination of just two Gaussian mixtures.

The assumption now is that sample from that Gaussian.

If the probability of picking Gaussian one is p, then the entire model looks like this (where N(μ,
σ 2 ) specifies a Gaussian distribution with mean μ and standard deviation σ):

13
Finding the maximum likelihood solution (actually the maximum log likelihood) to this
problem is then a case of computing the sum of the logarithm of Equation over all of the
training data, and differentiating it, which would be rather difficult.

Fortunately, there is a way around it.

The key insight that we need is that if we knew which of the two Gaussian components the
datapoint came from, then the computation would be easy.

The mean and standard deviation for each component could be computed from the datapoints
that belong to that component, and there would not be a problem.

Although we don’t know which component each datapoint came from, we can pretend we do,
by introducing a new variable f.

If f = 0 then the data came from Gaussian one, if f = 1 then it came from Gaussian two.

This is the typical initial step of an EM algorithm:

adding latent variables. Now we just need to work out how to optimise over them. This is the
time when the reason for the algorithm being called expectation-maximisation becomes clear.

We don’t know much about variable f , but we can compute its expectation from the data:

14
Computing the value of this expectation is known as the E-step.

Then this estimate of the expectation is maximised over the model parameters (the parameters
of the two Gaussians and the mixing parameter π), the M-step.

This requires differentiating the expectation with respect to each of the model parameters.
These two steps are simply iterated until the algorithm converges.

The trick with applying EM algorithms to problems is in identifying the correct latent variables
to include, and then simply working through the steps.

They are very powerful methods for a wide variety of statistical learning problems.

15