Module -3 : Estimation of Unknown Probability Density Functions:

Topics:
Maximum Likelihood Parameter Estimation
Maximum a Posteriori Probability estimation
Bayesian Inference
Maximum Entropy Estimation
Mixture Models
Naive-Bayes Classifier
The Nearest Neighbour Rule.

Introduction : Estimation of Unknown PDF

Many different distributions for random variables have parameters: the numbers that you
provide as input when you define a random variable. So far when we were working with
random variables, we either were explicitly told the values of the parameters or, we could
divine the values by understanding the process that was generating the random variables.
What if we don’t know the values of the parameters and we can’t estimate them from our own
expert knowledge?
What if instead of knowing the random variables, we have a lot of examples of data generated
with the same underlying distribution?
We learn formal ways of estimating parameters from data. These ideas are critical for artificial
intelligence. Almost all modern machine learning algorithms work like this: (1) specify a
probabilistic model that has parameters. (2) Learn the value of those parameters from data.
Parameters:
Given a model, the parameters are the numbers that yield the actual distribution. In the case of
a Bernoulli random variable, the single parameter was the value p. In the case of a Uniform
random variable, the parameters are the a and b values that define the min and max value. Here
is a list of random variables and the corresponding parameters

The pdf has to be estimated from the available data in many problems. There are various ways
to approach the problem.
Sometimes we may know the type of the pdf ([Link], Rayleigh), but we do not
know certain parameters, such as the mean values or the variances.
In contrast, in other cases, we may not have information about the type of the pdf but we may
know certain statistical parameters, such as the mean value and the variance.
Depending on the available information, different approaches can be adopted for this.
Let us Know about Probability Density Function.
The Probability Density Function(PDF) defines the probability function representing the
density of a continuous random variable lying between a specific range of values. the
probability density function produces the likelihood of values of the continuous random
variable. Sometimes it is also called a probability distribution function or just a probability
function. the actual truth is PDF (probability density function ) is defined for continuous
random variables, whereas PMF (probability mass function) is defined for discrete random
variables.
In the case of a continuous random variable, the probability taken by X on some given value x
is always 0. In this case, if we find P(X = x), it does not work. Instead of this, we must calculate
the probability of X lying in an interval (a, b). Now, we have to figure it for P(a< X< b), and
we can calculate this using the formula of PDF. The Probability density function formula is
given as,

,where
P(a ≤ X ≤ b) = P(a < X ≤ b) = P(a ≤ X < b) = P(a < X < b).

Probability Density Function Graph

The probability density function is defined as an integral of the density of the variable density
over a given range. It is denoted by f (x). This function is positive or non-negative at any point
of the graph, and the integral, more specifically the definite integral of PDF over the entire
space is always equal to one. The graph of PDFs typically resembles a bell curve, with the
probability of the outcomes below the curve. The below figure depicts the graph of a probability
density function for a continuous random variable x with function f(x).
Maximum Likelihood Parameter Estimation:
● Maximum likelihood estimation(ML Estimation, MLE) is a powerful parametric
estimation method commonly used in statistics fields.
● The idea in MLE is to estimate the parameter of a model where given data is likely to
be obtained.
● Consider the given formula of bayes theorem ,in which we have Likelihood ,Posterior,
Prior and evidence.

p ( x |  j ) P ( j )
P( j | x) =
p ( x)
●
● In statistical pattern recognition, statistical features of a given training sample are
extracted and used to form a recognition process.
● Pattern Recognition goal is equivalent to determining a discriminator function of
multiple categories.
● To generate a well-performed discriminator function, several criteria like maximum a
posteriori probability decision rule, minimum discriminator error decision rule and
Bayesian decision rule. etc will be used.
● Maximum likelihood estimation (MLE) is a technique used for estimating the
parameters of a given distribution, using some observed data.
● For example, if a population is known to follow a normal distribution but
the mean and variance are unknown.
 ● MLE can be used to estimate them using a limited sample of the population, by finding
particular values of the mean and variance so that the observation is the most likely
result to have occurred.
MLE is useful in a variety of contexts, ranging from econometrics to MRIs to satellite
imaging. It is also related to Bayesian statistics.
Formal definition
● Let x1, x2, . . . , xN be random samples drawn from pdf p(x ;Ɵ ).
● We form the joint pdf p(X;Ɵ ), where X {x1, . . . , xN } is the set of the samples.
Assuming statistical independence between the different samples, we have

● This is a function of Ɵ, and it is also known as the likelihood function of Ɵ with

respect to X.
● The maximum likelihood (ML) method estimates Ɵ so that the likelihood function takes
its maximum value, that is,

It can be written as
● In the maximum likelihood estimation method, we find the value of parameter theta
which maximizes the likelihood value. The optimal parameter theta with MLE method
is written as,

● When a parametric model q(x; theta) is derivable by theta, the following equation is
true. This equation is called a likelihood equation. It is a necessary condition of the
maximum likelihood estimation answer but not a sufficient condition.

Examples :
● A (possibly unfair) coin is flipped 100 times, and 61 heads are observed. The coin either
has probability 1/3,1/2 or 2/3 of flipping a head each time it is flipped. Which of the
three is the MLE
Example 2:
I have a bag that contains 3 balls. Each ball is either red or blue, but I have no information in
addition to this. Thus, the number of blue balls, call it θ, might be 0, 1, 2, or 3. I am allowed to
choose 4 balls at random from the bag with replacement. We define the random variables X1,
X2, X3, and X4 as follows.

Note that Xi's are i.i.d. and Xi∼Bernoulli(θ/3). After doing my experiment, I observe the
following values for Xi's.
x1=1,x2=0,x3=1,x4=1. Thus, I observe 3 blue balls and 1 red balls.
For each possible value of θ, find the probability of the observed sample,
(x1,x2,x3,x4)=(1,0,1,1). For which value of θ is the probability of the observed sample is the
largest.
The probability of observed sample for θ=0 and θ=3 is zero. This makes sense because our
sample included both red and blue balls.
From the table we see that the probability of the observed data is maximized for θ=2.
This means that the observed data is most likely to occur for θ=2. For this reason, we may
choose θ^=2 as our estimate of θ.
This is called the maximum likelihood estimate (MLE) of θ.
Maximum a Posteriori Probability Estimation:
• MLE is powerful when you have enough data.
• However, it doesn’t work well when observed data size is small.
• For example, if Liverpool only had 2 matches and they won the 2 matches, then the
estimated value of θ by MLE is 2/2 = 1. ( θ - Winning probability of Liverpool )
• It means that the estimation says Liverpool wins 100%, which is an unrealistic
estimation. MAP can help deal with this issue.
Why Is Maximum a Posteriori (MAP) Estimation the Ideal Estimation Method for Smaller
Datasets.
● It is ideal because it takes into account prior knowledge of an event. MLE does not and
is prone to overfitting.
● For this reason, MAP is considered a regularization of MLE.
● Adding the prior probability information reduces the overdependence on the observed
data for parameter estimation.
 ● MAP usually comes up in Bayesian setting. As the name suggests, it works on a
posterior distribution, not only the likelihood.
● Recall the Bayes’ rule, we could get the posterior as a product of likelihood and prior:
● P(Ɵ|X) =P(X|Ɵ) P(Ɵ) / P(X) ~ P(X|Ɵ) P(Ɵ)
Example :
Assume that we have a prior knowledge that Liverpool’s winning percentage for the past few
seasons were around 50%. Then, without the data from this season, we already have somewhat
idea of potential value of θ. Based (only) on the prior knowledge, the value of θ is most likely
to be 0.5, and less likely to be 0 or 1. On the other words, the probability of θ=0.5 is higher
than θ=0 or 1. Calling this as the prior probability P(θ).

Then, having the observed data D (30 win out of 38 matches) from this season, we can update
this P(θ) which is based only on the prior knowledge. The updated probability of θ given D is
expressed as P(θ|D) and called the posterior probability.
Now, we want to know the best guess of θ considering both our prior knowledge and the
observed data. It means maximising P(θ|D) and it’s the MAP estimation. Is given by
argmaxƟ P(Ɵ |D )

We need to use Bayes’ theorem : P(Ɵ|D)= P(D|Ɵ) P (Ɵ ) / P(D)

There’s P(D) in the equation, but P(D) is independent to the value of θ. Since we’re only
interested in finding θ maximizing P(θ|D), we can ignore P(D) in our maximization
argmaxƟ P(Ɵ |D ) = argmaxƟ P(D|Ɵ) P(Ɵ),

The equation above means that the maximization of the posterior probability P(θ|D) with
respect to θ is equal to the maximization of the product of Likelihood P(D|θ) and Prior
probability P(θ) with respect to θ.
Important method for calculating conditional probabilities is given by Bayes's formula.
The formula is based on the expression P(B) = P(B|A)P(A) + P(B|Ac)P(Ac), which simply states
that the probability of event B is the sum of the conditional probabilities of event B given that
event A has or has not occurred.
For independent events A and B, this is equal to
P(B)P(A) + P(B)P(Ac) = P(B)(P(A) + P(Ac)) = P(B)(1) = P(B), since the probability of an event
and its complement must always sum to 1.
The Bays formula in terms of A and B is given below.
P(A|B) = P(B|A) P(A) divides or / P(B|A)P(A) + P(B|Ac) P(Ac)
Example 2:
Suppose a voter poll is taken in three states. In state A, 50% of voters support the liberal
candidate, in state B, 60% of the voters support the liberal candidate, and in state C, 35% of
the voters support the liberal candidate. Of the total population of the three states, 40% live in
state A, 25% live in state B, and 35% live in state C. Given that a voter supports the liberal
candidate, what is the probability that she lives in state B.
Solution :
By Bayes's formula,
P(Voter lives in state B|Voter supports liberal candidate) =
P(Voter supports liberal candidate|Voter lives in state B)P(Voter lives in state B)/
(P(Voter supports lib. cand.|Voter lives in state A)P(Voter lives in state A) +
P(Voter supports lib. cand.|Voter lives in state B)P(Voter lives in state B) +
P(Voter supports lib. cand.|Voter lives in state C)P(Voter lives in state C))
= (0.60)*(0.25)/((0.50)*(0.40) + (0.60)*(0.25) + (0.35)*(0.35))
= (0.15)/(0.20 + 0.15 + 0.1225) = 0.15/0.4725 = 0.3175.
If we replace the likelihood in the MLE formula with the posterior, we get
P(Ɵ|X) =P(X|Ɵ) P(Ɵ) / P(X) ~ P(X|Ɵ) P(Ɵ)

Bayesian Inference:
Bayes’ theorem can be expressed through the following mathematical equation
P(A|B)= P(B|A) P(A) // P(B)
where A is an event and B is evidence. So, P(A) is the prior probability of event A and P(B) is
evidence of event B. Hence, P(B|A) is the likelihood. The denominator is a normalizing
constant. So, Bayes’ Theorem gives us the probability of an event based on our prior knowledge
of the conditions that might be related to the event and updates that conditional probability
when some new information or evidence comes up.

3 components of the Bayes’ theorem

• Prior
To express our beliefs about an unknown parameter θ we choose a distribution function called
the prior distribution. This distribution is chosen before we see any data or run any experiment.
• Likelihood
The probability of observing the actual data that we collected (head or tail), conditioned on a
value of the parameter theta (fairness of coin) and can be expressed as follows-

Posterior
Result or output of the Bayes’ Theorem. A posterior probability is the revised or updated
probability of an event occurring after taking into consideration new information. We calculate
the posterior probability p(θ|X) i.e., how probable is our hypothesis about θ given the observed
evidence.
In the Bayesian framework, we treat the unknown quantity, Θ, as a random variable. More
specifically, we assume that we have some initial guess about the distribution of Θ. This
distribution is called the prior distribution. After observing some data, we update the
distribution of Θ (based on the observed data). This step is usually done using Bayes' Rule.
That is why this approach is called the Bayesian approach.
Inference example using Frequentist vs Bayesian approach: Suppose my friend challenged me
to take part in a bet where I need to predict if a particular coin is fair or not. She told me “Well;
this coin turned up ‘Head’ 70% of the time when I flipped it several times. Now I am giving
you a chance to flip the coin 5 times and then you have to place your bet.” Now I flipped the
coin 5 times and Head came up twice and tail came up thrice. At first, I thought like a
frequentist.
So, θ is an unknown parameter which is a representation of fairness of the coin and can be
defined as
θ = {fair, loaded}.
Additionally, I assumed that the outcome variable X (whether head or tail) follows Binomial
distribution with the following functional representation

,
Now in our case n=5.
Now my likelihood function will
Now, I saw that head came up twice, so my X =[Link] X=2, f (θ|X=2) = 0.31 if θ =fair
= 0.13 if θ =loaded.
Therefore, using the frequentist approach I can conclude that maximum likelihood i.e., MLE
(theta hat) = fair.
MLE(Ɵ)=fair
Bayesian inference tell us the uncertainty in my prediction i.e., P(θ|X=2). The Bayesian
inference allows us to incorporate our knowledge/information about the unknown parameter θ
even before looking at any data. Here, suppose I know my friend pretty well and I can say with
90 % probability that she has given me a loaded coin.
Therefore, my prior P(loaded)=0.9. I can now update my prior belief with data and get the
posterior probability using Bayes’ Theorem

Numerator calculation will be as follows-

The denominator is a constant and can be calculated as the expression below. Please note that
we are here basically summing up the expression over all possible values of θ which is only 2
in this case i.e., fair or loaded.

Hence, after replacing X with 2 we can calculate the Bayesian probability of the coin being
loaded or fair.
Maximum Entropy Estimation.
ENTROPY ;
Entropy is defined as the randomness or measuring the disorder of the information being
processed in Machine Learning. Further, in other words, we can say that entropy is the
machine learning metric that measures the unpredictability or impurity in the system.
The concept of entropy is known from Shannon’s information theory.
It is a measure of the uncertainty concerning an event, or it is a measure of randomness of the
messages (feature vectors in our case) occurring at the output of a system.
If p(x) is the density function, the associated entropy H is given by

Assume now that p(x) is unknown but we know a number of related constraints (mean value,
variance, etc.). The maximum entropy estimate of the unknown pdf is the one that maximizes
the entropy, subject to the given constraints. According to the principle of maximum entropy
an estimate corresponds to the distribution that exhibits the highest possible randomness,
subject to the available constraints.
The maximum entropy principle is a rule which allows us to choose a ‘best’ from a number
of different probability distributions that all express the current state of knowledge. It tells us
that the best choice is the one with maximum entropy.
This will be the system with the largest remaining uncertainty, and not adding any
extra biases into analysis. We know that all systems tend toward maximal entropy
configurations over time, so the likelihood that the system is accurately represented by the
maximum entropy distribution is higher than the likelihood.

Applying the Maximum Entropy Principle

Applying the maximum entropy principle to a physical problem typically involves

algebraically solving a series of equations for a number of unknowns.

For instance, consider the discrete case to find out the probability of a quantity taking on values
{a, b, c, d…}. The probabilities will all add up to one; so the first equation is

p(a) + p(b) + p(c) + p(d) = 1.

Specific Example

Consider the probability of people buying apples, bananas, or oranges. If they bought one of
three in a particular supermarket.
1 = P apple + P banana + P oranges.

The apples cost a dollar each, bananas two dollars, and oranges three dollars, and the average
price of fruit bought in the supermarket is $[Link]
$1.75 = $1.00 Papple + $2.00 Pbanana + $3.00 Poranges.
That is what we call our constraint equation. This might be all the information you have. But
with two equations and three unknowns, it simply isn’t enough information to come up with a
unique solution. That’s where the maximum entropy principle comes in. The maximum entropy
principle narrows down the space of all the potentially possible solutions—and there are lots
to the one best solution; the one with the highest entropy.

Let the entropy of the system S. We know that Shannon entropy is defined as:

The logarithm, logb(1/p(A)i) represents the information in state i, so when that is multiplied by
p(A)i for each i, we get a measure of uncertainty. So our third equation is this one:

S = P apple log2 (1/ P apple) + P banana log2 (1/ P banana) + P orange log2 (1/ P orange).

Algebra Steps

Multiplying every term in your first equation by -$1.00 means that, when you add it to your
constraint equation, Papple falls out and you have an equation with just Pbanana and Porange.

$0.75 = $1.00Pbanana + $2.00Porange

so we can write Pbanana in terms of Porange like this:

Pbanana = 0.75 – 2.00 Porange.

We could write Papple in terms of Porange if, instead of multiplying equation 1 by -$1.00 to begin
with, we multiply by -$2.00. Then we’d we have
-$0.25 = -$1.00Papple+ 1.00Porange
which gives us
Papple = Porange + 0.25.

Now we can go back to our third equation, the one where all the probabilities add up to
entropy, and we can write all the other probabilities in terms of Porange using the two equations
we just derived

S = (Porange + 0.25) log2 (1/ (Porange0.25)) + (0.75 – 2 Porange )log2 (1/ (0.75- 2Porange)) +
Porange log2 (1/ Porange)

Now all that remains is to find the value of Porange so that S is maximized. You can use any of
a number of methods to do this; finding the critical points of the function is one good one.
We find that entropy is maximized when Porange = (3.25 – √3.8125) /6, which is about 0.216.
Using the equations above, we can conclude that Papple is 0.466, and Pbanana is 0.318.

Mixture Models

Methods for learning probabilistic models took the form of simple distributions like Bernoulli
or Gaussian. But often the data we’re trying to model is much more complex. For instance, it
might be multimodal. This means that there are several different modes, or regions of high
probability mass, and regions of smaller probability mass in between.

For instance, suppose we’ve collected the high temperatures for every day in March 2014 for
both Toronto and Miami, Florida, but forgot to write down which city is associated with each
temperature. The values are plotted in Figure 1 we can see the distribution has two modes. In
this situation, we might model the data in terms of a mixture of several components, where
each component has a simple parametric form (such as a Gaussian).

Fig 1: A histogram of daily high temperatures in ◦C for Toronto and Miami in March 2014.
The distribution clearly has two modes.

Each data point belongs to one of the components, and try to infer the distribution for each
component separately. In this example, the two mixture components should correspond to the
two cities. The model itself doesn’t know” anything about cities, Here the model is represented
and formulated in terms of latent variables, usually denoted z. These are variables which are
never observed and don’t know the correct values in advance. They are roughly analogous to
hidden units, in that the learning algorithm needs to figure out what they should represent,
without a human specifying it by hand. Variables which are always observed, or even
sometimes observed, are referred to as observables. In the above example, the city is the latent
variable and the temperature is observable.

In mixture models, the latent variable corresponds to the mixture component. It takes values
in a discrete set, which is denoted as {1………..k}. (For now, assume K is [Link] general,
a mixture model assumes the data are generated by the following process:
first we sample z, and then we sample the observables x from a distribution which depends
on z, i.e. p(z,x) = p(z) p(x | z)
In mixture models, p(z) is always a multinomial distribution. p(x | z) can take a variety of
parametric forms, We refer to such a model as a mixture of Gaussians.

Fig 2:An example of a univariate mixture of Gaussians model.

Figure shows an example of a mixture of Gaussians model with 2 components. It has
the following generative process:
[Link] probability 0.7, choose component 1, otherwise choose component 2.
2. If we chose component 1, then sample x from a Gaussian with mean 0 and standard
deviation 1.
[Link] we chose component 2, then sample x from a Gaussian with mean 6 and standard
deviation 2.

This can be written in a more compact mathematical notation:

z ~ Multinomial (0.7, 0.3)
x | z = 1 ~ Gaussian (0,1)
x | z = 2 ~ Gaussian (6,2)
For the general case,
z ~Multinomial(π)
x | z = k ~ Gaussian (µk, σk).
Here, π is a vector of probabilities (i.e. nonnegative values which sum to 1) known as the
mixing proportions. In general, the probability density function is computed (PDF) over x by
marginalizing out, or summing out, z:

The general problem of grouping data points into clusters, where data points in the same cluster
are more similar than data points in different clusters, is known as clustering. Learning a
mixture model is one approach to clustering, but we should mention that there are a number of
other approaches, most notably an algorithm called K-means.

Expectation-Maximization

What is the Expectation-Maximization (EM) AlgorithmIn most of the real-life problem

statements of Machine learning, it is very common that we have many relevant features available to
build our model but only a small portion of them are observable. Since we do not have the values for
the not observed (latent) variables, the Expectation-Maximization algorithm tries to use the existing
data to determine the optimum values for these variables and then finds the model parameters. It is
a latent variable model.
A latent variable model consists of observable variables along with unobservable variables.
Observed variables are those variables in the dataset that can be measured whereas unobserved
(latent/hidden) variables are inferred from the observed [Link] a mixture model, the
identity of the component that generated a given datapoint is a latent variable

• It can be used to find the local maximum likelihood (MLE) parameters

or maximum a posteriori (MAP) parameters for latent variables in a statistical or
mathematical model.
• It is used to predict these missing values in the dataset, provided we know the general
form of probability distribution associated with these latent variables.
 • In simple words, the basic idea behind the algorithm is to use the observable samples
of latent variables to predict the values of samples that are unobservable for learning.
This process is repeated until the convergence of the values occurs.
• Given a set of incomplete data, start with a set of initialized parameters.
• Expectation step (E – step): In this expectation step, by using the observed available
data of the dataset, we can try to estimate or guess the values of the missing data.
Finally, after this step, we get complete data having no missing values.
• Maximization step (M – step): Now, we have to use the complete data, which is
prepared in the expectation step, and update the parameters.
• Repeat step 2 and step 3 until we converge to our solution.

Let us understand the EM algorithm in a detailed manner:

• Initialization Step: In this step, we initialized the parameter values with a set of initial
values, then give the set of incomplete observed data to the system with the assumption
that the observed data comes from a specific model i.e, probability distribution.
• Expectation Step: In this step, by using the observed data to estimate or guess the
values of the missing or incomplete data. It is used to update the variables.
• Maximization Step: In this step, we use the complete data generated in
the “Expectation” step to update the values of the parameters i.e, update the
hypothesis.
Advantages and Disadvantages of EM algorithm

Advantages
• The basic two steps of the EM algorithm i.e, E-step and M-step are often pretty easy
for many of the machine learning problems in terms of implementation.
• The solution to the M-steps often exists in the closed-form.
• It is always guaranteed that the value of likelihood will increase after each iteration.

Disadvantages
• It has slow convergence.
• It converges to the local optimum only.
• It takes both forward and backward probabilities into account. This thing is in contrast
to that of numerical optimization which considers only forward probabilities.
Applications of EM Algorithm

The latent variable model has several real-life applications in Machine learning:

• Used to calculate the Gaussian density of a function, Helpful to fill in the missing
data during a sample.
• It finds plenty of use in different domains such as Natural Language Processing
(NLP), Computer Vision, etc.
• Used in image reconstruction in the field of Medicine and Structural
Engineering. Used for estimating the parameters of the Hidden Markov Model
(HMM) and also for some other mixed models like Gaussian Mixture Models, etc.
• Used for finding the values of latent variables.

EM for Mixture Models.

This is the basis of the ExpectationMaximization (E-M) algorithm. This algorithm is performed
two steps: computing the responsibilities, and applying the maximum likelihood update with
those responsibilities. More precisely, we apply the following algorithm: Repeat until
converged:

The Naive-Bayes Classifier

Naïve Bayes is a probabilistic machine learning algorithm based on the Bayes Theorem, used
in a wide variety of classification tasks.
Bayes Theorem
 Bayes’ Theorem is a simple mathematical formula used for calculating conditional
probabilities.
Conditional probability is a measure of the probability of an event occurring given that
another event has (by assumption, presumption, assertion, or evidence) occurred.
The formula is: —

Which tells us: how often A happens given that B happens, written P(A|B) also called posterior
probability, When we know: how often B happens given that A happens, written P(B|A) and
how likely A is on its own, written P(A) and how likely B is on its own, written P(B).
Assumptions Made by Naïve Bayes
The fundamental Naïve Bayes assumption is that each feature makes an:
• Independent
• Equal
contribution to the outcome.
Let us take an example to get some better intuition. Consider the car theft problem with
attributes Color, Type, Origin, and the target, Stolen can be either Yes or No.
Naïve Bayes Example

concerning our dataset, the concept of assumptions made by the algorithm can be understood
as:
 • We assume that no pair of features are dependent. For example, the color being ‘Red’
has nothing to do with the Type or the Origin of the car. Hence, the features are
assumed to be Independent.
• Secondly, each feature is given the same influence(or importance). For example,
knowing only Color and Type alone can’t predict the outcome perfectly. So none of
the attributes are irrelevant and assumed to be contributing Equally to the outcome.
Note: The assumptions made by Naïve Bayes are generally not correct in real-world situations.
The independence assumption is never correct but often works well in practice.
Here in our dataset, we need to classify whether the car is stolen, given the features of the
car. The columns represent these features and the rows represent individual entries. If we take
the first row of the dataset, we can observe that the car is stolen if the Color is Red, the Type
is Sports and the Origin is Domestic.
So we want to classify a Red Domestic SUV is getting stolen or not. Note that there is no
example of a Red Domestic SUV in our data set.
According to this example, Bayes theorem can be rewritten as:

P(y|x)= P(X|Y)P(Y) / P(X)

The variable y is the class variable(stolen?), which represents if the car is stolen or not given
the conditions. Variable X represents the parameters/features. X is given as,
X=(x1,x2,x3,…….,xn).
Here x1, x2…, xn represent the features, i.e they can be mapped to Colour, Type, and Origin.
By substituting for X and expanding using the chain rule we get,

P(y| x1,…..,xn) = P(x1|y) P (x2|y) ……..P(xn|y) P(y) // P(x1) P(x2)…..P(xn)

Now, you can obtain the values for each by looking at the dataset and substituting them into
the equation. For all entries in the dataset, the denominator does not change, it remains static.
Therefore, the denominator can be removed and proportionality can be injected.

In our case, the class variable(y) has only two outcomes, yes or no. There could be cases where
the classification could be multivariate. Therefore, we have to find the class variable(y) with
maximum probability.

using the above function, we can obtain the class, given the predictors/features.
The posterior probability P(y|X) can be calculated by first, creating a Frequency Table for
each attribute against the target. Then, molding the frequency tables to Likelihood Tables and
finally, use the Naïve Bayesian equation to calculate the posterior probability for each class.
The class with the highest posterior probability is the outcome of the prediction. Below are the
Frequency and likelihood tables for all three predictors.
Frequency and Likelihood tables of ‘Color’

Frequency and Likelihood tables of ‘Type’

Frequency and Likelihood tables of ‘Origin’

So in our example, we have 3 predictors X.

As per the equations discussed above, we can calculate the posterior probability P(Yes | X) as
:
Since 0.144 > 0.048, Which means given the features RED SUV and Domestic, our example
gets classified as ’NO’ the car is not stolen.

K-Nearest Neighbor(KNN)
o K-Nearest Neighbour is one of the simplest Machine Learning algorithms based on
Supervised Learning technique.
o K-NN algorithm assumes the similarity between the new case/data and available cases
and put the new case into the category that is most similar to the available categories.
o K-NN algorithm stores all the available data and classifies a new data point based on
the similarity. This means when new data appears then it can be easily classified into a
well suite category by using K- NN algorithm.
o K-NN algorithm can be used for Regression as well as for Classification but mostly it
is used for the Classification problems.
o K-NN is a non-parametric algorithm, which means it does not make any assumption
on underlying data.
o It is also called a lazy learner algorithm because it does not learn from the training set
immediately instead it stores the dataset and at the time of classification, it performs an
action on the dataset.
o KNN algorithm at the training phase just stores the dataset and when it gets new data,
then it classifies that data into a category that is much similar to the new data.
Example : Suppose, we have an image of a creature that looks similar to cat and dog, but we
want to know either it is a cat or dog. So for this identification, we can use the KNN algorithm,
as it works on a similarity measure. Our KNN model will find the similar features of the new
data set to the cats and dogs images and based on the most similar features it will put it in either
cat or dog category.
Suppose there are two categories, i.e., Category A and Category B, and we have a new data
point x1, so this data point will lie in which of these categories. To solve this type of problem,
we need a K-NN algorithm. With the help of K-NN, we can easily identify the category or class
of a particular dataset. Consider the below diagram:
How does K-NN work?
The K-NN working can be explained on the basis of the below algorithm:
o Step-1: Select the number K of the neighbors
o Step-2: Calculate the Euclidean distance of K number of neighbors
o Step-3: Take the K nearest neighbors as per the calculated Euclidean distance.
o Step-4: Among these k neighbors, count the number of the data points in each category.
o Step-5: Assign the new data points to that category for which the number of the
neighbor is maximum.
o Step-6: Our model is ready.
o Suppose we have a new data point and we need to put it in the required category.
Consider the below image:

o Firstly, we will choose the number of neighbors, so we will choose the k=5.
o Next, we will calculate the Euclidean distance between the data points. The Euclidean
distance is the distance between two points, which we have already studied in geometry.
It can be calculated as:
 o By calculating the Euclidean distance we got the nearest neighbors, as three nearest
neighbors in category A and two nearest neighbors in category B. Consider the below
image:

o As we can see the 3 nearest neighbors are from category A, hence this new data point
must belong to category A.

select the value of K in the K-NN Algorithm

o There is no particular way to determine the best value for "K", so we need to try some
values to find the best out of them. The most preferred value for K is 5.
o A very low value for K such as K=1 or K=2, can be noisy and lead to the effects of
outliers in the model.
o Large values for K are good, but it may find some difficulties.
o the classification error probability for the NN rule,PNN , is bounded by

o
where PB is the optimal Bayesian error. Thus,the error committed by the NN classifier
is (asymptotically) at most twice that of the optimal classifier. The asymptotic
o performance of the kNN is better than that of the NN,

ation Maximization for GMM

How Can We Fit a Mixture of Gaussians? How Can We Fit a Mixture of Gaussians? How Can

We Fit a Mixture of Gaussians?