lOMoARcPSD|29880857

MLT(Unit-2) - MLT

Machine learning techniques (Dr. A.P.J. Abdul Kalam Technical University)

Scan to open on Studocu

Studocu is not sponsored or endorsed by any college or university

Downloaded by Prabu S (arokiaprabus@[Link])
 lOMoARcPSD|29880857

Regression

Regression in machine learning is a technique used to find the relationships between

independent and dependent variables, with the main purpose of predicting an outcome. It
involves training a set of algorithms to reveal patterns that characterize the distribution of each
data point. With patterns identified, the model can then make accurate predictions for new data
points or input values.

Types of Regression

1. Linear Regression
2. Logistic Regression

Linear Regression

Linear regression is a type of supervised machine-learning algorithm that learns from the
labelled datasets and maps the data points with most optimized linear functions which can be
used for prediction on new datasets. It assumes that there is a linear relationship between the
input and output, meaning the output changes at a constant rate as the input changes. This
relationship is represented by a straight line.

For example we want to predict a student's exam score based on how many hours they
studied. We observe that as students study more hours, their scores go up. In the example of
predicting exam scores based on hours studied. Here

 Independent variable (input): Hours studied because it's the factor we control or
observe.

 Dependent variable (output): Exam score because it depends on how many hours were
studied.

Equation of the Best-Fit Line

For simple linear regression (with one independent variable), the best-fit line is represented
by the equation

y=mx+b

Where:

 y is the predicted value (dependent variable)

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

 x is the input (independent variable)

 m is the slope of the line (how much y changes when x changes)

 b is the intercept (the value of y when x = 0)

The best-fit line will be the one that optimizes the values of m (slope) and b (intercept) so that
the predicted y values are as close as possible to the actual data points.

Study Test Mean(X) Mean(Y) Deviations(X) Deviations(Y) Product of Sum of Square of

Hours score deviations Product of deviations
(X) (Y) deviations for X
2 40 4 50 -2 -10 20 40 4
4 50 0 0 0 0
6 60 2 10 20 4

Calculate m = Sum of product of deviations / Sum of square of deviation for X

Calculate b = Mean of Y – (m* Mean of X)

Calculations

 Sum of Product of Deviations = 20 + 0 + 20 = 40

 Sum of Square of Deviations for X = 4 + 0 + 4 = 8

m = Sum of Product of Deviations / Sum of Square of Deviations for X

m = 40/8 = 5

b=Mean(Y) − (m * mean(X)) =50− (5*4) =30

Final Regression Equation

Y=5X+30Y

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

Q1: Fit a linear regression model for data set (x, y): (1, 1.5), (2, 3.0), (3, 4.5), (4, 6.0) and
predict y for x = 5

Non-Linear Regression

Non-linear regression is a type of regression in machine learning where the relationship

between input XXX and output YYY is not a straight line. Instead, the data follows a curved
pattern.

In such cases, a straight line (linear regression) does not fit well, so we use equations like
polynomial, exponential, logarithmic, or other non-linear functions.

Multiple Linear Regression

Linear regression is a statistical method used for predictive analysis. It models the
relationship between a dependent variable and a single independent variable by fitting a
linear equation to the data. Multiple Linear Regression extends this concept by modelling
the relationship between a dependent variable and two or more independent variables. This
technique allows us to understand how multiple features collectively affect the outcomes.
Steps for Multiple Linear Regression
Steps to perform multiple linear regression are similar to that of simple linear Regression but
difference comes in the evaluation process. We can use it to find out which factor has the
highest influence on the predicted output and how different variables are related to each
other. Equation for multiple linear regression is:
y=β0+β1X1+β2X2+⋯+βnXn
Where:
 Y is the dependent variable

 X1,X2,⋯Xn are the independent variables

 β0 is the intercept

 β1,β2,⋯βn are the slopes

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

The goal of the algorithm is to find the best fit line equation that can predict the values based
on the independent variables. A regression model learns from the dataset with known X and y
values and uses it to predict y values for unknown X.

[Link] B0,B1,B2 using the given data.

Product 1 Product 2 Weekly Sales

Sales Sales (Y)
(X1) (X2)
1 4 1
2 5 6
3 8 8
4 2 12

𝟏 𝟏 𝟒 𝟏
𝟏 𝟐 𝟓 𝟔
𝑿=[ ] 𝒀=[ ]
𝟏 𝟑 𝟖 𝟖
𝟏 𝟒 𝟐 𝟏𝟐

Step 1. Transpose of X

𝟏 𝟏 𝟏 𝟏
𝑿` = [𝟏 𝟐 𝟑 𝟒]
𝟒 𝟓 𝟖 𝟐

Step 2. Multiply of X`.X

𝟏 𝟏 𝟒
𝟏 𝟏 𝟏 𝟏 𝟒 𝟏𝟎 𝟏𝟗
𝟏 𝟐 𝟓
X`.X = [𝟏 𝟐 𝟑 𝟒] . [ ] = [𝟏𝟎 𝟑𝟎 𝟒𝟔 ]
𝟏 𝟑 𝟖
𝟒 𝟓 𝟖 𝟐 𝟏𝟗 𝟒𝟔 𝟏𝟎𝟗
𝟏 𝟒 𝟐

Step 3. Multiply of X`.Y

𝟏
𝟏 𝟏 𝟏 𝟏 𝟐𝟕
𝟔
X`.Y = [𝟏 𝟐 𝟑 𝟒] . [ ] = [ 𝟖𝟓 ]
𝟖
𝟒 𝟓 𝟖 𝟐 𝟏𝟐𝟐
𝟏𝟐

Step 4. Inverse of (X`.X)-1

𝟒 𝟏𝟎 𝟏𝟗 𝟑. 𝟏𝟓𝟑 −𝟎. 𝟓𝟗𝟎 −𝟎. 𝟑𝟎𝟎

(X`.X)-1 = [𝟏𝟎 𝟑𝟎 𝟒𝟔 ] = [−𝟎. 𝟓𝟗𝟎 𝟎. 𝟐𝟎𝟒 𝟎. 𝟎𝟏𝟔 ]
𝟏𝟗 𝟒𝟔 𝟏𝟎𝟗 −𝟎. 𝟑𝟎𝟎 𝟎. 𝟎𝟏𝟔 𝟎. 𝟎𝟓𝟒

Step 4. Put in this equation β=(X`X)-1X`Y

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

𝟑. 𝟏𝟓𝟑 −𝟎. 𝟓𝟗𝟎 −𝟎. 𝟑𝟎𝟎 𝟐𝟕 −𝟏. 𝟔𝟗𝟗

β = [−𝟎. 𝟓𝟗𝟎 𝟎. 𝟐𝟎𝟒 𝟎. 𝟎𝟏𝟔 ] . [ 𝟖𝟓 ] = [ 𝟑. 𝟒𝟖𝟑 ]
−𝟎. 𝟑𝟎𝟎 𝟎. 𝟎𝟏𝟔 𝟎. 𝟎𝟓𝟒 𝟏𝟐𝟐 −𝟎. 𝟎𝟓𝟒

β0 = -1.699, β1 = 3.483, β2 = - 0.054

Logistic Regression

Logistic regression is a type of supervised machine-learning algorithm that also learns from
labelled datasets but is mainly used for classification problems instead of predicting continuous
values. It assumes that the output is categorical, such as Yes/No or 0/1, and maps the data
points using a logistic function (sigmoid curve) to estimate probabilities between 0 and 1. This
probability is then used to decide the class of new data points. For example, we may want to
predict whether a student will pass or fail based on how many hours they studied. We observe
that as study hours increase, the probability of passing also increases, which is captured by the
S-shaped logistic curve.

Sigmoid Function

Y = 1/1+e-(a0 + a1*X)

Where :

a0 → Intercept (similar to b in linear regression).

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

a1→ Coefficient/weight of the feature XXX.

X → Input (independent variable).

Output → A probability between 0 and 1.

Example :

Study Hours (X) Output(Y) = Pass/Fail

2 0
3 0
4 0
5 1
6 1
7 1
8 1

Given :

a0 = -1.5

a1 = 0.6

Input (X) = 5

1
y=
1+ ⅇ−(a0+a1X)

1
y=
1+ ⅇ−(−1.5+0.6x5)

1
y=
1 + ⅇ−(1.5)

1
y=
1 + ⅇ−(1.5)

1
y=
1 + 0.2231

1
y=
1.2231

y = 0.8175

Note : Value “Y” is greater than 0.5 then student is Pass.

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

What is Bayes theorem?

Bayes' theorem is a fundamental concept in probability theory that plays a crucial role in
various machine learning algorithms, especially in the fields of Bayesian statistics and
probabilistic modelling. It provides a way to update probabilities based on new evidence or
information. In the context of machine learning, Bayes' theorem is often used in Bayesian
inference and probabilistic models.
The theorem can be mathematically expressed as:
P(A∣B)=P(B∣A)∣P(A)/P(B)
Where

 P(A∣B) is the posterior probability of event A given event B.

 (B∣A) is the likelihood of event B given event A.

 P(A) is the prior probability of event A.

 P(B) is the total probability of event B.

In the context of modeling hypotheses, Bayes' theorem allows us to infer our belief in a
hypothesis based on new data. We start with a prior belief in the hypothesis, represented
by P(A), and then update this belief based on how likely the data are to be observed under
the hypothesis, represented by P(B∣A). The posterior probability P(A∣B) represents our
updated belief in the hypothesis after considering the data.

Key Terms Related to Bayes Theorem

1. Likelihood(P(B∣A)):

 Represents the probability of observing the given evidence (features) given that the
class is true.

 In the Naive Bayes algorithm, a key assumption is that features are conditionally
independent given the class label. In other words, Naive Bayes works best with
discrete features.

2. Prior Probability (P(A)):

 In machine learning, this represents the probability of a particular class before

considering any features.

 It is estimated from the training data.

3. Evidence Probability( P(B) ):

 This is the probability of observing the given evidence (features).

 It serves as a normalization factor and is often calculated as the sum of the joint
probabilities over all possible classes.

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

What is Naive Bayes Classifier?

The Naive Bayes classifier is a supervised learning algorithm used for solving classification
problems. It is based on Bayes’ Theorem, which calculates the probability of a certain event
occurring based on prior knowledge. The Naive Bayes algorithm assigns new data points to the
most likely class by comparing probabilities.

The algorithm is widely used in real-world scenarios, such as classifying emails into spam and
non-spam or identifying the sentiment of customer reviews. The strength of this algorithm lies
in how efficiently it handles data, even with limited training examples.

Why is it called Naive Bayes?

The Naive Bayes algorithm is called “naive” because it makes a strong assumption: all the
features in the dataset are independent of each other. This means that the presence of one
feature does not affect the presence of another.

In reality, many features are correlated, and this assumption may not always hold. However,
despite this simplification, the Naive Bayes algorithm often performs well in practice. Its
simplicity allows for quick computation, making it a popular choice for tasks like text
classification and spam detection.

Weather Temperature Play?

Sunny Hot No

Sunny Hot No

Overcast Hot Yes

Rainy Mild Yes

Rainy Cool Yes

Rainy Cool No

Overcast Cool Yes

Sunny Mild No

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

Sunny Cool Yes

Rainy Mild Yes

Sunny Mild Yes

Overcast Mild Yes

Dataset
Overcast Hot Yes
Consider a fictional dataset that
describes the weather conditions for
Rainy Mild No
playing a game of golf. Each row in the
dataset records the weather,
temperature, and whether the
conditions were suitable for playing. The goal is to predict whether it’s fit (“Yes”) or unfit (“No”)
for playing golf based on the weather conditions. Below is a tabular representation of our
dataset.

This dataset provides a simple example to illustrate how the Naive Bayes classifier calculates
probabilities. Later in the article, we will use this dataset to walk through examples and the
algorithm implementation.

How Bayes’ Theorem Works in Naive Bayes Classifier

In a Naive Bayes classifier, Bayes’ Theorem is used to calculate the probability of a

class based on given features. For instance, if we want to predict whether it is fit to play golf on
a Sunny day with Mild temperature, the classifier will compute two probabilities:

1. P(Play = Yes | Sunny, Mild) – Probability of playing on a sunny, mild day.

2. P(Play = No | Sunny, Mild) – Probability of not playing on a sunny, mild day.

Whichever probability is higher becomes the final prediction.

Naive Assumption

In the Naive Bayes algorithm, we assume that all features are independent of each other, which
simplifies the computation. This is known as the “naive” assumption. Although this assumption
may not always be true in practice, it often leads to good results.

Simplified Example Calculation

Let’s consider our dataset. To compute the probability of Play = Yes on a Sunny day with Mild
temperature, we need the following:

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

Similarly, the algorithm will calculate P(Play = No | Sunny, Mild). The class with the higher
probability will be the predicted outcome.

Bayes’ Theorem: Implementation (Example)

To demonstrate how Bayes’ Theorem works, let’s predict whether we can play outside on
a sunny day with mild temperature using the dataset.

The dataset has two features: Weather (Sunny, Overcast, Rainy) and Temperature (Hot, Mild,
Cool). The target variable is Play (Yes/No). We’ll calculate the probabilities step by step.

Goal:

Predict if Play = Yes or Play = No for Weather = Sunny and Temperature = Mild.

Step 1: Calculate Prior Probabilities

We first calculate the prior probabilities for each class:

Step 2: Calculate Likelihood Probabilities

Next, we calculate the likelihood probabilities for each feature value based on the given class:

 For Weather = Sunny:

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

Step 3: Apply Bayes’ Theorem

Now, we calculate the posterior probabilities for both classes using Bayes’ Theorem.

Step 4: Make the Prediction

Since P(Play = No | Sunny, Mild) is greater than P(Play = Yes | Sunny, Mild), the classifier will
predict:

Prediction: Play = No

Working of Naive Bayes Classifier

The Naive Bayes classifier works in two main stages: training and prediction. Below is a step-
by-step breakdown of how the algorithm classifies new data using the dataset we introduced
earlier.

Step 1: Training the Model

In the training phase, the algorithm calculates:

1. Prior Probability:

The probability of each class in the dataset.

Example:

2. Conditional Probability:

The probability of each feature value given a class.

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

Example:

3. Smoothing (Optional):

If a feature value has zero occurrences for a class, it can make the probability zero. Smoothing
techniques like Laplace Smoothing are used to handle this.

Step 2: Making Predictions

To predict the class of a new data point (e.g., a Sunny day with Mild temperature), the
classifier calculates the posterior probability for each class.

Summary of the Workflow

1. Training Phase: Calculate prior and conditional probabilities from the dataset.

2. Prediction Phase: Use Bayes’ Theorem to find the class with the highest probability.

3. Output: The predicted class is assigned to the new data point.

Advantages of Naive Bayes Classifier

 Simplicity and Ease of Implementation: Naive Bayes is straightforward to implement

and requires less computational power compared to other algorithms.

 Fast Training and Prediction: Since it involves simple probability calculations, Naive
Bayes is efficient for both training and making predictions, even on large datasets.

 Performs Well with High-Dimensional Data: The algorithm works effectively even
with datasets that contain many features, such as text classification where each word
becomes a feature.

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

 Works Well with Small Datasets: Naive Bayes can perform reliably even with limited
training data, as it generalizes well using prior probabilities.

 Handles Categorical and Text Data: It is widely used in spam filtering, sentiment
analysis, and news classification due to its efficiency with categorical data.

Disadvantages of Naive Bayes Classifier

 Naive Assumption of Feature Independence: The algorithm assumes that all features
are independent of each other, which is rarely true in real-world datasets. This can
reduce its accuracy if the features are highly correlated.

 Zero Probability Issue: If a feature value does not appear in the training data for a
given class, the algorithm assigns zero probability to that class. This can be mitigated
using Laplace Smoothing.

 Limited with Continuous Features: Naive Bayes struggles with continuous features
unless they follow a normal distribution, as required in Gaussian Naive Bayes.
Otherwise, data preprocessing may be necessary.

 Sensitive to Irrelevant Features: Including irrelevant or noisy features can negatively

impact the classifier’s performance, as the algorithm gives equal importance to all
features.

 Not Suitable for Complex Relationships: The algorithm may not perform well when
there are complex relationships between features, as it lacks the capability to model
feature interactions.

Applications of Naive Bayes Classifier

 Spam Filtering: Naive Bayes is at the heart of many spam detection systems. It classifies
emails as spam or non-spam based on the occurrence of specific words or patterns.

 Sentiment Analysis: It is used to determine the sentiment of customer reviews, social

media posts, or feedback by analyzing positive and negative words.

 Text Classification: Naive Bayes is effective for news categorization and classifying
documents into categories like sports, technology, or politics.

 Recommendation Systems: It helps in building recommendation engines by predicting

user preferences based on historical data.

 Medical Diagnosis: In healthcare, the algorithm can assist in predicting diseases based
on patient symptoms and medical history.

Types of Naive Bayes Model

There are several variations of the Naive Bayes algorithm, each designed to handle specific
types of data. Here are the most common types:

 Gaussian Naive Bayes (GNB):

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

 Suitable for continuous features that follow a normal distribution (bell curve).

 Example: Predicting stock prices or housing values using continuous data like price
or area.

 When to Use: Use when your features are continuous, like temperature or age, and
follow a normal distribution.

 Multinomial Naive Bayes (MNB):

 Works well with count data, where features represent the frequency of
occurrences (e.g., word counts in text).

 Example: Classifying documents or emails based on the frequency of words.

 When to Use: Ideal for text classification tasks, such as spam detection, where
features represent word frequencies.

 Bernoulli Naive Bayes (BNB):

 Suitable for binary features (yes/no, true/false) where features can take only two
values.

 Example: Spam filtering, where words are either present (1) or absent (0) in an
email.

 When to Use: Best suited for tasks with binary features, such as sentiment analysis
or spam filtering.

Bayesian Belief Network (BBN)

Bayesian Belief Network (BBN) is a graphical model that represents the probabilistic
relationships among variables. It is used to handle uncertainty and make predictions or
decisions based on probabilities.
 Graphical Representation: Variables are represented as nodes in a directed acyclic
graph (DAG), and their dependencies are shown as edges.

 Conditional Probabilities: Each node's probability depends on its parent nodes,

expressed as P(Variable | Parent)P(Variable | Parent).

 Probabilistic Model: Built from probability distributions, BBNs apply probability theory
for tasks like prediction and anomaly detection.

Bayesian Belief Networks are valuable tools for understanding and solving problems
involving uncertain events. They are also known as Bayes networks, belief networks,
decision networks, or Bayesian models.
(Note: A classifier assigns data in a collection to desired categories.)
 Consider this example:

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

 In the above figure, we have an alarm 'A' - a node, say installed in a house of a person 'gfg',
which rings upon two probabilities i.e burglary 'B' and fire 'F', which are - parent nodes of
the alarm node. The alarm is the parent node of two probabilities P1 calls 'P1' & P2 calls
'P2' person nodes.

 Upon the instance of burglary and fire, 'P1' and 'P2' call person 'gfg', respectively. But,
there are few drawbacks in this case, as sometimes 'P1' may forget to call the person 'gfg',
even after hearing the alarm, as he has a tendency to forget things, quick. Similarly, 'P2',
sometimes fails to call the person 'gfg', as he is only able to hear the alarm, from a certain
distance.

Calculating Conditional Probability of Events in a Bayesian Network

Find the probability that 'P1' is true (P1 has called 'gfg'), 'P2' is true (P2 has called 'gfg') when
the alarm 'A' rang, but no burglary 'B' and fire 'F' has occurred.
=> P ( P1, P2, A, ~B, ~F) [ where- P1, P2 & A are 'true' events and '~B' & '~F' are 'false'
events]
[ Note: The values mentioned below are neither calculated nor computed. They have observed
values ]
Burglary 'B' -
 P (B=T) = 0.001 ('B' is true i.e burglary has occurred)

 P (B=F) = 0.999 ('B' is false i.e burglary has not occurred)

Fire 'F' -
 P (F=T) = 0.002 ('F' is true i.e fire has occurred)

 P (F=F) = 0.998 ('F' is false i.e fire has not occurred)

Alarm 'A' -

B F P (A=T) P (A=F)

T T 0.95 0.05

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

B F P (A=T) P (A=F)

T F 0.94 0.06

F T 0.29 0.71

F F 0.001 0.999

 The alarm 'A' node can be 'true' or 'false' ( i.e may have rung or may not have rung). It has
two parent nodes burglary 'B' and fire 'F' which can be 'true' or 'false' (i.e may have
occurred or may not have occurred) depending upon different conditions.

Person 'P1' -

A P (P1=T) P (P1=F)

T 0.95 0.05

F 0.05 0.95

 The person 'P1' node can be 'true' or 'false' (i.e may have called the person 'gfg' or not) . It
has a parent node, the alarm 'A', which can be 'true' or 'false' (i.e may have rung or may
not have rung ,upon burglary 'B' or fire 'F').

Person 'P2' -

A P (P2=T) P (P2=F)

T 0.80 0.20

F 0.01 0.99

 The person 'P2' node can be 'true' or false' (i.e may have called the person 'gfg' or not). It
has a parent node, the alarm 'A', which can be 'true' or 'false' (i.e may have rung or may
not have rung, upon burglary 'B' or fire 'F').

Solution: Considering the observed probabilistic scan -

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

With respect to the question — P ( P1, P2, A, ~B, ~F) , we need to get the probability of 'P1'.
We find it with regard to its parent node - alarm 'A'. To get the probability of 'P2', we find it
with regard to its parent node — alarm 'A'.
We find the probability of alarm 'A' node with regard to '~B' & '~F' since burglary 'B' and fire
'F' are parent nodes of alarm 'A'.
From the observed probabilistic scan, we can deduce -
P ( P1, P2, A, ~B, ~F)
= P (P1/A) * P (P2/A) * P (A/~B~F) * P (~B) * P (~F)
= 0.95 * 0.80 * 0.001 * 0.999 * 0.998
= 0.00075

Expectation-Maximization Algorithm - ML

Last Updated : 08 Sep, 2025


The Expectation-Maximization (EM) algorithm is a powerful iterative optimization
technique used to estimate unknown parameters in probabilistic models, particularly when the
data is incomplete, noisy or contains hidden (latent) variables. It works in two steps:
 E-step (Expectation Step): Using the current parameter estimates, the algorithm calculates
the expected values of the missing or hidden variables. Essentially, it assigns probabilities or
"responsibilities" to different hidden outcomes given the observed data.
 M-step (Maximization Step): With these updated expectations from the E-step, the
algorithm then re-estimates the model parameters by maximizing the expected log-
likelihood. This improves how well the model explains the observed data.
These two steps are repeated until convergence, which typically means that:
 The parameter values stop changing significantly, or
 The log-likelihood improves only by a negligible amount.

Expectation
and Maximization in EM Algorithm

By iteratively repeating these steps the EM algorithm seeks to maximize the likelihood of the
observed data.
Key Terms in Expectation-Maximization (EM) Algorithm
Lets understand about some of the most commonly used key terms in the Expectation-
Maximization (EM) Algorithm:
 Latent Variables: Variables that are not directly observed but are inferred from the data.
They represent hidden structure (e.g., cluster assignments in Gaussian Mixture Models).

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

 Likelihood: The probability of the observed data given a set of model parameters. EM aims
to find parameter values that maximize this likelihood.
 Log-Likelihood: The natural logarithm of the likelihood function. It simplifies calculations
(turning products into sums) and is numerically more stable when dealing with very small
probabilities.
 Maximum Likelihood Estimation (MLE): A statistical approach to estimating parameters
by choosing the values that maximize the likelihood of observing the given data. EM extends
MLE to cases with hidden or missing variables.
 Posterior Probability: In Bayesian inference, this represents the probability of parameters
(or latent variables) given the observed data and prior knowledge. In EM, posterior
probabilities are used in the E-step to estimate the "responsibility" of each hidden variable.
 Convergence: The stopping criterion for the iterative process. EM is said to converge when
updates to parameters or improvements in log-likelihood become negligibly small, meaning
the algorithm has reached a stable solution.
Working of Expectation-Maximization (EM) Algorithm
Here's a step-by-step breakdown of the process:

EM Algorithm Flowchart

1. Initialization: The algorithm starts with initial parameter values and assumes the observed
data comes from a specific model.
2. E-Step (Expectation Step):
 Find the missing or hidden data based on the current parameters.
 Calculate the posterior probability of each latent variable based on the observed data.
 Compute the log-likelihood of the observed data using the current parameter estimates.
3. M-Step (Maximization Step):
 Update the model parameters by maximize the log-likelihood.
 The better the model the higher this value.
4. Convergence:
 Check if the model parameters are stable and converging.
 If the changes in log-likelihood or parameters are below a set threshold, stop. If not repeat
the E-step and M-step until convergence is reached

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

Support Vector Machine (SVM) Algorithm

Support Vector Machine (SVM) is a supervised machine learning algorithm used for
classification and regression tasks. It tries to find the best boundary known as hyperplane that
separates different classes in the data. It is useful when you want to do binary classification like
spam vs. not spam or cat vs. dog.
The main goal of SVM is to maximize the margin between the two classes. The larger the margin
the better the model performs on new and unseen data.

Key Concepts of Support Vector Machine

 Hyperplane: A decision boundary separating different classes in feature space and is
represented by the equation wx + b = 0 in linear classification.
 Support Vectors: The closest data points to the hyperplane, crucial for determining the
hyperplane and margin in SVM.
 Margin: The distance between the hyperplane and the support vectors. SVM aims to
maximize this margin for better classification performance.
 Kernel: A function that maps data to a higher-dimensional space enabling SVM to handle
non-linearly separable data.
 Hard Margin: A maximum-margin hyperplane that perfectly separates the data without
misclassifications.
 Soft Margin: Allows some misclassifications by introducing slack variables, balancing
margin maximization and misclassification penalties when data is not perfectly separable.
 C: A regularization term balancing margin maximization and misclassification penalties. A
higher C value forces stricter penalty for misclassifications.
 Hinge Loss: A loss function penalizing misclassified points or margin violations and is
combined with regularization in SVM.
 Dual Problem: Involves solving for Lagrange multipliers associated with support vectors,
facilitating the kernel trick and efficient computation.

How does Support Vector Machine Algorithm Work?

The key idea behind the SVM algorithm is to find the hyperplane that best separates two
classes by maximizing the margin between them. This margin is the distance from the
hyperplane to the nearest data points (support vectors) on each side.

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

Multiple hyperplanes separate the data from two classes

The best hyperplane also known as the "hard margin" is the one that maximizes the distance
between the hyperplane and the nearest data points from both classes. This ensures a clear
separation between the classes. So from the above figure, we choose L2 as hard margin. Let's
consider a scenario like shown below:

Selecting hyperplane for data with outlier

Here, we have one blue ball in the boundary of the red ball.
How does SVM classify the data?
The blue ball in the boundary of red ones is an outlier of blue balls. The SVM algorithm has the
characteristics to ignore the outlier and finds the best hyperplane that maximizes the margin.
SVM is robust to outliers.

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

Hyperplane which is the most optimized one

A soft margin allows for some misclassifications or violations of the margin to improve
generalization. The SVM optimizes the following equation to balance margin maximization
and penalty minimization:
Objective Function=(1margin)+λ∑penalty Objective Function=(margin1
)+λ∑penalty
The penalty used for violations is often hinge loss which has the following behavior:
 If a data point is correctly classified and within the margin there is no penalty (loss = 0).
 If a point is incorrectly classified or violates the margin the hinge loss increases
proportionally to the distance of the violation.
Till now we were talking about linearly separable data that seprates group of blue balls and
red balls by a straight line/linear line.
What if data is not linearly separable?
When data is not linearly separable i.e it can't be divided by a straight line, SVM uses a
technique called kernels to map the data into a higher-dimensional space where it becomes
separable. This transformation helps SVM find a decision boundary even for non-linear data.

Original 1D dataset for classification

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

A kernel is a function that maps data points into a higher-dimensional space without explicitly
computing the coordinates in that space. This allows SVM to work efficiently with non-linear
data by implicitly performing the mapping. For example consider data points that are not
linearly separable. By applying a kernel function SVM transforms the data points into a
higher-dimensional space where they become linearly separable.
 Linear Kernel: For linear separability.
 Polynomial Kernel: Maps data into a polynomial space.
 Radial Basis Function (RBF) Kernel: Transforms data into a space based on distances
between data points.

Types of Support Vector Machine (SVM)

Types of Support Vector Machine (SVM) include Linear SVM, used for linearly separable data, and

Non-Linear SVM, which handles complex data using kernel functions like RBF and polynomial.

These SVM types are widely applied in classification tasks such as text analysis and image

recognition.

Below, we provide a detailed explanation of each type.

Linear SVM : Linear SVM is used for data that are linearly separable i.e. for a dataset that can be

categorized into two categories by utilizing a single straight line. Such data points are termed as

linearly separable data, and the classifier is used described as a Linear SVM classifier.

Non-linear SVM: Non-Linear SVM is used for data that are non-linearly separable data i.e. a

straight line cannot be used to classify the dataset. For this, we use something known as a kernel

trick that sets data points in a higher dimension where they can be separated using planes or other

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

mathematical functions. Such data points are termed as non-linear data, and the classifier used is

termed as a Non-linear SVM classifier.

What is the Kernel method?

The kernel method is the mathematical technique that is used inmachine learning for analyzing
data. This method uses the Kernel function - that maps data from one space to another space.
It is generally used in Support Vector Machines (SVMs) where the algorithms classify data by
finding the hyperplane that separates the data points of different classes.
The most important benefit of the Kernel Method is that it can work with non-linearly separable
data, and it works with multiple Kernel functions - depending on the type of data.
Because the linear classifier can solve a very limited class of problems, the kernel trick is
employed to empower the linear classifier, enabling the SVM to solve a larger class of problems.

Source:eduCBA

What are the types of Kernel methods in SVM models?

Support vector machines use various kinds ofkernel methods in machine learning. Here are a
few of them:
1. Linear Kernel

K(x,y) = x.y

 No transformation → just the dot product.

 Used when data is linearly separable.
 Example: Text classification (spam filtering).

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

2. Polynomial Kernel
K(x,y) = (x.y + c)d

 Maps data into a higher dimension using polynomials.

 Good for non-linear patterns where relationships are polynomial.
 Parameters:
o c: trade-off parameter (bias)
o d: degree of the polynomial

3. Radial Basis Function (RBF) Kernel / Gaussian Kernel

 Maps data into infinite-dimensional space.

 Most popular kernel.
 Works well when classes are not linearly separable.
 Parameter:
o σ\sigma (or γ\gamma) → controls how far the influence of a single training
example reaches.

Advantages of SVM

SVM offers several advantages, including:

1. High effectiveness in high-dimensional spaces: SVM performs well in situations

where the number of features exceeds the number of samples.
2. Memory efficiency: SVM only uses a subset of the training data (support vectors),
making it memory-efficient.
3. Clear margin separation: SVM works well when there is a clear margin of separation
between classes.
4. Effectiveness with non-linear data: The kernel trick allows SVM to handle non-
linearly separable data efficiently.
Disadvantages of SVM

Despite its advantages, SVM also has some drawbacks:

1. Complex model interpretation: In non-linear cases, interpreting the model becomes

difficult.
2. Training time: SVM can take longer to train on large datasets compared to other
algorithms like decision trees.
3. Sensitivity to kernel choice: The performance of SVM is highly dependent on the
choice of kernel and regularization parameters.
4. Performance with overlapping classes: SVM may perform poorly when classes
overlap significantly.

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

Applications of SVM in Real World

As we have seen, SVMs depends on supervised learning algorithms. The aim of using SVM is to
correctly classify unseen data. SVMs have a number of applications in several fields.
Some common applications of SVM are-
 Face detection – SVMc classify parts of the image as a face and non-face and create a
square boundary around the face.
 Text and hypertext categorization – SVMs allow Text and hypertext categorization for
both inductive and transductive models. They use training data to classify documents into
different categories. It categorizes on the basis of the score generated and then compares
with the threshold value.
 Classification of images – Use of SVMs provides better search accuracy for image
classification. It provides better accuracy in comparison to the traditional query-based
searching techniques.
 Bioinformatics – It includes protein classification and cancer classification. We use SVM
for identifying the classification of genes, patients on the basis of genes and other biological
problems.
 Protein fold and remote homology detection – Apply SVM algorithms for protein remote
homology detection.
 Handwriting recognition – We use SVMs to recognize handwritten characters used
widely.
 Generalized predictive control(GPC) – Use SVM based GPC to control chaotic dynamics
with useful parameters.
Let us now see the above applications of SVM in detail-

1. Face Detection

It classifies the parts of the image as face and non-face. It contains training data of n x n pixels
with a two-class face (+1) and non-face (-1). Then it extracts features from each pixel as face or
non-face. Creates a square boundary around faces on the basis of pixel brightness and classifies
each image by using the same process.

Let’s discuss the learning rules in Neural Networks

2. Text and Hypertext Categorization

Allows text and hypertext categorization for both types of models; inductive and transductive. It
Uses training data to classify documents into different categories such as news articles, e-mails,
and web pages

Examples:
 Classification of news articles into “business” and “Movies”
 Classification of web pages into personal home pages and others
For each document, calculate a score and compare it with a predefined threshold value. When
the score of a document surpasses threshold value, then the document is classified into a
definite category. If it does not surpass threshold value then consider it as a general document.

Classify new instances by computing score for each document and comparing it with the learned
threshold.

In the text classification field, SVMs are used for sentiment analysis—like checking if a product
review is positive or negative. They are also applied in voice recognition systems, where spoken
words are converted into text. What makes SVM stand out is its ability to give good results even
when data is limited.

Downloaded by Prabu S (arokiaprabus@[Link])

 lOMoARcPSD|29880857

3. Classification of Images

SVMs can classify images with higher search accuracy. Its accuracy is higher than traditional
query-based refinement schemes

4. Bioinformatics

In the field of computational biology, the protein remote homology detection is a common
problem. The most effective method to solve this problem is using SVM. In last few years, SVM
algorithms have been extensively applied for protein remote homology detection. These
algorithms have been widely used for identifying among biological sequences. For example
classification of genes, patients on the basis of their genes, and many other biological problems.

5. Protein Fold and Remote Homology Detection

Protein remote homology detection is a key problem in computational biology. Supervised

learning algorithms on SVMs are one of the most effective methods for remote homology
detection. The performance of these methods depends on how the protein sequences modeled.
The method used to compute the kernel function between them.
Do you know about Kernel Functions in ML
6. Handwriting Recognition

We can also use SVMs to recognize hand-written characters that use for data entry and
validating signatures on documents.

Downloaded by Prabu S (arokiaprabus@[Link])