Supervised learning

Linear Regression Models: Least squares, single & multiple variables, Bayesian linear
regression, gradient descent, Linear Classification Models: Discriminant function –
Perceptron algorithm, Probabilistic discriminative model - Logistic regression, Probabilistic
generative model – Naive Bayes, Maximum margin classifier – Support vector machine,
Decision Tree

Introduction:
Supervised learning: classification is seen as supervised learning from examples.

● Supervision: The data (observations, measurements, etc.) are labeled with pre- defined classes. It is like
that a “teacher” gives the classes (supervision).
● Test data are classified into these classes too.
Unsupervised learning (clustering)
● Class labels of the data are unknown

● Given a set of data, the task is to establish the existence of classes or clusters in the data

Learning (training): Learn a model using the training data

Testing: Test the model using unseen test data to assess the model accuracy

Supervised Learning: Data and corresponding labels are given

Unsupervised Learning: Only data is given, no labels provided
Semi-supervised Learning: Some (if not all) labels are present
Linear Regression Models:
Linear regression is a statistical method that models the relationship between variables by fitting the best
straight line (or surface) to data points, allowing for prediction of a continuous outcome (dependent variable)
from one or more inputs (independent variables). It finds this line, represented by the equation

Department of CSE-AIML Machine Learning 24BEAML405

by minimizing the total distance (errors) between the line and the actual data points, a process often using
the least squares method.
Working

1. Identifies a linear trend: It assumes a straight-line relationship between variables, plotting them on a
graph (scatter plot).
2. Finds the best-fit line: It calculates the line that minimizes the squared differences (residuals) between
the predicted values and the actual data points.
3. Uses an equation: The line is defined by an equation.

Where X (or xi) are the inputs, Y (or y) is the prediction, a (or β1) is the slope, b (or β0) is the y intercept
Key components

Dependent Variable (Y): The outcome you want to predict (e.g., house price).

● Independent Variable(s) (X/xᵢ): The feature(s) used for prediction (e.g., square footage, number of
rooms).
● Slope (a or β1): How much Y changes for a one-unit change in X.
● Y- intercept (b or β0): Value of Y when x = 0
● Error (ε ): The difference between the predicted and actual values.

Types

● Simple Linear Regression: One independent variable predicting the dependent variable (e.g., height to
predict weight).
● Multiple Linear Regression: Multiple independent variables predicting the dependent variable (e.g.,
age, height, diet to predict weight).

Important assumptions

● Linearity: A straight-line relationship exists.

● Homoscedasticity: Errors have constant variance.

Department of CSE-AIML Machine Learning 24BEAML405

● Normality: Errors are normally distributed.

● Independence: Errors are not correlated.

Least Squares

The least squares method is a form of mathematical regression analysis used to determine the line of best
fit for a set of data, providing a visual demonstration of the relationship between the data points. Each
point of data represents the relationship between a known independent variable and an unknown
dependent variable. This method of regression analysis begins with a set of data points to be plotted on
an x-axis and y-axis graph. An analyst using the least squares method will generate a line of best fit that
explains the potential relationship between independent and dependent variables.

The regression line under the Least Squares method is calculated using the following
formula – y = a + bx
Where,
y = dependent variable
x = independent variable
a = y-intercept
b = slope of the line

The slope of line b is calculated using the following formula

Department of CSE-AIML Machine Learning 24BEAML405

Or

Y-intercept, ‘a’ is calculated using the following formula –

Least square regression line:

If the data shows a leaner relationship between two variables, the line that best fits this linear
relationship is known as a least-squares regression line, which minimizes the vertical distance from the
data points to the regression line. The term “least squares” is used because it is the smallest sum of
squares of errors, which is also called the "variance."
In regression analysis, dependent variables are illustrated on the vertical y-axis, while independent
variables are illustrated on the horizontal x-axis. These designations will form the equation for the line
of best fit, which is determined from the least squares method.
In contrast to a linear problem, a non-linear least-squares problem has no closed solution and is generally
solved by iteration.
The line of best fit is a straight line drawn through a scatter of data points that best represents the
relationship between them. Let us consider the following graph wherein a set of data is plotted along the
x and y-axis. These data points are represented using the blue dots. Three lines are drawn through these
points – a green, a red, and a blue line. The green line passes through a single point, and the red line
passes through three data points. In the above graph, the blue line represents the line of best fit as it lies
closest to all the values and the distance between the points outside the line to the line is minimal (i.e.,
the distance between the residuals to the line of best fit – also referred to as the sums of squares of
residuals). In the other two lines, the orange and the green, the distance between the residuals to the lines
is greater as compared to the blue line.
However, the blue line passes through four data points, and the distance between the residual points to the
blue line is minimal as compared to the other two lines.

Multiple regression:
Multiple regression is a statistical technique used to analyze the relationship between one dependent variable
and two or more independent variables. Its main objective is to predict the value of the dependent variable

Department of CSE-AIML Machine Learning 24BEAML405

using the known values of multiple predictors. The general form of the multiple regression equation is
Y=a+b1X1+b2X2+⋯+bnXn, where Y is the dependent variable, X₁ to Xₙ are the independent variables, a is
the intercept, and b₁ to bₙ are the regression coefficients. These coefficients represent the weight or
contribution of each independent variable to the prediction. The coefficients are calculated using the least
squares method, which minimizes the difference between the actual and predicted values. Multiple
regression overcomes the limitation of simple linear regression, which considers only one independent
variable, by allowing the analysis of multiple influencing factors simultaneously. This makes it more useful
for predicting outcomes in real-world problems where several variables affect the dependent variable.

Multiple Regression Equation: The linear regression equation is given by ( y = bx + a ), where y is the
dependent variable, x is the independent variable, b is the coefficient, and a is the intercept. In multiple
regression, there are several independent variables, so the equation becomes ( y = b 1x1 + b2x2 +…….. + bnxn
+ a). Here, y is the dependent variable, x₁, x₂, …, xₙ are the independent variables, and b₁, b₂, …, bₙ are their
corresponding coefficients representing their contribution to y. The intercept a represents the value of y
when all independent variables are zero. This equation allows prediction of the dependent variable using
multiple independent variables.

Example: A researcher studies the decline in students’ performance during online lectures using multiple
regression. The dependent variable is performance, and the independent variables include factors such as
lack of attention, internet addiction, and technology support. The regression equation is y=b1(attention)
+b2(internet addiction) +b3(technology support) +⋯ +bn x n + a, where each coefficient represents the
influence of each factor on performance.

Assumptions of multiple regression analysis:

 The variables used in the model should be relevant and reliable.
 The relationship between variables should be linear, not non-linear.
 The variables should follow a normal distribution.
 The variance should remain constant at all levels of the predicted variable.

Benefits of Multiple Regression Analysis (Simple Sentences):

● Multiple regression helps analyze several predictor variables at the same time.
● It improves reliability by using more than one independent variable.
● It reduces dependence on a single variable.
● It helps study more complex and meaningful hypotheses.

Bayesian Linear Regression:

Bayesian linear regression is a method in which the model parameters are treated as probability distributions
rather than fixed values. The dependent variable is expressed as a weighted sum of independent variables,
and the output is assumed to follow a normal (Gaussian) distribution. This method uses prior distribution and

Department of CSE-AIML Machine Learning 24BEAML405

observed data to calculate the posterior distribution of the regression coefficients. Conjugate priors help
simplify the calculations and obtain analytical solutions. Bayesian regression is especially useful when the
dataset is small or uncertain, and its main goal is to estimate the posterior distribution of the model
parameters for better prediction.

Bayesian Linear Regression – Posterior and Mathematical Model

In Bayesian linear regression, the posterior distribution is calculated using Bayes’ theorem:

Here, the posterior is the updated probability of the model parameters after observing data. The prior is the
initial belief about the parameters before seeing the data. The likelihood represents the probability of
observing the data given the parameters. The normalization term ensures the total probability equals one.
This is based on Bayes’ theorem:

In Bayesian regression, we start with prior estimates of parameters and update them as more data becomes
available. As the number of data points increases, the posterior estimates become more accurate and
approach the values obtained using Ordinary Least Squares (OLS).

The linear model is expressed as:

where w represents the weights and x represents the input variables. In Bayesian regression, the output y is
assumed to follow a Gaussian (normal) distribution:

This means the output is probabilistic rather than a single fixed value. Bayesian Ridge Regression also
assumes a prior distribution for weights:

Department of CSE-AIML Machine Learning 24BEAML405

 Thus, Bayesian linear regression combines prior knowledge and
observed data to estimate the posterior distribution of model
parameters, improving prediction accuracy as more data becomes available.

Real-Life Applications of Bayesian Linear Regression:

● Bayesian regression uses prior knowledge to improve predictions. For example, the sales of a new
product can be predicted using the sales data of similar products.

● When new sales data becomes available, the prior is updated to improve future predictions.

● It helps make better forecasts by combining prior knowledge and new data.

● It also helps prevent overfitting by regularizing the model when many variables are present, such as
season, holidays, and promotions.
Advantages of Bayesian Regression
● It works well even when the dataset is small.

● It is suitable for online learning, where data is updated continuously.

● It is mathematically strong and provides reliable predictions.

● It does not require complete data before training.

Disadvantages of Bayesian Regression

● The inference process can be slow and time-consuming.

● It is less useful when a large amount of data is available.

● In such cases, traditional regression methods may be more efficient.

Implementation of Bayesian Regression Using Python

Bayesian Ridge Regression can be implemented using the scikit-learn library in Python. It can be applied to
regression problems such as predicting house prices using datasets like the Boston Housing dataset. The
performance of the model is evaluated using the R² score, where a value close to 1 indicates a good model,
and a value near 0 or negative indicates poor prediction.
Important parameters of Bayesian Ridge Regression:

● n_iter: Number of iterations used to train the model. The default value is 100.

● tol: Tolerance value used to check convergence of the model. The default value is 1x10-3.

Department of CSE-AIML Machine Learning 24BEAML405

● alpha_1: Shape parameter of the Gamma distribution for alpha. The default value is 1x10-6.

● alpha_2: Inverse scale parameter for alpha. The default value is 1x10-6.

● lambda_1: Shape parameter of the Gamma distribution for lambda. The default value is 1x10-6

● lambda_2: Inverse scale parameter for lambda. The default value is 1x10-6.

These parameters help control the learning and regularization of the Bayesian Ridge Regression model.

Gradient Descent:

Gradient Descent is an optimization algorithm used to minimize the cost function. It works by repeatedly
adjusting the model parameters such as weights and biases. The adjustments are made in the direction that
reduces the error the most. It gradually reduces the difference between predicted and actual values. This
process continues until the model reaches the optimal parameter values.

Gradient Descent – Simple Explanation (Exam-Oriented)

Gradient Descent can be understood using the example of walking down a hill to reach the lowest point.
Initially, you start at the top of the hill, which is similar to starting with random parameter values. Then, you
check the slope to find the steepest downward direction, which represents calculating the gradient. Next, you
take a step in that direction, which means updating the model parameters. This process is repeated until you
reach the lowest point, where the error is minimum. Thus, gradient descent helps find the minimum value of
the loss function.

Learning Rate
Learning rate is a hyperparameter that controls the size of steps taken during gradient descent. It determines
how fast or slow the model learns.

Department of CSE-AIML Machine Learning 24BEAML405

● If the learning rate is too small, the model learns very slowly and takes more time to reach the minimum.

● If the learning rate is too large, the model may overshoot the minimum and fail to converge, causing
oscillations or instability.

● Choosing the correct learning rate helps the model converge faster and improves training efficiency.

Techniques to Handle Gradient Problems

● Weight Regularization: Helps keep weights in a proper range and improves learning.
● Gradient Clipping: Limits gradient values to prevent very large updates.
● Batch Normalization: Normalizes inputs to improve stability and reduce gradient problems.

Mathematics Behind Gradient Descent

We consider a linear regression model with one input feature.
Step 1: Write the Linear Model Equation
The predicted output is:

Where:

Department of CSE-AIML Machine Learning 24BEAML405

● w = weight (slope)
● b = bias (intercept)
● ŷ = predicted output

Step 2: Define the Loss Function (Error Function)

We use Mean Squared Error (MSE).
For one data point:

Purpose:
This function measures how wrong the prediction is.
Step 3: Find Gradient with Respect to Weight w
We differentiate the loss function.

Step 4: Find Gradient with Respect to Bias b

Department of CSE-AIML Machine Learning 24BEAML405

Step 5: Apply Gradient Descent Update Rule
Gradient Descent updates parameters to reduce error.
Update formula:

Gradient Descent Update:

Once the gradients are calculated we update the parameters w in the direction opposite to the gradient (to
minimize the loss function):
1. For positive gradient: 2. For negative gradient

Department of CSE-AIML Machine Learning 24BEAML405

Step 6: Repeat Until Error is Minimum
Repeat steps:

1. Predict output
2. Calculate error
3. Calculate gradients
4. Update w and b
Until loss becomes very small.
Step-by-Step Algorithm

Department of CSE-AIML Machine Learning 24BEAML405

Types of Gradients Descent:
1. Batch Gradient Descent
Batch Gradient Descent uses the entire training dataset to compute the gradient in each iteration.
It calculates the average gradient of all data points and updates the parameters once per iteration.

The update rule for batch gradient descent is:

Department of CSE-AIML Machine Learning 24BEAML405

Algorithmn
● Computes the gradient using all training examples.
● Averages the gradient over the full dataset.
● Updates theta once per epoch.
● Suitable for small to medium datasets.
Advantages:
● Gives accurate and stable gradient.
● Works well for small datasets.
● Suitable for smooth and convex error functions.
Disadvantages:
● Slow for large datasets.
● Requires high memory.
● Computationally expensive.
When to use:

● When dataset size is small or medium.

● When accuracy and stability are important.

2. Stochastic Gradient Descent (SGD)

Stochastic Gradient Descent uses only one training example to update parameters in each iteration.

Department of CSE-AIML Machine Learning 24BEAML405

The update rule for SGD is:

Algorithm

● Updates theta using one example at a time.

● Leads to faster but noisier updates.
● Useful for online learning and large datasets.
● More sensitive to learning rate.

Advantages:

● Faster than batch gradient descent.

● Requires less memory.
● Suitable for very large datasets.
● Can escape local minima.

Disadvantages:

● Gradient updates are noisy.

● May oscillate around minimum.
● Less stable convergence.

When to use:

● When dataset is large.

● For online learning problems.

Department of CSE-AIML Machine Learning 24BEAML405

3. Mini-Batch Gradient Descent
Mini-batch Gradient Descent uses a small group of training examples (batch) to update parameters.

Update rule for Mini-Batch Gradient Descent is:

● Splits data into mini-batches like 32 samples.

● Shuffles data for better generalization.
● Combines speed of SGD with stability of Batch GD.
● Supports parallel computation like GPUs.
● Splits data into mini-batches like 32 samples.

● Shuffles data for better generalization.

● Combines speed of SGD with stability of Batch GD.

● Supports parallel computation like GPUs.

Algorithms

● Splits data into mini-batches like 32 samples.

Department of CSE-AIML Machine Learning 24BEAML405

● Shuffles data for better generalization.
● Combines speed of SGD with stability of Batch GD.
● Supports parallel computation like GPUs.

Advantages:

● Faster than batch gradient descent.

● More stable than SGD.
● Efficient and widely used in practice.

Disadvantages:

● Requires selection of batch size.

● Slightly noisy updates.

When to use:

● Most commonly used method in deep learning.

● Suitable for large datasets.

4. Momentum-Based Gradient Descent

Momentum Gradient Descent adds a fraction of previous update to the current update.

Department of CSE-AIML Machine Learning 24BEAML405

Implementation

● Maintains a velocity vector v to smooth updates.

● gamma controls how much past gradients influence current step.
● Helps in faster convergence especially in noisy loss surfaces.
● Common in deep learning frameworks like TensorFlow and PyTorch
Advantages:

● Faster convergence.

● Reduces oscillations.

● Helps move quickly in correct direction.

Disadvantages:

● Requires tuning momentum parameter.

When to use:

● When gradient descent is slow.

● For deep neural networks.

5. Adagrad (Adaptive Gradient)

Adagrad adjusts the learning rate automatically based on past gradients.

Advantages:
● No need to manually tune learning rate.
● Good for sparse data.

Department of CSE-AIML Machine Learning 24BEAML405

Disadvantages:
● Learning rate becomes very small over time.
● May stop learning early.

6. RMSprop (Root Mean Square Propagation)

RMSprop uses a moving average of squared gradients to adjust learning rate.
Advantages:
● Prevents learning rate from becoming too small.
● Faster convergence than Adagrad.
Disadvantages:
● Requires parameter tuning.

7. Adam (Adaptive Moment Estimation)

Adam combines Momentum and RMSprop.
It uses:
● Moving average of gradients
● Moving average of squared gradients
Advantages:
● Fast convergence
● Stable and efficient
● Most widely used optimizer
Disadvantages:
● Slightly more complex
When to use:
● Best choice for most deep learning applications.

Comparison between the variants of Gradient Descent

Variant Data Used Convergence Memory Usage Efficiency Key Advantage

Batch Gradient Entire dataset Stable but slow High (entire Computationally Stable convergence,
Descent dataset) expensive global view of data

Stochastic Gradient One example Fast but noisy Low (one Less efficient Faster convergence,
Descent per iteration example) good for online learning

Mini-Batch Gradient Mini-batch of Faster and Medium (mini- Efficient, Balance of speed and
Descent data smoother batch) parallelizable stability

Department of CSE-AIML Machine Learning 24BEAML405

Variant Data Used Convergence Memory Usage Efficiency Key Advantage

Momentum-Based Entire dataset Faster and Medium (like Efficient with Accelerated
Gradient Descent or mini-batch smoother Mini-Batch) momentum convergence, smooth
updates

Comparison between Batch Gradient Descent and Stochastic Gradient Descent s

Aspect Batch Gradient Descent Stochastic Gradient Descent

(SGD)

Data Processing Uses the whole training dataset to Uses a single training sample to
compute the gradient. compute the gradient.

Convergence Speed Slower, takes longer to converge. Faster, converges quicker due to
frequent updates.

Convergence Accuracy More accurate, gives precise Less accurate due to noisy
gradient estimates. gradient estimates.

Computational and Requires significant computation Requires less computation and

Memory Requirements and memory. memory.

Optimization of Non- Can get stuck in local minima. Can escape local minima and
Convex Functions find the global minimum.

Suitability for Large Not ideal for very large datasets due Can handle large datasets
Datasets to slow computation. effectively.

Nature Deterministic: Same result for the Stochastic: Results can vary with
same initial conditions. different initial conditions.

Learning Rate Fixed learning rate. Learning rate can be adjusted

dynamically.

Shuffling of Data No need for shuffling. Requires shuffling of data before

each epoch.

Overfitting Can overfit if the model is too Can reduce overfitting due to
complex. more frequent updates.

Escape Local Minima Cannot escape shallow local minima. Can escape shallow local minima
more easily.

Computational Cost High due to processing the entire Low due to processing one
dataset at once. sample at a time.

Final Solution Tends to converge to the global May converge to a local

Department of CSE-AIML Machine Learning 24BEAML405

 minimum for convex loss functions. minimum or saddle point.

Linear Classification Models

Linear classification models are supervised machine learning algorithms used to classify data into different
categories. They separate classes using a straight decision boundary such as a line, plane, or hyperplane
based on weighted input features. Common examples include Logistic Regression, Linear Support Vector
Machine (SVM), and Perceptron. These models are simple, fast to train, efficient, and easy to interpret.

Discriminant Functions
A discriminant is a function which takes the input vector X and assigns it to one of the classes Ck. When the
decision surfaces are hyperplanes, we call them as linear discriminants.

Two Classes:
The simplest representation of linear discriminant function is

The input vector will be assigned to the class C1 if y(X)≥0 and C2 otherwise. The decision boundary will
be defined by the relation y(X)=0. Let XA, XB are where W is called the weight vector and W0 is bias. The
negative of the bias is called threshold the two points lying on the decision surface, then y(XA) =
y(XB)=0 and hence
WT(XA−XB) = 0. This means that the weight vector W is orthogonal to every vector in the decision surface,
i.e. W determines the direction of the decision surface. Let X be a point on the decision surface which is
closest to the origin. Thie means that this point can be represented as X=αW as the perpendicular weight
vector W will be the direction in which we will have the smallest vector passing through origin and the
hyperplace. As X lies on the hyperplace, we have y(X)=WTX+W0=0, i.e. WTX+W0=αWTW+W0=0 and
−w0 −w0
hence α=−W0/||W||2. The distance of the point X from the origin is ||X||=||αW||= ||W||=
¿∨W ∨¿ 2 ¿∨W ∨¿ 2
−w0
. Hence, the normal distance of the hyperplane from the origin is given by . This means that
¿∨W ∨¿ 2
the bias parameter W0 decides the location of the decision surface.
Another thing to note is that the value of y(X) gives the signed measure of the perpendicular distance r of
point X from the decision surface. Let X⊥ be the orthogonal projection of X on the decision surface, then

Department of CSE-AIML Machine Learning 24BEAML405

 W
as is the unit vector which is perpendicular to the decision surface. Multiplying both sides by WT and
‖W ‖
adding W0, we have

The geometrical representation of the result is shown in the below figure.

~
We can combine the bias into weight vector with the new weight vector represented as W = (W0, W) with
~
the updated input vector as X =(1,X) and the updated equation

Multiple Classes
A K-class discriminant (K>2) can be built by combining multiple two-class discriminant
functions. However, there are some limitations to it. Consider a one-versus-the-

Department of CSE-AIML Machine Learning 24BEAML405

rest classifier which separates a particular class Ck from points not in the class. For
a K-class problem, we will need a total of K−1 classifiers of such kind. The result of
this kind of classifiers is shown in the below figure. As shown in the left image, the
green region is classified both as class C1 and C2 and hence this region
is ambiguously classified.

Fig: Attempting to construct a k class discriminant from a set of 2 class discriminants leads to ambiguous
regions, shown in green. On the left is an example involving the use of 2 discriminants designed to
distinguish points in class Ck. On the right is an example involving three discriminant functions each of
which is used to separate a pair of classes Ck and Cj
Another approach is to use K(K−1)/2 discriminant functions, one for each class pair of
classes. This is called as one-versus-one classifier. Each point then can be classified according to the
majority vote amongst the discriminant functions. The right-hand side image in the above figure shows this
approach. The green region has one vote for each of the classes C1, C2 and C3 and hence being ambiguous.
These ambiguous classification regions can be avoided by using a single K-class discriminant comprising
of K linear functions of the form

The class assignment will be done on the basis of: assign a point X to class Ck if yk (X) > yj(X) for all j ≠
k. The decision boundary between class Ck and Cj is given by yk(X)=yj(X) and corresponds
to a (D−1)-dimensional hyperplane defined by

The decision boundary is same as the one for a two-class case. The decision region of such a discriminant
function is always singly connected and convex. For two points XA, XB which lie in the decision region Rk,
any point X which lies on the line connecting XA, XB can be expressed in the form

From the linearity of discriminant function

Department of CSE-AIML Machine Learning 24BEAML405

As both XA, XB lie inside Rk, it follows yk(XA)>yj(XA) and yk(XB)>yj(XB) for all j≠k. From this, we
get yk( X ) > yj( X ), and hence X lies in the region Rk. This means that the region Rk is singly connected and
convex.

Perceptron

The Perceptron was introduced by Frank Rosenblatt in 1957. It is based on the MCP neuron model. The
Perceptron is a supervised learning algorithm used for binary classification. It learns by processing one
training sample at a time and updates its weights accordingly.

Basic Components of Perceptron

The basic components of a perceptron are:

1. Input Layer: The input layer consists of one or more input neurons, which receive input signals from
the external world or from other layers of the neural network.

2. Weights: Each input neuron is associated with a weight, which represents the strength of the
connection between the input neuron and the output neuron.

3. Bias: A bias term is added to the input layer to provide the perceptron with additional flexibility in
modeling complex patterns in the input data.

4. Activation Function: The activation function determines the out put of the perceptron based on the
weighted sum of the inputs and the bias term. Common activation functions used in perceptrons include the
step function, sigmoid function, and ReLU function.

5. Output: The output of the perceptron is a single binary value, either 0 or 1, which indicates the class
or category to which the input data belongs.

6. Training Algorithm: The perceptron is typically trained using a supervised learning algorithm such as
the perceptron learning algorithm or backpropagation. During training, the weights and biases of the

Department of CSE-AIML Machine Learning 24BEAML405

perceptron are adjusted to minimize the error between the predicted output and the true output for a given set
of training examples.

Characteristics of the Perceptron Model

The following are the characteristics of a Perceptron Model:
● It is a machine learning algorithm that uses supervised learning of binary classifiers.

● In Perceptron, the weight coefficient is automatically learned.

● Initially, weights are multiplied with input features, and then the decision is made whether the neuron is
fired or not.
● The activation function applies a step rule to check whether the function is more significant than zero.

● The linear decision boundary is drawn, enabling the distinction between the two linearly separable classes
+1 and -1.
● If the added sum of all input values is more than the threshold value, it must have an output signal;
otherwise, no output will be shown.

Working of Perceptron Model

A Perceptron accepts inputs, moderates them with certain weight values, then applies the transformation
function to output the final result. The image below shows a Perceptron with a Boolean output.

Department of CSE-AIML Machine Learning 24BEAML405

A Boolean output is based on inputs such as salaried, married, age, past credit profile,
etc. It has only two values: Yes and No or True and False. The summation function “∑”
multiplies all inputs of “x” by weights “w” and then adds them up as follows:

Activation Functions of Perceptron

The activation function applies a step rule (convert the numerical output into +1 or -1) to check if
the output of the weighting function is greater than zero or not.

For example:
If ∑ wixi > 0 🡺 then final output “o” = 1 (issue bank loan)
Else, final output “o” = -1 (deny bank loan)
Step function gets triggered above a certain value of the neuron output; else it outputs zero. Sign Function
outputs +1 or -1 depending on whether neuron output is greater than zero or not. Sigmoid is the S-curve and
outputs a value between 0 and 1.
For example:
If ∑ wixi > 0 🡺 then final output “o” = 1 (issue bank loan)
Else, final output “o” = -1 (deny bank loan)
Step function gets triggered above a certain value of the neuron output; else it outputs zero. Sign Function
outputs +1 or -1 depending on whether neuron output is greater than zero or not. Sigmoid is the S-curve and
outputs a value between 0 and 1.

Output of Perceptron
Perceptron with a Boolean output:

Department of CSE-AIML Machine Learning 24BEAML405

Inputs: x1………………..xn

Output: o(x1……………..xn)

Weights: wi 🡺 contribution of input xi to the Perceptron output;

w0 🡺 bias or threshold

If ∑w.x > 0, output is +1, else -1. The neuron gets triggered only when weighted input
reaches a certain threshold value.

n output of +1 specifies that the neuron is triggered. An output of -1 specifies that the neuron did not get
triggered.
“sgn” stands for sign function with output +1 or -1.
Error in Perceptron
In the Perceptron Learning Rule, the predicted output is compared with the known output. If it does not
match, the error is propagated backward to allow weight adjustment to happen.

Perceptron: Decision Function

A decision function φ(z) of Perceptron is defined to take a linear combination of x and w vectors.

The value z in the decision function is given by:

Department of CSE-AIML Machine Learning 24BEAML405

The decision function is +1 if z is greater than a threshold θ, and it is -1 otherwise.

This is the Perceptron algorithm.

Bias Unit

For simplicity, the threshold θ can be brought to the left and represented as w0x0, where w0= -θ and x0= 1.

The value w0 is called the bias unit.

The decision function then becomes:

Output:
The figure shows how the decision function squashes wTx to either +1 or -1 and how it can be used to
discriminate between two linearly separable classes.

Department of CSE-AIML Machine Learning 24BEAML405

Types of Perceptron Models
There are two main types of perceptron models:
1. Single-Layer Perceptron
A single-layer perceptron is the simplest type of artificial neural network. It has only one layer between the
input and output. It is a feed-forward network and uses a threshold activation function. It is used to classify
linearly separable data. It can learn only linear patterns and produces binary output.
2. Multi-Layer Perceptron (MLP)
A multi-layer perceptron has one or more hidden layers between the input and output layers.
In the forward stage, data moves from the input layer to the output layer through hidden layers.
In the backward stage, weights and bias are updated to reduce errors using backpropagation.
It has higher processing power than a single-layer perceptron. It can learn both linear and non-linear
patterns and can implement logic gates such as AND, OR, XOR, XNOR, and NOR.
Advantages:
● A multi-layered perceptron model can solve complex non-linear problems.
● It works well with both small and large input data.
● Helps us to obtain quick predictions after the training.
● Helps us obtain the same accuracy ratio with big and small data.
Disadvantages:
● In multi-layered perceptron model, computations are time-consuming and complex.
● It is tough to predict how much the dependent variable affects each independent variable.
● The model functioning depends on the quality of training.

Build and Train Single Layer Perceptron Model

A Single Layer Perceptron is the simplest type of neural network with one layer of neurons connected to inputs. It works
by calculating a weighted sum of inputs and applying an activation function to produce the output.

● Initialize Parameters: Set weights and bias.

Department of CSE-AIML Machine Learning 24BEAML405

● Forward Pass: Compute the weighted sum of inputs.

● Activation: Use a step function to determine the output.

● Training: Adjust weights using the Perceptron Learning Rule based on errors.

Probabilistic Models:

Department of CSE-AIML Machine Learning 24BEAML405

Probabilistic models are machine learning models that use probability to make predictions. They handle
uncertainty in data and provide predictions based on likelihood. These models learn patterns from data and
apply them to new, unseen data. They are used in applications such as image recognition, speech
recognition, natural language processing, and recommendation systems.

Types of Probabilistic Models:

Probabilistic models are classified into three types:

1. Generative Models
Generative models learn the joint probability of input and output variables. They can generate new data
similar to the training data. These models are used in applications such as image generation, speech
synthesis, language translation, and text generation.

2. Discriminative Models
Discriminative models learn the probability of the output given the input. They focus on finding a decision
boundary to separate different classes. These models are used for accurate classification tasks such as image
recognition, speech recognition, and sentiment analysis.

3. Graphical Models
Graphical models use graphs to represent relationships between variables. They show how variables depend
on each other. These models are used in image recognition, natural language processing, and causal analysis.

Logistic regression

Logistic regression is a statistical analysis method to predict a binary outcome, such as yes or no, based
on prior observations of a data set.

A logistic regression model predicts a dependent data variable by analyzing the relationship between one
or more existing independent variables. For example, a logistic regression could be used to predict
whether a political candidate will win or lose an election or whether a high school student will be
admitted or not to a particular college. These binary outcomes allow straightforward decisions between
two alternatives.

A logistic regression model can take into consideration multiple input criteria. In the case of college
acceptance, the logistic function could consider factors such as the student's grade point average, SAT
score and number of extracurricular activities. Based on historical data about earlier outcomes involving
the same input criteria, it then scores new cases on their probability of falling into one of two outcome
categories.

Logistic Regression:

Department of CSE-AIML Machine Learning 24BEAML405

Logistic Regression is a supervised machine learning algorithm used for classification tasks. Unlike linear
regression, which predicts continuous values, logistic regression predicts the probability that an input
belongs to a particular class. It is mainly used for binary classification problems where the output has two
possible categories, such as Yes/No, True/False, or 0/1. It uses a sigmoid function to convert the input into a
probability value between 0 and 1.

Types of Logistic Regression

Logistic regression is classified into three types based on the type of output variable:

1. Binomial Logistic Regression:

This type is used when the dependent variable has only two possible categories, such as Yes/No, Pass/Fail,
or 0/1. It is the most commonly used type and is applied in binary classification problems.

2. Multinomial Logistic Regression:

This type is used when the dependent variable has three or more categories without any specific order. For
example, classifying animals as cat, dog, or sheep. It extends binary logistic regression to handle multiple
classes.

3. Ordinal Logistic Regression:

This type is used when the dependent variable has three or more categories with a natural order, such as low,
medium, and high. It considers the order of categories while making predictions.

Assumptions of Logistic Regression:

● Independent observations: Each data point should be independent. There should be no relationship or
dependence between samples.
● Binary dependent variable: The output variable should have only two categories, such as 0/1 or
Yes/No. For more than two categories, Softmax is used.
● Linearity with log odds: There should be a linear relationship between input variables and the log odds
of the output. This means the input variables affect the prediction in a linear way.
● No outliers: The dataset should not contain extreme outliers. Outliers can affect the accuracy of the
model.
● Large sample size: Logistic regression works better with a large amount of data. A large sample size
gives more reliable results.

Department of CSE-AIML Machine Learning 24BEAML405

Sigmoid Function:

1. The sigmoid function is an important part of logistic regression. It converts the model output into a
probability value between 0 and 1.
2. It takes any real number as input and maps it to a value between 0 and 1. It forms an S-shaped curve
called the sigmoid curve or logistic curve.
3. In logistic regression, a threshold value, usually 0.5, is used to classify the input. If the sigmoid output is
greater than or equal to 0.5, the input is classified as Class 1. If the output is less than 0.5, the input is
classified as Class 0.
4. This function helps convert continuous values into class labels for classification.

Assumptions of Logistic Regression

Understanding the assumptions behind logistic regression is important to ensure the model is applied
correctly, main assumptions are:
1. Independent observations: Each data point is assumed to be independent of the others means there
should be no correlation or dependence between the input samples.
2. Binary dependent variables: It takes the assumption that the dependent variable must be binary, means
it can take only two values. For more than two categories SoftMax functions are used.
3. Linearity relationship between independent variables and log odds: The model assumes a linear
relationship between the independent variables and the log odds of the dependent variable which means
the predictors affect the log odds in a linear way.
4. No outliers: The dataset should not contain extreme outliers as they can distort the estimation of the
logistic regression coefficients.
5. Large sample size: It requires a sufficiently large sample size to produce reliable and stable results.
Understanding Sigmoid Function
1. The sigmoid function is a important part of logistic regression which is used to convert the raw output of
the model into a probability value between 0 and 1.
2. This function takes any real number and maps it into the range 0 to 1 forming an "S" shaped curve called
the sigmoid curve or logistic curve. Because probabilities must lie between 0 and 1, the sigmoid function is
perfect for this purpose.
3. In logistic regression, we use a threshold value usually 0.5 to decide the class label.

● If the sigmoid output is same or above the threshold, the input is classified as Class 1.
● If it is below the threshold, the input is classified as Class 0.
This approach helps to transform continuous input values into meaningful class predictions.
How does Logistic Regression work?
Logistic regression model transforms the linear regression function continuous value output into categorical
value output using a sigmoid function which maps any real-valued set of independent variables input into a
value between 0 and 1. This function is known as the logistic function.
Suppose we have input features represented as a matrix:

Department of CSE-AIML Machine Learning 24BEAML405

and the dependent variable is Y having only binary value i.e 0 or 1.

then, apply the multi-linear function to the input variables X.

Here xi is the ith observation of X, wi=[w1,w2,w3,⋯,wm] is the weights or Coefficient and b is the bias term
also known as intercept. Simply this can be represented as the dot product of weight and bias.

At this stage, z is a continuous value from the linear regression. Logistic regression then applies the sigmoid
function to z to convert it into a probability between 0 and 1 which can be used to predict the class.
Now we use the sigmoid function where the input will be z and we find the probability between 0 and 1. i.e.
predicted y.

Department of CSE-AIML Machine Learning 24BEAML405

As shown above the sigmoid function converts the continuous variable data into the probability i.e between
0 and 1.

● σ(z) tends towards 1 as z→∞

● σ(z) tends towards 0 as z→−∞
● σ(z) is always bounded between 0 and 1
where the probability of being a class can be measured as:
Logistic Regression Equation and Odds:
It models the odds of the dependent event occurring which is the ratio of the probability of the event to the
probability of it not occurring:

Taking the natural logarithm of the odds gives the log-odds or logit:

then the final logistic regression equation will be:

This formula represents the probability of the input belonging to Class 1.

Likelihood Function for Logistic Regression

The goal is to find weights w and bias b that maximize the likelihood of observing the data.
For each data point i
● for y=1, predicted probabilities will be: p(X;b,w) =p(x)
● for y=0 The predicted probabilities will be: 1-p(X;b,w) = 1−p(x)

Department of CSE-AIML Machine Learning 24BEAML405

Taking natural logs on both sides:

This is known as the log-likelihood function.

Gradient of the log-likelihood function
To find the best w and b we use gradient ascent on the log-likelihood function. The gradient with respect to
each weight wj is:

Terminologies involved in Logistic Regression

Here are some common terms involved in logistic regression:
1. Independent Variables: These are the input features or predictor variables used to make predictions
about the dependent variable.

Department of CSE-AIML Machine Learning 24BEAML405

2. Dependent Variable: This is the target variable that we aim to predict. In logistic regression, the
dependent variable is categorical.
3. Logistic Function: This function transforms the independent variables into a probability between 0
and 1 which represents the likelihood that the dependent variable is either 0 or 1.
4. Odds: This is the ratio of the probability of an event happening to the probability of it not happening.
It differs from probability because probability is the ratio of occurrences to total possibilities.
5. Log-Odds (Logit): The natural logarithm of the odds. In logistic regression, the log-odds are
modeled as a linear combination of the independent variables and the intercept.
6. Coefficient: These are the parameters estimated by the logistic regression model which shows how
strongly the independent variables affect the dependent variable.
7. Intercept: The constant term in the logistic regression model which represents the log-odds when all
independent variables are equal to zero.
8. Maximum Likelihood Estimation (MLE): This method is used to estimate the coefficients of the
logistic regression model by maximizing the likelihood of observing the given data.
Implementation for Logistic Regression
Now, let's see the implementation of logistic regression in Python. Here we will be implementing two main
types of Logistic Regression:
1. Binomial Logistic regression:
In binomial logistic regression, the target variable can only have two possible values such as "0" or "1",
"pass" or "fail". The sigmoid function is used for prediction.

2. Multinomial Logistic Regression:

Target variable can have 3 or more possible types which are not ordered i.e types have no quantitative
significance like “disease A” vs “disease B” vs “disease C”.
In this case, the softmax function is used in place of the sigmoid function. Softmax function for K classes
will be:

Here K represents the number of elements in the vector z and i,j iterates over all the elements in the vector.
Then the probability for class c will be:

How to Evaluate Logistic Regression Model?

Evaluating the logistic regression model helps assess its performance and ensure it generalizes well to new,
unseen data. The following metrics are commonly used:
1. Accuracy: Accuracy provides the proportion of correctly classified instances.

Department of CSE-AIML Machine Learning 24BEAML405

2. Precision: Precision focuses on the accuracy of positive predictions.

3. Recall (Sensitivity or True Positive Rate): Recall measures the proportion of correctly predicted
positive instances among all actual positive instances.

4. F1 Score: F1 score is the harmonic mean of precision and recall.

[Link] Under the Receiver Operating Characteristic Curve (AUC-ROC): The ROC curve plots the true
positive rate against the false positive rate at various thresholds. AUC-ROC measures the area under this
curve which provides an aggregate measure of a model's performance across different classification
thresholds.
6. Area Under the Precision-Recall Curve (AUC-PR): Similar to AUC-ROC, AUC-PR measures the area
under the precision-recall curve helps in providing a summary of a model's performance across different
precision-recall trade-offs.

Comparison between Linear regression and Logistic regression

Aspect Linear Regression Logistic Regression

Linear regression is used to Logistic regression is used to

predict the continuous dependent predict the categorical dependent
variable using a given set of variable using a given set of
Definition independent variables. independent variables.

It is used for solving regression It is used for solving classification

Problem Type problem. problems.

In this we predict the value of In this we predict values of

Output Type continuous variables. categorical variables.

Curve/Model In this we find best fit line. In this we find S-Curve.

Department of CSE-AIML Machine Learning 24BEAML405

 Aspect Linear Regression Logistic Regression

Fitting

Least square estimation method Maximum likelihood estimation

Estimation is used for estimation of method is used for estimation of
Method accuracy. accuracy.

Output The output must be continuous Output must be categorical value

Example value such as price, age etc. such as 0 or 1, Yes or No, etc.

It required linear relationship

Relationship between dependent and It not required linear relationship.
Requirement independent variables.

There may be collinearity There should be little to no

between the independent collinearity between independent
Collinearity variables. variables.

Naive Bayes Classifiers

Naive Bayes is a machine learning classification algorithm that predicts the category of a data point using
probability. It assumes that all features are independent of each other. Naive Bayes performs well in many
real-world applications such as spam filtering, document categorisation and sentiment analysis.

Department of CSE-AIML Machine Learning 24BEAML405

Here:

● Original data has two classes: green circles (y = 1) and red squares (y = 2).
● Estimate probability distribution along the first dimension i.e P(x1∣y=1),P(x1∣y=2)P(x1∣y=1),P(x1
∣y=2)
● Estimate probability distribution along the second dimension i.e P(x2∣y=1),P(x2∣y=2)P(x2∣y=1),P(x2
∣y=2)
● Combine both dimensions using conditional independence i.e P(x∣y)=∏αP(xα∣y)P(x∣y)=∏αP(xα
∣y)
Key Features of Naive Bayes Classifiers
The main idea behind the Naive Bayes classifier is to use Bayes' Theorem to classify data based on the
probabilities of different classes given the features of the data. It is used mostly in high-dimensional text
classification
● The Naive Bayes Classifier is a simple probabilistic classifier and it has very few number of
parameters which are used to build the ML models that can predict at a faster speed than other classification
algorithms.
● It is a probabilistic classifier because it assumes that one feature in the model is independent of
existence of another feature. In other words, each feature contributes to the predictions with no relation
between each other.
● Naive Bayes Algorithm is used in spam filtration, Sentimental analysis, classifying articles and many
more.
Why it is Called Naive Bayes?
It is named as "Naive" because it assumes the presence of one feature does not affect other features. The
"Bayes" part of the name refers to its basis in Bayes’ Theorem.
Consider a fictional dataset that describes the weather conditions for playing a game of golf. Given the
weather conditions, each tuple classifies the conditions as fit(“Yes”) or unfit(“No”) for playing golf. Here is
a tabular representation of our dataset.

Outlook Temperature Humidity Windy Play Golf

0 Rainy Hot High False Yes

1 Rainy Hot High True No

2 Overcast Hot High False Yes

3 Sunny Mild High False No

4 Sunny Cool Normal False Yes

5 Sunny Cool Normal True No

Department of CSE-AIML Machine Learning 24BEAML405

 Outlook Temperature Humidity Windy Play Golf

6 Overcast Cool Normal True Yes

7 Rainy Mild6 High False No

8 Rainy Cool Normal False Yes

9 Sunny Mild Normal False Yes

10 Rainy Mild Normal True Yes

11 Overcast Mild High True Yes

12 Overcast Hot Normal False Yes

13 Sunny Mild High True No

The dataset is divided into two parts i.e feature matrix and the response vector.

● Feature matrix contains all the vectors(rows) of dataset in which each vector consists of the value
of dependent features. In above dataset, features are ‘Outlook’, ‘Temperature’, ‘Humidity’ and ‘Windy’.
● Response vector contains the value of class variable (prediction or output) for each row of feature
matrix. In above dataset, the class variable name is ‘Play golf’.
Assumption of Naive Bayes
The fundamental Naive Bayes assumption is that each feature makes an:

● Feature independence: This means that when we are trying to classify something, we assume that
each feature (or piece of information) in the data does not affect any other feature.
● Continuous features are normally distributed: If a feature is continuous, then it is assumed to be
normally distributed within each class.
● Discrete features have multinomial distributions: If a feature is discrete, then it is assumed to have
a multinomial distribution within each class.
● Features are equally important: All features are assumed to contribute equally to the prediction of
the class label.
● No missing data: The data should not contain any missing values.
Introduction to Bayes' Theorem
Bayes’ Theorem provides a principled way to reverse conditional probabilities. It is defined as:
P(y∣X)=P(X∣y)⋅P(y)P(X)P(y∣X)=P(X)P(X∣y)⋅P(y)
Where:

Department of CSE-AIML Machine Learning 24BEAML405

● P(y∣X)P(y∣X): Posterior probability, probability of class yy given features XX
● P(X∣y)P(X∣y): Likelihood, probability of features XX given class yy
● P(y)P(y): Prior probability of class yy
● P(X)P(X): Marginal likelihood or evidence
Naive Bayes Working

1. Terminology

Consider a classification problem (like predicting if someone plays golf based on weather). Then:

● yy is the class label (e.g. "Yes" or "No" for playing golf)

● X=(x1,x2,...,xn)X=(x1,x2,...,xn) is the feature vector (e.g. Outlook, Temperature, Humidity, Wind)
A sample row from the dataset:
X=(Rainy, Hot, High, False),y=NoX=(Rainy, Hot, High, False),y=No
This represents:
What is the probability that someone will not play golf given that the weather is Rainy, Hot, High humidity
and No wind?

2. The Naive Assumption

The "naive" in Naive Bayes comes from the assumption that all features are independent given the class.
That is:
P(x1,x2,...,xn∣y)=P(x1∣y)⋅P(x2∣y)⋯P(xn∣y)P(x1,x2,...,xn∣y)=P(x1∣y)⋅P(x2∣y)⋯P(xn∣y)
Thus, Bayes' theorem becomes:
P(y∣x1,...,xn)=P(y)⋅∏i=1nP(xi∣y)P(X)P(y∣x1,...,xn)=P(X)P(y)⋅∏i=1nP(xi∣y)
Since the denominator is constant for a given input, we can write:
P(y∣x1,...,xn)∝P(y)⋅∏i=1nP(xi∣y)P(y∣x1,...,xn)∝P(y)⋅∏i=1nP(xi∣y)

3. Constructing the Naive Bayes Classifier

We compute the posterior for each class yy and choose the class with the highest probability:
y^=arg⁡max⁡yP(y)⋅∏i=1nP(xi∣y)y^=argmaxyP(y)⋅∏i=1nP(xi∣y)
This becomes our Naive Bayes classifier.

4. Example: Weather Dataset

Let’s take a dataset used for predicting if golf is played based on:

Department of CSE-AIML Machine Learning 24BEAML405

● Outlook: Sunny, Rainy, Overcast
● Temperature: Hot, Mild, Cool
● Humidity: High, Normal
● Wind: True, False

Example Input: X=(Sunny,Hot,Normal,False)X=(Sunny,Hot,Normal,False)

Goal: Predict if golf will be played (Yes or No).

5. Pre-computation from Dataset

Class Probabilities:
From dataset of 14 rows:

● P(Yes)=914P(Yes)=149
● P(No)=514P(No)=145
Conditional Probabilities (Tables 1–4):

6. Calculate Posterior Probabilities

For Class = Yes:
P(Yes | today)∝29⋅29⋅69⋅69⋅914P(Yes | today)∝92⋅92⋅96⋅96⋅149
P(Yes | today)≈0.02116P(Yes | today)≈0.02116
For Class = No:
P(No | today)∝35⋅25⋅15⋅25⋅514P(No | today)∝53⋅52⋅51⋅52⋅145
P(No | today)≈0.0068P(No | today)≈0.0068

7. Normalize Probabilities

To compare:
P(Yes | today)=0.021160.02116+0.0068≈0.756P(Yes | today)=0.02116+0.00680.021
16≈0.756

Department of CSE-AIML Machine Learning 24BEAML405

P(No | today)=0.00680.02116+0.0068≈0.244P(No | today)=0.02116+0.00680.0068
≈0.244

8. Final Prediction

Since:
P(Yes | today)>P(No | today)P(Yes | today)>P(No | today)
The model predicts: Yes (Play Golf)

Naive Bayes for Continuous Features

For continuous features, we assume a Gaussian distribution:

P(xi∣y)=12πσy2exp⁡(−(xi−μy)22σy2)P(xi∣y)=2πσy21exp(−2σy2(xi−μy)2)
Where:

● μyμy is the mean of feature xixi for class yy

● σy2σy2 is the variance of feature xixi for class yy
This leads to what is called Gaussian Naive Bayes.
Types of Naive Bayes Model
There are three types of Naive Bayes Model :

1. Gaussian Naive Bayes

In Gaussian Naive Bayes, continuous values associated with each feature are assumed to be distributed
according to a Gaussian distribution. A Gaussian distribution is also called Normal distribution When
plotted, it gives a bell shaped curve which is symmetric about the mean of the feature values as shown
below:

2. Multinomial Naive Bayes

Multinomial Naive Bayes is used when features represent the frequency of terms (such as word counts) in a
document. It is commonly applied in text classification, where term frequencies are important.

3. Bernoulli Naive Bayes

Bernoulli Naive Bayes deals with binary features, where each feature indicates whether a word appears or
not in a document. It is suited for scenarios where the presence or absence of terms is more relevant than
their frequency. Both models are widely used in document classification tasks
Advantages
● Easy to implement and computationally efficient.
● Effective in cases with a large number of features.

Department of CSE-AIML Machine Learning 24BEAML405

● Performs well even with limited training data.
● It performs well in the presence of categorical features.
● For numerical features data is assumed to come from normal distributions
Disadvantages
● Assumes that features are independent, which may not always hold in real-world data.
● Can be influenced by irrelevant attributes.
● May assign zero probability to unseen events, leading to poor generalization.
Applications
● Spam Email Filtering: Classifies emails as spam or non-spam based on features.
● Text Classification: Used in sentiment analysis, document categorization and topic classification.
● Medical Diagnosis: Helps in predicting the likelihood of a disease based on symptoms.
● Credit Scoring: Evaluates creditworthiness of individuals for loan approval.
● Weather Prediction: Classifies weather conditions based on various factors.

Decision tree algorithm

Decision tree learning is a widely used and efficient technique for classification. It represents the
classification model in the form of a tree called a decision tree. The tree is built using a top-down, greedy
divide-and-conquer approach, where all training examples start at the root and are split into smaller groups.
The data is recursively partitioned based on selected attributes, which are chosen using impurity measures
such as information gain. The partitioning process stops when all examples belong to the same class, no
attributes are left, or no training examples remain, and the final node becomes a leaf representing the class
label.

Principle:
The decision tree algorithm is based on a greedy approach that builds the tree step by step. It constructs the
tree in a top-down recursive divide-and-conquer manner. Initially, all the training samples are placed at the
root node. The samples are then split into smaller groups based on selected attributes, which are chosen
using measures such as information gain. The process stops when all samples belong to the same class, no
attributes remain for further splitting, or no samples are left, and the final node is assigned a class label using
majority voting if needed.

Aim: find a small tree consistent with the training examples

Idea: (recursively) choose "most significant" attribute as root of (sub)tree

Department of CSE-AIML Machine Learning 24BEAML405

A decision tree from the loan data

Decision nodes and leaf nodes (classes)

Use the decision tree

Department of CSE-AIML Machine Learning 24BEAML405

From a decision tree to a set of rules
roots, which is better?

Fig. (B) seems to be better

Learning Decision Trees: Ex2

Example Problem: decide whether to wait for a table at a restaurant, based on the following attributes:
1. Alternate: is there an alternative restaurant nearby?
2. Bar: is there a comfortable bar area to wait in?
3. Fri/Sat: is today Friday or Saturday?
4. Hungry: are we hungry?
5. Patrons: number of people in the restaurant (None, Some, Full)
6. Price: price range ($, $$, $$$)
7. Raining: is it raining outside?
8. Reservation: have we made a reservation?
9. Type: kind of restaurant (French, Italian, Thai, Burger)

Department of CSE-AIML Machine Learning 24BEAML405

10. Wait Estimate: estimated waiting time (0-10, 10-30, 30-60, >60)

● Feature (Attribute)-based representations Examples described by feature(attribute)

values

● (Boolean, discrete, continuous)

● E.g., situations where I will/won't wait for a table:

Classification of examples is positive (T) or negative (F)

Expressiveness
Decision trees can express any function of the input attributes. E.g., for Boolean functions, truth table

row → path to leaf:

Department of CSE-AIML Machine Learning 24BEAML405

 give th

● Trivially, there is a consistent decision tree for any training set with one path to leaf for each example

(unless f nondeterministic in x) but it probably won't generalize to new examples

● Refer to find more compact decision trees

Validation set: the available data is divided into three subsets,

● a training set,

● a validation set and

● a test set.

● A validation set is used frequently for estimating parameters in learning algorithms.

● In such cases, the values that give the best accuracy on the validation set are used as the final parameter

values.

● Cross-validation can be used for parameter estimating as well.

● Classification measures

● Accuracy is only one measure (error = 1-accuracy).

● Accuracy is not suitable in some applications.

● In text mining, we may only be interested in the documents of a particular topic, which are only a small

portion of a big document collection.

● In classification involving skewed or highly imbalanced data, e.g., network intrusion and

financial fraud detections, we are interested only in the minority class.

o High accuracy does not mean any intrusion is detected.
o E.g., 1% intrusion. Achieve 99% accuracy by doing nothing.

Department of CSE-AIML Machine Learning 24BEAML405

● The class of interest is commonly called the positive class, and the rest negative classes.

● Precision and recall measures Used in information retrieval and text classification.

● We use a confusion matrix to introduce them.

An Example

● This confusion matrix gives

● precision p = 100% and

● recall r = 1%

● because we only classified one positive example correctly and no negative examples wrongly.

Note: precision and recall only measure classification on the positive class.
Support vector machines (SVM)
Support Vector Machine or SVM is one of the most popular Supervised Learning algorithms,
which is used for Classification as well as Regression problems. However, primarily, it is
used for Classification problems in Machine Learning.

The goal of the SVM algorithm is to create the best line or decision boundary that can

Department of CSE-AIML Machine Learning 24BEAML405

segregate n-dimensional space into classes so that we can easily put the new data point in the
correct category in the future. This best decision boundary is called a hyperplane.

SVM chooses the extreme points/vectors that help in creating the hyperplane. These extreme
cases are called as support vectors, and hence algorithm is termed as Support Vector
Machine. Consider the below diagram in which there are two different categories that are
classified using a decision boundary or hyperplane:

Example: SVM can be understood with the example that we have used in the KNN classifier.
Suppose we see a strange cat that also has some features of dogs, so if we want a model that
can accurately identify whether it is a cat or dog, so such a model can be created by using the
SVM algorithm. We will first train our model with lots of images of cats and dogs so that it
can learn about different features of cats and dogs, and then we test it with this strange
creature. So as support vector creates a decision boundary between these two data (cat and
dog) and choose extreme cases (support vectors), it will see the extreme case of cat and dog.
On the basis of the support vectors, it will classify it as a cat. Consider the below diagram:

SVM algorithm can be used for Face detection, image classification, text categorization, etc.

Department of CSE-AIML Machine Learning 24BEAML405

Types of SVM
SVM can be of two types:

Linear SVM: Linear SVM is used for linearly separable data, which means if a dataset can
be classified into two classes by using a single straight line, then such data is termed as
linearly separable data, and classifier is used called as Linear SVM classifier.

Non-linear SVM: Non-Linear SVM is used for non-linearly separated data, which means if a
dataset cannot be classified by using a straight line, then such data is termed as non-linear
data and classifier used is called as Non-linear SVM classifier.
Hyperplane and Support Vectors in the SVM algorithm:
Hyperplane: There can be multiple lines/decision boundaries to segregate the classes in n-
dimensional space, but we need to find out the best decision boundary that helps to classify
the data points. This best boundary is known as the hyperplane of SVM.
The dimensions of the hyperplane depend on the features present in the dataset, which means
if there are 2 features (as shown in image), then hyperplane will be a straight line. And if
there are 3 features, then hyperplane will be a 2-dimension plane. We always create a
hyperplane that has a maximum margin, which means the maximum distance between the
data points.
Support Vectors:
The data points or vectors that are the closest to the hyperplane and which affect the position
of the hyperplane are termed as Support Vector. Since these vectors support the hyperplane,
hence called a Support vector.
How does SVM works?
Linear SVM:
The working of the SVM algorithm can be understood by using an example. Suppose we have a dataset
that has two tags (green and blue), and the dataset has two features x1 and x2. We want a classifier that can
classify the pair (x1, x2) of coordinates in either green or blue. Consider the below image:

Department of CSE-AIML Machine Learning 24BEAML405

So as it is 2-d space so by just using a straight line, we can easily separate these two classes.
But there can be multiple lines that can separate these classes. Consider the below image:

Hence, the SVM algorithm helps to find the best line or decision boundary; this best boundary
or region is calledas a hyperplane. SVM algorithm finds the closest point of the lines from
both the classes. These points are calledsupport vectors. The distance between the vectors and
the hyperplane is called as margin. And the goal of SVM is to maximize this margin. The
hyperplane with maximum margin is called the optimal hyperplane.

Department of CSE-AIML Machine Learning 24BEAML405

Non-Linear SVM:
If data is linearly arranged, then we can separate it by using a straight line, but for non-linear
data, we cannot draw a single straight line. Consider the below image:

So to separate these data points, we need to add one more dimension. For linear data, we have
used two dimensions x and y, so for non-linear data, we will add a third dimension z. It can be
calculated as:
z=x2 +y2

By adding the third dimension, the sample space will become as below image:

Since we are in 3-d Space, hence it is looking like a plane parallel to the x-axis. If we convert it
in 2d space withz=1, then it will become as:

Department of CSE-AIML Machine Learning 24BEAML405

Hence we get a circumference of radius 1 in case of non-linear data.

now, SVM will divide the datasets into classes in the following way. Consider the below image:

Since we are in 3-d Space, hence it is looking like a plane parallel to the x-axis. If weconvert it
in 2d space with z=1, then it will become as:

Department of CSE-AIML Machine Learning 24BEAML405

Applications of SVM in Real World
As we have seen, SVMs depends on supervised learning algorithms. The aim of using SVM is to correctly
classify unseen data. SVM have a number of applications in several fields. Some common applications of
SVM are-

● Face detection – SVMs classify parts of the image as a face and non-face and create a
square boundary around the face.
● Text and hypertext categorization – SVMs allow Text and hypertext categorization for
both inductive and transductive models. They use training data to classify documents into
different categories. It categorizes on the basis of the score generated and then compares
with the threshold value.
● Classification of images – Use of SVMs provides better search accuracy for image
classification. It provides better accuracy in comparison to the traditional query- based
searching techniques.
● Bioinformatics – It includes protein classification and cancer classification. We use SVM
for identifying the classification of genes, patients on the basis of genes and other
biological problems.
● Protein fold and remote homology detection – Apply SVM algorithms for protein remote
homology detection.
● Handwriting recognition – We use SVMs to recognize handwritten characters used widely.

● Generalized predictive control (GPC) – Use SVM based GPC to control chaotic dynamics
with useful parameters.
Top of Form

Bottom of Form

Department of CSE-AIML Machine Learning 24BEAML405

 Bottom of Form

Department of CSE-AIML Machine Learning 24BEAML405