Module 3 : Linear Models

CO3: To apply Least Square Method, Regression Techniques and SVM for
Classification

Content:
• The least-squares method, Multivariate Linear Regression, Regularized
Regression, Using Least-Squares Regression for classification
• Support Vector Machines

Ichhanshu Jaiswal CSEDS VCET

Linear Models
• Linear models are models that shows linear relation between
independent and dependent variables.
• They assume that the relationship between your input features (X) and the
target output (Y) can be represented as a straight line (or a flat plane in
higher dimensions).
• A linear model calculates a weighted sum of the input features, plus a
constant called the bias (or intercept). The general formula looks like :

• y: The prediction.
• x: The input data (features).
• w: The weights (how much importance the model gives to each feature).
• b: The bias (the value of y when all x are zero).
Ichhanshu Jaiswal CSEDS VCET
 Linear Models
• Linear models are versatile and can be used for both predicting
numbers (Regression) and categorizing data (Classification).
Model Type Purpose How it works
Predicts a continuous value (e.g.,
Linear Regression Regression
predicting a house price.
Despite the name, it's used for
Logistic classification. It uses a "Sigmoid"
Classification
Regression function to turn the linear output into a
probability between 0 and 1.
SVM (Linear Finds the "widest" possible gap (margin)
Classification
Kernel) between two classes of data.

Ichhanshu Jaiswal CSEDS VCET

 Why to use Linear Models
• Interpretability: Linear models are easy to understand. Example
y=4x+3 , one can understand that O/P is 4times I/P plus [Link] is useful
for industries like finance or healthcare.
• Speed: They are incredibly fast to train and require very little
computing power.
• Baseline: They provide a basic check for any dataset, before you try
more complex models.
• Drawback: The biggest limitation is that they assume linearity. If
your data looks like a complex wave or a circle, a simple linear
model will likely struggle unless you transform the features first.

Ichhanshu Jaiswal CSEDS VCET

 Linear Regression
• Linear regression is a type of supervised machine-learning algorithm
that learns from the labelled datasets and maps the data points with
most optimized linear functions which can be used for prediction on
new datasets.
• It assumes that there is a linear relationship between the input and
output, meaning the output changes at a constant rate as the input
changes.
• This relationship is represented by a straight line.
• In linear regression, the best-fit line is the straight line that most
accurately represents the relationship between the independent
variable (input) and the dependent variable (output).
• Best fit line is the line that minimizes the difference between the actual
data points and the predicted values from the model.
Ichhanshu Jaiswal CSEDS VCET
Best fit line
Slope is given by

Y intercept is given by

Ichhanshu Jaiswal CSEDS VCET

How slope & Intercept are calculated ??
• The Ordinary Least Squares (OLS) method or least square method
is the most common strategy for "fitting" a linear regression line.
• The distance between an actual data point and the value predicted by
the line is called a residual.
• The goal of OLS is to find the specific line (the best weights W and
bias b) that makes the sum of the squares of these residuals as small
as possible.

• We need to minimize (Sum of square term ) = Σ ( Y-Y`)2

= Σ ( Y – (mx + b))2

Ichhanshu Jaiswal CSEDS VCET

To find best value of b

NowIchhanshu
divide byCSEDS
Jaiswal n VCET
Ichhanshu Jaiswal CSEDS VCET
To find the slope

Ichhanshu Jaiswal CSEDS VCET

Ichhanshu Jaiswal CSEDS VCET
Solve

Ichhanshu Jaiswal CSEDS VCET

 Regression Line
Ichhanshu Jaiswal CSEDS VCET
Ichhanshu Jaiswal CSEDS VCET
Multivariate Regression
• Multiple Linear Regression extends this concept by modelling the relationship between
a dependent variable and two or more independent variables.
• This technique allows us to understand how multiple features collectively affect the
outcomes.
• We can use it to find out which factor has the highest influence on the predicted output
and how different variables are related to each other.
• Equation for multiple linear regression is:
MVR are used for
1. Prediction:
2. Determine importance of
variable :Which factor influences
sales the most: social media ads,
TV spots, or price drops“ i.e which
I/P mostly control the O/P

Ichhanshu Jaiswal CSEDS VCET

• The goal of the algorithm is to find the best fit line equation that can
predict the values based on the independent variables.
• A regression model learns from the dataset with known X and y values
and uses it to predict y values for unknown X.
• In this we may have categorical data like Male/Female or name of location
all these are converted to numerical values either as 0/1 or using One
Hot Encoding.
• When two or more independent variables are highly correlated with each
other. It becomes difficult to find the individual contribution of each
variable to the dependent variable.
• Example: Don't use "Height in Inches" and "Height in Centimeters" in the
same dataset.
• In such case we can use Correlation matrix to find correlation between
two independent features, and if there exist high correlation we can either
drop of of feature or combine both features.
Ichhanshu Jaiswal CSEDS VCET
 Assumption of Multiple Linear Regression
• Linearity: The relationship between predictors and the outcome is a
straight line.
• No Multicollinearity: Your independent variables shouldn't be too
closely correlated with each other
• Independence: Observations are independent of one another.
• Homoscedasticity: The "errors" (residuals) should be consistent
across the data. i.e Free from outliers.

Ichhanshu Jaiswal CSEDS VCET

 Least Squares Error Function
• The goal of linear regression is to minimize the squared error:

• Error =(Y - Xβ)2 = (Y - Xβ)T (Y - Xβ) [As P2 = PTP]

• On Expanding we get : YT Y - 2βT XT Y + βT XT X β
• This represents the sum of squared differences between actual and
predicted values.
• To minimize the error function, take its derivative w.r.t. β and set it to
zero:
• ∂/∂β (YT Y - 2βT XT Y + βT XT X β) = 0
• Solving for β gives:
β = (XT X)-1 XT Y
Ichhanshu Jaiswal CSEDS VCET
Example

Ichhanshu Jaiswal CSEDS VCET

Ichhanshu Jaiswal CSEDS VCET
Ichhanshu Jaiswal CSEDS VCET
Ichhanshu Jaiswal CSEDS VCET
Ichhanshu Jaiswal CSEDS VCET
 Regularization
• Regularization is a technique used to prevent overfitting by adding a
penalty term to the Loss function, discouraging the model from assigning
too much importance to individual features or coefficients.
• Regularization helps control model complexity by preventing overfitting to
training data, resulting in better generalization to new data.
• Here penalty are imposed on large coefficients by constraining their
magnitudes, thereby preventing a model from becoming overly complex
and memorizing the training data instead of learning its underlying
patterns.
Focus on learn concepts and not Memorize

Ichhanshu Jaiswal CSEDS VCET

 BIAS VARIANCE
TRADE OFF

To reduce
overfitting
decrease the
value of slope (m)

For y = mx + b , here if m=0 means x has no contribution in O/P

if m= high means O/P to very high extent depends on x
Conclusion: value of “m” decides the contribution of x in deciding O/P
Ichhanshu Jaiswal CSEDS VCET
 Types of Regression Regularization
1. L2 Regression Regularization ( Ridge Regularization )

2. L1 Regression Regularization ( Lasso Regularization )

(Lambda): This is the "strength" of the penalty. If lambda = 0, it’s just

standard OLS regression. As lambda increases, the model becomes simpler
(biased) but more stable/generalize.
Ichhanshu Jaiswal CSEDS VCET
Consider these
two training
datapoints
P1= ( 1, 2.3 )
P2= (3,5.3 )
=2,03
Ichhanshu Jaiswal CSEDS VCET
L2 Regularization also known as Ridge Regularization

• Here we add a regularization term called as ƛ(Σmi2),

due to this overfitted line get shifted down thus
resulting in better approximation, i.e algorithm
perform better on test data.
• Due to this bias will increase little, but algorithm will
have low variance thus improving accuracy on test
data.
• Adding the term ƛ(Σmi2), to loss function is called
Ridge Regression
=2.03
Ichhanshu Jaiswal CSEDS VCET
• L2 regularization, also known as ridge regularization, is a technique used
in machine learning to prevent overfitting by adding a penalty to the loss
function proportional to the square of the magnitude of the coefficients
(weights) of the model.
• By penalizing large weights, L2 regularization forces the model to
prioritize smaller weights, reducing the likelihood of overfitting.
• Since the penalty is quadratic, it penalizes large weights more heavily,
leading to a model, that is smoother and less sensitive to individual data
points
• L2 regularization is widely used because it helps improve the model's
performance on unseen data by preventing overfitting while still allowing
for complex models.

Ichhanshu Jaiswal CSEDS VCET

• If value of lamda increase, then co-efficient get shrinking and
reach near to zero but not get zero.
• Bigger coefficient get shrink faster compared to smaller
coefficient.

Ichhanshu Jaiswal CSEDS VCET

 L1 Regularization ( Lasso Regularization )
(Least Absolute Shrinkage and Selection Operator)
• Here we add a regularization term called as ƛ(Σ|mi|), i.e m=|m1|+|m2| +.....+
|mn| due to overfitted line get shifted thus resulting in better

• Only difference in L2 and L1 is that in L2 we add sum of square of m & in LI

we add sum of mod of m values.
• Here when we increase value of lambda then co-efficient start shrinking
and it reaches to zero for some value of lambda.
• Thus, it helps in dimension reduction thus helping in better generalization.
• Here when we increase Lamba then data column having lower co-efficient
reduces to zero thus making model generalized and saving processing
power. Ichhanshu Jaiswal CSEDS VCET
Ridge Regression Lasso Regression

L2 Penalty: Ridge uses the square ofL1 Penalty: Lasso uses the absolute
the coefficients. values of coefficients.
It makes all coefficients smaller but
It completely removes unnecessary
doesn’t set them to zero. features by setting their coefficients
to zero.
It is good when all features matter It is best when we want to remove
but we want to reduce their impact irrelevant features
It works well when features are It is great for automatically choosing
related but shouldn’t be completely important features.
removed
It keeps all features which may not It can sometimes remove useful
help in high-dimensional data with features if not tuned properly.
irrelevant features.
Ichhanshu Jaiswal CSEDS VCET
Can we use Regression line for Classification ?

Note:
Every class will
have its own
Regression
Equation

Ichhanshu Jaiswal CSEDS VCET

Ichhanshu Jaiswal CSEDS VCET
Ichhanshu Jaiswal CSEDS VCET
Ichhanshu Jaiswal CSEDS VCET
Ichhanshu Jaiswal CSEDS VCET
 • Supose we want to classify data ( 5,3 ) we will just substitute its
value in y1, y2 and y3

The datapoint will belong to class

having largest Y value
Ichhanshu Jaiswal CSEDS VCET
 Linear Classification
• Linear classification separates data using a hyperplane.
Two main approaches:
1. -Modeling a discriminant function [ y= 𝑋𝛽 ] ( Linear Regression for
classification ), defines a decision boundary based on comparisons of
discriminant functions
2. -Directly modeling the hyperplane ( SVM )

Ichhanshu Jaiswal CSEDS VCET

 Linear Regression for classification
• A function is assigned to each class
• The class with the highest function value determines classification
• Goal here is to learn the discriminant functions δi.
• For a two-class classification problem with discriminant function δ1 and
δ2
• The boundary is where δ1 = δ2
• The function 𝑓(𝑋) is computed as:𝑓(𝑋)=𝑋𝛽
where 𝛽 is a 𝑝×𝐾 weight matrix.
• Prediction Rule: The final class label is determined using the argmax
function:
where𝑓𝑘(𝑋) is the score for class 𝑘.
Ichhanshu Jaiswal CSEDS VCET
 Drawbacks of linear model for classification
1. Predictions Outside the [0, 1] Range:
• Linear regression is designed to predict continuous values from - infinity to +
infinity . In classification, we usually want the probability that an observation
belongs to a certain class.
• A linear model can easily predict that a person has a -0.2 probability of having
a disease or a 1.4 probability of winning a race.
2. Sensitivity to Outliers:
• Linear regression works by minimizing the sum of squared errors. Because of
this, it is incredibly sensitive to "extreme" data points, even if those points are
correctly classified.
• If you add a data point very far to the right (a "clear" instance of the positive
class), the regression line will tilt to accommodate it thus disturbing the overall
classification
Ichhanshu Jaiswal CSEDS VCET
3. The "Masking" Problem (Multi-class)
• If you try to use linear regression for more than two classes (e.g.,
predicting if a fruit is an Apple, Orange, or Banana), the model
imposes an artificial order.
• By assigning Apple = 1, Orange = 2, and Banana = 3, you are telling
the model that an Orange is "halfway" between an Apple and a
Banana which will result in, linear fit will fail to capture the
boundaries between them.

Ichhanshu Jaiswal CSEDS VCET

Hyperplane

Ichhanshu Jaiswal CSEDS VCET

This means β is
perpendicular to
L.

Ichhanshu Jaiswal CSEDS VCET

Ichhanshu Jaiswal CSEDS VCET
Margin

Ichhanshu Jaiswal CSEDS VCET

 SVM
• SVM is a supervised
learning algorithm used
for classification and
regression.
• It finds the optimal
hyperplane that
maximizes the margin
between different
classes.

Ichhanshu Jaiswal CSEDS VCET

 SVM
• Works well for high-dimensional spaces and is effective when the
number of dimensions is greater than the number of samples.
• It tries to find the best boundary known as hyperplane that separates
different classes in the data.
• The main goal of SVM is to maximize the margin between the two
classes.
• The larger the margin the better the model performs on new and
unseen data.
• It depends on support vectors to define decision boundary and is
robust to outliers

Ichhanshu Jaiswal CSEDS VCET

• Support Vectors: The closest data points to the hyperplane, crucial for
determining the hyperplane and margin in SVM.
• Margin: The distance between the hyperplane and the support vectors.
SVM aims to maximize this margin for better classification performance.
• Kernel: A function that maps data to a higher-dimensional space
enabling SVM to handle non-linearly separable data.
• C: A regularization term balancing margin maximization and
misclassification penalties. A higher C value forces stricter penalty for
misclassifications.
• Hinge Loss: A loss function penalizing misclassified points or margin
violations and is combined with regularization in SVM.

Ichhanshu Jaiswal CSEDS VCET

• To maximize the margin, we need to minimize the norm of the weight
vector w. Specifically, the optimization problem is formulated as:

Ichhanshu Jaiswal CSEDS VCET

• Instead of just trying to classify data correctly, we maximize the margin
(the distance between the decision boundary and the nearest data
points).
• However, real-world data is often noisy and not perfectly separable, so a
strict margin might not work well.
• Some points lie inside the margin → we want to minimize their deviation.
• Some points are incorrectly classified → we need a trade-off
• Hard Margin: A maximum-margin hyperplane that perfectly separates
the data without misclassifications.
• Soft Margin: Allows some misclassifications by introducing slack
variables, balancing margin maximization and misclassification
penalties when data is not perfectly separable.

Ichhanshu Jaiswal CSEDS VCET

• We introduce slack
variables (ξi) to allow
flexibility:
• But in real-world data,
some ξi will be nonzero to
allow margin violations.
• Keep total slack under a
certain limit.

The parameter 𝐶 controls the trade-off between margin size\ and classification error
Ichhanshu Jaiswal CSEDS VCET
• Large C → Smaller margin, fewer training errors.
• Small C → Larger margin, allows some errors.
• If C → ∞, we recover the hard-margin SVM (strict separation).
• C determines how much we penalize margin violations
• If data has noise, strict separation may lead to overfitting.
• Allowing some errors (small C) leads to a more robust classifier.
• Hard-margin → Fits outliers too closely.
• Soft-margin → Finds a better overall separation.

Ichhanshu Jaiswal CSEDS VCET

 Kernal Trick in SVM
• When data isn't linearly separable in its current space, we project it
into a higher-dimensional space where a linear boundary can separate
it.
• Instead of doing the expensive math of the projection, we use a Kernel
function K(xi, xj) to calculate the dot product in that high-dimensional
space directly.
• By applying a kernel function, SVMs can implicitly map input data into
a higher-dimensional space where a linear separator (hyperplane) can
be used to divide the classes.
• This mapping is computationally efficient because it avoids the direct
calculation of the coordinates in this higher space.

Ichhanshu Jaiswal CSEDS VCET

• Kernels compute inner products between transformed vectors
without explicit feature expansion.
• Kernel trick: Avoids direct computation in high-dimensional space,
reducing complexity
• In non-linearly separable cases, we use kernel functions to map
input space into a higher-dimensional feature space.
• Mathematically:
• 𝐾(𝑥i,xj)=𝜙(𝑥i)⋅𝜙(xj)
• where
• 𝜙(𝑥i) is the feature mapping function.

Ichhanshu Jaiswal CSEDS VCET

Ichhanshu Jaiswal CSEDS VCET
Advantages of Kernal trick
• Enables SVM to handle complex decision boundaries
• Avoids explicit transformation into high-dimensional space
(saves computation)
• Works well with small-to-medium-sized datasets

Ichhanshu Jaiswal CSEDS VCET