Step-wise, Ridge, Lasso,
ElasticNet Regression
Step-wise regression
• Stepwise regression is a regression technique used for feature
selection, which aims to identify the subset of input features that are
most relevant for predicting the output variable.
• 3 types
• Forward Selection,
• Backward Elimination,
• Bidirectional Elimination
• Forward Selection: the algorithm starts with an empty set of features and
iteratively adds the most statistically significant feature to the model. This
process continues until no more features can be added without reducing
the model's performance.
• Backward Elimination: the algorithm starts with the full set of features and
iteratively removes the least statistically significant feature from the model.
This process continues until no more features can be removed without
reducing the model's performance.
• Bidirectional Elimination: a combination of forward and backward
selection, where the algorithm alternates between adding and removing
features until no more changes can be made to improve the model
performance.
Implementing forward stepwise regression
• Start with an empty set of features.
• Train the model using one feature at a time, starting with the most
statistically significant feature.
• Evaluate the performance of the model at each step and keep track of
the set of features that maximizes the performance.
• Continue adding features until no more features can be added
without reducing the model's performance.
Implementing backward stepwise regression
• Start with the full set of features.
• Train the model using all the features.
• Evaluate the performance of the model at each step and keep track of
the set of features that maximizes the performance.
• Remove the least statistically significant feature and repeat steps 2
and 3 until no more features can be removed without reducing the
model's performance.
Implementing stepwise regression with both
forward and backward selection
• Start with an empty or full set of features.
• Perform forward selection until no more features can be added
without reducing the model's performance.
• Perform backward elimination until no more features can be removed
without reducing the model's performance.
• Repeat steps 2 and 3 until no more changes can be made to improve
the model performance.
Steps to perform
• Load and Prepare the Data
• Preprocess the Data
• Select the Top Features
• Split the Data into Training and Testing Sets
• Perform default Linear Regression
• Perform Stepwise Regression
• Train and Evaluate the Model with Selected Features
Coefficients Significance
t-statistic and p-values:
• For a given predictor, the t-statistic (and its associated p-value) tests
whether or not there is a statistically significant relationship between a
given predictor and the outcome variable, that is whether or not the beta
coefficient of the predictor is significantly different from zero.
• The statistical hypotheses are as follow:
• Null hypothesis (H0): the coefficients are equal to zero (i.e., no relationship between
x and y)
• Alternative Hypothesis (Ha): the coefficients are not equal to zero (i.e., there is some
relationship between x and y)
Coefficients Significance
t-statistic and p-values:
• The t-statistic measures the number of standard deviations that 𝜃 is
away from 0. Thus a large t-statistic will produce a small p-value.
• Higher the t-statistic (and the lower the p-value), more significant the
predictor is.
• A statistically significant coefficient indicates that there is an
association between the predictor (x) and the outcome (y) variable.
Interpretation:
• If p-value < 0.05, reject the null hypothesis → statistically significant.
• If p-value > 0.05, you don’t have strong evidence that the model fits
better than a model with no predictors.
• [Link]
• [Link]
• [Link]
• [Link]
Record X₁ X₂ Y
1 2 1 10
2 4 3 20
3 6 5 30
4 8 10 40
You are given the following dataset with 4 records:
Perform feature selection based on correlation with the target variable Y.
Which feature(s) should be selected?
Gradient Descent in Linear Regression
A linear regression model attempts to explain the relationship between a dependent (output variables) variable and one
or more independent (predictor variable) variables using a straight line.
This straight line is represented using the following formula:
y=mx+c
Where, y: dependent variable
x: independent variable
m: Slope of the line (For a unit increase in the quantity of X, Y increases by m.1 = m units.)
c: y intercept
Step by Step Algorithm:
Step by Step Algorithm:
Step by Step Algorithm:
Let’s now consider a simple linear regression problem using 3 data points and perform 2 iterations of
gradient descent.
•Hypothesis: h(x)=mx+c
•Initial values: m=0, c=0
•Learning rate L=0.1
•Number of data points n=3
Step 1: Predictions & Errors
Using m=0, c=0:
x y Ypred h(x)=mx+c error=y−ypred
1 2 0 2
2 3 0 3
3 4 0 4
Step 1: Predictions & Errors
Using m=0, c=0:
x y Ypred h(x)=mx+c error=y−ypred
1 2 0 2
2 3 0 3
3 4 0 4
Step 3: Update m and c
Need for regularization
Need for regularization
Need for regularization
9
9
Cost Function of Linear Regression
Cost Function of Linear Regression
Regularization
Regularization
Ridge Regularization
Lasso regularization
Which technique, when to use
Ridge Regression-L2 Regularization
• Ridge regression advantage is to
avoid overfitting
• The model should generalize the
patterns well-for training and
testing
• Overfitting occurs when the
model works well on training data
and poorly on testing datasets
• Ridge regression adds a penalty
term to overcome overfitting-by
reducing weights and bias
• Least sum of squares is applied
to the best fit line Error
Salary
• Since the line is passing
through 3 training dataset
points, the sum of squared Training
samples
residuals =0.
Testing
• For the testing dataset, the samples
sum of residuals is large, so the
line is having higher variance
# years experience
• This is the case of overfitting
• Ridge regression works by Linear
Regression
increasing bias to improve
Salary Ridge
variance Regression
• Changes the slope of the model
Training
• Eventhough performance is samples
poor, works well for both Testing
training and testing samples samples
# years experience
• The slope is modified by adding Linear
Regression
the penalty term such that the
Salary Ridge
model becomes less sensitive Regression
to the changes in the
independent variable Training
samples
Testing
samples
# years experience
• As alpha increases, the slope of
regression line reduces and the line
becomes more horizontal
• It becomes less sensitive to
variations in the independent
variable
Lasso Regression
• The effect of alpha is
similar to the ridge
regression
Lasso Regression
• Lasso regression helps
overfitting and used for
feature selection
• Lasso regression is
useful if we have
several independent
insignificant variables
• Ridge regression can
reduce the slope closer
to the zero, but Lasso
regression can reduce
the slope exactly zero.
Regularization
Regularization is a technique used in machine learning and statistical
models to prevent overfitting—which occurs when a model learns the
training data too well, including noise and outliers, and performs poorly
on unseen data.
Regularization
• Why Regularization is Needed:
• Overfitting Control:
• A highly complex model (e.g., with many features or high-degree polynomials) can fit
training data very closely but may fail to generalize to test data.
• Regularization penalizes extreme weights or coefficients, effectively simplifying the
model.
• Bias-Variance Tradeoff:
• Regularization increases bias slightly but reduces variance significantly, leading to
better generalization.
• It shifts the model from overfitting (low bias, high variance) toward an optimal
tradeoff.
• Handling Multicollinearity:
• In linear models, regularization helps deal with correlated features, which can cause
instability in coefficient estimates.
Bias
• Definition: Bias refers to the error due to overly simplistic
assumptions in the learning algorithm.
• High Bias: Model is too simple to capture the underlying pattern in
the data (e.g., using linear regression for nonlinear data).
• Effect: Leads to underfitting — poor performance on both training
and test data.
• Example: Predicting house prices with only one feature (e.g., house
size), ignoring other important variables (e.g., location, number of
rooms).
Variance
• Definition: Variance refers to the error due to the model's sensitivity
to small fluctuations in the training data.
• High Variance: Model learns the training data too closely, including
noise and outliers.
• Effect: Leads to overfitting — good performance on training data but
poor generalization to new data.
Example: A decision tree with no depth limit that memorizes training
data exactly.
Bias-Variance Tradeoff
• Low Bias + High Variance: Overfitting
• High Bias + Low Variance: Underfitting
• Optimal Model: Strikes a balance between bias and variance,
minimizing total prediction error.
Regularization Techniques
Common Regularization Techniques:
• L1 Regularization (Lasso): Adds penalty λ * |weights|
• L2 Regularization (Ridge): Adds penalty λ * weights²
• Elastic Net: Combines L1 and L2 penalties
Regularization
Ridge Regularization
Lasso regularization
Which technique, when to use
• You are given the coefficients of a linear regression model trained on
a dataset: β0=2,β1=3,β2=−4,β3=1
• Compute the L2 penalty (Ridge penalty) for the model when the
regularization parameter λ=0.5
• Compute the L1 penalty (Lasso penalty) for the same coefficients
with λ=0.5
Solution
• [Link]
• [Link]
• [Link]
• Multiple linear regression
• Polynomial regression
• [Link]
Step-wise regression
• [Link]
Ridge
• [Link]
Lasso
• [Link]
from-scratch-using-python/
ElasticNet
• [Link]
Project – Assignment 1
• Title
• Team Name and Team Member Details
• SDG Goal No and Description
• Abstract
• Motivation
• Objectives ( 3 to 5 points)
• Dataset (Link and Description)
• References