Regression

Module 3
Regression
• Regression is a statistical method to model relationships between dependent
and independent variables.
• Helps in prediction and inference.
Types of regression
• Simple Linear Regression: Models the relationship between one
independent variable and one dependent variable.
• Multiple Linear Regression: Involves two or more independent variables
to predict the dependent variable.
• Other types: Logistic, Polynomial, Ridge, Lasso.
Simple linear regression
• Establishes a linear relationship between a dependent variable (Y) and one
independent variable (X).
• The equation is: Y = b0 + b1X + e, where:
• b0: Intercept (the value of Y when X = 0)
• b1: Slope (change in Y for a unit increase in X)
• e: Error term (random variability not captured by the model)
Components of Simple Linear Regression
• Dependent Variable (Y): The outcome variable we are trying to predict.
• Independent Variable (X): The predictor variable used to estimate Y.
• Intercept (b0): The starting value of Y when X is zero.
• Slope (b1): The rate of change of Y per unit change in X.
• Error term (e): Represents random fluctuations in the data.
Assumptions of Simple Linear Regression
• Linearity: The relationship between X and Y is linear.
• Independence: Observations are independent of each other.
• Homoscedasticity: Constant variance of errors.
• Normality of residuals: Errors should follow a normal distribution.
Assumptions of simple Linear Regression
1. Linearity
•Plot a scatter plot of the dependent variable (Y) vs. independent
variable (X).
•The relationship should appear as a straight-line trend.
•You can also check the residuals plot (Residuals vs. Fitted values) for a
random distribution.
2. Independence
•Durbin-Watson test: Used to check for autocorrelation in residuals. A
value close to 2 indicates no autocorrelation.
•Time Series Plot: If data is time-dependent, plot residuals over time to
check for patterns.
Assumptions of simple Linear Regression
3. Homoscedasticity (Constant Variance of Residuals)
•Plot Residuals vs. Fitted Values: Residuals should be randomly
scattered without forming patterns.
•Breusch-Pagan Test: A statistical test to check heteroscedasticity.

4. Normality of Residuals
•Histogram of Residuals: Residuals should form a normal (bell-
shaped) distribution.
•Q-Q Plot (Quantile-Quantile Plot): Residuals should align closely
with the diagonal line.
•Shapiro-Wilk Test: A statistical test for normality of residuals.
Goodness of fit (R square)
• R-squared (R2) is defined as a number that tells you how well the
independent variable(s) in a statistical model explains the variation in the
dependent variable.
• R^2 = Explained Variation / Total Variation.
• R^2 = 1 means perfect fit, R^2 = 0 means no predictive power.
Adjusted R square
• The adjusted R-squared compares the descriptive power of regression
models that include diverse numbers of predictors. This is often assessed
using measures like R-squared to evaluate the goodness of fit. Every
predictor added to a model increases R-squared and never decreases it.
Thus, a model with more terms may seem to have a better fit just for the
fact that it has more terms, while the adjusted R-squared compensates for
the addition of variables; it only increases if the new term enhances the
model above what would be obtained by probability and decreases when a
predictor enhances the model less than what is predicted by chance.
• In an overfitting condition, an incorrectly high value of R-squared is
obtained, even when the model actually has a decreased ability to predict.
This is not the case with the adjusted R-squared.
 Regression Metrics
• Mean Absolute Error:

Y is the Actual outcome, Y' is the predicted outcome, and N is the total number of
data points.
• Mean Squared Error:

In MSE, errors are squared, therefore it only assumes non-negative values, and it is
usually positive and non-zero.
Due to squared differences, it penalizes small errors also, and hence it leads to over-
estimation of how bad the model is.
R Squared score
• R squared error is also known as Coefficient of Determination, which is
another popular metric used for Regression model evaluation.
➢The R-squared metric enables us to compare our model with a constant
baseline to determine the performance of the model.
➢To select the constant baseline, we need to take the mean of the data and
draw the line at the mean.
Adjusted R square
• R square has a limitation of improvement of a score on increasing the terms,
even though the model is not improving, and it may mislead the data
scientists.
• To overcome the issue of R square, adjusted R squared is used, which will
always show a lower value than R². It is because it adjusts the values of
increasing predictors and only shows improvement if there is a real
improvement.
• n is the number of observations
• k denotes the number of independent variables
• Ra2 denotes the adjusted R2
Model selection techniques
a) Akaike Information Criterion (AIC)
• Evaluates model quality based on goodness of fit and complexity.
• Lower AIC is better (penalizes too many predictors).
• AIC=2p−2 Ln(L)
where:
• p = number of parameters,
• L = likelihood function.
Model selection techniques
b) Bayesian Information Criterion (BIC)
• Similar to AIC but imposes a higher penalty for complexity.
• Lower BIC is better for model selection.
BIC=p ln(n)-2 ln(L)
• where n = number of observations.
Model selection techniques
c) Cross-Validation (CV)
• Splits data into training and testing sets multiple times.
• Common method: k-Fold Cross-Validation (e.g., 5-fold, 10-fold).
• Helps prevent overfitting and ensures model generalizability.
d) Regularization Techniques
• Helps prevent overfitting in multiple regression models.
• Lasso Regression: Shrinks coefficients, some become zero (feature
selection).
• Ridge Regression: Shrinks coefficients but keeps all predictors.
Types of Regression Analysis
• Linear Regression
• Logistic Regression
• Polynomial Regression
• Support Vector Regression
• Decision Tree Regression
• Random Forest Regression
• Ridge Regression
• Lasso Regression
Linear Regression
• Linear regression is a statistical regression method which is used for
predictive analysis.
• It is one of the very simple and easy algorithms which works on regression
and shows the relationship between the continuous variables.
• Linear regression shows the linear relationship between the independent
variable (X-axis) and the dependent variable (Y-axis), hence called linear
regression.
• If there is only one input variable (x), then such linear regression is
called simple linear regression. And if there is more than one input variable,
then such linear regression is called multiple linear regression.
Logistic Regression
• Logistic regression is another supervised learning algorithm which is used
to solve the classification problems. In classification problems, we have
dependent variables in a binary or discrete format such as 0 or 1.
• Logistic regression algorithm works with the categorical variable such as 0
or 1, Yes or No, True or False, Spam or not spam, etc.
• It is a predictive analysis algorithm which works on the concept of
probability.
• Logistic regression uses sigmoid function or logistic function which is a
complex cost function. This sigmoid function is used to model the data in
logistic regression. The function can be represented as:
Plot for regression and logistic regression
Polynomial Regression
• Polynomial Regression is a type of regression which models the non-
linear dataset using a linear model.
• It is similar to multiple linear regression, but it fits a non-linear curve
between the value of x and corresponding conditional values of y.
• Suppose there is a dataset which consists of datapoints which are
present in a non-linear fashion, so for such case, linear regression will
not best fit to those datapoints. To cover such datapoints, we need
Polynomial regression.
• In Polynomial regression, the original features are transformed into
polynomial features of given degree and then modeled using a linear
model. Which means the datapoints are best fitted using a polynomial
line.
Polynomial Regression
• The equation for polynomial regression
also derived from linear regression
equation that means Linear regression
equation
• Y= b0+ b1x, is transformed into
Polynomial regression equation
• Y= b0+b1x+ b2x2+ b3x3+.....+ bnxn.
Here
Y is the predicted/target output,
b0, b1,... bn are the regression coefficients.
x is our independent/input variable.
The model is still linear as the coefficients
are still linear with quadratic
Support Vector Machines
• The goal of the SVM algorithm is to create the best line or decision
boundary that can segregate n-dimensional space into classes so that we can
easily put the new data point in the correct category in the future. This best
decision boundary is called a hyperplane.
• SVM chooses the extreme points/vectors that help in creating the
hyperplane. These extreme cases are called as support vectors, and hence
algorithm is termed as Support Vector Machine. In this two different
categories are classified using a decision boundary or hyperplane.
• SVM algorithm can be used for Face detection, image classification, text
categorization, etc.
Support Vector Machine
• Linear SVM: Linear SVM is used for linearly separable data, which means
if a dataset can be classified into two classes by using a single straight line,
then such data is termed as linearly separable data, and classifier is used
called as Linear SVM classifier.
• Non-linear SVM: Non-Linear SVM is used for non-linearly separated data,
which means if a dataset cannot be classified by using a straight line, then
such data is termed as non-linear data and classifier used is called as Non-
linear SVM classifier.
Support Vector Regression
• Support Vector Regression is a regression algorithm which works for
continuous variables. Below are some keywords which are used in Support
Vector Regression:
• Kernel: It is a function used to map a lower-dimensional data into higher
dimensional data.
• Hyperplane: In general SVM, it is a separation line between two classes, but
in SVR, it is a line which helps to predict the continuous variables and
cover most of the datapoints.
• Boundary line: Boundary lines are the two lines apart from hyperplane,
which creates a margin for datapoints.
• Support vectors: Support vectors are the datapoints which are nearest to the
hyperplane and opposite class.
Support Vector Regression

In SVR, we always try to determine a

hyperplane with a maximum margin, so that
maximum number of datapoints are covered
in that margin. The main goal of SVR is to
consider the maximum datapoints within the
boundary lines and the hyperplane (best-fit
line) must contain a maximum number of
datapoints.
Decision Tree Regression
• Decision Tree is a supervised learning algorithm which can be used for
solving both classification and regression problems.
• It can solve problems for both categorical and numerical data
• Decision Tree regression builds a tree-like structure in which each internal
node represents the "test" for an attribute, each branch represent the result of
the test, and each leaf node represents the final decision or result.
• A decision tree is constructed starting from the root node/parent node
(dataset), which splits into left and right child nodes (subsets of dataset).
These child nodes are further divided into their children node, and
themselves become the parent node of those nodes.
Decision Tree Regression

The Random Forest regression is an

ensemble learning method which combines
multiple decision trees and predicts the final
output based on the average of each tree
output.
Decision Tree Regression
• Random forest uses Bagging or
Bootstrap Aggregation technique
of ensemble learning in which
aggregated decision tree runs in
parallel and do not interact with
each other.
• With the help of Random Forest
regression, we can prevent
Overfitting in the model by creating
random subsets of the dataset.
 Ridge Regression
• Ridge regression is one of the most robust versions of linear regression in
which a small amount of bias is introduced so that we can get better long
term predictions.
• The amount of bias added to the model is known as Ridge Regression
penalty. We can compute this penalty term by multiplying with the lambda to
the squared weight of each individual features.
• The equation for ridge regression will be:
Ridge Regression
• A general linear or polynomial regression will fail if there is high
collinearity between the independent variables, so to solve such problems,
Ridge regression can be used.

• Ridge regression is a regularization technique, which is used to reduce the

complexity of the model. It is also called as L2 regularization.

• It helps to solve the problems if we have more parameters than samples.

Lasso Regression
• Lasso regression is another regularization technique to reduce the
complexity of the model.
• It is similar to the Ridge Regression except that penalty term contains only
the absolute weights instead of a square of weights.
• Since it takes absolute values, hence, it can shrink the slope to 0, whereas
Ridge Regression can only shrink it near to 0.
• It is also called as L1 regularization. The equation for Lasso regression
will be:
Clustering
• Clustering is an unsupervised machine learning technique designed to
group unlabeled examples based on their similarity to each other. If the
examples are labeled, this kind of grouping is called classification.
• Use cases
[Link] Segmentation
2. Statistical data analysis
[Link] network analysis
[Link] segmentation
[Link] detection
Clustering
Amazon
Netflix recommendation
system
Types of clustering
• Partitioning Clustering

• Density-Based Clustering

• Distribution Model-Based Clustering

• Hierarchical Clustering

• Fuzzy Clustering
Partitioning Clustering
• It is a type of clustering that divides the data into non-hierarchical groups. It
is also known as the centroid-based method. The most common example of
partitioning clustering is the K-Means algorithm
• In this type, the dataset is divided into a set of k groups, where K is used to
define the number of pre-defined groups. The cluster center is created in
such a way that the distance between the data points of one cluster is
minimum as compared to another cluster centroid.
• The similarity measure chosen for these algorithms are Euclidian distance,
Manhattan Distance or Minkowski Distance.
• The major drawback for centroid-based algorithms is the requirement that
we establish the number of clusters, “k,” either intuitively or scientifically
(using the Elbow Method) before any clustering machine learning system
starts allocating the data points.
K-Means clustering
• It is an iterative algorithm that divides the unlabeled dataset into k different
clusters in such a way that each dataset belongs only one group that has
similar properties.
• It is a centroid-based algorithm, where each cluster is associated with a
centroid. The main aim of this algorithm is to minimize the sum of distances
between the data point.
• Determines the best value for K center points or centroids by an iterative
process.
• Assigns each data point to its closest k-center. Those data points which are
near to the particular k-center, create a cluster.
K-means clustering
Step-1: Select the number K to decide the number of clusters.
Step-2: Select random K points or centroids.
Step-3: Assign each data point to their closest centroid, which will form
the predefined K clusters.
Step-4: Calculate the variance and place a new centroid of each cluster.
Step-5: Repeat the third steps, which means reassign each datapoint to
the new closest centroid of each cluster.
Step-6: If any reassignment occurs, then go to step-4 else go to FINISH.
Step-7: The model is ready.
Density based clustering
• The density-based clustering method connects the highly-dense areas into
clusters, and the arbitrarily shaped distributions are formed as long as the
dense region can be connected.
• This algorithm does it by identifying different clusters in the dataset and
connects the areas of high densities into clusters. The dense areas in data
space are divided from each other by sparser areas.
• These algorithms can face difficulty in clustering the data points if the
dataset has varying densities and high dimensions.
Distribution based clustering
• In the distribution model-based clustering method, the data is divided based
on the probability of how a dataset belongs to a particular distribution. The
grouping is done by assuming some distributions commonly Gaussian
Distribution.
• The example of this type is the Expectation-Maximization Clustering
algorithm that uses Gaussian Mixture Models (GMM).
Hierarchical clustering
• Hierarchical clustering can be used as an alternative for the partitioned
clustering as there is no requirement of pre-specifying the number of
clusters to be created.
• In this technique, the dataset is divided into clusters to create a tree-like
structure, which is also called a dendrogram.
• The observations or any number of clusters can be selected by cutting the
tree at the correct level. The most common example of this method is
the Agglomerative Hierarchical algorithm.
Distance measures in clustering
1. Euclidean Distance
The Euclidean distance is the most widely used distance measure in
clustering. It calculates the straight-line distance between two points in n-
dimensional space. The formula for Euclidean distance is:

where,
• p and q are two data points
• and n is the number of dimensions.
Distance measures in clustering
2. Manhattan Distance

• The Manhattan distance, is the total of the absolute differences between

their Cartesian coordinates, sometimes referred to as the L1 distance or city
block distance.
• When it comes to categorical data this metric is more effective than
Euclidean distance since it is less susceptible to outliers. The formula is: