MODULE 5
REGRESSION AND CLASSIFICATION TECHNIQUES
Module 5: Regression and Classification Techniques, Linear Regression , Simple Linear
Regression and Assumptions, Multivariate Linear Regression, Model Assessment and
Variable Importance, Subset Selection, Classification Techniques , Classification using
Logistic Regression.
Important Questions
1. What is the difference between regression and classification in data science?
2. What are the key assumptions of simple linear regression, and why are they
important?
3. How does multivariate linear regression extend simple linear regression, and what
challenges does it introduce?
4. What are the different ways to assess a regression model’s performance (e.g., R²,
RMSE, MAE)?
5. How do you determine the importance of variables in a regression model?
6. What is subset selection in regression, and how does it help in model building?
7. What is logistic regression, and how is it used for classification problems?
8. How do you evaluate the performance of a classification model (e.g., precision, recall,
F1-score, ROC curve)?
Regression and Classification Techniques
o In data science and machine learning, regression and classification are two primary
types of predictive modelling techniques used to make predictions based on data.
o While regression is used for continuous outcome prediction, classification is used
for predicting categorical outcomes. Here's a breakdown of both techniques.
Linear Regression
Definition: Linear regression is one of the most fundamental and widely used
techniques in statistics and machine learning. It is used for predicting a continuous
dependent (target) variable Y based on one or more independent (predictor)
variables (X1, X2, …., Xn). This statistical method models the relationship between a
target and the predictor variables.
Types of Regression(for syllabus)
➢ Simple Linear Regression(SLR) – Predicts a target variable using one
independent variable.
➢ Multiple Linear Regression
➢ Multivariate Linear Regression(MLR) - Predicts a target variable using more
independent variables.
Simple Linear Regression
➢ Definition: Simple linear regression models the relationship between a single
independent variable X and a dependent variable Y using a linear equation.
➢ Expression/Equation:
The equation for simple linear regression is:
➢ Evaluation Metrics:
3. Residual Plot:
• A residual plot shows the difference between the actual and predicted values.
• It is used to check for homoscedasticity and independence of errors.
• A good residual plot should show no patterns, indicating that the residuals are
randomly distributed and that the model is well specified.
➢ Applications of Linear Regression:
1. In Sales Forecasting, it is used to predict future sales based on advertising
spend (e.g., TV, social media)
2. In real estate price prediction, is it used to estimate house prices based
on size (square footage)
3. In academic performance predictions, it is used to predict exam scores
from hours studied.
4. In Height-Weight Relationship, it is used to predict a person’s weight
based on their height
5. In Manufacturing Quality Control, it is used to predict number of defects
based on machine run time or load
6. In Energy Consumption Estimation, it is used to predict energy usage
based on outside temperature
7. In Fuel Efficiency Modelling, it is used to predict fuel consumption based
on speed or weight
8. In Website Traffic Prediction, it is used to predict number of clicks from
number of ad impressions
➢ Assumptions for Linear Regression:
✓ For the simple linear regression model to produce reliable and valid results,
certain assumptions must be satisfied.
✓ These assumptions ensure that the model is properly specified and the
statistical tests (e.g., hypothesis tests on the coefficients) are valid.
✓ Violating these assumptions can lead to biased, inefficient, or misleading
results.
1. Linearity:
o The relationship between the independent variable X and the dependent
variable Y must be linear.
o In other words, the model assumes that a straight line can best describe the
relationship between the two variables.
o How to check: Plot a scatter plot of the data to visually check for a linear
relationship between X and Y. If the relationship appears curved or non-
linear, simple linear regression may not be appropriate.
2. Independence of Errors:
o The residuals (errors) should be independent of each other. This means
that the error for one observation should not influence the error for
another observation.
o This assumption is important for the validity of hypothesis tests and
confidence intervals.
o How to check: This can be checked using Durbin-Watson test or by
plotting residuals versus time (if the data is time-series data). If there is a
pattern (e.g., autocorrelation), then this assumption is violated.
3. Homoscedasticity:
o Homoscedasticity means that the variance of the residuals (errors) should
remain constant for all levels of the independent variable X.
o In other words, the spread of the residuals should not increase or decrease
systematically as the value of X changes.
o How to check: Plot the residuals versus the fitted values. If the plot shows
a fan or cone shape (i.e., the spread of residuals increases or decreases with
X), the assumption of homoscedasticity is violated. This issue is called
heteroscedasticity.
4. Normality of Errors:
o The residuals (errors) should be normally distributed.
o This assumption allows us to make statistical inferences, such as
hypothesis tests for the regression coefficients.
o If the errors are not normally distributed, the confidence intervals and
significance tests may not be valid.
o How to check: You can use a histogram or a Q-Q plot (Quantile-Quantile
plot) to visually assess the normality of the residuals. Additionally, a
Shapiro-Wilk test or Kolmogorov-Smirnov test can be used to formally test
for normality.
5. No Multicollinearity:
o In simple linear regression, this assumption is naturally satisfied because
there is only one independent variable.
o However, if there were multiple independent variables (in multiple linear
regression), the independent variables should not be highly correlated with
each other.
o High correlation between predictors can cause instability in the estimated
coefficients, making them unreliable.
o How to check: In simple linear regression, multicollinearity is not an issue.
However, in multiple linear regression, you can check for multicollinearity
using Variance Inflation Factor (VIF).
Expression/Mathematical Formulation