MODULE III -REGRESSION
Prediction - Correlation - The Regression Line - The Method of Least Squares -
Least Squares Regression - Visual Diagnostics -Numerical Diagnostics -
Inference for Regression - A Regression Model - Inference for the True Slope -
Prediction Intervals - Multiple Regression.
Prediction
• In the essentials of data science, "prediction"
refers to the use of historical data, algorithms,
and machine learning to forecast future events
or behaviors, a process known as predictive
analytics.
• This involves creating mathematical models that
identify patterns and relationships in past data to
anticipate risks,improve decision-making, and
optimize operations
Step Description Examples
Will a customer churn? What will sales
1. Problem Definition Define what you want to predict
be next month?
2. Data Collection Gather relevant data Databases, APIs, sensors, surveys
Handle missing values, encode
3. Data Preparation Clean and structure data
categories, scale/normalize
Customer age, transaction frequency,
4. Feature Engineering & Selection Choose/create the right variables
seasonality
Regression, Classification, Time Series,
5. Model Selection Pick the right model type
Clustering, Anomaly Detection
Train Decision Tree, Neural Network,
6. Model Training Fit the model to training data
Random Forest, etc.
Accuracy, Precision, Recall, RMSE, AUC
7. Model Evaluation Test model performance
depending on problem type
Use trained model to make predictions Predict churn risk, forecast sales, detect
8. Prediction (Inference)
on new data fraud
9. Deployment Put model into production for real use API, dashboard, application
Track performance, update model as
10. Monitoring & Maintenance Detect data drift, retrain with new data
needed
➢ Predictive analytics is the process of using data to forecast future outcomes.
➢ The process uses data analysis, machine learning, artificial intelligence, and statistical models to
find patterns that might predict future behavior.
➢ Organizations can use historic and current data to forecast trends and behaviors seconds, days, or
years into the future with a great deal of precision.
➢ Data scientists use predictive models to identify correlations between different elements in
selected datasets.
Uses and examples of predictive analytics
Conversion and
Fraud detection Risk reduction
purchase prediction
• Predictive analytics examines all • Companies can take actions, • Credit scores, insurance
actions on a company’s network like retargeting online ads to claims, and debt collections
in real time to pinpoint visitors, with data that all use predictive analytics to
abnormalities that indicate fraud
and other vulnerabilities.
predicts a greater likelihood assess and determine the
of conversion and purchase likelihood of future defaults.
intent.
Operational improvement Customer segmentation Maintenance forecasting
• Companies use predictive • By dividing a customer base • Organizations use data to
analytics models to forecast into specific groups,
inventory, manage resources,
predict when routine
marketers can use predictive equipment maintenance
and operate more efficiently. analytics to make forward-
will be required and can
looking decisions to tailor
content to unique audiences. then schedule it before a
problem or malfunction
arises.
Model Type Purpose Output Common Algorithms Applications
Sales forecasting,
Linear Regression,
Predict continuous house price
Regression Numeric (e.g., 45.6) Decision Trees,
numerical values prediction, demand
Neural Networks
estimation
Logistic Regression,
Decision Trees, Spam detection, loan
Categorize data into Categorical (e.g.,
Classification Random Forests, default prediction,
predefined classes Yes/No, A/B/C)
SVM, Neural customer churn
Networks
Stock price
Forecast values from ARIMA, Exponential
Numeric sequence forecasting, weather
Time Series sequential, time- Smoothing, LSTM
over time prediction, energy
based data (Neural Networks)
consumption
Customer
Group similar data K-Means, segmentation,
Cluster labels (e.g.,
Clustering points without Hierarchical market basket
Group 1, Group 2)
predefined labels Clustering, DBSCAN analysis, grouping
outliers
Fraud detection,
Binary Isolation Forest, One-
Detect unusual or equipment failure
Anomaly Detection (Normal/Anomaly) or Class SVM,
rare data points monitoring, intrusion
score Autoencoders
detection
MODULE III -REGRESSION
Prediction - Correlation - The Regression Line - The Method of Least Squares -
Least Squares Regression - Visual Diagnostics -Numerical Diagnostics -
Inference for Regression - A Regression Model - Inference for the True Slope -
Prediction Intervals - Multiple Regression.
Correlation
• Correlation refers to a process for establishing the
relationships between two variables.
• The relationship between two variables in a single
number called the correlation coefficient.
• The correlation coefficient is usually represented
using the symbol r, and it ranges from -1 to +1.
• A correlation coefficient quite close to 0, but either
positive or negative, implies little or no relationship
between the two variables.
• A correlation coefficient close to plus 1 means a
positive relationship between the two variables,
with increases in one of the variables being
associated with increases in the other variable.
• A correlation coefficient close to -1 indicates a negative relationship between two variables, with
an increase in one of the variables being associated with a decrease in the other variable.
• There exists a correlation between two variables X and Y, which means the value of one variable is
found to change in one direction, the value of the other variable is found to change either in the
same direction (i.e. positive change) or in the opposite direction (i.e. negative change).
Types of Correlation
• Positive Correlation – when the values of
the two variables move in the same
direction so that an increase/decrease in
the value of one variable is followed by an
increase/decrease in the value of the
other variable.
• Negative Correlation – when the values of
the two variables move in the opposite
direction so that an increase/decrease in
the value of one variable is followed by
decrease/increase in the value of the
other variable.
• No Correlation – when there is no linear
dependence or no relation between the
two variables.
• Negative Correlation Examples
• Time Spent Running vs. Body Fat
• Time Spent Watching TV vs. Exam Scores
• Positive Correlation Examples
• Temperature vs. Ice Cream Sales
• You feel more financially secure as you save more money.
• No Correlation Examples
• Coffee Consumption vs. Intelligence
Pearson Correlation Coefficient Formula
• Pearson Correlation Coefficient (PCC) is used for measuring the
strength and direction of a linear relationship between two variables
• Pearson Correlation coefficient used for linear dependency between
the data sets.
• The value of the coefficient lies between -1 to +1. When the
coefficient comes down to zero.
• If we get the value of +1, then the data are positively correlated, and -
1 has a negative correlation.
Pearson Correlation Coefficient Table
Example:
•r = 0.85 suggests a strong positive
correlation such as more study time leading
to better test scores.
•r = -0.75 shows a strong negative
correlation like the inverse relationship
between outdoor temperature and heating
costs.
Example 1
Calculate the correlation coefficient for the following data by the help of Pearson's correlation
coefficient formula: X = 21, 31, 25, 40, 47, 38 and Y = 70, 55, 60, 78, 66, 80
• ∑xy= 13937, ∑x=202, ∑y=409, ∑x² =7280, ∑y² =28365
Put n =6 all the values in the Pearson's correlation coefficient formula:-
r= 𝑛 σ 𝑥 𝑦 − σ 𝑥 σ 𝑦 /√ 𝑛 σ 𝑥 ² − σ 𝑥 ² ൣ𝑛 σ 𝑦 ² − σ 𝑦 ²]
r= 6(13937) - (202)(409) / √ [6(7280)-(202)²][6(28365)-(409)²]
r= 1004 / √[2876][2909]
r=1004 / 2892.452938
r=0.3471
The correlation coefficient is 0.3471
REGRESSION LINE
➢ The regression line is a straight line that best fits the data points on a scatter plot.
➢ It shows the general trend of the relationship between the independent variable 𝑋 and the dependent
variable 𝑌.
Dependent and Independent Variables
• The dependent variable is the one that we want to forecast
or analyze.
• The independent variable is hypothesized to affect the
dependent variable.
• Example: we wish to analyze income and we choose the
variable individual’s education that most affects income.
y is income and x is individual’s education
23
Common Types of Regression Models
Linear Regression:
• The simplest and most widely used model, which assumes a linear (straight-line) relationship
between variables.
Polynomial Regression:
• Used when the relationship between variables is not a straight line, but a curve.
Logistic Regression:
• Specifically for predicting binary outcomes (0 or 1, yes or no) by modeling the probability of an
event occurring.
Ridge Regression and Lasso Regression:
• Techniques that help to prevent overfitting by adding a penalty to the model's coefficients,
particularly useful in high-dimensional data.
Decision Tree Regression and Random Forest Regression:
• Ensemble methods that use multiple decision trees to make predictions, improving accuracy and
robustness.
25
MODULE III -REGRESSION
Prediction - Correlation - The Regression Line - The Method of Least Squares -
Least Squares Regression - Visual Diagnostics -Numerical Diagnostics -
Inference for Regression - A Regression Model - Inference for the True Slope -
Prediction Intervals - Multiple Regression.
The Method of Least Square
• The Method of Least Squares is a statistical technique that finds the best-fitting line or curve to a set
of data points by minimizing the sum of the squared differences (residuals or errors) between the
observed data values and the values predicted by the curve.
• The Least Square method is a popular mathematical approach used in data fitting, regression analysis,
and predictive modeling.
• It helps find the best-fit line or curve that minimizes the sum of squared differences between the
observed data points and the predicted values.
• In regression, the difference between the observed value of the dependent variable(Yi ) and the
predicted value(predicted) is called the residuals.
Steps to find the line of Best Fit by using the Least Squares Method :
Step 1: Denote the independent variable values as xi and the dependent ones as yi.
Step 2: Calculate the average values of Xi and Yi as X and Y.
Step 3: Presume the equation of the line of best fit as y = mx + c, where m is the slope of the line and c
represents the intercept of the line on the Y-axis.
Step 4: The slope m can be calculated from the following formula:
Step 5: The intercept c is calculated from the following formula:
c = Y - mX
37
38
39
Real-Life Application of the Least Squares Method
➢ Predicting housing prices using linear regression.
➢ Estimating trends in the financial market.
➢ Fitting Curves to experimental data in physics and chemistry.
➢ Calibrating sensors in engineering.
MODULE III -REGRESSION
Prediction - Correlation - The Regression Line - The Method of Least Squares -
Least Squares Regression - Visual Diagnostics -Numerical Diagnostics -
Inference for Regression - A Regression Model - Inference for the True Slope -
Prediction Intervals - Multiple Regression.
Visual Diagnostics
• Visual diagnostics in data science use charts and graphs to check a model's performance, find
hidden patterns in complex data, and evaluate the validity of assumptions.
• These visualizations go beyond raw data, helping identify issues like over-fitting, under-
fitting, or anomalies by revealing patterns that numerical analysis might miss.
• Residual plots help us make visual assessments of the quality of a linear regression
analysis. Such assessments are called diagnostics.
42
Feature Analysis and Selection
Visualization Type Description
Visualize distribution, central tendency, spread, and outliers. Violin
Box Plots / Violin Plots
plots also show kernel density.
Show frequency distribution of a single feature to identify
Histograms
concentrations or gaps in values.
Display all pairwise scatter plots in a grid to help detect correlations
Scatter Plot Matrices (SPLOMs)
and feature relationships.
Map features onto a circle; data points are "pulled" towards
Radial Visualizations (Radviz) features with higher values, revealing clusters and feature
influence.
43
Model Evaluation and Tuning
Visualization Type Description
Plot residuals vs. fitted values in regression. Random scatter
Residual Plots
around zero indicates a good fit.
Detecting Non-linearity (via Residual Curved pattern suggests missing non-linear relationships in
Plots) the model.
Uneven spread (e.g., flaring) in residuals indicates
Detecting Heteroscedasticity
inconsistent error variance.
Show training and validation scores across hyperparameter
Validation Curves
values; visualize bias-variance tradeoff.
Compare how different models partition data; helps in model
Model Comparison Plots
selection.
Visualize performance of models across hyperparameter
Hyperparameter Heatmaps
combinations from grid search.
44
The residual regression line for the data in the table is y=2.15*x+2.35
45
The residual regression line for the data in the table is y=1.75*x+2.75
46
Example of estimating the heights of adult children (the response) based on
the midparent height (the predictor)
➢ This residual plot indicates that linear regression was a reasonable method of estimation.
➢ Notice how the residuals are distributed fairly symmetrically above and below the horizontal line at 0,
corresponding to the original scatter plot being roughly symmetrical above and below.
➢ Notice also that the vertical spread of the plot is fairly even across the most common values of the
children’s heights. 47
48
Detecting Nonlinearity
Example :Here is a regression of age (the response) on length (the predictor). The correlation between the two
variables is substantial, at 0.83.
➢ While you can spot the non-linearity in the original scatter, it is more clearly evident in the residual
plot.
➢ At the low end of the lengths, the residuals are almost all positive; then they are almost all negative;
then positive again at the high end of lengths
➢ That means it would have been better to use a curve instead of a straight line to estimate the ages.
➢ When a residual plot shows a pattern, there may be a non-linear relation between the variables.
Model Evaluation tools
Tool What It Does Common Visuals
Helps evaluate and tune ML Residual plots, validation curves,
Yellowbrick
models confusion matrices
Custom plots like residuals,
Matplotlib Basic plotting library
learning curves
Distribution and residual plots,
Seaborn Makes statistical plots easier
heatmaps
Plotly Interactive charts ROC curves, confusion matrices
Loss/accuracy curves,
TensorBoard Visualize deep learning training
embeddings
50
MODULE III -REGRESSION
Prediction - Correlation - The Regression Line - The Method of Least Squares -
Least Squares Regression - Visual Diagnostics -Numerical Diagnostics -
Inference for Regression - A Regression Model - Inference for the True Slope -
Prediction Intervals - Multiple Regression.
Numerical Diagnostics
Diagnostic Type Description Purpose
Mean / Median Central tendency measures Understand feature center values
Standard Deviation Measure of spread or variability Assess data dispersion
Measures linear relationship
Correlation Coefficient Identify feature dependencies
between variables
Statistical significance of
P-values Test hypotheses about features
relationships
Proportion of variance explained by
R-squared Measure model goodness-of-fit
model
Root Mean Squared Error / Mean Evaluate regression prediction
RMSE / MAE
Absolute Error accuracy
Counts of true/false Evaluate classification
Confusion Matrix
positives/negatives performance
Assess model’s positive prediction
Precision / Recall Classification metrics
quality
52
MODULE III -REGRESSION
Prediction - Correlation - The Regression Line - The Method of Least Squares -
Least Squares Regression - Visual Diagnostics -Numerical Diagnostics -
Inference for Regression - A Regression Model - Inference for the True Slope -
Prediction Intervals - Multiple Regression.
Conditions for inference
The observations are independent.
The relationship is indeed linear.
The standard deviation of y,σ, is the same for all values of x.
The response y varies normally
around its mean.
Using residual plots to check for regression validity
The residuals (y−ŷ) give useful information about the contribution of individual data points to the
overall pattern of scatter.
We view the residuals in a residual plot:
If residuals are scattered randomly around 0 with uniform variation, it indicates that the data fit a
linear model, have normally distributed residuals for each value of x, and constant standard
deviation σ.
Residuals are randomly scattered
→ good!
Curved pattern
→ the relationship is not linear.
Change in variability across plot
→σ not equal for all values of x.
What is the relationship between
the average speed a car is
driven and its fuel efficiency?
We plot fuel efficiency (in miles
per gallon, MPG) against average
speed (in miles per hour, MPH)
for a random sample of 60 cars.
The relationship is curved.
When speed is log transformed
(log of miles per hour, LOGMPH)
the new scatterplot shows a
positive, linear relationship.
Residual plot:
The spread of the residuals is
reasonably random—no clear pattern.
The relationship is indeed linear.
But we see one low residual (3.8, −4)
and one potentially influential point
(2.5, 0.5).
Normal quantile plot for residuals:
The plot is fairly straight, supporting
the assumption of normally distributed
residuals.
➔ Data okay for inference.
Inference for Regression
Metric What It Measures Simple Explanation
How much of the variation in Y is Closer to 1 means the model fits
R-squared (R²)
explained by X data well
Like R² but adjusts for number of Prevents overestimating fit when
Adjusted R-squared
predictors adding useless features
Average squared difference
Lower is better; big errors
Mean Squared Error (MSE) between actual and predicted
penalized more
values
Same as MSE but in original units
Root Mean Squared Error (RMSE) Square root of MSE
of Y
Average absolute difference
Lower is better; less sensitive to
Mean Absolute Error (MAE) between actual and predicted
big errors
values
Differences between actual and Analyze patterns to check model
Residuals
predicted values assumptions
59
MODULE III -REGRESSION
Prediction - Correlation - The Regression Line - The Method of Least Squares -
Least Squares Regression - Visual Diagnostics -Numerical Diagnostics -
Inference for Regression - A Regression Model - Inference for the True Slope -
Prediction Intervals - Multiple Regression.
A Regression Model
A regression model is a statistical model that estimates the relationship between one dependent variable and one
or more independent variables using a line (or a plane in the case of two or more independent variables).
The model tries to find the best-fitting line (or curve) that describes how changes in the independent
variable(s) affect the dependent [Link] line is determined by estimating coefficients (like slope
and intercept) that minimize the difference between the observed data points and the predicted values
from the [Link] predicted value can be used to forecast or infer outcomes for new data.
61
62
Step Description
Clean and preprocess your
Step 1: Collect and Prepare Data dataset, ensuring variables are
relevant and properly formatted.
Decide whether a simple,
Step 2: Choose the Right Model multiple, or polynomial
regression fits your data.
Use tools like Python (Scikit-
Step 3: Fit the Model learn, Statsmodels), R, or Excel
to run the regression.
Check R-squared (fit), p-values
Step 4: Analyze the Model (significance), and residuals
(error distribution).
Use testing data or cross-
Step 5: Validate and Predict validation, then apply the model
for forecasting or decisions.
63
Types of Regression Models
Regression Type Description Example
Involves one independent and Predicting house price based on
Simple Linear Regression
one dependent variable. square footage.
Uses two or more independent Predicting sales using advertising
Multiple Linear Regression variables to predict the dependent spend, number of salespeople,
variable. and store location.
Uses higher-degree terms (e.g., Predicting growth rates that
Polynomial Regression (X^2), (X^3)) to capture non-linear accelerate or decelerate over
relationships. time.
Used for binary outcomes
Predicting whether a customer will
Logistic Regression (yes/no, true/false). Not a
buy a product (1) or not (0).
traditional regression line.
64
MODULE III -REGRESSION
Prediction - Correlation - The Regression Line - The Method of Least Squares -
Least Squares Regression - Visual Diagnostics -Numerical Diagnostics -
Inference for Regression - A Regression Model - Inference for the True Slope -
Prediction Intervals - Multiple Regression.
Inference for the True Slope
In regression analysis, inference for the true slope means testing whether the independent variable
actually has a statistically significant effect on the dependent variable in the population
66
67
Confidence Interval (CI)
confidence interval gives you a range where the true value is likely to be.
If you say "95% confidence interval," it means if you repeated the study many times, 95% of those intervals would
contain the true value.
Example:Imagine you want to find out how much studying affects test scores. You collect data and find that for every
extra hour studied, the test score goes up by 2 points. But you’re not 100% sure — so you calculate a confidence
interval of 1.5 to 2.5 points per hour. This means you are pretty sure the real effect is somewhere between 1.5 and
2.5 points for each hour studied.
Bootstrapping
Bootstrapping is a way to check how good your estimate is by pretending to do the experiment many times,
using your own data again and again with random picks.
This helps you figure out how much your estimate might change if you did the study again.
Example:You take your original 30 students' study and test data and create many new "fake" datasets by
randomly picking students from your original group (sometimes picking the same student more than once). For
each fake dataset, you calculate the slope again. After doing this 1,000 times, you look at all those slopes to
see the range where most fall — this range helps you create your confidence interval.
68
69
70
Prediction Interval
A Prediction Interval gives a range where you expect a new individual observation (response value) to fall,
based on a given predictor value x∗x^*x∗. It accounts for both the uncertainty in estimating the regression line
and the natural variability of individual outcomes around the line.
71
Multiple Regression
• Multiple Regression is a statistical technique that models the relationship between one dependent variable (als
called the outcome or response variable) and two or more independent variables (also called predictors or
explanatory variables).
• To predict the value of a dependent variable based on several independent variables.
• To understand how each independent variable is associated with the dependent variable, holding other
variables constant.
72
73
74
75