DATA SCIENCE & MACHINE LEARNING

Regression Analysis
Complete Study Notes

Simple & Multiple Regression | Model Evaluation

Residual Plots | Distribution Plots
Polynomial Regression | Pipelines
In-Sample Evaluation | Prediction & Decision Making

Comprehensive Reference Notes

Model Development · Evaluation · Prediction

Regression Analysis — Complete Study Notes Data Science & Machine Learning

Table of Contents
1. Simple Linear Regression 3

2. Multiple Linear Regression 4

3. Model Evaluation using Visualization 5

4. Residual Plots 6

5. Distribution Plots 7

6. Polynomial Regression & Pipelines 8

7. Measures for In-Sample Evaluation 10

8. Prediction & Decision Making 11

For educational use Page 2

Regression Analysis — Complete Study Notes Data Science & Machine Learning

CHAPTER 1
Simple Linear Regression 1

1.1 What is Simple Linear Regression?

Simple Linear Regression (SLR) models the relationship between one independent variable (X) and one
dependent/target variable (Y) by fitting a straight line through the data. It is the foundation of all
regression techniques.

Equation of the regression line

Y = b0 + b1X + ε

Key Terms
Term Symbol Meaning

Intercept b■ Value of Y when X = 0 (where line crosses Y-axis)

Slope b■ Change in Y for a one-unit increase in X

Error term ε Random noise / unexplained variation in Y

Fitted value ■ Predicted value of Y for a given X

Residual e Observed Y − Predicted ■ (actual minus predicted)

1.2 Least Squares Estimation

The Ordinary Least Squares (OLS) method finds the line that minimises the sum of squared residuals
(SSR). The formulas for the slope and intercept are:

OLS coefficient formulas

b■ = Σ[(x■ − x■)(y■ − ■)] / Σ[(x■ − x■)²] | b■ = ■ − b■ ·
x■

■ Key Assumptions
Assumptions of SLR: (1) Linearity — true relationship is linear. (2) Independence — observations
are independent. (3) Homoscedasticity — constant variance of errors. (4) Normality — residuals are
normally distributed. (5) No measurement error in X.

For educational use Page 3

Regression Analysis — Complete Study Notes Data Science & Machine Learning

CHAPTER 2
Multiple Linear Regression 2

2.1 Extending to Multiple Predictors

Multiple Linear Regression (MLR) extends SLR to include two or more independent variables (X■, X■,
…, X■). Each predictor has its own coefficient, measuring its effect on Y while holding all other predictors
constant (partial effect).

General MLR equation

Y = b■ + b■X■ + b■X■ + … + b■X■ + ε

2.2 Model Development Steps

Step Stage Activity

Step 1 Data Collection & Cleaning Gather data, handle missing values, remove outliers.

Step 2 Exploratory Data Analysis Scatter plots, correlation matrix, distribution checks.

Step 3 Feature Selection Choose relevant predictors; avoid multicollinearity.

Step 4 Model Fitting Use OLS (or sklearn LinearRegression) to estimate coefficients.

Step 5 Evaluation Assess R², MSE, MAE, residual plots.

Step 6 Prediction Use the fitted model on new/unseen data.

2.3 Multicollinearity

■ Watch Out: Multicollinearity

Multicollinearity occurs when two or more predictors are highly correlated with each other. It inflates
standard errors and makes coefficient interpretation unreliable. Detection: Variance Inflation Factor
(VIF). If VIF > 10 for a variable, consider removing it or using regularisation (Ridge/Lasso).

For educational use Page 4

Regression Analysis — Complete Study Notes Data Science & Machine Learning

CHAPTER 3
Model Evaluation using Visualization 3

3.1 Why Visualise?

Numerical metrics (R², MSE) give an overall summary but can miss systematic patterns. Visual diagnostics
reveal non-linearity, heteroscedasticity, outliers, and influential points that numbers alone cannot
capture.

3.2 Core Visualisation Toolkit

Plot Type What to Look For Good Sign

Scatter: Y vs ■ Points close to 45° diagonal line Tight cluster on the line

Residual vs Fitted Random scatter around zero No pattern / funnel shape

Q-Q Plot Points on the diagonal reference lineStraight line → normal residuals

Scale-Location Flat red line, equal spread Homoscedasticity confirmed

Cook's Distance Points below 0.5 or 1.0 threshold No highly influential obs.

Partial Regression Slope of each predictor controlling others

Linear partial relationships

3.3 Regression Plot (seaborn)

A regression plot overlays the best-fit line with a confidence interval band on the scatter plot, giving an
immediate visual sense of the linear fit quality.

import seaborn as sns

import [Link] as plt

[Link](x='engine-size', y='price', data=df)

[Link]('Engine Size')
[Link]('Car Price')
[Link]('Simple Regression: Engine Size vs Price')
[Link]()

The shaded band around the regression line represents the 95% confidence interval for the mean
prediction. A narrow band indicates high confidence in the fit. Points far from the line are potential
outliers.

For educational use Page 5

Regression Analysis — Complete Study Notes Data Science & Machine Learning

CHAPTER 4
Residual Plots 4

4.1 What is a Residual?

A residual is the difference between the observed value and the value predicted by the model. Residuals
are the leftover variation that the model does not explain.

Residual formula
e■ = y■ − ■■

4.2 Residual vs Fitted Plot

This is the most important diagnostic plot. Residuals are plotted on the Y-axis and fitted (predicted) values
on the X-axis.

Pattern Observed Diagnosis Remediation

Random scatter around 0 Model is correct ✓ No action needed

U-shape or arch Non-linearity Add polynomial terms

Fan / funnel shape Heteroscedasticity Log-transform Y; WLS

Points far from 0 Outliers present Investigate / remove

Systematic trend Omitted variable Add missing predictor

4.3 Creating Residual Plots (Code)

from sklearn.linear_model import LinearRegression
import numpy as np
import [Link] as plt

model = LinearRegression()
[Link](X_train, y_train)
y_pred = [Link](X_train)

residuals = y_train - y_pred

[Link](y_pred, residuals, alpha=0.6)

[Link](y=0, color='red', linestyle='--')
[Link]('Fitted Values')
[Link]('Residuals')
[Link]('Residual vs Fitted Plot')
[Link]()

■ Key Rule
Rule of Thumb: If residuals show any systematic pattern (curved, fanned, trending), the model's
assumptions are violated and the model needs to be revised. Ideally residuals should look like random
white noise centred at zero.

For educational use Page 6

Regression Analysis — Complete Study Notes Data Science & Machine Learning

CHAPTER 5
Distribution Plots 5

5.1 Purpose of Distribution Plots

Distribution plots compare the distribution of predicted values (■) against the distribution of actual
values (Y). If the model is a good fit, the two distributions should closely overlap.

5.2 Creating a Distribution Plot

import seaborn as sns
import [Link] as plt

ax = [Link](y_train, color='blue', label='Actual Values', shade=True)

[Link](y_pred, color='red', label='Predicted Values', shade=True, ax=ax)

[Link]('Price')
[Link]('Density')
[Link]('Actual vs Predicted Distribution')
[Link]()
[Link]()

5.3 Interpreting Distribution Plots

Observation Interpretation

Peaks coincide perfectly Excellent fit — model captures the data distribution well

Predicted peak shifted left/right Systematic bias — model over- or under-predicts

Predicted distribution is narrower Model underestimates variance / too smooth

Predicted distribution is wider Model overestimates variance / too noisy

Multiple predicted modes missing Model cannot capture multimodality

■ Pro Tip
Use both KDE (Kernel Density Estimate) and histogram overlays for a complete picture. KDE
smooths the distribution; histograms show the raw bin counts. Both together provide the most
informative comparison.

For educational use Page 7

Regression Analysis — Complete Study Notes Data Science & Machine Learning

CHAPTER 6
Polynomial Regression & Pipelines 6

6.1 Limitations of Linear Regression

Real-world data often exhibits curvilinear (non-linear) relationships. A straight line cannot capture these
patterns, resulting in high bias. Polynomial regression extends linear regression by adding higher-degree
terms (X², X³, …) as additional features.

6.2 Polynomial Regression Model

nth-degree polynomial regression

Y = b■ + b■X + b■X² + b■X³ + … + b■X■ + ε

Degree Name Curve Shape When to Use

1 Linear Straight line Linear relationship

2 Quadratic U-shape / inverted-U Single peak or valley

3 Cubic S-curve / inflection Multiple inflection points

n≥4 Higher Complex wiggly curve Risk of overfitting — use with care

6.3 Implementing Polynomial Regression

from [Link] import PolynomialFeatures
from sklearn.linear_model import LinearRegression
import numpy as np

# Step 1: Generate polynomial features (degree=2)

poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X) # X: original feature array

# Step 2: Fit linear regression on transformed features

model = LinearRegression()
[Link](X_poly, y)

# Step 3: Predict
y_pred = [Link](X_poly)

6.4 Scikit-Learn Pipelines

A Pipeline chains multiple preprocessing and modelling steps into a single, reusable object. This
eliminates data leakage, simplifies code, and makes the workflow reproducible. All steps except the last
must implement transform(); the last step must implement fit().

from [Link] import Pipeline

from [Link] import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression

pipe = Pipeline([

For educational use Page 8

Regression Analysis — Complete Study Notes Data Science & Machine Learning

('scaler', StandardScaler()), # Step 1: Standardise features

('poly', PolynomialFeatures(degree=2)),# Step 2: Polynomial features
('model', LinearRegression()) # Step 3: Fit regression
])

[Link](X_train, y_train)
y_pred = [Link](X_test)

■ Benefits
Advantages of Pipelines: (1) Prevents data leakage — scaling is fit only on training data. (2) Single
fit/predict call handles all steps. (3) Easy integration with GridSearchCV for hyperparameter tuning. (4)
Serialisable — save and reload the entire workflow with joblib.

6.5 Overfitting vs Underfitting

OVERFITTING (High Degree) UNDERFITTING (Low Degree)
• Low training error • High training error
• High test error • High test error
• Model too complex • Model too simple
• Memorises noise • Misses patterns
• High variance • High bias
• Solution: reduce degree / regularise • Solution: increase degree / add features

For educational use Page 9

Regression Analysis — Complete Study Notes Data Science & Machine Learning

CHAPTER 7
Measures for In-Sample Evaluation 7

7.1 Why Numerical Metrics Matter

In-sample evaluation metrics quantify model performance on the training data. While they do not replace
out-of-sample testing, they provide essential baselines and guide model selection and improvement.

7.2 Key Regression Metrics

Metric Formula Range Interpretation

R² 1 − SSres/SStot 0 to 1 Proportion of Y variance explained by the model.

(Coefficient of (can be negative) R²=1 is perfect; R²=0 means no better than mean.
Determination)

Adjusted R² 1−[(1−R²)(n−1)/(n−k−1)] ≤ R² Penalises for adding useless predictors. Preferred for MLR model com

MSE Σ(y■−■■)² / n 0 to ∞ Average squared error. Penalises large errors heavily. Same unit² as
(Mean Squared Error)

RMSE √MSE 0 to ∞ Same unit as Y. More interpretable than MSE. Lower is better.
(Root MSE)

MAE Σ|y■−■■| / n 0 to ∞ Average absolute deviation. Robust to outliers. Lower is better.

(Mean Absolute Error)

7.3 Computing Metrics in Python

from [Link] import r2_score, mean_squared_error, mean_absolute_error
import numpy as np

r2 = r2_score(y_test, y_pred)
mse = mean_squared_error(y_test, y_pred)
rmse = [Link](mse)
mae = mean_absolute_error(y_test, y_pred)

print(f'R² = {r2:.4f}')
print(f'MSE = {mse:.2f}')
print(f'RMSE = {rmse:.2f}')
print(f'MAE = {mae:.2f}')

■ Decision Rule
Model Comparison Rule: When comparing two models, prefer the one with higher R² / Adjusted R²
AND lower MSE / RMSE / MAE. Use Adjusted R² (not plain R²) when adding or removing predictors
in MLR, because plain R² always increases when more variables are added.

For educational use Page 10

Regression Analysis — Complete Study Notes Data Science & Machine Learning

CHAPTER 8
Prediction & Decision Making 8

8.1 Making Predictions with a Fitted Model

Once a regression model is fitted and validated, it can generate predictions for new, unseen input data.
There are two important types of predictions:

Prediction Type What it Estimates Confidence Interval

Mean Response (E[Y|X]) Average value of Y for given X Narrower — less uncertainty

Individual Response (Y|X) Wider — includes random error ε

Value for one specific new observation

8.2 Prediction Code Example

import numpy as np
from sklearn.linear_model import LinearRegression

# Train the model

model = LinearRegression()
[Link](X_train, y_train)

# Predict on test set

y_pred = [Link](X_test)

# Predict a single new observation

new_data = [Link]([[2.5, 3, 1500]]) # Example: 3 features
predicted_price = [Link](new_data)
print(f'Predicted Price: {predicted_price[0]:.2f}')

8.3 Decision Making Framework

Model predictions alone are insufficient for business decisions. A structured framework should be followed:

# Stage Description

1 Define Objective What decision will the prediction inform? (e.g., set a price)

2 Validate Model Check R², RMSE, residual plots. Is the model fit for purpose?

3 Generate Prediction Apply model to new input data using .predict()

4 Assess Uncertainty Report confidence/prediction intervals, not just point estimates.

5 Domain Sanity Check Does the prediction make business/domain sense?

6 Make Decision Use prediction + uncertainty + domain knowledge together.

7 Monitor & Iterate Track real outcomes vs predictions; retrain when drift is detected.

8.4 Extrapolation Warning

For educational use Page 11

Regression Analysis — Complete Study Notes Data Science & Machine Learning

■ Extrapolation Risk
Extrapolation — predicting outside the range of training data — is dangerous. The model has no
information about patterns beyond its training range, and predictions can become wildly inaccurate.
Always check that new input values lie within the range of X values seen during training (min ≤
Xnew ≤ max).

8.5 Train / Test Split Best Practices

Training Set Rules Testing Set Rules
• Use 70–80% data for training • Never use test data during training
• Use 20–30% data for testing • Report metrics on the test set
• Shuffle data before splitting • Use cross-validation for small datasets
• Set random_state for reproducibility • K-Fold CV: k=5 or k=10 is standard
• Stratify if classes are imbalanced • Evaluate final model on held-out test set once

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.2, random_state=42
)

Quick Reference Summary

Topic Key Concept Python Tool

Simple Regression Y = b■ + b■X LinearRegression()

Multiple Regression Y = b■ + b■X■ + … + b■X■ LinearRegression()

Residual Plot Residuals vs Fitted — check randomness

[Link](■, e)

Distribution Plot Compare actual vs predicted KDE [Link]()

Polynomial Regression Add X², X³ terms PolynomialFeatures()

Pipeline Chain preprocessing + model Pipeline([...])

R² Score Variance explained (0–1) r2_score()

MSE / RMSE Average squared / root error mean_squared_error()

Prediction [Link](X_new) .predict()

Train/Test Split 80% train, 20% test train_test_split()

For educational use Page 12