Linear & Polynomial Regression — Revision Notes

LINEAR & POLYNOMIAL

REGRESSION
Complete End-to-End Notes

Simple Linear · Multiple Linear · Polynomial

Formulas · Weights & Derivations · Assumptions · Metrics · Code

Page 1
 Linear & Polynomial Regression — Revision Notes

Contents
1. The Big Picture
2. Simple Linear Regression
3. How the Best-Fit Line is Found (OLS & Gradient Descent)
4. Multiple Linear Regression
5. Polynomial Regression
6. Model Evaluation Metrics
7. Assumptions of Linear Regression
8. Bias-Variance & Regularization (Ridge, Lasso, Elastic Net)
9. One-Page Cheat Sheet

Page 2
 Linear & Polynomial Regression — Revision Notes

1. The Big Picture

Regression is about drawing the best possible line (or curve) through your data so you can predict a continuous
number — price, temperature, salary, score — from one or more input features.

Type Inputs Shape it fits Formula

Simple Linear Regression 1 feature A straight line y = b₀ + b₁x

Multiple Linear Regression 2+ features A flat plane / hyperplane y = b₀ + b₁x₁ + b₂x₂ + … + bₙxₙ

Polynomial Regression 1+ features, curved A curve y = b₀ + b₁x + b₂x² + … + bₙxⁿ

The one idea that ties all three together

• All three are “linear models” in the statistical sense — they're linear in their weights (b₀, b₁, b₂…), even Polynomial
Regression, which just feeds in x, x², x³ as if they were separate features.
• “Training” a regression model always means one thing: find the weights that make the predicted line/curve as close
as possible to the real data points.
• “As close as possible” is measured by a cost function — almost always Mean Squared Error (MSE).

Page 3
 Linear & Polynomial Regression — Revision Notes

2. Simple Linear Regression

Simple Linear Regression predicts one output (y) from exactly one input (x), by fitting the straight line that best passes
through the data points.

2.1 The equation

y = b0 + b1 * x

Symbol Name Meaning

y Target / dependent variable The value you're trying to predict

x Feature / independent variable The input you're predicting from

Predicted value of y when x = 0 —

b0 Intercept (bias)
where the line crosses the y-axis

How much y changes for a 1-unit

b1 Slope (weight / coefficient)
increase in x

Reading the slope

• b1 > 0 → y increases as x increases (positive relationship)
• b1 < 0 → y decreases as x increases (negative relationship)
• b1 = 0 → x has no linear effect on y at all

2.2 Finding the weights: Ordinary Least Squares (OLS)

“Best fit” has a precise mathematical meaning: the line that minimizes the sum of squared vertical distances between
each real point and the line's prediction. Those distances are called residuals.

Cost function — Mean Squared Error (MSE)

J(b0, b1) = (1/n) * SUM( yi - y_hat_i )^2

where y_hat_i = b0 + b1*xi

We square the residuals so positive and negative errors don't cancel out, and so bigger mistakes are punished more
heavily. Minimizing J with respect to b0 and b1 (using calculus — setting partial derivatives to zero) gives a direct,
closed-form formula for the weights:

Closed-form solution

b1 = SUM( (xi - x_mean) * (yi - y_mean) ) / SUM( (xi - x_mean)^2 )

b0 = y_mean - b1 * x_mean

● b1's numerator is essentially the covariance between x and y — how much they move together.
● b1's denominator is the variance of x — how spread out x is.
● So slope = covariance(x, y) / variance(x).

Page 4
 Linear & Polynomial Regression — Revision Notes

2.3 Worked example

Hours studied (x) vs. exam score (y) for 5 students: (1, 50), (2, 55), (3, 65), (4, 70), (5, 80).

import numpy as np

x = [Link]([1, 2, 3, 4, 5])
y = [Link]([50, 55, 65, 70, 80])

x_mean, y_mean = [Link](), [Link]()

b1 = [Link]((x - x_mean) * (y - y_mean)) / [Link]((x - x_mean) ** 2)

b0 = y_mean - b1 * x_mean

print(f"b0 (intercept) = {b0:.2f}")

print(f"b1 (slope) = {b1:.2f}")
# b0 ~= 42.0, b1 ~= 7.5
# Interpretation: each extra hour of study adds ~7.5 points to the exam score

2.4 Code: scikit-learn

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

X = [Link]([1, 2, 3, 4, 5]).reshape(-1, 1) # sklearn needs a 2D array of features

y = [Link]([50, 55, 65, 70, 80])

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.2, random_state=42
)

model = LinearRegression()
[Link](X_train, y_train)

print("Intercept (b0):", model.intercept_)

print("Slope (b1):", model.coef_[0])

y_pred = [Link](X_test)

Page 5
 Linear & Polynomial Regression — Revision Notes

3. How the Best-Fit Line is Found

There are two ways any linear model actually learns its weights. Both minimize the same cost function (MSE) — they
just get there differently.

3.1 Method 1 — Ordinary Least Squares (closed-form / “normal equation”)

Solve directly for the weights using linear algebra — no iteration, no guessing. For multiple features this generalizes to
the Normal Equation:

beta = (X^T X)^(-1) X^T y

Symbol Meaning

Feature matrix (n rows x p features, plus a column of 1s for

X
the intercept)

y Target vector (n x 1)

beta Vector of all the weights, [b0, b1, b2, ..., bp]

X^T Transpose of X

( )^(-1) Matrix inverse

Pros & cons of the closed form

• Pros: exact answer in one step, no learning rate to tune, no risk of not converging.
• Cons: computing (X^T X)^(-1) is roughly O(p^3) — becomes very slow once you have thousands of features; fails if
X^T X isn't invertible (perfect multicollinearity).

3.2 Method 2 — Gradient Descent

Start with random weights, then repeatedly nudge them in the direction that reduces the cost function the fastest,
until the cost stops improving. This is the method that scales to huge datasets and is the backbone of how neural
networks learn too.

The update rule

wj := wj - alpha * dJ/dwj

Symbol Meaning

wj The j-th weight (b0, b1, b2, ...)

alpha Learning rate — how big a step to take each round

The gradient — the slope of the cost function with respect to

dJ/dwj
that weight

“Update to” — all weights are updated simultaneously each

:=
iteration
For simple linear regression, the two gradients work out to:

Page 6
 Linear & Polynomial Regression — Revision Notes

dJ/db0 = -(2/n) * SUM( yi - y_hat_i )

dJ/db1 = -(2/n) * SUM( (yi - y_hat_i) * xi )

Choosing the learning rate

Learning rate What happens

Too small Converges, but painfully slowly — wastes compute

Overshoots the minimum, can bounce around or diverge

Too large
entirely

Just right Steadily decreases the cost each iteration until it flattens out

Code: gradient descent from scratch

import numpy as np

x = [Link]([1, 2, 3, 4, 5], dtype=float)

y = [Link]([50, 55, 65, 70, 80], dtype=float)
n = len(x)

b0, b1 = 0.0, 0.0 # start at zero

alpha = 0.01 # learning rate
epochs = 1000

for _ in range(epochs):
y_pred = b0 + b1 * x
d_b0 = -(2/n) * [Link](y - y_pred)
d_b1 = -(2/n) * [Link]((y - y_pred) * x)
b0 -= alpha * d_b0
b1 -= alpha * d_b1

print(f"b0 = {b0:.2f}, b1 = {b1:.2f}") # converges close to the OLS answer

3.3 OLS vs. Gradient Descent

OLS (Normal Equation) Gradient Descent

Type of solution Exact, one step Approximate, iterative

Speed on small data Fast Fast

Speed on large data / many features Slow (matrix inversion is expensive) Scales well

Needs a learning rate? No Yes — must be tuned

Works for other models (logistic

No, only linear regression Yes — the universal workhorse
regression, neural nets)?

Page 7
 Linear & Polynomial Regression — Revision Notes

4. Multiple Linear Regression

Multiple Linear Regression is the exact same idea as simple linear regression, except now there are two or more input
features. Instead of fitting a line, you're fitting a flat plane (2 features) or a hyperplane (3+ features) through the data.

4.1 The equation

y = b0 + b1*x1 + b2*x2 + ... + bn*xn

In matrix form, this is written compactly as:

y = X * beta

Symbol Meaning

x1, x2, ..., xn The n input features

The weight/coefficient for each feature — its effect on y,

b1, b2, ..., bn
holding all other features constant

b0 The intercept — predicted y when every feature is 0

n_samples x (n_features + 1) matrix — includes a column of

X
1s for the intercept

The full weight vector [b0, b1, ..., bn], found via the Normal
beta
Equation from Section 3

4.2 Interpreting the weights

The key phrase to remember

• Each coefficient bi is the change in y for a 1-unit increase in xi, holding every other feature constant.
• This “holding constant” part is what makes multiple regression powerful — it isolates each feature's individual effect,
even when features are related to each other.
• Coefficients are only directly comparable in size if the features were scaled first (see the Feature Scaling notes) —
otherwise a feature measured in millions will naturally get a tiny coefficient.

4.3 Multicollinearity — the thing that breaks multiple regression

Multicollinearity happens when two or more input features are highly correlated with each other. It doesn't hurt
prediction accuracy much, but it makes individual coefficients unstable and hard to interpret — the model can't tell
which of the correlated features deserves the credit.

Detecting it: Variance Inflation Factor (VIF)

VIF_i = 1 / (1 - R_i^2)

where R_i^2 comes from regressing feature xi against all the other features.

Page 8
 Linear & Polynomial Regression — Revision Notes

VIF value Interpretation

1 No correlation with other features

1–5 Moderate, usually acceptable

5 – 10 High — worth investigating

Severe multicollinearity — consider removing or combining

> 10
features
● Fix by dropping one of the correlated features, combining them into a single feature, or using Ridge
Regression (Section 8), which handles multicollinearity natively.

4.4 Worked example — house prices

Predicting house price from size (sq. ft.) and number of bedrooms.

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

data = [Link]({
'size_sqft': [750, 800, 850, 1200, 1500, 1800, 2000, 2400],
'bedrooms': [1, 1, 2, 2, 3, 3, 4, 4],
'price': [150000, 160000, 175000, 220000, 260000, 300000, 340000, 400000],
})

X = data[['size_sqft', 'bedrooms']]
y = data['price']

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.25, random_state=42
)

model = LinearRegression()
[Link](X_train, y_train)

print("Intercept (b0):", model.intercept_)

print("Coefficients (b1, b2):", model.coef_)
# e.g. b1 ~= 140 -> each extra sq. ft. adds ~$140 to price, holding bedrooms constant
# e.g. b2 ~= 5000 -> each extra bedroom adds ~$5,000, holding size constant

print("Predicted price:", [Link]([[1600, 3]]))

4.5 Checking for multicollinearity in code

from [Link].outliers_influence import variance_inflation_factor

import pandas as pd

vif_data = [Link]()
vif_data['feature'] = [Link]
vif_data['VIF'] = [variance_inflation_factor([Link], i) for i in range([Link][1])]
print(vif_data)

Page 9
 Linear & Polynomial Regression — Revision Notes

5. Polynomial Regression
When the relationship between x and y is curved rather than a straight line, Polynomial Regression fits a curve by
adding powers of x as extra features.

5.1 The equation

y = b0 + b1*x + b2*x^2 + b3*x^3 + ... + bn*x^n

n here is the degree of the polynomial — it controls how curvy the fitted line is allowed to be.

The trick: it's still “linear” regression

• Polynomial Regression is NOT a different algorithm — it's Multiple Linear Regression in disguise.
• You create new columns x^2, x^3, ... from your original x, and then fit an ordinary linear regression on top of [x, x^2,
x^3, ...].
• It's called “linear” because the model is linear in the weights (b0, b1, b2…) — even though the resulting curve is not a
straight line.

5.2 Choosing the degree

Degree Effect Risk

A straight line — same as simple linear

1 Underfits if the true pattern is curved
regression

A gentle curve — usually the sweet spot Low, if chosen based on validation
2–3
for mildly non-linear data performance

A wiggly curve that can bend to pass Overfits badly — great on training data,
High (6+)
through almost every point terrible on new data
Pick the degree using validation-set performance (or cross-validation), not just how well it fits the training data — a
higher degree will always fit the training set at least as well, which is exactly why you can't trust training error alone
to choose it.

5.3 Worked example — by hand feature construction

import numpy as np
from sklearn.linear_model import LinearRegression

x = [Link]([1, 2, 3, 4, 5], dtype=float).reshape(-1, 1)

y = [Link]([2, 6, 14, 28, 45], dtype=float) # a curved relationship

# Manually build polynomial features up to degree 2

X_poly = [Link]([x, x**2]) # columns: [x, x^2]

model = LinearRegression()
[Link](X_poly, y)

print("Intercept (b0):", model.intercept_)

Page 10
 Linear & Polynomial Regression — Revision Notes

print("Coefficients (b1, b2):", model.coef_)

# fitted curve: y = b0 + b1*x + b2*x^2

5.4 Code: scikit-learn's PolynomialFeatures + Pipeline

In practice, you almost never build the powers by hand — sklearn's PolynomialFeatures does it for you, and wrapping
it in a Pipeline keeps training and prediction consistent.

import numpy as np
from [Link] import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from [Link] import Pipeline
from sklearn.model_selection import train_test_split

x = [Link]([1, 2, 3, 4, 5, 6, 7, 8]).reshape(-1, 1)
y = [Link]([2, 6, 14, 28, 45, 66, 91, 120])

X_train, X_test, y_train, y_test = train_test_split(

x, y, test_size=0.25, random_state=42
)

poly_model = Pipeline([
('poly_features', PolynomialFeatures(degree=2, include_bias=False)),
('linear_reg', LinearRegression())
])

poly_model.fit(X_train, y_train)
y_pred = poly_model.predict(X_test)

coefs = poly_model.named_steps['linear_reg'].coef_
intercept = poly_model.named_steps['linear_reg'].intercept_
print(f"y = {intercept:.2f} + {coefs[0]:.2f}*x + {coefs[1]:.2f}*x^2")

5.5 Comparing degrees visually (code)

import [Link] as plt

x_line = [Link]([Link](), [Link](), 100).reshape(-1, 1)

for degree in [1, 2, 4, 8]:

model = Pipeline([
('poly', PolynomialFeatures(degree=degree, include_bias=False)),
('lr', LinearRegression())
])
[Link](x, y)
[Link](x_line, [Link](x_line), label=f'degree={degree}')

[Link](x, y, color='black', label='data')

[Link]()
[Link]()
# Low degree -> underfits (misses the curve)
# Very high degree -> overfits (wiggles through every point)

Page 11
 Linear & Polynomial Regression — Revision Notes

6. Model Evaluation Metrics

Once weights are fitted, you need a number that tells you how good the model actually is. These are the standard
metrics for any regression model — simple, multiple, or polynomial.

6.1 Error-based metrics

Metric Formula Notes

MAE — Mean Absolute Error (1/n) SUM |yi - y_hat_i| Same units as y, robust to outliers

Punishes big errors more; this is what

MSE — Mean Squared Error (1/n) SUM (yi - y_hat_i)^2
training minimizes

Back in the same units as y — the most

RMSE — Root Mean Squared Error sqrt(MSE)
commonly reported metric

6.2 R² — how much variance is explained

R^2 = 1 - ( SUM(yi - y_hat_i)^2 / SUM(yi - y_mean)^2 )

= 1 - (SS_residual / SS_total)

R² value Meaning

1.0 The model explains 100% of the variance — a perfect fit

The model is no better than just predicting the mean of y

0.0
every time

The model is worse than predicting the mean — something is

Negative
badly wrong

Adjusted R² — the fairer version for multiple regression

Plain R² always goes up (or stays the same) every time you add a feature, even a useless random one — which makes
it a misleading way to compare models with different numbers of features. Adjusted R² fixes this by penalizing extra
features that don't actually help.

Adj_R^2 = 1 - [ (1 - R^2) * (n - 1) / (n - p - 1) ]

where n = number of samples, p = number of features. Use Adjusted R² whenever you're comparing Simple vs.
Multiple vs. Polynomial models with different numbers of terms.

6.3 Code: computing all of them

from [Link] import mean_absolute_error, mean_squared_error, r2_score

import numpy as np

mae = mean_absolute_error(y_test, y_pred)

mse = mean_squared_error(y_test, y_pred)

Page 12
 Linear & Polynomial Regression — Revision Notes

rmse = [Link](mse)
r2 = r2_score(y_test, y_pred)

n, p = X_test.shape[0], X_test.shape[1]
adj_r2 = 1 - (1 - r2) * (n - 1) / (n - p - 1)

print(f"MAE: {mae:.2f}")
print(f"MSE: {mse:.2f}")
print(f"RMSE: {rmse:.2f}")
print(f"R2: {r2:.3f}")
print(f"Adj R2: {adj_r2:.3f}")

Which metric to lead with

• Reporting to a non-technical audience → RMSE (same units as y, easy to explain) or R² (as a %).
• Data has extreme outliers you don't want to dominate the score → MAE.
• Comparing models with different numbers of features → Adjusted R².

Page 13
 Linear & Polynomial Regression — Revision Notes

7. Assumptions of Linear Regression

Linear regression's weights and p-values are only trustworthy if the data roughly satisfies five assumptions. Violating
them doesn't always break predictions, but it can badly mislead interpretation and confidence intervals.

Assumption What it means How to check If violated

The true relationship

between X and y is actually Plot residuals vs. predicted Add polynomial/interaction
1. Linearity linear (or polynomial, if values — should show no terms, or use a non-linear
you've added polynomial pattern model
terms)

Durbin-Watson test; a
Residuals (errors) are not Use time-series-aware
2. Independence concern mainly in time series
correlated with each other models (ARIMA, etc.)
data

The spread of residuals is

Plot residuals vs. predicted Log-transform the target, or
3. Homoscedasticity constant across all predicted
values use weighted least squares
values (no “funnel” shape)

Transform the target (log,

Residuals are approximately
4. Normality of residuals Q-Q plot, Shapiro-Wilk test Box-Cox); matters most for
normally distributed
small samples

Input features aren't highly VIF (see Section 4.3), Drop/combine features, or
5. No multicollinearity
correlated with each other correlation matrix use Ridge Regression

7.1 Residual plots — your main diagnostic tool

A residual is simply the leftover error: residual = y_actual - y_predicted. Plotting residuals against predicted values is
the single most useful diagnostic in regression — it can reveal non-linearity, heteroscedasticity, and outliers all in one
chart.

import [Link] as plt

residuals = y_test - y_pred

[Link](y_pred, residuals)
[Link](y=0, color='red', linestyle='--')
[Link]('Predicted values')
[Link]('Residuals')
[Link]('Residual plot')
[Link]()

# What a HEALTHY residual plot looks like: a random, flat cloud around 0
# A curve shape -> linearity assumption violated
# A funnel/cone shape -> homoscedasticity assumption violated

Page 14
 Linear & Polynomial Regression — Revision Notes

8. Bias-Variance & Regularization

8.1 Underfitting vs. overfitting

Underfitting (high bias) Good fit Overfitting (high variance)

Model too simple (e.g. Complexity matches the true Model too complex (e.g.
Cause
degree 1 on curved data) pattern degree 10 on simple data)

Training error High Low Very low

Test error High Low High

Add features, increase Regularize, reduce degree,

Fix —
polynomial degree get more data
This tradeoff — the bias-variance tradeoff — is exactly why choosing the polynomial degree (Section 5.2) matters so
much: too low underfits, too high overfits.

8.2 Regularization — penalizing overly large weights

Regularization fights overfitting by adding a penalty term to the cost function that discourages the weights from
growing too large. Large weights are usually a sign the model is straining to fit noise.

Ridge Regression (L2 penalty)

J(beta) = MSE + lambda * SUM( bj^2 )

● Shrinks all coefficients toward zero, but never exactly to zero.

● Great when you have multicollinearity — it stabilizes the coefficients.
● lambda (also written alpha in sklearn) controls the strength: 0 = plain linear regression, large = heavy
shrinkage.

Lasso Regression (L1 penalty)

J(beta) = MSE + lambda * SUM( |bj| )

● Can shrink some coefficients exactly to zero — which makes it a built-in feature selector.
● Good when you suspect only a handful of features actually matter.

Elastic Net

J(beta) = MSE + lambda1 * SUM(|bj|) + lambda2 * SUM(bj^2)

A blend of Ridge and Lasso — gets Lasso's feature-selection behavior while keeping Ridge's stability when features are
correlated.

Code

from sklearn.linear_model import Ridge, Lasso, ElasticNet

Page 15
 Linear & Polynomial Regression — Revision Notes

ridge = Ridge(alpha=1.0).fit(X_train, y_train)

lasso = Lasso(alpha=0.1).fit(X_train, y_train)
enet = ElasticNet(alpha=0.1, l1_ratio=0.5).fit(X_train, y_train)

# alpha (== lambda) is usually chosen via cross-validation:

from sklearn.linear_model import RidgeCV
ridge_cv = RidgeCV(alphas=[0.01, 0.1, 1.0, 10.0]).fit(X_train, y_train)
print("Best alpha:", ridge_cv.alpha_)

Ridge (L2) Lasso (L1) Elastic Net

Shrinks coefficients to exactly

No Yes Yes
0?

Doubles as feature selection? No Yes Yes

Many small/medium effects, Correlated features AND you

Best when Few features actually matter
correlated features want selection

Page 16
 Linear & Polynomial Regression — Revision Notes

9. One-Page Cheat Sheet

The three models

Model Equation Weights found via

b1 = cov(x,y)/var(x), b0 = y_mean -
Simple Linear y = b0 + b1*x
b1*x_mean

beta = (X^T X)^(-1) X^T y (or gradient

Multiple Linear y = b0 + b1*x1 + ... + bn*xn
descent)

Same as multiple linear, on features [x,

Polynomial y = b0 + b1*x + b2*x^2 + ... + bn*x^n
x^2, ..., x^n]

Gradient descent, in one line

wj := wj - alpha * dJ/dwj (repeat until cost stops decreasing)

Metrics, in one line each

Metric One-line meaning

MAE Average absolute error, in y's units

Average squared error (RMSE brings it back to y's units);

MSE / RMSE
punishes big misses

% of variance in y explained by the model (1 = perfect, 0 = no

R²
better than the mean)

R² penalized for extra features — use this to compare models

Adjusted R²
fairly

The 5 assumptions, in one line each

● Linearity — the relationship really is a straight line (or polynomial curve)
● Independence — residuals aren't correlated with each other
● Homoscedasticity — residual spread stays constant across predictions
● Normality — residuals are roughly bell-shaped
● No multicollinearity — features aren't highly correlated with each other

Regularization, in one line each

Ridge Lasso Elastic Net

Best of both — shrinks and can drop

Shrinks weights, keeps all features Shrinks weights, can drop features to 0
features

Page 17
 Linear & Polynomial Regression — Revision Notes

Degree / complexity check

Remember this forever

• Underfitting: training error AND test error are both high → model is too simple, add complexity.
• Overfitting: training error is low but test error is high → model is too complex, regularize or simplify.
• The right model: training and test error are both low and close to each other.

End of notes — happy revising!

Page 18