Ridge Regression
Problem with Linear Regression
• When predictors (independent variables) are highly correlated
(multicollinearity), the least squares estimates become unstable.
• Overfitting occurs when the model fits noise in the training data.
• Solution: Ridge regression adds a penalty term to the loss function to
shrink the coefficients, improving stability and generalization.
Multicollinearity
• Multicollinearity denotes when two or
more predictors have a near-linear
relationship.
What is Multicollinearity?
• Situation where predictor variables (independent variables) are highly correlated with
each other.
• Predictors carry overlapping information about the response variable.
• Example: Height (cm) and Height (inches) in the same model.
Example
𝐻𝑒𝑖𝑔ℎ𝑡𝑐𝑚 = 2.54 X 𝐻𝑒𝑖𝑔ℎ𝑡𝑖𝑛𝑐ℎ
• Suppose we build a model:
𝑊𝑒𝑖𝑔ℎ𝑡 = θ + 𝜃1 ⋅ 𝐻𝑒𝑖𝑔ℎ𝑡𝑐𝑚 + 𝜃2 ⋅ 𝐻𝑒𝑖𝑔ℎ𝑡𝑖𝑛𝑐ℎ
• Since 𝐻𝑒𝑖𝑔ℎ𝑡𝑐𝑚 and 𝐻𝑒𝑖𝑔ℎ𝑡𝑖𝑛𝑐ℎ contain the same information, the regression can’t
uniquely assign weights.
• Mathematically, 𝑋 𝑇 𝑋 becomes singular (non-invertible).
• The model will try to “balance” the coefficients, often giving:
• Huge positive value to one feature
• Huge negative value to the other
• Net effect → predictions still work, but coefficients are unstable and meaningless.
Other Examples of Multicollinearity
House Price Prediction
• Size of the house in square feet
• Number of rooms
• These two variables are not identical, but they are strongly correlated—larger houses
typically have more rooms. This can introduce collinearity in regression.
Employee Performance Prediction
• Years of experience
• Age of employee
• These aren’t the same but tend to move together, creating multicollinearity.
Strategies to Address Multicollinearity
• Collect more diversified data.
• If you’re modeling delivery times, don’t just collect data for short-
distance deliveries with large inventories;
• also include long-distance deliveries, small inventories, etc.
• Collecting more data is not always be a viable fix, however, such as when
multicollinearity is intrinsic to the data studied.
Strategies to Address Multicollinearity
• Other options for fixing multicollinearity include
➢increasing sample size,
➢ reducing the number of independent variables,
➢or simply deploying a different model.
• Such fixes do not always succeed in eliminating multicollinearity, however,
and ridge regression serves as another method for regularizing a model to
address multicollinearity.
What is Ridge regression
• Ridge regression is a statistical regularization technique.
• It corrects for overfitting on training data in machine learning models.
• Ridge regression—also known as L2 regularization—is one of several
types of regularization for linear regression models.
What is Ridge regression
• Ridge regression specifically corrects for multicollinearity in regression
analysis.
• This is useful when developing machine learning models that have a
large number of parameters, particularly if those parameters also have
high weights.
The problem: multicollinearity
• A standard, multiple-variable linear regression equation is:
Ordinary Least
Squares (OLS)
Linear
Regression
Form
Matrix Form
(compact OLS
representation)
Cost Function
• 𝐽 𝜃 is the cost function (or objective function) used in regression.
• It measures how well the model with parameters 𝜃 fits the training data.
In OLS (Linear Regression):
where:
• 𝑌𝑖 =actual value (target)
• 𝑌𝑖 = 𝜃0 + 𝜃1 𝑋𝑖1 + ⋯ + 𝜃𝑛 𝑋𝑖𝑛 =predicted value
• 𝑚= number of training examples
Residual Sum of Squares (RSS)
• RSS as a measure of fit
• The Residual Sum of Squares (RSS) tells us how far the predicted
values 𝑌𝑖 are from the actual values 𝑌𝑖 .
• Residual = 𝑌𝑖 − 𝑌𝑖 .
OLS principle
• OLS is the method used in linear regression to find the best-fit line.
• It does this by minimizing RSS over all possible values of coefficients (𝜃).
• That is:
𝑚
min 𝑅𝑆𝑆 = min ( 𝑌𝑖 − 𝜃0 + 𝜃1 𝑋𝑖1 + 𝜃2 𝑋𝑖2 + ⋯ + 𝜃𝑛 𝑋𝑖𝑛 )2
𝜃 𝜃
𝑖=1
Connection with MSE:
• Mean Squared Error (MSE) cost function.
• Whether you minimize RSS or MSE, the optimal 𝜃 is the same.
• MSE is RSS normalized by number of samples, giving an average error per
observation.
How ridge regression works: the regularization algorithm
• When initially developing predictive models, we often need to compute
coefficients, as coefficients are not explicitly stated in the training data.
• To estimate coefficients, we can use a standard ordinary least squares
(OLS) matrix coefficient estimator.
θ = ൫𝑋 𝑇 𝑋)−1 𝑋 𝑇 𝑌
Best-Fitting Line
• This formula aims to find the best-fitting line for a given dataset by
calculating coefficients for each independent variable that collectively
result in the smallest residual sum of squares (also called the sum of
squared errors).
Problems with OLS
• High coefficient estimates can often be symptomatic of overfitting.
• If two or more variables share a high, linear correlation, OLS may return
erroneously high-value coefficients.
• When one or more coefficients are too high, the model’s output becomes
sensitive to minor alterations in the input data.
• In other words, the model has overfitted on a specific training set and fails
to accurately generalize on new test sets. Such a model is considered
unstable.
Solution - Ridge regression
• Ridge regression modifies OLS by calculating coefficients that account for
potentially correlated predictors.
• Specifically, ridge regression corrects for high-value coefficients by
introducing a regularization term (often called the penalty term) into the
RSS function.
• This penalty term is the sum of the squares of the model’s coefficients.
Ridge regression
• The L2 penalty term is inserted as the end of the RSS function, resulting in
a new formulation, the ridge regression estimator.
• Therein, its effect on the model is controlled by the hyperparameter
lambda (λ):
Ridge Regression Objective (RSS + L2 penalty):
Ridge Regression Objective (RSS + L2 penalty):
L2 Penalty
• It is represented in the formulation:
• The L2 penalty term is inserted as the end of the RSS function,
resulting in a new formulation, the ridge regression estimator.
How Ridge Regression Shrinks Coefficients
• Note that ridge regression does not shrink every coefficient by the same
value.
• Rather, coefficients are shrunk in proportion to their initial size.
• As λ increases, high-value coefficients shrink at a greater rate than low-
value coefficients.
• High-value coefficients are thus penalized greater than low-value
coefficients.