Scribd
Upload
4 views3 pages

Regularization

Regularization is a technique in machine learning that adds a penalty to the loss function to prevent overfitting and improve model performance on unseen data. It includes methods like LASSO (L1) and Ridge (L2) regression, which control model complexity and handle irrelevant features differently. The regularization parameter (λ) determines the strength of regularization, influencing the balance between model simplicity and complexity.

Uploaded by

avatarpj2
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
4 views3 pages

Regularization

Regularization is a technique in machine learning that adds a penalty to the loss function to prevent overfitting and improve model performance on unseen data. It includes methods like LASSO (L1) and Ridge (L2) regression, which control model complexity and handle irrelevant features differently. The regularization parameter (λ) determines the strength of regularization, influencing the balance between model simplicity and complexity.

Uploaded by

avatarpj2
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Regularization in Machine Learning

Definition
Regularization is a technique used to prevent overfitting by adding a penalty term to the loss
function. It discourages overly complex models and improves performance on unseen data.

Importance of Regularization
• Prevents overfitting by reducing model complexity
• Improves generalization on new data
• Controls large weight values
• Makes the model more stable and reliable

Types of Regularization
1. LASSO Regression (L1 Regularization)

• Adds the absolute value of coefficients as a penalty


• Can shrink some coefficients exactly to zero
• Performs feature selection
• Useful when many irrelevant features are present

2. Ridge Regression (L2 Regularization)

• Adds the square of coefficients as a penalty


• Shrinks coefficients but does not make them zero
• Handles multicollinearity effectively
• Keeps all features in the model

Regularization Parameter (λ)


• Controls the strength of regularization
• Large λ → simpler model
• Small λ → complex model
Regularization – Line Diagram
Loss
↑
| L1 (LASSO)
| /\
| / \
| / \
| / \
| / \
|----●----------●----→ Weight (w)
| L2 (Ridge)
| \____/
|

• X-axis → Weight / Coefficient (w)

• Y-axis → Loss / Cost

• Sharp V shape → L1 Regularization (can make weights exactly zero)

• Smooth U shape → L2 Regularization (shrinks weights, not zero)

Regularization – Line Diagram

Error / Loss
↑
|
| Without Regularization
| _________
| /
| /
| /
| /
| /
|_____/__________________→ Model Complexity
\
\
\_____ With Regularization

Without regularization, as model complexity increases, training error decreases but the model
overfits.
With regularization, a penalty is added which controls model complexity and prevents
overfitting.
Thus, regularization improves generalization on unseen data.

You might also like