Lecture 1.5 and 1.

6
• Regression
• Linear Regression
• Intuition
• Loss Function
• Regularization
• Lasso Regularization
• Ridge Regularization

Dr. Mainak Biswas

 Regression
• Regression is a statistical method for modeling
the relationship between a dependent
variable (target) and one or more
independent variables (predictors)
• In linear regression, the relationship is
modeled as a straight line
• Intuition in regression is the idea of using a
linear approach to predict a continuous value
for a data point by generalizing over the data

Dr. Mainak Biswas

 Linear Regression
• Linear regression assumes a linear relationship
between the dependent variable 𝑦 and the
independent variable(s) 𝑥
• The equation of a simple linear regression model is
𝑦 = 𝑚𝑥 + 𝑐
𝑚 (𝑤1 ): slope of the line (weight of the predictor)
𝑐 (𝑜𝑟 𝑤0 ): y-intercept
𝑥: input feature(s)
𝑦: predicted output (target)
• For multiple predictors (features), the equation
becomes
𝑦 = 𝑤0 + 𝑤1 𝑥1 + 𝑤2 𝑥2 + ⋯ + 𝑤𝑛 𝑥𝑛

Dr. Mainak Biswas

 Loss Function
• The performance of the linear regression model
is evaluated using the cost function, commonly
the Mean Squared Error (MSE)
𝑁1 2
𝐸 𝑤0 , 𝑤1 , … , 𝑤𝑛 = 𝑖=1 𝑦𝑖 − 𝑦𝑖
2𝑁
where,
𝐸: Loss function
𝑁: Number of data points
𝑦𝑖 :Predicted value
𝑦𝑖 : Actual value

Dr. Mainak Biswas

 Example 1
SL No. 𝑥 (data) 𝑦𝑎𝑐𝑡𝑢𝑎𝑙
1 1 2
2 2 3

𝑦 = 𝑤1 𝑥 + 𝑤0
Initial Setup
𝑤0 = 0
𝑤1 = 0
𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝑟𝑎𝑡𝑒 (𝜂) = 0.1
1 𝑁 2
𝐿𝑜𝑠𝑠 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛: 𝐸 𝑤0 , 𝑤1 , … , 𝑤𝑛 = 𝑖=1 𝑦𝑖 − 𝑦𝑖
2𝑁

Dr. Mainak Biswas

 Iteration 1
1 𝑁 2
𝑤0 = 0, 𝑤1 = 0, 𝐸 𝑤0 , 𝑤1 , … , 𝑤𝑛 = 𝑖=1 𝑤1 𝑥𝑖 + 𝑤0 − 𝑦𝑖
2𝑁
𝑁
𝛿𝐸 1
= (𝑦𝑖 − 𝑦𝑖 )
𝛿𝑤0 𝑁
𝑖=1
𝑁
𝛿𝐸 1
= (𝑦𝑖 − 𝑦𝑖 )𝑥𝑖
𝛿𝑤1 𝑁
𝑖=1
𝑁
𝛿𝐸 1 1
= (𝑦𝑖 − 𝑦𝑖 ) = −2 + −3 = −2.5
𝛿𝑤0 𝑁 2
𝑖=1
𝑁
𝛿𝐸 1 1
= 𝑦𝑖 − 𝑦𝑖 𝑥𝑖 = −2 × 1 + −3 × 2 = −4
𝛿𝑤1 𝑁 2
𝑖=1

𝑥 (data) 𝑦𝑎𝑐𝑡𝑢𝑎𝑙 𝑦 = 𝑤1 𝑥 + 𝑤0 Residual Error 𝒘𝟏 = 𝒘𝟏 − 𝜼𝜵𝑬 𝒘𝟏 𝒘𝟎 = 𝒘𝟎 − 𝜼𝜵𝑬 𝒘𝟎

1 2 0x1+0=0 0 – 2 = -2
0 – 0.1x-4= 0 – 0.1x-2.5=
2 3 0x2+0=0 0 – 3 = -3 0.25
0.4

Dr. Mainak Biswas

 Iteration 2
𝑁
𝛿𝐸 1 1
= (𝑦𝑖 − 𝑦𝑖 ) = −1.35 − 1.95 = −1.65
𝛿𝑤0 𝑁 2
𝑖=1

𝑁
𝛿𝐸 1 1
= 𝑦𝑖 − 𝑦𝑖 𝑥𝑖 = −1.35 × 1 − 1.95 × 2 = −2.625
𝛿𝑤1 𝑁 2
𝑖=1

𝑥 (data) 𝑦𝑎𝑐𝑡𝑢𝑎𝑙 𝑦 = 𝑤1 𝑥 + 𝑤0 Residual Error 𝒘𝟏 = 𝒘𝟏 − 𝜼𝜵𝑬 𝒘𝟏 𝒘𝟎 = 𝒘𝟎 − 𝜼𝜵𝑬 𝒘𝟎

1 2 0.4 x 1 + 0.25 = 0.65 – 2 = -1.35

0.65 0.4 – 0.1x-2.625= 0 .25– 0.1x-1.65=
0.6625 0.415
2 3 0.4 x 2 + 0.25 = 1.05 – 3 = -1.95
1.05

Dr. Mainak Biswas

 Iteration 10
• 𝑤0 = 0.7322, 𝑤1 = 1.1411,

𝑦 = 1.1411𝑥 + 0.7322

Dr. Mainak Biswas

 Regularization
• Regularization is a technique used in machine learning
and statistics to prevent overfitting of models by
adding a penalty term to the model's loss function
• Regularization provides this increased generalizability
at the sake of increased training error
• Regularization is a technique used to reduce errors by
fitting the function appropriately on the given training
set and avoiding overfitting. The commonly used
regularization techniques are :
– Lasso Regularization – L1 Regularization
– Ridge Regularization – L2 Regularization

Dr. Mainak Biswas

Overfitting

Dr. Mainak Biswas

 Role of Regularization
• Complexity Control: Regularization helps control model complexity by preventing
overfitting to training data, resulting in better generalization to new data.
• Preventing Overfitting: One way to prevent overfitting is to use regularization,
which penalizes large coefficients and constrains their magnitudes, thereby
preventing a model from becoming overly complex and memorizing the training
data instead of learning its underlying patterns.
• Balancing Bias and Variance: Regularization can help balance the trade-off
between model bias (underfitting) and model variance (overfitting) in machine
learning, which leads to improved performance.
• Feature Selection: Some regularization methods, such as L1 regularization (Lasso),
promote sparse solutions that drive some feature coefficients to zero. This
automatically selects important features while excluding less important ones.
• Handling Multicollinearity: When features are highly correlated (multicollinearity),
regularization can stabilize the model by reducing coefficient sensitivity to small
data changes.
• Generalization: Regularized models learn underlying patterns of data for better
generalization to new data, instead of memorizing specific examples.

Dr. Mainak Biswas

 L1 (LASSO) Regularization
• A regression model which uses the L1 Regularization
technique is called LASSO(Least Absolute Shrinkage and
Selection Operator) regression
• Lasso Regression adds the “absolute value of magnitude”
of the coefficient as a penalty term to the loss function(L)
𝑁 𝑚
1 2
𝐸 𝑤0 , 𝑤1 , … , 𝑤𝑛 = 𝑦𝑖 − 𝑦𝑖 +λ 𝑤𝑖
2𝑁
𝑖=1 𝑖=1
λ: Regularization parameter
∣ 𝑤𝑖 ∣: Absolute value of the i-th weight (coefficient) of the
model
m: Number of features

Dr. Mainak Biswas

 How it affects the gradient?
• In Lasso regularization, the gradient of the cost function
with respect to a parameter 𝑤 includes the term 𝜆⋅sign(𝑤)
• When computing the gradient of the cost function with
respect to 𝑤w, the derivative of the absolute value is not
defined at 𝑤=0, but it is approximated using the sign
function:
𝛿𝑤
= 𝑠𝑖𝑔𝑛(𝑤)
𝛿𝑤
When 𝑤>0, 𝜆 pushes 𝑤 to decrease
When 𝑤<0, 𝜆 pushes 𝑤 to increase
When 𝑤=0, the penalty term does not contribute to the
gradient

Dr. Mainak Biswas

 Example 2
SL No. 𝑥 (data) 𝑦𝑎𝑐𝑡𝑢𝑎𝑙
1 1 2
2 2 3

𝑦 = 𝑤1 𝑥 + 𝑤0
Initial Setup
𝑤0 = 0
𝑤1 = 0
𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝑟𝑎𝑡𝑒 (𝜂) = 0.1
𝑅𝑒𝑔𝑢𝑙𝑎𝑟𝑖𝑧𝑎𝑡𝑖𝑜𝑛 𝑃𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟 𝜆 = 0.1
𝐿𝑜𝑠𝑠 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛:
𝑁 𝑚
1 2
𝐸 𝑤0 , 𝑤1 = 𝑦𝑖 − 𝑦𝑖 +λ 𝑤𝑖
2𝑁
𝑖=1 𝑖=1

Dr. Mainak Biswas

 Iteration 1
𝑤0 = 0, 𝑤1 = 0,
1 𝑁
𝐸 𝑤0 , 𝑤1 =
2𝑁 𝑖=1 𝑤1 𝑥𝑖 + 𝑤0 − 𝑦𝑖 2 +𝜆( 𝑤0 + 𝑤1 )

𝑁
𝛿𝐸 1
= (𝑦𝑖 − 𝑦𝑖 ) + 𝜆 𝑠𝑖𝑔𝑛 (𝑤0 )
𝛿𝑤0 𝑁
𝑖=1
𝑁
𝛿𝐸 1
= (𝑦𝑖 − 𝑦𝑖 )𝑥𝑖 + 𝜆 𝑠𝑖𝑔𝑛 (𝑤1 )
𝛿𝑤1 𝑁
𝑖=1
𝑁
𝛿𝐸 1 1
= (𝑦𝑖 − 𝑦𝑖 ) = −2 + −3 + 0.1 × 0 = −2.5
𝛿𝑤0 𝑁 2
𝑖=1
𝑁
𝛿𝐸 1 1
= 𝑦𝑖 − 𝑦𝑖 𝑥𝑖 = −2 × 1 + −3 × 2 + 0.1 × 0 = −4
𝛿𝑤1 𝑁 2
𝑖=1

𝑥 (data) 𝑦𝑎𝑐𝑡𝑢𝑎𝑙 𝑦 = 𝑤1 𝑥 + 𝑤0 Residual Error 𝒘𝟏 = 𝒘𝟏 − 𝜼𝜵𝑬 𝒘𝟏 𝒘𝟎 = 𝒘𝟎 − 𝜼𝜵𝑬 𝒘𝟎

1 2 0x1+0=0 0 – 2 = -2
0 – 0.1x-4= 0 – 0.1x-2.5=
2 3 0x2+0=0 0 – 3 = -3 0.25
0.4
Dr. Mainak Biswas
 Iteration 2
𝑁
𝛿𝐸 1 1
= (𝑦𝑖 − 𝑦𝑖 ) = −1.35 − 1.95 + 0.1 × 1 = −1.55
𝛿𝑤0 𝑁 2
𝑖=1
𝑁
𝛿𝐸 1 1
= 𝑦𝑖 − 𝑦𝑖 𝑥𝑖 = −1.35 × 1 − 1.95 × 2 + 0.1 × 1 = −2.525
𝛿𝑤1 𝑁 2
𝑖=1

𝑥 (data) 𝑦𝑎𝑐𝑡𝑢𝑎𝑙 𝑦 = 𝑤1 𝑥 + 𝑤0 Residual Error 𝒘𝟏 = 𝒘𝟏 − 𝜼𝜵𝑬 𝒘𝟏 𝒘𝟎 = 𝒘𝟎 − 𝜼𝜵𝑬 𝒘𝟎

1 2 0.4 x 1 + 0.25 = 0.65 – 2 = -1.35

0.65 0.4 – 0.1x-2.525= 0 .25– 0.1x-1.55=
0.6525 0.405
2 3 0.4 x 2 + 0.25 = 1.05 – 3 = -1.95
1.05

Dr. Mainak Biswas

 Iteration 10
• 𝑤0 = 0.6874, 𝑤1 = 1.1227

𝑦 = 1.1227𝑥 + 0.6874

Dr. Mainak Biswas

 L2 (Ridge) Regularization
• Ridge regression, also known as L2 regularization, is a
technique used to improve the performance and
generalization of linear regression models
• It penalizes large coefficients to reduce overfitting while
ensuring that all features contribute to the prediction
𝑁 𝑚
1
𝐸 𝑤0 , 𝑤1 , … , 𝑤𝑛 = 𝑦𝑖 − 𝑦𝑖 2 +λ 𝑤𝑖2
2𝑁
𝑖=1 𝑖=1
λ: Regularization parameter
𝑤𝑖2 : square of the magnitude of coefficients
m: Number of features

Dr. Mainak Biswas

 How it affects the gradient?
• If 𝜆=0: Ridge regression reduces to ordinary
least squares (OLS) regression with no
regularization
• If 𝜆 is very large: Coefficients shrink
significantly, which may lead to Underfitting
(model is too simple)
• Optimal 𝜆: A balance between overfitting and
Underfitting is achieved through cross-
validation to determine the best 𝜆

Dr. Mainak Biswas

 Example 3
SL No. 𝑥 (data) 𝑦𝑎𝑐𝑡𝑢𝑎𝑙
1 1 2
2 2 3

𝑦 = 𝑤1 𝑥 + 𝑤0
Initial Setup
𝑤0 = 0
𝑤1 = 0
𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝑟𝑎𝑡𝑒 (𝜂) = 0.1
𝑅𝑒𝑔𝑢𝑙𝑎𝑟𝑖𝑧𝑎𝑡𝑖𝑜𝑛 𝑃𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟 𝜆 = 0.1
𝐿𝑜𝑠𝑠 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛:
𝑁 𝑚
1
𝐸 𝑤0 , 𝑤1 = 𝑦𝑖 − 𝑦𝑖 2
+λ 𝑤𝑖2
2𝑁
𝑖=1 𝑖=1

Dr. Mainak Biswas

 Iteration 1
𝑤0 = 0, 𝑤1 = 0,
1
𝐸 𝑤0 , 𝑤1 = 𝑁
𝑖=1 𝑤1 𝑥𝑖 + 𝑤0 − 𝑦𝑖 2 +𝜆(𝑤02 + 𝑤12 )
2𝑁

𝑁
𝛿𝐸 1
= (𝑦𝑖 − 𝑦𝑖 ) + 2𝜆𝑤0
𝛿𝑤0 𝑁
𝑖=1
𝑁
𝛿𝐸 1
= (𝑦𝑖 − 𝑦𝑖 )𝑥𝑖 + 2𝜆𝑤1
𝛿𝑤1 𝑁
𝑖=1
𝑁
𝛿𝐸 1 1
= (𝑦𝑖 − 𝑦𝑖 ) = −2 + −3 + 2 × 0.1 × 0 = −2.5
𝛿𝑤0 𝑁 2
𝑖=1
𝑁
𝛿𝐸 1 1
= 𝑦𝑖 − 𝑦𝑖 𝑥𝑖 = −2 × 1 + −3 × 2 + 2 × 0.1 × 0 = −4
𝛿𝑤1 𝑁 2
𝑖=1

𝑥 (data) 𝑦𝑎𝑐𝑡𝑢𝑎𝑙 𝑦 = 𝑤1 𝑥 + 𝑤0 Residual Error 𝒘𝟏 = 𝒘𝟏 − 𝜼𝜵𝑬 𝒘𝟏 𝒘𝟎 = 𝒘𝟎 − 𝜼𝜵𝑬 𝒘𝟎

1 2 0x1+0=0 0 – 2 = -2
0 – 0.1x-4= 0 – 0.1x-2.5=
2 3 0x2+0=0 0 – 3 = -3 0.25
0.4
Dr. Mainak Biswas
 Iteration 2
𝑁
𝛿𝐸 1 1
= (𝑦𝑖 − 𝑦𝑖 ) = −1.35 − 1.95 + 2 × 0.1 × 0.25 = −1.60
𝛿𝑤0 𝑁 2
𝑖=1
𝑁
𝛿𝐸 1 1
= 𝑦𝑖 − 𝑦𝑖 𝑥𝑖 = −1.35 × 1 − 1.95 × 2 + 2 × 0.1 × 0.4 = −2.545
𝛿𝑤1 𝑁 2
𝑖=1

𝑥 (data) 𝑦𝑎𝑐𝑡𝑢𝑎𝑙 𝑦 = 𝑤1 𝑥 + 𝑤0 Residual Error 𝒘𝟏 = 𝒘𝟏 − 𝜼𝜵𝑬 𝒘𝟏 𝒘𝟎 = 𝒘𝟎 − 𝜼𝜵𝑬 𝒘𝟎

1 2 0.4 x 1 + 0.25 = 0.65 – 2 = -1.35

0.65 0.4 – 0.1x-2.545= 0 .25– 0.1x-1.60=
0.6545 0.41
2 3 0.4 x 2 + 0.25 = 1.05 – 3 = -1.95
1.05

Dr. Mainak Biswas

 Iteration 10
w0​=3.5089
w1=0.8912

Dr. Mainak Biswas