Standardization

 Standardization transforms the data to have a mean of 0 and a standard deviation of 1,

making it follow a standard normal distribution.
 Formula for Standardization:
𝒙 𝝁
𝒙𝒔𝒕𝒅 = 𝝈
where 𝒙 is the original value
𝝁 is the mean of the data and 𝝈 is the standard deviation of the data
 Standardization is useful when the data follows a Gaussian (normal) distribution.
 It maintains the distribution shape and is robust to outliers compared to normalization.
 Used in algorithms like Logistic Regression, Linear Regression, Neural Networks, and
Principal Component Analysis (PCA), where assumptions about data distribution are
important.
 Example:
Original data: [10, 20, 30, 40, 50]
Mean, 𝝁 = 30, Standard deviation, 𝜎 = 15
Standardized data: [−1.33,−0.67,0,0.67,1.33]
ML / Module-I / Jasaswi Prasad Mohanty 141
 Normalization vs Standardization

ML / Module-I / Jasaswi Prasad Mohanty 142

 Distribution of Data
 A statistical distribution, or probability
distribution, describes how values are distributed for a
field. In other words, the statistical distribution shows
which values are common and uncommon.
 In data science, a uniform distribution is a probability
distribution where all possible outcomes are equally
likely to occur.
 Uniform distribution is a probability distribution that
asserts that the outcome possibilities for a discrete set
of data are the same for each value.
 Normal distribution (Gaussian Distribution) is a
probability distribution that is symmetric about the mean
showing that data near the mean are more frequent in
occurrence than data far from the mean.

ML / Module-I / Jasaswi Prasad Mohanty 143

 Overview of Bias Variance
 In supervised machine learning, an algorithm learns a model from training data.
 The goal of any supervised machine learning algorithm is to best estimate the mapping function
(f) for the output variable (Y) given the input data (X).
 The mapping function is often called the target function because it is the function that a given
supervised machine learning algorithm aims to approximate.
 The prediction error for any machine learning algorithm can be broken down into three parts:
• Bias Error
• Variance Error
• Irreducible Error
 The irreducible error cannot be reduced regardless of what algorithm is used. It is the error
introduced from the chosen framing of the problem and may be caused by factors like unknown
variables that influence the mapping of the input variables to the output variable.

ML / Module-I / Jasaswi Prasad Mohanty 144

 Bias
 Bias are the simplifying assumptions made by a model to make the target function easier to
learn.
 Bias is the difference between the average prediction of our model and the correct value which
we are trying to predict. Model with high bias pays very little attention to the training data and
oversimplifies the model. It always leads to high error on training and test data.
 High bias (overly simple) can cause an algorithm to miss the relevant relations between
features and target outputs (underfitting).
 Generally, linear algorithms have a high bias making them fast to learn and easier to understand
but generally less flexible. In turn, they have lower predictive performance on complex problems
that fail to meet the simplifying assumptions of the algorithms’ bias.
 Low Bias: Suggests less assumptions about the form of the target function.
 High-Bias: Suggests more assumptions about the form of the target function.
 Examples of low-bias machine learning algorithms : Decision Trees, k-Nearest Neighbors
and SVM.
 Examples of high-bias machine learning algorithms : Linear Regression, Linear Discriminant
Analysis and Logistic Regression.
ML / Module-I / Jasaswi Prasad Mohanty 145
 Variance
 Variance is the amount that the estimate of the target function will change if different training data
was used.
 The variance is an error from sensitivity to small fluctuations in the training set.
 Variance is the variability of model prediction for a given data point or a value which tells us
spread of our data.
 Model with high variance pays a lot of attention to training data and does not generalize on the
data which it hasn’t seen before. As a result, such models perform very well on training data but
has high error rates on test data.
 We should expect the machine learning algorithm, to have some variance. Ideally, it should not
change too much from one training dataset to the next, meaning that the algorithm is good at
picking out the hidden underlying mapping between the inputs and the output variables.
 Low Variance: Suggests small changes to the estimate of the target function with changes to the
training dataset.
 High Variance: Suggests large changes to the estimate of the target function with changes to
the training dataset.

ML / Module-I / Jasaswi Prasad Mohanty 146

 Variance
 Generally, nonlinear machine learning algorithms that have a lot of flexibility have a high
variance. For example, decision trees have a high variance, that is even higher if the trees are
not pruned before use.
 Examples of low-variance machine learning algorithms : Linear Regression, Linear Discriminant
Analysis and Logistic Regression.
 Examples of high-variance machine learning algorithms include: Decision Trees, k-Nearest
Neighbors and Support Vector Machines.

ML / Module-I / Jasaswi Prasad Mohanty 147

 Bias Vs Variance
 Bias
• Difference between predicted value of the
model and actual (target) value of the data.
 Variance
• How much scatter the predicted values are
between each other.

ML / Module-I / Jasaswi Prasad Mohanty 148

 Underfitting vs Overfitting
 In supervised learning, underfitting happens when a model unable to capture the underlying
pattern of the data.
 These models usually have high bias and low variance.
 It happens when we have very less amount of data to build an accurate model or when we try to
build a linear model with a nonlinear data.
 These kind of models are very simple to capture the complex patterns in data like Linear and
logistic regression.
 Overfitting means that error on the training data is very low, but error on new instances is high.
 In supervised learning, overfitting happens when our model captures the noise along with the
underlying pattern in data.
 It happens when we train our model a lot over noisy dataset.
 These models have low bias and high variance.
 These models are very complex like Decision trees which are prone to overfitting.

ML / Module-I / Jasaswi Prasad Mohanty 149

 Regression Model

• Training error is high • Training error is low • Training error is lowest

• Test error is high • Test error is moderate • Test error is high
• Under fitting condition • Just right condition • Over fitting condition
• High bias • Low bias • High variance
• Low variance

ML / Module-I / Jasaswi Prasad Mohanty 150

 Classification Model

Model 1 Model 2 Model 3

• Training error: 1% • Training error: 25% • Training error: < 10%
• Test error: 20% • Test error: 26% • Test error: 10%
• Low bias • High bias • Low bias
• High variance • High variance • Low variance

ML / Module-I / Jasaswi Prasad Mohanty 151

 Bias vs Variance

ML / Module-I / Jasaswi Prasad Mohanty 152

 Error vs Model Complexity

High Bias Low Bias

Low Variance High Variance

ML / Module-I / Jasaswi Prasad Mohanty 153

 Bias variance trade off
 If our model is too simple and has very few parameters, then it may have high bias and low
variance.
 If our model has large number of parameters, then it’s going to have high variance and low bias.
 We need to find the right/good balance without overfitting and underfitting the data.
 This tradeoff in complexity is why there is a tradeoff between bias and variance. An algorithm
can’t be more complex and less complex at the same time.
 To build a good model, we need to find a good balance between bias and variance such that it
minimizes the total error.
 An optimal balance of bias and variance would never overfit or underfit the model.

ML / Module-I / Jasaswi Prasad Mohanty 154

 Bias variance trade off
 The goal of any supervised machine learning algorithm is to achieve low bias and low variance.
In turn the algorithm should achieve good prediction performance.
 General trend in the examples:
• Linear machine learning algorithms often have a high bias but a low variance.
• Nonlinear machine learning algorithms often have a low bias but a high variance.
 The parameterization of machine learning algorithms is often a battle to balance out bias and
variance.
 There is no escaping the relationship between bias and variance in machine learning.
• Increasing the bias will decrease the variance.
• Increasing the variance will decrease the bias.
 There is always a trade-off between
• complex hypotheses that fit the training data well
• Simpler hypotheses that may generalize better.
 As the amount of training data increases, the generalization error decrease.
ML / Module-I / Jasaswi Prasad Mohanty 155
 Addressing Underfitting
 Increase Model Complexity
• Replace a simple model (e.g., linear regression) with a more sophisticated one (e.g.,
decision trees, neural networks).
• Use advanced algorithms like ensemble learning (e.g., Random Forests, Gradient
Boosting).
 Add Features
• Feature engineering: Create new, meaningful features that better represent the
problem.
• Polynomial features: Introduce interaction terms or higher-degree polynomial terms
for better fit.
 Reduce Regularization strength (e.g., lower the alpha or lambda parameter).
 Tune Hyperparameters
• Adjust parameters such as learning rate, number of epochs, tree depth, or kernel
parameters (for SVMs).
ML / Module-I / Jasaswi Prasad Mohanty 156
 Addressing Overfitting
 Reduce number of features.
• Manually select which feature to keep.
• Model selection algorithm
 Regularization
• Keep all features, but reduce magnitude / values of parameters 𝜃 , 𝜃 , . . . ,𝜃
• Works well when we have a lot of features, each of which contributes a bit to
predict 𝑦.
• Small values for parameters 𝜃 , 𝜃 , . . . ,𝜃
“Simpler” hypothesis
Less prone to overfitting

ML / Module-I / Jasaswi Prasad Mohanty 157

 Regularization
 Complex models with large number of parameters can lead to large variance, i.e. over-
fitting.
 Overfitting occurs when a model performs well on training data but poorly on unseen
test data because it has learned the noise and specific details in the training data rather
than the general underlying patterns.
 Regularization in machine learning is a technique used to prevent overfitting by adding
a penalty to the model's loss function.
 Regularization can be used to address over-fitting, however degree of regularization
can once again lead to high variance or bias.
 Goal of Regularization:
• Improve the model's ability to generalize to unseen data.
• Avoid overly complex models that fit the training data too closely.

ML / Module-I / Jasaswi Prasad Mohanty 158

 Regularizaion
 It is a technique to prevent overfitting in ML models, especially in regression
models like MLR.
 Overfitting happens when a model is too complex and captures not just the
underlying pattern but also the noise in the data, leading to poor performance on
unseen data.
 Regularization adds a penalty to the loss function which helps in reducing the
complexity of the model by shrinking the magnitude of the coefficients. This
results in a simpler, generalization model.
 𝐶𝑜𝑠𝑡 = 𝑆𝑆𝐸 + 𝜆 ∑ |𝛽 | [First term is sum of squared error and the second term is the
regularizing term.]
• 𝜆 is the tuning parameter which controls the amount of shrinkage.

ML / Module-I / Jasaswi Prasad Mohanty 159

 Common Types of Regularization
1. L1 Regularization (Lasso):
 Adds the absolute value of coefficients as penalty: 𝜆 ∑ 𝑤
 Encourages sparsity, making some weights exactly zero, effectively performing
feature selection.
2. L2 Regularization (Ridge):
 Adds the squared value of coefficients as penalty: 𝜆 ∑ 𝑤
 Discourages large weights and leads to smoother models.
3. Elastic Net:
 Combines L1 and L2 penalties: 𝜆 ∑ 𝑤 + 𝜆 ∑ 𝑤
 Balances sparsity and smoothness.

ML / Module-I / Jasaswi Prasad Mohanty 160

 Ridge Regression
 Consider a dataset consisting of Weight and Size measurement from a bunch of mice.
 Since these data looks linear we can use Linear Regression to model the relationship between
Weight and Size.
 We will fit a line to the data using Least Squares.

Size
Size

Weight
Weight
ML / Module-I / Jasaswi Prasad Mohanty 161
 Least Square Line
 We will end up with the equation for the line
𝑆𝑖𝑧𝑒 = 0.9 + 0.75 𝑊𝑒𝑖𝑔ℎ𝑡
 To find the size of a mouse of weight 2.5, we have to put the Weight 2.5 in the above equation
and find its size.
 When we have a lot of data points we are confident that Least Squares Line accurately reflects
the relationship between Weight and Size.

Size
Size

Weight Weight
ML / Module-I / Jasaswi Prasad Mohanty 162
 Finding a new Line
 Since the new line overlaps the two points the minimum sum of
square residuals = 0.
 In this case

Size
Size = 0.4 + 1.3 * Weight
 Now we got the testing data shown in green dots.
 The sum of the residuals for the green points is large. This means the
red line has high variance. The red line is Over Fit (for training data
the model gives good result, for test data the error is more) to the
Training Data.
 Any model should be generalized model (having low variance and Weight
also should have low bias).
 The main idea behind Ridge Regression is to find a line that does not
fit the training data as well.

Size
 We introduce a small amount of Bias (for training data the error is
more) into how the line is fit to the data.
 In return for that small amount of Bias we get significant amount drop
in variance. Weight
ML / Module-I / Jasaswi Prasad Mohanty 163
 How Ridge Regression Works
 Least Square minimizes the sum of the squared residuals.
 Ridge Regression minimizes the sum of the squared residuals
plus   (slope)2
 This adds a penalty to the least square method.  determines
how severe the penalty is.

Size
 Assuming  = 1:
• For the Least Square Line the sum of squared residuals
plus the ridge regression penalty = 02 + 02 + (1.3)2 =
1.69
• For the Ridge Regression Line the sum of squared
residuals plus the ridge regression penalty = (0.3)2 + Weight
(0.1)2 + (0.8)2 = 0.74
 If we choose to minimize the sum of the squared residuals
plus   (slope)2 we would choose the ridge regression line.

ML / Module-I / Jasaswi Prasad Mohanty 164

 Slope and Prediction Sensitivity

 When the slope of the line is steep then the prediction of

size is very sensitive to relative small change in Weight.
 When the slope of the line is small, then the predictions
of size are less sensitive to relative small change in
Weight.

Size
 The Ridge Regression Line has a smaller slope in
comparison to Least Square Line hence less sensitive to
Weight than the Least Square Line.
 Ridge Regression penalizes the higher slopes.

Weight

ML / Module-I / Jasaswi Prasad Mohanty 165

 Value of 
 Value of  can be from 0 to ∞.
 When  is 0, the Ridge Regression Penalty is also 0.
 When the value of  increases the slope decreases. The larger we make , the slope
asymptotically close to 0.
 The larger  gets, our predictions for size become less sensitive to Weight.
 We use cross validation method to determine the value of 

ML / Module-I / Jasaswi Prasad Mohanty 166

 Ridge Regression for smaller data set
 We know the equation of the line
𝑆𝑖𝑧𝑒 = 𝑦_𝑖𝑛𝑡𝑒𝑟𝑐𝑒𝑝𝑡 + 𝑠𝑙𝑜𝑝𝑒 × 𝑊𝑒𝑖𝑔ℎ𝑡
 We need at least two points to solve for the parameters y-intercept and slope.
 If we have only one data point, then we would not be able to solve for these parameters.
Because we would get infinite number of lines passing through a single data point with zero sum
of the residuals.
 Similarly, to solve for three parameters, Least Square needs at least three data points.
 Like that if we have a equation of 1000 parameters, we need at least 1000 data points to
estimate all the parameters.
 So, what should we do when we have 1000 parameters and 500 data points?
• We use Ridge Regression
• It turns out that by adding the Ridge Regression penalty we can solve for all 1000 parameter
with only 500 data points.
 When the sample size is relatively small, then Ridge Regression can improve predictions made
from new data (reduce variance) by making the predictions less sensitive to the Training Data.
ML / Module-I / Jasaswi Prasad Mohanty 167
 Ridge Regression
 Least Square Regression (LSR) estimates often low prediction accuracy when the data is non-
linear.
 Prediction accuracy can sometimes be improved by shrinking some of the coefficients .
 Ridge Regression is used for achieving this.
 This method can determine the coefficients when some of the features are highly correlated
unlike LSR.
 In Ridge Regression the coefficients 𝛽 𝑗 = 0, 1, 2, . . . , 𝑝 of then linear model 𝑌 = 𝛽 + ∑ 𝛽𝑋
are obtained by minimizing

𝑅𝑆𝑆 𝛽 = [𝑦 − (𝛽 + 𝛽 𝑋 )] + 𝜆 𝛽 −−−−−−−−−−−− −(1)

where 𝜆 is a parameter to be specified.

L2 regularization
 If 𝜆 is large, ∑ 𝛽 to be small, some of 𝛽 ’s should be small. or L2 norm

ML / Module-I / Jasaswi Prasad Mohanty 168

 Ridge Regression Steps
1. The original data matrix is given by
𝑥 𝑥 … 𝑥 … 𝑥
⋮ ⋮ ⋱ ⋮ ⋱ ⋮
𝑋= 𝑥 𝑥 … 𝑥 … 𝑥
⋮ ⋮ ⋱ ⋮ ⋱ ⋮
𝑥 𝑥 … 𝑥 … 𝑥
Subtract mean of each column from the individual elements of the corresponding column, that is
transform each data 𝑥 to 𝑥 − 𝑥 , 𝑖 = 1, 2, 3, . . . , 𝑁; 𝑗 = 1, 2, 3, . . . , 𝑝 where 𝑥 = ∑ 𝑥
The updated X (Normalized X) is given by
𝑥 −𝑥 𝑥 −𝑥 … 𝑥 −𝑥 … 𝑥 −𝑥
⋮ ⋮ ⋱ ⋮ ⋱ ⋮
𝑋 = 𝑥 −𝑥 𝑥 −𝑥 … 𝑥 −𝑥 … 𝑥 −𝑥 −−−−−−−−−−−− −(2)
⋮ ⋮ ⋱ ⋮ ⋱ ⋮
𝑥 −𝑥 𝑥 −𝑥 … 𝑥 −𝑥 … 𝑥 −𝑥
2. Denote 𝛽 = (𝛽 𝛽 . . . 𝛽 ) −−−−−−−−−−−− −(3)

ML / Module-I / Jasaswi Prasad Mohanty 169

 Ridge Regression Steps
3. Deonte response vector 𝑦 = (𝑦 − 𝛽 𝑦 −𝛽 . . . 𝑦 − 𝛽 ) −−−−−−−−−−−− −(4)

where average response 𝛽 = ∑ 𝑦 −−−−−−−−−−−− −(5)

With the above notations, equation (1) reduces to
𝑅𝑆𝑆 𝛽 = 𝑦 − 𝑋𝛽 (𝑦 − 𝑋𝛽) + 𝜆𝛽 𝛽
We need to determine the optimal 𝛽 say 𝛽 , by minimizing 𝑅𝑆𝑆 𝛽 over all 𝛽𝜖𝑅
𝜕𝑅𝑆𝑆 𝛽
= −2𝑋 𝑦 − 𝑋𝛽 + 𝜆2𝛽 = 0 ⇒ −𝑋 𝑦 + 𝑋 𝑋𝛽 + 𝜆𝛽 = 0 ⇒ 𝑋 𝑋𝛽 + 𝜆𝛽 = 𝑋 𝑦
𝜕𝛽
⇒ 𝛽 𝑋 𝑋 + 𝜆𝐼 = 𝑋 𝑦 ⇒ 𝛽 = 𝑋 𝑋 + 𝜆𝐼 𝑋 𝑦 −−−−−−−−−−−− −(6)
NOTE:
1. The solution 𝛽 obtained in (6) adds a positive constant 𝜆 to the diagonals elements of 𝑋 𝑋
before inversion which makes 𝑋 𝑋 + 𝜆𝐼 non-singular (invertible) even if 𝑋 𝑋 is not of full
column rank (columns are independent).
2. Ridge regression shrinks the coefficients of some of the predictors which has less significant
effect on the response.

ML / Module-I / Jasaswi Prasad Mohanty 170

 Ridge Regression: Algorithm
Input:{(𝑥 , 𝑦 )} , where 𝑥 𝜖𝑅 , 𝑦 𝜖𝑅 and a new predictor vector 𝑥 𝜖𝑅
Output: 𝑦 𝜖 𝑅
1. Choose 𝜆 > 0
2. Create the matrix X, the vector  and the vector y defined in equation (2), (3) and (4) respectively.
𝟏
3. Compute 𝜷 = 𝑿𝑻 𝑿 + 𝝀𝑰 (𝑿𝑻 𝒚)[As per equation (6)]

4. Compute 𝜷 = ∑ 𝑦
1
𝑥
1
5. Create the 𝑥 = = 𝑥
𝑥 ⋮
𝑥
𝜷
6. Compute 𝛽 =
𝜷
7. Return 𝛽 . 𝑥

ML / Module-I / Jasaswi Prasad Mohanty 171

 Regularized Linear Regression: Gradient Descent Method
 The cost function J  = [∑ ℎ (𝑥 ) − 𝑦 +𝜆∑ 𝜃 ] where 𝑦 ( ) = ℎ (𝑥 ) =
 0 +  1𝑥 .
 Gradient Descent Steps:
Repeat{

0 = 0 − 𝛼 ∑ 𝑦( ) − 𝑦 𝑥 ()

𝑗 = 𝑗 − 𝛼[ ∑ 𝑦 −𝑦 𝑥 + 𝑗] for 𝑗 = 1, 2, . . . , 𝑝
}
𝜆 1
𝑗 = 𝑗(1 − 𝛼 ) − 𝛼 𝑦 −𝑦 𝑥
𝑚 𝑚

 As 1 − 𝛼 < 1, the value of the parameter 𝑗 is shrinking.

ML / Module-I / Jasaswi Prasad Mohanty 172

Least Absolute Selection and Shrinkage Operator (LASSO)
 Some predictors used in a multiple linear regression model are not associated with the response.
 Including such irrelevant variables leads to unnecessary complexity and reduction of prediction
accuracy.
 Sometimes the prediction accuracy and computation time can be improved by setting some of
the coefficients to zero.
 LASSO is a method for reducing some of the coefficients of multiple linear regression model to
zero.
 In other words this method selects the features which significantly affects the response. So, it is
also used as a feature selection method.
 The difference between ridge and lasso regression is that it tends to make coefficients to
absolute zero as compared to Ridge which never sets the value of coefficient to absolute zero.

ML / Module-I / Jasaswi Prasad Mohanty 173

 LASSO
 In Ridge Regression the coefficients 𝛽 𝑗 = 0, 1, 2, . . . , 𝑝 of then linear model 𝑌 = 𝛽 + ∑ 𝛽𝑋
using LASSO are obtained by minimizing

𝑅𝑆𝑆 𝛽 = [𝑦 − (𝛽 + 𝛽 𝑋 )] + 𝜆 |𝛽 | , 𝜆 > 0 −−−−−−−−−−−− −(1)

where 𝜆 is a parameter to be specified. L1 regularization

Minimizing 𝑅𝑆𝑆 𝛽 is equivalent to minimizing or L1 norm

𝑅𝑆𝑆 𝛽 = [𝑦 − (𝛽 + 𝛽 𝑋 )]

Subject to ∑ |𝛽 | ≤ 𝑡
where 𝑡>0

ML / Module-I / Jasaswi Prasad Mohanty 174

 Limitations of LASSO
• Lasso sometimes struggles with some types of data. If the number of
predictors (p) is greater than the number of observations (n), Lasso will pick
at most n predictors as non-zero, even if all predictors are relevant (or may
be used in the test set).
• If there are two or more highly collinear variables, then LASSO regression
selects one of them randomly which is not good for the interpretation of data.
• Lasso does not have the close form solution since the penalty is not
differentiable.

ML / Module-I / Jasaswi Prasad Mohanty 175

 Problem

ML / Module-I / Jasaswi Prasad Mohanty 176