Overfitting, regularization, and

model validation
Ngô Minh Nhựt

2025
 Content
❑ Overfitting
❑ Regularization
❑ Model validation

2025 Machine learning 2

 Overfitting

When there are many features:

‣ Model may overfit with training data (training cost ~ 0)
‣ Could not generalize for new samples (testing cost >> 0)

Source: Andrew Ng

2025 Machine learning 3

 Overfitting
❑ Logistic regression

Source: Andrew Ng

2025 Machine learning 4

 Overfitting

Source: ChatGPT

2025 Machine learning 5

 Overfitting
❑ Solutions:
■ Reduce number of features
● Select features through data analysis
● Select features through model validation
■ Regularization
● Reduce effect of parameters to output
● Good in case features contributes slightly to output

2025 Machine learning 6

 Feature selection
❑ Learn structure, distribution and relationship between
input and output
■ Tools like histogram or scatter plot may be helpful

Source: Internet

2025 Machine learning 7

 Feature selection
❑ Evaluate importance of feature using
correlation
■ Apply to linear relationship
■ Determine correlation inside features
■ Determine correlation between each feature
and output
❑ Feature that is highly correlated with others
can be eliminated
❑ Feature that is highly correlated with output
is more important

Source: Internet

2025 Machine learning 8

 Feature selection
❑ Steps to select features
■ Learn structure, distribution and relationship between
input and output
■ Evaluate importance of features using correlation
❑ Model evaluation
■ Test model with selected features
■ Repeat feature selection until model is good enough

2025 Machine learning 9

 Regularization

Source: Andrew Ng

2025 Machine learning 10

 Regularization
❑ When values of parameters become small
■ Produce simpler models
■ Reduce overfitting

Without regularization: the model overfits by assigning large weights to features,

leading to high variance.

With regularization: the model penalizes large weights, leading to better

generalization.

2025 Machine learning 11

 L2 Regularization
❑ Cost function

• Lambda: weight decay

• When lambda is bigger, model becomes simpler

2025 Machine learning 12

 L2 Regularization

If lambda too big?

Source: Andrew Ng

2025 Machine learning 13

 L2 Regularization

Source: ChatGPT

2025 Machine learning 14

 L2 Regularization
❑ Vector gradient

Gradient descent algorithm works as usual

2025 Machine learning 15

 L2 Regularization
❑ Logistic regression – cost function

2025 Machine learning 16

 L2 Regularization
❑ Logistic regression – vector gradient

2025 Machine learning 17

 L1 Regularization
❑ Cost function

• Adds penalty on sum of absolute weights

• Effect: force some weights to become exactly zero
• Perform feature selection by eliminating irrelevant
features

2025 Machine learning 18

 L1 an L2 Regularization

Source: ChatGPT

2025 Machine learning 19

 Overfitting
❑ Other reasons for overfitting:
■ Training set size is too small
■ Training samples do not represent all possible input data
■ Training samples contain large amount of irrelevant
information or noisy data
■ Model is overtrained

2025 Machine learning 20

 Overfitting
❑ Other solutions for overfitting:
■ Data augmentation
● Increase number of training samples by slightly changing
available samples
■ Early stopping during training
● Training is stopped before the model learns noise in data
■ Ensembling
● Combine predictions from several models to get more
accurate results

2025 Machine learning 21

 Model validation
❑ Learning process makes model fit with training data
❑ Can learnt model generalize for new samples?
■ Validate the model with unseen data
❑ Solution:
■ Learn, validate, and select the best model
■ Test if the selected model works well with new data

Train Validate Test

2025 Machine learning 22

 Dataset
❑ Split dataset into 3 subsets: train, validation, and test
❑ With large dataset, the ratio is: 60% : 20% : 20%

Test
20%

Validation Train
20% 60%

2025 Machine learning 23

 Cost function
Train error

Cross-validation error

Test error

2025 Machine learning 24

 k-fold cross validation
❑ Randomly split dataset into k parts
❑ Learn and validate k times. k is usually 10

#1 1 2 … k-1 k

Validation Training

#2 1 2 … k-1 k

#i …

#k 1 2 … k-1 k

2025 Machine learning 25

 k-fold cross validation

Source: Internet

2025 Machine learning 26

 Model finetuning
❑ Model has various parameters
❑ Learning algorithm has various parameters as well,
namely hyper parameters
❑ What should we do when model does not work well?
■ Collect more data
■ Reduce number of features
■ Try with new features
■ Switch to other models or hypotheses
■ Reduce weight decay
■ Increase weight decay
2025 Machine learning 27
 Model finetuning

Source: Internet

2025 Machine learning 28

 Bias and variance
❑ Bias: model error on training set
❑ Variance: difference between model error on
validation set and that on training set
❑ Adjust bias and variance until they reach minimal

Source: Andrew Ng

2025 Machine learning 29

 Bias and variance

Source: Internet

2025 Machine learning 30

 Bias and variance

Source: ChatGPT

2025 Machine learning 31

 Model evaluation
❑ Measurements
■ Precision, recall
■ Accuracy
■ F1-score
❑ Depending on problems, we need some suitable
measures

2025 Machine learning 32

 Model evaluation
True positive
❑ Precision =
Predicted positive

True positive
❑ Recall =
Positive

2 x Precision x Recall
❑ F1 − score =
Precision +Recall

True positive + True negative

❑ Accuracy =
Positive + Negative

Source: Wikipedia
2025 Machine learning 33
 Confusion matrix
Predicted
Cat Dog Tiger Wolf
Total
Cat 6 0 3 1 10
Dog 2 4 0 4 10
Actual

Tiger 3 3 3 0 9
Wolf 1 4 1 2 8
Total 12 11 7 7

>>> [Link].classification_report
>>> y_true = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …]
>>> y_pred = [0, 0, 0, 0, 0, 0, 2, 2, 2, 3, …]
>>> target_names = [‘Cat', ‘Dog', ‘Tiger’, ‘Wolf’]

2025 Machine learning 34

 Confusion matrix
❑ Example code

from [Link] import classification_report

y_true = [0, 0, 1, 1, 2, 2]
y_pred = [0, 1, 1, 1, 2, 0]
target_names = ["Class A", "Class B", "Class C"]

print(classification_report(y_true, y_pred, target_names=target_names))

2025 Machine learning 35

 Confusion matrix
Predicted
Cat Dog Tiger Wolf
Total
Cat 6 0 3 1 10
Dog 2 4 0 4 10
Actual

Tiger 3 3 3 0 9
Wolf 1 4 1 2 8
Total 12 11 7 7

❑ Macro average is calculated over all

classes regardless their proportions
❑ Weighted average is calculated over
all classes taking their proportions into
account

2025 Machine learning 36

 Quiz
❑ 1. Which of the following scenarios indicates an overfitting
model?
■ A) Low training loss, high test loss
■ B) High training loss, low test loss
■ C) Both training and test loss are high
■ D) Both training and test loss are low
❑ 2. What is the primary purpose of L2 regularization in machine
learning?
■ A) Encourages sparsity by setting some weights to zero
■ B) Penalizes large weights to prevent overfitting
■ C) Speeds up training without affecting model complexity
■ D) Removes unnecessary features from the dataset

2025 Machine learning 37

 Quiz
❑ 3. In early stopping, at which point should you stop
training?
■ A) When training loss is minimized
■ B) When validation loss starts increasing
■ C) When both training and validation loss reach zero
■ D) After a fixed number of epochs, regardless of loss behavior
❑ 4. Which technique helps evaluate a model's generalization
ability without using separate training and test sets?
■ A) Dropout Regularization
■ B) L1/L2 Regularization
■ C) k-Fold Cross-Validation
■ D) Batch Normalization

2025 Machine learning 38

 Exercise
Predicted
Cat Dog Tiger Wolf
Total
Cat 6 0 3 1 10
Dog 2 4 0 4 10
Actual

Tiger 3 3 3 0 9
Wolf 1 4 1 2 8
Total 12 11 7 7

❑ Calculate precision, recall and F1-score of class dog

2025 Machine learning 39

 Exercise
Predicted
Cat Dog Tiger Wolf
Total
Cat 6 0 3 1 10
Dog 2 4 0 4 10
Actual

Tiger 3 3 3 0 9
Wolf 1 4 1 2 8
Total 12 11 7 7

❑ Calculate macro average and weighted average of

precision

2025 Machine learning 40