Scribd
Upload
15 views7 pages

Confusion Matrix and Overfitting Analysis

The document outlines an experiment to implement a Confusion Matrix for evaluating classification models and to simulate overfitting effects. It includes prerequisites, key terms, an experimental setup, and a detailed procedure using the CIFAR10 dataset with a neural network model. The experiment aims to visualize model performance and overfitting through accuracy metrics and confusion matrix calculations.

Uploaded by

manyaprakash31
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
15 views7 pages

Confusion Matrix and Overfitting Analysis

The document outlines an experiment to implement a Confusion Matrix for evaluating classification models and to simulate overfitting effects. It includes prerequisites, key terms, an experimental setup, and a detailed procedure using the CIFAR10 dataset with a neural network model. The experiment aims to visualize model performance and overfitting through accuracy metrics and confusion matrix calculations.

Uploaded by

manyaprakash31
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

4/22/25, 8:52 PM DeepLearningLab2.

ipynb - Colab

keyboard_arrow_down Experiment 2
Deploy the Confusion matrix and simulate for Overfitting

Objective:

[Link] understand and implement the Confusion Matrix as a performance evaluation metric for
classification models.

[Link] simulate and analyze the effects of overfitting on model performance.

Prerequisites:

Basic understanding of Machine Learning concepts, especially classification.


Familiarity with Python programming and libraries like NumPy, Pandas, Scikit-learn, and
Matplotlib.
A working Python environment (e.g., Jupyter Notebook, Google Colab).

Key Terms:

Confusion Matrix: A table summarizing the performance of a classification model,


showing the counts of true positive, true negative, false positive, and false negative
predictions.
True Positive (TP): Correctly predicted positive instances.
True Negative (TN): Correctly predicted negative instances.
False Positive (FP) (Type I Error): Incorrectly predicted positive instances (actually
negative).
Overfitting: A phenomenon where a model learns the training data too well, including
noise, and performs poorly on unseen data.
Regularization: Techniques used to prevent overfitting (e.g., L1/L2 regularization, dropout).

Experimental Setup:

[Link]: Choose a suitable dataset for classification (e.g., Iris, Breast Cancer, or a synthetic
dataset). If using a real-world dataset, ensure proper data preprocessing (handling missing
values, encoding categorical features).

[Link] Selection: Select a classification algorithm (e.g., Logistic Regression, Decision Tree,
Random Forest, Support Vector Machine).

[Link]-Test Split: Divide the dataset into training and testing sets (e.g., 80% for training, 20% for
testing).

[Link] 1/7
4/22/25, 8:52 PM [Link] - Colab

[Link] Training: Train the chosen model on the training data.

[Link]: Use the trained model to make predictions on the testing data.

[Link] Matrix Calculation: Calculate the confusion matrix based on the actual and
predicted labels.

[Link] Metrics Calculation: Calculate accuracy, precision, recall, and F1-score from the
confusion matrix.

[Link] Simulation: Train the model with increasing model complexity (e.g., increasing tree
depth in a decision tree, using a polynomial kernel of higher degree in SVM) or by training on a
smaller subset of the training data. Observe the performance on both training and testing sets.

Theory:

The confusion matrix provides a detailed breakdown of the model's performance beyond simple
accuracy. Understanding the different components (TP, TN, FP, FN) is crucial for evaluating the
model's strengths and weaknesses, especially in imbalanced datasets. Overfitting occurs when
a model becomes too complex and starts memorizing the training data, leading to poor
generalization to new data. This is often indicated by high accuracy on the training set but low
accuracy on the testing set.

Experimental Procedure:

Step 1: Import neccesary libraries

import tensorflow as tf

from [Link] import cifar10

from [Link] import Sequential


from [Link] import Dense, Flatten
from [Link] import to_categorical
from [Link] import confusion_matrix

import numpy as np
import [Link] as plt

Step 2: Load and preprocess the CIFAR10 dataset:

(x_train, y_train), (x_test, y_test)= cifar10.load_data()


x_train= x_train/255.0
x_test= x_test/255.0
y_train= to_categorical(y_train)
y_test= to_categorical(y_test)

[Link] 2/7
4/22/25, 8:52 PM [Link] - Colab

Downloading data from [Link]


170498071/170498071 ━━━━━━━━━━━━━━━━━━━━ 2s 0us/step

Step 3: Build a simple neural network model:

model = Sequential([
Flatten(input_shape=(32,32,3)),
Dense(128, activation='relu'),
Dense(10, activation='softmax')
])

/usr/local/lib/python3.11/dist-packages/keras/src/layers/reshaping/[Link]: Use
super().__init__(**kwargs)

 

[Link](optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

Step 4: Train the model:

history = [Link](x_train, y_train, epochs=10, validation_data=(x_test, y_test))

Epoch 1/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 14s 8ms/step - accuracy: 0.2936 - loss: 1.9858 - val_a
Epoch 2/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.3719 - loss: 1.7501 - val_a
Epoch 3/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 21s 8ms/step - accuracy: 0.3964 - loss: 1.6945 - val_a
Epoch 4/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.4142 - loss: 1.6468 - val_a
Epoch 5/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.4188 - loss: 1.6235 - val_a
Epoch 6/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 22s 8ms/step - accuracy: 0.4286 - loss: 1.6072 - val_a
Epoch 7/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 21s 9ms/step - accuracy: 0.4331 - loss: 1.5939 - val_a
Epoch 8/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.4404 - loss: 1.5734 - val_a
Epoch 9/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.4396 - loss: 1.5682 - val_a
Epoch 10/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 14s 9ms/step - accuracy: 0.4414 - loss: 1.5599 - val_a

 

Step 5: Simulate overfitting by increasing the number of epochs significantly:

history_overfit = [Link](x_train, y_train, epochs=50, validation_data=(x_test, y_test)

Epoch 1/50 
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.4951 - loss: 1.4221 - va
Epoch 2/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.4984 - loss: 1.4052 - va
Epoch 3/50
[Link] 3/7
4/22/25, 8:52 PM [Link] - Colab
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.4942 - loss: 1.4143 - va 
Epoch 4/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.4988 - loss: 1.4059 - va
Epoch 5/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 24s 10ms/step - accuracy: 0.4914 - loss: 1.4096 - v
Epoch 6/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.5000 - loss: 1.4002 - va
Epoch 7/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 21s 8ms/step - accuracy: 0.4972 - loss: 1.4054 - va
Epoch 8/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.4983 - loss: 1.4019 - va
Epoch 9/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.5004 - loss: 1.3984 - va
Epoch 10/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.5004 - loss: 1.3976 - va
Epoch 11/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 14s 9ms/step - accuracy: 0.4976 - loss: 1.4114 - va
Epoch 12/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 19s 8ms/step - accuracy: 0.4991 - loss: 1.4040 - va
Epoch 13/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.5010 - loss: 1.3918 - va
Epoch 14/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 7ms/step - accuracy: 0.5021 - loss: 1.3986 - va
Epoch 15/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.5008 - loss: 1.3991 - va
Epoch 16/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 21s 8ms/step - accuracy: 0.5011 - loss: 1.3922 - va
Epoch 17/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.4981 - loss: 1.4032 - va
Epoch 18/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 19s 8ms/step - accuracy: 0.5066 - loss: 1.3870 - va
Epoch 19/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 21s 8ms/step - accuracy: 0.4995 - loss: 1.3968 - va
Epoch 20/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.5004 - loss: 1.3967 - va
Epoch 21/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.5016 - loss: 1.3896 - va
Epoch 22/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 22s 9ms/step - accuracy: 0.5042 - loss: 1.3908 - va
Epoch 23/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 14s 9ms/step - accuracy: 0.4991 - loss: 1.3968 - va
Epoch 24/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 19s 8ms/step - accuracy: 0.5009 - loss: 1.3936 - va
Epoch 25/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.5007 - loss: 1.3906 - va
Epoch 26/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 21s 8ms/step - accuracy: 0.5046 - loss: 1.3803 - va
Epoch 27/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.5050 - loss: 1.3869 - va
Epoch 28/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.5027 - loss: 1.3956 - va
Epoch 29/50

Step 6: Evaluate the model and generate the confusion matrix:


loss,accuracy = [Link](x_test, y_test, verbose=0) 
print('Test Loss:', loss)

[Link] 4/7
4/22/25, 8:52 PM [Link] - Colab

print('Test Accuracy:', accuracy)

Test Loss: 1.6039667129516602


Test Accuracy: 0.4422000050544739

y_pred = [Link](x_test)
y_pred_classes = [Link](y_pred, axis=1)
y_true_classes = [Link](y_test, axis=1)
cm = confusion_matrix(y_true_classes, y_pred_classes)
print("Confusion Matrix: ",cm)

313/313 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step


Confusion Matrix: [[628 44 73 23 12 23 36 61 32 68]
[ 73 518 28 19 10 33 48 46 17 208]
[103 18 340 58 73 90 180 109 6 23]
[ 49 21 91 204 31 250 223 91 8 32]
[ 64 7 182 46 230 61 231 163 7 9]
[ 40 8 93 129 39 404 139 122 6 20]
[ 11 11 74 71 55 59 656 48 4 11]
[ 43 12 66 43 55 103 51 586 6 35]
[300 79 27 27 17 50 46 21 342 91]
[ 93 157 21 35 1 34 40 85 20 514]]

Step 7: Plot training and validation to visualize overfitting:

[Link](history_overfit.history['accuracy'])
[Link](history_overfit.history['val_accuracy'])
[Link]('Model accuracy')
[Link]('Accuracy')
[Link]('Epoch')
[Link](['Train', 'Validation'], loc='upper left')
[Link]()

[Link] 5/7
4/22/25, 8:52 PM [Link] - Colab

Precautions and Sources of Error:

Data Leakage: Ensure that information from the testing set does not leak into the training
set.

Imbalanced Datasets: If the dataset has a skewed class distribution, consider using
appropriate evaluation metrics (e.g., precision, recall, F1-score) and techniques (e.g.,
oversampling, undersampling).

Random Initialization: Be mindful of the random initialization of model parameters and use
a fixed random seed for reproducibility.

Insufficient Data: Limited training data can exacerbate overfitting.

Result:

The output should include:

[Link] confusion matrix.

[Link] performance metrics (accuracy, precision, recall, F1-score).

[Link] showing the training and testing accuracy/loss as a function of model complexity or
training data size, demonstrating the effects of overfitting.

Short Questions:

[Link] 6/7
4/22/25, 8:52 PM [Link] - Colab

[Link] are the advantages of using a confusion matrix over simple accuracy?

[Link] the difference between Type I and Type II errors.

[Link] does overfitting affect the model's performance on training and testing data?

[Link] are some common techniques for preventing overfitting?

[Link] can you interpret the values in a confusion matrix to understand the model's behavior?

[Link] 7/7

Common questions

Powered by AI

By examining the confusion matrix, you can identify biases by observing which classes have high false positive or false negative rates. For instance, if certain classes consistently show a high number of misclassifications, this imbalance indicates the model is biased towards or against these classes. Adjustments can then be made, like class rebalancing, to address these biases .

Imbalances in datasets can lead to models that are biased towards the majority class, resulting in poor predictive performance for the minority class. This skew can lead to misleading accuracy when the minority class is the target of interest. Techniques such as oversampling minority classes, undersampling majority classes, or using more sophisticated measures like F1-score for model evaluation are key to mitigating these issues .

A confusion matrix provides a detailed breakdown of a model's performance by displaying the counts of true positives, true negatives, false positives, and false negatives. This granularity allows for an understanding of specific types of misclassifications which accuracy alone cannot provide. For example, it reveals whether the model is good at differentiating between all classes or just one, and is especially valuable in evaluating performance on imbalanced datasets .

Data preprocessing is crucial in avoiding overfitting by ensuring that the model learns relevant patterns and not noise. Techniques include handling missing values, normalizing data scales, and encoding categorical variables. Proper preprocessing ensures that the data quality is high and consistent, which stabilizes model training and improves performance on unseen data, thus reducing overfitting .

Overfitting occurs when a model performs exceedingly well on the training dataset, indicating that it has learned the specific details and noise in the training data. However, this leads to poor performance on the testing dataset, where the accuracy significantly drops, showing that the model fails to generalize well to new, unseen data .

Precision is more critical than recall in scenarios where the cost of false positives is high. For example, in email spam detection, precision is prioritized as marking non-spam emails incorrectly as spam (false positive) could lead to missing important emails. Conversely, recall is prioritized in medical diagnoses where missing a condition (false negative) could be more detrimental .

Using a fixed random seed is important for ensuring the reproducibility of results in machine learning experiments. It stabilizes the randomness in procedures like weight initialization and dataset shuffling, making comparisons between model versions or hyperparameter settings valid and reliable .

Visual tools such as plots showing training and validation accuracy/loss across epochs are effective in assessing overfitting. These graphs help visualize trends where training accuracy continues to increase while validation accuracy stagnates or declines, indicating overfitting. Such plots complement confusion matrices in providing a comprehensive view of a model's generalization ability .

Regularization techniques, such as L1 and L2 regularization, help prevent overfitting by adding a penalty on the size of the coefficients of features. This discourages overly complex models, reducing their variance without significantly increasing bias. Techniques like dropout randomly ignore neurons during training, which prevents the model from becoming too dependent on any single feature, further reducing overfitting .

Training a model for too many epochs can lead to overfitting, where the model becomes highly accurate on the training set yet performs poorly on the testing set due to its memorization of the training data, including its noise. This phenomenon indicates that the model has lost its generalization power. This is evident in decoration of performance metrics and confusion matrix results that show inconsistencies .

You might also like