SCSB4008 INTELLIGENT SYSTEMS ENGINEERING

UNIT 2
MACHINE LEARNING AND DATA MINING
Introduction to machine learning and data mining-Supervised and unsupervised learning
algorithms-Feature extraction and dimensionality reduction-Data pre-processing and cleaning
techniques- Evaluation and validation of machine learning models-metrics-Cross-validation-
Over fitting and under fitting

INTRODUCTION TO MACHINE LEARNING AND DATA SCIENCE

Machine Learning and Data Science are at the core of modern Intelligent Systems
Engineering. These fields empower machines to learn from data, make decisions, and adapt
over time without being explicitly programmed for every task. This chapter introduces the
foundational principles of Data Science and Machine Learning, their interrelationship, core
methodologies, and their importance in building intelligent systems.

What is Data Science?

Data Science is an interdisciplinary field that combines techniques from statistics,

computer science, and domain-specific knowledge to extract insights and knowledge from
structured and unstructured data.

Key Components:

• Data Collection: Gathering raw data from various sources such as sensors, logs, APIs,
and databases.

• Data Cleaning and Preparation: Removing inconsistencies, handling missing values,

and transforming data for analysis.

• Exploratory Data Analysis (EDA): Visualizing and summarizing data to understand

patterns and distributions.

• Data Modeling and Algorithms: Applying machine learning or statistical models.

• Communication of Results: Using visualization and storytelling to convey insights

effectively.

Applications:

• Fraud detection, business intelligence, healthcare analytics, recommendation systems,

social media monitoring, etc.
What is Machine Learning?

Machine Learning (ML) is a branch of artificial intelligence (AI) that enables systems
to learn patterns from data and make decisions or predictions without being explicitly
programmed.

Types of Machine Learning:

• Supervised Learning: Learning from labeled data.

o Examples: Linear Regression, Decision Trees, Support Vector Machines (SVM)

• Unsupervised Learning: Learning from data without labels.

o Examples: K-means clustering, Principal Component Analysis (PCA)

• Semi-supervised Learning: Combination of a small amount of labeled data and a large
amount of unlabeled data.

• Reinforcement Learning: Learning through trial and error based on feedback from the
environment.

Applications:
• Speech and image recognition, spam filtering, predictive maintenance, personalized
recommendations, autonomous vehicles.
Relationship Between Data Science and Machine Learning

Data Science is a broader field that includes data acquisition, cleaning, exploration,
visualization, and modeling. Machine Learning forms the core of the modeling and prediction
part in Data Science.

Example:
In a movie recommendation system:

• Data Science handles collecting user data, cleaning, analyzing user preferences.

• Machine Learning is used to train algorithms that predict user interests.

SUPERVISED ALGORITHMS AND UNSUPERVISED ALGORITHMS

Machine learning algorithms are broadly classified into two categories:

• Supervised Learning

• Unsupervised Learning

These approaches differ based on whether the data used for training contains labels (i.e., target
outputs).
Supervised Learning

Supervised learning involves training a model using labeled data. The algorithm learns
a mapping function from input to output based on example input-output pairs. Once trained, it
can predict outcomes for unseen data.

Common Applications:

• Classification (e.g., spam detection, image recognition)

• Regression (e.g., stock price prediction, sales forecasting)

Types of Supervised Algorithms

Linear Regression
Linear regression is a regression algorithm that models the relationship between one
or more independent variables (X) and a dependent variable (Y) using a linear equation:

Use Cases:

• Predicting housing prices

• Sales forecasting

• Estimating exam scores based on study hours

Advantages:

• Simple and easy to interpret

• Computationally efficient

Limitations:

• Assumes linearity
• Sensitive to outliers

Multiple Linear Regression

Multiple linear regression involves more than one independent variable and one
dependent variable. The equation for multiple linear regression is:
 The goal of the algorithm is to find the best Fit Line equation that can predict the values
based on the independent variables. In regression set of records are present with X and Y values
and these values are used to learn a function so if you want to predict Y from an unknown X
this learned function can be used. In regression we have to find the value of Y, So, a function
is required that predicts continuous Y in the case of regression given X as independent features.

DECEISION TREE CLASSIFIER

A decision tree in machine learning is a supervised learning algorithm used for both
classification and regression tasks. It visually represents a series of decisions as a tree-like
structure, guiding data through a process of splitting and branching until a final outcome is
reached.

Fig DECISION TREE

Key Concepts:

• Supervised Learning: Decision trees learn from labeled data, where the desired outcome
is known.

✓ Classification: Predicting a categorical outcome (e.g., spam/not spam, cat/dog).

✓ Regression: Predicting a continuous numerical outcome (e.g., price of a house,
temperature).

Tree Structure:

• Root Node: The starting point of the tree, representing the entire dataset.

• Internal Nodes: Represent decision points or tests on attributes.

• Branches: Connect nodes and represent the outcome of a decision.

• Leaf Nodes: Terminal nodes that provide the final prediction or classification.

• Splitting: The process of dividing data based on attribute values to create branches.
• Greedy Search: Decision tree algorithms use a greedy approach to find the best split
at each node, optimizing for a specific criterion like information gain or Gini impurity.

Information Gain:

Given a set of examples D, we first compute its entropy:

If we make attribute Ai, with v values, as the root of the current tree, this will partition D into
v subsets D1, D2 …, Dv. The expected entropy if Ai is used as the current root:
v | Dj |
entropyAi ( D) =  | D |  entropy( D )
j =1
j

Information gained by selecting attribute Ai to branch or to partition the data is

gain( D, Ai ) = entropy ( D) − entropy Ai ( D)

We evaluate every attribute. We choose the attribute with the highest gain to branch/split the
current tree.

Let us consider the Table 2.1, and let us calculate the Entropy and Information gain to decide
the parameter for the first split
 Table 2.1 Dataset for Loan Approval

6 6 9 9
entropy( D) = −  log 2 −  log 2 = 0.971
15 15 15 15
6 9
entropyOwn _ house ( D) =  entropy( D1 ) +  entropy( D2 )
15 15
6 9
=  0 +  0.918
15 15
= 0.551
5 5 5
entropyAge ( D) =  entropy( D1 ) +  entropy( D2 ) +  entropy( D3 )
15 15 15
5 5 5
=  0.971 +  0.971 +  0.722
15 15 15
= 0.888

From the above calculations it is clear that Own_house has the highest information
gain and hence it is chosen for the split. The split is done as in Fig 2.1
 Fig Decision Tree for Table2.1

How it works: 1. Start at the root node:

The entire dataset is considered.

2. Select the best attribute to split on:

The algorithm chooses the attribute that best separates the data based on a chosen
metric.

3. Create branches:
Based on the attribute's values, the data is split into different branches.

4. Repeat steps 2 and 3 for each branch:

This process continues recursively until a stopping criterion is met (e.g., reaching a
maximum depth, having a minimum number of samples in a leaf, or achieving a pure split).

5. Assign a prediction to each leaf node:

The final leaf nodes represent the predicted outcome for the data points that reach them.

Advantages of Decision Trees:

• Easy to understand and interpret: The tree structure makes it easy to visualize the
decision-making process.

• Can handle both categorical and numerical data: Decision trees can be applied to a
wide range of datasets.

• Feature selection: Decision trees can implicitly identify important features by how they
are used to split the data.
• Non-parametric: They don't assume any specific data distribution.
Disadvantages of Decision Trees:
• Prone to overfitting: Decision trees can create overly complex structures
that fit the training data too closely, leading to poor generalization on new
data.
• Can be unstable: Small changes in the data can lead to significantly
different tree structures.
• Pruning: Techniques like pruning are often necessary to prevent
overfitting.

Example:
Imagine predicting whether someone will play sports based on the weather.
A decision tree might first split based on whether it's "Sunny" or "Rainy". Then,
if it's sunny, it might split based on humidity (High or Low). Finally, if it's rainy,
it might check if it's "Cold" or "Warm". The leaf nodes would then indicate
whether or not the person is likely to play sports.

Naïve Bayes Classifier

Naive Bayes is a classification algorithm that uses probability to predict
which category a data point belongs to, assuming that all features are unrelated.
This article will give you an overview as well as more advanced use and
implementation of Naive Bayes in machine learning.
It is called “naive” because it assumes that all features are independent of
each other, an assumption that is rarely true in real-world data, but still performs
well in many cases.
Bayes’ Theorem goes as

In the context of classification

Key Features of Naive Bayes Classifiers

The main idea behind the Naive Bayes classifier is to use Bayes' Theorem to classify data
based on the probabilities of different classes given the features of the data. It is used mostly
in high-dimensional text classification

• The Naive Bayes Classifier is a simple probabilistic classifier and it has very few
number of parameters which are used to build the ML models that can predict at a faster
speed than other classification algorithms.

• It is a probabilistic classifier because it assumes that one feature in the model is

independent of existence of another feature. In other words, each feature contributes to
the predictions with no relation between each other.

• Naïve Bayes Algorithm is used in spam filtration, Sentimental analysis, classifying

articles and many more.

Consider a fictional dataset that describes the weather conditions for playing a game of tennis.
Given the weather conditions, each tuple classifies the conditions as fit(“Yes”) or unfit(“No”)
for playing tennis. Here is a tabular representation of our dataset.

The dataset is divided into two parts, namely, feature matrix and the response vector.

• Feature matrix contains all the vectors(rows) of dataset in which each vector consists
of the value of dependent features. In above dataset, features are ‘Outlook’,
‘Temperature’, ‘Humidity’ and ‘Windy’.

• Response vector contains the value of class variable(prediction or output) for each row
of feature matrix. In above dataset, the class variable name is ‘Play golf’.
The Learning Phase
Unsupervised Algorithms
Unsupervised learning is a part of machine learning which works differently from
supervised because there is no teacher(supervisor) involved to guide the machine. In this
approach the machine is given with data that has no labels or categories. It analyzes the data
on its own to find patterns, groups or relationships without any prior knowledge. The machine
learns by discovering hidden structures within the data without being told what the correct
output should be.

For example, unsupervised learning can analyze animal data and group the animals by
their traits and behavior. These groups might represent different species which allows the
machine to organize animals without any prior labels or categories.

Fig Illustration of Unsupervised Learning

 Imagine that a machine learning model is trained on many unlabeled images of dogs
and cats. The model has never seen any labeled example that says “dog” or “cat” before so it
doesn’t know how these animals look like.

Now, if a new image is given to the model that contains both dogs and cats it won’t be
able to directly label them as “dog” or “cat.” It will group parts of the image based on
similarities and differences in features like shape or texture. It might separate the image into
two groups one with dog-like features and other with cat-like features.

This happens because unsupervised learning doesn’t rely on prior knowledge or training
with labeled data. It finds patterns and organizes data on its own helps in discovering
information that wasn’t given before.

Types of Unsupervised Learning

Unsupervised learning is divided into two categories of algorithms:

1. Clustering

A clustering is used to group similar data points together. Clustering algorithms work by
repeatedly moving data points closer to to the center of their group (cluster) and farther from
points in other groups. This helps the algorithm to create clear and meaningful clusters. Some
popular clustering algorithms include:

1. K-means clustering

2. Hierarchical clustering

3. Principal Component Analysis (PCA)

4. Singular Value Decomposition (SVD)

5. Independent Component Analysis
6. Gaussian Mixture Models (GMMs)

7. Density-Based Spatial Clustering of Applications with Noise (DBSCAN)

K-mean Clustering

K-means is a partitional clustering [Link] the set of data points (or instances) D be

{x1, x2, …, xn},

where xi = (xi1, xi2, …, xir) is a vector in a real-valued space X  Rr, and r is the number
of attributes (dimensions) in the data.
The k-means algorithm partitions the given data into k clusters. Each cluster has a
cluster center, called centroid.k is specified by the user.
Given k, the k-means algorithm works as follows:
1) Randomly choose k data points (seeds) to be the initial centroids, cluster centers
 2) Assign each data point to the closest centroid

3) Re-compute the centroids using the current cluster memberships.

4) If a convergence criterion is not met, go to 2).

Fig K-means Algorithm

Hierarchical Clustering
Hierarchical clustering is an unsupervised machine learning algorithm that groups data
into a tree of nested clusters. The main types include agglomerative and divisive. Hierarchical
cluster analysis helps find patterns and connections in datasets. Results are presented in a
dendrogram diagram showing the distance relationships between clusters.

Fig Dendrogram
Types of hierarchical clustering

1. Agglomerative (bottom up) clustering:

It builds the dendrogram (tree) from the bottom level, and merges the most
similar (or nearest) pair of clusters. Itstops when all the data points are merged into a
single cluster (i.e., the root cluster).

2. Divisive (top down) clustering:

It starts with all data points in one cluster, the root. It splits the root into a set of
child clusters. Each child cluster is recursively divided further. It stops when only
singleton clusters of individual data points remain, i.e., each cluster with only a single
point

Agglomerative (bottom up) clustering:

It is more popular then divisive methods. At the beginning, each data point forms a
cluster (also called a node). It merges nodes/clusters that have the least distance. The merging
goes on. Eventually all nodes belong to one cluster

Fig Algorithm for Agglomerative Clustering

Measuring the distance of two clusters

There are a few ways to measure distances of two clusters namely,

❑ Single link

❑ Complete link

❑ Average link

❑ Centroids

Single Link

The distance between two clusters is the distance between two closest data points in the
two clusters, one data point from each cluster. It can find arbitrarily shaped clusters, but it may
cause the undesirable “chain effect” by noisy points
Complete Link

The distance between two clusters is the distance of two furthest data points in the two
clusters. It is sensitive to outliers because they are far away.

Average link:

Average link is a compromise between the sensitivity of complete-link clustering to

outliers and the tendency of single-link clustering to form long chains that do not correspond
to the intuitive notion of clusters as compact, spherical objects. In this method, the distance
between two clusters is the average distance of all pair-wise distances between the data points
in two clusters.
Centroid method

In this method, the distance between two clusters is the distance between their centroids

2. Association rule learning

An association rule learning used to find patterns and relationships between different
items in a dataset. It looks for rules like “people who buy X often also buy Y”.

Some common Association rule learning algorithms include:

1. Apriori Algorithm

2. Eclat Algorithm

3. FP-Growth Algorithm

FEATURE EXTRACTION AND DIMENSIONLAITY REDUCTION

Feature extraction is a technique that reduces the dimensionality or complexity of data

to improve the performance and efficiency of machine learning (ML) algorithms. This process
facilitates ML tasks and improves data analysis by simplifying the dataset to include only its
significant variables or attributes.

An artificial intelligence (AI) model’s performance relies on the quality of its training
data. Machine learning models go through preprocessing to help ensure that data is in a suitable
format for efficient model training and performance. Feature extraction is a crucial part of the
pre-processing workflow.

During the extraction process, unstructured data is converted into a more structured and
usable format to enhance the data quality and model interpretability. Feature extraction is a
subset of feature engineering, the broader process of creating, modifying and selecting features
within raw data to optimize model performance.
How Dimensionality Reduction Works?
 Imagine a dataset where each data point exists in a 3D space defined by axes X, Y and
Z. If most of the data variance occurs along X and Y then the Z-dimension may contribute very
little to understanding the structure of the data.

Fig Dimensionality Reduction

• Before Reduction You can see that Data exist in 3D (X,Y,Z). It has high redundancy
and Z contributes little meaningful information

• On the right after reducing the dimensionality the data is represented in lower-
dimensional spaces. The top plot (X-Y) maintains the meaningful structure while the
bottom plot (Z-Y) shows that the Z-dimension contributed little useful information.

This process makes data analysis more efficient, improving computation speed and
visualization while minimizing redundancy

Dimensionality Reduction Techniques

Dimensionality reduction techniques can be broadly divided into two categories:

1. Feature Selection
Feature selection chooses the most relevant features from the dataset without altering them. It
helps remove redundant or irrelevant features, improving model efficiency. Some common
methods are:

• Filter methods rank the features based on their relevance to the target variable.

• Wrapper methods use the model performance as the criteria for selecting features.
• Embedded methods combine feature selection with the model training process.
2. Feature Extraction

Feature extraction involves creating new features by combining or transforming the original
features. These new features retain most of the dataset’s important information in fewer
dimensions. Common feature extraction methods are:

1. Principal Component Analysis (PCA): Converts correlated variables into

uncorrelated 'principal components, reducing dimensionality while maintaining as
much variance as possible enabling more efficient analysis.
2. Missing Value Ratio: Variables with missing data beyond a set threshold are removed,
improving dataset reliability.
3. Backward Feature Elimination: Starts with all features and removes the least
significant ones in each iteration. The process continues until only the most impactful
features remain, optimizing model performance.

4. Forward Feature Selection: Forward Feature Selection Begins with one feature, adds
others incrementally and keeps those improving model performance.

5. Random Forest: Random forest Uses decision trees to evaluate feature importance,
automatically selecting the most relevant features without the need for manual coding,
enhancing model accuracy.

6. Factor Analysis: Groups variables by correlation and keeps the most relevant ones for
further analysis.

7. Independent Component Analysis (ICA): Identifies statistically independent

components, ideal for applications like ‘blind source separation’ where traditional
correlation-based methods fall short.

Dimensionality Reduction Real World Examples

Dimensionality reduction plays a important role in many real-world applications such as text
categorization, image retrieval, gene expression analysis and more. Here are a few examples:

1. Text Categorization: With vast amounts of online data dimensionality reduction helps
classify text documents into predefined categories by reducing the feature space like
word or phrase features while maintaining accuracy.

2. Image Retrieval: As image data grows indexing based on visual content like color,
texture, shape rather than just text descriptions has become essential. This allows for
better retrieval of images from large databases.

3. Gene Expression Analysis: Dimensionality reduction accelerates gene expression

analysis help to classify samples like leukemia by identifying key features, improve
both speed and accuracy.
 4. Intrusion Detection: In cybersecurity dimensionality reduction helps analyze user
activity patterns to detect suspicious behaviors and intrusions by identifying optimal
features for network monitoring.

Advantages of Dimensionality Reduction

High dimensionality makes models inefficient. Let's now summarize the key advantages of
reducing dimensionality.

• Faster Computation: With fewer features machine learning algorithms can process
data more quickly. This results in faster model training and testing which is particularly
useful when working with large datasets.
• Better Visualization: As we saw in the earlier figure reducing dimensions makes it
easier to visualize data and reveal hidden patterns.

• Prevent Overfitting: With few features models are less likely to memorize the training
data and overfit. This helps the model generalize better to new, unseen data improve its
ability to make accurate predictions.

Disadvantages of Dimensionality Reduction

• Data Loss & Reduced Accuracy: Some important information may be lost during
dimensionality reduction and affect model performance.

• Choosing the Right Components: Deciding how many dimensions to keep is difficult
as keeping too few may lose valuable information while keeping too many can led to
overfitting.

DATA PRE-PROCESSING AND CLEANING TECHNIQUES

Data preprocessing and cleaning are crucial steps in machine learning, involving
transforming raw data into a usable format for algorithms. This process includes handling
missing values, removing outliers, scaling numerical features, and encoding categorical
variables. Effective preprocessing ensures data quality, consistency, and suitability for analysis,
leading to improved model performance.

1. Handling Missing Values:

• Removal: Delete rows or columns with missing data, but this can lead to information
loss.

• Imputation: Replace missing values with statistical measures like mean, median, or
mode, or with more sophisticated methods like K-Nearest Neighbors.
2. Outlier Detection and Removal/Treatment:

• Outlier Detection: Identify extreme values that deviate significantly from the norm
using techniques like z-score, IQR, or visualization methods.

• Outlier Treatment: Decide whether to remove outliers or transform them (e.g., using
winsorizing or capping) based on the context.

Fig Data Preprocessing Techniques

3. Feature Scaling:

• Normalization: Rescale features to a range between 0 and 1, useful for algorithms

sensitive to feature magnitudes.

• Standardization: Transform features to have a mean of 0 and a standard deviation of

1, suitable for algorithms assuming normally distributed data.

4. Categorical Encoding:

• One-Hot Encoding: Create binary columns for each category, representing the
presence or absence of a specific category.

• Label Encoding: Assign a unique numerical value to each category, suitable for
algorithms that can handle ordinal data.
5. Data Transformation:

• Log Transformation: Apply a logarithmic function to reduce skewness and stabilize

variance.

• Box-Cox Transformation: A more general power transformation that can handle

different types of data distributions.

6. Data Reduction:

• Feature Selection: Choose the most relevant features for the model, reducing
dimensionality and complexity.

• Dimensionality Reduction: Use techniques like PCA to reduce the number of features
while retaining important information.

7. Data Integration:
• Merging Data: Combine data from multiple sources into a unified dataset.

• Data Cleansing: Correct errors, inconsistencies, and remove duplicates.

8. Data Discretization:

• Binning: Group continuous data into intervals or bins, often for visualization or to
simplify analysis.

By applying these techniques, data preprocessing and cleaning ensures that the data is of
high quality, consistent, and suitable for machine learning algorithms, ultimately leading to
better model performance and more reliable results.

EVALUATION AND VALIDATION OF MACHINE LEARNING MODELS

Model evaluation refers to the process of assessing a machine learning model’s

performance and reliability using specific metrics. It involves testing the model on unseen data
to ensure its predictions are accurate and meaningful. This step is crucial to determine how
well the model can generalize beyond the training dataset. Model evaluation often uses
techniques like cross-validation, hold-out validation, and metrics like accuracy, precision,
recall, and F1 score.

While often used interchangeably, model evaluation differs from model validation.
Validation typically occurs during the model training phase to fine-tune parameters, whereas
evaluation happens after training to measure final performance. Together, they ensure the
model is both optimized and trustworthy.

Why is Model Evaluation Important?

Model evaluation is essential for determining the reliability and accuracy of machine
learning models. Without evaluation, models risk overfitting (performing well on training data
but poorly on new data) or underfitting (failing to capture the underlying patterns). Reliable
evaluation methods help identify and address these issues.
Another critical aspect is generalizability—ensuring the model performs consistently
on unseen datasets. This is especially vital for applications like medical diagnostics or financial
predictions, where errors can have significant consequences. Proper evaluation builds trust in
the model and lays the foundation for its successful deployment in real-world scenarios.

Key Evaluation Techniques

Model evaluation relies on several techniques to measure performance and ensure

reliability. These techniques are designed to balance bias, variance, and computational
efficiency, helping to identify how well a model generalizes to unseen data.

Train-Test Split

The train-test split method divides the dataset into two subsets: training data to build
the model and test data to evaluate it. Typically, the split ratio is 80:20 or 70:30. While
straightforward and computationally efficient, this method has limitations, such as high
variance in results when data is limited. Moreover, a single split may not capture the full range
of variability in the dataset, potentially leading to biased evaluations.
Cross-Validation

Cross-validation improves upon the train-test split by ensuring more robust evaluation.
The most common type is k-fold cross-validation, where the dataset is divided into k subsets
(folds). Each fold serves as a testing set once, while the remaining folds are used for training.
This process repeats k times, and the results are averaged for a comprehensive performance
measure. Stratified k-fold cross-validation ensures that each fold maintains the same class
distribution, making it particularly useful for imbalanced datasets. Cross-validation reduces
variance and provides a more reliable estimate of a model’s generalizability.
Holdout Validation

Holdout validation involves setting aside a portion of the dataset as a final validation
set after training and testing. This method is particularly useful for models undergoing
hyperparameter tuning, as it helps avoid overfitting to the training and test sets. However, it
requires a sufficiently large dataset to allocate separate subsets for training, testing, and
validation.

Each of these techniques offers unique benefits and trade-offs, and the choice depends
on the size of the dataset, the complexity of the model, and the desired evaluation rigor.
EVALUATION METRICS FOR CLASSIFICATION MODELS

Evaluating the performance of classification models requires specific metrics to provide

a holistic view of their accuracy and reliability. These metrics are crucial for understanding
how well the model predicts outcomes, particularly in various real-world applications.

Accuracy

Accuracy measures the proportion of correct predictions out of all predictions made by the
model. It is calculated as:

Limitations: Accuracy can be misleading when dealing with imbalanced datasets. For
instance, in a fraud detection scenario where only 1% of transactions are fraudulent, a model
predicting all transactions as non-fraudulent would achieve 99% accuracy but fail to detect any
fraud.

Precision and Recall

• Precision: The ratio of true positives to all positive predictions. It emphasizes

minimizing false positives, making it crucial in scenarios like spam detection, where a
false positive could mean labeling an important email as spam.

• Recall (Sensitivity): The ratio of true positives to all actual positives. It is critical in
cases like medical diagnosis, where missing a positive case (e.g., a disease) could have
severe consequences.

When to prioritize: Use precision when the cost of false positives is high (e.g., spam
detection). Prioritize recall when false negatives have greater consequences (e.g., identifying
diseases).

F1 Score

The F1 score is the harmonic mean of precision and recall, offering a balanced metric,
especially when class distributions are uneven:

It is particularly useful in situations where both false positives and false negatives need to be
minimized, such as fraud detection.
Confusion Matrix

A confusion matrix provides a detailed breakdown of predictions:

• True Positives (TP): Correctly predicted positives.

• True Negatives (TN): Correctly predicted negatives.

• False Positives (FP): Incorrectly predicted positives.

• False Negatives (FN): Incorrectly predicted negatives.

This matrix helps in calculating all other metrics and offers a clear visualization of model
performance.

A hospital uses an AI based system to detect whether a patient has a

rare disease. Out of 1000 patients tested, 100 actually have the disease. The
model’s confusion matrix is as follows:

Predicted Predicted
Positive Negative
Actual Positive 80 20
Actual 50 850
Negative

Predicted Positive Predicted Negative

Actual Positive TP = 80 FN = 20
Actual Negative FP = 50 TN = 850

Evaluation Metrics

• Accuracy = (TP + TN) / Total

= (80 + 850) / 1000
= 0.93 or 93%
• Precision = TP / (TP + FP)
= 80 / (80 + 50)
= 0.615 or 61.5%
• Recall (Sensitivity) = TP / (TP + FN)
= 80 / (80 + 20)
= 0.80 or 80%
• F1-Score = 2 × (Precision × Recall) / (Precision + Recall)
= 2 × (0.615 × 0.8) / (0.615 + 0.8)
= 0.696 or 69.6%
AUC-ROC Curve

The AUC-ROC (Area Under the Receiver Operating Characteristic Curve) evaluates a
model’s ability to distinguish between classes. The ROC curve plots the true positive rate
(recall) against the false positive rate at various thresholds.

• AUC (Area Under Curve): A higher AUC indicates better model performance. An
AUC of 1.0 represents a perfect model, while 0.5 suggests random guessing.

Importance: AUC-ROC is critical for binary classification problems where balancing

sensitivity and specificity is key, such as credit risk assessment.

Evaluation Metrics for Regression Models

Evaluating regression models involves measuring how accurately a model predicts

continuous outcomes. Different metrics address specific aspects of error distribution and
magnitude, helping refine model performance for various applications.

Mean Absolute Error (MAE)

The Mean Absolute Error (MAE) is a straightforward metric that calculates the average
absolute difference between predicted and actual values. It is expressed as:

Mean Squared Error (MSE)

The Mean Squared Error (MSE) computes the average of squared differences between
predicted and actual values:

Root Mean Squared Error (RMSE)

Root Mean Squared Error (RMSE) is the square root of MSE, which brings the error back to
the same units as the target variable:
Mean Absolute Percentage Error (MAPE)

MAPE measures errors as a percentage of the actual values, making it intuitive for comparing
model performance across datasets of different scales:

OVERFITTING AND UNDERFITTING

In machine learning, the goal is to build models that can generalize well to new, unseen
data, not just perform well on the data they were trained on. However, machine learning models
can encounter two common pitfalls: overfitting and underfitting, both of which hinder a model's
ability to make accurate predictions.

1. Overfitting

Overfitting occurs when a machine learning model learns the training data too well,
including the noise and irrelevant details. This can result in poor performance on new data.

Causes of overfitting
Overfitting can be caused by complex models, insufficient or noisy training data, or
excessive training time.
How to detect overfitting

Overfitting can be detected by observing a significant difference in performance

between training and unseen data, analyzing learning curves, or using cross-validation.
Preventing overfitting

To prevent overfitting, you can increase data size, simplify the model, apply
regularization, use early stopping, utilize cross-validation, or perform feature selection.

2. Underfitting

Underfitting occurs when a model is too simple to capture patterns in the training data,
leading to poor performance on both the training and test sets. This means the model fails to
learn relationships between features and labels.

Causes of underfitting

Underfitting can result from a simplistic model, insufficient features, inadequate

training time, or excessive regularization.
How to detect underfitting

Underfitting is indicated by poor performance on both training and test data, learning
curves showing consistently high error rates, or by comparing performance to a more complex
model.

Achieving a good fit in a machine learning model involves finding the right balance
between underfitting and overfitting. This balance ensures the model captures patterns without
being overly sensitive to noise, making it capable of generalizing effectively to new data.
To intuitively grasp the concepts of overfitting and underfitting, consider the behaviors
of three different students A, B, and C each with distinct learning styles and test performances.

Fig Illustration for Underfit and Overfit

Student A: Disengaged Learner — Underfitting

Student A shows minimal interest in the learning process. She is distracted during lectures,
does not make an effort to understand or memorize the content, and displays a general lack of
academic engagement. Unsurprisingly, her scores are consistently poor:

• Class Test Score: ~50%

• Final Test Score: ~47%

This reflects a classic case of underfitting. In machine learning, underfitting occurs when
a model is too simple to capture the underlying patterns in the data. Just like Student A, the
model fails to perform well both during training and testing because it has not effectively
“learned” the data.

Student B: Rote Learner — Overfitting

Student B memorizes lessons thoroughly, focusing on repeating what was taught without
developing a deeper understanding of the concepts. His performance in class assessments is
excellent, but he struggles in the final test, which likely includes unfamiliar problems requiring
conceptual application:

• Class Test Score: ~98%

• Final Test Score: ~69%

This scenario mirrors overfitting in machine learning. An overfit model performs very well
on the training data but fails to generalize to new, unseen data. Student B, like an overfit model,
performs perfectly in familiar environments (class tests) but falters when confronted with
variation.

Student C: Conceptual Learner — Good Fit

Student C exemplifies effective learning. She focuses on conceptual understanding rather

than surface-level memorization. This enables her to apply knowledge flexibly across different
contexts. Her performance is consistent:
• Class Test Score: ~92%
• Final Test Score: ~89%

This behavior reflects a well-generalized model in machine learning — one that balances
bias and variance and performs reliably on both training and testing data. Student C’s learning
strategy leads to true comprehension, much like a model that has learned the real structure of
the data.

Key Takeaways

Student Learning Style Class Test Final Test ML Analogy

A Not interested ~50% ~47% Underfitting

B Memorization only ~98% ~69% Overfitting

C Conceptual understanding ~92% ~89% Good generalization

This analogy helps demystify overfitting and underfitting by grounding them in

everyday learning experiences. Just as students benefit from meaningful, concept-based study
strategies, machine learning models must also strike a balance , learning enough to generalize,
but not so much that they simply memorize the training data.