Introduction to Machine Learning

Machine learning (ML) allows computers to learn and make decisions without being
explicitly programmed. It involves feeding data into algorithms to identify patterns and
make predictions on new data. It is used in various applications like image recognition,
speech processing, language translation, recommender systems, etc. In this article, we
will see more about ML and its core concepts.
Machine Learning Foundations
Foundations
Logic and reasoning: Ancient Greek philosophers like Aristotle introduced concepts of
logical reasoning that influenced the development of AI and machine learning.
Neural networks: In 1943, Walter Pitts and Warren McCulloch published the first
mathematical model of a neural network, providing a framework for how artificial
neurons could mimic human thought.
Biological inspiration: Donald Hebb's 1949 book, "The Organization of Behavior,"
proposed a theory of how brain cells communicate, which became a foundational concept
for modifying the relationships between nodes in artificial neural networks.
Probability and statistics: Mathematical concepts like Bayes' Theorem (1812) and Least
Squares were developed in the 18th and 19th centuries, providing the statistical and
mathematical underpinnings for machine learning algorithms.

Machine Learning Life Cycle?

Problem Definition
Data Preparation
Model Development
Model Deployment
Monitoring and Maintenance
Problem Definition
First step in the machine learning life cycle
Helps decide what problem you want to solve
Defines the output, scope, and objective of the task
Lays the foundation for building the model
Problem definition must be clear and concise
Data Preparation
Data preparation means getting data ready for the model. It involves three simple steps:

Collect the data

Clean and pre-process the data

Select the best features to train the model

It also includes exploring the data to understand patterns and creating new features from
existing ones.
Data Collection
Data Collection in ML

After defining the problem, the next step is collecting data from various sources to train
the model.

Key things to consider:

Relevance - Data must relate to the problem statement

Quality & Quantity - Better data = better model performance

Variety - Diverse data helps the model recognize more patterns

Sources of Data:

Primary Data - Collected specifically for the problem (e.g., surveys)

Secondary Data - Already existing data (e.g., databases, Kaggle)

Data Pre-processing

Data pre-processing, also known as data wrangling, is the process of cleaning raw data to
improve the accuracy and performance of the machine learning model. Since raw data is
often messy and unstructured, it is important to fix issues like missing values, duplicate
data, invalid data, and noise before using it for analysis. This step makes the data more
clean, consumable, and useful for building a better model.

Analyzing Data

After cleaning the data, the next step is to visualize and explore it. Tools like Power BI
and Tableau are used to find patterns and trends. This helps in making better choices for
feature engineering and model selection.

Feature Engineering and Selection

A feature is a measurable quantity used to train the model. Feature Engineering is

creating or improving features to better understand the data. Feature Selection is picking
the most relevant features for the problem. Both processes help in reducing dataset size
making the model training easier and efficient.

Model Development

Model Selection

Choosing the right model depends on the data characteristics, problem complexity and
desired outcomes. This step directly affects the performance and results of the model.

Model Training

The algorithm is fed with pre-processed data to learn patterns and relationships.
Consistently adjusting parameters improves prediction rate and accuracy, making the
model reliable in real-world scenarios.
Model Evaluation

The model is evaluated using metrics like accuracy, precision, recall and F1 score. If
performance is not satisfactory, hyper-parameters are tuned to improve accuracy. If still
not satisfied, we go back to model selection and retrain the model.

Model Deployment

The trained model is integrated into real-world systems and made available to users.
Before deploying, two things are checked:

Portable — Can be transferred from one machine to another

Scalable — Performance is maintained without redesigning

Monitor and Maintenance

The model is continuously monitored for issues and performance. If an issue is found, the
model is retrained with new data or architecture is modified. If the issue cannot be solved,
it becomes a new problem statement and the ML life cycle starts again.

Applications
Healthcare Diagnostics
Fraud Detection in Finance
Product Recommendations
Speech and Language Processing
Autonomous Vehicles

Overview of ML

Why do we need Machine Learning?

Traditional programming cannot handle complex tasks or large data efficiently. ML

solves this by learning from examples and making predictions without fixed rules.
Reasons why ML is Important

Solving Complex Problems ML learns from data and predicts outcomes for complex tasks
like image recognition, speech recognition and language translation.

Handling Large Data ML quickly processes huge amounts of data and provides valuable
insights and real-time predictions like fraud detection and personalized recommendations.

Automate Repetitive Tasks ML automates time-consuming tasks with high accuracy,

reducing manual work and errors like spam filtering and chat-bots.

Personalized User Experience ML analyzes user behavior to deliver tailored

recommendations like Netflix suggesting movies and e-commerce recommending
products.

Self-Improvement ML models get smarter over time with more data like voice assistants
learning accents and self-driving cars improving decisions.

What Makes a Machine Learn?

A machine learns by finding patterns in data and improving without being explicitly
programmed. Here is how:

Data Input - Needs quality data like text, images or numbers

Algorithms - Mathematical methods to find patterns

Model Training - Adjusts settings to better predict outcomes

Feedback Loop - Corrects errors by comparing predictions with actual results

Iteration - Repeats training to refine predictions

Evaluation - Tested on unseen data for real-world performance

Importance of Data in ML

Data is the foundation of ML without which models cannot learn or perform.

Provides examples for models to learn patterns

High quality data improves model performance

Helps models understand real-world scenarios

Drives continuous improvement through feedback loops

Design of a Learning System

ML system design follows an iterative lifecycle with core phases to ensure the system is
reliable, scalable and delivers real-world value

Problem Definition

Define what the system needs to do (spam detection, price prediction etc.)

Set performance metrics like accuracy, precision, F1-score

Identify constraints like resources, latency and ethical considerations

Data Processing Pipeline

Collect diverse and relevant data from various sources

Clean the data by handling missing values, duplicates and outliers

Feature Engineering - create and select most relevant features

Model Development

Model Selection - choose algorithm based on problem type

Training - feed data to model and minimize error

Evaluation - test on unseen data to avoid over-fitting

Hyper-parameter Tuning - optimize model for better performance

Model Deployment

Integrate the model into applications via APIs

Serving decide between batch processing or real-time predictions

Monitoring and Maintenance

Monitor accuracy, latency and resource usage

Detect Drift - changes in data or relationships

Retrain with new data to keep model accurate

Applications

Healthcare - disease detection and medical image analysis

Finance - fraud detection and suspicious pattern recognition

E-commerce - personalized product recommendations

NLP - translation, virtual assistants and sentiment analysis

Autonomous Vehicles - real-time decision making for self-driving cars

Limitations

Data Dependency - needs high quality and unbiased data

Computational Cost - requires significant processing power

Black Box Effect - deep learning models lack transparency

Risk of Bias - can amplify biases from training data

Maintenance - needs regular monitoring and retraining

Types of Machine Learning

Supervised Learning
Unsupervised Learning
Semi-supervised Learning
Reinforcement Learning

Supervised Learning

Supervised learning algorithms or methods are the most commonly used ML algorithms.
This method or learning algorithm takes the data sample i.e. the training data and its
associated output i.e. labels or responses with each data sample during the training
process.
The main objective of supervised learning algorithms is to learn an association between
input data samples and corresponding outputs after performing multiple training data
instances.
For example, we have
x: Input variables and
Y: Output variable
Now, apply an algorithm to learn the mapping function from the input to output as
follows −
Y=f(x)
Now, the main objective would be to approximate the mapping function so well that even
when we have new input data (x), we can easily predict the output variable (Y) for that
new input data.
It is called supervised because the whole process of learning can be thought as it is being
supervised by a teacher or supervisor. Examples of supervised machine learning
algorithms includes Decision tree, Random Forest, KNN, Logistic Regression etc.
Based on the ML tasks, supervised learning algorithms can be divided into the following
two broad classes −
Classification
Regression

Classification
The key objective of classification-based tasks is to predict categorial output labels or
responses for the given input data. The output will be based on what the model has
learned in the training phase. As we know the categorial output responses means
unordered and discrete values, hence each output response will belong to a specific class
or category. We will discuss Classification and associated algorithms in detail in the
upcoming chapters also.
Classification Models
Followings are some common classification models −
Logistic Regression
Decision Trees
Random Forest
K-nearest Neighbour
Support Vector Machine
Naive Bayes
Linear Discriminant Analysis
Neural Networks

Regression
The key objective of regression-based tasks is to predict output labels or responses, which
are continuous numeric values, for the given input data. The output will be based on what
the model has learned in its training phase. Basically, regression models use the input data
features (independent variables) and their corresponding continuous numeric output
values (dependent or outcome variables) to learn specific associations between inputs and
corresponding outputs. We will discuss regression and associated algorithms in detail in
further chapters.

Regression Models
Followings are some common regression models −
Linear Regression
Ridge regression
Decision Trees
Random Forest
K-nearest Neighbour
Neural Network Regression

Unsupervised Learning

As the name suggests, unsupervised learning is opposite to supervised ML methods or

algorithms in which we do not have any supervisor to provide any sort of guidance.
Unsupervised learning algorithms are handy in the scenario in which we do not have the
liberty, like in supervised learning algorithms, of having pre-labelled training data and we
want to extract useful pattern from input data.
For example, it can be understood as follows
Suppose we have − x: Input variables, then there would be no corresponding output
variable and the algorithms need to discover the interesting pattern in data for learning.
Examples of unsupervised machine learning algorithms includes K-means clustering, K-
nearest neighbours etc.
Based on the ML tasks, unsupervised learning algorithms can be divided into the
following broad classes −
Clustering
Association
Dimensionality Reduction

Clustering
Clustering methods are one of the most useful unsupervised ML methods. These
algorithms used to find similarity as well as relationship patterns among data samples and
then cluster those samples into groups having similarity based on features. The real-world
example of clustering is to group the customers by their purchasing behavior.

Clustering Models
Followings are some common clustering models −
K-Means Clustering
Hierarchical Clustering
Mean-shift Clustering
DBSCAN Clustering
HDBSCAN Clustering
BIRCH Clustering
Affinity Propagation
Agglomerative Clustering

Association
Another useful unsupervised ML method is Association which is used to analyze large
dataset to find patterns which further represents the interesting relationships between
various items. It is also termed as Association Rule Mining or Market basket
analysis which is mainly used to analyze customer shopping patterns.
Association Models
Followings are some common association models −
Apriori Algorithm
Eclat algorithm
FP-growth algorithm
Dimensionality Reduction
This unsupervised ML method is used to reduce the number of feature variables for each
data sample by selecting set of principal or representative features. A question arises here
is that why we need to reduce the dimensionality? The reason behind is the problem of
feature space complexity which arises when we start analyzing and extracting millions of
features from data samples. This problem generally refers to curse of dimensionality.
PCA (Principal Component Analysis), K-nearest neighbors and discriminant analysis are
some of the popular algorithms for this purpose.
Dimensionality Reduction Models
Followings are some common dimensionality Reduction models −
Principal Component Analysis(PCA)
Autoencoders
Singular value decomposition (SVD)
Anomaly Detection
This unsupervised ML method is used to find out the occurrences of rare events or
observations that generally do not occur. By using the learned knowledge, anomaly
detection methods would be able to differentiate between anomalous or a normal data
point. Some of the unsupervised algorithms like clustering, KNN can detect anomalies
based on the data and its features.

Semi-supervised Learning
Semi-supervised learning algorithms or methods are neither fully supervised nor fully
unsupervised. They basically fall between the two i.e. supervised and unsupervised
learning methods. These kinds of algorithms generally use small supervised learning
component i.e. small amount of pre-labeled annotated data and large unsupervised
learning component i.e. lots of unlabeled data for training. We can follow any of the
following approaches for implementing semi-supervised learning methods −
The first and simple approach is to build the supervised model based on small amount of
labeled and annotated data and then build the unsupervised model by applying the same
to the large amounts of unlabeled data to get more labeled samples. Now, train the model
on them and repeat the process.
The second approach needs some extra efforts. In this approach, we can first use the
unsupervised methods to cluster similar data samples, annotate these groups and then use
a combination of this information to train the model.
Reinforcement Learning

Reinforcement learning methods are different from previously studied methods and very
rarely used also. In this kind of learning algorithms, there would be an agent that we want
to train over a period of time so that it can interact with a specific environment. The agent
will follow a set of strategies for interacting with the environment and then after
observing the environment it will take actions regards the current state of the
environment. The following are the main steps of reinforcement learning methods −
Step 1 − First, we need to prepare an agent with some initial set of strategies.
Step 2 − Then observe the environment and its current state.
Step 3 − Next, select the optimal policy regards the current state of the environment and
perform important action.
Step 4 − Now, the agent can get corresponding reward or penalty as per accordance with
the action taken by it in previous step.
Step 5 − Now, we can update the strategies if it is required so.
Step 6 − At last, repeat steps 2-5 until the agent got to learn and adopt the optimal
policies.

Reinforcement Learning Models

Following are some common reinforcement learning algorithms −
Q-learning
Markov Decision Process (MDP)
SARSA
DQN
DDPG

Supervised Machine Learning

What is Supervised Machine Learning?
Supervised learning, also known as supervised machine learning, is a type of machine
learning that trains the model using labeled datasets to predict outcomes. A Labeled
dataset is one that consists of input data (features) along with corresponding output data
(targets).
The main objective of supervised learning algorithms is to learn an association between
input data samples and corresponding outputs after performing multiple training data
instances.
How does Supervised Learning Work?
In supervised machine learning, models are trained using a dataset that consists of input-
output pairs.
The supervised learning algorithm analyzes the dataset and learns the relation between the
input data (features) and correct output (labels/ targets). In the process of training, the
model estimates the algorithm's parameters by minimizing a loss function. The loss
function measures the difference between the model's predictions and actual target values.
The model iteratively updates its parameters until the loss/ error has been sufficiently
minimized.
Once the training is completed, the model parameters have optimal values. The model has
learned the optimal mapping/ relation between the inputs and targets. Now, the model can
predict values for the new and unseen input data.
Types of Supervised Learning Algorithm
Supervised machine learning is categorized into two types of problems − classification
and regression.
Classification
The key objective of classification-based tasks is to predict categorical output labels or
responses for the given input data such as true-false, male-female, yes-no etc. As we
know, the categorical output responses mean unordered and discrete values; hence, each
output response will belong to a specific class or category.
Some popular classification algorithms are decision trees, random forests, support vector
machines (SVM), logistic regression, etc.
Regression
The key objective of regression-based tasks is to predict output labels or responses, which
are continuous numeric values, for the given input data. Basically, regression models use
the input data features (independent variables) and their corresponding continuous
numeric output values (dependent or outcome variables) to learn specific associations
between inputs and corresponding outputs.
Some popular regression algorithms are linear regression, polynomial regression, Laso
regression, etc.
Algorithms for Supervised Learning
Supervised learning is one of the important models of learning involved in training
machines. This chapter talks in detail about the same.
There are several algorithms available for supervised learning. Some of the widely used
algorithms of supervised learning are as shown below −
Linear Regression
k-Nearest Neighbors
Decision Trees
Naive Bayes
Logistic Regression
Support Vector Machines
Random Forest
Gradient Boosting
Let's discuss each of the above mentioned supervised machine learning algorithms in
detail.

Linear Regression
Linear regression is a type of algorithm that tries to find the linear relation between input
features and output values for the prediction of future events. This algorithm is widely
used to perform stock analysis, weather forecasting and others.
K-Nearest Neighbors
The k-Nearest Neighbors (kNN) is a statistical technique that can be used for solving
classification and regression problems. This algorithm classifies or predicts values for
new data by mathematically calculating the nearest distance with other points in training
data.
Let us discuss the case of classifying an unknown object using kNN. Consider the
distribution of objects as shown in the image given below −
The diagram shows three types of objects, marked in red, blue and green colors. When
you run the kNN classifier on the above dataset, the boundaries for each type of object
will be marked as shown below −

Now, consider a new unknown object you want to classify as red, green or blue. This is
depicted in the figure below.
As you see it visually, the unknown data point belongs to a class of blue objects.
Mathematically, this can be concluded by measuring the distance of this unknown point
with every other point in the data set. When you do so, you will know that most of its
neighbors are blue in color. The average distance between red and green objects would
definitely be more than the average distance between blue objects. Thus, this unknown
object can be classified as belonging to blue class.
The kNN algorithm can also be used for regression problems. The kNN algorithm is
available as ready-to-use in most of the ML libraries.
Decision Trees
A Decision tree is a tree-like structure used to make decisions and analyze the possible
consequences. The algorithm splits the data into subsets based on features, where each
parent node represents internal decisions and the leaf node represents final prediction.
A simple decision tree in a flowchart format is shown below −
You would write a code to classify your input data based on this flowchart. The flowchart
is self-explanatory and trivial. In this scenario, you are trying to classify an incoming
email to decide when to read it.
In reality, the decision trees can be large and complex. There are several algorithms
available to create and traverse these trees. As a Machine Learning enthusiast, you need
to understand and master these techniques of creating and traversing decision trees.
Naive Bayes
Naive Bayes is used for creating classifiers. Suppose you want to sort out (classify) fruits
of different kinds from a fruit basket. You may use features such as color, size, and shape
of fruit; for example, any fruit that is red in color, round in shape, and about 10 cm in
diameter may be considered an Apple. So to train the model, you would use these features
and test the probability that a given feature matches the desired constraints. The
probabilities of different features are then combined to arrive at the probability that a
given fruit is an Apple. Naive Bayes generally requires a small number of training data
for classification.
Logistic Regression
Logistic regression is a type of statistical algorithm that estimates the probability of
occurrence of an event.
Look at the following diagram. It shows the distribution of data points in the XY plane.
From the diagram, we can visually inspect the separation of red and green dots. You may
draw a boundary line to separate out these dots. Now, to classify a new data point, you
will just need to determine on which side of the line the point lies.
Support Vector Machines
Support Vector Machines (SVM) algorithm can be typically used for both classification
and regression. For classification tasks, the algorithm creates a hyperplane to separate
data into classes. While for regression, the algorithm tries to fit a regression line with
minimal error.
Look at the following distribution of data. Here the three classes of data cannot be
linearly separated. The boundary curves are non-linear. In such a case, finding the curve's
equation becomes a complex job.
The Support Vector Machines (SVM) come in handy in determining the separation
boundaries in such situations.
Random Forest
Random forest is also a supervised learning algorithm that is flexible for classification
and regression. This algorithm is a combination of multiple decision trees which are
merged to improve the accuracy of prediction .
The following diagram illustrates how the Random Forest Algorithm works −

Gradient Boosting
Gradient boosting combines weak learners(decision trees), to create a strong model. It
builds new models that correct errors of the previous ones. The goal of this algorithm is to
minimize the loss function. It can be efficiently used for classification and regression
tasks.
Advantages of Supervised Learning
Supervised learning algorithms are one of the most popular among the machine learning
models. Some benefits are:-
The goal in supervised learning is well-defined, which improves the prediction accuracy.
Models trained using supervised learning are effective at predicting and classification
since they use labeled datasets.
It can be highly versatile, i.e., applied to various problems, like spam detection, stock
prices, etc.
Disadvantages of Supervised Learning
Though supervised learning is the most used, it comes with certain challenges too. Some
of them are:
Supervised learning requires a large amount of labeled data for the model to train
effectively. It is practically very difficult to collect such huge data; it is expensive and
time-consuming.
Supervised learning cannot predict accurately if the test data is different from the training
data.
Accurately labeling the data is complex and requires expertise and effort.
Applications of Supervised learning
Supervised learning models are widely used in many applications in various sectors,
including the following-
Image recognition − A model is trained on a labeled dataset of images, where each
image is associated with a label. The model is fed with data, which allows it to learn
patterns and features. Once trained, the model can now be tested using new, unseen data.
This is widely used in applications like facial recognition and object detection.
Predictive analytics − Supervised learning algorithms are used to train labeled historical
data, allowing the model to learn patterns and relations between input features and output
to identify trends and make accurate predictions. Businesses use this method to make
data-driven decisions and enhance strategic planning.
What is Unsupervised Machine Learning?
Unsupervised learning, also known as unsupervised machine learning, is a type of
machine learning that learns patterns and structures within the data without human
supervision. Unsupervised learning uses machine learning algorithms to analyze the data
and discover underlying patterns within unlabeled data sets.
Unlike supervised machine learning, unsupervised machine learning models are trained
on unlabeled dataset. Unsupervised learning algorithms are handy in scenarios in which
we do not have the liberty, like in supervised learning algorithms, of having pre-labeled
training data and we want to extract useful patterns from input data.
We can summarize unsupervised learning as −
a machine learning approach or type that
uses machine learning algorithms
to find hidden patterns or structures
within the data without human supervision.
There are many approaches that are used in unsupervised machine learning. Some of the
approaches are association, clustering, and dimensionality reduction. Some examples of
unsupervised machine learning algorithms include K-means clustering, K-nearest
neighbors, etc.
In regression, we train the machine to predict a future value. In classification, we train the
machine to classify an unknown object in one of the categories we define. In short, we
have been training machines so that it can predict Y for our data X. Given a huge data set
and not estimating the categories, it would be difficult for us to train the machine using
supervised learning. What if the machine can look up and analyze the big data running
into several Gigabytes and Terabytes and tell us that this data contains so many distinct
categories?
As an example, consider the voters data. By considering some inputs from each voter
(these are called features in AI terminology), let the machine predict that there are so
many voters who would vote for X political party and so many would vote for Y, and so
on. Thus, in general, we are asking the machine given a huge set of data points X, What
can you tell me about X?. Or it may be a question like What are the five best groups we
can make out of X?. Or it could be even like What three features occur together most
frequently in X?.
This is exactly what Unsupervised Learning is all about.
How does Unsupervised Learning Work?
In unsupervised learning, machine learning algorithms (called self-learning algorithms)
are trained on unlabeled data sets i.e, the input data is not categorized. Based on the tasks,
or machine learning problems such as clustering, associations, etc. and the data sets, the
suitable algorithms are chosen for the training.
In the training process, the algorthims learn and infer their own rules on the basis of the
similarities, patterns and differences of data points. The algorithms learn without any
labels (target values) or pre-training.
The outcome of this training process of algorithm with data sets is a machine learning
model. As the data sets are unlabeled (no target values, no human supervision), the model
is unsupervised machine learning model.
Now the model is ready to perform the unsupervised learning tasks such as clustering,
association, or dimensionality reduction.
Unsupervised learning models is suitable complex tasks, like organizing large datasets
into clusters.
Unsupervised Machine Learning Methods
Unsupervised learning methods or approaches are broadly categorized into three
categories − clustering, association, and dimensionality reduction. Let us discuss these
methods briefly and list some related algorithms −
Clustering
Clustering is a technique used to group a set of objects or data points into clusters based
on their similarities. The goal of this technique is to make sure that the data points within
the same cluster should have more similarities than those in other clusters.
Clustering is sometimes called unsupervised classification because it produces the same
result as classification does but without having predefined classes.
Clustering is one of the popular unsupervised learning approaches. There are several
unsupervised learning algorithms used for clustering like −
K-Means Clustering − This algorithm is used to assign data points to one among the K
clusters based on its distance from the center of the cluster. After assigning each data
point to a cluster, new centroids are recalculated. This is an iterative process until the
centroids no longer change. This shows that the algorithm is efficient and the clusters are
stable.
Mean Shift Algorithm − It is a clustering technique that identifies clusters by finding high
data density areas. It is an iterative process, where mean of each data point is shifted
towards the densest area of the data.
Gaussian Mixture Models − It is a probabilistic model that is a combination of multiple
Gaussian distributions. These models are used to determine which determination a given
data belongs to.
Association Rule Mining
This is rule based technique that is used to discover associations between parameters in
large dataset. It is popularly used for Market Basket Analysis, allows companies to make
decisions and recommendation engines. One of the main algorithms that is used for
Association Rule Mining is the Apriori algorithm.
Apriori Algorithm
Apriori algorithm is a technique used in unsupervised learning to identify data points that
are frequently repeated and discover association rules within transactional data.
 Dimensionality Reduction
As the name suggests, dimensionality reduction is used to reduce the number of feature
variables for each data sample by selecting set of principal or representative features.
A question arises here is that, why we need to reduce the dimensionality? The reason
behind this is the problem of feature space complexity which arises when we start
analyzing and extracting millions of features from data samples. This problem generally
refers to "curse of dimensionality". Some popular algorithms in unsupervised learning
that are used for dimensionality reduction are −
Principle Component Analysis
Missing Value Ratio
Singular Value Decomposition
Autoencoders
Algorithms for Unsupervised Learning
Algorithms are very important part in machine learning model training. A machine
learning algorithm is a set of instructions that a program follows to analyze the data and
produce the outcomes. For specific tasks, suitable machine learning algorithms are
selected and trained on the data.
Algorithms used in unsupervised learning generally fall under one of the three categories
− clustering, association, or dimensionality reduction. The following are the most used
unsupervised learning algorithms −
K-Means Clustering
Hierarchical Clustering
Mean-shift Clustering
DBSCAN Clustering
HDBSCAN Clustering
BIRCH Clustering
Affinity Propagation
Agglomerative Clustering
Apriori Algorithm
Eclat algorithm
FP-growth algorithm
Principal Component Analysis(PCA)
Autoencoders
Singular value decomposition (SVD)
Advantages of Unsupervised Learning
Unsupervised learning has many advantages that make it particularly purposeful in
various tasks −
No labeled data required − Unsupervised learning doesn't require a labeled dataset for
training, which makes it easier and cheaper to use.
Discovers hidden patterns − It helps in recognizing patterns and relationships in large
data, which can lead to gaining insights and efficient decision-making.
Suitable for complex tasks − It is efficiently used for various complex tasks like
clustering, anomaly detection, and dimensionality reduction.
Disadvantages of Unsupervised Learning
While unsupervised learning has many advantages, some challenges can occur too while
training the model without human intervention. Some of the disadvantages of
unsupervised learning are:
Difficult to evaluate − Without labeled data and predefined targets, it would be difficult
to evaluate the performance of unsupervised learning algorithms.
Inaccurate outcomes − The outcome of an unsupervised learning algorithm might be
less accurate, especially if the input data has noise and also since the data is not labeled,
the algorithms do not know the exact output.
Applications of Unsupervised Learning
Unsupervised learning provides a path for businesses to identify patterns in large volumes
of data. Some real-world applications of unsupervised learning are:
Customer Segmentation − In business and retail analysis, unsupervised learning is used
to group customers into segments based on their purchases, past activity, or preferences.
Anomaly Detection − Unsupervised learning algorithms are used in anomaly detection to
identify unusual patterns, which is crucial for fraud detection in financial transactions and
network security.
Recommendation Engines − Unsupervised learning algorithms help to analyze large
customer data to gain valuable insights and understand patterns. This can help in target
marketing and personalization.
Natural Language Processing− Unsupervised learning algorithms are used for various
applications. For example, google used to categorize articles in the news section.
What is Anomaly Detection?
This unsupervised ML method is used to find out occurrences of rare events or
observations that generally do not occur. By using the learned knowledge, anomaly
detection methods would be able to differentiate between anomalous or normal data
points.
Some of the unsupervised algorithms, like clustering and KNN, can detect anomalies
based on the data and its features.
Supervised Vs. Unsupervised Learning
Supervised learning algorithms are trained using labeled data. But there might be cases
where data might not be labeled, so how do you gain insights from data that is unlabeled
and messy? Well, to solve these types of cases, unsupervised learning is used.

Machine Learning - Applications

Machine learning has become the ubiquitous technology that has impacted many aspects
of our lives, from business to healthcare to entertainment. Machine learning helps make
decisions and find all possible solutions to a problem which improves the efficiency of
work in every sector.
Some of the successful machine learning applications are chatbots, language translation,
face recognition, recommendation systems, autonomous vehicles, object detection,
medical image analysis, etc. Here are some popular applications of machine learning –

Image and Speech Recognition

Natural Language Processing
Finance Sector
E-commerce and Retail
Automotive Sector
Computer Vision
Manufacturing and Industries
Healthcare Sector
Let us discuss all applications of machine learning in detail −
Image and Speech Recognition
Image and speech recognition are two areas where machine learning has significantly
improved. Machine learning algorithms are used in applications such as facial
recognition, object detection, and speech recognition to accurately identify and classify
images and speech.
Natural Language Processing
Natural Language Processing (NLP) is a field of computer science that deals with the
interaction between computers and humans using natural language. NLP uses machine
learning algorithms to identify parts of speech, sentiment and other aspects of text. It
analyzes, understands, and generates human language. It is currently all over the internet
which includes translation software, search engines, chatbots, grammar correction
software and voice assistants, etc.
Here is a list of some applications of machine learning in natural language processing −
Sentiment Analysis
Speech synthesis
Speech recognition
Text classification
Chatbots
Language translation
Caption generation
Document summarization
Question answering
Autocomplete in search engines
Finance Sector
The role of machine learning in finance is to maintain secure transactions. Also, in
trading, the data is converted to information for the decision-making process. Some
applications of machine learning in the finance sector are –

Fraud Detection
Machine learning is widely used in the finance industry for fraud detection. Fraud
detection is a process of using a machine learning model to monitor transactions and
understand patterns in the dataset to identify fraudulent and suspicious activities.
Machine learning algorithms can analyze vast amounts of transactional data to detect
patterns and anomalies that may indicate fraudulent activity, helping to prevent financial
losses and protect customers.
Algorithmic Trading
Machine learning algorithms are used to identify complex patterns in the large dataset to
discover trading signals which might not be possible for humans.
Some other applications of machine learning in the finance sector are as follows −
Stock market analysis and forecasting
Credit risk assessment and management
Security analysis and portfolio optimization
Asset evaluation and management
E-commerce and Retail
Machine learning is used to enhance the business in e-commerce and retail sector through
recommendation systems and target advertising which improve user experience. Machine
learning makes the process of marketing easy by performing repetitive tasks. Some tasks
where Machine learning is applied are:
Recommendation Systems
Recommendation systems are used to provide personalized recommendations to users
based on their past behavior and preferences and previous interaction with the website.
Machine learning algorithms are used to analyze user data and generate recommendations
for products, services, and content.
Demand Forecasting
Companies use machine learning to understand the future demand for their product or
services based on various factors like market trends, customer behavior and historical
data regarding sales.
Customer Segmentation
Machine learning can be used to segment customers into particular groups with similar
characteristics. The purpose of customer segmentation is to understand customer behavior
and target them with personalized experience.

Automotive Sector
Who would have thought of a car that would move independently without driving?
Machine learning enabled manufacturers to improve the performance of existing products
and vehicles. One massive innovation is the development of autonomous vehicles also
called drive less vehicles which can sense its environment and drive for itself passing the
obstacles without human assistance. It uses machine learning algorithms for continuous
analysis of the surroundings and predicting possible outcomes.
Computer Vision
Computer vision is an application of machine learning that uses algorithms and neural
networks to teach computers to derive meaningful information from digital images and
videos. Computer vision is applied in face recognition, to diagnose diseases based on
MRI scans, and autonomous vehicles.
Object detection and recognition
Image classification and recognition
Faicial recognition
Autonomous vehicles
Object segmentation
Image reconstruction
Manufacturing and Industries
Machine learning is also used in manufacturing and industries to keep a check on the
working conditions of machinery. Predictive Maintenance is used to identify defects in
operational machines and equipment to avoid unexpected outages. This detection of
anomalies would also help with regular maintenance.
Predictive maintenance is a process of using machine learning algorithms to predict
when maintenance will be required on a machine, such as a piece of equipment in a
factory. By analyzing data from sensors and other sources, machine learning algorithms
can detect patterns that indicate when a machine is likely to fail, enabling maintenance to
be performed before the machine breaks down.
Healthcare Sector
Machine learning has also found many applications in the healthcare industry. For
example, machine learning algorithms can be used to analyze medical images and detect
diseases such as cancer or to predict patient outcomes based on their medical history and
other factors.
Some applications of machine learning in healthcare are discussed below −
Medical Imaging and Diagnostics
Machine learning in medical imaging is used to analyze the patterns in the image that
indicate the presence of a particular disease.
 Drug Discovery
Machine learning techniques are used to analyze vast datasets, to predict the biological
activity of compounds, and to identify potential drugs for a disease by analyzing its
chemical structures.
Disease Diagnosis
Machine learning may also be used to identify some types of diseases. Breast cancer,
heart failure, Alzheimer's disease, and pneumonia are some examples of such diseases
that can be identified using machine learning algorithms.
These are just a few examples of the many applications of machine learning. As machine
learning continues to evolve and improve, we can expect to see it used in more areas of
our lives, improving efficiency, accuracy, and convenience in a variety of industries.
Tools overview for machine learning
Machine Learning (ML) has become a cornerstone in the software industry,
revolutionizing everything from predictive analytics to automation. With the growing
demand for intelligent systems, various tools have emerged to assist developers, data
scientists, and organizations build efficient machine-learning models.

TensorFlow
PyTorch
Scikit-learn
Keras
Apache Spark (MLlib)
[Link]
 RapidMiner
Weka
Microsoft Azure Machine Learning
Google Cloud AI Platform
TensorFlow
TensorFlow, developed by Google, is one of the most popular open-source machine
learning frameworks. It provides comprehensive libraries for building and deploying deep
learning models, making it suitable for large-scale machine learning projects.
Key Features:
Supports both high-level and low-level APIs
Flexible architecture for deployment on CPUs, GPUs, and TPUs
TensorFlow Lite for mobile and IoT devices
TensorFlow Extended (TFX) for production ML pipelines
Use Cases:
Image recognition
natural language processing (NLP)
deep neural networks
PyTorch
PyTorch is another widely-used open-source machine learning library, particularly
favored in academic and research circles. Developed by Facebook's AI Research lab, it is
known for its flexibility and ease of use.
Key Features:
Dynamic computational graph (eager execution) for greater flexibility
Strong integration with Python
Extensive support for deep learning models
TorchScript for transitioning models from research to production
Use Cases:
Research in neural networks
Computer Vision
Reinforcement Learning
Scikit-learn
Scikit-learn is a powerful Python library that provides simple and efficient tools for data
mining, machine learning, and data analysis. It is particularly useful for beginners in
machine learning due to its user-friendly interface.
Key Features:
Simple API for a wide variety of algorithms
Extensive documentation and user community
Supports preprocessing, clustering, regression, and classification
Model evaluation metrics and cross-validation techniques
Use Cases:
Predictive analytics
Data Classification
Clustering
Keras
Keras is a high-level deep learning API written in Python, and it is tightly integrated with
TensorFlow. It is designed to be easy to use, modular, and extensible, allowing
developers to quickly prototype machine learning models.
Key Features:
User-friendly API with minimalistic design
Support for multiple backends (TensorFlow, Theano, CNTK)
Ability to run seamlessly on CPUs and GPUs
Built-in support for training and evaluating neural networks
Use Cases:
Neural networks for classification
Regression
Forecasting
Apache Spark (MLlib)
Apache Spark is an open-source distributed computing system that is widely used for big
data processing. Its machine learning library, MLlib, is designed for scalable and fault-
tolerant machine learning.
Key Features:
Distributed and parallelized processing of large datasets
Built-in algorithms for classification, clustering, and regression
Support for various data sources, including HDFS and cloud storage
Integration with Hadoop and Kubernetes
Use Cases:
Big data analytics
Large-scale machine learning
Distributed Computing
[Link]
[Link] is an open-source machine learning platform that offers both an enterprise and a
community version. It is known for its autoML capabilities, making it easier for non-
experts to build robust models.

Key Features:
Automated machine learning (H2O AutoML)
Supports gradient boosting machines, deep learning, and generalized linear models
Scalable for distributed processing
Easy integration with Spark, Python, and R
Use Cases:
Financial services
Healthcare
Marketing analytics
RapidMiner
RapidMiner is a robust data science platform that provides an integrated environment for
machine learning, data preparation, model validation, and deployment. Its drag-and-drop
interface makes it highly accessible to users without coding experience.
Key Features:
No-code and low-code environments for model creation
Extensive library of pre-built machine learning models
Strong capabilities for data preprocessing and visualization
Integration with Python, R, and deep learning frameworks
Use Cases:
Predictive maintenance
Customer analytics
Fraud detection
Weka
Weka, developed by the University of Waikato in New Zealand, is a collection of
machine learning algorithms for data mining tasks. It’s primarily used for educational and
research purposes but is also effective in production systems.
Key Features:
Wide range of algorithms for classification, regression, and clustering
Data preprocessing tools
GUI-based environment for ease of use
Java API for integration into software applications
Use Cases:
Academic research
Text mining
Educational machine learning

Microsoft Azure Machine Learning

Microsoft Azure Machine Learning is a cloud-based service that provides a platform for
building, deploying, and managing machine learning models. It offers an end-to-end
solution for machine learning workflows.
Key Features:
Pre-built machine learning models and templates
Automated machine learning capabilities
Easy deployment of models in the cloud
Support for Python, R, and integration with other Azure services
Use Cases:
Predictive modeling
Anomaly detection
Recommendation systems
Google Cloud AI Platform
Google Cloud AI Platform is a comprehensive suite of cloud-based tools for building and
deploying machine learning models. It leverages the power of Google’s infrastructure to
provide scalable solutions for machine learning applications.
Key Features:
Support for custom models using TensorFlow, PyTorch, and Scikit-learn
AutoML for training high-quality models with minimal expertise
Integration with BigQuery for large-scale data analysis
Managed Jupyter Notebooks for experimentation
Use Cases:
Retail, healthcare
Financial services for predictive analytics
Personalized recommendations
Unit II Supervised Learning – I: Simple Linear Regression – Multiple Linear
Regression – Polynomial Regression – Ridge Regression – Lasso Regression –
Evaluating Regression Models – Model Selection – Bagging – Ensemble Methods.

What is Regression Analysis?

In machine learning, regression analysis is a statistical technique that predicts continuous
numeric values based on the relationship between independent and dependent variables.
The main goal of regression analysis is to plot a line or curve that best fit the data and to
estimate how one variable affects another.
Regression analysis is a fundamental concept in machine learning and it is used in many
applications such as forecasting, predictive analytics, etc.
In machine learning, regression is a type of supervised learning. The key objective of
regression-based tasks is to predict output labels or responses, which are continuous
numeric values, for the given input data. The output will be based on what the model has
learned in the training phase.
Regression models use the input data features (independent variables) and their
corresponding continuous numeric output values (dependent or outcome variables) to
learn specific associations between inputs and corresponding outputs.
Terminologies Used In Regression Analysis
Let us understand some basic terminologies used in regression analysis before going into
further detail. The following are some important terminologies −
Independent Variables − These variables are used to predict the value of the dependent
variable. These are also called predictors. In dataset, these are represented as features.
Dependent Variables − These are the variables whose values we want to predict. These
are the main factors in regression analysis. In dataset, these are represented as target
variables
Regression line − It is a straight line or curve that a regressor plots to fit the data points
best.
Overfitting and underfitting − Overfitting is when the regression model works well
with the training dataset but not with the testing dataset. It's also referred to as the
problem of high variance. Underfitting is when the model doesn't work well with training
datasets. It's also referred to as the problem of high bias.
Outliers − These are data points that don't fit the pattern of the rest of the data. They are
the extremely high or extremely low values in the data set.
Multicollinearity − multicollinearity occurs when independent variables (features) have
dependency among them.
How Does Regression Work?
Regression in machine learning is a supervised learning. Basically, regression is a
statistical technique that finds a relationship between dependent and independent
variables. To implement regression in machine learning, a regression algorithm is trained
with a labeled dataset. The dataset contains features (independent variables) and target
values (dependent variable).
During the training phase, the regression algorithm learns the relation between
independent variables (predictors) and dependent variables (target).
The regression models predict new values based on the learned relation between
predictors and targets during the training.
Types of Regression in Machine Learning
Generally, the classification of regression methods is done based on the three metrics −
the number of independent variables, type of dependent variables, and shape of the
regression line.
There are numerous regression techniques used in machine learning. However, the
following are commonly used types of regression −
Linear Regression
Logistic Regression
Polynomial Regression
Lasso Regression
Ridge Regression
Decision Tree Regression
Random Forest Regression
Support Vector Regression
Linear Regression
Linear regression in machine learning is defined as a statistical model that analyzes the
linear relationship between a dependent variable and a given set of independent variables.
The linear relationship between variables means that when the value of one or more
independent variables will change (increase or decrease), the value of the dependent
variable will also change accordingly (increase or decrease).
In machine learning, linear regression is used for predicting continuous numeric values
based on learned linear relation for new and unseen data. It is used in predictive
modeling, financial forecasting, risk assessment, etc.

What is Linear Regression?

Linear regression is a statistical technique that estimates the linear relationship between a
dependent and one or more independent variables. In machine learning, linear regression
is implemented as a supervised learning approach. In machine learning, labeled datasets
contain input data (features) and output labels (target values). For linear regression in
machine learning, we represent features as independent variables and target values as the
dependent variable.
For the simplicity, take the following data (Single feature and single target)

Square Feet (X) House Price (Y)

1300 240

1500 320

1700 330

1830 295
 1550 256

2350 409

1450 319

In the above data, the target House Price is the dependent variable represented by X, and
the feature, Square Feet, is the independent variable represented by Y. The input features
(X) are used to predict the target label (Y). So, the independent variables are also known
as predictor variables, and the dependent variable is known as the response variable.
So lets define linear regression in machine learning as follows:
In machine learning, linear regression uses a linear equation to model the relationship
between a dependent variable (Y) and one or more independent variables (Y).
The main goal of the linear regression model is to find the best-fitting straight line (often
called a regression line) through a set of data points.

Line of Regression
A straight line that shows a relation between the dependent variable and independent
variables is known as the line of regression or regression line.

Furthermore, the linear relationship can be positive or negative in nature as explained

below −
 Positive Linear Relationship
A linear relationship will be called positive if both independent and dependent variable
increases. It can be understood with the help of the following graph −

Negative Linear Relationship

A linear relationship will be called positive if the independent increases and the
dependent variable decreases. It can be understood with the help of the following graph −
Linear regression is of two types, "simple linear regression" and "multiple linear
regression".
Simple Linear Regression
What is Simple Linear Regression?
Simple linear regression is a statistical and supervised learning method in which a single
independent variable (also known as a predictor variable) is used to predict the dependent
variable. In other words, it models the linear relationship between the dependent variable
and a single independent variable.
Simple linear regression in machine learning is a type of linear regression. When the
linear regression algorithm deals with a single independent variable, it is known as simple
linear regression. When there is more than one independent variable (feature variables), it
is known as multiple linear regression.
Independent Variable
The feature inputs in the dataset are termed as the independent variables. There is only a
single independent variable in simple linear regression. An independent variable is also
known as a predictor variable as it is used to predict the target value. It is plotted on a
horizontal axis.
Dependent Variable
The target value in the dataset is termed as the dependent variable. It is also known as a
response variable or predicted variable. It is plotted on a vertical axis.

Line of Regression
In simple linear regression, a line of regression is a straight line that best fits the data
points and is used to show the relationship between a dependent variable and an
independent variable.
Graphical Representation
The following graph depicts the simple linear regression model −
In the above image, the straight line represents the simple linear regression line where
Ŷ is the predicted value, and Y is dependent variable (target) and X is independent
variable (input).
Simple Linear Regression Model
A simple linear regression model in machine learning can be represented as the following
mathematical equation −
Y=w0+w1X+ϵ
Where
Y is the dependent variable (target).
X is the independent variable (feature).
w0 is the y-intercept of the line.
w1 is the slope of the line, representing the effect of X on Y.
ε is the error term, capturing the variability in Y not explained by X.

How Simple Linear Regression Works?

The main of simple linear regression is to find the best fit line (a straight line) through the
data points that minimizes the difference between the actual values and predicted values.
Defining Hypothesis Function
In simple linear regression, the hypothesis is that there is a linear relation between the
dependent variable (output/ target) and the independent variable (input). This linear
relation can be represented using a linear equation −
Y^=w0+w1X
With different values of parameters w0 and w1 there are multiple linear equations (straight
lines). The set of all such linear equations (all straight lines) is termed hypothesis space.
Now, the main aim of the simple linear regression model is to find the best-fit line in
Hypothesis space (set of all straight lines).
Finding the Best Fit Line
Now the task is to find the best fit line (line of regression). To do this, we define a cost
function or loss function that measure the the difference between the actual values and
predicted values.
To find the best fit line, the simple linear regression model initializes (with default
values) the parameters of the regression line. This regression line (with initialized
parameters) is used to find the predicted values for the given input values.
Loss Function for Simple Linear Regression
Now using the input and predicted values, we compute the loss function. The loss
function is used to find the optimal values of the parameters.
The loss function finds the difference between the input value and predicted value. There
are different loss functions such as mean squared error (MSE), mean absolute error
(MEA), R-squared, etc. used in simple linear regression. The most commonly used loss
function is mean squared error.
The loss function for simple linear regression in terms of mean squared error is as follows
−
J(w0,w1)=12n∑i=1n(Yi−Y^i)2
Optimization
The optimal values of parameters are those values that minimize the cost function.
Finding the optimal values is an iterative process in which the parameters are updated
iteratively.
There are many optimization techniques applied in simple linear regression. Gradient
Descent is a simple and most common optimization technique used in simple linear
regression.
A linear equation with optimal parameter values is the best fit line(regression line) and it
is the final solution for a simple linear regression problem. This line is used to predict
new and unseen data.
Assumptions of Simple Linear Regression
There are some assumptions about the dataset that are made by the simple linear
regression model. The following are some assumptions −
Linearity − This assumption assumes that the relationship between the dependent and
independent variables is linear. That means the dependent variable changes linearly as the
independent variable changes. A scatter plot will show the linearity in the dataset.
Homoskedasticity − For all observations, the variance of the residuals is the same. This
assumption relates to the squared residuals.
Independence − The examples (observations or X and Y pairs) are independent. There is
no collinearity in data so the residuals will not be correlated. To check this, we example
the scatter plot of residuals vs. fits.
Normality − Model Residuals are normally distributed. Residuals are the differences
between the actual and predicted values. To check for the normality, we examine the
histogram of residuals. The histogram should be approximately normally distributed.
Implementation of Simple Linear Regression Algorithm using Python
To implement the simple linear regression algorithm, we are taking a dataset with two
variables: Years Experience (independent variable) and Salary (dependent variable).
Here, we are using the following dataset. The dataset contains 30 examples of data points.
You can create a CSV file and store these data points in it.
Salary_Data.csv

Years of Experience Salary

1.1 39343

1.3 46205

1.5 37731

2 43525

2.2 39891

2.9 56642

3 60150

3.2 54445
3.2 64445

3.7 57189

3.9 63218

4 55794

4 56957

4.1 57081

4.5 61111

4.9 67938

5.1 66029

5.3 83088

5.9 81363

6 93940

6.8 91738

7.1 98273

7.9 101302

8.2 113812

8.7 109431

9 105582

9.5 116969

9.6 112635

10.3 122391
 10.5 121872

What is the purpose of this implementation?

The purpose of building this simple linear regression model is to determine which line
best represents the relationship between the two variables.
The following are the steps to implement the simple linear regression model in Python −
Step 1: Data Preparation
Data preparation or pre-processing is the initial step. We have our dataset as a CSV file
named "Salary_Data.csv," as discussed above.
We need to import python libraries prior to importing the dataset and building the simple
linear regression model.
import numpy as np
import [Link] as plt
import pandas as pd
Load the dataset
dataset = pd.read_csv('Salary_Data.csv')
The dependent variable (X) and independent variable (Y) must then be extracted from the
provided dataset. Years of experience (YearsExperience) is the independent variable,
and Salary is the dependent variable.
X = [Link][:, :-1].values
y = [Link][:, -1].values
Let's check the first five examples of the dataset.
print([Link]())

Output
0 1.1 39343.0
1 1.3 46205.0
2 1.5 37731.0
3 2.0 43525.0
4 2.2 39891.0
Lets check if the dataset is linear or not
[Link](X, y, color="green")
[Link]("Salary vs Experience")
[Link]("Years of Experience")
[Link]("Salary (INR)")
[Link]()
Output

The above graph shows that the dependent and independent variables are linearly
dependent. So we can apply the simple linear regression on the dataset to find the best
relation between these variables.
Split the dataset into training and testing sets
The training set and test set will then be divided into two groups. We will use 80%
observations for the training set and 20% observations for the test set out of the total 30
observations we have. So there will be 24 observation in training set and 6 observation in
test set. We divide our dataset into training and test sets so that we can use one set to train
and the other to test our model.
# Split the dataset into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
Here, X_train represents the input feature of the training data and y_train represents the
output variable (target variable).
Step 2: Model Training (Fitting the Simple Linear Regression to Training Set)
The next step is fitting our model with the training dataset. We will use scikit-learn's
LinearRegression class to train a simple linear regression model on the training data. The
code for this is as follows −
from sklearn.linear_model import LinearRegression
# Create a linear regression object
regressor= LinearRegression()
[Link](X_train, y_train)
The fit() method is used to fit the linear regression object (regressor) to the training data.
The model learns the relation between the predictor variable (X_train), and the target
variable (y_train).
Step 3: Model Testing
Once the model is trained, we can use it to make predictions on the test data. The code for
this is as follows −
y_pred = [Link](X_test)
df = [Link]({'Actual Values':y_test, 'Predicted Values':y_pred})
print(df)
Output
Actual Values Predicted Values
0 60150.0 54093.648425
1 93940.0 82416.119864
2 57081.0 64478.554619
3 116969.0 115459.003211
4 56957.0 63534.472238
5 121872.0 124899.827024
The above output shows actual values and predicted values of Salary in the test set.
Here, X_test represents the input feature of the test data and y_pred represents the
predicted output variable (target variable).
Similarly, you can test the model with training data.
y_pred = [Link](X_train)
df = [Link]({'Real Values':y_test, 'Predicted Values':y_pred})
print(df)
Output
Real Values Predicted Values
0 57189.0 60702.225094
1 64445.0 55981.813188
2 63218.0 62590.389857
3 122391.0 123011.662261
4 91738.0 89968.778915
5 43525.0 44652.824612
6 61111.0 68254.884145
7 56642.0 53149.566044
8 66029.0 73919.378433
9 83088.0 75807.543195
10 46205.0 38044.247943
11 109431.0 107906.344160
12 98273.0 92801.026059
13 37731.0 39932.412705
14 54445.0 55981.813188
15 39891.0 46540.989374
16 101302.0 100353.685109
17 55794.0 63534.472238
18 81363.0 81472.037483
19 39343.0 36156.083180
20 113812.0 103185.932253
21 67938.0 72031.213670
22 112635.0 116403.085592
23 105582.0 110738.591304
Step 4: Model Evaluation
We need to evaluate the performance of the model to determine its accuracy. We will use
the mean squared error (MSE), root mse (RMSE), mean average error (MAE), and the
coefficient of determination (R^2) as evaluation metrics. The code for this is as follows −
from [Link] import mean_squared_error
from [Link] import mean_absolute_error
from [Link] import r2_score

# get the predicted values for test dat

y_pred = [Link](X_test)
mse = mean_squared_error(y_test, y_pred)
print("mse", mse)
rmse = mean_squared_error(y_test, y_pred, squared=False)
print("rsme", rmse)
mae = mean_absolute_error(y_test, y_pred)
print("mae", mae)
r2 = r2_score(y_test, y_pred)
print("r2", r2)
Output
mse: 46485664.99327367
rsme: 6818.0396737826095
mae: 6015.513730219523
r2: 0.9399326805390613
Here, y_test represents the actual output variable of the test data.
Step 5: Visualize Training Set Results (with Regression Line)
Now, let's visualize the results on the training set and the regression line.
We use the scatter plot to plot the actual values (input and target values) in the training
set. We also plot a straight line (regression line) for actual values (input) and predicted
values of the training set.
y_pred = [Link](X_train)
[Link](X_train, y_train, color="green", label="training data points (actual)")
[Link](X_train, y_pred, color="blue",label="training data points (predicted)")
[Link](X_train, y_pred, color="red")
[Link]("Salary vs Experience (Training Dataset)")
[Link]("Years of Experience")
[Link]("Salary(In Rupees)")
[Link]()
[Link]()
Output
The above graph shows the line of regression (straight line in red color), actual values (in
green color), and predicted values (in blue color) for the training set.
Step 6: Visualize the Test Set Results (with Regression Line)
Now, let's visualize the results on the test set and the regression line.
We use the scatter plot to plot the actual values (input and target values) in the test set.
We also plot a straight line (regression line) for actual values (input) and predicted values
of the test set.
y_pred = [Link](X_test)
[Link](X_test, y_test, color="green", label="test data points (actual)")
[Link](X_test, y_pred, color="blue",label="test data points (predicted)")
[Link](X_test, y_pred, color="red")
[Link]("Salary vs Experience (Test Dataset)")
[Link]("Years of Experience")
[Link]("Salary(In Rupees)")
[Link]()
[Link]()
Output
The above graph shows the line of regression (straight line in red color), actual values (in
green color), and predicted values (in blue color) for the test set.
Multiple Linear Regression
Multiple linear regression in machine learning is a supervised algorithm that models the
relationship between a dependent variable and multiple independent variables. This
relationship is used to predict the outcome of the dependent variable.
Multiple linear regression is a type of linear regression in machine learning. There are
mainly two types of linear regression algorithms −
simple linear regression − it deals with two features (one dependent variable and one
independent variable).
multiple linear regression − deals with more than two features (one dependent variable
and more than one independent variables).

What is Multiple Linear Regression?

In machine learning, multiple linear regression (MLR) is a statistical technique that is
used to predict the outcome of a dependent variable based on the values of multiple
independent variables. The multiple linear regression algorithm is trained on data to learn
a relationship (known as a regression line) that best fits the data. This relation describes
how various factors affect the result. This relation is used to forecast the value of
dependent variable based on the values of independent variables.
In linear regression (simple and multiple), the dependent variable is continuous (numeric
value) and independent variables can be continuous or discreet (numeric value).
Independent variables can also be categorical (gender, occupation), but they need to be
converted to numerical values first.
Multiple linear regression is basically the extension of simple linear regression that
predicts a response using two or more features. Mathematically we can represent the
multiple linear regression as follows −
Consider a dataset having n observations, p features i.e. independent variables and y as
one response i.e. dependent variable the regression line for p features can be calculated as
follows −
h(xi)=w0+w1xi1+w2xi2+⋅⋅⋅+wpxip
Here, h(xi) is the predicted response value and w0,w1,w2....wp are the regression
coefficients.
Multiple Linear Regression models always includes the errors in the data known as
residual error which changes the calculation as follows −
yi=w0+w1xi1+w2xi2+⋅⋅⋅+wpxip+ei
We can also write the above equation as follows −
yi=h(xi)+eiorei=yi−h(xi)
Assumptions of Multiple Linear Regression
The following are some assumptions about the dataset that are made by the multiple
linear regression model −
Linearity
The relationship between the dependent variable (target) and independent (predictor)
variables is linear.
Independence
Each observation is independent of others. The value of the dependent variable for one
observation is independent of the value of another.

Homoscedasticity
For all observations, the variance of the residual errors is similar across the value of each
independent variable.
Normality of Errors
The residuals (errors) are normally distributed. The residuals are differences between the
actual and predicted values.
No Multicollinearity
The independent variables are not highly correlated with each other. Linear regression
models assume that there is very little or no multi-collinearity in the data.
No Autocorrelation
There is no correlation between residuals. This ensures that the residuals (errors) are
independent of each other.
Fixed Independent Variables
The values of independent variables are fixed in all repeated samples.
Violations of these assumptions can lead to biased or inefficient estimates. It is essential
to validate these assumptions to ensure model accuracy.
Implementing Multiple Linear Regression in Python
To implement multiple linear regression in Python using Scikit-Learn, we can use the
same Linear Regression class as in simple linear regression, but this time we need to
provide multiple independent variables as input.
Step 1: Data Preparation
We use the dataset named [Link] with 50 examples. It contains four predictor
(independent) variables and a target (dependent) variable. The following table represents
the data in [Link] file.
[Link]

R&D Marketing
Administration State Profit
Spend Spend

New
165349.2 136897.8 471784.1 192261.8
York

162597.7 151377.6 443898.5 California 191792.1

153441.5 101145.6 407934.5 Florida 191050.4

New
144372.4 118671.9 383199.6 182902
York

142107.3 91391.77 366168.4 Florida 166187.9

 New
131876.9 99814.71 362861.4 156991.1
York

134615.5 147198.9 127716.8 California 156122.5

130298.1 145530.1 323876.7 Florida 155752.6

New
120542.5 148719 311613.3 152211.8
York

123334.9 108679.2 304981.6 California 149760

101913.1 110594.1 229161 Florida 146122

100672 91790.61 249744.6 California 144259.4

93863.75 127320.4 249839.4 Florida 141585.5

91992.39 135495.1 252664.9 California 134307.4

119943.2 156547.4 256512.9 Florida 132602.7

New
114523.6 122616.8 261776.2 129917
York

78013.11 121597.6 264346.1 California 126992.9

New
94657.16 145077.6 282574.3 125370.4
York

91749.16 114175.8 294919.6 Florida 124266.9

New
86419.7 153514.1 0 122776.9
York

76253.86 113867.3 298664.5 California 118474

New
78389.47 153773.4 299737.3 111313
York
73994.56 122782.8 303319.3 Florida 110352.3

67532.53 105751 304768.7 Florida 108734

New
77044.01 99281.34 140574.8 108552
York

64664.71 139553.2 137962.6 California 107404.3

75328.87 144136 134050.1 Florida 105733.5

New
72107.6 127864.6 353183.8 105008.3
York

66051.52 182645.6 118148.2 Florida 103282.4

New
65605.48 153032.1 107138.4 101004.6
York

61994.48 115641.3 91131.24 Florida 99937.59

New
61136.38 152701.9 88218.23 97483.56
York

63408.86 129219.6 46085.25 California 97427.84

55493.95 103057.5 214634.8 Florida 96778.92

46426.07 157693.9 210797.7 California 96712.8

New
46014.02 85047.44 205517.6 96479.51
York

28663.76 127056.2 201126.8 Florida 90708.19

44069.95 51283.14 197029.4 California 89949.14

New
20229.59 65947.93 185265.1 81229.06
York
 38558.51 82982.09 174999.3 California 81005.76

28754.33 118546.1 172795.7 California 78239.91

27892.92 84710.77 164470.7 Florida 77798.83

23640.93 96189.63 148001.1 California 71498.49

New
15505.73 127382.3 35534.17 69758.98
York

22177.74 154806.1 28334.72 California 65200.33

New
1000.23 124153 1903.93 64926.08
York

1315.46 115816.2 297114.5 Florida 49490.75

0 135426.9 0 California 42559.73

New
542.05 51743.15 0 35673.41
York

0 116983.8 45173.06 California 14681.4

You can create a CSV file and store the above data points in it.
We have our dataset as [Link] file. We will use it to understand the implementation of
the multiple linear regression in Python.
We need to import libraries before loading the dataset.
# import libraries
import numpy as np
import [Link] as plt
import pandas as pd
Load the dataset
We load our dataset as a Pandas Data frame named <string>dataset. Now let's create a list
of independent values (predictors) and put them in a variable called X.</string>
The independent values are 'R&D Spend', 'Administration', 'Marketing Spend'. We are not
using the independent variable 'State' for sake of simplicity.
We put the dependent variable values to a variable y.
# load dataset
dataset = pd.read_csv('[Link]')
X = dataset[['R&D Spend', 'Administration', 'Marketing Spend']]
y = dataset['Profit']
Let's check first five examples (rows) of input features (X) and target (y) −
[Link]()
Output
R&D Spend Administration Marketing Spend
0 165349.20 136897.80 471784.10
1 162597.70 151377.59 443898.53
2 153441.51 101145.55 407934.54
3 144372.41 118671.85 383199.62
4 142107.34 91391.77 366168.42
[Link]()
Output
Profit
192261.83
191792.06
191050.39
182901.99
166187.94
Split the dataset into training and test sets
Now, we split the dataset into a training set and a test set. Both the X(independent values)
and y (dependent values) are divided into two sets - training and test. We will use 20% for
the test set. In such a way out of 50 feature vectors (observations/ examples), there will be
40 feature vectors in training set and 10 feature vectors in test set.
# Split the dataset into training and test sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
Here X_train and X_test represent input features in training set and test set,
where y_train and y_test represent target values (output) in traning and test set.
Step 2: Model Training
The next step is to fit our model with training data. We will use linear_model class
from sklearn module. We use the Linear Regression() method of linear_model class to
create a linear regression object, here we name it as regressor.
# Fit Multiple Linear Regression to the Training set
from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
[Link](X_train, y_train)
The regressor object has fit() method. The fit() method is used to fit the linear regression
object, regressor to the training data. The model learns the relation between the predictor
variable (X_train), and the target variable (y_train).
Step 3: Model Testing
Now our model is ready to use for prediction. Let's test our regressor model on test data.
We use predict() method to predict the results for the test set. It takes input features
(X_test) and return the redicted values.
y_pred = [Link](X_test)
df = [Link]({'Real Values':y_test, 'Predicted Values':y_pred})
print(df)
Output
Real Values Predicted Values
23 108733.99 110159.827849
43 69758.98 59787.885207
26 105733.54 110545.686823
34 96712.80 88204.710014
24 108552.04 114094.816702
39 81005.76 84152.640761
44 65200.33 63862.256006
18 124266.90 129379.514419
47 42559.73 45832.902722
17 125370.37 130086.829016
You can compare the actual values and predicted values.
Step 4: Model Evaluation
We now evaluate our model to check how accurate it is. We will use mean square error
(MSE), root mean square error (RMSE), mean absolute error (MAE), and R2-score
(Coefficient of determination).
from [Link] import mean_squared_error, root_mean_squared_error,
mean_absolute_error, r2_score
# Assuming you have your true y values (y_test) and predicted y values (y_pred)
mse = mean_squared_error(y_test, y_pred)
rmse = root_mean_squared_error(y_test, y_pred)
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print("Mean Squared Error (MSE):", mse)
print("Root Mean Squared Error (RMSE):", rmse)
print("Mean Absolute Error (MAE):", mae)
print("R-squared (R2):", r2)
Output
Mean Squared Error (MSE): 72684687.6336162
Root Mean Squared Error (RMSE): 8525.531516193943
Mean Absolute Error (MAE): 6425.118502810154
R-squared (R2): 0.9588459519573707
You can examine the above metrics. Our model shows an R-squared score of around
0.96, which means that 96% of data points are scattered around the fitted regression line.
Another interpretation is that 96% of the variation in the output variables is explained by
the input variables.
Step 5: Model Prediction for New Data
Let's use our regressor model to predict profit values based on R&D Spend,
Administration and Marketing Spend.
['R&D Spend','Administration','Marketing Spend']=[166343.2, 136787.8, 461724.1]
// predict profit when R&D Spend is 166343.2, Administration is 136787.8 and
Marketing Spend is 461724.1
new_data =[[166343.2, 136787.8, 461724.1]]
profit = [Link](new_data)
print(profit)
Output
[193053.61874652]
The model predicts the profit value is approximately 192090.567 for the above three
values.
Model Parameters (Coefficients and Intercept)
The model parameters (intercept and coefficients) describe the relation between a
dependent variable and the independent variables.
Our regression model for the above use case,
Y=w0+w1X1+w2X2+w2X2
w0 is intercept and w1,w2,w3 are coefficients of X1,X2,X3 respectively.
Here,
X1 represents R&D Spend,
X2 represents Administration, and
X3 represents Marketing Spend.
Let's first compute the intercept and coefficients.
print("coefficients: ", regressor.coef_)
print("intercept: ", regressor.intercept_)
Output
coefficients: [ 0.81129358 -0.06184074 0.02515044]
intercept: 54946.94052163202
The above output shows the following -
w0 = 54946.94052163202
w1 = 0.81129358
w2 = -0.06184074
w3 = 0.02515044
Result Explanation
We have calculated intercept (w0) and coefficients (w1, w2, w3).
The coefficients are as follows -
R&D Spend: 0.81129358
Administration: -0.06184074
Marketing Spend: 0.02515044
This shows that if R&D Spend is increased by 1 USD, the Profit will increase by
0.81851334 USD.
The result shows that when Administration spend is increased by 1 USD, the Profit will
decrease by 0.03124763 USD.
And when Marketing Spend increases by 1 USD, the Profit increases by 0.02042286
USD.
Let's verify the result,
In step 5, we have predicted Profit for new data as 193053.61874652
Here,
new_data =[[166343.2, 136787.8, 461724.1]]
Profit = 54946.94052163202+ 0.81129358*166343.2 - 0.06184074* 136787.8 +
0.02515044 * 461724.1
Profit = 193053.616257
Which is approximately the same as model prediction. Why approximately? Because of
residual error.
residual error = 193053.61874652 - 193053.616257
residual error = 0.00248952
Applications of Multiple Linear Regression
The following are some commonly used applications of multiple linear regression −

Application Description

Predicting stock prices, forecasting exchange rates, assessing credit

Finance
risk.

Predicting sales, customer churn, and marketing campaign

Marketing
effectiveness.

Predicting house prices based on factors like size, location, and

Real Estate
number of bedrooms.

Predicting patient outcomes, analyzing the impact of treatments,

Healthcare
and identifying risk factors for diseases.

Forecasting economic growth, analyzing the impact of policies, and

Economics
predicting inflation rates.

Social Modeling social phenomena, predicting election outcomes, and

Sciences understanding human behavior.
Challenges of Multiple Linear Regression
The following are some common challenges faced by multiple linear regression in
machine learning −

Challenge Description

High correlation between independent variables, leading to

Multicollinearity unstable model coefficients and difficulty in interpreting the impact
of individual variables.

The model fits the training data too closely, leading to poor
Overfitting
performance on new, unseen data.

The model fails to capture the underlying patterns in the data,

Underfitting
resulting in poor performance on both training and test data.

Multiple linear regression assumes a linear relationship between the

Non-linearity independent and dependent variables. Non-linear relationships can
lead to inaccurate predictions.

Outliers can significantly impact the model's performance,

Outliers
especially in small datasets.

Missing Data Missing data can lead to biased and inaccurate results.

Difference Between Simple and Multiple Linear Regression

The following table highlights the major differences between simple and multiple linear
regression −

Simple Linear
Feature Multiple Linear Regression
Regression

Independent
One Two or more
Variables

Model
y = w1x + w0 y=w0+w1x1+w2x2+ ... +wpxp
Equation
 Complexity Less complex More complex due to multiple variables

Predicting house prices Predicting sales based on advertising

based on square expenditure, price, and competitor
Real-world
footage, predicting sales activity, predicting student performance
Applications
based on advertising based on study hours, attendance, and
expenditure IQ

Model
Easier to interpret More complex to interpret due to
Interpretatio
coefficients multiple variables
n

Polynomial Regression in Machine Learning

What is Polynomial Regression?
Polynomial Linear Regression is a type of regression analysis in which the relationship
between the independent variable and the dependent variable is modeled as an n-th degree
polynomial function. Polynomial regression allows for a more complex relationship
between the variables to be captured beyond the linear relationship in simple linear
regression and multiple linear regression.
Why Polynomial Regression?
In machine learning (ML) and data science, choosing between a linear regression or
polynomial regression depends upon the characteristics of the dataset. A non-linear
dataset can't be fitted with a linear regression. If we apply linear regression to a nonlinear
dataset, it will not be able to capture the non-linear patterns in the data.
Look at the below diagram to understand why we need polynomial regression for non-
linear data.
The above diagram shows the simple linear model hardly fits the data points whereas the
polynomial model fits most of the data points.

Equation of Polynomial Regression Model

In machine learning, the general formula for polynomial regression of degree n is as
follows −
y=w0+w1x+w2x2+w3x3+…+wnxn+ϵ
Where
y is the dependent variable (output).
x is the independent variable (input).
w0,w1,w2,…,wn are the coefficients (parameters) of the model.
n is the degree of the polynomial (the highest power of x).
ϵ is the error term or residual, representing the difference between the observed value and
the model's prediction.
For a quadratic (second-degree) polynomial regression, the formula would be:
y=w0+w1x+w2x2+ϵ
This would fit a parabolic curve to the data points.
How does Polynomial Regression Work?
In machine learning, the polynomial regression actually works in a similar way as linear
regression works. It is modeled as multiple linear regression. The input feature is
transformed into polynomial features of higher degrees (x2,x3,...,xn). These features are
now treated as separate independent variables as in multiple linear regression. Now, a
multiple linear regressor is trained on these transformed polynomial features.
The polynomial regression is a special case of multiple linear regression but there is a
difference that multiple linear regression assumes linearity of input features. Here, in
polynomial regression, the transformed polynomial features are dependent on the original
input feature.
Implementation of Polynomial Regression using Python
Let's implement polynomial regression using Python. We will use a well known machine
learning Python library, Scikit-learn for building a regression model.
Step 1: Data Preparation
In machine learning model building, the data preparation is very important step. Let's
prepare our data first. We will be using a dataset named ice_cream_selling_data.csv. It
contains 49 data examples. It has an input feature/ independent variable (Temperature
(C)) and target feature/ dependent variable (Ice Cream Sales (units)).
The following table represents the data in ice_cream_selling_data.csv file.
ice_cream_selling_data.csv

Temperature (C) Ice Cream Sales (units)

-4.662262677 41.84298632

-4.316559447 34.66111954

-4.213984765 39.38300088

-3.949661089 37.53984488

-3.578553716 32.28453119

-3.455711698 30.00113848

-3.108440121 22.63540128

-3.081303324 25.36502221

-2.672460827 19.22697005

-2.652286793 20.27967918
-2.651498033 13.2758285

-2.288263998 18.12399121

-2.11186969 11.21829447

-1.818937609 10.01286785

-1.66034773 12.61518115

-1.326378983 10.95773134

-1.173123268 6.68912264

-0.773330043 9.392968661

-0.673752802 5.210162615

-0.149634867 4.673642541

-0.036156498 0.328625517

-0.033895286 0.897603187

0.008607699 3.165600008

0.149244574 1.931416029

0.688780908 2.576782245

0.693598873 4.625689458

0.874905029 0.789973651

1.024180814 2.313806358

1.240711619 1.292360811

1.359812674 0.953115312

1.740000012 3.782570136
 1.850551926 4.857987801

1.999310369 8.943823209

2.075100597 8.170734936

2.31859124 7.412094028

2.471945997 10.33663062

2.784836463 15.99661997

2.831760211 12.56823739

2.959932091 21.34291574

3.020874314 20.11441346

3.211366144 22.8394055

3.270044068 16.98327874

3.316072519 25.14208223

3.335932412 26.10474041

3.610778478 28.91218793

3.704057438 17.84395652

4.130867961 34.53074274

4.133533788 27.69838335

4.899031514 41.51482194

Note − Create a CSV file with the above data and save it as ice_cream_selling_data.csv.
Import Python libraries and packages for data preparation
Let's first import libraries and packages required in the data preparation step. We use
Python pandas for reading CSV files. We use NumPy to convert the pandas data frame to
NumPy array. Input and output features are NumPy arrays. We
use preprocessing package from the Scikit-learn library for preprocessing related tasks
such as transforming input feature to polynomial features.
import numpy as np
import pandas as pd
import [Link] as plt
from [Link] import PolynomialFeatures
Load the dataset
Load the ice_cream_selling_data.csv as a pandas dataframe. Learn more about data
loading here.
data = pd.read_csv('/ice_cream_selling_data.csv')
[Link]()
Output
Temperature (C) Ice Cream Sales (units)
0 -4.662263 41.842986
1 -4.316559 34.661120
2 -4.213985 39.383001
3 -3.949661 37.539845
4 -3.578554 32.284531
Let's create independent variable (X) and the dependent variable (y).
X = [Link][:, 0].[Link](-1, 1)
y = [Link][:, 1].values
Visualize the original datapoints
Let's visualize the original data points to get some insight.
# Visualize the original data points
[Link](X, y, color="green")
[Link]("Original Data")
[Link]("Temperature (C)")
[Link]("Ice Cream Sales (units)")
[Link]()
Output
The above graph shows a parabolic curve (polynomial with degree 2) that will fit the
datapoints.
So the relationship between the dependent variable ("Ice Cream Sales (units)") and
independent variable ("Temperature (C)") can be modeled using polynomial regression
of degree 2.
Create a polynomial features object
Now, let's create a polynomial feature object with degree 2. We will
use PolynomialFeatures class from [Link] module to create the feature
object.
degree = 2 # Degree of the polynomial
poly_features = PolynomialFeatures(degree=degree)
Let's now transform the input data to include polynomial features
X_poly = poly_features.fit_transform(X)
Here X_poly is transformed polynomial features of original input features (X). The
transformed data is of (49, 3) shape.
Step 2: Model Training
We have created polynomial features. Now, let's build out the model. We
use LinearRegression class from sklearn.linear_model module. As we already discussed,
Polynomial regression is a special type of linear regression.
Let's create a linear regression object lr_model and train (fit) the model with data.
from sklearn.linear_model import LinearRegression
lr_model = LinearRegression()
#Now, fit the model (linear regression object) on the data
lr_model.fit(X_poly, y)
So far, we have trained our regression model lr_model
Step 3: Model Prediction and Testing
Now, we can use our model to predict the output. Before going to predict for new data,
let's predict for the existing data.
# Generate predictions
y_pred = lr_model.predict(X_poly)
df = [Link]({'Actual Values':y, 'Predicted Values':y_pred})
print(df)

Output
Actual Values Predicted Values
0 41.842986 46.564507
1 34.661120 40.600548
2 39.383001 38.915089
3 37.539845 34.749272
4 32.284531 29.331940
5 30.001138 27.649735
6 22.635401 23.192862
7 25.365022 22.863178
8 19.226970 18.222266
9 20.279679 18.009098
10 13.275828 18.000794
11 18.123991 14.418541
12 11.218294 12.853070
13 10.012868 10.504868
14 12.615181 9.364587
15 10.957731 7.264266
16 6.689123 6.437055
17 9.392969 4.683654
18 5.210163 4.337906
19 4.673643 3.116139
20 0.328626 2.983983
21 0.897603 2.981829
22 3.165600 2.944811
23 1.931416 2.869446
24 2.576782 3.251711
25 4.625689 3.259923
26 0.789974 3.630683
27 2.313806 4.026226
28 1.292361 4.744891
29 0.953115 5.213321
30 3.782570 7.055902
31 4.857988 7.690948
32 8.943823 8.616039
33 8.170735 9.118494
34 7.412094 10.874961
35 10.336631 12.092557
36 15.996620 14.843721
37 12.568237 15.287199
38 21.342916 16.539614
39 20.114413 17.156188
40 22.839406 19.171090
41 16.983279 19.818497
42 25.142082 20.335157
43 26.104740 20.560474
44 28.912188 23.826884
45 17.843957 24.998282
46 34.530743 30.764287
47 27.698383 30.802396
48 41.514822 42.821195
You can compare the predicted values with actual values.
Step 4: Evaluating Model Performance
To evaluate the model performance, the best metric is the R-squared score (Coefficient of
determination). It measures the proportion of the variance in the dependent variable that is
predictable from the independent variables.
from [Link] import r2_score
# get the predicted values for test dat
y_pred = lr_model.predict(X_poly)
r2 = r2_score(y, y_pred)
print(r2)
Output
0.9321137090423877
The r2_score is the most common metric used to evaluate a regression model. The high
score indicates a better fit of the model with data. 1 represent perfect fit and 0 represents
no relation between the predicted values and actual values.
Result Explanation − You can examine the above metrics. Our model shows an R-
squared score of around 0.932, which means that approximately 93% of data points are
scattered around the fitted regression curve. Another interpretation is that 93% of the
variation in the output variables is explained by the input variables.
Step 5: Visualize the polynomial regression results
Let's visualize the regression results for better understanding. We use the pyplot module
from the Matplotlib library to plot the graph.
import [Link] as plt
# Visualize the polynomial regression results
[Link](X, y, color="green")
[Link](X, y_pred, color='red', label=f'Polynomial Regression (degree={degree})')
[Link]("Temperature (C)")
[Link]("Ice Cream Sales (units)")
[Link]()
[Link]('Polynomial Regression')
[Link]()
Output
The above graph shows that the polynomial regression with degree 2 fits well with the
original data. The polynomial curve (parabola), in red color, represents the best-fit
regression curve. This regression curve is used to predict the value. The graph also shows
that the predicted values are close to the actual values.
Step 5: Model Prediction for New Data
Up to now, we have predicted the values in the dataset. Let's use our regression model to
predict new, unseen data.
Let's take the Temperature (C) as 1.9929C and predict the units of Ice Cream Sales.
# Predict a new value
X_new = [Link]([[1.9929]]) # Example value to predict
X_new_poly = poly_features.transform(X_new)
y_new_pred = lr_model.predict(X_new_poly)
print(y_new_pred)
Output
[8.57450466]
The above result shows that the predicted value of Ice cream sales is 8.57450466.

Ridge Regression

Ridge regression, also known as L2 regularization, is a technique used in linear

regression to address the problem of multicollinearity among predictor variables.
Multicollinearity occurs when independent variables in a regression model are
highly correlated, which can lead to unreliable and unstable estimates of regression
coefficients.
Ridge regression mitigates this issue by adding a regularization term to the
ordinary least squares (OLS) objective function, which penalizes
large coefficients and thus reduces their variance.

What is ridge regression?

How Ridge Regression Addresses Overfitting and Multicollinearity?

Overfitting occurs when a model becomes too complex and fits the noise in the training
data, leading to poor generalization on new data. Ridge regression combats overfitting
by adding a penalty term (L2 regularization) to the ordinary least squares (OLS)
objective function.

Imagine your model is overreacting to tiny details in the data (like memorizing noise).
Ridge regression "calms it down" by shrinking the model's weights (coefficients) toward
zero. Think of it like adjusting a volume knob to get the perfect sound level—not too
loud (overfitting), not too quiet (underfitting).

This penalty discourages the model from using large values for the coefficients (the
numbers multiplying the features). It forces the model to keep these coefficients
small. By making the coefficients smaller and closer to zero, ridge regression simplifies
the model and reduces its sensitivity to random fluctuations or noise in the data. This
makes the model less likely to overfit and helps it perform better on new, unseen data,
improving its overall accuracy and reliability.
For Example - We are predicting house prices based on multiple features such as
square footage, number of bedrooms, and age of the house:

Price=1000 Size−500⋅Age+Noise

Ridge might adjust it to:

Price=800⋅Size−300⋅Age+Less Noise

As lambda increases the model places more emphasis on shrinking the coefficients of
highly correlated features, making their impact smaller and more stable. This reduces
the effect of multicollinearity by preventing large fluctuations in coefficient estimates
due to correlated predictors.

Mathematical Formulation of Ridge Regression Estimator

Consider the multiple linear regression model:.

y= Xβ+ ϵ
where:
y is an n×1 vector of observations,
X is an n×p matrix of predictors,
β is a p×1 vector of unknown regression coefficients,
ϵ is an n×1 vector of random errors.

The ordinary least squares (OLS) estimator of β is given by:

^β =¿
OLS

In the presence of multicollinearity, X ' X is nearly singular, leading to unstable

estimates. ridge regression addresses this issue by adding a penalty term kI, where k is
the ridge parameter and I is the identity matrix. The ridge regression estimator is:
^β =¿
k

This modification stabilizes the estimates by shrinking the coefficients, improving

generalization and mitigating multicollinearity effects.
Bias-Variance Tradeoff in Ridge Regression

Ridge regression allows control over the bias-variance trade-off. Increasing the value of
λ increases the bias but reduces the variance, while decreasing λ does the opposite. The
goal is to find an optimal λ that balances bias and variance, leading to a model that
generalizes well to new data.

As we increase the penalty level in ridge regression, the estimates of β gradually

change. The following simulation illustrates how the variation in β is affected by
different penalty values, showing how estimated parameters deviate from the true
values.

Bias-Variance Tradeoff in Ridge Regression

Ridge regression introduces bias into the estimates to reduce their variance. The mean
squared error (MSE) of the ridge estimator can be decomposed into bias and variance
components:
MSE( β^ k )=Bias ( β^ k )+ Var ( β^ k )
2

Bias: Measures the error introduced by approximating a real-world problem, which may
be complex, by a simplified model. In ridge regression, as the regularization parameter
k increases, the model becomes simpler, which increases bias but reduces variance.
Variance: Measures how much the ridge regression model's predictions would vary if
we used different training data. As the regularization parameter k decreases, the model
becomes more complex, fitting the training data more closely, which reduces bias but
increases variance.
Irreducible Error: Represents the noise in the data that cannot be reduced by any
model.
As k increases, the bias increases, but the variance decreases. The optimal value
of k balances this tradeoff, minimizing the MSE.
Selection of the Ridge Parameter in Ridge Regression

Choosing an appropriate value for the ridge parameter k is crucial in ridge regression, as
it directly influences the bias-variance tradeoff and the overall performance of the
model. Several methods have been proposed for selecting the optimal ridge parameter,
each with its own advantages and limitations. Methods for Selecting the Ridge
Parameter are:
Cross-Validation
Cross-validation is a common method for selecting the ridge parameter by dividing data
into subsets. The model trains on some subsets and validates on others, repeating this
process and averaging the results to find the optimal value of k.
K-Fold Cross-Validation: The data is split into K subsets, training on K-1 folds and
validating on the remaining fold. This is repeated K times, with each fold serving as the
validation set once.
Leave-One-Out Cross-Validation (LOOCV) A special case of K-fold where K equals
the number of observations, training on all but one observation and validating on the
remaining one. It’s computationally intensive but unbiased.
Generalized Cross-Validation (GCV)

Generalized Cross-Validation is an extension of cross-validation that provides a more

efficient way to estimate the optimal k without explicitly dividing the data. GCV is
based on the idea of minimizing a function that approximates the leave-one-out cross-
validation error. It is computationally less intensive and often yields similar results to
traditional cross-validation methods.
Information Criteria
Information criteria such as the Akaike Information Criterion (AIC) and the Bayesian
Information Criterion (BIC) can also be used to select the ridge parameter. These
criteria balance the goodness of fit of the model with its complexity, penalizing models
with more parameters.
Empirical Bayes Methods

Empirical Bayes methods involve estimating the ridge parameter by treating it as a

hyperparameter in a Bayesian framework. These methods use prior distributions and
observed data to estimate the posterior distribution of the ridge parameter.
Empirical Bayes Estimation: This method involves specifying a prior distribution
for k and using the observed data to update this prior to obtain a posterior distribution.
The mode or mean of the posterior distribution is then used as the estimate of k.
Stability Selection

Stability selection improves ridge parameter robustness by subsampling data and fitting
the model multiple times. The most frequently selected parameter across all subsamples
is chosen as the final estimate.
Practical Considerations for Selecting Ridge Parameter

Tradeoff Between Bias and Variance: The choice of the ridge parameter k involves a
tradeoff between bias and variance. A larger k introduces more bias but reduces
variance, while a smaller k reduces bias but increases variance. The optimal k balances
this tradeoff to minimize the mean squared error (MSE) of the model.
Computational Efficiency: Some methods for selecting k, such as cross-validation and
empirical Bayes methods, can be computationally intensive, especially for large
datasets. Generalized cross-validation and analytical methods offer more
computationally efficient alternatives.
Interpretability: The interpretability of the selected ridge parameter is also an
important consideration. Methods that provide explicit criteria or formulas for
selecting k can offer more insight into the relationship between the data and the model.
Read about Implementation of Ridge Regression from Scratch using Python.
Applications of Ridge Regression
Forecasting Economic Indicators: Ridge regression helps predict economic factors
like GDP, inflation, and unemployment by managing multicollinearity between
predictors like interest rates and consumer spending, leading to more accurate forecasts.
Medical Diagnosis: In healthcare, it aids in building diagnostic models by controlling
multicollinearity among biomarkers, improving disease diagnosis and prognosis.
Sales Prediction: In marketing, ridge regression forecasts sales based on factors like
advertisement costs and promotions, handling correlations between these variables for
better sales planning.
Climate Modeling: Ridge regression improves climate models by eliminating
interference between variables like temperature and precipitation, ensuring more
accurate predictions.
Risk Management: In credit scoring and financial risk analysis, ridge regression
evaluates creditworthiness by addressing multicollinearity among financial ratios,
enhancing accuracy in risk management.
Advantages and Disadvantages of Ridge Regression

Advantages:

Stability: Ridge regression provides more stable estimates in the presence of

multicollinearity.
Bias-Variance Tradeoff: By introducing bias, ridge regression reduces the variance of
the estimates, leading to lower MSE.
Interpretability: Unlike principal component regression, ridge regression retains the
original predictors, making the results easier to interpret.
Disadvantages:

Bias Introduction: The introduction of bias can lead to underestimation of the true
effects of the predictors.
Parameter Selection: Choosing the optimal ridge parameter k can be challenging and
computationally intensive.
Not Suitable for Variable Selection: Ridge regression does not perform variable
selection, meaning all predictors remain in the model, even those with negligible
effects.
Lasso Regression in Machine learning
What is Lasso Regression?
Lasso Regression is a regression method based on Least Absolute
Shrinkage and Selection Operator and is used in regression analysis for variable
selection and regularization. It helps remove irrelevant data features and prevents
overfitting. This allows features with weak influence to be clearly identified as the
coefficients of less important variables are shrunk toward zero.
In this guide, we will understand the core concepts of lasso regression as well as how it
works to mitigate overfitting.
Table of Content
Understanding Lasso Regression
Bias-Variance Tradeoff in Lasso Regression
When to use Lasso Regression
Advantages of Lasso Regression
Disadvantages

Understanding Lasso Regression

Lasso Regression is a regularization technique used to prevent overfitting. It improves
linear regression by adding a penalty term to the standard regression equation. It works by
minimizing the sum of squared differences between the observed and predicted values by
fitting a line to the data.
However in real-world datasets features have strong correlations with each other known
as multicollinearity where Lasso Regression actually helps.
For example, if we're predicting house prices based on features like location, square
footage and number of bedrooms. Lasso Regression can identify most important features.
It might determine that location and square footage are the key factors influencing price
while others has less impact. By making coefficient for the bedroom feature to zero it
simplifies the model and improves its accuracy.
Bias-Variance Tradeoff in Lasso Regression
The bias-variance tradeoff refers to the balance between two types of errors in a model:
Bias: Error caused by over simplistic assumptions of the data.
Variance: Error caused by the model being too sensitive to small changes in the training
data.
When implementing Lasso Regression the L1 regularization penalty reduces variance by
making the coefficients of less important features to zero. This prevents overfitting by
ensuring model doesn't fit to noise in the data.
However increasing regularization strength i.e raising the lambda value can increase
bias. This happens because a stronger penalty can cause the model to oversimplify
making it unable to capture the true relationships in the data leading to underfitting.
Thus the goal is to choose right lambda value that balances both bias and variance
through cross-validation.

Bias Variance Tradeoff

Understanding Lasso Regression Working
Lasso Regression is an extension of linear regression. While traditional linear regression
minimizes the sum of squared differences between the observed and predicted values to
find the best-fit line, it doesn’t handle the complexity of real-world data well when many
factors are involved.
Ordinary Least Squares (OLS) Regression
It builds on Ordinary Least Squares (OLS) Regression method by adding a penalty
term. The basic equation for OLS is:
min RSS=Σ ( y ᵢ− ^y ᵢ)²
Where
y i is the observed value.
^y ᵢ is the predicted value for each data point i .
 Penalty Term for Lasso Regression
In Lasso regression a penalty term is added to the OLS equation. Penalty is the sum of the
absolute values of the coefficients. Updated cost function becomes:
RSS+ λ × ∑ ∣ β i ∣
Where,
β i represents the coefficients of the predictors
λ is the tuning parameter that controls the strength of the penalty. As λ increases more
coefficients are pushed towards zero
Shrinking Coefficients:
Key feature of Lasso is its ability to make coefficients of less important features to zero.
This removes irrelevant features from the model helps in making it useful for high-
dimensional data with many predictors relative to the number of observations.
Selecting the optimal λ :
Selecting correct lambda value is important. Cross-validation techniques are used to find
the optimal value helps in balancing model complexity and predictive performance.
Primary objective of Lasso regression is to minimize residual sum of squares
(RSS) along with a penalty term multiplied by the sum of the absolute values of the
coefficients.

Graphical Representation of Lasso Regression

In the plot, the equation for the Lasso Regression of cost function combines the residual
sum of squares (RSS) and an L1 p enalty on the coefficients β j .
RSS measures: Squared difference between expected and actual values is measured.
L1 penalty: Penalizes absolute values of the coefficients making some of them to zero
and simplifying the model. Strength of L1 penalty is controlled by the lambda parameter.
y-axis: Represents value of the cost function which Lasso Regression tries to minimize.
x-axis: Represents value of the lambda (λ) parameter which controls the strength of the
L1 penalty in the cost function.
Green to orange curve: This curve shows how the cost function (on the y-axis) changes
as lambda (on the x-axis) increases. As lambda grows the curve shifts from green to
orange. This indicates that the cost function value increases as the L1 penalty becomes
stronger helps in pushing more coefficients toward zero.
When to use Lasso Regression
Lasso Regression is useful in the following situations:
Feature Selection: It automatically selects most important features by reducing the
coefficients of less significant features to zero.
Collinearity: When there is multicollinearity it can help us by reducing the coefficients
of correlated variables and selecting only one of them.
Regularization: It helps preventing overfitting by penalizing large coefficients which is
useful when the number of predictors is large.
Interpretability: Compared to traditional linear regression models that have all features
lasso regression generates a model with fewer non-zero coefficients making model
simpler to understand.
For its implementation refer to:
Implementation of Lasso Regression From Scratch using Python
Lasso Regression in R Programming
Advantages of Lasso Regression
Feature Selection: It removes the need to manually select most important features hence
the developed regression model becomes simpler and more explainable.
Regularization: It constrains large coefficients so a less biased model is generated which
is robust and general in its predictions.
Interpretability: This creates another models helps in making them simpler to
understand and explain which is important in fields like healthcare and finance.
Handles Large Feature Spaces: It is effective in handling high-dimensional data such as
images and videos.
Disadvantages
Selection Bias: Lasso may randomly select one variable from a group of highly
correlated variables which leads to a biased model.
Sensitive to Scale: It is sensitive to features with different scales as they can impact the
regularization and affect model's accuracy.
Impact of Outliers: It can be easily affected by the outliers in the given data which
results to overfitting of the coefficients.
Model Instability: It can be unstable when there are many correlated variables which
causes it to select different features with small changes in the data.
Tuning Parameter Selection: Analyzing different λ (alpha) values may be problematic
but can be solved by cross-validation.

Evaluating Regression Models

Here is a list of 13 evaluation metrics
Mean Absolute Error (MAE)
Mean Bias Error (MBE)
Relative Absolute Error (RAE)
Mean Absolute Percentage Error (MAPE)
Mean Squared Error (MSE)
Root Mean Squared Error (RMSE)
Relative Squared Error (RSE)
Normalized Root Mean Squared Error (NRMSE)
Relative Root Mean Squared Error (RRMSE)
Root Mean Squared Logarithmic Error (RMSLE)
Hyber Loss
Log Cosh Loss
Quantile Loss
Mean Absolute Error (MAE)
Mean absolute error, or L1 loss, stands out as one of the simplest and easily
comprehensible loss functions and evaluation metrics. It computes by averaging the
absolute differences between predicted and actual values across the dataset.
Mathematically, it represents the arithmetic mean of absolute errors, focusing solely on
their magnitude, irrespective of direction. A lower MAE indicates superior model
accuracy.
MAE formula is:
where
y_i = actual value
y_hat_i = predicted value
n = sample size
Python Code:
import numpy as np
def mean_absolute_error(true, pred):
"""
Calculates the Mean Absolute Error (MAE) between the true and predicted values.
Args:
true ([Link]): An array of true values.
pred ([Link]): An array of predicted values.
Returns:
float: The Mean Absolute Error.
"""
mae = [Link]([Link](true - pred))
return mae
Pros of the MAE Evaluation Metric
It is an easy-to-calculate evaluation metric.
All the errors are weighted on the same scale since absolute values are taken.
It is useful if the training data has outliers as MAE does not penalize high errors caused
by outliers.
It provides an even measure of how well the model is performing.
Cons of the MAE evaluation metric
Sometimes the large errors coming from the outliers end up being treated as the same as
low errors.
MAE follows a scale-dependent accuracy measure using the same scale as the data being
measured. Hence it cannot be used to compare series’ using different measures.

One of the main disadvantages of MAE is that it is not differentiable at zero. Many
optimization algorithms tend to use differentiation to find the optimum value for
parameters in the evaluation metric.
It can be challenging to compute gradients in MAE.
Mean Bias Error (MBE)
In “Mean Bias Error,” bias reflects the tendency of a measurement process to
overestimate or underestimate a parameter. It has a single direction, positive or negative.
Positive bias implies an overestimated error, while negative bias implies an
underestimated error. Mean Bias Error (MBE) calculates the mean difference between
predicted and actual values, quantifying overall bias without considering absolute values.
Similar to MAE, MBE differs in not taking the absolute value. Caution is needed with
MBE, as positive and negative errors can cancel each other out.

The formula for MBE:

def mean_bias_error(true, pred):

bias_error = true - pred
mbe_loss = [Link]([Link](diff) / [Link])
return mbe_loss

MBE LOSS VS PREDICTION

Pros of the MBE Evaluation Metric

MBE is a good measure if you want to check the direction of the model (i.e. whether
there is a positive or negative bias) and rectify the model bias.
Cons of the MBE Evaluation Metric
It is not a good measure in terms of magnitude as the errors tend to compensate each
other.
It is not highly reliable because sometimes high individual errors produce low MBE.
As an evaluation metric, it can be consistently wrong in one direction. For example, if
you’re trying to predict traffic patterns it always shows lower traffic than what is
observed.

Relative Absolute Error (RAE)

Relative root mean square error Absolute Error is calculated by dividing the total
absolute error by the absolute difference between the mean and the actual value.
The formula for RAE is:

where y_bar is the mean of the n actual values.

RAE measures the performance of a predictive model and is expressed in terms of a
ratio. The value of RAE can range from zero to one. A good model will have values
close to zero, with zero being the best value. This error shows how the mean residual
relates to the mean deviation of the target function from its mean.
def relative_absolute_error(true, pred):
true_mean = [Link](true)
squared_error_num = [Link]([Link](true - pred))
squared_error_den = [Link]([Link](true - true_mean))
rae_loss = squared_error_num / squared_error_den
return rae_loss

RAE LOSS VS PREDICTION

Pros of the RAE Evaluation Metric
RAE can be used to compare models where errors are measured in different units.
In some cases, RAE is reliable as it offers protection from outliers.
Cons of the RAE Evaluation Metric
One main drawback of RAE is that it can be undefined if the reference forecast is
equal to the ground truth.
Mean Absolute Percentage Error (MAPE)
Calculate Mean Absolute Percentage Error (MAPE) by dividing the absolute
difference between the actual and predicted values by the actual value. This absolute
percentage is averaged across the dataset. MAPE, also known as Mean Absolute
Percentage Deviation (MAPD), increases linearly with error. Lower MAPE values
indicate better model performance.

def mean_absolute_percentage_error(true, pred):

abs_error = ([Link](true - pred)) / true
sum_abs_error = [Link](abs_error)
mape_loss = (sum_abs_error / [Link]) * 100
return mape_loss

MAPE LOSS VS PREDICTION

Pros of the MAPE Evaluation Metric

MAPE is independent of the scale of the variables since its error estimates are in
terms of percentage.
All errors are normalized on a common scale and it is easy to understand.
As MAPE uses absolute percentage errors, the problem of positive values and
negative values canceling each other out is avoided.
Cons of the MAPE Evaluation Metric
MAPE faces a critical problem when the denominator becomes zero, resulting in a
“division by zero” challenge.
MAPE exhibits bias by penalizing negative errors more than positive errors,
potentially favoring methods with lower values.
Mean Squared Error (MSE)
MSE is one of the most common regression loss functions and an important error metric.
In Mean Squared Error, also known as L2 loss, we calculate the error by squaring the
difference between the predicted value and actual value and averaging it across the
dataset.
You Should know Linear Regression in Machine Learning
MSE is also known as Quadratic loss as the penalty is not proportional to the error but to
the square of the error. Squaring the error gives higher weight to the outliers, which
results in a smooth gradient for small errors.
Optimization algorithms benefit from this penalization for large errors as it helps find the
optimum values for parameters using the least squares method. MSE will never be
negative since the errors are squared. The value of the error ranges from zero to infinity.
MSE increases exponentially with an increase in error. A good model will have an MSE
value closer to zero, indicating a better goodness of fit to the data.

def mean_squared_error(true, pred):

squared_error = [Link](true - pred)
sum_squared_error = [Link](squared_error)
mse_loss = sum_squared_error / [Link]
return mse_loss

MSE LOSS [Link]

Pros of the MSE Evaluation Metric
MSE values are expressed in quadratic equations. Hence when we plot it, we get
a gradient descent with only one global minima.
For small errors, it converges to the minima efficiently. There are no local minima.
MSE penalizes the model for having huge errors by squaring them.
It is particularly helpful in weeding out outliers with large errors from the model by
putting more weight on them.
Cons of the MSE Evaluation Metric
One of the advantages of MSE becomes a disadvantage when there is a bad prediction.
The sensitivity to outliers magnifies the high errors by squaring them.
MSE will have the same effect for a single large error as too many smaller errors. But
mostly we will be looking for a model which performs well enough on an overall level.
MSE is scale-dependent as its scale depends on the scale of the data. This makes it highly
undesirable to compare different measures.
When a new outlier is introduced into the data, the model will try to take in the outlier. By
doing so it will produce a different line of best fit which may cause the final results to be
skewed.
Root Mean Squared Error (RMSE)
Root Mean Square Error in Machine Learning (RMSE) is a popular metric used
in machine learning and statistics to measure the accuracy of a predictive model. It
quantifies the differences between predicted values and actual values, squaring the errors,
taking the mean, and then finding the square root. RMSE provides a clear understanding
of the model’s performance, with lower values indicating better predictive accuracy
relative root mean square error.
It is computed by taking the square root of MSE. RMSE is also called the Root Mean
Square Deviation. It measures the average magnitude of the errors and is concerned with
the deviations from the actual value. RMSE value with zero indicates that the model has a
perfect fit. The lower the RMSE, the better the model and its predictions. A higher
relative root mean square error in machine learning indicates that there is a large
deviation from the residual to the ground truth. RMSE can be used with different features
as it helps in figuring out if the feature is improving the model’s prediction or not.

def root_mean_squared_error(true, pred):

squared_error = [Link](true - pred)
sum_squared_error = [Link](squared_error)
rmse_loss = [Link](sum_squared_error / [Link])
return rmse_loss

Pros of the RMSE Evaluation Metric

RMSE is easy to understand.
It serves as a heuristic for training models.
It is computationally simple and easily differentiable which many optimization algorithms
desire.
RMSE does not penalize the errors as much as MSE does due to the square root.
Cons of the RMSE Metric
Like MSE, RMSE is dependent on the scale of the data. It increases in magnitude if the
scale of the error increases.
One major drawback of RMSE is its sensitivity to outliers and the outliers have to be
removed for it to function properly.
RMSE increases with an increase in the size of the test sample. This is an issue when we
calculate the results on different test samples.

Relative Squared Error (RSE)

To calculate Relative Squared Error, you take the Mean Squared Error (MSE) and divide
it by the square of the difference between the actual and the mean of the data. In other
words, we divide the MSE of our model by the MSE of a model that uses the mean as the
predicted value.

def relative_squared_error(true, pred):

true_mean = [Link](true)
squared_error_num = [Link]([Link](true - pred))
squared_error_den = [Link]([Link](true - true_mean))
rse_loss = squared_error_num / squared_error_den
return rse_loss
The output value of RSE is expressed in terms of ratio. It can range from zero to one. A
good model should have a value close to zero while a model with a value greater than 1 is
not reasonable.

RSE LOSS VS PREDICTION

Pros of the RSE Evaluation Metric
RSE is not scale-dependent. Hence it can be used to compare models where errors are
measured in different units.
RSE is not sensitive to the mean and the scale of predictions.
Cons of the RSE Evaluation Metric
RSE does not distinguish between underestimation and overestimation errors, as it only
considers the squared differences between y_pred and true values. This means that a
model that consistently overestimates or underestimates can still have a low RSE value.
Like the Mean Squared Error (MSE), RSE is also heavily influenced by outliers in the
data points. A few extreme errors can significantly increase the RSE value, even if the
model performs well on the majority of the data.
When the RSE value is much greater than 1, it becomes difficult to interpret the degree of
poor performance. An RSE of 2 or 10 indicates that the model performs worse than the
mean prediction baseline, but the magnitude of the difference is not clear.
The interpretation of RSE depends on the performance of the mean prediction baseline
for the target values. If the mean prediction itself is a poor baseline, the RSE values may
not provide a meaningful comparison.
Although RSE is scale-independent in terms of the target variable’s units, it can still be
sensitive to the scale of the target values. If the target variable has a small range, small
errors can result in large RSE values, making the metric less informative.
For regression analysis problems with strictly non-negative target values (e.g., count data
or positive values), the mean prediction baseline may not be a meaningful or appropriate
baseline for comparison with the independent variables.
The interpretation of RSE can also depend on the specific test set used for evaluation. If
the test set is not representative of the overall data distribution, the RSE values may not
accurately reflect the model’s performance.
Normalized Root Mean Squared Error (NRMSE)
The Normalized RMSE is generally computed by dividing a scalar value. It can be in
different ways like,
RMSE / maximum value in the series
RMSE / mean
RMSE / difference between the maximum and the minimum values (if mean is zero)
RMSE / standard deviation
RMSE / interquartile range

# implementation of NRMSE with standard deviation

def normalized_root_mean_squared_error(true, pred):
squared_error = [Link]((true - pred))
sum_squared_error = [Link](squared_error)
rmse = [Link](sum_squared_error / [Link])
nrmse_loss = rmse/[Link](pred)
return nrmse_loss
 NRMSE LOSS VS. PREDICTION
Opting for the interquartile range can be the most suitable choice, especially when dealing
with outliers. NRMSE proves effective for comparing models with different dependent
variables or when modifications like log transformation or standardization occur. This
metric addresses scale-dependency issues, facilitating comparisons across models of
varying scales or datasets.
Relative Root Mean Squared Error (RRMSE)
Relative Root Mean Squared Error (RRMSE) is a variant of Root Mean Square Error in
Machine Learning (RMSE), gauging predictive model accuracy relative to the target
variable range. It normalizes RMSE by the target variable range and presents it as a
percentage for easy cross-dataset or cross-variable comparison. RRMSE, a dimensionless
form of RMSE, scales residuals against actual values, allowing comparison of different
measurement techniques.
Excellent when RRMSE < 10%
Good when RRMSE is between 10% and 20%
Fair when RRMSE is between 20% and 30%
Poor when RRMSE > 30%
def relative_root_mean_squared_error(true, pred):
num = [Link]([Link](true - pred))
den = [Link]([Link](pred))
squared_error = num/den
rrmse_loss = [Link](squared_error)
return rrmse_loss

RRMSE LOSS VS PREDICTION

Root Mean S quared Logarithmic Error (RMSLE)
Root Mean Squared Logarithmic Error is calculated by applying log to the actual and the
predicted values and then taking their differences. RMSLE is robust to outliers where the
small and the large errors are treated evenly.
It penalizes the model more if the predicted value is less than the actual value while the
model is less penalized if the predicted value is more than the actual value. It does not
penalize high errors due to the log. Hence the model has a larger penalty for
underestimation than overestimation. This can be helpful in situations where we are not
bothered by overestimation but underestimation is not acceptable.

def root_mean_squared_log_error(true, pred):

square_error = [Link](([Link](true + 1) - [Link](pred + 1)))
mean_square_log_error = [Link](square_error)
rmsle_loss = [Link](mean_square_log_error)
return rmsle_loss
Pros of the RMSLE Evaluation Metric
RMSLE is not scale-dependent and is useful across a range of scales.
It is not affected by large outliers.
It considers only the relative error between the actual value and the predicted value.
Cons of the RMSLE Evaluation Metric
It has a biased penalty where it penalizes underestimation more than overestimation.
Huber Loss
What if you want a function that learns about the outliers as well as ignores them? Well,
Huber loss is the one for you. Huber loss is a combination of both linear and quadratic
scoring methods. It has a hyperparameter delta (𝛿) which can be tuned according to the
data. The loss will be linear (L1 loss) for values above delta and quadratic (L2 loss) for
values below it. It balances and combines good properties of both MAE (Mean Absolute
Error) and MSE (Mean Squared Error).
In other words, for loss values less than delta, MSE will be used and for loss values
greater than delta, MAE will be used. The choice of delta (𝛿) is extremely critical
because it defines our choice of the outlier. Huber loss reduces the weight we put on
outliers for larger loss values by using MAE while for smaller loss values it maintains a
quadratic function using MSE.

def huber_loss(true, pred, delta):

huber_mse = 0.5 * [Link](true - pred)
huber_mae = delta * ([Link](true - pred) - 0.5 * ([Link](delta)))
return [Link]([Link](true - pred) <= delta, huber_mse, huber_mae)
Pros of the Huber Loss Evaluation Metric
It is differentiable at zero.
Outliers are handled properly due to the linearity above the delta.
The hyperparameter, 𝛿 can be tuned to maximize model accuracy.
Cons of the Huber Loss Evaluation Metric
The additional conditionals and comparisons make Huber loss computationally expensive
for large datasets.
To maximize model accuracy, 𝛿 needs to be optimized and it is an iterative process.
It is differentiable only once.
Log Cosh Loss
Log cosh calculates the logarithm of the hyperbolic cosine of the error. This function is
smoother than quadratic loss. It works like MSE but is not affected by large prediction
errors. It is quite similar to Huber loss in the sense that it is a combination of both linear
and quadratic scoring methods.

def log_cosh(true, pred):

logcosh = [Link]([Link](pred - true))
logcosh_loss = [Link](logcosh)
return logcosh_loss

Pros of the Log Cosh Loss Evaluation Metric

It has the advantages of Huber loss while being twice differentiable everywhere. Some
optimization algorithms like XGBoost favor double differentials over functions like
Huber which can be differentiable only once.
It requires fewer computations than Huber.

Cons of the Log Cosh Loss Evaluation Metric

It is less adaptive as it follows a fixed scale.
Compared to Huber loss, the derivation is more complex and requires much in-depth
study.
Quantile Loss
The quantile regression loss function is applied to predict quantiles. The quantile is the
value that determines how many values in the group fall below or above a certain limit. It
estimates the conditional median or quantile of the response (dependent) variables across
values of the predictor (independent) variables. The loss function is an extension of MAE
except for the 50th percentile, where it is MAE. It provides prediction intervals even for
residuals with non-constant variance and it does not assume a particular parametric
distribution for the response.
γ represents the required quantile. The quantile values are selected based on how we want
to weigh the positive and the negative errors. Unlike the squared difference loss used in
linear regression models, this loss function is based on absolute differences.
Loss Function
In the loss function above, γ has a value between 0 and 1. When there is an
underestimation, the first part of the formula will dominate and for overestimation, the
second part will dominate. The chosen value of quantile(γ) gives different penalties for
over-prediction and under prediction. When γ = 0.5, underestimation and overestimation
are penalized by the same factor, and the median is obtained. When the value of γ is
larger, overestimation is penalized more than underestimation. For example, when γ =
0.75 the model will penalize overestimation and it will cost three times as much as
underestimation. Optimization algorithms based on gradient descent learn from the
quantiles instead of the mean.

𝛾 represents the required quantile. The quantiles values are selected based on how we
want to weigh the positive and the negative errors.
In the loss function above, 𝛾 has a value between 0 and 1. When there is an
underestimation, the first part of the formula will dominate and for overestimation, the
second part will dominate. The chosen value of quantile(𝛾) gives different penalties for
over-prediction and under prediction. When 𝛾 = 0.5, underestimation and overestimation
are penalized by the same factor and the median is obtained. When the value of 𝛾 is
larger, overestimation is penalized more than underestimation. For example, when 𝛾 =
0.75 the model will penalize overestimation and it will cost three times as much as
underestimation. Optimization algorithms based on gradient descent learn from the
quantiles instead of the mean.

def quantile_loss(true, pred, gamma):

val1 = gamma * [Link](true - pred)
val2 = (1-gamma) * [Link](true - pred)
q_loss = [Link](true >= pred, val1, val2)
return q_loss
Pros of the Quantile Loss Evaluation Metric
It is particularly useful when we are predicting an interval instead of point estimates.
This function can also be used to calculate prediction intervals in neural nets and tree-
based models.
It is robust to outliers.
Cons of the Quantile Loss Evaluation Metric
Quantile loss is computationally intensive.
If we use a squared loss to measure the efficiency or if we are to estimate the mean, then
quantile loss will be worse.
Conclusion
This comprehensive guide navigated through diverse regression loss functions, shedding
light on their applications, advantages, and drawbacks. The article demystified complex
metrics like MAE, MBE, RAE, MAPE, MSE, RMSE (the root mean squared error), RSE,
NRMSE, RRMSE, and RMSLE, and introduced specialized losses like Huber, Log Cosh,
and Quantile. It emphasized the nuanced factors influencing loss function selection, from
algorithm types to outlier handling. Additionally, it covered the coefficient of
determination (R-squared), and r2_score function from [Link] import, and
adjusted r-squared, which are important evaluation metrics for assessing the performance
of machine learning algorithms in regression problems.

Model Selection for Machine Learning

Machine learning (ML) is a field that enables computers to learn patterns from data and
make predictions without being explicitly programmed. However, one of the most crucial
aspects of machine learning is selecting the right model for a given problem. This process
is called model selection. The choice of model significantly affects the accuracy,
efficiency and reliability of predictions. A bad model can cause overfitting or
underfitting and sometimes even lead to increased computational costs.
In this article, we are going to deeply explore into the process of model selection, its
importance and techniques used to determine the best-performing machine learning
model for different problems.

Importance of Model Selection

Model selection is a key step in machine learning because it affects how well a system
can learn from data and make accurate predictions. Different models have different ways
of processing data and choosing the right one ensures that the system works efficiently. A
simple model cannot capture details and has poor accuracy, while a model too complex
might overfit that is doing very well on training data but fails on new data. The goal is to
find a model that learns patterns effectively without being too simple or too complex.
Proper model selection involves experimenting with different models and comparing their
performance using evaluation metrics such as accuracy, precision, recall or mean squared
error. These metrics help in determining which model is best suited for a given task.
Apart from performance metrics, other factors such as training time, dataset size and
available computing power also play a crucial role in choosing the right model.
Selecting an appropriate model not only improves prediction accuracy but also enhances
efficiency, making the system faster and more reliable. This ensures that AI-driven
applications perform well in real-world scenarios.
Steps in Model Selection
Understanding the Problem and Data
Before selecting a model, it is important to first analyze the problem we are trying to
solve. The initial step is to determine whether it is a regression problem, where the goal is
to predict continuous values like house prices. If the task involves predicting categorical
labels, such as distinguishing between spam and non-spam emails, it falls
under classification problem. On the other hand, if the objective is to group similar data
points, like segmenting customers based on behavior, then it is a clustering problem.
Understanding the type of problem helps in choosing the most suitable machine learning
model.
Another important point is a bit about the nature of the dataset itself. One has to check for
missing values, the number of numerical and categorical variables and the distribution of
data. Understanding the type of problem and the dataset helps in choosing the most
suitable machine learning model.
Selecting Suitable Models
After understanding the problem, we then choose a best model that should solve the
problem. Different types of models work better for different kinds of problems:
For Regression: Linear Regression, Decision Trees, Random Forest, Neural Networks.
For Classification: Logistic Regression, Support Vector Machines (SVM), k-Nearest
Neighbors (k-NN), Neural Networks.
For Clustering: k-Means, Hierarchical Clustering, DBSCAN.

Model Evaluation
Once we have identified the right models, we must rank each one according to how well
it does the job. The most common method is to split the dataset into two parts.
Training Set: The data used to train a machine learning model by learning patterns and
relationships.
Testing Set: This checks how well a model performs over new, unseen data.
We use k-fold cross-validation to further improve the evaluation. In k-fold cross-
validation, the data is split into k subsets. The model is trained on k-1 subsets and tested
on the remaining one, repeating the process k times. This way, our evaluation is not
biased by a particular train-test split.
Different machine learning problems require different evaluation metrics.
For Regression Problems: We make use of Mean Squared Error (MSE), Mean Absolute
Error (MAE) and R-squared.
For Classification Problems: We make use of Accuracy, Precision, Recall and F1-score.
After evaluating the models, we compare them to identify the one that satisfies
performance and computational efficiency.
Model Selection Techniques in Machine Learning
Grid Search
One of the simplest and most commonly used model selection techniques is grid search.
In this approach, systematically different combinations of hyperparameters are tried and
that gives the best performance chosen. It can be effective, but the main drawback will be
computationally intensive, especially for complex models and many parameters.
Random Search
Similar to grid search, random search doesn't check all possible combinations. Instead, it
randomly chooses a subset of the hyperparameter combinations. The random search
method often runs much faster than the grid search method and yet achieves equally good
results.
Bayesian Optimization
Bayesian optimization is a smarter approach to model selection. Instead of just randomly
searching for the best hyperparameters, it uses probability models to predict which
parameters are likely to perform best and focuses on evaluating those. This method is
efficient and often finds better results than grid or random search.
Cross-Validation Based Selection
This method involves using cross-validation to evaluate multiple models and selecting the
one with the best average performance. Instead of relying on a single train-test
split, cross-validation divides the dataset into multiple parts and trains the model on
different subsets. This helps to ensure that the model’s performance is not just due to a
specific split of data. By averaging the results from different splits, we get how well the
model will perform on new, unseen data. This approach reduces the risk of overfitting and
helps in selecting a good model.
Bagging
What is Bagging?
Bagging is a machine learning ensemble method aimed at improving the reliability
and accuracy of predictive models. It involves generating several subsets of the training
data using random sampling with replacement. These subsets are then used to train
multiple base models, such as decision trees or neural networks.
When making predictions, the outputs from these base models are combined, often
through averaging (for regression) or voting (for classification), to produce the final
prediction. Bagging reduces overfitting by creating diversity among the models and
enhances overall performance by lowering variance and increasing robustness.
Implementation Steps of Bagging
Here’s a general outline of implementing Bagging:
Dataset Preparation: Clean and preprocess your dataset. Split it into training and test
sets.
Bootstrap Sampling: Randomly sample from the training data with replacement to create
multiple bootstrap samples. Each sample typically has the same size as the original
dataset.
Model Training: Train a base model (e.g., decision tree, neural network) on each
bootstrap sample. Each model is trained independently.
Prediction Generation: Use each trained model to predict the test data.
Combining Predictions: Aggregate the predictions from all models using methods like
majority voting for classification or averaging for regression.
Evaluation: Assess the ensemble’s performance on the test data using metrics like
accuracy, F1 score, or mean squared error.
Hyperparameter Tuning: Adjust the hyperparameters of the base models or the
ensemble as needed, using techniques like cross-validation.
Deployment: Once satisfied with the ensemble’s performance, deploy it to make
predictions on new data.
Understanding Ensemble Learning
To increase performance overall, ensemble learning integrates the predictions of several
models. By combining the insights from multiple models, this method frequently
produces forecasts that are more accurate than those of any one model alone.
Popular ensemble methods include:
Bagging: Involves training multiple base models on different subsets of the training data
created through random sampling with replacement.
Boosting: A sequential method where each model focuses on correcting the errors of its
predecessors, with popular algorithms like AdaBoost and XGBoost.
Random Forest: An ensemble of decision trees, each trained on a random subset of
features and data, with final predictions made by aggregating individual tree predictions.
Stacking: Combines the predictions of multiple base models using a meta-learner to
produce the final prediction.
Benefits of Bagging
Variance Reduction: By training multiple models on different data subsets, Bagging
reduces variance, leading to more stable and reliable predictions.
Overfitting Mitigation: The diversity among base models helps the ensemble generalize
better to new data.
Robustness to Outliers: Aggregating multiple models’ predictions reduces the impact
of outliers and noisy data points.
Parallel Training: Training individual models can be parallelized, speeding up the
process, especially with large datasets or complex models.
Versatility: Bagging can be applied to various base learners, making it a flexible
technique.
Simplicity: The concept of random sampling with replacement and combining
predictions is easy to understand and implement.

Applications of Bagging
Bagging, also known as Bootstrap Aggregating, is a versatile technique used across many
areas of machine learning. Here’s a look at how it helps in various tasks:
Classification: Bagging combines predictions from multiple classifiers trained on
different data splits, making the overall results more accurate and reliable.
Regression: In regression problems, bagging helps by averaging the outputs of multiple
regressors, leading to smoother and more accurate predictions.
Anomaly Detection: By training multiple models on different data subsets, bagging
improves how well anomalies are spotted, making it more resistant to noise and outliers.
Feature Selection: Bagging can help identify the most important features by training
models on different feature subsets. This reduces overfitting and improves model
performance.
Imbalanced Data: In classification problems with uneven class distributions, bagging
helps balance the classes within each data subset. This leads to better predictions for less
frequent classes.
Building Powerful Ensembles: Bagging is a core part of complex ensemble methods like
Random Forests and Stacking. It trains diverse models on different data subsets to
achieve better overall performance.
Time-Series Forecasting: Bagging improves the accuracy and stability of time-series
forecasts by training on various historical data splits, capturing a wider range of patterns
and trends.
Clustering: Bagging helps find more reliable clusters, especially in noisy or high-
dimensional data. This is achieved by training multiple models on different data subsets
and identifying consistent clusters across them.
Differences Between Bagging and Boosting

Let us now explore difference between bagging and boosting.

Feature Bagging Boosting

Type of
Parallel ensemble method Sequential ensemble method
Ensemble
Trained in parallel on Trained sequentially, correcting
Base Learners
different subsets of the data previous mistakes
Weighting of All data points equally Misclassified points given more
Data weighted weight
Reduction of
Mainly reduces variance Mainly reduces bias
Bias/Variance
Handling of
Resilient to outliers More sensitive to outliers
Outliers
Robustness Generally robust Less robust to outliers
Model Training Generally slower due to
Can be parallelized
Time sequential training
AdaBoost, Gradient Boosting,
Examples Random Forest
XGBoost

Conclusion

Bagging is a powerful yet simple ensemble method that strengthens model

performance by lowering variation, enhancing generalization, and increasing
resilience. Its ease of use and ability to train models in parallel make it popular
across various applications.

Elevate your machine learning skills with our ‘Mastering Bagging Techniques ‘
course! Learn how to leverage this powerful ensemble method to boost model
performance, reduce overfitting, and tackle real-world challenges with
confidence—enroll today and unlock your full potential!
Ensemble Methods in Machine learning
Ensemble methods combine multiple machine learning models to improve overall
accuracy and reliability. They work by aggregating the predictions of "weak learners" to
create a single, more powerful "strong learner" that is less prone to individual model
errors. The three main types are bagging, boosting, and stacking, which differ in how they
combine models.
In machine learning, a model is trained to make predictions or classify data based on
patterns in a dataset. However, a single model can sometimes have limitations, such as
overfitting, where the model performs well on training data but poorly on new data.
Ensemble methods offer a solution by combining multiple models to improve accuracy
and reduce errors. By using the strengths of different models together, ensemble methods
create a more reliable and robust prediction system than any single model on its own.
What are Ensemble Methods?
Ensemble methods are techniques in machine learning that combine the predictions of
multiple models to improve overall accuracy. The idea is simple: rather than relying on a
single model, ensemble methods leverage the strengths of several models to create a more
powerful prediction system.
By combining different models, ensemble methods address some of the limitations of
individual models. For example, if one model has a tendency to overfit or underperform
on certain data points, the ensemble can balance out these weaknesses and make more
accurate predictions. In essence, ensemble methods help create a “team” of models,
working together for better results.
Why Use Ensemble Learning?
Ensemble learning offers several benefits that make it popular in machine learning:
Improved Accuracy and Generalization – By combining multiple models, ensemble
methods often achieve higher accuracy than individual models, making predictions more
reliable.
Reduced Variance and Overfitting – Ensembles help reduce overfitting by balancing
out the weaknesses of individual models. This makes ensemble models perform better on
new, unseen data.
Robustness to Noise in Data – Ensemble methods are more resilient to noisy or complex
data. With multiple models contributing to the final prediction, they can better handle
variations and inconsistencies in the data.
Types of Ensemble Models
Ensemble models combine the predictions of multiple individual models to improve
accuracy and stability. Here’s a breakdown of popular ensemble methods, each offering
unique ways to enhance model performance:
Bagging (Bootstrap Aggregation)
Concept: Bagging creates multiple versions of the original dataset by randomly sampling
data with replacement. Each subset is used to train a different model, which reduces
variance and enhances stability.
How It Works: Multiple models (usually decision trees) are trained independently on
different subsets of the data. The predictions are then averaged (for regression) or voted
on (for classification) to make the final decision.

Vnm Popular Example: Random Forest

Purpose: Combines multiple decision trees to create a powerful and more
generalizable model.
Implementation: In Python, use RandomForestClassifier or
RandomForestRegressor from scikit-learn.

Boosting
Concept: Boosting aims to build a strong model by training models sequentially. Each
new model focuses on correcting the errors of the previous ones, progressively reducing
bias.
How It Works: Models are trained one after another, with each new model adjusting to
the errors of its predecessors. By focusing on misclassified data points, boosting creates a
robust model.
Popular Examples:
 AdaBoost: Adjusts weights for misclassified points, emphasizing harder-to-
classify examples.
Gradient Boosting: Uses gradient descent to minimize error, popular for complex
tasks.
Implementation: In Python, use AdaBoostClassifier, GradientBoostingClassifier,
or libraries like XGBoost and LightGBM for optimized boosting algorithms.

Stacking
Concept: Stacking combines multiple models by using a meta-model (or “super learner”)
to integrate their outputs. This meta-model learns from each individual model’s
predictions to make a final, optimized prediction.
How It Works: The base models generate predictions on the training data, and the meta-
model then learns from these predictions to make a final decision. Stacking leverages the
strengths of diverse models to increase accuracy.
Popular Example: Combining decision trees, logistic regression, and support vector
machines with a meta-model (such as linear regression) to produce the final result.
Implementation: Use StackingClassifier or StackingRegressor from scikit-learn to stack
models effectively.
Voting
Concept: Voting aggregates predictions from multiple models, making a final decision
based on a majority or weighted vote.
How It Works: In hard voting, the final class is determined by majority vote, while soft
voting takes the average probabilities of each class. This method is commonly used in
classification tasks where predictions from models like logistic regression, k-nearest
neighbors, and decision trees are combined.
Popular Example: Hard voting for discrete class predictions or soft voting for
probability-based outcomes.
Implementation: Use the VotingClassifier in scikit-learn, which supports both hard and
soft voting.
Weighted Ensemble
Concept: Weighted ensembles assign varying importance to models based on their
accuracy or reliability, giving more influence to models that perform better.
How It Works: Each model is assigned a weight proportional to its accuracy. The final
prediction is then a weighted combination, with higher-performing models contributing
more to the outcome.
Popular Example: Custom ensembles where models with greater accuracy receive
higher weights to boost overall performance.
Implementation: Weighted ensembles can be created using custom code or by specifying
weights in scikit-learn’s VotingClassifier.
Main Challenge for Developing Ensemble Models?
While ensemble methods are powerful, they come with their own set of challenges that
can impact model development and usability:
Increased Computational Cost – Ensemble models often require multiple individual
models, which can significantly increase computation time and resources. Training and
deploying these models can be more resource-intensive compared to single models,
especially for large datasets.
Interpretability – Ensemble models, particularly complex ones like Random Forests or
stacked models, can be difficult to interpret. Unlike simpler models, it’s challenging to
understand how each individual model contributes to the final decision, which can make
the ensemble harder to explain to stakeholders.
Complexity in Implementation – Implementing ensemble models can be technically
challenging, especially when combining different types of models. Ensuring that each
model works together efficiently requires careful design and tuning, which may be
difficult for beginners or small teams.
Risk of Overfitting with Complex Ensembles – Although ensembles are designed to
reduce overfitting, overly complex ensembles can still overfit if not properly tuned. When
too many models are combined without careful evaluation, there’s a risk of creating a
model that performs well on training data but poorly on new data.
Conclusion
Ensemble methods combine multiple models to improve accuracy and robustness, making
them effective in complex machine learning tasks. Techniques like bagging, boosting, and
stacking help address limitations of single models, enhancing performance.
While ensembles come with challenges like higher computational costs and reduced
interpretability, these can be managed with careful tuning. Ensemble methods will
continue to play a vital role in advancing machine learning, enabling reliable and high-
performance models across various fields.

Unit III - Supervised Learning – II: Classification – Logistic Regression – Decision Tree
Regression and Classification – Random Forest Regression and Classification – Support
Vector Machine Regression and Classification - Evaluating Classification Models.

Classification in Machine Learning

Classification may be defined as the process of predicting class or category from
observed values or given data points. The categorized output can have the form such as
"Black" or "White" or "spam" or "no spam".
Classification in machine learning is a supervised learning technique where an algorithm
is trained with labeled data to predict the category of new data.
Mathematically, classification is the task of approximating a mapping function (f) from
input variables (X) to output variables (Y). It is basically belongs to the supervised
machine learning in which targets are also provided along with the input data set.
An example of classification problem can be the spam detection in emails. There can be
only two categories of output, "spam" and "no spam"; hence this is a binary type
classification.
To implement this classification, we first need to train the classifier. For this example,
"spam" and "no spam" emails would be used as the training data. After successfully train
the classifier, it can be used to detect an unknown email.
Types of Learners in Classification
We have two types of learners in respective to classification problems −
Lazy Learners − As the name suggests, such kind of learners waits for the testing data to
be appeared after storing the training data. Classification is done only after getting the
testing data. They spend less time on training but more time on predicting. Examples of
lazy learners are K-nearest neighbor and case-based reasoning.
Eager Learners − As opposite to lazy learners, eager learners construct classification
model without waiting for the testing data to be appeared after storing the training data.
They spend more time on training but less time on predicting. Examples of eager learners
are Decision Trees, Nave Bayes and Artificial Neural Networks (ANN).
Classification Algorithms in Machine Learning
The classification algorithm is a type of supervised learning technique that involves
predicting a categorical target variable based on a set of input features. It is commonly
used to solve problems such as spam detection, fraud detection, image recognition,
sentiment analysis, and many others.
The goal of a classification model is to learn a mapping function (f) between the input
features (X) and the target variable (Y). This mapping function is often represented as a
decision boundary, which separates different classes in the input feature space. Once the
model is trained, it can be used to predict the class of new, unseen examples.
The followings are some important ML classification algorithms −
Logistic Regression
K-Nearest Neighbors (KNN)
Support Vector Machine (SVM)
Decision Tree
Nave Bayes
Random Forest
We will be discussing all these classification algorithms in detail in further chapters.
However let's discuss these algorithms in brief as follows –
zss
Logistic Regression
Logistic Regression is a popular algorithm used for binary classification problems, where
the target variable is categorical with two classes. It models the probability of the target
variable given the input features and predicts the class with the highest probability.
Logistic regression is a type of generalized linear model, where the target variable follows
a Bernoulli distribution. The model consists of a linear function of the input features,
which is transformed using the logistic function to produce a probability value between 0
and 1.

K-Nearest Neighbors (KNN)

K-Nearest Neighbors (KNN) is a supervised learning algorithm that can be used for both
classification and regression problems. The main idea behind KNN is to find the k-nearest
data points to a given test data point and use these nearest neighbors to make a prediction.
The value of k is a hyperparameter that needs to be tuned, and it represents the number of
neighbors to consider.
For classification problems, the KNN algorithm assigns the test data point to the class that
appears most frequently among the k-nearest neighbors. In other words, the class with the
highest number of neighbors is the predicted class.
For regression problems, the KNN algorithm assigns the test data point the average of the
k-nearest neighbors' values.
Support Vector Machine (SVM)
Support Vector Machines (SVMs) are powerful yet flexible supervised machine learning
algorithm which is used for both classification and regression. But generally, they are
used in classification problems. In 1960s, SVMs were first introduced but later they got
refined in 1990 also. SVMs have their unique way of implementation as compared to
other machine learning algorithms. Now a days, they are extremely popular because of
their ability to handle multiple continuous and categorical variables.

Decision Tree
The Decision Tree algorithm is a hierarchical tree-based algorithm that is used to classify
or predict outcomes based on a set of rules. It works by splitting the data into subsets
based on the values of the input features. The algorithm recursively splits the data until it
reaches a point where the data in each subset belongs to the same class or has the same
value for the target variable. The resulting tree is a set of decision rules that can be used
to make predictions or classify new data.
Nave Bayes
The Nave Bayes algorithm is a classification algorithm based on Bayes' theorem. The
algorithm assumes that the features are independent of each other, which is why it is
called "naive." It calculates the probability of a sample belonging to a particular class
based on the probabilities of its features. For example, a phone may be considered as
smart if it has touch-screen, internet facility, good camera, etc. Even if all these features
are dependent on each other, but all these features independently contribute to the
probability of that the phone is a smart phone.
Random Forest
Random Forest is a machine learning algorithm that uses an ensemble of decision trees to
make predictions. The algorithm was first introduced by Leo Breiman in 2001. The key
idea behind the algorithm is to create a large number of decision trees, each of which is
trained on a different subset of the data. The predictions of these individual trees are then
combined to produce a final prediction.
Applications of Classification in Machine Learning
Some of the most important applications of classification algorithms are as follows −
Speech Recognition
Handwriting Recognition
Biometric Identification
Document Classification
Image Classification
Spam Filtering
Fraud Detection
Facial Recognition
Building a Classication Model in Machine Learning
Let us now take a look at the steps involved in building a classification model −
Data Preparation
The first step is to collect and preprocess the data. This involves cleaning the data,
handling missing values, and converting categorical variables to numerical values.
Feature Extraction/Selection
The next step is to extract or select relevant features from the data. This is an important
step because the quality of the features can greatly impact the performance of the model.
Some common feature selection techniques include correlation analysis, feature
importance ranking, and principal component analysis.
Model Selection
Once the features are selected, the next step is to choose an appropriate classification
algorithm. There are many different algorithms to choose from, each with its own
strengths and weaknesses. Some popular algorithms include logistic regression, decision
trees, random forests, support vector machines, and neural networks
Model Training
After selecting a suitable algorithm, the next step is to train the model on the labeled
training data. During training, the model learns the mapping function between the input
features and the target variable. The model parameters are adjusted iteratively to
minimize the difference between the predicted outputs and the actual outputs.
Model Evaluation
Once the model is trained, the next step is to evaluate its performance on a separate set of
validation data. This is done to estimate the model's accuracy and generalization
performance. Common evaluation metrics include accuracy, precision, recall, F1-score,
and area under the receiver operating characteristic (ROC) curve.
Hyperparameter Tuning
In many cases, the performance of the model can be further improved by tuning its
hyperparameters. Hyperparameters are settings that are chosen before training the model
and control aspects such as the learning rate, regularization strength, and the number of
hidden layers in a neural network. Grid search, random search, and Bayesian optimization
are some common techniques used for hyperparameter tuning.
Model Deployment
Once the model has been trained and evaluated, the final step is to deploy it in a
production environment. This involves integrating the model into a larger system, testing
it on realworld data, and monitoring its performance over time.
Introduction to Logistic Regression
Logistic regression is a supervised learning classification algorithm used to predict the
probability of a target variable. The nature of target or dependent variable is dichotomous,
which means there would be only two possible classes.
In simple words, the dependent variable is binary in nature having data coded as either 1
(stands for success/yes) or 0 (stands for failure/no).
Mathematically, a logistic regression model predicts P(Y=1) as a function of X. It is one
of the simplest ML algorithms that can be used for various classification problems such as
spam detection, Diabetes prediction, cancer detection etc.

Types of Logistic Regression

Generally, logistic regression means binary logistic regression having binary target
variables, but there can be two more categories of target variables that can be predicted by
it. Based on those number of categories, Logistic regression can be divided into following
types −
Binary or Binomial
In such a kind of classification, a dependent variable will have only two possible types
either 1 and 0. For example, these variables may represent success or failure, yes or no,
win or loss etc.
Multinomial
In such a kind of classification, dependent variable can have 3 or more possible unordered
types or the types having no quantitative significance. For example, these variables may
represent "Type A" or "Type B" or "Type C".
Ordinal
In such a kind of classification, dependent variable can have 3 or more possible ordered
types or the types having a quantitative significance. For example, these variables may
represent "poor" or "good", "very good", "Excellent" and each category can have the
scores like 0,1,2,3.
Logistic Regression Assumptions
Before diving into the implementation of logistic regression, we must be aware of the
following assumptions about the same −
In case of binary logistic regression, the target variables must be binary always and the
desired outcome is represented by the factor level 1.
There should not be any multi-collinearity in the model, which means the independent
variables must be independent of each other .
We must include meaningful variables in our model.
We should choose a large sample size for logistic regression.
Binary Logistic Regression Model
The simplest form of logistic regression is binary or binomial logistic regression in which
the target or dependent variable can have only 2 possible types either 1 or 0. It allows us
to model a relationship between multiple predictor variables and a binary/binomial target
variable. In case of logistic regression, the linear function is basically used as an input to
another function such as in the following relation −
hθ(x)=g(θTx)0hθ1hθ(x)=g(θTx)0hθ1
Here, is the logistic or sigmoid function which can be given as follows −
g(z)=11+e−z=θTg(z)=11+e−z=θT
To sigmoid curve can be represented with the help of following graph. We can see the
values of y-axis lie between 0 and 1 and crosses the axis at 0.5.
The classes can be divided into positive or negative. The output comes under the
probability of positive class if it lies between 0 and 1. For our implementation, we are
interpreting the output of hypothesis function as positive if it is 0.5, otherwise negative.
We also need to define a loss function to measure how well the algorithm performs using
the weights on functions, represented by theta as follows −
=()=()
J(θ)=1m.(−yTlog(h)−(1−y)Tlog(1−h))J(θ)=1m.(−yTlog(h)−(1−y)Tlog(1−h))
Now, after defining the loss function our prime goal is to minimize the loss function. It
can be done with the help of fitting the weights which means by increasing or decreasing
the weights. With the help of derivatives of the loss function w.r.t each weight, we would
be able to know what parameters should have high weight and what should have smaller
weight.
The following gradient descent equation tells us how loss would change if we modified
the parameters −
()θj=1mXT(())()θj=1mXT(())
Implementation of Binary Logistic Regression Model in Python
Now we will implement the above concept of binomial logistic regression in Python. For
this purpose, we are using a multivariate flower dataset named iris which have 3 classes
of 50 instances each, but we will be using the first two feature columns. Every class
represents a type of iris flower.
First, we need to import the necessary libraries as follows −
import numpy as np
import [Link] as plt
import seaborn as sns
from sklearn import datasets
Next, load the iris dataset as follows −
iris = datasets.load_iris()
X = [Link][:, :2]
y = ([Link] != 0) * 1
We can plot our training data s follows −
[Link](figsize=(6, 6))
[Link](X[y == 0][:, 0], X[y == 0][:, 1], color='g', label='0')
[Link](X[y == 1][:, 0], X[y == 1][:, 1], color='y', label='1')
[Link]();

Next, we will define sigmoid function, loss function and gradient descend as follows −
class LogisticRegression:
def __init__(self, lr=0.01, num_iter=100000, fit_intercept=True, verbose=False):
[Link] = lr
self.num_iter = num_iter
self.fit_intercept = fit_intercept
[Link] = verbose
def __add_intercept(self, X):
intercept = [Link](([Link][0], 1))
return [Link]((intercept, X), axis=1)
def __sigmoid(self, z):
return 1 / (1 + [Link](-z))
def __loss(self, h, y):
return (-y * [Link](h) - (1 - y) * [Link](1 - h)).mean()
def fit(self, X, y):
if self.fit_intercept:
X = self.__add_intercept(X)
Now, initialize the weights as follows −
[Link] = [Link]([Link][1])
for i in range(self.num_iter):
z = [Link](X, [Link])
h = self.__sigmoid(z)
gradient = [Link](X.T, (h - y)) / [Link]
[Link] -= [Link] * gradient
z = [Link](X, [Link])
h = self.__sigmoid(z)
loss = self.__loss(h, y)
if([Link] ==True and i % 10000 == 0):
print(f'loss: {loss} \t')
With the help of the following script, we can predict the output probabilities −
def predict_prob(self, X):
if self.fit_intercept:
X = self.__add_intercept(X)
return self.__sigmoid([Link](X, [Link]))
def predict(self, X):
return self.predict_prob(X).round()
Next, we can evaluate the model and plot it as follows −
model = LogisticRegression(lr=0.1, num_iter=300000)
preds = [Link](X)
(preds == y).mean()

[Link](figsize=(10, 6))
[Link](X[y == 0][:, 0], X[y == 0][:, 1], color='g', label='0')
[Link](X[y == 1][:, 0], X[y == 1][:, 1], color='y', label='1')
[Link]()
x1_min, x1_max = X[:,0].min(), X[:,0].max(),
x2_min, x2_max = X[:,1].min(), X[:,1].max(),
xx1, xx2 = [Link]([Link](x1_min, x1_max), [Link](x2_min, x2_max))
grid = np.c_[[Link](), [Link]()]
probs = model.predict_prob(grid).reshape([Link])
[Link](xx1, xx2, probs, [0.5], linewidths=1, colors='red');

Multinomial Logistic Regression Model

Another useful form of logistic regression is multinomial logistic regression in which the
target or dependent variable can have 3 or more possible unordered types i.e. the types
having no quantitative significance.
Implementation of Multinomial Logistic Regression Model in Python
Now we will implement the above concept of multinomial logistic regression in Python.
For this purpose, we are using a dataset from sklearn named digit.
First, we need to import the necessary libraries as follows −
Import sklearn
from sklearn import datasets
from sklearn import linear_model
from sklearn import metrics
from sklearn.model_selection import train_test_split
Next, we need to load digit dataset −
digits = datasets.load_digits()
Now, define the feature matrix(X) and response vector(y)as follows −
X = [Link]
y = [Link]
With the help of next line of code, we can split X and y into training and testing sets −
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=1)
Now create an object of logistic regression as follows −
digreg = linear_model.LogisticRegression()
Now, we need to train the model by using the training sets as follows −
[Link](X_train, y_train)
Next, make the predictions on testing set as follows −
y_pred = [Link](X_test)
Next print the accuracy of the model as follows −
print("Accuracy of Logistic Regression model is:",
metrics.accuracy_score(y_test, y_pred)*100)

Output
Accuracy of Logistic Regression model is: 95.6884561891516

Decision Tree: Regression & Classification

What is a Decision Tree?

A Decision Tree is a supervised learning algorithm that makes decisions by splitting data
into branches based on feature conditions, forming a tree-like structure.

Works for both Classification (predicting categories) and Regression (predicting continuous
values)

Mimics human decision-making logic

Highly interpretable — "White Box" model

Tree Structure Components

 [Root Node]
(Best Feature)
/ \
[Internal Node] [Internal Node]
(Feature Split) (Feature Split)
/ \ |
[Leaf] [Internal] [Leaf]
(Output) (Feature) (Output)
/ \
[Leaf] [Leaf]
(Output)(Output)

Component Description

Root Node Top node — best splitting feature for entire dataset

Internal
A feature with a condition/split
Node

Branch /
Outcome of a condition (yes/no, ≤, >)
Edge

Leaf Node Terminal node — final prediction output

Depth Number of levels from root to deepest leaf

Sub-tree Any sub-section of the full tree

Decision Tree: Classification

Goal

Predict a discrete class label (e.g., spam/not spam, disease/no disease).

How It Works
At each node, split data to maximize class purity leaves should contain mostly one class.

Splitting Criteria

Gini Impurity

Measures the probability of incorrectly classifying a randomly chosen element.

Gini(t) = 1 − Σ p(i|t)²

Gini = 0 → Pure node (all one class)

Gini = 0.5 → Maximum impurity (equal classes, binary)

Gini for a split:

Gini_split = (n_left/n) × Gini(left) + (n_right/n) × Gini(right)

Choose the split with minimum Gini impurity.

Example (Binary: Yes/No):

Node has 4 Yes, 6 No (total 10)

Gini = 1 − [(4/10)² + (6/10)²] = 1 − [0.16 + 0.36] = 0.48

Entropy & Information Gain

Based on Information Theory — measures disorder/uncertainty.

Entropy(t) = −Σ p(i|t) × log₂(p(i|t))

Entropy = 0 → Pure node

Entropy = 1 → Maximum impurity (binary, equal split)

Information Gain:
IG(split) = Entropy(parent) − Σ (nᵢ/n) × Entropy(childᵢ)

Choose the split with maximum Information Gain.

Example:

Parent: 5 Yes, 5 No → Entropy = −(0.5 log₂0.5 + 0.5 log₂0.5) = 1.0

After split:

Left: 4 Yes, 1 No → Entropy = 0.722

Right: 1 Yes, 4 No → Entropy = 0.722

IG = 1.0 − [(5/10)(0.722) + (5/10)(0.722)] = 0.278

Gain Ratio (C4.5)

Corrects Information Gain's bias toward features with many values.

Gain Ratio = Information Gain / Split Information

Split Info = −Σ (nᵢ/n) × log₂(nᵢ/n)

Chi-Square (CHAID)

Statistical test — measures significance of split.

χ² = Σ (Observed − Expected)² / Expected

Higher chi-square = more significant split.

Prediction in Classification

Traverse the tree from root → leaf based on feature values. Leaf node returns the majority
class of training samples in that leaf.

Decision Tree: Regression

Goal

Predict a continuous numerical value (e.g., house price, temperature, salary).

How It Works

Splits data to minimize variance/error in output values — leaves should have similar
target values.

Splitting Criteria for Regression

Mean Squared Error (MSE) — Most Common

MSE(t) = (1/n) Σ (yᵢ − ȳ)²

MSE for a split:

MSE_split = (n_left/n) × MSE(left) + (n_right/n) × MSE(right)

Choose the split with minimum weighted MSE.

Mean Absolute Error (MAE)

MAE(t) = (1/n) Σ |yᵢ − ȳ|

More robust to outliers than MSE.

Variance Reduction
Variance Reduction = Var(parent) − Σ (nᵢ/n) × Var(childᵢ)

Maximize variance reduction at each split — equivalent to minimizing MSE.

Prediction in Regression

Traverse tree → reach leaf node. Leaf node returns the mean of all training target values
in that leaf.

Aspect Classification Tree Regression Tree

Target Discrete class labels Continuous values

Leaf Output Majority class Mean of values

Split Criterion Gini / Entropy / Gain Ratio MSE / MAE / Variance

Evaluation Metric Accuracy, F1, AUC RMSE, MAE, R²

Purity Measure Class homogeneity Value similarity

Example Spam detection House price prediction

Building a Decision Tree — Step by Step

Start with the entire dataset at the root node.

Select Best Feature using the splitting criterion (Gini / IG / MSE).

Split the Data into subsets based on the feature's values.

Categorical → one branch per category

Numerical → binary split using a threshold (e.g., age ≤ 30)

Repeat recursively for each child node on its subset.

 Stop when a stopping condition is met (see below).

Assign Output to each leaf:

Classification → majority class

Regression → mean value

Condition Description

Max depth reached Tree won't grow beyond set depth

Min samples per split Node won't split if fewer than n samples

Min samples per leaf Leaf must have at least n samples

Pure node All samples in node belong to same class

Split doesn't improve criterion beyond

No improvement
threshold

Overfitting in Decision Trees

A fully grown tree memorizes training data — performs poorly on unseen data.

Why It Happens

Tree grows too deep

Learns noise and outliers as patterns

Very small leaf nodes (1-2 samples)

Solution: Pruning
Pruning Techniques

Pre-Pruning (Early Stopping)

Stop tree growth during construction.

Set max_depth

Set min_samples_split

Set min_samples_leaf

Set min_impurity_decrease

Post-Pruning (Reduced Error / Cost Complexity)

Grow full tree → trim back branches after construction.

Cost Complexity Pruning (ccp_alpha):

T α (T) = Error(T) + α × |leaves(T)|

α = 0 → Full tree (no pruning)

Increase α → More pruning → Simpler tree

Choose optimal α via cross-validation

Reduced Error Pruning:

Replace a sub tree with a leaf if it doesn't reduce validation error

Feature Splits: Numerical vs Categorical

Numerical Features

Sort unique values → evaluate midpoints as thresholds

Binary split: feature ≤ threshold (left), feature > threshold (right)

 e.g., Age ≤ 30 → Left branch, Age > 30 → Right branch

Categorical Features

Multiway split: One branch per category (ID3 approach)

Binary split: Group categories into two sets (CART approach)

e.g., Color = {Red, Blue} vs {Green}

Split
Algorithm Tree Type Splits Notes
Criterion

Information
ID3 Classification Multiway Only categorical features
Gain

C4.5 Gain Ratio Classification Multiway Handles numerical + missing

C5.0 Gain Ratio Classification Multiway Faster, more memory efficient

CART Gini / MSE Both Binary only Scikit-learn uses CART

CHAID Chi-Square Classification Multiway Statistical approach

Decision Boundary

Classification

Creates rectangular/axis-aligned boundaries in feature space

Each region corresponds to one predicted class

Regression

Creates a step function piecewise constant predictions

Each leaf region predicts a constant mean value

 Parameter Purpose Default

max_depth Maximum tree depth None (unlimited)

min_samples_split Min samples to split a node 2

min_samples_leaf Min samples in a leaf 1

max_features Features considered per split None (all)

Split measure
criterion gini / mse
(gini/entropy/mse)

ccp_alpha Cost complexity pruning 0.0

max_leaf_nodes Max number of leaves None

min_impurity_decrease Min gain to split 0.0

Advantages

Interpretable - easy to visualize and explain

No feature scaling needed (no normalization/standardization)

Handles both numerical and categorical data

Handles non-linear relationships naturally

Implicit feature selection - unimportant features are ignored

Fast prediction - O(log n) at prediction time

Handles missing values (C4.5, CHAID)

Disadvantages

Over-fitting - especially with deep trees

High variance - small data changes → very different trees

Biased toward high-cardinality features (Information Gain)

Axis-aligned splits only - struggles with diagonal decision boundaries

Unstable - not robust; sensitive to noisy data

Greedy algorithm - local optimal splits, not global optimum

Feature Importance

Decision trees naturally provide feature importance scores.

Importance(feature) = Σ (weighted impurity decrease for all splits on that feature)

Higher score = feature used more for splitting = more important

Used extensively for feature selection

Evaluation Metrics

Classification Tree

Metric Formula

Accuracy (TP+TN) / Total

Precision TP / (TP+FP)

Recall TP / (TP+FN)

F1 Score 2 × (P×R)/(P+R)

AUC-ROC Area under ROC curve

Confusion Matrix Full TP/FP/TN/FN breakdown

Regression Tree

Metric Formula

MSE Mean Squared Error

RMSE √MSE

MAE Mean Absolute Error

R² 1 − SS_res/SS_tot

Penalizes extra
Adjusted R²
features

Method Idea Improvement

Train multiple trees on bootstrap

Bagging Reduces variance
samples

Reduces correlation between

Random Forest Bagging + random feature subsets
trees

Gradient Boosting Trees trained sequentially on residuals Reduces bias

XGBoost Regularized gradient boosting Faster, better performance

Weighted resampling of misclassified

AdaBoost Reduces bias
points

Random Forest:Regression & Classification

What is Random Forest?

Random Forest is a supervised ensemble learning algorithm that builds multiple decision
trees and combines their outputs for a final prediction.
Proposed by Leo Breiman in 2001

Based on two core ideas: Bagging + Random Feature Selection

Works for both Classification and Regression

One of the most powerful and widely used ML algorithms

"A forest of random decision trees is stronger than any single tree"

Why Random Forest? — Problem with Single Decision

Tree

Problem in Decision Tree Random Forest Solution

High variance – over fits easily Averaging multiple trees reduces variance

Unstable sensitive to data changes Diversity across trees adds stability

Greedy splits local optima Different trees capture different patterns

Single perspective on data Multiple trees = multiple perspectives

Core Concepts

Bagging (Bootstrap Aggregating)

Introduced by Breiman (1996) — foundation of Random Forest.

Steps:

From original dataset of n samples, create B bootstrap samples

Each bootstrap sample: draw n samples with replacement

~63.2% unique samples per bootstrap, ~36.8% are duplicates

Train one decision tree on each bootstrap sample

Aggregate all tree predictions

Why it works: Each tree sees a slightly different dataset → diversity → reduced variance
when combined.

Random Feature Subspace (Feature Randomness)

At each split in each tree, only a random subset of features is considered.

Classification: √p features (p = total features)

Regression: p/3 features

This is what makes it "Random" Forest — not just bagging

Why it works: Prevents strong features from dominating every tree → decorrelates trees
→ better ensemble.

Out-of-Bag (OOB) Samples

The ~36.8% samples NOT used in a bootstrap sample = OOB samples for that tree.

Used as a built-in validation set no need for separate test set

OOB error = average prediction error on OOB samples across all trees

Very accurate estimate of generalization error

Random Forest Architecture

Original Dataset (n samples, p features)

|
┌────┴─────────────────┐
│ Bootstrap │
│ Sampling │
└────┬─────────────────┘
|
┌─────┼─────┐
↓ ↓ ↓
D₁ D₂ ... Dᴮ ← B Bootstrap Datasets
| | |
T₁ T₂ ... Tᴮ ← B Decision Trees
(random (random (random (each with random
feature) feature) feature) feature subsets)
| | |
P₁ P₂ ... Pᴮ ← B Predictions
|
┌────┴──────────────┐
│ Aggregation │
│ Classification │ → Majority Vote
│ Regression │ → Mean Average
└───────────────────┘
|
Final Prediction

Random Forest: Classification

Goal

Predict a discrete class label.

How Trees Vote

Each tree casts one vote for a class → Majority Vote wins.

Final Class = mode(T₁(x), T₂(x), ..., Tᴮ(x))

Example (5 trees predicting spam/not-spam):

T₁ → Spam
T₂ → Not Spam
T₃ → Spam
T₄ → Spam
T₅ → Not Spam
Final → Spam (3 votes vs 2)

Probability Estimation

Each tree predicts class probabilities → average probabilities across trees.

P(class=k | x) = (1/B) Σ Pᵢ(class=k | x)

Splitting Criterion (for each tree)

Gini Impurity (default in scikit-learn)

Entropy / Information Gain

Random Forest: Regression

Goal

Predict a continuous numerical value.

How Trees Combine

Each tree predicts a number → Average of all predictions.

Final Value = (1/B) Σ Tᵢ(x)

Example (5 trees predicting house price):

T₁ → ₹45L
T₂ → ₹48L
T₃ → ₹44L
T₄ → ₹47L
T₅ → ₹46L
Final → (45+48+44+47+46)/5 = ₹46L

Splitting Criterion (for each tree)

MSE — Mean Squared Error (default)

MAE — Mean Absolute Error

Variance Reduction

Aspect Classification RF Regression RF

Target Discrete class labels Continuous values

Aggregation Majority voting Mean averaging

Class label /
Tree output Numeric value
probabilities

Split criterion Gini / Entropy MSE / MAE

Default features/split √p p/3

Evaluation Accuracy, F1, AUC RMSE, MAE, R²

Example use Disease diagnosis Stock price prediction

Step-by-Step: Building a Random Forest

Step 1 - Set Parameters Choose: number of trees (B), max features per split, max depth,
etc.

Step 2 - Bootstrap Sampling For each tree i from 1 to B:

Sample n rows with replacement from training data → Dataset Dᵢ

Step 3 - Grow Each Tree For each node in tree Tᵢ:

Randomly select m features (m = √p or p/3)

Find best split among those m features

Split the node

Repeat until stopping condition met

No pruning - trees grown fully (depth controlled by hyper-params)

Step 4 - OOB Evaluation Use samples not in Dᵢ to estimate error of tree Tᵢ.

Step 5 - Aggregate Predictions

Classification → majority vote

Regression → mean

Each sample is OOB for roughly 1/3 of trees.

For each sample xᵢ:

Collect predictions only from trees where xᵢ was OOB

Aggregate those predictions

Compare to true label → OOB error for xᵢ

OOB Error = (1/n) Σ Loss(yᵢ, OOB_prediction(xᵢ))

Advantage: Free internal cross-validation — no need for a separate validation split.

Feature Importance in Random Forest

Mean Decrease in Impurity (MDI)

Importance(f) = Σ over all trees Σ over all nodes using f (weighted impurity decrease)

Fast to compute
Biased toward high-cardinality features

Mean Decrease in Accuracy (MDA / Permutation Importance)

Compute OOB accuracy normally

Randomly permute (shuffle) feature f in OOB data

Measure drop in OOB accuracy

Larger drop = more important feature

Importance(f) = Accuracy_normal − Accuracy_permuted(f)

Slower but more reliable

Not biased by feature cardinality

Available via permutation_importance in scikit-learn

Hyperparameters

Parameter Description Typical Values

n_estimators Number of trees 100–1000

max_features Features per split √p (clf), p/3 (reg), or float

Max depth of each

max_depth None or 5–30
tree

Min samples to split a

min_samples_split 2–20
node

min_samples_leaf Min samples in a leaf 1–20

Whether to use
bootstrap True
bootstrap

oob_score Compute OOB score True/False

 Parameter Description Typical Values

Parameter Description

criterion Split quality measure

max_leaf_nodes Max leaves per tree

Min gain threshold to

min_impurity_decrease
split

Handle imbalanced
class_weight
classes

ccp_alpha Cost complexity pruning

Parallel jobs (-1 = all

n_jobs
cores)

random_state Reproducibility seed

Effect of n_estimators

More trees → better performance up to a point

After ~200-500 trees, returns diminish

More trees = more computation (but parallelizable)

Never overfits by adding more trees — only plateaus

Bias-Variance Trade-off in Random Forest

Single Deep
Random Forest
Tree

Bias Low Low (trees still deep)

Low (averaging
Variance High
reduces it)

Overfitting High Much lower

Key insight: Each tree has low bias (deep, complex) but high variance. Averaging B
uncorrelated trees reduces variance by factor of B without increasing bias.

Var(mean of B trees) = ρσ² + (1−ρ)σ²/B

where ρ = correlation between trees.

Lower correlation between trees → better variance reduction

Random feature selection reduces ρ → key benefit

Handling Special Cases

Imbalanced Classes

Use class_weight='balanced'

Use class_weight={0:1, 1:10} (manual)

Use Balanced Random Forest - undersamples majority class in each bootstrap

Use Easy Ensemble — multiple balanced subsets

Missing Values

Random Forest can handle missing values via:

Median/mode imputation before training

Proximity-based imputation (iterative RF approach)

Built-in handling in some implementations (XGBoost style)

High Dimensionality

Random feature selection naturally handles it

Performs well even when p >> n

Categorical Features

Scikit-learn requires encoding (OrdinalEncoder, OneHotEncoder)

Some implementations (R's randomForest) handle natively

Proximity Matrix

A unique feature of Random Forest — measures similarity between samples.

Proximity(xᵢ, xⱼ) = (times xᵢ and xⱼ end in same leaf) / (total trees)

Uses:

Outlier detection (low proximity to all others)

Missing value imputation

Data visualization (MDS on proximity matrix)

Clustering

Evaluation Metrics
 Metric Description

Accuracy Overall correct predictions

Precision TP / (TP + FP)

Recall / Sensitivity TP / (TP + FN)

F1 Score Harmonic mean of P & R

AUC-ROC Area under ROC curve

OOB Score Internal validation accuracy

Confusion Matrix Class-wise breakdown

Log Loss Probabilistic accuracy

Metric Description

MSE Mean Squared Error

RMSE Root Mean Squared Error

MAE Mean Absolute Error

R² Variance explained

Adjusted R² R² penalized for features

OOB R² Score Internal validation R²

MAPE Mean Absolute % Error

Advantages

High accuracy — one of the best off-the-shelf algorithms

Robust to over-fitting — averaging prevents memorization

Handles high dimensionality — feature randomness helps

Built-in feature importance — MDI and MDA

OOB validation — free cross-validation

No feature scaling needed

Handles missing values reasonably well

Parallelizable — trees built independently

Works well on small and large datasets

Robust to outliers — averaging smooths extreme predictions

Handles non-linear relationships naturally

Disadvantages

Less interpretable than single decision tree — "Black Box"

Slow prediction for real-time systems (B trees to traverse)

Memory intensive — stores B full trees

Biased toward features with more levels (MDI importance)

Poor extrapolation in regression — can't predict beyond training range

Not ideal for very sparse data (e.g., text)

Hyperparameter tuning needed for best performance

Variants of Random Forest

Extra Trees (Extremely Randomized Trees)

Random threshold for splits (not optimal threshold)

Even more randomness → lower variance, slightly higher bias

Faster to train than standard RF

Balanced Random Forest

Undersamples majority class in each bootstrap

Designed for imbalanced classification

Weighted Random Forest

Assigns weights to samples during bootstrap

Useful for cost-sensitive learning

Rotation Forest

Applies PCA to feature subsets before tree building

Captures rotated feature relationships

Random Forest with Oblique Splits

Uses linear combinations of features at splits

Can capture diagonal decision boundaries

Real-World Applications

Domain Application

Finance Credit scoring, fraud detection, stock prediction

Healthcare Disease diagnosis, drug discovery, survival analysis

E-commerce Recommendation, churn prediction, price optimization

 Domain Application

Computer Vision Object detection (older systems), image classification

NLP Sentiment analysis, text classification

Remote Sensing Land cover classification, satellite image analysis

Manufacturing Predictive maintenance, quality control

Ecology Species distribution modeling

Support Vector Machine: Regression & Classification

What is SVM?

Support Vector Machine is a supervised learning algorithm used for both classification
and regression tasks.

Proposed by Vapnik & Chervonenkis (1963), popularized in 1990s

Core idea: Find the optimal hyperplane that best separates or fits data

Based on Statistical Learning Theory and Structural Risk Minimization

Extremely effective in high-dimensional spaces

Core Intuition

Imagine two groups of points on a table. You can draw many lines separating them SVM
finds the line that is farthest from both groups simultaneously maximizing the safety
margin.
 Key Terminology

Term Definition

Hyper-plane Decision boundary separating classes (line in 2D, plane in 3D, hyper-plane in nD)

Support Vectors Data points closest to the hyper-plane they define and support the margin

Margin Distance between the two margin hyper-planes (2/||w||)

Hard Margin No misclassification allowed only for linearly separable data

Soft Margin Allows some misclassification for real-world noisy data

Kernel Function that transforms data into higher dimensions

w Weight vector normal to the hyper-plane

b Bias term shifts the hyper-plane

Hyper-plane Equation

w·x+b=0

w = weight vector (normal to hyper-plane)

x = input feature vector

b = bias

Margin Hyper-planes

Positive hyper-plane: w · x + b = +1

Negative hyper-plane: w · x + b = −1

Margin width = 2 / ||w||

Classification Rule
ŷ = sign(w · x + b)

ŷ = +1 → Class 1

ŷ = −1 → Class −1

SVM: Classification

Objective: Maximize margin = Maximize 2/||w|| = Minimize ||w||²/2

Optimization Problem:

Minimize: ½ ||w||² Subject to: yᵢ(w · xᵢ + b) ≥ 1 for all i

yᵢ = true class label (+1 or −1)

Constraint ensures all points correctly classified and outside margin

Soft Margin SVM (Non-Separable Data — C-SVM)

Real data is noisy and not perfectly separable. Soft margin introduces slack variables ξᵢ.

Slack Variable ξᵢ:

ξᵢ = 0 → point correctly classified, outside margin

0 < ξᵢ ≤ 1 → point inside margin but correct side

ξᵢ > 1 → point misclassified

Optimization Problem:

Minimize: ½ ||w||² + C Σ ξᵢ Subject to: yᵢ(w · xᵢ + b) ≥ 1 − ξᵢ and ξᵢ ≥ 0

C Parameter (Regularization)

C Value Effect

Large C Small margin, less misclassification, risk of over-fitting

 C Value Effect

Small C Large margin, more misclassification allowed, better generalization

C→∞ Hard margin SVM

Dual Formulation

SVM optimization is often solved in dual form using Lagrange multipliers αᵢ:

Maximize: Σαᵢ − ½ Σᵢ Σⱼ αᵢαⱼ yᵢyⱼ (xᵢ · xⱼ) Subject to: 0 ≤ αᵢ ≤ C and Σαᵢyᵢ = 0

Key insight: Only support vectors have αᵢ > 0 — rest are zero. This leads directly to the
Kernel Trick.

The Kernel Trick

Real data is often not linearly separable in original space.

Solution: Map data to a higher-dimensional space where it becomes linearly separable —

without explicitly computing the transformation.

K(xᵢ, xⱼ) = φ(xᵢ) · φ(xⱼ)

Instead of computing φ(x) explicitly, use a kernel function that computes the dot product
in the transformed space directly.

2D non-separable data
↓ Kernel φ(x)
3D linearly separable data
↓ Find hyperplane
Project back to 2D → non-linear boundary

Kernel Functions
 Linear Kernel

K(xᵢ, xⱼ) = xᵢ · xⱼ

No transformation — original feature space

Fast, works well for linearly separable data

Best for: text classification, high-dimensional data

Polynomial Kernel

K(xᵢ, xⱼ) = (γ xᵢ · xⱼ + r)^d

d = degree, γ = scale, r = coeff0

Captures feature interactions up to degree d

Best for: image processing, NLP with feature interactions

RBF / Gaussian Kernel (Most Popular)

K(xᵢ, xⱼ) = exp(−γ ||xᵢ − xⱼ||²)

γ = 1/(2σ²) controls the influence radius

Maps to infinite-dimensional space

Best for: non-linear data, general purpose

Sigmoid Kernel

K(xᵢ, xⱼ) = tanh(γ xᵢ · xⱼ + r)

Similar to neural network activation

Not always positive semi-definite

Best for: neural network-like behavior

Laplacian Kernel

K(xᵢ, xⱼ) = exp(−γ ||xᵢ − xⱼ||)

Less sensitive to parameter γ than RBF

Robust to outliers

Custom Kernels

Any function satisfying Mercer's Theorem (positive semi-definite) can be used as a

kernel.

Kernel Comparison

Decision
Kernel Parameters When to Use
Boundary

Linearly separable,
Linear Linear C
high-dim

Polynomia
Curved C, d, γ, r Interactions matter
l

General purpose
RBF Smooth non-linear C, γ
default

Neural-net-like
Sigmoid S-shaped C, γ, r
problems

γ (Gamma) Parameter in RBF Kernel

Controls the influence radius of each support vector.

 γ Value Effect

Small radius, each point influences only nearby → complex boundary,

Large γ
overfitting

Small γ Large radius, points influence widely → smooth boundary, underfitting

SVM: Regression (SVR)

Support Vector Regression (SVR)

Instead of finding a hyper-plane that separates classes, SVR finds a hyper-plane that best
fits the data within a tube of width ε.

ε-Insensitive Tube

ε-tube
───────────────── f(x) + ε (upper boundary)
─ ─ ─ ─ ─ ─ ─ ─ f(x) (regression line)
───────────────── f(x) − ε (lower boundary)
Points INSIDE tube → zero loss
Points OUTSIDE tube → penalized

ε-Insensitive Loss Function

L(y, f(x)) = max(0, |y − f(x)| − ε)

If prediction error ≤ ε → no penalty

If prediction error > ε → penalized by the excess

SVR Optimization Problem

Minimize: ½ ||w||² + C Σ (ξᵢ + ξᵢ*) Subject to: yᵢ − (w · xᵢ + b) ≤ ε + ξᵢ (w · xᵢ + b) − yᵢ ≤ ε

+ ξᵢ* ξᵢ, ξᵢ* ≥ 0

ξᵢ = slack for points above the tube

ξᵢ* = slack for points below the tube

SVR Parameters

Parameter Effect

C Penalty for points outside tube — higher C = less tolerance

ε (epsilon) Width of the insensitive tube — larger ε = more tolerance

Kernel Determines shape of the regression function

Classification vs Regression SVM

Aspect SVM Classification SVR (Regression)

Find optimal separating hyper- Find optimal fitting hyper-

Goal
plane plane

Margin Maximize margin between classes Maximize tube width (ε)

Loss Hinge loss ε-insensitive loss

Output Class label Continuous value

Support Vectors Points on/inside margin Points on/outside ε-tube

Key Parameter C C, ε

Evaluation Accuracy, F1, AUC RMSE, MAE, R²

Loss Functions in SVM

Hinge Loss (Classification)

L(y, f(x)) = max(0, 1 − y·f(x))

Zero loss when correctly classified with margin

Linear penalty for margin violations

Squared Hinge Loss

L(y, f(x)) = max(0, 1 − y·f(x))²

Differentiable, heavier penalty for large violations

ε-Insensitive Loss (SVR)

L = max(0, |y − f(x)| − ε)

Zero loss within ε-tube

Linear penalty outside

Multiclass SVM

SVM is inherently binary. Extended to multiclass:

One-vs-Rest (OvR)

Train k classifiers (one per class)

Classifier i: class i vs all others

Predict: class with highest decision score

One-vs-One (OvO)

Train k(k−1)/2 classifiers

One for every pair of classes

Predict: majority vote across all classifiers

Default in scikit-learn's SVC

Crammer-Singer (Direct Multiclass)

Single optimization over all classes simultaneously

More principled but computationally heavier

Decision Function

Classification

f(x) = w · x + b = Σ αᵢ yᵢ K(xᵢ, x) + b

Sign gives class label

Magnitude gives distance from hyperplane (confidence)

Probability Estimation (Platt Scaling)

SVM doesn't natively output probabilities Platt scaling fits a sigmoid on top:

P(y=1|x) = 1 / (1 + exp(A·f(x) + B))

Enable with probability=True in scikit-learn (slower training).

Hyperparameters

Parameter Applies To Description Typical Values

Regularization — margin vs
C SVC, SVR 0.01 to 1000
error trade-off

linear, rbf, poly,

kernel Both Kernel function
sigmoid
 Parameter Applies To Description Typical Values

RBF, Poly, scale, auto, 0.001–

gamma Kernel coefficient
Sigmoid 10

degree Polynomial Degree of polynomial 2, 3, 4

coef0 Poly, Sigmoid Independent term in kernel 0.0 to 1.0

epsilon SVR ε-tube width 0.01 to 1.0

class_weight SVC Handle imbalanced data balanced, dict

Feature Scaling CRITICAL for SVM

SVM is very sensitive to feature scale always scale before training.

Scaler Formula When to Use

General purpose — most

StandardScaler (x − μ) / σ
common

(x − min) / (max −
MinMaxScaler When distribution unknown
min)

RobustScaler (x − median) / IQR When outliers are present

Kernel Selection Guide

Is data linearly separable?

↓ Yes ↓ No
Linear Kernel Many features (text)?
↓ Yes ↓ No
Linear Kernel Try RBF first
↓
Tune C and γ
↓
Still not good?
 ↓
Try Polynomial Kernel

Advantages

Effective in high-dimensional spaces even when d > n

Memory efficient only support vectors stored

Versatile different kernels for different problems

Global optimum guaranteed convex optimization

Robust to over-fitting in high dimensions (with proper C)

Works well with small datasets

Clear geometric interpretation

Effective for non-linear data via kernel trick

Disadvantages

Slow on large datasets — O(n² to n³) training time

Feature scaling mandatory

Kernel and hyper-parameter selection is non-trivial

Black box with kernel trick - hard to interpret

Doesn't scale well with n > 100,000 samples

No native probability output - needs Platt scaling

Real-World Applications

Domain Application

Text Classification Spam detection, sentiment analysis, topic classification

 Domain Application

Image Recognition Face detection, handwriting recognition, object classification

Protein classification, gene expression analysis, cancer

Bioinformatics
detection

Finance Credit risk, stock trend prediction, fraud detection

Medical Diagnosis Disease classification from medical images

NLP Named entity recognition, document categorization

Remote Sensing Satellite image land-use classification

Anomaly Detection One-Class SVM for novelty/outlier detection

One-Class SVM (Novelty Detection)

Special variant trained on only one class to detect outliers/anomalies.

Finds a boundary enclosing the training data new points outside = anomalies

Used for:

Fraud detection (only normal transactions available)

Fault detection in manufacturing

Intrusion detection

Evaluating Classification Models

Why Model Evaluation Matters?

Building a model is only half the job - knowing how well it performs and where it fails is
equally critical.

A model with 99% accuracy can still be completely useless (class imbalance)

Different problems need different metrics (medical diagnosis ≠ spam detection)

Evaluation guides model selection, tuning, and deployment decisions

Term Full Name Meaning

TP True Positive Predicted Positive, Actually Positive ✓

TN True Negative Predicted Negative, Actually Negative ✓

Predicted Positive, Actually Negative ✗ (Type I

FP False Positive
Error)

False Predicted Negative, Actually Positive ✗ (Type

FN
Negative II Error)

Example (Disease Detection)

Predicted Disease Predicted Healthy

Actual Disease 90 (TP) 10 (FN)
Actual Healthy 20 (FP) 180 (TN)
Total = 300

Core Metrics Derived from Confusion Matrix

Accuracy

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Proportion of all correct predictions.

Example: (90 + 180) / 300 = 90%

When to use: Balanced classes, equal cost for all errors.

When NOT to use: Imbalanced datasets.

Problem: 95% of emails are not spam → predicting "not spam" always gives 95%
accuracy completely useless model.

Precision (Positive Predictive Value)

Precision = TP / (TP + FP)

Of all predicted positives, how many are actually positive?

Example: 90 / (90 + 20) = 81.8%

Answers: "When I predict positive, how often am I right?"

High precision is critical when: False positives are costly

Spam detection (legitimate emails marked as spam)

Legal document review (flagging innocent documents)

Recall (Sensitivity / True Positive Rate)

Recall = TP / (TP + FN)

Of all actual positives, how many did I correctly identify?

Example: 90 / (90 + 10) = 90%

Answers: "Of all real positives, how many did I catch?"

High recall is critical when: False negatives are costly

Cancer detection (missing a real case is dangerous)

Fraud detection (missing actual fraud is costly)

COVID screening (missing infected person spreads disease)

Specificity (True Negative Rate)

Specificity = TN / (TN + FP)

Of all actual negatives, how many did I correctly identify?

Example: 180 / (180 + 20) = 90%

Answers: "Of all real negatives, how many did I correctly reject?"

Complement of False Positive Rate.

Specificity = 1 − FPR

F1 Score

F1 = 2 × (Precision × Recall) / (Precision + Recall)

Harmonic mean of Precision and Recall — balances both.

Example: 2 × (0.818 × 0.90) / (0.818 + 0.90) = 0.857

Why harmonic mean, not arithmetic?

Arithmetic mean of P=1.0, R=0.0 → 0.5 (misleadingly good)

Harmonic mean of P=1.0, R=0.0 → 0.0 (correctly bad)

Harmonic mean punishes extreme imbalances between P and R

When to use: Imbalanced classes, both FP and FN matter equally.

F-Beta Score

Fβ = (1 + β²) × (Precision × Recall) / (β² × Precision + Recall)

Generalization of F1 — control the weight of Recall vs Precision.

β value Emphasis Use Case

β = 0.5 Precision 2× more Spam detection

β=1 Equal (F1 Score) General purpose

β=2 Recall 2× more Disease screening

False Positive Rate (FPR)

FPR = FP / (FP + TN)

Of all actual negatives, how many were wrongly predicted positive?

Example: 20 / (20 + 180) = 10%

Also called: Fall-out or 1 − Specificity

False Negative Rate (FNR)

FNR = FN / (FN + TP)

Of all actual positives, how many were wrongly predicted negative?

Example: 10 / (10 + 90) = 10%

Also called: Miss Rate or 1 − Recall

False Discovery Rate (FDR)

FDR = FP / (FP + TP)

Of all predicted positives, how many are wrong?

FDR = 1 − Precision

Negative Predictive Value (NPV)

NPV = TN / (TN + FN)

Of all predicted negatives, how many are actually negative?

Metric Formula Focus

Accuracy (TP+TN)/Total Overall correctness

Precision TP/(TP+FP) Predicted positives

Recall TP/(TP+FN) Actual positives

Specificity TN/(TN+FP) Actual negatives

F1 Score 2PR/(P+R) Balance P & R

FPR FP/(FP+TN) False alarm rate

FNR FN/(FN+TP) Miss rate

NPV TN/(TN+FN) Predicted negatives

FDR FP/(FP+TP) False discovery

Precision-Recall Tradeoff

Precision and Recall are in direct conflict improving one hurts the other.

High Threshold (strict positive prediction):

→ Fewer positives predicted
→ Precision ↑ (those predicted are more likely correct)
→ Recall ↓ (miss more actual positives)

Low Threshold (lenient positive prediction):

→ More positives predicted
→ Precision ↓ (more false alarms)
→ Recall ↑ (catch more actual positives)

Decision Threshold

Default threshold = 0.5. Adjust based on problem:

Cancer detection → Lower threshold (0.3) → Higher recall

Spam filter → Higher threshold (0.7) → Higher precision

AUC interpretation: Probability that the model ranks a random positive higher than a
random negative.

Advantages of AUC-ROC:

Threshold-independent

Works across all classification thresholds

Good for balanced datasets

Disadvantage: Misleading with highly imbalanced classes.

Precision-Recall Curve

Plots Precision vs Recall at every threshold.

Precision
|●────────
| \
| \ ← Good Model
0.5 | \
| ●
|_____________
0.0 0.5 1.0
Recall

AUC-PR (Area Under PR Curve)

Better than ROC-AUC for imbalanced datasets

Higher AUC-PR = better model at handling minority class

Baseline = prevalence (% of positive class)

When to prefer PR curve over ROC:

Class imbalance (fraud detection, rare disease)

Care more about positive class performance

Log Loss (Cross-Entropy Loss)

Measures the quality of probability predictions — penalizes confident wrong

predictions.
Log Loss = −(1/n) Σ [yᵢ log(pᵢ) + (1−yᵢ) log(1−pᵢ)]

Prediction Actual Loss

Very low (correct &

0.99 1
confident)

0.51 1 Medium (correct but unsure)

0.49 1 Medium (wrong but unsure)

Very high (wrong &

0.01 1
confident)

Lower log loss = better model.

Range: 0 (perfect) to ∞

Heavily penalizes confident wrong predictions

Used when probability calibration matters (medical risk scoring)

Matthews Correlation Coefficient (MCC)

One of the best single metrics for imbalanced binary classification.

MCC = (TP×TN − FP×FN) / √[(TP+FP)(TP+FN)(TN+FP)(TN+FN)]

MCC Value Interpretation

+1 Perfect prediction

Random
0
prediction

−1 Perfectly wrong

Advantages:

Uses all four confusion matrix values

Works well even with severe class imbalance

More informative than F1 for imbalanced data

Single balanced metric

Cohen's Kappa

Measures agreement between predicted and actual labels, correcting for chance.

κ = (Po − Pe) / (1 − Pe)

Po = observed accuracy

Pe = expected accuracy by chance

Interpretatio
Kappa n

Almost
> 0.8
perfect

0.6 – 0.8 Substantial

0.4 – 0.6 Moderate

0.2 – 0.4 Fair

< 0.2 Slight

0 Random

Multiclass Classification Metrics

Multiclass Confusion Matrix (3-class example)

 Strategy How When to Use

Macro Average metric across classes equally Equal importance to all classes

Micro Aggregate TP/FP/FN then compute When overall performance matters

Weighted Average weighted by class support Imbalanced classes

Samples Per sample average (multilabel) Multilabel classification

Example (F1 Score):

Class A: F1 = 0.90, support = 100

Class B: F1 = 0.70, support = 200
Class C: F1 = 0.80, support = 50

Macro F1 = (0.90 + 0.70 + 0.80) / 3 = 0.80

Weighted F1 = (0.90×100 + 0.70×200 + 0.80×50) / 350 = 0.777

Calibration

A calibrated model's predicted probabilities match actual frequencies.

"If my model predicts 70% for 100 samples → ~70 should actually be positive"

Reliability Diagram (Calibration Curve)

X-axis: Mean predicted probability

Y-axis: Fraction of positives

Perfect calibration = diagonal line

Calibration Methods

Platt Scaling — Fits logistic regression on top of outputs

Isotonic Regression — Non-parametric calibration

Temperature Scaling — Single parameter scaling (neural nets)

Brier Score

Brier = (1/n) Σ (pᵢ − yᵢ)²

Measures accuracy of probability predictions

Range: 0 (perfect) to 1 (worst)

Lower = better calibrated model

Cross-Validation for Evaluation

K-Fold Cross-Validation

Split data into K equal folds

Train on K-1 folds, test on 1 fold

Repeat K times (each fold as test once)

Average metrics across K runs

K=5:
Fold 1: [Test][Train][Train][Train][Train]
Fold 2: [Train][Test][Train][Train][Train]
Fold 3: [Train][Train][Test][Train][Train]
Fold 4: [Train][Train][Train][Test][Train]
Fold 5: [Train][Train][Train][Train][Test]
─────────────────────────────────
Average accuracy, F1, AUC...

Stratified K-Fold

Preserves class distribution in each fold

Essential for imbalanced datasets

Default choice for classification evaluation

Leave-One-Out (LOO)
K = n (each sample is test once)

Computationally expensive

Used for very small datasets

Handling Imbalanced Classes

Problem

Dataset: 950 negative, 50 positive → predicting all negative = 95% accuracy (useless).

Better Metrics for Imbalance

F1 Score — balances precision and recall

AUC-PR — focuses on minority class

MCC — balanced, uses all 4 confusion matrix cells

Balanced Accuracy = (Recall + Specificity) / 2

Resampling Strategies

Method Approach Effect

Create synthetic minority

Oversampling (SMOTE) Increases minority class
samples

Remove majority class

Undersampling Reduces majority class
samples

Penalize majority class errors Adjusts learning

Class weights
more emphasis

Trade precision for

Threshold tuning Adjust decision boundary
recall
UNIT-4

CLUSTERING
Clustering is an unsupervised machine learning technique used to group similar
data points together without using labelled data. It helps discover hidden
patterns or natural groupings in datasets by placing similar data points into the
same cluster.

Discover the natural grouping or structure in unlabelled data without

predefined categories.
Data points are assigned to clusters based on similarity or distance measures.
Uses Euclidean distance, cosine similarity or other metrics depending on data
type and clustering method

Types of Clustering
Hard Clustering
Hard clustering assigns each data point to exactly one cluster. A data point
cannot belong to multiple clusters, making the grouping clear and easy to
interpret.

Each data point belongs to only one cluster

No overlap between clusters
Simple and easy to interpret

Example
If customers are divided into two clusters, each customer belongs completely to
either Cluster 1 or Cluster 2. A customer cannot belong to both clusters at the
same time.

Common Uses
Market segmentation: Businesses group customers with similar buying
behaviour to design targeted marketing strategies.
Customer grouping: Companies organize customers into clear categories for
better service and analysis.
Document clustering: Documents with similar topics or keywords are grouped
together for easier organization.

Limitation
Cannot represent overlapping groups: Hard clustering cannot handle
situations where a data point may logically belong to multiple groups.

Soft Clustering
Soft clustering allows a data point to belong to multiple clusters with different
probabilities. Instead of assigning a strict cluster, it gives a degree of
membership to each cluster.
Example
A data point may belong 70% to Cluster 1 and 30% to Cluster 2, indicating
that it shares characteristics with both groups.

Use Cases
Overlapping class boundaries: Useful when data points cannot be clearly
separated into distinct groups.
Customer personas: Helps represent customers who share traits with multiple
behavioral groups.
Medical diagnosis: Patients may show symptoms related to multiple conditions.

Benefits
Captures ambiguity: Represents uncertainty when cluster boundaries are not
clear.
Models gradual transitions: Allows smooth transitions between clusters instead
of strict separation.

Clustering Methods
Centroid based Clustering
Centroid based clustering group data points around central points called
centroids. Each cluster is represented by a central point (centroid or medoid),
and data points are assigned to the nearest center.

Algorithms:
K-means: Iteratively assigns points to nearest centroid and recalculates
centroids to minimize intra cluster variance.
K-medoids: Similar to K-means but uses actual data points (medoids) as
centers, robust to outliers.

Advantages:
Fast and scalable for large datasets.
Simple to implement and interpret.

Limitations:
Requires choosing number of clusters in advance
Sensitive to initialization and outliers.
Not suitable for non-spherical clusters.

Density based Clustering

Density based clustering identifies clusters as regions where data points are
densely packed together. Points in low density areas are treated as noise.

Algorithms:
DBSCAN: Groups points with sufficient neighbors; labels sparse points as
noise.
OPTICS: Extends DBSCAN to handle varying densities.

Advantages:
Handles clusters of varying shapes and sizes.
Does not require cluster count upfront.
Effective in noisy datasets.

Limitations:
Difficult to choose parameters like epsilon and min points.
Less effective for varying density clusters (except OPTICS).

Connectivity based Clustering

Connectivity based or Hierarchical clustering builds clusters by gradually
merging or splitting groups of data points. It creates a tree like structure called
a dendrogram that shows relationships between clusters.

Approaches:
Agglomerative: Starts with each point as a cluster and merges them step by
step.
Divisive: Starts with one cluster and splits it into smaller clusters.

Advantages:
Provides a full hierarchy, easy to visualize
No need to specify number of clusters upfront

Limitations
Computationally intensive for large datasets
Merging/splitting decisions are irreversible
Choosing parameters can be difficult

Distribution-based Clustering

Distribution based clustering assumes that data points come from a mixture of
probability distributions. Each cluster is modelled as a statistical distribution.

Algorithm:
Gaussian Mixture Model (GMM): Fits data as a weighted mixture of Gaussian
distributions, assigns data points based on likelihood

Advantages:
Works well for clusters that are not perfectly circular.
Provides probabilistic memberships
Suitable for overlapping clusters

Limitations:
Requires specifying number of components
Computationally more expensive
Sensitive to initialization

Fuzzy Clustering

Fuzzy clustering allows data points to belong to multiple clusters with

different degrees of membership. It is useful when cluster boundaries are not
clear.

Algorithm:
Fuzzy C-Means: Similar to K-means but with fuzzy memberships updated
iteratively

Advantages:
Models data ambiguity explicitly
Useful for complex or imprecise data

Limitations:
Choosing fuzziness parameter can be tricky
Slightly higher computational cost

Applications
Customer Segmentation: Group customers based on behaviour or
demographics.
Anomaly Detection: Detect unusual activities in finance, security or sensor
data.
Image Segmentation: Divide images into meaningful regions for computer
vision tasks.
Recommendation Systems: Group similar users or items for personalized
suggestions.
Market Basket Analysis: Identify products frequently purchased together.

K means Clustering

K-Means Clustering groups similar data points into clusters without needing
labelled data. It is used to uncover hidden patterns when the goal is to
organize data based on similarity.

Helps identify natural groupings in unlabeled datasets

Works by grouping points based on distance to cluster centers
Commonly used in customer segmentation, image compression and pattern
discovery
Useful when you need structure from raw, unorganized data

Working of K-Means Clustering

Suppose we are given a data set of items with certain features and values for
these features like a vector. The task is to categorize those items into groups.

To achieve this we will use the K-means algorithm. "k" represents the number
of groups or clusters we want to classify our items into.

The algorithm will categorize the items into " k" groups or clusters of similarity.
To calculate that similarity we will use the Euclidean distance as a
measurement. The algorithm works as follows:
Initialization: We begin by randomly selecting k cluster centroids.
Assignment Step: Each data point is assigned to the nearest centroid, forming
clusters.
Update Step: After the assignment, we recalculate the centroid of each cluster
by averaging the points within it.
Repeat: This process repeats until the centroids no longer change or the
maximum number of iterations is reached.
The goal is to partition the dataset into k clusters such that data points within
each cluster are more similar to each other than to those in other clusters.

Selecting the right number of clusters is important for meaningful segmentation

to do this we use Elbow Method for optimal value of k in K-Means which is a
graphical tool used to determine the optimal number of clusters (k) in K-means.

Uses of K-Means Clustering

K-Means is popular in a wide variety of applications due to its simplicity,
efficiency and effectiveness. Here’s why it is widely used:

Data Segmentation: One of the most common uses of K-Means is segmenting

data into distinct groups. For example, businesses use K-Means to group
customers based on behaviour, such as purchasing patterns or website
interaction.
Image Compression: K-Means can be used to reduce the complexity of images
by grouping similar pixels into clusters, effectively compressing the image.
This is useful for image storage and processing.
Anomaly Detection: K-Means can be applied to detect anomalies or outliers by
identifying data points that do not belong to any of the clusters.
Document Clustering: In natural language processing (NLP), K-Means is used
to group similar documents or articles together. It’s often used in applications
like recommendation systems or news categorization.
Organizing Large Datasets: When dealing with large datasets, K-Means can
help in organizing the data into smaller, more manageable chunks based on
similarities, improving the efficiency of data analysis.

Implementation of K-Means Clustering

We will be using blobs datasets and show how clusters are made
using Python programming language.
Step 1: Importing the necessary libraries
We will be importing the following libraries.

Numpy: for numerical operations (e.g., distance calculation).

Matplotlib: for plotting data and results.
Scikit learn: to create a synthetic dataset using make_blobs

import numpy as np
import [Link] as plt
from [Link] import make_blobs

Step 2: Creating Custom Dataset

We will generate a synthetic dataset with make_blobs.

make_blobs(n_samples=500, n_features=2, centers=3): Generates 500 data

points in a 2D space, grouped into 3 clusters.
[Link](X[:, 0], X[:, 1]): Plots the dataset in 2D, showing all the points.
[Link](): Displays the plot

X,y = make_blobs(n_samples = 500,n_features = 2,centers = 3,random_state = 23)

fig = [Link](0)
[Link](True)
[Link](X[:,0],X[:,1])
[Link]()

Output:

Step 3 :Feature Scaling using StandardScaler

from [Link] import StandardScaler

scaler = StandardScaler()

X = scaler.fit_transform(X)

Step 4: Initializing Random Centroids

We will randomly initialize the centroids for K-Means clustering

[Link](23): Ensures reproducibility by fixing the random seed.

The for loop initializes k random centroids, with values between -2 and 2, for
a 2D dataset.

k=3
clusters = {}
[Link](23)

for idx in range(k):

center = 2*(2*[Link](([Link][1],))-1)
points = []
cluster = {
'center' : center,
'points' : []
}
clusters[idx] = cluster

Output:

Step 5: Plotting Random Initialized Center with Data Points

We will now plot the data points and the initial centroids.

[Link](): Plots a grid.

[Link](center[0], center[1], marker='*', c='red'): Plots the cluster
center as a red star (* marker).

[Link](X[:,0],X[:,1])
[Link](True)
for i in clusters:
center = clusters[i]['center']
[Link](center[0],center[1],marker = '*',c = 'red')
[Link]()
Output:

Step 6: Defining Euclidean Distance

To assign data points to the nearest centroid, we define a distance function:

[Link](): Computes the square root of a number or array element-wise.

[Link](): Sums all elements in an array or along a specified axis

def distance(p1,p2):
return [Link]([Link]((p1-p2)**2))

Step 7: Creating Assign and Update Functions

Next, we define functions to assign points to the nearest centroid and update
the centroids based on the average of the points assigned to each cluster.

[Link](dis): Appends the calculated distance to the list dist.

curr_cluster = [Link](dist): Finds the index of the closest cluster by
selecting the minimum distance.
new_center = [Link](axis=0): Calculates the new centroid by taking
the mean of the points in the cluster.

def assign_clusters(X, clusters):

for idx in range([Link][0]):
dist = []

curr_x = X[idx]

for i in range(k):
dis = distance(curr_x,clusters[i]['center'])
[Link](dis)
curr_cluster = [Link](dist)
clusters[curr_cluster]['points'].append(curr_x)
return clusters
def update_clusters(X, clusters):
for i in range(k):
points = [Link](clusters[i]['points'])
if [Link][0] > 0:
new_center = [Link](axis =0)
clusters[i]['center'] = new_center

clusters[i]['points'] = []
return clusters

Step 8: Predicting the Cluster for the Data Points

We create a function to predict the cluster for each data point based on the
final centroids.
[Link]([Link](dist)): Appends the index of the closest cluster (the
one with the minimum distance) to pred.

def pred_cluster(X, clusters):

pred = []
for i in range([Link][0]):
dist = []
for j in range(k):
[Link](distance(X[i],clusters[j]['center']))
[Link]([Link](dist))
return pred

Step 9: Assigning, Updating and Predicting the Cluster Centers

The assign and update steps are repeated multiple times until the cluster
centers stabilize or a maximum number of iterations is reached.

assign_clusters(X, clusters): Assigns data points to the nearest centroids.

update_clusters(X, clusters): Recalculates the centroids.
pred_cluster(X, clusters): Predicts the final clusters for all data points.

clusters = assign_clusters(X,clusters)
clusters = update_clusters(X,clusters)
pred = pred_cluster(X,clusters)

Step 10: Plotting Data Points with Predicted Cluster Centers

Finally, we plot the data points, colored by their predicted clusters, along with
the updated centroids.

center = clusters[i]['center']: Retrieves the center (centroid) of the current

cluster.
[Link](center[0], center[1], marker='^', c='red'): Plots the cluster
center as a red triangle (^ marker).

[Link](X[:,0],X[:,1],c = pred)
for i in clusters:
center = clusters[i]['center']
[Link](center[0],center[1],marker = '^',c = 'red')
[Link]()

Output:

Challenges with K-Means Clustering

Choosing the Right Number of Clusters (kk): One of the biggest challenges
is deciding how many clusters to use.
Sensitive to Initial Centroids: The final clusters can vary depending on the
initial random placement of centroids.
Non-Spherical Clusters: K-Means assumes that the clusters are spherical and
equally sized. This can be a problem when the actual clusters in the data are of
different shapes or densities.
Outliers: K-Means is sensitive to outliers, which can distort the centroid and,
ultimately, the clusters.
Density based clustering

It is a density-based clustering algorithm that groups data points that are

closely packed together and marks outliers as noise based on their density in
the feature space. It identifies clusters as dense regions in the data space
separated by areas of lower density. Unlike K-Means or hierarchical clustering
which assumes clusters are compact and spherical, DBSCAN perform well in
handling real-world data irregularities such as:

Arbitrary-Shaped Clusters : Clusters can take any shape not just circular or
convex.
Noise and Outliers: It effectively identifies and handles noise points without
assigning them to any cluster.
The figure above shows a data set with clustering algorithms: K-Means and
Hierarchical handling compact, spherical clusters with varying noise tolerance
while DBSCAN manages arbitrary-shaped clusters and noise handling.

Key Parameters in DBSCAN

eps: This defines the radius of the neighborhood around a data point. If the
distance between two points is less than or equal to eps they are considered
neighbours. A common method to determine eps is by analyzing the k-
distance graph. Choosing the right eps is important:
If eps is too small most points will be classified as noise.
If eps is too large clusters may merge and the algorithm may fail to distinguish
between them.
MinPts: This is the minimum number of points required within the eps radius
to form a dense region. A general rule of thumb is to set MinPts >= D+1
where D is the number of dimensions in the dataset.
For most cases a minimum value of MinPts = 3 is recommended.

How Does It Work

It works by categorizing data points into three types:

Core points which have a sufficient number of neighbours within a specified

radius (eplison)
Border points which are near core points but lack enough neighbours to be
core points themselves
Noise points which do not belong to any cluster.
By iteratively expanding clusters from core points and connecting density-
reachable points, DBSCAN forms clusters without relying on rigid
assumptions about their shape or size.
Steps in the DBSCAN Algorithm

Identify Core Points : For each point in the dataset count the number of points
within its eps neighborhood. If the count meets or exceeds MinPts mark the
point as a core point.
Form Clusters: For each core point that is not already assigned to a cluster
create a new cluster. Recursively find all density-connected points i.e points
within the eps radius of the core point and add them to the cluster.
Density Connectivity : Two points a and b are density-connected if there exists a
chain of points where each point is within the eps radius of the next and at
least one point in the chain is a core point. This chaining process ensures that
all points in a cluster are connected through a series of dense regions.
Label Noise Points: After processing all points any point that does not belong to
a cluster is labeled as noise.

Implementation Algorithm In Python

Step 1: Importing Libraries
We import all the necessary library like numpy , matplotlib and scikit-learn.

import [Link] as plt

import numpy as np
from [Link] import DBSCAN
from sklearn import metrics
from [Link] import make_blobs
from [Link] import StandardScaler
from sklearn import datasets
from [Link] import adjusted_rand_score

Step 2: Preparing Dataset

We will create a dataset of 4 clusters using make_blob. The dataset have 300
points that are grouped into 4 visible clusters.

X, y_true = make_blobs(n_samples=300, centers=4,

cluster_std=0.50, random_state=0)

Step 3: Applying DBSCAN Clustering

Now we apply DBSCAN clustering on our data, count it and visualize it using
the matplotlib library.

eps=0.3: The radius to look for neighboring points.

min_samples: Minimum number of points required to form a dense region a
cluster.
labels: Cluster numbers for each point. -1 means the point is considered noise.

db = DBSCAN(eps=0.3, min_samples=10).fit(X)
core_samples_mask = np.zeros_like(db.labels_, dtype=bool)
core_samples_mask[db.core_sample_indices_] = True
labels = db.labels_
n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)
unique_labels = set(labels)
colors = ['y', 'b', 'g', 'r']
print(colors)
for k, col in zip(unique_labels, colors):
if k == -1:

col = 'k'

class_member_mask = (labels == k)
xy = X[class_member_mask & core_samples_mask]
[Link](xy[:, 0], xy[:, 1], 'o', markerfacecolor=col,
markeredgecolor='k',
markersize=6)

xy = X[class_member_mask & ~core_samples_mask]

[Link](xy[:, 0], xy[:, 1], 'x', markerfacecolor=col,
markeredgecolor='k',
markersize=6)

[Link]('number of clusters: %d' % n_clusters_)

[Link]()
Output:
As shown in above output image cluster are shown in different colours like
yellow, blue, green and red.

Step 4: Evaluation Metrics

We will use the Silhouette score and Adjusted rand score for evaluating
clustering algorithms.
Silhouette's score is in the range of -1 to 1. A score near 1 denotes the best
meaning that the data point i is very compact within the cluster to which it
belongs and far away from the other clusters. The worst value is -1. Values
near 0 denote overlapping clusters.
Adjusted Rand Score is in the range of 0 to 1. More than 0.9 denotes excellent
cluster recovery and above 0.8 is a good recovery. Less than 0.5 is considered
to be poor recovery.

from sklearn import metrics

sc = metrics.silhouette_score(X, labels)
print("Silhouette Coefficient:%0.2f" % sc)
ari = metrics.adjusted_rand_score(y_true, labels)
print("Adjusted Rand Index: %0.2f" % ari)
Output:
Coefficient:0.13
Adjusted Rand Index: 0.31

Black points represent outliers. By changing the eps and the MinPts we can
change the cluster configuration.

DBSCAN and K-Means are both clustering algorithms that group together
data that have the same characteristic. However they work on different
principles and are suitable for different types of data. We prefer to use
DBSCAN when the data is not spherical in shape or the number of classes is
not known beforehand.

DBSCAN K-Means

It is very sensitive to the

In DBSCAN we need not specify the number
number of clusters (k), which
of clusters.
must be specified in advance.

Clusters formed in DBSCAN can be of any Clusters formed are spherical

arbitrary shape. or convex in shape

It does not work well with

It can work well with datasets having noise outliers data. Outliers can skew
and outliers the clusters in K-Means to a
very large extent.

In DBSCAN two parameters are required for In K-Means only one

training the Model parameter is required is for
 DBSCAN K-Means

training the model

As it can identify clusters of arbitrary shapes and effectively handle noise. K-

Means on the other hand is better suited for data with well-defined, spherical
clusters and is less effective with noise or complex cluster structures.

Introduction to Dimensionality Reduction

Dimensionality reduction is a technique used to reduce the number of features

in a dataset while preserving important information. It transforms high-
dimensional data into a lower-dimensional space for simpler representation.

Reduces computation time by lowering the number of features

Helps prevent over-fitting by removing irrelevant data
Improves data visualization and understanding
Working
Imagine a dataset where each data point exists in a 3D space defined by axes X,
Y and Z. If most of the data variance occurs along X and Y then the Z-
dimension may contribute very little to understanding the structure of the data.

Before Reduction we can see that data exist in 3D (X,Y,Z). It has high
redundancy and Z contributes little meaningful information
On the right after reducing the dimensionality the data is represented in lower-
dimensional spaces. The top plot (X-Y) maintains the meaningful structure
while the bottom plot (Z-Y) shows that the Z-dimension contributed little
useful information.
This process makes data analysis more efficient hence improving computation
speed and visualization while minimizing redundancy

Dimensionality Reduction Techniques

Dimensionality reduction techniques can be broadly divided into two
categories:
 Feature Selection
Feature selection chooses the most relevant features from the dataset without
altering them. It helps remove redundant or irrelevant features, improving
model efficiency. Some common methods are:
Filter methods: Rank the features based on their relevance to the target
variable.
Wrapper methods: Use the model performance as the criteria for selecting
features.
Embedded methods: Combine feature selection with the model training
process.
Missing Value Ratio: Variables with missing data beyond a set threshold are
removed, improving dataset reliability.
Backward Feature Elimination: Starts with all features and removes the
least significant ones in each iteration. The process continues until only the
most impactful features remain, optimizing model performance.
Forward Feature Selection: It begins with one feature, adds others
incrementally and keeps those improving model performance.
Random Forest: It uses decision trees to evaluate feature importance,
automatically selecting the most relevant features without the need for manual
coding, enhancing model accuracy.

Feature Extraction
Feature extraction involves creating new features by combining or
transforming the original features. These new features retain most of the
dataset’s important information in fewer dimensions. Common feature
extraction methods are:
Principal Component Analysis (PCA): Converts correlated variables into
uncorrelated principal components hence reducing dimensionality while
maintaining as much variance as possible enabling more efficient analysis.
Factor Analysis: Groups variables by correlation and keeps the most relevant
ones for further analysis.
Independent Component Analysis (ICA): Identifies statistically independent
components, ideal for applications like ‘blind source separation’ where
traditional correlation-based methods fall short.

Real World Use Case

Text categorization: Reduces feature space (words/phrases) to classify
documents accurately from large datasets.
Image retrieval: Uses visual features like color, texture, and shape to
improve search in large image databases.
Gene expression analysis: Identifies key features to classify samples like
leukemia with better speed and accuracy.
Intrusion detection: Analyzes activity patterns to detect threats by selecting
important features for monitoring.

Advantages
Reduces computation time as models process fewer features.
Makes data easier to visualize and understand patterns.
Helps reduce overfitting and improves model generalization.

Disadvantages
May lead to loss of important information from the data.
Choosing the right number of dimensions can be challenging.
Excessive reduction can negatively affect model accuracy.
Collaborative Filtering

A recommendation system analyses user behaviour and past activity to

understand preferences and suggest relevant content, products or ideas. By
tracking what users watch, click or interact with, it identifies patterns and
continuously improves recommendations to enhance user experience and
engagement.

Tracks user activity like views, clicks and interactions

Identifies patterns in user preferences
Predicts likes and dislikes based on past behavior
Recommends similar or relevant content
Continuously updates suggestions with new data
Improves personalization and user engagement

Collaborative Filtering
Collaborative filtering works by identifying users with similar preferences and
recommending items based on what those similar users like. Instead of using
item features, it groups users into clusters and suggests content according to
the shared preferences of each group.

Focuses on user behaviour, not item features

Finds users with similar interests
Groups users into clusters based on preferences
Recommends items liked by similar users
Types of Collaborative Filtering Techniques

Memory-Based: Uses user-item data directly to make recommendations

Model-Based: Builds predictive models using machine learning
Hybrid: Combines multiple approaches for better results
Deep Learning: Uses neural networks for more advanced recommendations

Measuring Similarity in Collaborative Filtering

Collaborative filtering works by comparing user preferences and identifying
similarities in their ratings. Based on these similarities, the system predicts
what a user might like or dislike and recommends items accordingly.
Example:

User 1 and User 2 have nearly similar ratings (both liked Movie 1), showing
similar preferences
Based on this, Movie 3 (liked by User 2) can be recommended to User 1
Similarly, Movie 4 (liked by User 1) can be recommended to User 2
User 1 and User 3 have opposite tastes, so their recommendations will differ
User 3 and User 4 share similar low ratings for Movie 2
Since User 3 disliked Movie 4, it can be predicted that User 4 may also dislike
Movie 4
Cosine Similarity in Collaborative Filtering
Cosine similarity measures how similar two users are based on their ratings. A
higher cosine value means users have similar preferences, while a lower value
means they are different. Missing values are often treated as 0 to simplify
calculations.
Measures similarity between users using their rating patterns
Higher cosine value means more similar users
Lower cosine value means less similar users
Missing ratings can be filled with 0 for easy calculation
Helps in recommending items liked by similar users

Rounding the Data

Rounding is used to simplify rating data by converting it into binary values.
Ratings below 3 are set to 0 (dislike), and ratings 3 or above are set to 1 (like).
This makes comparison between users faster and easier.

Converts complex ratings into simple 0 and 1 values

Improves readability of the data
Makes similarity comparison more efficient
Helps clearly identify similar user groups

Example:
After rounding, User 1 and User 2 show similar patterns (more 1s) which
means similar preferences
User 3 and User 4 show similar patterns (more 0s) which means similar dislikes

Normalizing Rating
Normalization adjusts user ratings by subtracting each user’s average rating
from their given ratings. This converts values into positive and negative scores,
making it easier to compare user preferences fairly.

Subtracts the user’s average rating from each rating

Produces positive (above average) and negative (below average) values
Removes bias of users who rate consistently high or low
Helps group users with similar rating patterns
Improves accuracy of recommendations by better similarity detection

Advantages
Unlike content-based systems, it does not rely on limited item features making
it more flexible in different use cases
It can handle a wide variety of data since it learns from user interactions instead
of predefined content
It provides strong personalization by recommending items based on similar
users’ preferences
It adapts easily to changes in user behaviour over time, improving
recommendations continuously
It performs well when large amounts of user data are available, increasing
accuracy and relevance

Challenges
As the number of users and items increases, computation and storage
requirements grow significantly, making the system slower
Scalability becomes a major issue with large datasets, affecting performance
and accuracy
Relies heavily on historical data, so it may struggle when there is limited or
new user data
Tends to recommend similar types of items repeatedly, reducing diversity in
recommendations
May not capture changing interests instantly if recent data is limited
UNIT-5
Association Rule
Association rules are a fundamental concept used to find relationships,
correlations or patterns within large sets of data items. They describe how
often item sets occur together in transactions and express implications of the
form:

Where X and Y are disjoint sets of items. This rule suggests that when items
in XX appear, items in YY tend to appear as well. Association rules originated
from market basket analysis and help retailers and analysts understand customer
behavior by discovering item associations in transaction data. For example, a
rule stating

Key Components

Antecedent (X): The "if" part representing one or more items found in
transactions.
Consequent (Y): The "then" part, representing the items likely to be purchased
when antecedent items appear.
Rules are evaluated based on metrics that quantify their strength and usefulness:

Rule Evaluation Metrics

 Support: Fraction of transactions containing the itemsets in both X and
Y.

Support measures how frequently the combination appears in the data.

Confidence: Probability that transactions with X also include Y.

Confidence measures the reliability of the inference.

Lift: The ratio of observed support to that expected if X and Y were

independent

Lift > 1 implies a positive association — items occur together more than
expected.
Lift = 1 implies independence.
Lift < 1 implies a negative association.

Example Transaction Data

Considering the rule:

Implementation
Step 1: Install and Import Libraries

We will install and import all the required libraries such as pandas,
mixtend, matplotlib, networkx.

!pip install pandas mlxtend matplotlib seaborn networkx

import pandas as pd
from [Link] import TransactionEncoder
from mlxtend.frequent_patterns import apriori, association_rules
import [Link] as plt
import seaborn as sns
import networkx as nx

Step 2: Load and Preview Dataset

We will upload the dataset,

data = pd.read_csv("Groceries_dataset.csv")

print([Link]())

output

Step 3: Prepare Data for Apriori Algorithm

Apriori requires this one-hot encoded format where columns = items and rows =
transactions with True/False flags.

transactions = [Link]('Member_number')[
'itemDescription'].apply(list).[Link]()

te = TransactionEncoder()
te_ary = [Link](transactions).transform(transactions)
df = [Link](te_ary, columns=te.columns_)
[Link]()
Output:

Step 4: Generate Frequent Itemsets

Finds itemsets appearing in ≥ 1% of all transactions.

use_colnames=True to keep item names readable.

frequent_itemsets = apriori(df, min_support=0.01, use_colnames=True)

print(frequent_itemsets.head())

Output:

Step 5: Generate Association Rules

Extract rules with confidence ≥ 30%.

Rules DataFrame includes columns like antecedents, consequents, support,
confidence and lift.

rules = association_rules(
frequent_itemsets, metric="confidence", min_threshold=0.3)

print([Link]())

Output:

Step 6: Visualize Top Frequent Items

Visualizes the 10 most purchased items.
Helps understand popular products in the dataset.

item_frequencies = [Link]().sort_values(ascending=False)

[Link](figsize=(10, 6))
[Link](x=item_frequencies.head(10).values,
y=item_frequencies.head(10).index)
[Link]('Top 10 Frequent Items')
[Link]('Frequency')
[Link]('Items')
[Link]()
Output:

Step 7: Scatter Plot of Rules (Support vs Confidence)

Shows the relationship between support and confidence for rules.

Color encodes the strength of rules via lift.

[Link](figsize=(8, 6))
scatter = [Link](rules['support'], rules['confidence'],
c=rules['lift'], cmap='viridis', alpha=0.7)
[Link](scatter, label='Lift')
[Link]('Support')
[Link]('Confidence')
[Link]('Scatter Plot of Association Rules')
[Link]()

Output:
Step 8: Heatmap of Confidence for Selected Rules

Shows confidence values between top antecedent and consequent itemsets.

A quick way to identify highly confident rules.

rules['antecedents_str'] = rules['antecedents'].apply(
lambda x: ', '.join(list(x)))
rules['consequents_str'] = rules['consequents'].apply(
lambda x: ', '.join(list(x)))

top_ants = [Link]('antecedents_str')['support'].sum().nlargest(10).index
top_cons = [Link]('consequents_str')['support'].sum().nlargest(10).index

filtered = rules[(rules['antecedents_str'].isin(top_ants)) &

(rules['consequents_str'].isin(top_cons))]
heatmap_data = [Link](
index='antecedents_str', columns='consequents_str', values='confidence')

[Link](figsize=(12, 8))
[Link](heatmap_data, annot=True, cmap='YlGnBu',
linewidths=0.5, cbar_kws={'label': 'Confidence'})
[Link]('Heatmap of Confidence for Top Association Rules')
[Link]('Consequents')
[Link]('Antecedents')
[Link]()

Output:

Use Cases
Market Basket Analysis : Identifies products often bought together to improve
store layouts and promotions (e.g., bread and butter).
Recommendation Systems : Suggests related items based on buying patterns (e.g.,
accessories with laptops).
Fraud Detection: Detects unusual transaction patterns indicating fraud.
Healthcare Analytics : Finds links between symptoms, diseases and treatments
(e.g., symptom combinations predicting a disease).

Advantages
Interpretable and Easy to Explain : Rules offer clear “if-then” relationships
understandable to non-technical stakeholders.
Unsupervised Learning : Works well on unlabeled data to find hidden patterns
without prior knowledge.
Flexible Data Types: Effective on transactional, categorical and binary data.
Helps in Feature Engineering : Can be used to create new features for downstream
supervised models.

Limitations
Large Number of Rules : Can generate many rules, including trivial or redundant
ones, making interpretation hard.
Support Threshold Sensitivity : High support thresholds miss interesting but
infrequent patterns; low thresholds generate too many rules.
Not Suitable for Continuous Variables : Requires discretization or binning before
use with numerical attributes.
Computationally Expensive : Performance degrades on very large or dense
datasets due to combinatorial explosion.
Statistical Significance : High confidence doesn’t guarantee a meaningful rule;
domain knowledge is essential to validate findings.
ECLAT Algorithm

ECLAT stands for Equivalence Class Clustering and bottom-up Lattice

Traversal. It is a data mining algorithm used to find frequent item sets in a
dataset. These frequent itemsets are then used to create association rules
which helps to identify patterns in data. It is an improved alternative to the
Apriori algorithm by providing better scalability and computational efficiency.

What Makes ECLAT Different from Apriori?

The main difference between the two lies in how they store and search
through the data:

Apriori uses a horizontal format where each transaction is a row and it follows
a breadth-first search (BFS) strategy. This means it scans the database
multiple times to find frequent item combinations.
ECLAT on the other hand uses a vertical format where each item is linked to a
list of transaction IDs (TIDs). It uses a depth-first search (DFS) strategy which
requires fewer scans and makes it faster and more memory-efficient.
This vertical approach significantly reduces the number of database scans
making ECLAT faster and more memory-efficient especially for large
datasets.
 Aspect Apriori ECLAT

Horizontal (transactions Vertical (items linked to

Data Format as rows) transaction IDs)

Search Breadth-First Search

Depth-First Search (DFS)
Strategy (BFS)

Database
Multiple scans required Fewer scans needed
Scans

Memory
Less memory-efficient More memory-efficient
Efficiency

Slower, especially with Faster due to vertical

Speed large datasets representation

Working
Let’s walk through an example to better understand how ECLAT algorithm
works. Consider the following transaction dataset represented in a Boolean
matrix:
 Brea
Transaction ID d Butter Milk Coke Jam

T1 1 1 0 0 1

T2 0 1 0 1 0

T3 0 1 1 0 0

T4 1 1 0 1 0

T5 1 0 1 0 0

T6 0 1 1 0 0

T7 1 0 1 0 0

T8 1 1 1 0 1

T9 1 1 1 0 0

The core idea of the ECLAT algorithm is based on the interaction of datasets
to calculate the support of itemsets, avoiding the generation of subsets that are
not likely to exist in the dataset. Here’s a breakdown of the steps:
Step 1: Create the Tidset

The first step is to generate the tidset for each individual item. A tidset is
simply a list of transaction IDs where the item appears. For example: k = 1,
minimum support = 2

Item Tidset

Bread {T1, T4, T5, T7, T8, T9}

Butter {T1, T2, T3, T4, T6, T8, T9}

Milk {T3, T5, T6, T7, T8, T9}

Coke {T2, T4}

Jam {T1, T8}

Step 2: Calculate the Support of Itemsets by Intersecting Tidsets

ECLAT then proceeds by recursively combining the tid sets. The support of
an item set is determined by the intersection of tid sets. For example: k = 2
 Item Tidset

{Bread, Butter} {T1, T4, T8, T9}

{Bread, Milk} {T5, T7, T8, T9}

{Bread, Coke} {T4}

{Bread, Jam} {T1, T8}

{Butter, Milk} {T3, T6, T8, T9}

{Butter, Coke} {T2, T4}

{Butter, Jam} {T1, T8}

{Milk, Jam} {T8}

Step 3: Recursive Call and Generation of Larger Itemsets

The algorithm continues recursively by combining pairs of itemsets (k-

itemsets) checking the support by intersecting the tidsets. The recursion
continues until no further frequent itemsets can be generated. Now k = 3
 Item Tidset

{Bread, Butter,
{T8, T9}
Milk}

{Bread, Butter, Jam} {T1, T8}

Step 4: Stop When No More Frequent Itemsets Can Be Found

The algorithm stops once no more itemset combinations meet the minimum
support threshold. k = 4

Item Tidset

{Bread, Butter, Milk,

{T8}
Jam}

We stop at k = 4 because there are no more item-tidset pairs to combine. Since

minimum support = 2, we conclude the following rules from the given
dataset:-

Items Bought Recommended Products

Bread Butter

Bread Milk

Bread Jam
 Items Bought Recommended Products

Butter Milk

Butter Coke

Butter Jam

Bread and
Milk
Butter

Bread and
Jam
Butter

Step 1: Import Packages and Dataset

We will import necessary libraires and provide the dataset.

from collections import defaultdict

from itertools import combinations
transactions = {
"T1": ["Bread", "Butter", "Jam"],
"T2": ["Butter", "Coke"],
"T3": ["Butter", "Milk"],
"T4": ["Bread", "Butter", "Coke"],
"T5": ["Bread", "Milk"],
"T6": ["Butter", "Milk"],
"T7": ["Bread", "Milk"],
"T8": ["Bread", "Butter", "Milk", "Jam"],
"T9": ["Bread", "Butter", "Milk"]
}
min_support = 2

Step 2: Generate Tidsets (Vertical representation)

Purpose: create a mapping item -> set_of_tids (the vertical format ECLAT
uses).
Benefit: intersections of these tidsets are quick to compute and give support
counts.
def generate_tidsets(transactions):
item_tidset = defaultdict(set)
for tid, items in [Link]():
for item in items:
item_tidset[item].add(tid)
return item_tidset

item_tidset = generate_tidsets(transactions)

for item, tidset in item_tidset.items():

print(item, ":", sorted(tidset))

Step 3: Prepare a sorted list of items

Purpose: convert the item_tidset dict into a sorted list of (item, tidset) pairs.
Tip: sorting by tidset size (ascending) often helps pruning and makes
intersections cheaper earlier.

items = sorted(item_tidset.items(), key=lambda x: len(x[1]))

Step 4: Implement recursive ECLAT

Recursively build larger itemsets by intersecting tidsets (depth-first). How it
works:

Pop one (item, tidset) from the list.

If len(tidset) >= min_support, record the itemset (prefix + item).
Build a suffix by intersecting this tidset with each remaining item's tidset;
keep intersections that meet min_support.
Recurse on the suffix to extend the current itemset.

Step 5: Run ECLAT and collect frequent itemsets

Call the recursive function, then inspect the found frequent itemsets (with
support counts).

item_tidset = generate_tidsets(transactions)
items = sorted(item_tidset.items(), key=lambda x: len(x[1]))
frequent_itemsets = {}
eclat([], items, min_support, frequent_itemsets)

print("Frequent itemsets (as list) -> support count")

for itemset, support in sorted(frequent_itemsets.items(), key=lambda x: (-len(x[0]), -
x[1], sorted(list(x[0])))):
print(list(itemset), "=>", support)

Applications
Market Basket Analysis: Identifying frequently purchased items together.
Recommendation Systems: Suggesting products based on past purchase
patterns.
Medical Diagnosis: Finding co-occurring symptoms in medical records.
Web Usage Mining: Analyzing web logs to understand user behavior.
Fraud Detection: Discovering frequent patterns in fraudulent activities.

Advantages
Efficient in Dense Datasets: Performs better than Apriori in datasets with
frequent co-occurrences.
Memory Efficient: Uses vertical representation, reducing redundant scans.
Fast Itemset Intersection: Computing itemset support via TID-set
intersections is faster than scanning transactions repeatedly.
Better Scalability: Can handle larger datasets due to its depth-first search
mechanism.

Disadvantages
High Memory Requirement: Large TID sets can consume significant
memory.
Not Suitable for Sparse Data: Works better in dense datasets, but
performance drops for sparse datasets where intersections result in small
itemsets.
Sensitive to Large Transactions: If a transaction has too many items its
corresponding TID-set intersections can be expensive.

Foundational Concepts

Itemset

A collection of one or more items appearing together in a transaction.

1-itemset: {bread}

2-itemset: {bread, butter}

k-itemset: Any itemset with k items

Transaction Database

A collection of transactions where each transaction is a set of items.

T1: {bread, milk, butter}

T2: {bread, eggs}
T3: {milk, butter, cheese}
T4: {bread, milk, butter, eggs}

Frequent Itemset

An itemset whose support ≥ minimum support threshold set by the user.

Infrequent / Rare Itemset

An itemset whose support falls below the minimum threshold.

Core Measures & Interestingness

Support

Fraction of transactions containing the item set.

Support(A) = |T(A)| / |T|

Confidence

Probability that B occurs given A occurs.

Confidence(A→B) = Support(A∪B) / Support(A)

Lift

How much more likely B is given A, compared to B alone.

Lift(A→B) = Confidence(A→B) / Support(B)

Conviction

Measures how often the rule is wrong.

Conviction = (1 - Support(B)) / (1 - Confidence(A→B))

Higher conviction = stronger rule

Leverage

Difference between observed and expected frequency.

Leverage(A→B) = Support(A∪B) − Support(A) × Support(B)

Leverage = 0 → A and B are independent

Jaccard Coefficient

Similarity between two itemsets.

Jaccard(A,B) = Support(A∪B) / (Support(A) + Support(B) − Support(A∪B))

Kulczynski Measure

Average of two conditional probabilities.

Kulc(A,B) = ½ × [P(A|B) + P(B|A)]

Null-transaction invariant (not affected by empty transactions)

Cosine Measure (IS Measure)

Cosine(A,B) = Support(A∪B) / √(Support(A) × Support(B))

The Apriori Principle (Downward Closure)

The most fundamental concept in ARL:

If an itemset is frequent → all its subsets are frequent If an itemset is infrequent → all its
supersets are infrequent

This enables pruning eliminating huge numbers of candidate itemsets without scanning
the database, making algorithms computationally feasible.

Candidate Generation & Pruning

Join Step

Combine two frequent (k-1)-item sets that share the same first (k-2) items to generate a
candidate k-item set.

Prune Step

Remove any candidate whose any (k-1)-subset is infrequent — using the Apriori
principle.

Support Counting

Scan the database to verify which candidates actually meet minimum support.

Data Representations

Horizontal Format (Transaction-based)

T1 → {A, B, C}
T2 → {A, C}
T3 → {B, D}

Used by Apriori and FP-Growth.

Vertical Format (TID-list)

A → {T1, T2}
B → {T1, T3}
C → {T1, T2}

Used by ECLAT — support = size of TID intersection.

Bit Vector Representation

Items represented as binary vectors across transactions.

Fast bitwise AND operations for support counting.

FP-Tree Concepts

FP-Tree (Frequent Pattern Tree)

A compressed prefix-tree structure storing the database in only 2 scans.

Nodes contain:

Item name

Count (support)

Node-link (pointer to same item elsewhere in tree)

Header Table: Links all occurrences of each item across the tree.

Conditional Pattern Base

For item X — all prefix paths in the FP-Tree leading to X, with adjusted counts.

Conditional FP-Tree

FP-Tree built from the conditional pattern base of item X — used to recursively mine
patterns.
Mining Order

Items mined in reverse order of frequency (least frequent first) to keep conditional trees
small.

Lattice Theory in ARL

Itemset Lattice

A hierarchical structure showing all possible subsets and supersets of itemsets.

{A, B, C}
/ | \
{A, B} {A, C} {B,C}
| \/ | \/ |
{A} {B} {C}
|
{}

Bottom-up traversal → finds small frequent item sets first (Apriori style)

Top-down traversal → starts from maximal item sets

Depth-first → ECLAT, H-Mine approach

Maximal Frequent Itemset

A frequent item set with no frequent superset most compact representation of all frequent
item sets.

Closed Frequent Itemset

A frequent item set with no superset with the same support lossless compression of all
frequent item sets.
Closed ⊇ Maximal (all maximal item sets are closed but not vice versa)

Rule Generation Concepts

Antecedent & Consequent

Antecedent (LHS): The "if" part → {bread, milk}

Consequent (RHS): The "then" part → {butter}

Rule Strength

Determined by support, confidence, and lift together — not any single measure alone.

Redundant Rules

Rule A→B is redundant if a more specific rule A'→B (where A' ⊂ A) has equal or higher
confidence.

Rule Pruning

Remove redundant and uninteresting rules using confidence-based or lift-based filtering.

Null Transaction Problem

Null Transaction

A transaction that contains neither A nor B — irrelevant to the A→B relationship.

Problem: Measures like support and confidence are inflated by null transactions in sparse
datasets.

Null-invariant measures (not affected):

Kulczynski

Cosine (IS measure)

Jaccard

All-confidence

Not null-invariant:

Support, Confidence, Lift, Conviction, Leverage

Thresholds & Parameter Selection

Minimum Support (min_sup)

Too high → misses rare but important patterns

Too low → generates exponentially many item sets

Minimum Confidence (min_conf)

Too high → misses valid rules

Too low → too many trivial rules

Support-Confidence Framework

The standard threshold pair used to filter rules — but does NOT guarantee interestingness
(a rule can have high support & confidence yet be misleading if lift < 1).

Types of Patterns

Positive Association Rule

A → B (presence of A implies presence of B)

Negative Association Rule

A → ¬B (presence of A implies absence of B) Requires modified support/confidence
definitions.

Approximate Rule

Holds with some tolerance — used in noisy or probabilistic data.

Actionable Rule

A rule that provides useful, actionable business insight — the ultimate goal.

Trivial Rule

Technically valid but provides no new information (e.g., stating the obvious).

Multilevel & Multidimensional Concepts

Concept Hierarchy

Items organized into abstraction levels:

Electronics → Laptops → Dell Laptops → Dell XPS 15

Rules mined at different levels = Generalized Association Rules.

Cross-level Rules

Rules spanning different abstraction levels:

{Dell Laptops} → {Mouse} (mix of specific and general)

Multidimensional Association Rules

Rules involving multiple attributes/dimensions:

age(25-35) ∧ occupation(engineer) → buys(smartphone)

Sequential Pattern Mining (Extension of ARL)

Considers order of itemsets across time — not just co-occurrence.

{buy camera} → {buy memory card} → {buy printer}

Key algorithms:

GSP (Generalized Sequential Patterns) — Apriori-based

PrefixSpan — Projection-based, like FP-Growth for sequences

SPADE — Vertical format, like ECLAT for sequences

Constraint-Based Mining

Mining rules under user-defined constraints to reduce search space and improve
relevance.

Types of Constraints:

Type Example

Data constraint Only transactions from Electronics dept

Dimension constraint Rules must involve "price" attribute

Rule constraint Consequent must be a single item

Interestingness constraint Lift > 1.5

Anti-monotone Max(price) ≤ 100 — if violated, all supersets also violate

Monotone Min(price) ≥ 10 — if satisfied, all supersets also satisfy

Succinct Can be directly pushed into itemset generation

Convertible Satisfied by ordering items appropriately

Evaluation & Validation Concepts

Objective Measures

Computed from data — support, confidence, lift, leverage, conviction.

Subjective Measures

Based on user knowledge:

Unexpectedness — Rule contradicts prior belief

Actionability — Rule can drive a useful decision

Interestingness

A rule is interesting if it is: unexpected, actionable, and novel.

Lift vs. Other Measures

Lift handles some null-transaction issues but not all

No single measure works best for all scenarios

Use multiple measures together for robust rule evaluation

Scalability Concepts

Sampling

Mine rules on a random sample → verify on full database. Faster but may miss some
patterns.

Partitioning

Divide DB into partitions → mine each → combine. Any globally frequent itemset must
be locally frequent in at least one partition.
Incremental Mining

Update rules when new transactions are added without re-mining from scratch.

Parallel & Distributed Mining

Distribute the database and mining across multiple processors/nodes — essential for big
data ARL.

Apriori Algorithm

Apriori Algorithm is a data mining technique used to identify items that

frequently appear together in large datasets. It helps discover relationships and
association rules between items, making it widely used in market basket
analysi

For Example:
If customers often buy bread and butter together in a grocery store, the store
can place these items nearby or create combo offers to improve sales and
customer experience.

Finds frequent item combinations in datasets

Discovers association rules between items
Widely used in market basket analysis
Helps businesses improve recommendations and sales strategies

Working
Identifying Frequent Item-Sets
The Apriori algorithm starts by looking through all the data to count how
many times each single item appears. These single items are called 1-Item-
Sets.
Next it uses a rule called minimum support this is a number that tells us how
often an item or group of items needs to appear to be important. If an item
appears often enough meaning its count is above this minimum support it is
called a frequent Item-Set.

Creating Possible Item Group

After finding the single items that appear often enough (frequent 1-item
groups) the algorithm combines them to create pairs of items (2-item groups).
Then it checks which pairs are frequent by seeing if they appear enough times
in the data.
This process keeps going step by step making groups of 3 items, then 4 items
and so on. The algorithm stops when it can’t find any bigger groups that
happen often enough.

Removing Infrequent Item Groups

The Apriori algorithm uses a helpful rule to save time. This rule says, if a
group of items does not appear often enough then any larger group that
includes these items will also not appear often.
Because of this, the algorithm does not check those larger groups. This way it
avoids wasting time looking at groups that won’t be important making the
whole process faster.

Generating Association Rules

The algorithm makes rules to show how items are related.
It checks these rules using support, confidence and lift to find the strongest
ones.

Key Metrics of Apriori Algorithm

Support

Support measures how frequently an item or item-set appears in the dataset

relative to the total number of transactions.

Indicates the overall occurrence of an item-set

Higher support means the item-set appears more frequently

Confidence

Confidence measures the likelihood that item Y is purchased when item X is

purchased.

Indicates the strength of association between items

Shows how often items occur together

Lift
Lift measures how much more likely two items are purchased together
compared to random chance.

Evaluates the strength of item relationships

Lift greater than 1 indicates a positive association
Example
Let's understand the concept of apriori Algorithm with the help of an example.
Consider the following dataset and we will find frequent Item-Sets and
generate association rules for them:

Step 1 : Setting the parameters

Minimum Support Threshold: 50% (item must appear in at least 3/5
transactions). This threshold is formulated from this formula:
Minimum Confidence Threshold: 70% ( You can change the value of
parameters as per the use case and problem statement ). This threshold is
formulated from this formula:
Step 2: Find Frequent 1-Item-Sets
Let's count how many transactions include each item in the dataset
(calculating the frequency of each item).

All items have support ≥ 50%, so they qualify as frequent 1-Item-Sets. If any
item has support < 50%, It will be omitted from the frequent 1- Item-Sets.

Step 3: Generate Candidate 2-Item-Sets

Combine the frequent 1-Item-Sets into pairs and calculate their support. For
this use case we will get 3 item pairs ( bread,butter) , (bread,milk) and
(butter,milk). Their support values are calculated similarly to Step 2.

Frequent 2-Item-Sets: {Bread, Milk} meets the 50% minimum support

threshold. However, {Bread, Butter} and {Butter, Milk} do not meet the
threshold, so they are omitted.

Step 4: Generate Candidate 3-Item-Sets

The Apriori Algorithm generates candidate 3-itemsets only from frequent 2-
itemsets. Since only {Bread, Milk} satisfies the minimum support threshold in
Step 3, there is no valid 3-itemset can be generated.

Step 5: Generate Association Rules

Now we generate association rules from the frequent itemsets and calculate
their confidence values.

Rule 1: If Bread implies Butter

If a customer buys Bread, they are likely to buy Butter as well.

Support of {Bread, Butter} = 2.

Support of {Bread} = 4.
Confidence = 2/4 = 50% (Fails threshold)

Rule 2: Butter implies Bread

If a customer buys Butter, they are likely to buy Bread as well.

Support of {Bread, Butter} = 2.

Support of {Butter} = 3.
Confidence = 2/3 = 66.67% (Fails threshold).

Rule 3: Bread implies Milk

If a customer buys Bread, they are likely to buy Milk as well.

Support of {Bread, Milk} = 3.

Support of {Bread} = 4.
Confidence = 3/4 = 75% (Passes threshold).
The Apriori Algorithm, as demonstrated in the bread-butter example, is
widely used in modern startups like Zomato, Swiggy and other food delivery
platforms. These companies use it to perform market basket analysis which
helps them identify customer behaviour patterns and optimise
recommendations.

Applications
Used in e-commerce to recommend products that are frequently bought
together
Helps food delivery platforms identify popular meal combinations and combo
offers
Enables streaming services to recommend related movies and shows
Assists financial services in analyzing spending patterns and personalized
offers
Supports travel platforms in creating combined travel and hotel packages
Used in health and fitness applications for personalized recommendations
based on user behaviour