Data Analytics using Python Unit – 3 Understanding Supervised Learning

Contents
Understanding Supervised Learning................................................................................. 2
Overview and Concepts of Supervised Learning........................................................... 2
Classification: ........................................................................................................... 2
Regression:............................................................................................................... 3
Difference between Classification and Regression ....................................................... 4
Components (Process) of Supervised Machine Learning: ............................................. 5
Features ................................................................................................................... 5
Labels: ...................................................................................................................... 6
Dataset .......................................................................................................................... 6
Training Data, Test Data, validation Data ................................................................. 8
Overfitting, Underfitting ............................................................................................ 10
What is Underfitting? ............................................................................................. 10
What is Overfitting? ............................................................................................... 10
Python Code for Simple Linear Regression ................................................................. 11
Cost/Loss functions: ................................................................................................... 12
Regression Cost Functions ......................................................................................... 13
Mean Squared Error (MSE)..................................................................................... 13
Mean Absolute Error (MAE) ................................................................................... 15
Root Mean Square Error (RMSE) ............................................................................ 16
Understanding R2 ................................................................................................... 17
Classification: ............................................................................................................. 18
Types of Classification ............................................................................................ 18
Classification Algorithms ........................................................................................ 19
Logistic Regression in Machine Learning .................................................................... 20
Types of Logistic Regression ................................................................................... 21
Assumptions of Logistic Regression ........................................................................ 21
Understanding Sigmoid Function ........................................................................... 22
Python Code for Classification(Binary Logistic Regression – Single Input) .............. 23
Confusion Matrix:................................................................................................... 24
Metrics based on Confusion Matrix Data ............................................................... 24
Example of above Metrics: ..................................................................................... 26

Shri [Link] college of CS and BM Page | 1

Data Analytics using Python Unit – 3 Understanding Supervised Learning

3. Understanding Supervised Learning

❖ Overview and Concepts of Supervised Learning
▪ Supervised Learning is the most common branch of Machine Learning.
▪ It is defined by its use of labeled datasets to train algorithms that classify data or
predict outcomes accurately.
▪ Supervised machine learning algorithms make it easier for organizations to create
complex models that can make accurate predictions.
▪ As a result, they are widely used across various industries and fields, including
healthcare, marketing, financial services, and more.
▪ The data used in supervised learning is labelled — meaning that it contains
examples of both inputs (called features) and correct outputs (labels).
▪ The algorithms analyse a large dataset of these training pairs to infer what a
desired output value would be when asked to make a prediction on new data.
▪ For instance, let’s pretend you want to teach a model to identify pictures of trees.
▪ You provide a labelled dataset that contains many different examples of types of
trees and the names of each species.
▪ You let the algorithm try to define what set of characteristics belongs to each tree
based on the labelled outputs.
▪ You can then test the model by showing it a tree picture and asking it to guess
what species it is.
▪ If the model provides an incorrect answer, you can continue training it and
adjusting its parameters with more examples to improve its accuracy and
minimize errors.
▪ Once the model has been trained and tested, you can use it to make predictions
on unknown data based on the previous knowledge it has learned.
▪ There are two types of supervised learning algorithms:
• Classification
• Regression

 Classification:
▪ Classification is a type of supervised machine learning where algorithms learn
from the data to predict an outcome or event in the future.
▪ For example: A bank may have a customer dataset containing credit history,
loans, investment details, etc. and they may want to know if any customer will
default. In the historical data, we will have Features and Target.
▪ Features will be attributes of a customer such as credit history, loans,
investments, etc.
▪ Target will represent whether a particular customer has defaulted in the past
(normally represented by 1 or 0 / True or False / Yes or No).

Shri [Link] college of CS and BM Page | 2

Data Analytics using Python Unit – 3 Understanding Supervised Learning
▪ Classification algorithms are used for predicting discrete outcomes, if the
outcome can take two possible values such as True or False, Default or No Default,
Yes or No, it is known as Binary Classification.
▪ When the outcome contains more than two possible values, it is known as
Multiclass Classification.
▪ There are many machine learning algorithms that can be used for classification
tasks.
▪ Some of them are:
• Logistic Regression
• Decision Tree Classifier
• K Nearest Neighbor Classifier
• Random Forest Classifier
• Neural Networks

 Regression:
▪ Regression is a type of supervised machine learning where algorithms learn from
the data to predict continuous values such as sales, salary, weight, or
temperature.
▪ For example: A dataset containing features of the house such as lot size, number
of bedrooms, number of baths, neighbourhood, etc. and the price of the house.
▪ A Regression algorithm can be trained to learn the relationship between the
features and the price of the house.
▪ There are many machine learning algorithms that can be used for regression
tasks.
▪ Some of them are:
• Linear Regression
• Multiple Linear Regression
• Ridge Regression
• Lasso Regression
• Polynomial Regression
• Decision Tree Regressor
• K Nearest Neighbor Regressor
• Random Forest Regressor

Shri [Link] college of CS and BM Page | 3

Data Analytics using Python Unit – 3 Understanding Supervised Learning

❖ Difference between Classification and Regression

Regression Algorithms Classification Algorithms

The output variable must be either The output variable has to be a discrete
continuous nature or real value. value.

The regression algorithm’s task is The classification algorithm’s task

mapping input value (x) with continuous mapping the input value of x with the
output variable (y). discrete output variable of y.

They are used with continuous data. They are used with discrete data.

Classification tries to find the decision

It attempts to find the best fit line, which
boundary, which divides the dataset into
predicts the output more accurately.
different classes.

Classification algorithms solve

Regression algorithms solve regression
classification problems like identifying
problems such as house price predictions
spam e-mails, spotting cancer cells, and
and weather predictions.
speech recognition.

We can further divide Regression We can further divide Classification

algorithms into Linear and Non-linear algorithms into Binary Classifiers and
Regression. Multi-class Classifiers.

Shri [Link] college of CS and BM Page | 4

Data Analytics using Python Unit – 3 Understanding Supervised Learning
❖ Components (Process) of Supervised Machine Learning:

Figure 3.1 Components of Supervised Machine Learning

 Features
▪ A feature is an individual measurable property or characteristic of the
phenomenon you are observing.
▪ In simple terms, these are the inputs or variables used to make a prediction.
▪ They are also known as independent variables, predictors, or attributes.
▪ Features are selected and pre-processed to ensure they are in a format the
computer can understand (often converting text to numbers).

➢ Characteristics of Features
• Measurable: Features are quantifiable properties that can be measured
and recorded.
• Independent: Each feature should ideally be independent of the others,
providing unique information to the model.
• Varied Types: Features can be numerical (e.g., age, height), categorical
(e.g., gender, color), or even text-based (e.g., reviews, comments).
Shri [Link] college of CS and BM Page | 5
Data Analytics using Python Unit – 3 Understanding Supervised Learning
➢ Examples of Features
• In a dataset predicting house prices, features might include the number of
bedrooms, square footage, and location.
• For a spam email classifier, features could be the presence of certain
keywords, the length of the email, and the sender's address.
 Labels:
▪ A label, also known as the target variable or dependent variable, is the output
that the model is trained to predict.
▪ In supervised learning, labels are the known outcomes that the model learns to
associate with the input features during training.

➢ Characteristics of Labels
• Dependent: Labels depend on the input features and are the result of the
model's prediction.
• Categorical or Numerical: Labels can be categorical (e.g., spam or not
spam) or numerical (e.g., price of a house).

➢ Examples of Labels
• In a house price prediction model, the label would be the actual price of
the house.
• For a spam email classifier, the label would be whether the email is spam
or not.
 Dataset
▪ A dataset is a structured collection of related data, usually organized in rows and
columns that represents information about a specific category or domain.
▪ It forms the foundation for many operations, techniques and models used across
industries.

Figure 3.2 Components of Dataset

Shri [Link] college of CS and BM Page | 6

Data Analytics using Python Unit – 3 Understanding Supervised Learning
▪ For example, A student dataset may include rows for each student and columns
for attributes like name, age, grade and marks.

➢ Importance of Dataset:
Some reasons why datasets are important in analysis and machine learning
are:
• Analysis: Provide the raw material required for analysis and decision
making.
• Training Machine Learning Models: Enable the training and testing of
machine learning and AI models.
• Discover Patterns and Correlations: Help uncover patterns, correlations
and insights across domains.
• Innovation: Support research and development in industries like
healthcare, finance and education.
• Evaluation and Standards: Allow reproducibility and benchmarking in
academic and professional projects.

➢ Types of Datasets
There are various types of datasets available out there. Some of them are:
• Numerical Dataset: Contains numeric data points that can be analysed
using mathematical or statistical methods. For example, temperature
dataset.
• Categorical Dataset: Represents discrete categories or groups such as
color, gender, occupation or sports.
• Time Series Dataset: Records data over a period of time to track trends or
changes. For example, stock prices.
• Ordered Dataset: Contains ranked or ordinal data where the order matters
but not the exact difference between values. Example can be customer
reviews, survey ratings or movie rankings.
• Image Dataset: Consists of images used for classification, recognition or
analysis tasks. For example, medical imaging for disease detection.
• Web Dataset: Collected from APIs or web sources, usually stored in
structured formats like JSON for further analysis.
• File based Dataset: Stored in files such as CSV, Excel (.xlsx) or text files for
easy access and manipulation.

➢ Properties of Dataset
Here are the key properties that define a dataset:
• Center of Data: Refers to the "middle" value of a dataset, usually measured
using mean, median or mode. It helps identify where most of the values lie
and gives a sense of the average data point.
Shri [Link] college of CS and BM Page | 7
Data Analytics using Python Unit – 3 Understanding Supervised Learning
• Skewness of Data: This indicates how symmetrical the data distribution
is. A perfectly symmetrical distribution like a normal distribution has a
skewness of 0 while positive or negative skewness indicates a tilt in one
direction.
• Spread: This describes how much the data points vary from the
center. Common measures include standard deviation or variance, which
quantify how far individual points deviate from the average.
• Outliers: These are data points that fall significantly outside the overall
pattern. Identifying outliers can be important as they might influence
analysis results and require further investigation.
• Correlation: It shows how strongly variables are related. A positive
correlation means both increase together, a negative correlation means
they move in opposite directions and no correlation means no clear
relationship.
• Probability distribution: Understanding the distribution like
normal, uniform, binomial helps us predict how likely it is to find certain
values within the data and choose appropriate statistical methods for
analysis.
➢ Features of a Dataset
Some possible features of a dataset are:
• Numerical Features: These may include numerical values such as height,
weight and so on. These may be continuous over an interval or discrete
variables.
• Categorical Features: These include multiple classes or categories such as
gender, colour and so on.
• Size of the Data: It refers to the number of entries and features it contains
in the file containing the Dataset.
• Data Entries: These refer to the individual values of data present in the
Dataset.
• Target Variable: This is the main feature in a dataset that we want to
predict or explain using the other features.

 Training Data, Test Data, validation Data

▪ Training, validation and testing sets are three essential components in building
reliable machine learning models.
▪ The training set teaches the model patterns, the validation set helps fine-tune
hyperparameters and prevent overfitting and the testing set evaluates how well
the model performs on completely unseen data.
▪ Training: used to learn patterns
▪ Validation: used to tune and optimize the model
▪ Testing: used to measure final performance and generalization
Shri [Link] college of CS and BM Page | 8
Data Analytics using Python Unit – 3 Understanding Supervised Learning
▪ The Data Split (The 80/20 Rule):
▪ You never use 100% of your data for training. If you do, you will not know if the
model “learned” or just "memorized" the answers.
▪ Data is usually split into:
▪ Training Set (~70-80%): Used to build the model and adjust its internal weights.
▪ Validation Set (~10-15%): Used to fine-tune "hyperparameters" (like how fast the
model learns) and prevent overfitting.
▪ Test Set (~10-15%): Kept in a "vault" until the very end to see how the model
performs on completely unseen data.

➢ Training Data:
• The training set is the portion of the dataset used to fit the machine
learning model.
• During training, the algorithm learns patterns, relationships and
parameters (such as weights in neural networks or coefficients in
regression models) directly from this data.
• The model repeatedly adjusts itself to minimize the training error using
optimization techniques like gradient descent.
▪ The model has full access to the labels and features in this dataset.
▪ Learning happens exclusively on the training set.
▪ Performance on training data does not reflect real-world
performance.

➢ Validation Data:
• The validation set is a separate subset of data used to tune model
hyperparameters and make design decisions during training.
• Unlike the training set, it is not used to update model weights directly.
• Instead, it provides an unbiased estimate of model performance during
development.
▪ Used during model selection and optimization
▪ Helps detect overfitting early
▪ Often involved in techniques like early stopping

➢ Test Data:
• The testing set is a completely independent subset used to evaluate the
final model’s performance after all training and tuning are complete.
• It simulates how the model will perform on unseen, real-world data and
provides the most reliable estimate of generalization.
▪ Used only once (or very sparingly)
▪ No model decisions should be based on test results
▪ Represents real deployment conditions
Shri [Link] college of CS and BM Page | 9
Data Analytics using Python Unit – 3 Understanding Supervised Learning
❖ Overfitting, Underfitting
▪ Machine learning models should learn useful patterns from training data.
▪ When a model learns too little or too much, we get underfitting or overfitting.
▪ Underfitting means that the model is too simple and does not cover all real
patterns in the data.
▪ Overfitting means that the model learns not just the underlying pattern, but also
noise or random quirks in the training data. model memorizes training data.
▪ A good model finds the right spot, it is complex enough to capture real patterns,
but not so complex that it “memorizes” noise

Figure 3.3 Overfitting and Underfitting

 What is Underfitting?
▪ Underfitting happens when the model fails to learn important patterns.
▪ It performs poorly on both training and testing data.
▪ Underfitting happens due to:
• Model is too simple
• Very high regularization
• Features are weak or missing
• Not enough training
• High bias
▪ Example: Trying to predict house prices, which depend on size, location, age, etc.,
using only the number of rooms. The model is too simple (high bias) and will likely
make poor predictions for most houses.

 What is Overfitting?
▪ Overfitting happens when the model learns too much from the training data,
including noise and outliers.
▪ It performs very well on training data but poorly on test data.
▪ Overfitting happens due to:
• Model too complex
• Too many features
• Very little data
Shri [Link] college of CS and BM Page | 10
Data Analytics using Python Unit – 3 Understanding Supervised Learning
• No regularization
• High variance
▪ Example: Creating a highly complex polynomial regression model that perfectly
passes through every single data point in your house price training set. This model
might have near-zero error on the training data, but it will perform terribly on
new houses because it has learned the noise specific to the training set (high
variance).

❖ Python Code for Simple Linear Regression

Figure 3.4 Python Code for Simple Linear Regression Model

Shri [Link] college of CS and BM Page | 11

Data Analytics using Python Unit – 3 Understanding Supervised Learning

Figure 3.5 Scatter Plot for Regression Model

❖ Cost/Loss functions:
▪ The loss function quantifies how much a model‘s prediction deviates from the
ground truth for one particular object.
▪ So, when we calculate loss, we do it for a single object in the training or test sets.
▪ The term cost is often used interchangeably with loss. However, the cost function
measures the model’s error over a group of data points, while the loss function
measures the error for a single data instance.
▪ In machine learning, a cost function tells you how wrong your model’s predictions
are compared to the true values.
▪ A high cost means the model is making large errors, a low cost means it is doing
well.
▪ The lower the cost, the better the model is at predicting reality

➢ How it works (The "Distance" Concept)

• When you build your regression model for Age, Experience, and Salary, the
model draws a line.
• The actual data is where the dots are and the prediction is where the line
is.
• The distance between a dot and the line is the Error (also called a Residual).
• The Cost Function takes all those distances, combines them into a single
number, and gives you a "Penalty Score."

➢ Why do we need it?

• A model "learns" by trying to make this Cost Function as small as possible.
▪ The model starts with a random line.
▪ It calculates the Cost (total error).
▪ It adjusts the weights (beta values) for Age and Experience to try and
lower that cost.
▪ It repeats this until it finds the line with the lowest possible cost.

Shri [Link] college of CS and BM Page | 12

Data Analytics using Python Unit – 3 Understanding Supervised Learning
➢ Types of cost Function
• Regression cost functions measure "distance" (how far off the number is)
• Classification cost functions measure "divergence" (how wrong the
probability is).

❖ Regression Cost Functions

▪ Regression models deal with predicting a continuous value for example salary of
an employee, price of a car, loan prediction, etc.
▪ A cost function used in the regression problem is called “Regression Cost
Function”.
▪ They are calculated on the distance-based error.
▪ The most used Regression cost functions are below:
 Mean Squared Error (MSE)
▪ This is the "standard" for Linear Regression.
▪ It squares the difference between the actual and predicted values.
▪ Best for: Most standard regression tasks.
▪ The Mean Squared Error (MSE) is the most common cost function used for
regression problems.
▪ It measures the average squared difference between the actual values and the
predicted values.

Figure 3.6 Mean Squared Error (MSE)

▪ Formula:

▪ Where:

Shri [Link] college of CS and BM Page | 13

Data Analytics using Python Unit – 3 Understanding Supervised Learning
➢ Working of MSE
• Mathematical Operation: MSE squares the residuals (𝑦 − 𝑦̂)²
This serves two purposes:
▪ It turns all negative errors into positive numbers.
▪ It creates a "Penalty" system for the model.

• The Cost Function Shape (Parabola): When you plot the errors, they form
a smooth U-shaped Parabola.
▪ The Global Minimum: The bottom of this parabola represents the
point of lowest error (the "Best Fit" line).
▪ Smoothness: Unlike a "V" shape, the parabola is smooth at the
bottom, making it perfectly differentiable.
• Sensitivity to Outliers: Because of the squaring, MSE penalizes large errors
heavily.
▪ A mistake of 10 results in a penalty of 100.
▪ A mistake of 2 results in a penalty of only 4.

• The Pull Effect: The model is so "scared" of these large penalties that it will
actually shift its line toward outliers to minimize that squared error, even if
it makes the predictions for normal data points less accurate.

➢ Interpretation of Mean Squared Error

• The Interpreting MSE involves understanding the magnitude of the error
and its implications for the model's performance.
• A lower MSE indicates that the model's predictions are closer to the actual
values, signifying better accuracy.
• Conversely, a higher MSE suggests that the model's predictions deviate
further from the true value, indicating poorer performance.

➢ Significance of Mean Squared Error

• The Mean Squared Error is widely used in various fields, including statistics,
machine learning, and econometrics, due to its several important
properties:
▪ It provides the quantitative measure of the accuracy of the predictive
models.
▪ It penalizes large errors more heavily than small errors, making it
sensitive to the outliers.
▪ It is mathematically convenient and easy to interpret, making it a
preferred choice for evaluating model performance.

Shri [Link] college of CS and BM Page | 14

Data Analytics using Python Unit – 3 Understanding Supervised Learning
➢ Applications of Mean Squared Error
• The Mean Squared Error is extensively used in various applications,
including:
▪ Regression analysis: Assessing the goodness of fit of the regression
models.
▪ Model evaluation: Comparing the performance of the different
machine learning algorithms.
▪ Optimization: Minimizing MSE during the model training to improve
predictive accuracy.
▪ Predictive modeling: Evaluating the accuracy of the regression and
forecasting models.
▪ Image processing: Assessing the quality of the image reconstruction
and restoration algorithms.
▪ Financial modeling: Analyzing the performance of the investment
strategies and risk models.

 Mean Absolute Error (MAE)

▪ Instead of squaring the difference, it simply takes the absolute value (the positive
distance).
▪ Best for: Data with many outliers or "noisy" data.
▪ Key Feature: It is more "robust," meaning one or two weird data points won't pull
the entire regression line toward them as much as they would in MSE.
▪ Formula:
MAE = (1/n) * Σ|y – ŷ|

➢ Working of MAE (Mean Absolute Error)

• Mathematical Operation: MAE takes absolute value ∣ 𝑦 − 𝑦̂ ∣
▪ Purpose: By using absolute values instead of squares, it treats all
errors proportionally. It does not "blow up" the importance of
large errors.

• The Cost Function Shape (The V-Shape)

▪ Visual Shape: When plotted, MAE creates a sharp V-shaped graph.
▪ The Sharp Corner: Unlike the smooth MSE curve, the MAE graph has
a sharp point at the bottom (the vertex).
▪ Calculus Note: Because of this sharp point, it is non-differentiable at
the exact bottom. This can make it slightly trickier for some
optimization algorithms to "find" the perfect center compared to a
smooth parabola.

Shri [Link] college of CS and BM Page | 15

Data Analytics using Python Unit – 3 Understanding Supervised Learning
• Robustness to Outliers:
▪ The "Fair" Penalty: MAE is robust to outliers. It does not penalize
them heavily.
• An error of 2 becomes a penalty of 2.
• An error of 10 becomes a penalty of 10.
▪ The Result: Because the penalty does not explode (like 100 in MSE),
the model stays focused on the median of the data. It ignores "crazy"
outliers and keeps the line aligned with most of the "normal" data
points.

 Root Mean Square Error (RMSE)

▪ The Root Mean Squared Error (RMSE) is a variant of MSE that calculates the
square root of the average squared difference between actual and predicted
values.
▪ It is often preferred over MSE as it provides an interpretable measure of the error
in the same units as the original data.
▪ Formula:

➢ Working of RMSE:
• Mathematical Operation: Square Root of the Mean Squared Error (MSE)
▪ Purpose: It "undoes" the squaring process to return the error value
to the original units of your data (e.g., if you are predicting salary in
Dollars, RMSE gives you the error in Dollars, whereas MSE gives it in
"Squared Dollars").
▪ Always Positive: Because it involves squaring and then a square root,
RMSE will always be a positive number

• Interpretation & Use Case:

▪ Unit Consistency: It is expressed in the same units as the target
variable (e.g., kg, cm, dollars, or liters). This makes it the best metric
to explain to a client or a boss...
▪ Reporting: While MSE is often used behind the scenes to "train" the
model, RMSE is almost always used to describe the model's
performance.

Shri [Link] college of CS and BM Page | 16

Data Analytics using Python Unit – 3 Understanding Supervised Learning
• Relationship with Outliers:
▪ Inherited Sensitivity: Even though we take the square root at the
end, the errors were still squared first.
▪ The Result: Like MSE, RMSE is still sensitive to outliers. If your dataset
has a few very large errors, the RMSE will be significantly higher than
the MAE (Mean Absolute Error).
 Understanding R2
▪ The most important thing we do after making any model is evaluating the model.
▪ We have different evaluation matrices for evaluating the model.
▪ However, the choice of evaluation matrix to use for evaluating the model depends
upon the type of problem we are solving whether it's a regression, classification,
or any other type of problem.

➢ What is R-Squared ?
• R-squared (R^2), also known as the Coefficient of Determination, is a
statistical measure that represents the goodness of fit of a regression
model.
• The value of R-square lies between 0 to 1.
• Where we get R-square equals 1 when the model perfectly fits the data and
there is no difference between the predicted value and actual value.
• However, we get R-square equals 0 when the model does not predict any
variability in the model and it does not learn any relationship between the
dependent and independent variables.

➢ The "Dumb" Model vs. Your Model

• To understand 𝑅^2, imagine two different ways to predict something (like
Salary):
▪ The Average (Baseline): If you knew nothing about a person's
experience, your best guess would be the average salary of everyone
in your dataset. This would be a flat, horizontal line
▪ Regression Line: This is the line you built using y = mx + c. It uses
"Experience" to make a better guess.
• R^2 calculates how much closer your regression line is to the actual data
points compared to the "Average" line.
• Formula:

• Sum of Squared Residuals (SSR or SSE)

∑(𝑦 − 𝑦^)2 Error made by your model

Shri [Link] college of CS and BM Page | 17

Data Analytics using Python Unit – 3 Understanding Supervised Learning
• Total Sum of Squares (TSS)
The total variation in the data.
∑(𝑦 − 𝑦ˉ)2 Error made by the baseline model that always predicts the
mean
➢ Interpreting the Score
• 𝑅^2 is usually a value between 0 and 1 (though in rare, very bad cases, it
can be negative).
• R^2 = 0: Your model is no better than just guessing the average. It explains
0% of the data.
• R^2 = 1: Your model is perfect. Every single data point falls exactly on your
regression line.
• R^2 = 0.85: This means 85% of the variation in Salary is explained by the
input (Experience). The other 15% is "noise" or factors you haven't
measured yet.
• A high R^2 does not always mean the model is "good." If you have very few
data points, you might get a high R^2 by accident (overfitting).
• Always use both: When you build a model, you should always report both
,the actual error and the R^2 (to show the strength of the relationship).

❖ Classification:
▪ Classification is a supervised machine learning technique used to predict labels or
categories based on input data.
▪ The goal is to assign each data point to a predefined class, such as spam vs. non-
spam emails or diseased vs. healthy patients.
▪ A classification model might be trained on dataset of images labeled as either
dogs or cats and it can be used to predict the class of new and unseen images as
dogs or cats based on their features such as color, texture or shape.

 Types of Classification
▪ When we talk about classification in machine learning, we are talking about the
process of sorting data into categories based on specific features or
characteristics.
▪ Following are the types:
1. Binary Classification
• In binary classification, the goal is to sort the data into two distinct
categories.
• Think of it like a simple choice between two options.
• Imagine a system that sorts emails into either spam or not spam.

Shri [Link] college of CS and BM Page | 18

Data Analytics using Python Unit – 3 Understanding Supervised Learning
• It works by looking at different features of the email like certain keywords
or sender details and decides whether it’s spam or not. It only chooses
between these two options.
2. Multiclass Classification
• Instead of just two categories, the data needs to be sorted into more than
two categories.
• The model picks the one that best matches the input.
• Think of an image recognition system that sorts pictures of animals into
categories like cat, dog and bird.

Figure 3.7Types of Classification

3. Multi-Label Classification
• In multi-label classification single piece of data can belong to multiple
categories at once.
• Unlike multiclass classification where each data point belongs to only one
class, multi-label classification allows datapoints to belong to multiple
classes.
• A movie recommendation system could tag a movie as both action and
comedy. The system checks various features (like movie plot, actors or
genre tags) and assigns multiple labels to a single piece of data, rather than
just one.
 Classification Algorithms
▪ There are various types of classifiers algorithms. Some of them are :
1. Linear Classifiers:
• Linear classifier models create a linear decision boundary between classes.
• They are simple and computationally efficient.
Shri [Link] college of CS and BM Page | 19
Data Analytics using Python Unit – 3 Understanding Supervised Learning
• Some of the linear classification models are as follows:
▪ Logistic Regression
▪ Support Vector Machines having kernel = 'linear'
▪ Single-layer Perceptron
▪ Stochastic Gradient Descent (SGD) Classifier
2. Non-linear Classifiers:
• Non-linear models create a non-linear decision boundary between classes.
• They can capture more complex relationships between input features and
target variable.
• Some of the non-linear classification models are as follows:
▪ K-Nearest Neighbours
▪ Kernel SVM
▪ Naive Bayes
▪ Decision Tree Classification
▪ Ensemble learning classifier
▪ Random Forests

❖ Logistic Regression in Machine Learning

▪ Logistic Regression is a supervised machine learning algorithm used for
classification problems.
▪ Unlike linear regression which predicts continuous values it predicts the
probability that an input belongs to a specific class.
▪ It is used for binary classification where the output can be one of two possible
categories such as Yes/No, True/False or 0/1.
▪ It uses sigmoid function to convert inputs into a probability value b/w 0 and 1.

Figure 3.8 Logistic Regression

Shri [Link] college of CS and BM Page | 20

Data Analytics using Python Unit – 3 Understanding Supervised Learning
 Types of Logistic Regression
▪ Logistic regression can be classified into three main types based on the nature of
the dependent variable.

➢ Binomial Logistic Regression:

• This type is used when the dependent variable has only two possible
categories. Examples include Yes/No, Pass/Fail or 0/1. It is the most
common form of logistic regression and is used for binary classification
problems.

➢ Multinomial Logistic Regression:

• This is used when the dependent variable has three or more possible
categories that are not ordered. For example, classifying animals into
categories like "cat," "dog" or "sheep." It extends the binary logistic
regression to handle multiple classes.

➢ Ordinal Logistic Regression:

• This type applies when the dependent variable has three or more
categories with a natural order or ranking. Examples include ratings like
"low," "medium" and "high." It takes the order of the categories into
account when modeling.

Figure 3.9 Types of Logistic Regression

 Assumptions of Logistic Regression

▪ Understanding the assumptions behind logistic regression is important to ensure
the model is applied correctly, main assumptions are:
▪ Independent observations: Each data point is assumed to be independent of the
others means there should be no correlation or dependence between the input
samples.

Shri [Link] college of CS and BM Page | 21

Data Analytics using Python Unit – 3 Understanding Supervised Learning
▪ Binary dependent variables: It takes the assumption that the dependent variable
must be binary, means it can take only two values.
▪ Linearity relationship between independent variables and log odds: The model
assumes a linear relationship between the independent variables and the log
odds of the dependent variable which means the predictors affect the log odds in
a linear way.
▪ No outliers: The dataset should not contain extreme outliers as they can distort
the estimation of the logistic regression coefficients.
▪ Large sample size: It requires a sufficiently large sample size to produce reliable
and stable results.
 Understanding Sigmoid Function
▪ The sigmoid function is an important part of logistic regression which is used to
convert the raw output of the model into a probability value between 0 and 1.
▪ This function takes any real number and maps it into the range 0 to 1 forming an
"S" shaped curve called the sigmoid curve or logistic curve. Because probabilities
must lie between 0 and 1, the sigmoid function is perfect for this purpose.
▪ In logistic regression, we use a threshold value usually 0.5 to decide the class
label.
• If the sigmoid output is same or above the threshold, the input is classified
as Class 1.
• If it is below the threshold, the input is classified as Class 0.

Shri [Link] college of CS and BM Page | 22

Data Analytics using Python Unit – 3 Understanding Supervised Learning
 Python Code for Classification(Binary Logistic Regression – Single Input)

Figure 3.10 python code for classification

Figure 3.11 Logistic Regression Graph

Shri [Link] college of CS and BM Page | 23

Data Analytics using Python Unit – 3 Understanding Supervised Learning
 Confusion Matrix:
▪ Confusion matrix is a simple table used to measure how well a classification model
is performing.
▪ It compares the predictions made by the model with the actual results and shows
where the model was right or wrong.
▪ This helps you understand where the model is making mistakes so you can
improve it.
▪ It breaks down the predictions into four categories:
• True Positive (TP): The model correctly predicted a positive outcome i.e the
actual outcome was positive.
• True Negative (TN): The model correctly predicted a negative outcome i.e
the actual outcome was negative.
• False Positive (FP): The model incorrectly predicted a positive outcome i.e
the actual outcome was negative. It is also known as a Type I error.
• False Negative (FN): The model incorrectly predicted a negative outcome
i.e the actual outcome was positive. It is also known as a Type II error

Figure 3.12 Confusion Matrix

 Metrics based on Confusion Matrix Data

1. Accuracy
• Accuracy shows how many predictions the model got right out of all the
predictions.
• It gives idea of overall performance but it can be misleading when one class
is more dominant over the other.
• For example a model that predicts the majority class correctly most of the
time might have high accuracy but still fail to capture important details
about other classes. It can be calculated using the below formula:

2. Precision
• Precision focus on the quality of the model’s positive predictions.
• It tells us how many of the "positive" predictions were actually correct.
Shri [Link] college of CS and BM Page | 24
Data Analytics using Python Unit – 3 Understanding Supervised Learning
• It is important in situations where false positives need to be minimized such
as detecting spam emails or fraud. The formula of precision is:

3. Recall
• Recall measures how good the model is at predicting positives.
• It shows the proportion of true positives detected out of all the actual
positive instances.
• High recall is essential when missing positive cases has significant
consequences like in medical tests.

4. F1-Score
• F1-score combines precision and recall into a single metric to balance their
trade-off.
• It provides a better sense of a model’s overall performance particularly for
imbalanced datasets.
• It is helpful when both false positives and false negatives are important
though it assumes precision and recall are equally important but in some
situations, one might matter more than the other.

5. Type 1 and Type 2 error

• Type 1 error:
▪ It occurs when the model incorrectly predicts a positive instance but
the actual instance is negative. This is also known as a false positive.
▪ Type 1 Errors affect the precision of a model which measures the
accuracy of positive predictions.
▪ A Type I Error is often called a "False Alarm." It occurs when your
model predicts that something is happening (Positive), but in reality,
nothing is happening (Negative).

• Type 2 error:
▪ This occurs when the model fails to predict a positive instance even
though it is actually positive. This is also known as a false negative.
▪ Type 2 Errors impact the recall of a model which measures how well
the model identifies all actual positive cases.
Shri [Link] college of CS and BM Page | 25
Data Analytics using Python Unit – 3 Understanding Supervised Learning
▪ A Type II Error is a "Miss." It happens when the model predicts that
nothing is happening (Negative), but in reality, something important
is happening (Positive).

 Example of above Metrics:

▪ Confusion matrix for disease prediction:
Predicted: Healthy (0) Predicted: Disease (1)

Actual: Healthy (0) 3 (TN) 1 (FP)

Actual: Disease (1) 2 (FN) 4 (TP)

1. Precision (Exactness)
• Question: "Of all the people the model predicted as having the disease,
how many actually had it?"
• Logic: It looks at the Predicted Disease column.

• Calculation: 4/(4+1)
• Meaning: When the model says a patient is sick, it is right 80% of the time.
High precision means fewer "False Alarms" (Type I Errors).

2. Recall / Sensitivity (Completeness)

• Question: "Of all the people who actually had the disease, how many did
the model correctly find?"
• Logic: It looks at the Actual Disease row.

• Calculation: 4/(4 + 2)
• Meaning: The model caught 67% of the total sick people. Low recall is
dangerous in medicine because it means you missed 33% of the sick
patients (Type II Errors).

Shri [Link] college of CS and BM Page | 26

Data Analytics using Python Unit – 3 Understanding Supervised Learning
3. F1-Score (The Balance)
• Question: "How is the overall balance between Precision and Recall?"
• Logic: It is the harmonic mean. We use it instead of a simple average
because it penalizes extreme values (if one is very low, the F1-Score drops
significantly).

• Meaning: An F1-Score of 0.73 shows a decent balance, but the low recall is
pulling it down.

4. Type I Error (False Positive)

• Matrix Value: 1
• Meaning: The model gave a "False Alarm." It predicted the disease for a
patient who is actually healthy.

5. Type II Error (False Negative)

• Matrix Value: 2
• Meaning: The model "Missed" the disease. It predicted the patient was
healthy when they were actually sick.

6. Accuracy (Overall Correctness)

• Question: "Out of all 10 cases, how many did the model get right (both sick
and healthy)?"

• Calculation: (4+3)/10
• Meaning: The model is correct 70% of the time across the whole dataset.

Shri [Link] college of CS and BM Page | 27