UNIT-II

SUPERVISED LEARNING

Regression: Linear Regression-Logistic Regression- Polynomial Regression.

Classification: MINIST- Multiclass Classification- Multilabel Classification-
Multioutput Classification
Supervised Machine Learning
Supervised machine learning is a fundamental approach for machine learning and artificial
intelligence. It involves training a model using labeled data, where each input comes with a
corresponding correct output. The process is like a teacher guiding a student—hence the term
"supervised" learning. In this article, we'll explore the key components of supervised learning,
the different types of supervised machine learning algorithms used, and some practical
examples of how it works.

Supervised Machine Learning

What is Supervised Machine Learning?
As we explained before, supervised learning is a type of machine learning where a model is
trained on labeled data—meaning each input is paired with the correct output. the model learns
by comparing its predictions with the actual answers provided in the training data. Over time,
it adjusts itself to minimize errors and improve accuracy. The goal of supervised learning is to
make accurate predictions when given new, unseen data. For example, if a model is trained to
recognize handwritten digits, it will use what it learned to correctly identify new numbers it
hasn't seen before.
Supervised learning can be applied in various forms, including supervised learning
classification and supervised learning regression, making it a crucial technique in the field of
artificial intelligence and supervised data mining.
A fundamental concept in supervised machine learning is learning a class from examples. This
involves providing the model with examples where the correct label is known, such as learning
to classify images of cats and dogs by being shown labeled examples of both. The model then
learns the distinguishing features of each class and applies this knowledge to classify new
images.
How Supervised Machine Learning Works?
Where supervised learning algorithm consists of input features and corresponding output
labels. The process works through:
 Training Data: The model is provided with a training dataset that includes input data
(features) and corresponding output data (labels or target variables).
 Learning Process: The algorithm processes the training data, learning the relationships
between the input features and the output labels. This is achieved by adjusting the model's
parameters to minimize the difference between its predictions and the actual labels.
After training, the model is evaluated using a test dataset to measure its accuracy and
performance. Then the model's performance is optimized by adjusting parameters and using
techniques like cross-validation to balance bias and variance. This ensures the model
generalizes well to new, unseen data.
In summary, supervised machine learning involves training a model on labeled data to learn
patterns and relationships, which it then uses to make accurate predictions on new data.
Let's learn how a supervised machine learning model is trained on a dataset to learn a mapping
function between input and output, and then with learned function is used to make predictions
on new data:

In the image above,

 Training phase involves feeding the algorithm labeled data, where each data point is paired
with its correct output. The algorithm learns to identify patterns and relationships between
the input and output data.
 Testing phase involves feeding the algorithm new, unseen data and evaluating its ability to
predict the correct output based on the learned patterns.
Types of Supervised Learning in Machine Learning
Now, Supervised learning can be applied to two main types of problems:
 Classification: Where the output is a categorical variable (e.g., spam vs. non-spam emails,
yes vs. no).
 Regression: Where the output is a continuous variable (e.g., predicting house prices, stock
prices).
While training the model, data is usually split in the ratio of 80:20 i.e. 80% as training data and
the rest as testing data. In training data, we feed input as well as output for 80% of data. The
model learns from training data only. We use different supervised learning
algorithms (which we will discuss in detail in the next section) to build our model. Let's first
understand the classification and regression data through the table below:

Both the above figures have labelled data set as follows:

 Figure A: It is a dataset of a shopping store that is useful in predicting whether a customer
will purchase a particular product under consideration or not based on his/ her gender, age,
and salary.
Input: Gender, Age, Salary
Output: Purchased i.e. 0 or 1; 1 means yes the customer will purchase and 0 means that
the customer won't purchase it.
 Figure B: It is a Meteorological dataset that serves the purpose of predicting wind speed
based on different parameters.
Input: Dew Point, Temperature, Pressure, Relative Humidity, Wind Direction
Output: Wind Speed
Practical Examples of Supervised learning
Few practical examples of supervised machine learning across various industries:
 Fraud Detection in Banking: Utilizes supervised learning algorithms on historical
transaction data, training models with labeled datasets of legitimate and fraudulent
transactions to accurately predict fraud patterns.
 Parkinson Disease Prediction: Parkinson’s disease is a progressive disorder that affects
the nervous system and the parts of the body controlled by the nerves.
 Customer Churn Prediction: Uses supervised learning techniques to analyze historical
customer data, identifying features associated with churn rates to predict customer retention
effectively.
 Cancer cell classification: Implements supervised learning for cancer cells based on their
features, and identifying them if they are ‘malignant’ or ‘benign.
 Stock Price Prediction: Applies supervised learning to predict a signal that indicates
whether buying a particular stock will be helpful or not.
Supervised Machine Learning Algorithms
Supervised learning can be further divided into several different types, each with its own
unique characteristics and applications. Here are some of the most common types of supervised
learning algorithms:
 Linear Regression: Linear regression is a type of supervised learning regression algorithm
that is used to predict a continuous output value. It is one of the simplest and most widely
used algorithms in supervised learning.
 Logistic Regression : Logistic regression is a type of supervised learning classification
algorithm that is used to predict a binary output variable.
 Decision Trees : Decision tree is a tree-like structure that is used to model decisions and
their possible consequences. Each internal node in the tree represents a decision, while each
leaf node represents a possible outcome.
 Random Forests : Random forests again are made up of multiple decision trees that work
together to make predictions. Each tree in the forest is trained on a different subset of the
input features and data. The final prediction is made by aggregating the predictions of all
the trees in the forest.
 Support Vector Machine(SVM) : The SVM algorithm creates a hyperplane to segregate
n-dimensional space into classes and identify the correct category of new data points. The
extreme cases that help create the hyperplane are called support vectors, hence the name
Support Vector Machine.
 K-Nearest Neighbors (KNN) : KNN works by finding k training examples closest to a
given input and then predicts the class or value based on the majority class or average value
of these neighbors. The performance of KNN can be influenced by the choice of k and the
distance metric used to measure proximity.
 Gradient Boosting : Gradient Boosting combines weak learners, like decision trees, to
create a strong model. It iteratively builds new models that correct errors made by previous
ones.
 Naive Bayes Algorithm: The Naive Bayes algorithm is a supervised machine learning
algorithm based on applying Bayes' Theorem with the “naive” assumption that features are
independent of each other given the class label.

Regression in machine learning

Regression in machine learning refers to a supervised learning technique where the goal is
to predict a continuous numerical value based on one or more independent features. It finds
relationships between variables so that predictions can be made. we have two types of
variables present in regression:
 Dependent Variable (Target): The variable we are trying to predict e.g house price.
 Independent Variables (Features): The input variables that influence the prediction e.g
locality, number of rooms.
Regression analysis problem works with if output variable is a real or continuous value such
as “salary” or “weight”. Many different regression models can be used but the simplest model
in them is linear regression.
Types of Regression
Regression can be classified into different types based on the number of predictor variables
and the nature of the relationship between variables:
1. Simple Linear Regression
Linear regression is one of the simplest and most widely used statistical models. This
assumes that there is a linear relationship between the independent and dependent variables.
This means that the change in the dependent variable is proportional to the change in the
independent variables. For example predicting the price of a house based on its size.
2. Multiple Linear Regression
Multiple linear regression extends simple linear regression by using multiple independent
variables to predict target variable. For example predicting the price of a house based on
multiple features such as size, location, number of rooms, etc.
3. Polynomial Regression
Polynomial regression is used to model with non-linear relationships between the dependent
variable and the independent variables. It adds polynomial terms to the linear regression
model to capture more complex relationships. For example when we want to predict a non-
linear trend like population growth over time we use polynomial regression.
4. Ridge & Lasso Regression
Ridge & lasso regression are regularized versions of linear regression that help avoid
overfitting by penalizing large coefficients. When there’s a risk of overfitting due to too many
features we use these type of regression algorithms.
5. Support Vector Regression (SVR)
SVR is a type of regression algorithm that is based on the Support Vector Machine
(SVM) algorithm. SVM is a type of algorithm that is used for classification tasks but it can
also be used for regression tasks. SVR works by finding a hyperplane that minimizes the sum
of the squared residuals between the predicted and actual values.
6. Decision Tree Regression
Decision tree Uses a tree-like structure to make decisions where each branch of tree
represents a decision and leaves represent outcomes. For example predicting customer
behavior based on features like age, income, etc there we use decison tree regression.
7. Random Forest Regression
Random Forest is a ensemble method that builds multiple decision trees and each tree is
trained on a different subset of the training data. The final prediction is made by averaging
the predictions of all of the trees. For example customer churn or sales data using this.
Regression Evaluation Metrics
Evaluation in machine learning measures the performance of a model. Here are some popular
evaluation metrics for regression:
 Mean Absolute Error (MAE): The average absolute difference between the predicted
and actual values of the target variable.
 Mean Squared Error (MSE): The average squared difference between the predicted and
actual values of the target variable.
 Root Mean Squared Error (RMSE): Square root of the mean squared error.
 Huber Loss: A hybrid loss function that transitions from MAE to MSE for larger errors,
providing balance between robustness and MSE’s sensitivity to outliers.
 R2 – Score: Higher values indicate better fit ranging from 0 to 1.
Regression Model Machine Learning
Let's take an example of linear regression. We have a Housing data set and we want to
predict the price of the house. Following is the python code for it.
import matplotlib
[Link]('TkAgg') # General backend for plots

import [Link] as plt

import numpy as np
from sklearn import datasets, linear_model
import pandas as pd

# Load dataset
df = pd.read_csv("[Link]")

# Extract features and target variable

Y = df['price']
X = df['lotsize']

# Reshape for compatibility with scikit-learn

X = X.to_numpy().reshape(len(X), 1)
Y = Y.to_numpy().reshape(len(Y), 1)

# Split data into training and testing sets

X_train = X[:-250]
X_test = X[-250:]
Y_train = Y[:-250]
Y_test = Y[-250:]

# Plot the test data

[Link](X_test, Y_test, color='black')
[Link]('Test Data')
[Link]('Size')
[Link]('Price')
[Link](())
[Link](())

# Train linear regression model

regr = linear_model.LinearRegression()
[Link](X_train, Y_train)

# Plot predictions
[Link](X_test, [Link](X_test), color='red', linewidth=3)
[Link]()
Output:
 Here in this graph we plot the test data. The red line indicates the best fit line for predicting
the price.
To make an individual prediction using the linear regression model:
print("Predicted price for a lot size of 5000: " + str(round([Link]([[5000]])[0][0])))
Applications of Regression
 Predicting prices: Used to predict the price of a house based on its size, location and
other features.
 Forecasting trends: Model to forecast the sales of a product based on historical sales
data.
 Identifying risk factors: Used to identify risk factors for heart patient based on patient
medical data.
 Making decisions: It could be used to recommend which stock to buy based on market
data.
Advantages of Regression
 Easy to understand and interpret.
 Robust to outliers.
 Can handle both linear relationships easily.
Disadvantages of Regression
 Assumes linearity.
 Sensitive to situation where two or more independent variables are highly correlated with
each other i.e multicollinearity.
 May not be suitable for highly complex relationships.
Conclusion
Regression in machine learning is a fundamental technique for predicting continuous
outcomes based on input features. It is used in many real-world applications like price
prediction, trend analysis and risk assessment. With its simplicity and effectiveness
regression is used to understand relationships in data.
Multiclass Classification vs Multi-label Classification
Last Updated : 06 Jan, 2024


Multiclass classification is a machine learning task where the goal is to assign instances to
one of multiple predefined classes or categories, where each instance belongs to exactly one
class. Whereas multilabel classification is a machine learning task where each instance can
be associated with multiple labels simultaneously, allowing for the assignment of multiple
binary labels to the instance. In this article we are going to understand the multi-class
classification and multi-label classification, how they are different, how they are evaluated,
how to choose the best method for your problem, and much more.

What is Multiclass Classification?

Multiclass classification is a machine learning challenge focused on categorizing data into
more than two classes. While binary classification involves distinguishing between only two
classes, multiclass classification expands this scope to involve distinguishing between multiple
classes. In essence, the goal is to train a model that can effectively sort instances into various
predefined categories, providing a nuanced solution for scenarios where items can belong to
more than two exclusive groups. This approach is commonly employed in tasks such as
handwriting recognition, email categorization, and image classification involving more than
two distinct categories.
Multiclass classification is a type of machine learning task where the goal is to categorize
instances into one of several predefined classes. Unlike binary classification, where there are
only two possible outcomes, multiclass classification involves distinguishing between multiple
classes or categories. The fundamental idea is to teach a model to assign the most appropriate
class label to each instance based on its features.
Multiclass classification finds application in a wide range of real-world scenarios. Consider
email categorization, where emails need to be sorted into categories like "spam," "ham" (non-
spam), or "important." Another classic example is handwritten digit recognition, where the task
is to identify which digit (0 through 9) is written in a given image. Other applications include
speech recognition, sentiment analysis, and image classification into multiple categories.
Model Training Techniques:
Training a multiclass classification model involves employing specific techniques to ensure
accurate class assignment. One common approach is to use softmax activation in the output
layer of the neural network. Softmax converts the raw model outputs into probabilities,
assigning higher probabilities to the correct classes. Additionally, categorical cross-entropy
loss is often used as the objective function during training. This loss function measures the
dissimilarity between the predicted probabilities and the actual class labels, guiding the model
to minimize errors and improve accuracy.
Evaluation Metrics:
To assess the performance of a multiclass classification model, various evaluation metrics like
accuracy, precision, recall (sensitivity) and F1 score.
Understanding these concepts is crucial for practitioners working on multiclass classification
problems, as they form the foundation for designing effective models and assessing their
accuracy in real-world applications.
What is Multi-label Classification?
Multi-label classification is a machine learning paradigm where instances can be associated
with multiple labels simultaneously. Unlike traditional classification tasks, where an instance
is assigned a single exclusive label, multi-label classification recognizes the possibility for
instances to exhibit characteristics that span across various categories. The goal is to develop
models capable of accurately predicting and assigning a set of relevant labels to each instance,
reflecting the complex relationships and diversity inherent in real-world datasets. This
approach acknowledges the overlapping nature of labels, providing a more realistic
representation of the multifaceted attributes present in the data.
Multi-label classification is a machine learning task where instances can be associated with
multiple labels simultaneously. This differs from multiclass classification, where each instance
is assigned to one and only one class. In multi-label scenarios, an instance may exhibit
characteristics that correspond to several different categories, making the task more intricate
and reflecting the complexity often found in real-world data.
Multi-label classification is highly applicable in diverse scenarios where instances can possess
multiple attributes or labels. Examples include:
 Document Tagging: Assigning multiple tags or topics to a document, such as labeling an
article as both "technology" and "business."
 Image Classification with Multiple Labels: Identifying and labeling multiple objects or
features within an image, like recognizing both "cat" and "outdoor" in a photograph.
Model Training Techniques:
Training models for multi-label classification involves specific techniques to accommodate the
simultaneous assignment of multiple labels to instances:
 Sigmoid Activation: In the output layer of the neural network, sigmoid activation is often
used. Unlike softmax in multiclass scenarios, sigmoid independently activates each output
node, producing a value between 0 and 1, representing the likelihood of the corresponding
label being present.
 Binary Cross-Entropy Loss: This loss function is employed during training to measure
the dissimilarity between the predicted probabilities and the actual presence or absence of
each label. It guides the model to minimize errors in its multi-label predictions.
Evaluation Metrics:
Assessing the performance of a multi-label classification model requires specific metrics
tailored to handle the complexity of multiple labels per instance:
 Hamming Loss: This metric calculates the fraction of labels that are incorrectly predicted.
It provides a comprehensive measure of overall model performance in terms of label
accuracy.
 Precision at k: Precision at k evaluates the precision of the top-k predicted labels,
recognizing that not all labels need to be considered. It accounts for scenarios where only
the most relevant labels are of interest.
 Recall at k: Similar to precision at k, recall at k assesses the recall of the top-k predicted
labels. It focuses on capturing the relevant labels among the top predictions.
Understanding these nuances of multi-label classification is essential for practitioners working
on tasks where instances can belong to multiple categories simultaneously, ensuring effective
model design and evaluation in complex real-world scenarios.
Differences between Multi class and Multi label Classification
Features Multi class classification. Multi label classification

The output is a single class The output is a set of binary

Output Structure: label assigned to each values indicating the
Features Multi class classification. Multi label classification

instance, indicating the most presence or absence of each

probable or correct class. label for each instance.
Instances can be associated
with multiple labels
simultaneously.

The model outputs a binary

the model assigns a single
vector for each instance,
class label to each instance
where each element
based on the class with the
corresponds to a label,
highest probability or
indicating whether it is
confidence.
Model Output: present or not.

Techniques like softmax

Techniques like sigmoid
activation and categorical
activation and binary cross-
cross-entropy loss are
entropy loss are employed,
commonly used for training
treating each label
models to handle multiple
independently.
Training Techniques: classes.

Each instance is assigned to Instances can be associated

one and only one class, with multiple labels,
making the classification allowing for overlapping or
Class Assignment: mutually exclusive. shared characteristics.

Metrics like Hamming loss,

Metrics such as accuracy,
precision at k, and recall at k
precision, recall, and F1
are more appropriate, as they
score are commonly used to
account for the presence of
assess the overall
multiple labels for each
performance of the model.
Evaluation Metrics: instance.

Can be more complex due to

Generally considered
the need to capture
simpler as it involves
dependencies and
assigning instances to
correlations between
exclusive classes.
Model Complexity: multiple labels.

Suited for more complex

Typically used for simpler
scenarios where instances
problems where instances
can exhibit characteristics of
belong to mutually
multiple labels
exclusive categories.
Problem Complexity: simultaneously.
Choosing Between Multi-Class and Multi-Label Classification
When embarking on a classification task, one of the foundational decisions is whether to opt
for multi-class or multi-label classification, and this choice significantly influences the model's
performance and relevance to real-world scenarios.
 Assess whether the instances in your dataset belong to mutually exclusive classes (Multi-
Class) or if they can have multiple labels simultaneously (Multi-Label). Understanding the
nature of labels is fundamental in choosing the appropriate classification approach.
 Examine the relationships between labels. If the labels are independent or weakly
correlated, multi-class classification may be suitable. For strong correlations or overlapping
characteristics, multi-label classification is more appropriate.
 Gauge the complexity of your classification problem. Multi-class classification is generally
simpler as it deals with exclusive categorization. If the problem is inherently complex and
instances can have diverse characteristics, opt for multi-label classification.
 Consider domain-specific requirements and constraints. Some domains naturally lend
themselves to one approach over the other based on the inherent characteristics of the data
and the specific objectives of the task.
In conclusion, the choice between multi-class and multi-label classification should be made
considering the intricacies of the problem, the nature of the data, and the specific requirements
of the application. Each approach has its merits, and selecting the most suitable classification
method is pivotal for achieving optimal model performance in diverse real-world scenarios.