Thynk Unlimited
SUPERVISED
LEARNING:
CLASSIFICATION
Thynk Unlimited
OBJECTIVE OF THE
CHAPTER
1. To provide an
understanding of
popular classifiers
and their work
principls.
Thynk Unlimited
WHAT IS SUPERVISED
CLASSIFICATION?
Supervised classification is a machine learning
technique
Uses labeled datasets to train models.
Each data instance consists of:
Input features
A known class label
The model learns to assign correct labels to new
data
Thynk Unlimited
DEFINITION OF
CLASSIFICATION
A classification problem focuses on
predicting a category or class label for an
input based on the training data.
Example of categorial input:
Tumor type: malignant or benign
Color: red or blue
Thynk Unlimited
PRINCIPLES OF SUPERVISED
CLASSIFICATION
Supervised classification is a type of
machine learning where a model is
trained using labeled data. This means
that each training data point already
has a known class or category, and
the goal of the model is to learn
patterns that allow it toPerformance
correctly
Tracking Strategic Alignment
classify new, unseen data.
Thynk Unlimited
KEY
CHARACTERISTICS
•Uses •Predicts •Requires •Widely
labeled categorical human- used in
training (discrete) defined prediction
and
data outcomes labels
decision-
making
systems
Thynk Unlimited
The core principle of supervised
LABELED
classification is the use of labeled TRAINING
datasets, where each input is
paired with a correct output (class DATA
label).
Example:
Email text → Spam or Not
Spam
Student record → Passed or
Failed
Thynk Unlimited
Supervised classification relies on
identifying relationships between FEATURE
features (input variables) and
class labels (output). –LABEL
Features may include numerical or
categorical attributes that help RELATIONSHIP
distinguish one class from another.
Example:
Features: age, income,
spending habits
Label: High-risk or Low-risk
customer
Thynk Unlimited
Learning a Decision Boundary
The classifier learns a decision boundary that separates different
classes in the data space.
Simple models use straight lines
Complex models use curves or multi-dimensional boundaries
This boundary helps the model decide which class a new data
point belongs to.
Generalization
A key principle is that the model should generalize well, meaning
it performs accurately on new, unseen data, not just the training
data.
Overfitting (memorizing data) is avoided to ensure reliable
predictions.
Thynk Unlimited
Evaluation and Feedback
The model’s performance is evaluated using metrics such as:
Accuracy
Precision
Recall
Confusion matrix
Human-Guided Learning
Since labels are provided by humans or predefined rules,
supervised classification depends heavily on data quality,
including:
Correct labeling
Balanced classes
Clean and relevant features
Thynk Unlimited
COMMON
CLASSIFICATION
METHODS
Classification methods are algorithms used in supervised
learning to assign data into predefined categories or
classes. Each method follows a different approach in
learning patterns from labeled data.
Decision Trees Naive Bayes (KNN) (SVM)
Decision Trees
Definition: A non-parametric supervised learning method that
predicts the value of a target variable by learning simple decision
rules inferred from the data features. It partitions the feature space
into a set of rectangles and fits a simple model (like a majority vote)
in each one.
Supporting Theory & Proof: The core of a decision tree is the
Splitting Criterion, typically Gini Impurity or Entropy.
* Entropy (H): Measures the disorder in a set S.
* Proof of Concept: The algorithm uses Information Gain, which is
the reduction in entropy after a split. By maximizing Information Gain
at each node, the tree mathematically proves it is creating the most
homogeneous (pure) child nodes possible. Strategic Alignment
Example: Decision Tree: The "Logic Gate" Imagine a simple rulebook.
Naive Bayes
Definition: A probabilistic classifier based on applying Bayes'
Theorem with the "naive" assumption of conditional independence
between every pair of features given the value of the class variable.
Supporting Theory & Proof: It is rooted in the Maximum A Posteriori
(MAP) decision rule.
* Bayes' Theorem:
* The "Naive" Proof: By assuming P(x_i | y, x_1, \dots, x_{i-1}) = P(x_i |
y), we simplify the complex joint probability into a product of
individual probabilities:
Example: Naive Bayes: The "Frequency Counter" Imagine you have a basket of 100
Strategic fruits.
Alignment
80% of the "Bumpy" fruits you've seen are Oranges, and 90% of "Smooth" fruits are
Apples.
K-Nearest Neighbors (KNN)
Definition: A type of instance-based learning where the function is
only approximated locally and all computation is deferred until
classification. An object is classified by a plurality vote of its
neighbors.
Supporting Theory & Proof: KNN is supported by the Cover-Hart
Theorem, which provides an upper bound on the error rate.
* The Proof: As the number of training samples n \to \infty, the error
rate of the 1-Nearest Neighbor classifier is guaranteed to be no
worse than twice the Bayes Error Rate (the minimum possible error
rate for any classifier).
Example: K-Nearest Neighbors (KNN): The "Sorting Map" Imagine a table whereAlignment
Strategic you’ve
placed all your previously identified fruits. Apples are clustered on the left; Oranges are
clustered on the right.
Support Vector Machines (SVM)
Definition: A classifier that finds the optimal hyperplane which
maximizes the margin between two classes in a high-dimensional
space.
Supporting Theory & Proof: SVM is based on Structural Risk
Minimization (SRM) rather than Empirical Risk Minimization.
* The Math: It solves a constrained optimization problem:
* The Proof: By minimizing \|w\|, we are maximizing the margin
(\frac{2}{\|w\|}). According to Vapnik-Chervonenkis (VC) Theory,
maximizing this margin minimizes the "VC dimension," which
mathematically guarantees a lower bound on the generalization
error (reducing overfitting).
Strategic Alignment
Example: Support Vector Machines (SVM): The "Clear Boundary" Imagine the same
table with Apples and Oranges. They are a bit mixed in the middle.
Thynk Unlimited
REAL-WORLD
APPLICATION
Spam Detection Fraud Detection Image
Classification
Thynk Unlimited
REAL-WORLD APPLICATION: SPAM DETECTION
Spam detection refers to the process of identifying and
categorizing electronic messages—primarily emails—as either
spam (unwanted or malicious) or ham (legitimate).
Email Classification Common Algorithms Industry Standard
Major email services like
Incoming emails are Naive Bayes Classifier
Gmail, Outlook, and Yahoo
analyzed based on
Mail heavily rely on
content, sender, and Support Vector
sophisticated classification
structure to determine if Machines (SVM)
systems to protect users
they are legitimate or from phishing attempts,
spam. malware, and unsolicited
advertisements.
Thynk Unlimited
REAL-WORLD APPLICATION: FRAUD DETECTION
refers to the use of machine learning and data analysis techniques
to categorize transactions or activities as either fraudulent or non-
fraudulent based on historical patterns and anomalies.
Key Approaches in Classification-Based Fraud Detection:
The model is trained on a labeled dataset where each transaction is
marked as "fraud" or "non-fraud.
Common algorithms include:
Logistic Regression Decision Trees & Support Vector Machines
Random Forests (SVM) and Neural Networks
REAL-WORLD APPLICATION: IMAGE CLASSIFICATION
Image classification is a cornerstone of computer vision, enabling
machines to "see" and interpret the visual world by assigning
predefined categories to images.
Single-label classification: Multi-label classification: Binary classification:
REAL-WORLD APPLICATION: IMAGE CLASSIFICATION
Image classification is a cornerstone of computer vision, enabling
machines to "see" and interpret the visual world by assigning
predefined categories to images.
Face Recognition Medical Image Diagnosis Everyday Integration
Platforms like Google
Used in security AI assists doctors in
Photos and Facebook
systems, smartphone detecting
leverage image
unlocking, and social abnormalities in X-rays,
classification to
media tagging, these MRIs, and CT scans,
automatically organize
models identify classifying images as
photos, tag friends,
individuals by indicative of diseases
and enable searching
classifying unique like cancer or
for specific objects or
facial features. pneumonia, aiding
scenes within vast
early diagnosis.
image libraries.
CONCLUSION:
Classification is undeniably a foundational and indispensable task within the realm of
supervised machine learning. Its ability to categorize data into discrete classes
underpins countless technological advancements, making it a critical component of
modern AI systems across various industries.
The diversity of classification algorithms, from the straightforward Decision Tree to
the probabilistic Naïve Bayes, highlights the need to select the right tool for the right
problem. Each algorithm comes with its own strengths and weaknesses, making
algorithm selection a key step in model development.
From detecting spam in our inboxes to identifying fraudulent financial activities and
recognizing objects in complex images, classification's real-world applications are
vast and continue to expand. As AI evolves, so too will the sophistication and reach of
classification techniques, further integrating them into the fabric of our digital lives.