Chapter 4

Classification
Definition of Classification

Classification is a data mining and machine learning technique used to categorize data into
predefined classes or groups. It is a supervised learning method, meaning the model is trained
on labeled data (data with known outcomes).

Example:

● Email Filtering: Classifying emails as spam or not spam.

● Medical Diagnosis: Classifying a patient as diabetic or non-diabetic based on test results.

Model Construction in Classification

Model construction is the first major step in the classification process.

It involves training a machine learning algorithm on a labeled dataset to learn patterns and
relationships between input features (X) and output labels (Y).
Step-by-Step Breakdown:

1. Data Collection

● Collect labeled data (i.e., data with known outcomes).

● The dataset should have:

○ Features: The independent variables (e.g., age, income)

○ Labels: The target/class to predict (e.g., will buy = Yes/No)

2. Data Preprocessing

● Handle missing values: Fill or remove rows with missing data.

● Convert categorical data to numeric (e.g., one-hot encoding).

● Normalize/scale features if needed.

● Split dataset into:

○ Training set (e.g., 80%)
○ Testing set (e.g., 20%)
3. Choose a Classification Algorithm

Some common algorithms used:

● Decision Tree-A tree-based model that splits data based on feature values to make decisions in a
flowchart-like structure.

● Naïve Bayes–A probabilistic classifier based on Bayes’ Theorem assuming feature independence.

● Logistic Regression-A statistical model that uses a sigmoid function to estimate the probability of a
binary outcome.

● K-Nearest Neighbors (KNN)-A lazy learning algorithm that classifies a data point based on the
majority class of its K closest neighbors.

● Support Vector Machine (SVM)-A powerful classifier that finds the optimal hyperplane to separate
data points of different classes.
4. Train the Model

● The algorithm learns from the training data by analyzing feature-label relationships.

● It adjusts internal parameters to minimize error in predicting the class.

Example:

In a decision tree, the model chooses the best features and thresholds to split the data into
pure groups (i.e., all "Yes" or "No").

5. Model Evaluation (Optional but Critical)

● After construction, the model is tested on unseen data to see how well it generalizes.

● Accuracy, precision, recall, and F1 score are commonly used metrics.

Model Usage in Classification

Once a classification model is constructed (trained), the next phase is Model Usage, also called the prediction or
inference phase.

This is where the model is used to classify new, unseen data based on what it has learned during training.

Example-An e-commerce website uses the model to predict if a new visitor will make a purchase based on their
browsing pattern.

Key Purpose:

To apply the trained model to make predictions or classifications on fresh input data

What Happens in Model Usage

1. Input New Data

You provide the model with new instances (data rows) without labels — the model needs to predict those labels
Example:

A trained spam detection model receives a new email and must decide:

● Is it Spam or Not Spam?

2. Preprocess the Input (Same as Training)

The new data must be prepared in the same format used during training:

● Features should be encoded the same way

● Scaling or normalization should match
● Same number and type of inputs

3. Model Applies Its Learned Rules or Patterns

Depending on the algorithm:

● Decision Tree follows a path down the tree based on feature values
● Naïve Bayes calculates probabilities and chooses the most likely class
● KNN finds closest neighbors and picks the most common label among them

4. Output Prediction

The model returns:

● A class label (e.g., "Diabetic", "Not Diabetic")

● Sometimes, a probability score (e.g., 90% chance of fraud)
Example-
Preparing Data for Classification and Prediction

Preparing data is a significant part of the classification and prediction process. This
involves several key activities:

● Data Cleaning: This involves handling missing values and removing "noise" from
the data. Noise can be removed using smoothing techniques, and missing values
are often replaced with the most common value for that particular attribute.
● Relevance Analysis: This analysis is used to determine if two attributes in a
database are related, helping to identify and remove irrelevant attributes.
● Data Transformation and Reduction: Data can be transformed using methods like
normalization, which scales all attribute values to a small, specific range.
Generalization can also be used to transform data by summarizing it to a higher
concept
Examination of Classification and Prediction Strategies

Accuracy

● Classifier Accuracy: This refers to a classifier's ability to correctly predict the class label.
● Predictor Accuracy: This refers to how well a predictor can forecast the value of a predicted attribute for
new data

Speed

● Speed refers to the computational cost of generating and using the classifier or predictor.

Robustness

● Robustness is the ability of a classifier or predictor to make accurate predictions from noisy data.

Scalability

● Scalability is the ability to easily build a classifier or predictor given a large amount of data.

Interpretability

● Interpretability refers to how well the classifier or predictor is understood.

4. Decision Tree Learning

● Decision Tree Learning is a supervised machine learning technique used primarily for classification and
sometimes regression.
● It builds a model in the form of a tree structure, where each node represents a decision or test on an
attribute, and each branch represents the result of that test.
● The final leaf node shows the output class or value.

Example:

● A school uses a decision tree to determine if a student should get a scholarship:

○ Income < ₹2 lakh? ➝ Yes

○ Marks > 85%? ➝ Yes

○ Extra-curricular activities? ➝ Maybe

Components of a Decision Tree:
information Entropy
Information Entropy is a measure of uncertainty in data. In the context of classification, it
measures the spread of labels.

● Low Entropy means the data labels are very uniform. ( data belongs to one class)

For example, a dataset where 99 out of 100 samples are "Negative" has very low
entropy.

● High Entropy means the labels are in disorder. (dataset is mixed — no single label)

For example, a dataset with 45 "Positive" and 55 "Negative" samples has very high
entropy.
How a Decision Tree is Constructed
1. Choose the Best Attribute (Feature)
● At each step, the algorithm selects the most significant feature to split the data.
● Selection is based on criteria like:

○ Information Gain (ID3)

○ Gain Ratio (C4.5)
○ Gini Index (CART)

2. Split the Dataset

● Divide the data based on the values of the selected feature.
● Create child nodes for each subset.

3. Repeat Recursively
● Repeat the above steps for each child node using the remaining features.
4. Stop Condition
● All data in a subset belongs to the same class
● No more attributes to split
● Tree reaches a maximum depth (optional to avoid overfitting)

Example: Student Exam Result Predictor

Common Algorithms for Building Trees

Advantages Disadvantages

● Easy to understand and interpret ● Deep trees may memorize the data and not work
well on new data.
● Works with both numerical and categorical data
● Small changes in data can completely change
● No need for data scaling or normalization the tree.

● If one class appears more, the tree may unfairly

● Handles missing values well favor it.
Bayes’ Theorem
Bayes’ Theorem is a fundamental concept in probability and statistics. It allows
us to update the probability of a hypothesis when new evidence is observed.
The formula is:

Where:
Medical Test Example (Disease Detection)

Suppose there’s a disease that affects 1 in 1,000 people.

● Prior Probability (P(Disease)) = 0.001

● The test for this disease is:

○ 99% accurate if you have the disease (True Positive Rate = 0.99)

○ 5% false positive rate (it shows positive even if you don’t have it = 0.05)

Question:

If a person tests positive, what is the probability that they actually have the disease?
Calculate each term
● P(Disease) = 0.001

● P(No Disease) = 0.999

● P(Positive|Disease) = 0.99

● P(Positive|No Disease) = 0.05

Now,
Naïve Bayes Algorithm

The Naïve Bayes algorithm is a probabilistic AI algorithm used for various classification tasks like spam
filtering, document classification, and sentiment prediction. It's based on Bayes' Theorem.

The algorithm is called "Naïve" because it assumes that the features used in the model are independent
of one another

Benefits of Naïve Bayes:

● It is very quick and can predict the class of a test dataset easily.
● It is effective for multi-class prediction issues.
● It performs well even with less training data, assuming the assumption of feature autonomy holds.
● It is particularly effective with clear-cut input factors and mathematical factors.

Limitations of Naïve Bayes:

● If a variable in the test data set was absent in the training data set, the model will assign it a "zero
likelihood" and won't be able to make a prediction. This is known as "Zero Recurrence".
● It is considered a poor assessor, and you should not take the likelihood results too seriously.
● It assumes that all features are autonomous.
So, if an email contains the word “offer,” there is an 82% chance it’s spam.

So, if an email contains the word “offer,” there is an 82% chance it’s spam.