UNIT 5 – CLASSIFICATION
1. Introduction
Classification is a supervised learning technique used to predict categorical outcomes (labels) for data
instances based on past observations. It assigns data to predefined groups or classes.
Example: Predict whether a new email is Spam or Not Spam.
2. Basic Terminology
Term Meaning
Training Data Data with known labels used to train the model.
Testing Data Data used to test the model’s predictive power.
Features Independent variables or attributes.
Target Variable Dependent variable to be predicted.
Model Evaluation Measurement of prediction accuracy and reliability.
3. Naïve Bayes Classifier
Based on Bayes’ Theorem of conditional probability, it assumes independence between features. It is
widely used for text classification and spam detection.
Bayes’ Theorem: P(C|X) = [P(X|C) * P(C)] / P(X)
Steps:
1. Calculate prior probabilities for each class.
2. Calculate conditional probabilities for each attribute given the class.
3. Apply Bayes’ Theorem to compute posterior probability.
4. Assign the class with the highest posterior probability.
4. Nearest Neighbour Classifier (k-NN)
An instance-based learning algorithm that classifies a new sample based on the majority class of its k
nearest neighbours.
Steps:
1. Choose k.
2. Calculate distances (e.g., Euclidean).
3. Select k nearest neighbours.
4. Assign the most frequent class.
5. Decision Tree Classifier
A tree-structured model for classification. Internal nodes test attributes, branches show outcomes, and
leaves represent class labels.
Algorithm Criterion Used
ID3 Information Gain
C4.5/C5.0 Gain Ratio
CART Gini Index
Entropy(S) = -Σ p_i log■(p_i)
6. Overfitting and Underfitting
Overfitting occurs when a model memorizes noise in training data. Underfitting happens when a model
is too simple to capture relationships.
Condition Description Example
Overfitting Excellent on training, poor on test Deep tree memorizes noise
Underfitting Poor on both train and test Too shallow tree
7. Confusion Matrix
Predicted Positive Predicted Negative
Actual Positive True Positive (TP) False Negative (FN)
Actual Negative False Positive (FP) True Negative (TN)
8. Evaluation Metrics
Metric Formula Interpretation
Accuracy (TP + TN)/(TP + TN + FP + FN) Overall correctness
Precision TP/(TP + FP) Proportion of true positives among predicted positives
Recall TP/(TP + FN) Ability to detect positives
F1 Score 2 × (Precision×Recall)/(Precision+Recall) Balance between precision & recall
Specificity TN/(TN + FP) Ability to detect negatives
9. Model Evaluation
Use Train–Test Split and Cross-Validation to evaluate generalization. Apply ROC Curve and AUC for
performance visualization.
10. Summary Table
Concept Description Example Algorithm
Naïve Bayes Probabilistic, independent features Spam Filter
k-NN Distance-based instance learning Recommender System
Decision Tree Hierarchical rule-based classification Credit Risk Analysis
Overfitting Model too complex Deep Decision Tree
Evaluation Metrics Accuracy, Precision, Recall, F1 Model Evaluation
11. Visual Summary
Input Data → Training Algorithm (Naïve Bayes / k-NN / Decision Tree) → Model Creation → Testing
and Evaluation → Best Classifier → Prediction or Decision