Classification

Introduction
Classification is a Supervised Machine Learning technique where a model learns from labeled
training data and then categorizes new data into predefined classes. The goal is to predict discrete
labels (categories or classes) based on input features.

Examples of Classification

●​ Spam Detection: Classify emails as Spam or Not Spam.

●​ Disease Prediction: Predict whether a patient has Diabetes or No Diabetes.
●​ Sentiment Analysis: Classify customer reviews as Positive, Negative, or Neutral.
●​ Image Recognition: Identify objects in images (e.g., Cat, Dog, Bird).

Types of Classification

Classification problems can be categorized based on the number of target classes and the
algorithm used.

(A) Based on Number of Classes

1.​ Binary Classification → Only two classes

○​ Example: Spam vs. Not Spam, Fraud vs. Non-Fraud
○​ Algorithms: Logistic Regression, SVM, Decision Tree
2.​ Multi-Class Classification → More than two classes
○​ Example: Handwritten digit recognition (0-9), Flower classification (Setosa,
Versicolor, Virginica in Iris dataset)
○​ Algorithms: Random Forest, kNN, Neural Networks
3.​ Multi-Label Classification → Each data point can belong to multiple categories
○​ Example: Movie genres (Action, Comedy, Drama)
○​ Algorithms: Neural Networks, Decision Trees

(B) Based on Classification Algorithms

1.​ Logistic Regression

○​ Used for Binary Classification
○​ Example: Predicting if a customer will buy a product (Yes/No)
2.​ k-Nearest Neighbors (kNN)
○​ Classifies based on the majority class of nearest neighbors
○​ Example: Identifying handwritten digits
 3.​ Decision Tree
○​ Splits data into branches based on conditions
○​ Example: Loan approval based on income, credit score, and employment
4.​ Support Vector Machine (SVM)
○​ Creates a decision boundary (hyperplane) to separate classes
○​ Example: Cancer detection (Malignant vs. Benign)
5.​ Naïve Bayes
○​ Based on Bayes’ Theorem (probability-based classification)
○​ Example: Spam email classification
6.​ Random Forest
○​ A collection of multiple Decision Trees
○​ Example: Credit Card Fraud Detection
7.​ Neural Networks & Deep Learning
○​ Used for complex tasks like image recognition, speech processing
○​ Example: Facial recognition system

Choosing the Right Classification Algorithm

Scenario Recommended Algorithm

Small dataset with linear decision boundary Logistic Regression, SVM

Large dataset with complex relations Neural Networks, Random Forest

Need fast and interpretable results Decision Tree, Naïve Bayes

Handling noisy data k-NN, Random Forest

Logistic Regression
Logistic Regression is a Supervised Learning algorithm used for Binary
Classification (Yes/No, 0/1, True/False). Unlike Linear Regression, which predicts
continuous values, Logistic Regression predicts probabilities and maps them to a
class label.

Email Spam Detection → (Spam = 1, Not Spam = 0)

Disease Prediction → (Diabetic = 1, Not Diabetic = 0)
Loan Approval → (Approved = 1, Not Approved = 0)

Types of Logistic Regression

1.​ Binary Logistic Regression → 2 Classes (Yes/No)
2.​ Multinomial Logistic Regression → More than 2 Classes (Red, Green,
Blue)
3.​ Ordinal Logistic Regression → Ordered Categories (Low, Medium, High)

Implementing Logistic Regression in R

Step 1: Load Required Packages and Data

# Load necessary library

library(datasets)

# Load the iris dataset

data(iris)

# View the first few rows of the dataset

head(iris)

Step 2: Convert the Target Variable to Binary

Since Logistic Regression is used for binary classification, we will modify the
Species column to classify Setosa vs. Non-Setosa.

# Convert Species to a binary outcome (1 for "setosa", 0 for others)

iris$BinarySpecies <- ifelse(iris$Species == "setosa", 1, 0)

# Check the modified dataset

head(iris)

Step 3: Split the Data into Training and Testing Sets

We divide the dataset into 80% training and 20% testing data.

# Load the library for splitting data

[Link](123) # Set seed for reproducibility

library(caTools)

# Split the data into training (80%) and testing (20%)

split <- [Link](iris$BinarySpecies, SplitRatio = 0.8)

train_data <- subset(iris, split == TRUE)

test_data <- subset(iris, split == FALSE)

# Check the distribution

table(train_data$BinarySpecies)

table(test_data$BinarySpecies)

Step 4: Build the Logistic Regression Model

Now, we use the glm() function to build the logistic regression model.

# Build logistic regression model

model <- glm(BinarySpecies ~ [Link] + [Link] + [Link] +
[Link], data = train_data, family = binomial)

# Display model summary

summary(model)

Advantages of Logistic Regression

Simple and easy to interpret

Works well for binary classification

Provides probability estimates

Less prone to overfitting than complex models

Limitations of Logistic Regression

Not suitable for highly complex patterns

Cannot handle non-linear relationships well

Assumes independence among observations

Support Vector Machine (SVM)

SVM finds the best boundary (hyperplane) that separates different classes.

library(e1071) # For SVM

library(caret) # For evaluation

[Link](123)
data(iris)

indexes <- createDataPartition(iris$Species, p = 0.7, list = FALSE)

train_data <- iris[indexes, ]

test_data <- iris[-indexes, ]

Train the SVM Model

svm_model <- svm(Species ~ ., data = train_data)

Predict and Evaluate

svm_predictions <- predict(svm_model, test_data)

confusionMatrix(svm_predictions, test_data$Species)

Confusion Matrix shows how many instances were correctly/incorrectly classified.​

Accuracy tells the percentage of correct predictions.

K-Nearest Neighbors (KNN)

KNN classifies a point based on how its nearest neighbors are classified.

library(class) # For KNN

normalize <- function(x) { (x - min(x)) / (max(x) - min(x)) }

iris_norm <- [Link](lapply(iris[1:4], normalize))

iris_norm$Species <- iris$Species

Normalize the Features (Important for KNN)

normalize <- function(x) { (x - min(x)) / (max(x) - min(x)) }

iris_norm <- [Link](lapply(iris[1:4], normalize))

iris_norm$Species <- iris$Species

Split into Train/Test

train_data <- iris_norm[indexes, 1:4]

train_labels <- iris_norm[indexes, 5]

test_data <- iris_norm[-indexes, 1:4]

test_labels <- iris_norm[-indexes, 5]

Run KNN Model (k = 5)

knn_pred <- knn(train = train_data, test = test_data, cl = train_labels, k = 5)

confusionMatrix(knn_pred, test_labels)

Naïve Bayes Classifier

Naïve Bayes is a probabilistic classifier based on Bayes’ theorem, assuming

feature independence.

library(e1071)

library(caret)

[Link](123)

# Split data

index <- createDataPartition(iris$Species, p = 0.7, list = FALSE)

train <- iris[index, ]

test <- iris[-index, ]

# Train Naive Bayes model

nb_model <- naiveBayes(Species ~ ., data = train)

# Predict

nb_pred <- predict(nb_model, test)

# Evaluation

confusionMatrix(nb_pred, test$Species)

Decision Tree Classifier

Decision Trees split data based on feature values to classify.

library(rpart)

# Train Decision Tree model

tree_model <- rpart(Species ~ ., data = train, method = "class")

# Predict
tree_pred <- predict(tree_model, test, type = "class")

# Evaluation

confusionMatrix(tree_pred, test$Species)

Random Forest Classifier

Random Forest is an ensemble method that builds many decision trees and
averages their predictions.

library(randomForest)

# Train Random Forest model

rf_model <- randomForest(Species ~ ., data = train)

# Predict

rf_pred <- predict(rf_model, test)

# Evaluation

confusionMatrix(rf_pred, test$Species)

Model Evaluation

The confusion matrix shows how many predictions were correct/incorrect.

You can compare accuracy, precision, and recall between models.

Confusion Matrix and Statistics Reference

Prediction setosa versicolor virginica

setosa 15 0 0

versicolor 0 14 1

virginica 0 1 14

Accuracy : 0.955