UNIT 3

1. What is Classification in Data Mining?

➔ Classification is the process of predicting the category or class label of data objects
based on a training dataset.

2. Define Decision Tree in Classification.

➔ A Decision Tree is a tree-like structure where each internal node represents a test
on an attribute, branches represent outcomes, and leaf nodes represent class labels.

3. What is the purpose of a Training Dataset in Classification?

➔ The Training Dataset is used to build the classification model by learning patterns
between attributes and class labels.

4. Define Overfitting in Classification.

➔ Overfitting occurs when a model fits the training data too well but performs poorly
on unseen data.

5. What is Naïve Bayes Classifier?

➔ Naïve Bayes is a probabilistic classifier based on applying Bayes’ theorem,
assuming independence between features.

6. What is Support Vector Machine (SVM)?

➔ SVM is a classification algorithm that finds the optimal hyperplane which separates
different classes in the feature space.

7. What is meant by Class Label in Classification?

➔ A Class Label is the predefined category to which a data object is assigned (e.g.,
"spam" or "not spam").

8. Define Model Evaluation in Classification.

➔ Model Evaluation assesses the performance of a classification model using metrics
like accuracy, precision, recall, and F1-score.
 9. What is Pruning in Decision Trees?
➔ Pruning removes sections of the tree that do not provide power in classifying
instances to reduce overfitting.

10. What is K-Nearest Neighbor (K-NN) Classifier?

➔ K-NN classifies an object based on the majority class among its K nearest
neighbors in the feature space.

11. What is meant by Attribute Selection in Classification?

➔ Attribute Selection chooses the most relevant attributes from the dataset to improve
model accuracy and reduce complexity.

12. What is the role of Probability in Naïve Bayes Classification?

➔ Probability is used to compute the likelihood of an instance belonging to a class
based on the observed attribute values.

13. What is Entropy in Decision Tree algorithms?

➔ Entropy measures the impurity or disorder in a dataset and helps decide the
attribute that splits the dataset best.

4 MARKS

1. Explain the working of the Decision Tree Classification algorithm with an

example.
➔
A Decision Tree builds a tree where:
o Internal nodes represent attribute tests (e.g., Is Age > 30?).
o Branches represent possible attribute values.
o Leaves represent class labels (e.g., “Yes” or “No”).
Example:
To classify whether a person buys a computer:
o If Age > 30 and Income = High → Class: Yes.
The algorithm splits the dataset by choosing attributes that maximize
information gain (based on Entropy) until pure leaves are formed.
This approach helps visually understand classification rules.

2. Describe Naïve Bayes Classifier with its formula and a real-world example.
➔
 Naïve Bayes applies Bayes’ theorem assuming independence between attributes:
Formula:
P(C∣X)=P(X∣C)⋅P(C)P(X)P(C|X) = \frac{P(X|C) \cdot P(C)}
{P(X)}P(C∣X)=P(X)P(X∣C)⋅P(C)
Where:
o CCC = Class
o XXX = Feature vector
Example:
In spam filtering, classify emails as “Spam” or “Not Spam” based on words
like “Offer”, “Free”.
It calculates probabilities for each class and selects the class with the highest
posterior probability.
Simple, fast, and works well for text classification.

3. Explain Support Vector Machine (SVM) and its application with an example.
➔
SVM finds an optimal hyperplane that separates classes in feature space.
o Maximizes the margin between support vectors of each class.
Example:
In medical diagnosis, classify tumor as “Malignant” or “Benign” based on
features like size, shape.
SVM performs well with high-dimensional data and avoids overfitting using
regularization.

4. What is K-Nearest Neighbor (K-NN) Classifier? Explain its working with an

example.
➔
K-NN classifies a data point based on majority class of its K nearest neighbors using a
distance metric (e.g., Euclidean distance).
Example:
For K = 3, if 2 out of 3 neighbors are labeled “Spam”, the new email is classified as
“Spam”.
Simple, no training phase, suitable for small datasets, but computationally expensive
for large datasets.

5. Explain the concept of Overfitting and how to prevent it in Classification

Models.
➔
Overfitting occurs when a model learns noise or random fluctuations in training data.
Prevention Techniques:
o Pruning (for Decision Trees): Removes less important branches.
o Cross-validation: Validates model performance on unseen data.
o Regularization (in SVM): Controls complexity of the model.
Example:
 A Decision Tree grown fully without pruning perfectly classifies training data
but performs poorly on test data.

6. What is Attribute Selection in Classification? Why is it important?

➔
Attribute Selection chooses the most relevant features to build an effective
classification model.
Importance:
o Reduces dimensionality.
o Improves model accuracy.
o Speeds up training.
Example:
For a customer churn model, selecting relevant attributes like “Last Purchase
Date” and “Monthly Spend” over irrelevant ones like “Customer ID” improves
model performance.

7. Describe the role of Entropy and Information Gain in building a Decision Tree.
➔
o Entropy: Measures impurity in the dataset.
Entropy=−∑P(i)log⁡2P(i)\text{Entropy} = -\sum P(i) \log_2
P(i)Entropy=−∑P(i)log2P(i)
o Information Gain: Measures the reduction in entropy after splitting data on
an attribute.
IG(Attribute)=Entropy(parent)−Weighted Entropy(children)\
text{IG(Attribute)} = \text{Entropy(parent)} - \text{Weighted
Entropy(children)}IG(Attribute)=Entropy(parent)−Weighted Entropy(children
)
The attribute with the highest Information Gain is chosen for splitting at each
node.
Example:
Splitting “Income” yields the highest information gain in predicting loan
defaults.

8. Discuss the advantages and disadvantages of Naïve Bayes Classifier.

➔
Advantages:
o Simple and fast.
o Works well for text classification and spam detection.
o Handles missing data well.
Disadvantages:
o Assumes attribute independence, which is often unrealistic.
o Performance degrades when attributes are correlated.
Example: Despite its simplicity, Naïve Bayes performs surprisingly well in
email spam filtering.
 9. Explain Cross-Validation and its importance in Classification models.
➔
Cross-Validation splits data into training and validation sets multiple times to assess
model performance.
Types:
o K-Fold Cross-Validation.
o Leave-One-Out Cross-Validation.
Importance:
o Prevents overfitting.
o Provides a better estimate of real-world performance.
Example: 10-fold cross-validation averages accuracy over 10 experiments.

10. Explain the working of Logistic Regression in Classification.

➔
Logistic Regression models the probability of a binary outcome:
P(Y=1∣X)=11+e−(β0+β1X1+...+βnXn)P(Y=1|X) = \frac{1}{1 + e^{-(\beta_0 + \
beta_1 X_1 + ... + \beta_n X_n)}}P(Y=1∣X)=1+e−(β0+β1X1+...+βnXn)1
Example:
Predicting whether a customer will churn (Yes/No) based on attributes like usage
frequency, customer age.
It outputs probabilities which are thresholded to classify into classes.
Simple yet powerful for binary classification.

7 MARKS

1. Explain the complete workflow of building a Classification Model in Data

Mining.
➔
The workflow of a classification model includes:
1. Data Collection: Gather relevant dataset from different sources (e.g.,
customer data, sales records).
2. Data Preprocessing: Handle missing data, remove noise, normalize or
standardize features.
3. Feature Selection: Select the most relevant attributes to reduce
dimensionality and improve performance.
4. Model Selection: Choose a suitable classification algorithm (e.g., Decision
Tree, SVM, K-NN, Naïve Bayes).
5. Training the Model: Apply the training data to build the model using
supervised learning.
6. Model Evaluation: Validate using test data and evaluate metrics like
accuracy, precision, recall, and F1-score.
7. Model Deployment: Use the model to classify new/unseen data.
This process ensures an efficient and accurate classification solution.
2. Discuss the advantages and limitations of Decision Trees in Classification.
➔
Advantages:
o Easy to understand and visualize.
o Requires little data preprocessing.
o Handles both numerical and categorical data.
Limitations:
o Prone to overfitting if not pruned properly.
o Small variations in data may lead to different trees.
o Not suitable for continuous variable prediction.
Example: Useful in customer credit scoring but may overfit if too deep.

3. Explain how Support Vector Machine (SVM) works with a kernel function and
its importance.
➔
SVM works by finding the optimal hyperplane that separates classes with maximum
margin.
o Kernel Function:
Transforms data into higher-dimensional space where linear separation is
possible.
Common Kernels:
 Linear Kernel
 Polynomial Kernel
 Radial Basis Function (RBF) Kernel
Example:
For non-linearly separable data (e.g., concentric circles), an RBF
kernel projects data into higher space where separation is feasible.
Kernel avoids explicit feature mapping and makes SVM powerful for
complex datasets.

4. Describe the process of Model Evaluation using Confusion Matrix.

➔
A Confusion Matrix compares actual vs predicted class labels:

Predicted Positive Predicted Negative

Actual Positive True Positive (TP) False Negative (FN)
Actual Negative False Positive (FP) True Negative (TN)
Key Metrics:

o Accuracy = (TP + TN) / Total Instances

o Precision = TP / (TP + FP)
o Recall = TP / (TP + FN)
 o F1-score = 2 × (Precision × Recall) / (Precision + Recall)
This evaluation helps assess model performance beyond simple accuracy.

5. Explain the difference between Binary Classification and Multi-class

Classification with examples.
➔
o Binary Classification:
Two classes (e.g., spam vs not spam in email filtering).
o Multi-class Classification:
More than two classes (e.g., classifying handwritten digits 0–9 in digit
recognition).
Example:
o Binary: Predicting “Loan Approved” or “Not Approved”.
o Multi-class: Classifying types of fruits: Apple, Banana, Orange.

6. Describe Ensemble Learning and explain Bagging and Boosting with examples.
➔
Ensemble Learning: Combines multiple models to improve accuracy.
o Bagging (Bootstrap Aggregating):
Multiple models trained on random samples → Aggregated by voting (e.g.,
Random Forest).
Example: Voting among several decision trees to classify customer churn.
o Boosting:
Sequentially trains weak learners where each focuses on previous errors (e.g.,
AdaBoost, Gradient Boosting).
Example: Improving model accuracy in predicting credit card fraud by
correcting misclassified instances.

7. Explain the limitations of K-Nearest Neighbor (K-NN) and how to mitigate them.
➔
Limitations:
o Computationally expensive for large datasets.
o Sensitive to irrelevant/noisy features.
o Requires proper selection of K value.
Mitigation:
o Use dimensionality reduction (e.g., PCA).
o Normalize feature scales.
o Optimize K using cross-validation.
Example: Using PCA to reduce feature set before K-NN improves speed and
accuracy.
8. What is Logistic Regression? Explain the use of Sigmoid Function in it.
➔
Logistic Regression predicts probability of binary outcomes.
Formula:
P(Y=1∣X)=11+e−(β0+β1X1+...+βnXn)P(Y=1|X) = \frac{1}{1 + e^{-(\beta_0 + \
beta_1 X_1 + ... + \beta_n X_n)}}P(Y=1∣X)=1+e−(β0+β1X1+...+βnXn)1
o Sigmoid Function: Maps any real value to (0,1) → Probability of belonging
to class 1.
Example: Predict customer churn (Yes/No) based on usage patterns.
It provides interpretable results with probabilities.

9. Explain how Pruning prevents overfitting in Decision Tree Classifier.

➔
Pruning removes branches that have little importance.
o Pre-Pruning: Stop splitting when further splits do not improve performance.
o Post-Pruning: Remove branches after the full tree is grown.
Example: If a branch classifies only one data point, it is pruned to prevent
noise fitting.
Result: Improved generalization and better performance on unseen data.

10. Discuss the importance of Hyperparameter Tuning in Classification algorithms.

➔
Hyperparameters control the learning process (e.g., K in K-NN, max depth in
Decision Tree).
o Proper tuning improves model accuracy and prevents overfitting.
Methods:
o Grid Search.
o Random Search.
Example:
o Selecting optimal K in K-NN via cross-validation ensures a balance between
underfitting and overfitting.
Hyperparameter tuning is crucial for achieving optimal model performance.