Study Guide: Classification Methods

Summarized from Lectures

1 Classification Concepts
Classification is the process of predicting categorical labels. The methods cov-
ered include:
• Bayes Classifier (Naı̈ve Bayes)

• Logistic Regression
• Linear Discriminant Analysis (LDA)
• Quadratic Discriminant Analysis (QDA)
• K-Nearest Neighbors (KNN)

• Support Vector Machines (SVM)

2 Bayes Theorem and Naı̈ve Bayes Classifier

2.1 Bayes Theorem
P (B|A)P (A)
P (A|B) =
P (B)
Naı̈ve Bayes assumes feature independence:

P (X|Y = k)P (Y = k)
P (Y = k|X) =
P (X)

Example: Predicting a patient’s health status based on nasal congestion using

historical probabilities.

2.2 Validation Methods

• Confusion Matrix: Measures accuracy using TP, TN, FP, FN.
• Precision, Recall, and Specificity: Used to evaluate classifier perfor-
mance.
• ROC Curve: Plots True Positive Rate vs False Positive Rate.

1
3 Logistic Regression
3.1 Concept
Models the probability of a class using the logistic function:

eβ0 +β1 X
P (Y = 1|X) =
1 + eβ0 +β1 X
Example: Predicting coronary heart disease risk based on tobacco consump-
tion.

3.2 Advantages and Drawbacks

• Simple and interpretable.
• Assumes linearity and independence among predictors.
• Struggles with well-separated classes.

4 Linear Discriminant Analysis (LDA)

4.1 Concept
1
δk (X) = X T Σ−1 µk − µTk Σ−1 µk + log πk
2
Example: Assigning patients to a risk group based on blood pressure and
cholesterol.

4.2 Quadratic Discriminant Analysis (QDA)

Like LDA but allows different covariance matrices for each class.

5 K-Nearest Neighbors (KNN)

5.1 Concept
Assigns a label based on the majority class among the K nearest points:
Kj
P (Y = j|X) =
K
Example: Classifying a new tobacco user based on past smoker data.

5.2 Effect of K
• Small K: High variance, overfitting.
• Large K: High bias, underfitting.

2
6 Support Vector Machines (SVM)
6.1 Concept
Finds the optimal hyperplane that maximizes the margin:

wT X + b = 0

Example: Separating heart disease patients based on multiple health indica-

tors.

6.2 Kernel Trick

Transforms data into higher dimensions to allow non-linear separation.

7 Exam Questions (Multiple Choice)

1. Which of the following classifiers assumes feature independence?
• (A) Logistic Regression
• (B) Naı̈ve Bayes
• (C) SVM
• (D) KNN
2. In Logistic Regression, the odds ratio is given by:
P (Y =1|X)
• (A) 1−P (Y =1|X)

• (B) P (Y = 1|X) − P (Y = 0|X)

• (C) eβ0 +β1 X
• (D) log(P (Y = 1|X))
3. Which algorithm is considered non-parametric?
• (A) Naı̈ve Bayes
• (B) Logistic Regression
• (C) KNN
• (D) LDA
4. The decision boundary in LDA is:
• (A) Quadratic
• (B) Linear
• (C) Non-linear
• (D) Stepwise

3
 5. What does the parameter K in KNN influence?
• (A) The complexity of the decision boundary
• (B) The number of features in the dataset
• (C) The distance metric used
• (D) The model’s training time
6. In SVM, a hyperplane:
• (A) Maximizes classification accuracy
• (B) Separates data by maximizing margin
• (C) Is always a straight line
• (D) Must pass through all points
7. Which validation metric is best for imbalanced data?
• (A) Accuracy
• (B) Recall
• (C) Precision
• (D) F1-score
8. The ROC curve is used to:

• (A) Measure class probabilities

• (B) Select the best K in KNN
• (C) Compare classifiers’ performance
• (D) Optimize hyperparameters in SVM

9. What does a high bias model tend to do?

• (A) Overfit the training data
• (B) Generalize well on new data
• (C) Have high variance
• (D) Require a large dataset

10. The main drawback of KNN is:

• (A) Poor interpretability
• (B) Sensitivity to outliers
• (C) Inability to handle categorical data
• (D) High computational cost

4
8 Exam Questions (Multiple Choice with An-
swers and Explanations)
1. Which of the following classifiers assumes feature independence?
Correct answer: (B) Naı̈ve Bayes
Explanation: The Naı̈ve Bayes classifier assumes that features are con-
ditionally independent given the class. This simplifies probability calcula-
tions but can be inaccurate if features are correlated. Logistic Regression
(A), SVM (C), and KNN (D) do not make this assumption.
2. In Logistic Regression, the odds ratio is given by:
P (Y =1|X)
Correct answer: (A) 1−P (Y =1|X)
Explanation: The odds ratio expresses the relative likelihood of an event
occurring compared to not occurring. It is calculated as the ratio of the
probability of the positive class to the probability of the negative class.
Option (C) is the exponentiated logit function, while (B) and (D) do not
correctly represent the odds ratio.
3. Which algorithm is considered non-parametric?
Correct answer: (C) KNN
Explanation: K-Nearest Neighbors does not assume a fixed functional
form for the data distribution and relies on storing and comparing data
points directly. Logistic Regression (B) and Naı̈ve Bayes (A) are para-
metric models, while LDA (D) assumes a Gaussian distribution for each
class.
4. The decision boundary in LDA is:
Correct answer: (B) Linear
Explanation: Linear Discriminant Analysis (LDA) assumes that differ-
ent classes have the same covariance matrix, leading to a linear decision
boundary. Quadratic decision boundaries (A) are found in QDA, while
non-linear and stepwise (C, D) do not apply.
5. What does the parameter K in KNN influence?
Correct answer: (A) The complexity of the decision boundary
Explanation: A small K results in a highly flexible and complex bound-
ary (prone to overfitting), while a large K smooths the boundary (prone
to underfitting). Distance metric (C) is separate from K, and (B) and (D)
do not directly relate to K.
6. In SVM, a hyperplane:
Correct answer: (B) Separates data by maximizing margin
Explanation: SVM seeks the hyperplane that maximizes the margin be-
tween classes, making it robust to noise. It does not necessarily maximize
accuracy (A), is not always a straight line (C, due to kernel methods), and
does not need to pass through all points (D).

5
 7. Which validation metric is best for imbalanced data?
Correct answer: (D) F1-score
Explanation: The F1-score balances precision and recall, making it suit-
able for imbalanced datasets. Accuracy (A) can be misleading, recall (B)
favors positive classes, and precision (C) does not consider false negatives.

8. The ROC curve is used to:

Correct answer: (C) Compare classifiers’ performance
Explanation: The ROC curve plots the true positive rate against the
false positive rate for different thresholds, helping compare classifier per-
formance. It does not directly measure class probabilities (A), optimize
KNN (B), or adjust SVM parameters (D).

9. What does a high bias model tend to do?

Correct answer: (B) Generalize well on new data
Explanation: High bias models have simpler assumptions and generalize
better but may underfit. Overfitting (A) is due to high variance, (C) is
incorrect, and (D) is unrelated.

10. The main drawback of KNN is:

Correct answer: (D) High computational cost
Explanation: KNN requires storing all training data and computing
distances for each prediction, making it slow. It is interpretable (A),
handles categorical data (C), and while it is sensitive to outliers (B), its
biggest issue is computation time.