KNN, Naive Bayes &
Decision Tree
Cheatsheet
Questions
Q1. Suppose, you have been given the
following data where x1 and x2 are the 2
input variables and Class is the dependent
variable.
What will be the class of a new data point
x1=1 and x2=1 in 5-NN (k nearest
neighbour with k=5) using euclidean
distance measure?
A. + Class
B. - Class
C. Cannot be determined
Q2. The table below shows the test set for a 1-NN classifier
that uses Manhattan distance, i.e., the distance between two
points at coordinates p and q is |p-ql. The only attribute, X, is
real-valued, and the label, Y, has two classes, 0 and 1.
Suppose a subset containing n≤ 8 examples is selected from
this set to train the classifier, and the accuracy of the
classifier is 100 percent when tested on this set (with all 8
examples). What is the smallest possible value for n? In
case of ties in distance, use the example with smallest X
value as the neighbor.
Q3. KNN has curse of dimensionality: high dimensions
cause?
A) Distances become similar, neighbors meaningless
B) Faster computation
C) Automatic feature selection
D) Linear separability
Q4. For regression KNN (k=3), neighbor values are: 2.1, 3.4,
2.8. What is going to be prediction?
A) Mean: 2.77
B) Median: 2.8
C) Mode: none
D) Weighted mean (needs distances)
Q5. Unscaled features in K-NN with different ranges cause?
A) High-range features dominate distance
B) No effect
C) Automatic normalization
D) Reduces bias
Q6. A 1D classification problem has training
points: A(2, C1), B(5, C2), C(6, C2), D(10, C1). A
new point X_new is at x = 4. Using K = 3 and
Manhattan distance, which class is assigned to
X_new?
A. C1
B. Classification is not possible because K = 3 is
an odd number.
C. Classification is not possible because of the tie
in distance between A and C.
D. C2
Q7. In a 2D feature space, a new instance X_new = (5, 1)
needs to be classified. The distances to its three closest
neighbors are d1 = 1.0, d2 = 1.5, and d3 = 2.0. Their
respective class labels are C_A, C_B, C_A. If you use
distance-weighted KNN with K = 3, what is the weighted
score for class C_A?
A. 2.0
B. 1.43
C. 1.5
D. 1.83
Q8. When using a Naive Bayes classifier for prediction, what
rule is used to assign a class to a new observation?
A. Minimum Error Rule
B. Maximum a Posteriori (MAP) Rule
C. Minimum Distance Rule
D. Maximum Likelihood Rule
Q9. Which of the following is/are the assumption of the Naive
Bayes classifier?
A. Each feature is independent of all other features given the
class.
B. All features are equally important in predicting the class
label.
C. There can be missing values in the training data.
D. Class labels are mutually exclusive
Q10. To prevent computational underflow when calculating
the likelihood term,
in high-dimensional spaces (where the product is a
multiplication of many small probabilities), Naive Bayes
implementations often calculate the score by summing which
transformed values?
A. The square roots of the probabilities.
B. The reciprocals of the probabilities.
C. The exponentials of the probabilities.
D. The logarithms of the probabilities.
Q11. Suppose you have a three class problem where class
label y belongs to (0, 1, 2) and each training example X has
3 binary attributes X1, X2, X3 belongs to (0,1). How many
parameters do you need to know to classify an example
using the Naive Bayes classifier?
(a) 5
(b) 9
(c) 11
(d) 13
Q12. Estimate P(Xj | Y) and P(Y) directly from the training data data by counting
with/without Laplace smoothing.
P(play)
P(¬ play)
P(Sky = sunny | play)
P(Sky sunny | ¬ play)
P(Humid = high | play)
P(Humid = high | ¬ play)
Q13. Consider given dataset:
Which among the following
split-points for the feature1 would
give the best split according to the
information gain measure?
(a) 14.6
(b) 16.05
(c) 16.85
(d) 17.35
Q14. For the same dataset, which among the
following split-points for feature2 would give the
best split according to the gini index measure?
Q15. Which of the following describes the key mechanism by which a Decision
Tree classifies a new instance?
A. By finding the hyperplane that maximizes the margin between the two
closest classes.
B. By calculating the kernel similarity between the new instance and stored
support vectors.
C. By recursively partitioning the feature space into hyper-rectangles such that
each region primarily contains data from a single class.
D. By computing the posterior probability P(C|X) using Bayes' Theorem under
the conditional independence assumption.
Q16. Information Gain (IG) tends to be biased toward attributes with a large
number of distinct values (high cardinality). The standard modification used to
counteract this bias and avoid selecting meaningless splits is the Gain Ratio.
How does Gain Ratio modify the Information Gain calculation
A. It multiplies the Information Gain by the Gini Index of the parent node.
B. It divides the Information Gain by the Split Information (Split Entropy), which
penalizes attributes that create many small branches.
C. It subtracts a penalty based on the split's depth from the Information Gain.
D. It uses the Euclidean distance instead of the logarithmic function for
calculating purity.
Q17. How are continuous attributes typically handled by Decision Tree
algorithms to determine the best split?
A. A threshold value T is found (e.g., X ≤ T vs. X > T) by examining
the midpoint between two adjacent, distinct values of the feature that belong to
different classes.
B. The continuous feature is ignored since Entropy and Gini Index are only
defined for discrete variables.
C. The mean and variance of the feature are calculated and used in a Gaussian
distribution assumption.
D. They are discretized into categorical ranges (bins) of equal width before the
algorithm starts.
Q18. While solving a classification problem, you use a pure, binary decision tree
constructed by the standard greedy procedure we outlined in class. While your
training accuracy is perfect, your validation accuracy is unexpectedly low. Which
of the following, in isolation, is likely to improve your validation accuracy in most
real-world applications?
A. Lift your data into a quadratic feature space
B. Select a random subset of the features and use only those in your tree
C. Normalize each feature to have variance 1
D. Prune the tree, using validation to decide how to prune
Q19. Suppose you have a binary classification problem with 3-dimensional
feature vectors x R3. You are given 50 positive and 50 negative training
examples, and want to build a decision tree classifier. Consider 4 possible splits
at the root node, and find which of the below splits gives the highest
information?
Q20. If a feature perfectly splits the data into pure nodes, the Information Gain
is:
A) Maximum, equal to parent's entropy
B) Minimum, zero
C) Negative
D) Half of parent's entropy