Exam Practice Questions

Q1 Answer True/False in the following:

1. Consider a cancer diagnosis classification problem where almost all the people being diagnosed
don’t have cancer. The probability of correct classification is the most important metric to
optimize.
2. Consider a cancer diagnosis classification problem where almost all the people being diagnosed
don’t have cancer. For this dataset, Receiver Operating Characteristic (ROC) curve can measure
the true performance.

3. Logistic Regression can be used for classification.

4. Logistic Regression with no polynomial will always give a linear decision boundary.

5. By normalizing data, gradient decent algorithm will converge faster.

6. We can get one global-minima if we solve a linear regression problem by minimizing the sum of
squared errors using gradient descent.

7. In gradient decent based algorithms, choosing a small learning rate may cause the model fail to
converge.

8. For linear-SVM with large C = infinity value giving a linearly separable dataset, the training error
is guaranteed to be zero.

9. Even Stochastic gradient descent is faster to update the model parameters (comparing to Batch
gradient descent). Batch gradient descent takes longer time to converge.

10. The only way to solve logistic regression problem is using gradient based solutions as there is no
known analytical way to solve it.

11. Support vector machines, like logistic regression models, give a probability distribution over the
possible labels given an input example.

12. Given m data points, the training error converges to the true error as m → ∞.

13. Adding more features to a linear regression model always increase model variance.

14. Leave-one-out cross validation generally gives less accurate estimates of true test error than 10-
fold cross validation.

15. When the feature space is larger, over fitting is more likely.

16. If a learning algorithm is suffering from high bias, getting more training data is likely to help.

1
 17. If we use feature scaling, it’s guaranteed that gradient descent-based algorithms will converge.

18. In gradient based algorithms, early stopping criteria can work as regularization.
19. Decision trees with depth one will always give a linear decision boundary.

20. Making a decision tree deeper will assure better fit but reduce robustness/generalization.

21. K-means automatically adjusts the number of clusters.

22. For two runs of K-Mean, with same K and initialized from same clusters’ centroids. It is expected
that both runs give same clustering results.

23. Subsequent principle components are always orthogonal to each other.

24. The largest eigenvector of the covariance matrix is the direction of minimum variance in the data

25. Dimensionality reduction can be used as pre-processing for machine learning algorithms like
decision trees, SVM, etc.

26. Random forest is an ensemble learning method that attempts to lower the bias error of decision
trees.

27. In SVMs, the values of αi for non-support vectors are 0.

28. For k-means algorithm, finding the optimal number of clusters using Silhouette Score method is
more accurate than the elbow method.

Q2 Circle the correct answer(s).

1. Which statement is WRONG about PCA:

(a) PCA seeks for linear transformations to a lower dimensional space.
(b) PCA first component tries to capture the least of data variance.
(c) PCA is an unsupervised method.
(d) PCA can be used as pre-processing step for other machine learning algorithms.

2. Which one of these statements is TRUE about decision tree:

(a) A tree with depth of 3 has higher variance than a tree with depth of 1.
(b) A tree with depth of 3 has higher bias than a tree with depth 1.
(c) A tree with depth of 3 always has higher training error than a tree with depth 1.
(d) A tree with depth of 3 never has higher test error than a tree with depth 1.

3. Consider a point that is correctly classified and distant from the decision boundary.
(a) SVM’s decision boundary might be unaffected by this point, but the one learned by logistic
regression will be affected.

2
 (b) SVM’s decision boundary will be affected by this point, and the one learned by logistic
regression will be affected.
(c) SVM’s decision boundary might be unaffected by this point, but the one learned by logistic
regression will be unaffected.
(d) SVM’s decision boundary will be affected by this point, but the one learned by logistic
regression will be unaffected.

4. Suppose we want to compute 10-Fold Cross-Validation error on 100 training examples. We need
to compute error N1 times, and the Cross-Validation error is the average of the errors. To compute
each error, we need to build a model with data of size N2, and test the model on the data of size
N3. What are the appropriate numbers for N1, N2, N3?
(a) N1 = 1, N2 = 90, N3 = 10
(b) N1 = 10, N2 = 100, N3 = 10
(c) N1 = 10, N2 = 90, N3 = 10
(d) N1 = 10, N2 = 100, N3 = 100

5. To Maximize the margin in SVM, we want to:

1
(a) max 2 ‖𝜃‖2 such that 𝜃 𝑇 𝑋𝑖 ≤ 1 𝑖𝑓 𝑦𝑖 = 1 𝑎𝑛𝑑 𝜃 𝑇 𝑋𝑖 ≥ −1 𝑖𝑓 𝑦𝑖 = −1
𝜃
1
(b) min such that 𝜃 𝑇 𝑋𝑖 ≥ 1 𝑖𝑓 𝑦𝑖 = 1 𝑎𝑛𝑑 𝜃 𝑇 𝑋𝑖 ≤ −1 𝑖𝑓 𝑦𝑖 = −1
𝜃 2‖𝜃‖2
1
(c) min 2 ‖𝜃‖2 such that 𝜃 𝑇 𝑋𝑖 ≥ 1 𝑖𝑓 𝑦𝑖 = 1 𝑎𝑛𝑑 𝜃 𝑇 𝑋𝑖 ≤ −1 𝑖𝑓 𝑦𝑖 = −1
𝜃
(d) None of above

6. Which of the following is true about “Ridge” or “Lasso” regression methods in case of feature
selection?
(e) Ridge regression uses subset selection of features
(f) Lasso regression uses subset selection of features
(g) Both use subset selection of features
(h) None of the above

7. You train a learning algorithm and find that it has unacceptable high error on the test set. You
plot the learning curve and obtain the figure below. What you can conclude form the graph:

(a) Algorithm has high bias.

(b) Algorithm has high variance.
(c) Neither
(d) Can’t tell

3
Q3 Decision tree.
For this problem you will respond to several questions concerning the dataset shown
below. You will be using a decision tree to classify whether or not an advertisement
was clicked based on its size, position, and whether or not it played a sound.

H(S) = -p+ log2 p+ - p- log2 p-

|Sv|
𝐺𝑎𝑖𝑛(𝑆, 𝐴) = 𝐸𝑛𝑡𝑟𝑜𝑝𝑦 (𝑆) − ∑ Entropy(Sv)
|S|
vvalues(A)

(a) [4 points] What is the initial entropy of Clicked?

(b) [8 points] Assume that Position is chosen for the root of the decision tree. What is the information
gain associated with this attribute?

(c) [8 points] Draw the full decision tree learned from this data (without any pruning).

4
Q4 Short answers.
1. Mention two main problems in K-means algorithm, what do you suggest to solve each problem?

2. The K-Means algorithm will in general converge to a local-optima rather than a

global one. Given this, how would you adapt the algorithm to increase the chances of finding a
good solution?

3. Random forest is an ensemble method that uses bagging and random selection features, at which
level the random selection features happens?

4. Describe how random forest algorithm works.

5. The goal of this problem is to correctly classify test data points, given a training data set. For this
problem, assume that we are training an SVM with a quadratic kernel– that is, our kernel function
is a polynomial kernel of degree 2. You are given the data set presented in Figure 1 (page-7). The
slack penalty C will determine the location of the separating hyperplane. Please answer the
following questions qualitatively. Give a one sentence answer/justification for each and draw
your solution in the appropriate part of the Figure at the end of the problem.
a. Where would the decision boundary be for very large values of C (i.e., C → ∞)?
(remember that we are using an SVM with a quadratic kernel.) Draw on the figure below.
Justify your answer.

b. For C ≈ 0, indicate in the figure below, where you would expect the decision boundary to
be? Justify your answer.

c. Which of the two cases above (1 & 2) would you expect to work better in the
classification task? Why?

d. Draw (add) a data point which will not change the decision boundary learned for very
large values of C. Justify your answer.

e. Draw (add) a data point which will significantly change the decision boundary learned
for very large values of C. Justify your answer.

5
 Figure 1

6. What is the leave-one-out cross-validation error estimate for maximum margin separation in the
following figure? (we are asking for a number), Justify your answer.

6
7. Consider the data shown below to answer question 3.1 – 3.2:

a. In Panel A, Sketch (on the panel A) a pair of eigenvectors that you would expect to
obtain from a Principal Components Analysis (PCA) of the data. Indicate which
eigenvector you would expect to have the larger corresponding eigenvalue

b. Do you think it is appropriate to use PCA to reduce the dimensionality of the dataset
shown in panel B? Why or why not?

8. Suppose that you have a high dimensional data, describe how you can use PCA to reduce data
dimensions so you can visualize it?