Assignment 2: Classification

Q1. For QDA with p = 1 and K classes, prove that the Bayes classifier is not linear in x.
Further, show that the decision boundary between any two classes k and ℓ is given by
an equation that is quadratic in x.
Q2. Suppose we collect data for a group of students in a statistics class with variables

X1 = hours studied, X2 = undergrad GPA, Y = receive an A.

We fit a logistic regression and obtain the estimated coefficients

β̂0 = −6, β̂1 = 0.05, β̂2 = 1.

The estimated variance-covariance matrix of the coefficients is

 
1.44 −0.01 0.10
Var(β̂) = −0.01 0.0004 0.002 .
0.10 0.002 0.09

(a) Estimate the probability that a student who studies for 40 hours and has an
undergrad GPA of 3.5 gets an A in the class.
(b) How many hours would the student in part (a) need to study to have a 50%
chance of getting an A in the class?
(c) Calculate a 95% confidence interval for the predicted probability that the student
in part (a) gets an A.

Q3. Suppose that we wish to predict whether a given stock will issue a dividend this year
(“Yes” or “No”) based on X, last year’s percent profit. We examine a large number of
companies and discover that the mean value of X for companies that issued a dividend
was X̄ = 10, while the mean for those that didn’t was X̄ = 0. In addition, the variance
of X for these two sets of companies was σ̂ 2 = 36. Finally, 80% of companies issued
dividends. Predict the probability that a company will issue a dividend this year given
that its percentage profit was X = 4 last year.
Q4. Derive expressions for
Pr(Y = k | X = x)
log
Pr(Y = K | X = x)
for LDA, QDA, Naive Bayes in terms of appropriate means, covariances and prior
distributions. Compare with the expression for this ratio for logistic regression.

1
Q5. Consider a binary response variable Y ∈ {0, 1} and a single predictor X ∈ R. Suppose
we model
exp(β0 + β1 x)
Pr(Y = 1 | X = x) = .
1 + exp(β0 + β1 x)

(a) Derive the likelihood function L(β0 , β1 ) and the corresponding log-likelihood ℓ(β0 , β1 )
for n independent observations (xi , yi ).
(b) Show that the score equations (first derivatives of the log-likelihood with respect
to β0 and β1 ) are
n
X n
X
(yi − πi ) = 0, (yi − πi )xi = 0,
i=1 i=1

where πi = Pr(Y = 1 | X = xi ).
(c) Explain why these equations generally do not admit a closed-form solution and
describe how iterative methods (e.g., Newton-Raphson) are used to obtain the
MLEs.
(d) Suppose n = 3 with observations (x1 , y1 ) = (0, 0), (x2 , y2 ) = (1, 1), (x3 , y3 ) =
(2, 1). Set up the system of score equations for this dataset and compute the first
iteration of the Newton-Raphson method starting from β0 = 0, β1 = 0.

Q6. Consider a logistic regression model with a binary response Y ∈ {0, 1} and two pre-
dictors X1 and X2 . Suppose the data consist of two groups, with n1 observations in
group 1 and n2 observations in group 2.

(a) Write down the logistic regression model for Y in terms of β0 , β1 , β2 .

(b) Derive the likelihood function and the log-likelihood function for this two-group
dataset.
(c) Derive the detailed expression for the z-statistic for testing H0 : βj = 0 (j = 1, 2)
in terms of β̂j and the estimated standard errors.
(d) Discuss the interpretation of the z-statistic in terms of odds ratios for each pre-
dictor.

Q7. Show how to solve the generalized eigenvalue problem

max aT Ba subject to aT W a = 1
a

by transforming it to a standard eigenvalue problem.

Challenge Problems
Q8. Suppose we have features x ∈ Rp , a two-class response, with class sizes N1 , N2 .

2
 (a) Show that the LDA rule classifies to class 2 if

xT Σ̂−1 (µ̂2 − µ̂1 ) > 12 (µ̂2 + µ̂1 )T Σ̂−1 (µ̂2 − µ̂1 ) − log(N2 /N1 ),

and to class 1 otherwise.

(b) Consider minimization of the least squares criterion
N
X
(yi − β0 − xTi β)2 . (4.55)
i=1

Show that the solution β̂ satisfies

 


N − 2 Σ̂ + N Σ̂B β̂ = N (µ̂2 − µ̂1 ) (after simplification),

where
N1 N2
Σ̂B = (µ̂2 − µ̂1 )(µ̂2 − µ̂1 )T . (4.56)
N2
(c) Hence show that
Σ̂B β̂ is in the direction of (µ̂2 − µ̂1 ),
and thus
β̂ ∝ Σ̂−1 (µ̂2 − µ̂1 ). (4.57)
Therefore the least-squares regression coefficient is identical to the LDA coeffi-
cient, up to a scalar multiple.
(d) Show that this result holds for any (distinct) coding of the two classes.
(e) Find the solution β̂0 (up to the same scalar multiple as in (c)), and hence the
predicted value
fˆ(x) = β̂0 + xT β̂.
Consider the following rule: classify to class 2 if fˆ(x) > 0 and to class 1 otherwise.
Show this is not the same as the LDA rule unless the classes have equal numbers
of observations.

Q9. Suppose we transform the original predictors X to Y via linear regression. In detail,
let
Y = X(X T X)−1 X T Y = X B̂,
where Y is the indicator response matrix. Similarly, for any input x ∈ Rp , we get a
transformed vector
ŷ = B̂ T x ∈ RK .
Show that LDA using Ŷ is identical to LDA in the original space.

3
 Coding
Q10. This question should be answered using the Weekly data set, which is part of the ISLP
package. This data contains 1,089 weekly returns for 21 years, from the beginning of
1990 to the end of 2010.

(a) Produce some numerical and graphical summaries of the Weekly data. Do there
appear to be any patterns?
(b) Use the full data set to perform a logistic regression with Direction as the re-
sponse and the five lag variables plus Volume as predictors. Use the summary
function to print the results. Do any of the predictors appear to be statistically
significant? If so, which ones?
(c) Compute the confusion matrix and overall fraction of correct predictions. Explain
what the confusion matrix is telling you about the types of mistakes made by
logistic regression.
(d) Now fit the logistic regression model using a training data period from 1990 to
2008, with Lag2 as the only predictor. Compute the confusion matrix and the
overall fraction of correct predictions for the held-out data (that is, the data from
2009 and 2010).
(e) Repeat (d) using LDA.
(f) Repeat (d) using QDA.
(g) Repeat (d) using naive Bayes.
(h) Which of these methods appears to provide the best results on this data?
(j) Experiment with different combinations of predictors, including possible transfor-
mations and interactions, for each of the methods. Report the variables, method,
and associated confusion matrix that appears to provide the best results on the
held-out data.

Q11. Using the Boston data set, fit classification models in order to predict whether a given
suburb has a crime rate above or below the median. Explore logistic regression, LDA,
QDA and naive Bayes models using various subsets of the predictors. Describe your
findings.