Quiz-3 Solutions

Question 1
Question Text: ”I collect a set of data (n = 100 observations) containing a single predictor and a quan-
titative response. I then fit a linear regression model to the data, as well as a separate cubic regression,
i.e. Y = β0 + β1 X + β2 X 2 + β3 X 3 + ϵ. Suppose that the true relationship between X and Y is linear, i.e.
Y = β0 + β1 X + ϵ. Consider the training residual sum of squares (RSSL) for the linear regression, and also
the training RSS (RSSC) for the cubic regression. Select one: a. RSSL and RSSC would be the same b.
Insufficient information c RSSL is higher Od. RSSC is higher”

Correct Answer
RSSL is higher

Detailed Solution
This question compares the training residual sum of squares (RSS) for two different models: a simple
linear regression (RSSL) and a more complex cubic regression (RSSC).

1. Model Flexibility: The cubic model (Y = β0 + β1 X + β2 X 2 + β3 X 3 + ϵ) is more flexible than the

linear model (Y = β0 + β1 X + ϵ). The linear model is ”nested” within the cubic one; it’s a special case
of the cubic model where β2 = 0 and β3 = 0.
2. Training Error (RSS): When fitting models to the training data, a more flexible model will always
achieve an RSS that is less than or equal to the RSS of a simpler, nested model. The cubic model has
more ”levers” (coefficients) to pull to fit the 100 training observations as closely as possible, even if the
true relationship is linear.

3. Conclusion: The cubic model will ”overfit” the training data more than the linear model, resulting
in a smaller residual sum of squares. Therefore, the RSS for the cubic model (RSSC) will be lower
than the RSS for the linear model (RSSL). This means RSSL is higher.

Question 2
Question Text: ”for any n ∗ d matrix D and scalar λ > 0: Select one Oa. DT D is always orthogonal Ob.
(λId + D−1 D)−1 DT = DT (λIn + DDT )−1 c (λId + D2 )−1 DT = DT (λIn + DDT )−1 d. (λId + DT D)−1 DT =
DT (λIn + DDT )−1 ”

Correct Answer
(λId + DT D)−1 DT = DT (λIn + DDT )−1

1
Detailed Solution
This question asks to identify a correct matrix identity. The correct choice is a standard and useful identity,
sometimes called the ”push-through” identity, which is related to the Woodbury matrix identity.
We can prove it by multiplying both sides by (λIn + DDT ) on the right:

1. Left-Hand Side (LHS):

(λId + DT D)−1 DT × (λIn + DDT )

= (λId + DT D)−1 (λDT + DT DDT )
= (λId + DT D)−1 (λId DT + DT DDT )
= (λId + DT D)−1 (λId + DT D)DT
= Id DT
= DT

2. Right-Hand Side (RHS):

DT (λIn + DDT )−1 × (λIn + DDT )

= DT In
= DT

Since both sides simplify to DT , the identity is true. This identity is very common in machine learning,
particularly in the derivation of Ridge Regression and Gaussian Processes.

Question 3
Question Text: ”Consider the n ∗ d matrix D. Construct the (n + d) ∗ (n + d) matrix B as follows:

0 DT
 

D 0

Note that the matrix B is square and symmetric. Let q ′ and p′ be the left and right singular vectors of D
with singular value σ. Select one: 0 a. DDT = I Ob. DT p′ = σq ′ OC D is singular d. DT q ′ = σp′ ”

Correct Answer
DT q ′ = σp′ If multicorrect also mark option b
Detailed Solution
This question relates the singular value decomposition (SVD) of a matrix D to the properties of a related
block matrix B.

1. SVD Definitions: By standard definition, if σ is a singular value of D, with a corresponding left

singular vector q ′ and right singular vector p′ , the following two relationships hold:
• Dp′ = σq ′
• DT q ′ = σp′
2. Analyzing the Options:
• a. DDT = I: This is not generally true.

2
 • b. DT p′ = σq ′ : This is not a standard SVD definition.
• c. D is singular: Not necessarily true.
• d. DT q ′ = σp′ : This directly matches the second SVD definition listed above.

Therefore, DT q ′ = σp′ is the correct statement.

Question 4
Question Text: ”Suppose that you add a small amount of noise to each entry of an n∗d matrix D with rank
r << d and n >> d. The noise is drawn from a Gaussian distribution, whose variance λ > 0 is much smaller
than the smallest non-zero singular value of D. The non-zero singular values of D are σ11 , σ22 , ......, σrr The
rank of the modified matrix, D′ is Select one: a. d-n Ob. d+n cd Od. 2d”

Correct Answer
d

Detailed Solution
This question asks for the rank of a matrix D′ which is created by adding a small amount of Gaussian noise
to every entry of an n × d matrix D.

1. Matrix Dimensions: The matrix D is n × d, with n ≫ d.

2. Rank Definition: The rank of a matrix is the number of linearly independent columns (or rows).
The maximum possible rank of an n × d matrix is min(n, d). Since n ≫ d, the maximum possible rank
is d.
3. Original Rank: The original matrix D has rank r ≪ d, meaning it is ”rank-deficient”. Its columns
are not linearly independent.
4. Effect of Noise: When you add a small amount of random noise (drawn from a continuous distri-
bution like a Gaussian) to every element of a matrix, you are perturbing all its entries. This random
perturbation will, with probability 1, break any pre-existing linear dependencies.
5. New Rank: The new matrix D′ will almost certainly have linearly independent columns. Since it is
an n × d matrix, the maximum number of linearly independent columns it can have is d.

Therefore, the rank of the modified matrix D′ will be d.

Question 5
Question Text: ”Consider the unconstrained optimization problem of minimizing the Frobenius norm
||D − U V T ||2F , which is equivalent to SVD. Here, D is an n ∗ d data matrix, U is an nk matrix, and V is a
d ∗ k matrix. Use differential calculus to show that the ontimal solution satisfies the following conditions:
Select one: a. DV = U V −1 V Ob. DV = U T V T V Oc DV = U V V d. DV = U V T V ”

Correct Answer
DV = U V T V

3
Detailed Solution
This question asks for a condition that the optimal solution to the minimization problem ||D − U V T ||2F must
satisfy. This is a matrix factorization problem, and we can find the conditions by using differential calculus.
We can find the optimal U by taking the partial derivative of the objective function L = ||D − U V T ||2F
with respect to U and setting it to zero.

1. Objective Function:
L(U, V ) = ||D − U V T ||2F
∂L ∂
2. Find the Gradient ∂U : We can use the matrix calculus rule: ∂X ||Y − XZ||2F = −2(Y − XZ)Z T .

• Here Y = D, X = U , Z = V T .
• ∂L
∂U = −2(D − U V T )(V T )T
• ∂L
∂U = −2(D − U V T )V
• ∂L
∂U = −2DV + 2U V T V
3. Set Gradient to Zero: To find the minimum, we set the gradient to 0:

−2DV + 2U V T V = 0
2U V T V = 2DV
U V T V = DV

This matches option (d).

Question 6
Question Text: ”Consider the following matrix A whose SVD is given by the following:
 −1
√1

√
2 √ 2   
 √1 2 4 0 1 0
 2  0 2 0 1
1 1 √ √
2 2
!  !
  1  −1
√ √1
 1  −1
√ √1
1 0 4 0 2 2 −1 0 4 0 2 2
Then, the inverse of A is Select one: a. Does not exist b. 1 √1 √1
c 1 √1 √1
0 1 0 2 2 2
0 1 0 2 2 2
Od. I”

Correct Answer
(Note: The PDF shows the correct answer as ’b’ but with a typo from the original matrix. The LaTeX for
the PDF’s choice is below, though it’s mathematically flawed as noted in the solution.)
 −1
√1

  1  √2 √ 2
1 0 4 0  √1 2
0 1 0 12
 2 
1 1 √ √
2 2

4
Detailed Solution
This question asks for the inverse of a matrix A given its SVD.

1. SVD Formula: The SVD is given as A = U ΣV T .

2. Inverse Formula: The inverse of A is A−1 = (U ΣV T )−1 . Using the property (XY Z)−1 = Z −1 Y −1 X −1 ,
this becomes:
A−1 = (V T )−1 Σ−1 U −1

3. Orthogonal Property: In SVD, U and V are orthogonal matrices. This means U −1 = U T and
V −1 = V T . Therefore, (V T )−1 = (V −1 )−1 = V . So, the formula simplifies to: A−1 = V Σ−1 U T .

4. Identifying Components: From the question:

 −1 1 
√ √
2 √2
• U =  √12 2


√1 √1
2 2
 
4 0
• Σ=
0 2
 
1 0
• VT = (the identity matrix I)
0 1
5. Calculating Inverse Components:
 
1 0
• V = (V ) = I = I =
T T T
0 1
 −1  1 
4 0 0
• Σ−1 = = 4
0 2−1 0 12
• U T is the transpose of U .
6. Assembling the Inverse:   1 
1 0 0
A−1 = 4
1 UT
0 1 0 2

Note on Quiz Errors: This question has significant formatting errors in the PDF.
• The U matrix is shown as 3 × 2, which would make A a non-square 3 × 2 matrix, meaning its inverse
does not exist (option a).

• However, the quiz indicates a different correct answer.

• The provided correct answer incorrectly copies the original U matrix at the end, instead of the correct
U T . This is a typo in the quiz’s answer key, but it is the intended answer.

Question 7
Question Text: ”The Frobenius norm of the inverse of a d d square matrix A is equal to Select one: a. the
sum of squared inverses of the eigen values of A. b. the sum of squared inverses of the singular values of A.
c. the sum of squared inverses of the singular values of AT, d. None of the above”

Correct Answer
the sum of squared inverses of the singular values of A.

5
Detailed Solution
This question asks for the definition of the Frobenius norm of a matrix inverse, ||A−1 ||F . This was the one
question marked incorrect on the quiz.

1. Frobenius Norm: The squared Frobenius P P norm of any matrix M , denoted ||M ||2F , is the sum of the
2 2
squares of all its elements (||M ||F = i j Mij ).
2. Norm and Singular Values: A key property of the Frobenius norm is that it is also equal to the
sum of the squares of the matrix’s singular values (σi ).
X
||M ||2F = σi (M )2
i

3. Singular Values of A−1 : If the singular values of the invertible square matrix A are σ1 , σ2 , ..., σd ,
then the singular values of its inverse A−1 are 1/σ1 , 1/σ2 , ..., 1/σd .
4. Putting It Together: Let’s apply the property from step 2 to the matrix A−1 :
• ||A−1 ||2F = i (singular values of A−1 )2i
P

• ||A−1 ||2F = i (1/σi (A))2

P

5. Conclusion: The squared Frobenius norm of A−1 is the ”sum of squared inverses of the singular
values of A.” The question asks for the norm (||A−1 ||F ), but the options are all in the form of a sum,
which corresponds to the squared norm. This implies a slight ambiguity in the question, but option
(b) is the correct corresponding relationship.

• Option (a) is incorrect because this property does not hold for eigenvalues in the general case.
• Option (c) is technically a true statement (as A and AT have the same singular values), but (b) is the
direct and standard relationship.

• The user’s answer (d) ”None of the above” was incorrect because (b) is the correct answer.