Scribd
Upload
13 views3 pages

LASSO Regression Problem Set Analysis

The document presents a problem set focused on LASSO regression, outlining various exercises related to the estimation and properties of the LASSO estimator under specific assumptions about the data. It includes tasks such as demonstrating properties of normal distributions, inequalities involving estimation errors, and comparisons with ordinary least squares (OLS) regression. The final question prompts a discussion on the conditions under which LASSO may outperform OLS and its implications for the Gauss-Markov Theorem.

Uploaded by

hennah360
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
13 views3 pages

LASSO Regression Problem Set Analysis

The document presents a problem set focused on LASSO regression, outlining various exercises related to the estimation and properties of the LASSO estimator under specific assumptions about the data. It includes tasks such as demonstrating properties of normal distributions, inequalities involving estimation errors, and comparisons with ordinary least squares (OLS) regression. The final question prompts a discussion on the conditions under which LASSO may outperform OLS and its implications for the Gauss-Markov Theorem.

Uploaded by

hennah360
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Problem set: least absolute

shrinkage and selection operator


(LASSO) regression

Emmanuel Guerre
Queen Mary University of London

NB: the potential mistakes in the exercises below are part


of the problem set.

In this problem, (Yi , Xi0 )0 is an i.i.d vector where Yi is of dimension


1, and Xi is deterministic with dimension Kn ≤ n which grows with the
sample size. It is supposed that

Yi = Xi0 β + ui , where ui ∼ N (0, 1) is independent of Xi .

It is assumed that all the entries of β except k are 0. The econometrician


does not know which entries are not equal to 0 but it is supposed for the
sake of simplicity that this is the k ∗ th first one, that is

|β1 | > 0, . . . , |βk∗ | > 0, βk∗ +1 = · · · = βKn = 0. (1)

For the sake of simplicity, it is assumed that ni=1 Xi Xi0 /n is the identity
P
matrix of dimension Kn × Kn .
An econometrician proposes to implement the LASSO estimator
( n Kn
)  1/2
1 X
0 2 2c n
X K n
βb = arg min (Yi − Xi b) + √ |bk | where cn = 2 log .
b∈RKn n i=1 n k=1 log Kn

1. Preliminary question: show that for any b = (b1 , . . . , bKn )0


n Kn
01
X X
b Xi Xi0 b = b2k . (2)
n i=1 k=1

2. Show that the variables Zk = √1n ni=1 Xik ui have a standard


P
normal distribution. Are they independent?

1
3. Show that
Kn
2Kn exp (−c2n /2)
  X
P max |Zk | ≥ cn ≤ P (|Zk | ≥ cn ) ≤ √ → 0.
k∈[1,Kn ]
k=1
c n 2π

R ∞Z has a standard normal distribution, P (|Z| ≥ c) =


Hint: if c > 0 and
2P (Z ≥ c) = √22π c tt exp (−t2 /2) dt. Using that d exp (−t2 /2) /dt =
−t exp (−t2 /2) and that t ≥ c in the integral above, show that P (Z ≥ c) ≥
√1 exp (−c2 /2) /c.
2π

4. Show that
n Kn n Kn
1 X 2 2c X
n 1X 2 2cn X
Yi − Xi0 βb + √ βbk ≤ (Yi − Xi0 β) + √ |βk | .
n i=1 n k=1 n i=1 n k=1

Show that this inequality


2 implies that the in sample estimation error
Pn 0b 0
i=1 Xi β − Xi β /n satisfies

n
1 X  0 b 2
Xi β − β
n i=1
n K n n K n K
2 X 1 X 2cn X 2cn X
≤ √ √ Xik ui βk − βk −
b √ βk +
b √ |βk | .
n k=1 n i=1 n k=1 n k=1


5. Let En = maxk∈[1,Kn ] |Zk | ≤ cn and suppose that En is true. Use
(1) and the last inequality of Question 4 to show that
n
1 X  0 b 2
Xi β − β
n i=1
( k∗ Kn
) ( k∗ Kn
)
2cn X b X 2c n
X  X
≤√ βk − βk + βbk + √ |βk | − βbk − βbk
n k=1 k=k∗ +1
n k=1 k=k∗ +1
k∗
4cn X b
≤√ βk − βk ,
n k=1

and use (2) to get that


k ∗ Kn k∗
X 2 X
2 4cn X b
βk − βk +
b βk ≤ √
b βk − βk .
k=1 ∗
k=k +1
n k=1

2
Hint: recall that ||z1 | − |z2 || ≤ |z1 − z2 | for any real numbers z1 and
z2 .

(6) Using the Cauchy-Schwarz inequality ((z10 z2 )2 ≤ z10 z1 ·z20 z2 ), shows


that
k∗ k∗  2 1/2
!
X √ X
βbk − βk ≤ k ∗ βbk − βk .
k=1 k=1

Show that the last inequality in Question 5 gives, when En is true,

Kn  √ Kn 
!1/2
X 2 4cn k ∗ X 2
βbk − βk ≤ √ βbk − βk .
k=1
n k=1

(7) Show that


n
1 X  0 b 2 16c2n k ∗
Xi β − β ≤ (1 + oP (1)) .
n i=1 n

(8) Suppose that βe is the OLS obtained from regressing Yi on Xi .


Show that the distribution of
n 
X  2
Xi0 βe − β
i=1

is the one of a Chi square with Kn degrees of freedom, with mean Kn


and variance 2Kn .

(9) Show that


n
1 X  0 e 2 Kn
Xi β − β = (1 + oP (1))
n i=1 n

Can you explain when the LASSO improves on the OLS which uses all
the variables? Does this result contradict the Gauss-Markov Theorem,
ie that the OLS estimator is the best linear unbiased one?

You might also like