See discussions, stats, and author profiles for this publication at: [Link]

net/publication/272002062

Tuning Parameter Selection for the Adaptive Lasso using ERIC

Article  in  Journal of the American Statistical Association · November 2014

DOI: 10.1080/01621459.2014.951444

CITATIONS READS
15 653

3 authors, including:

David I Warton
UNSW Sydney
111 PUBLICATIONS   9,507 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure View project

All content following this page was uploaded by David I Warton on 10 March 2015.

The user has requested enhancement of the downloaded file.

 1 Tuning Parameter Selection for the Adaptive Lasso
2 using ERIC

3 Francis K.C. Hui∗1,2 , David I. Warton1,3 , and Scott D. Foster2,4

1
4 School of Mathematics and Statistics, The University of New South

5 Wales, Sydney, Australia

2
6 CSIRO Computational Informatics, Australia
3
7 Evolution & Ecology Research Centre, The University of New South

8 Wales, Sydney, Australia

4
9 CSIRO Wealth from Oceans Flagship, Australia

10 Abstract

11 The adaptive lasso is a commonly applied penalty for variable selection in

12 regression modeling. Like all penalties though, its performance depends criti-

13 cally on the choice of tuning parameter. One method for choosing the tuning

14 parameter is via information criteria, such as those based on AIC and BIC.

15 However, these criteria were developed for use with unpenalized maximum like-

16 lihood estimators, and it is not clear that they take into account the effects of
∗
Corresponding author: fhui28@[Link]; School of Mathematics and Statistics, The University
of New South Wales, 2052, Sydney, Australia
17 penalization. In this article, we propose the Extended Regularized Information

18 Criterion (ERIC) for choosing the tuning parameter in adaptive lasso regres-

19 sion. ERIC extends the BIC to account for the effect of applying the adaptive

20 lasso on the bias-variance tradeoff. This leads to a criterion whose penalty for

21 model complexity is itself a function of the tuning parameter.

22 We show the tuning parameter chosen by ERIC is selection consistent when

23 the number of variables grows with sample size, and that this consistency holds

24 in a wider range of contexts compared to using BIC to choose the tuning pa-

25 rameter. Simulation show that ERIC can significantly outperform BIC and

26 other information criteria proposed (for choosing the tuning parameter) in se-

27 lecting the true model. For ultra high-dimensional data (p > n), we consider a

28 two-stage approach combining sure independence screening with adaptive lasso

29 regression using ERIC, which is selection consistent and performs strongly in

30 simulation.

31 Keywords: BIC, consistency, high-dimensional data, information criteria,

32 penalized likelihood, regularization parameter, variable selection

33 1 Introduction

34 Penalized likelihood functions are a powerful approach to variable selection in re-

35 gression modeling. One commonly applied penalty is the adaptive lasso, based on a
36 weighted sum of the absolute value of the coefficients (Zou, 2006). Like all penalties
37 though, its performance depends critically on the choice of tuning parameter. This
38 is because only a certain range of the tuning parameter values leads to selection con-
39 sistency i.e., asymptotically identifies the true model. In this article, we focus on
40 using information criteria for tuning parameter selection in adaptive lasso regression.

2
41 Other methods of choosing the tuning parameter include cross validation (Zou, 2006)
42 and selection stability (Sun et al., 2013), for instance.
43 The two most common used criteria for selecting the tuning parameter are those based
44 on AIC (Akaike, 1974) and BIC (Schwarz, 1978). However, both criteria were designed
45 for regression models with unpenalized maximum likelihood estimators, so their use
46 in the penalized likelihood setting somewhat lacks motivation (Bühlmann and van de
47 Geer, 2011). It is not clear that either criterion account for the prior information
48 introduced by the adaptive lasso penalty, and the effect of this penalization on the
49 bias-variance tradeoff.
50 Recently, several modifications to BIC have been proposed, with the aim been to
51 construct a criterion for selecting the tuning parameter that is selection consistent
52 in high-dimensional settings (e.g., Wang et al., 2009; Wang and Zhu, 2011; Fan and
53 Tang, 2012). These BIC-type criteria were formulated by establishing the condi-
54 tions necessary to achieve consistency i.e., considering changes in the criterion for
55 the cases of overfitting (tuning parameter too small) or underfitting (tuning param-
56 eter too large), rather than by directly considering the effects of penalization on the
57 bias-variance tradeoff. To this end, Konishi et al. (2004) formulated a BIC-type cri-
58 terion for choosing smoothing parameters in radial basis function network models, by
59 directly applying Laplace’s approximation on the joint distribution of the likelihood
60 and the basis functions which assumed Gaussian priors (similar to ridge regression,
61 Hoerl and Kennard, 1970). Kawano (2012) adopted the same approach to derive a
62 criterion for choosing the tuning parameter in bridge regression (Frank and Friedman,
63 1993), although the resulting criterion had a rather complicated form and there was
64 no consideration of selection consistency.
65 In this article, we propose the Extended Regularized Information Criterion (ERIC)

3
66 for selecting the tuning parameter in adaptive lasso regression. ERIC is an extension
67 of BIC to account for the effect of applying the adaptive lasso, and in particular the
68 effect of the Laplace prior on coefficients not shrunk to zero. This leads to a criterion
69 whose penalty for model complexity is itself a function of the tuning parameter. For
70 generalized linear models (GLMs, McCullagh and Nelder, 1989) with the adaptive
71 lasso, we show the tuning parameter chosen by ERIC is selection consistent when
72 the number of variables grows at a smaller rate than sample size, and that this
73 consistency holds in a wider range of contexts compared to using BIC to choose the
74 tuning parameter. We use ERIC in conjunction with a hybrid estimator, whereby
75 the final coefficient estimates are based on applying unpenalized maximum likelihood
76 to the selected covariates (Efron et al., 2004). Simulations show that ERIC can
77 significantly outperform BIC and some recently proposed modifications in selecting
78 the true model.
79 We conclude the introduction by reviewing two methods of variable selection that
80 have been recently developed for ultra high-dimensional settings i.e., p > n, and
81 which are outside of the penalized likelihood framework. First, the Extended BIC
82 (EBIC, Chen and Chen, 2008) is a modification of BIC which imposes a prior dis-
83 tribution to account for the increasing complexity of the model space. Although
84 EBIC is used in conjunction with lasso or SCAD (Fan and Li, 2001) penalties for
85 instance, it only uses penalized regression for the purpose of fitting a sequence of
86 models quickly. That is, EBIC is constructed from the unpenalized likelihood at each
87 of the subset models on the regularization path, and it was not originally designed
88 for tuning parameter selection. It has been proven though that EBIC is selection
89 consistent in high-dimensional linear models (Chen and Chen, 2008; Luo and Chen,
90 2013) and GLMs (Chen and Chen, 2012; Luo and Chen, 2011). Also, Kim et al.

4
 91 (2012) considered generalized information criterion, with EBIC as a special case, and
92 established selection consistency under quite general conditions. Second, Sure Inde-
93 pendence Screening (Fan and Lv, 2008; Fan et al., 010b) is based on using marginal
94 statistics for each covariate e.g., marginal likelihood ratio values, to screen out those
95 which are uninformative. For ultra high-dimensional data, we show that a two-stage
96 approach combining SIS and adaptive lasso regression using ERIC is selection con-
97 sistent and performs strongly in simulation.

98 2 The Adaptive Lasso

99 We apply the adaptive lasso to generalized linear models. Let {(xi , yi ); i = 1, . . . , n}

100 be a sample of independent and identically distributed observations, where y is a
101 univariate response and x a p-dimensional covariate vector. The number of covariates,
102 p, is allowed to grow with sample size. The conditional density of yi given xi is
103 assumed to come from the exponential family of distributions,

   
1 1 0 0
f (yi |xi , β, φ) = exp (yi θi − b(θi )) + c(yi , φ) = exp (yi xi β − b(xi β)) + c(yi , φ) ,
φ φ

104 for suitably chosen functions b(·) and c(·, ·), where β = (β1 , . . . , βp ) denotes the
105 regression coefficients, θ = x0 β is the canonical parameter, and E(y|x) = b0 (θ) = µ.
106 We use the canonical link function, g(µ) = θ, with the nuisance parameter φ either
107 known in advance or requiring estimation. Finally, we assume the covariates have
108 been standardized to have mean zero and standard deviation one.
109 Let β̃ be the maximum likelihood estimate (MLE) of β. Note that β̃ is, under gen-
110 eral regularity conditions, well-defined provided p < n. The penalized log-likelihood

5
111 function for the adaptive lasso is given by

p
X
`p (β) = `(y|β) − λ w̃j |βj |, (1)
j=1

n
log f (yi |xi , β, φ), and w̃j = 1/|β̃j |γ with γ > 0 the power pa-
P
112 where `(y|β) =
i=1
113 rameter. The penalized estimates, denoted as β̂λ , are obtained by maximizing (1).
114 In a Bayesian context, the adaptive Lasso can be regarded as imposing independent
115 Laplace priors on the coefficients (Park and Casella, 2008),

λw̃j −λw̃j |βj |/φ

ρ(βj |λ) = e for j = 1, . . . , p, (2)
2φ

116 such that maximizing (1) is equivalent to using the posterior mode estimates. Note
117 the scaling by 1/φ in (2) is necessary to cancel out the 1/φ present in the exponential
118 family – this ensures the nuisance parameter does not enter into the calculation of
119 β̂λ . Besides, in penalized likelihood analysis one usually estimates φ separately after
120 the penalized coefficients have been estimated. For example in linear models with
121 unknown φ ≡ σ 2 , the variance is estimated afterwards as σ̂λ2 = (1/n)ky − x0 β̂λ k2
122 (Wang et al., 2007).
123 The adaptive lasso introduces prior information which purposefully biases the MLEs,
124 such that some of the unpenalized estimates are shifted to zero. Given an appropriate
125 choice of λ, only the truly zero coefficients are asymptotically shifted to zero, leading
126 to selection consistency. The penalization also reduces the variance of the mean
127 response – for any fixed sample size, larger values of λ lead to greater bias and less
128 variability in predicted values.

6
129 3 The Extended Regularized Information Crite-

130 rion

131 One approach for choosing the tuning parameter, λ, is via information criteria, with
132 the most commonly used based on BIC (Wang et al., 2007; Zhang et al., 2010),

BIC(λ) = −2`(y|β̂λ ) + |αλ | log(n),

133 where αλ = {j : β̂λ,j 6= 0} is the active set excluding the intercept, and |αλ | is
134 its size. It has been shown that BIC(λ) is selection consistent in the case of fixed p
135 (Zhang et al., 2010). Furthermore, in Section 4 we show BIC(λ) is consistent provided
136 p/nκ → 0 with κ < 1/2. As with all information criteria, BIC(λ) can be interpreted
137 in terms of a bias-variance tradeoff – the first term measures bias or goodness-of-fit,
138 and the second term measures variance of the predicted values (Hastie et al., 2009).
139 As more covariates are removed from the model, the goodness-of-fit term increases as
140 we move away from the MLEs of the full model, while the variance penalty decreases
141 due to reduced model complexity.
142 The BIC was designed for model selection with unpenalized maximum likelihood es-
143 timators (Schwarz, 1978). It is based on an asymptotic approximation of the model
144 likelihood, and is derived by applying Laplace’s approximation and assuming the
145 uninformative priors on all coefficients, ρ(βj ) = O(1); j = 1, . . . , p. Since BIC(λ)
146 directly adapts the form of BIC to the penalized likelihood setting for choosing λ, it
147 is not clear that the criterion takes into account the effect of applying the adaptive
148 lasso on the bias-variance tradeoff. This is most clearly seen if we consider an interval
149 [λa , λb ], with λa < λb , for which |αλ | remains unchanged. As we increase λ within

7
150 this interval, the bias of the adaptive lasso estimates not shrunk to zero is increased,
151 and this is captured by a corresponding increase in −2`(y|β̂λ ). At the same time
152 however, the variance of these adaptive lasso estimates ( and hence the variance of
153 the predicted response) decreases, yet this is not accounted for since |αλ | log(n) re-
154 mains unchanged. The decrease in variance of the non-zero estimates is a result of
155 the increased information from the Laplace prior, as captured by the normalization
156 constant in equation (2), λw̃j /(2φ). This suggests that the variance penalty in BIC(λ)
157 should be modified accordingly to |αλ | log(n)−|αλ | log(λ/φ) = |αλ | log(nφ/λ). There-
158 fore, we propose the following Extended Regularized Information Criterion (ERIC)
159 for tuning parameter selection in adaptive lasso GLMs,

ERICν (λ) = −2`(y|β̂λ ) + 2ν|αλ | log(nφ/λ). (3)

160 From (3), ERICν (λ) adjusts the penalty for model complexity in BIC(λ) by accounting
161 for the effect of prior information introduced by the adaptive lasso. Coefficients
162 shrunk to zero contribute towards the bias term but not the variance term in (3). On
163 the other hand, for any fixed sample size, coefficients not shrunk to zero have less
164 variance than if they were estimated with no penalty i.e., their variance is less than
165 the the variance of the MLEs calculated from the sub-model αλ . Therefore ERICν (λ)
166 accounts for this by reducing the variance penalty to |αλ | log(nφ/λ). Large values
167 of λ incur greater shrinkage and a smoother (less variable) mean response, so the
168 variance penalty in ERICν (λ) decreases to reflect this. The additional parameter,
169 ν > 0, provides flexibility to control the severity of penalization (similar to EBIC,
170 Chen and Chen, 2008). We choose between ν = 0.5 or 1, with the former better
171 suited to high-dimensional data.

8
172 It is important to emphasize that the way ERICν (λ) penalizes for model complexity is
173 fundamentally different to BIC(λ) and its modifications proposed for high-dimensional
174 settings – for a fixed sample size, BIC(λ) penalizes a constant value for every new
175 covariate entered into the model. In contrast, ERICν (λ) has a ‘dynamic’ variance
176 penalty that depends on λ itself, meaning it also depends on how complex the model
177 is already.
178 On the surface then, given the reduced variance penalty, it appears ERICν (λ) has a
179 tendency to select larger models compared to BIC(λ). However, it turns the opposite
180 actually occurs – when we consider the relative values of a criteria, which are of more
181 importance in model selection, it turns out that ERICν (λ) penalizes more severely
182 for overfitting than BIC(λ). This point is detailed at the end of Section 4.
183 Regarding large sample behavior, the form of ERICν (λ) assumes λ grows with sample
184 size, since the effect of penalization on the bias-variance tradeoff is non-trivial in such
185 case. Note that λ must grow with sample size anyway to achieve selection consistency
186 (Zou, 2006; Zou and Zhang, 2009). On the other hand, if λ = O(1) and ν = 0.5 then
187 log(λ/φ) = O(1) and β̂λ tends to the maximum likelihood estimates for the sub-
188 model αλ , in probability. In other words, if adaptive lasso penalty is asymptotically
189 negligible, then ERICν (λ) reduces to the BIC with unpenalized MLEs.
190 As is typically done for linear models where φ ≡ σ 2 is unknown, we estimate the
191 variance term in ERICν (λ) as σ̂λ2 = (1/n)ky − x0 β̂λ k2 .

192 4 Selection Consistency

193 In this section, we show that ERICν (λ) is selection consistent for adaptive lasso GLMs
194 where p grows at a smaller rate than sample size i.e., p/nκ → 0 and κ < 1. This

9
195 is often referred to literature as a diverging number of parameters setting, after Fan
196 and Peng (2004). We use pn to reflect this, although for other quantities e.g., αλ , the
197 subscript is suppressed for clarity of notation. Let β 0 = (β10 , . . . , βp0 ) denote the true
198 parameter values, with the true model identified by α0 = {j : βj0 6= 0} and p0 = |α0 |.
199 Assume the following regularity conditions are satisfied.

200 (A1) The range of tuning parameters considered lies in the interval λ ∈ [λmin , λmax ],
201 where λmax /n → 0.

202 (A2) limn→∞ log(pn )/ log(n) = κ, where κ ∈ [0, 1).

203 (A3) For all i = 1, . . . , n, j = 1, . . . , pn in the model matrix x, |xij | = O(1). Further-
204 more, for any model α identified in the interval [λmin , λmax ], Σα = (1/n)x0α xα
205 has minimum and maximum eigenvalues bounded below and above by c1 ad c2
206 respectively, where 0 < c1 < c2 < ∞.
p
207 (A4) limn→∞ n/(p0 log(n)) minj∈α0 {|βj0 |} → ∞.
p √
208 (A5) (a) λ p0 /n → 0; (b) λ(n/pn )γ/2 / npn → ∞.

209 Condition (A1) ensures the maximum tuning parameter considered does not grow at a
210 faster rate than the sample size. Moreover, for each model α identified in [λmin , λmax ],
211 there exists a unique population parameter to which the unpenalized MLEs β̃α con-
212 verges to in probability. If α ⊃ α0 , then this population parameter will coincide with
213 β 0 , but if α 6⊃ α0 then this will not be the case. Condition (A2) permits the number
214 of covariates to diverge with sample size, with the same rate of divergence as in Wang
215 et al. (2009); Zou and Zhang (2009). Along with a mild regularity condition on the
216 likelihood function (see Supplementary Material A), Condition (A3) ensure the Fisher
217 information matrix exists and is non-singular for each n. Note that under Conditions

10
 p
218 (A2) and (A3), the MLEs for the full model are well-defined, n/pn -consistent, and
219 thus weights w̃j = 1/|β̃j |γ can be calculated for use in the adaptive lasso.
220 The size of the true model, p0 , is permitted to grow with sample size, provided it
221 satisfies Conditions (A4) and (A5) (and obviously Condition (A2)). Condition (A4)
222 is similar to one seen in Wang et al. (2009), and permits the truly non-zero coefficients
223 to tend to zero. In other words, it allows competing underfitted models to get close
224 to true model at a certain rate. When the number of covariates is fixed, Condition
√
225 (A5a) simplifies to λ/ n → 0 and λn(γ−1)/2 → ∞, which is the condition necessary
226 for the adaptive lasso to achieve consistency in the fixed p setting (Zou, 2006). On
227 the other hand, when the number of covariates diverges according to Condition (A2),
228 we have the following result,

229 Lemma 1. Assume Conditions (A1)-(A5) are satisfied, then the adaptive lasso esti-
230 mates β̂λ must satisfy:
p
231 • Estimation consistency: kβ̂λ − β 0 k = Op ( pn /n).

232 • Selection consistency: P (β̂λαc0 = 0) → 1.

233 All proofs may be found in Supplementary Material A. Lemma 1 is a generalization

234 of the consistency result for the adaptive elastic net (Zou and Zhang, 2009) to non-
235 Gaussian responses and GLMs.
236 To study the asymptotic behavior of ERICν (λ), we first define a proxy version of the
237 criterion as follows,

ERIC∗ν (λ) = −2`(y|β̃αλ ) + 2ν|αλ | log(nφ/λ).

11
238 where β̃αλ is the MLE under model αλ . For linear models, we estimate the variance
239 as σ̃α2 λ = (1/n)ky − x0αλ β̃αλ k2 where xαλ is the model matrix for αλ . By examining
240 the behavior of proxy ERIC∗ν (λ) for the cases of underfitted and overfitted models,
241 we can infer what happens to ERICν (λ) in the large sample limit. We partition the
242 interval [λmin , λmax ] into three segments, analogous to Zhang et al. (2010),

Underfitted Models: Λ− = {λ : αλ 6⊃ α0 }

True Model: Λ0 = {λ : αλ = α0 }

Overfitted Models: Λ+ = {λ : αλ ! α0 },

243 The three partitions are closely related to Condition (A5). For λ ∈ Λ0 , both Condi-
244 tions (A5a) and (A5b) must be satisfied. For λ ∈ Λ− , Condition (A5b) is satisfied
245 but not (A5a) i.e., λ is growing too quickly and removing one or more truly non-zero
246 coefficients from the model. For λ ∈ Λ+ , Condition (A5a) is satisfied but not (A5b)
247 i.e., λ is growing too slowly to achieve the correct amount of sparsity. Using proxy
248 ERIC∗ν (λ) and the partitions above, we have the following lemma,

249 Lemma 2. Assume conditions (A1)-(A4) are satisfied, and that there exists a λ0 ∈ Λ0
250 satisfying condition (A5). Then,

(a) P inf λ∈Λ− minαλ 6⊃α0 ERIC∗ν (λ) > ERIC∗ν (λ0 ) → 1.


251

252 (b) If γ ≥ 2κ/(ν(1 − κ)) − 1 also, then

P inf λ∈Λ+ minαλ !α0 ERIC∗ν (λ) > ERIC∗ν (λ0 ) → 1.


253

254 From parts (a) and (b) of Lemma 2, we can not asymptotically choose a λ that
255 identifies an overfitted or underfitted model respectively, since we could always select
256 λ0 to produce a lower value of proxy ERIC∗ν (λ). In Lemma 2b, the condition on γ

12
257 reduces to a simple inequality for the two cases of ν = 0.5 (γ ≥ (5κ − 1)/(1 − κ)) and
258 ν = 1 (γ ≥ (3κ − 1)/(1 − κ)). In simulations, choosing γ is straightforward since κ is
259 known. In real applications, we suggest trying several values of γ e.g., γ = 1, 2, 4, 6,
260 while taking into account the dimensionality of the dataset at hand. Since λ0 satisfies
261 Condition (A5), then based on Lemmas 1 and 2 we obtain the following result,

262 Theorem 1. Let λ̂ be the tuning parameter chosen by minimizing ERICν (λ). Under
263 conditions (A1)-(A5), then P (αλ̂ = α0 ) → 1.

264 If the true model is contained within the set of candidate models, then Theorem
265 1 guarantees it is selected by ERICν (λ). Note that for fixed p, the same property
266 is shared by BIC(λ) (Zhang et al., 2010, Theorem 1B). However, if the number of
267 covariates increases with sample size, then BIC(λ) may be selection inconsistent as
268 formalized below,

269 Corollary 1. Let λ̂ be the tuning parameter chosen by minimizing BIC(λ). If κ > 0.5
270 in Condition (A2), then P (αλ̂ = α0 ) 6→ 1.

271 The corollary above extends the result shown in Chen and Chen (2008) to the pe-
272 nalized likelihood setting. It also implies that ERICν (λ) is selection consistent for a
273 wider range of settings compared to BIC(λ).
274 To further understand how ERICν (λ) and BIC(λ) penalize differently for models that
275 lie apart on the interval [λmin , λmax ], consider the ratio in variance penalties (RVP)
276 between an incorrect model with λ ∈ Λ+ ∪ Λ− and a correct model with λ0 ,

|αλ | |αλ | log(n/λ)

RVPBIC = ; RVPERICν =
p0 p0 log(n/λ0 )

277 where for simplicity we have assumed φ = 1. For BIC(λ), and more generally for
278 all criteria with fixed variance penalties, the RVP depends solely on the sizes of

13
279 the true and overfitted models. In contrast, the dynamic variance penalty means
280 RVPERICν > RVPBIC when λ < λ0 , implying that ERICν (λ) penalizes more severely
281 than BIC(λ) and its modifications for overfitted models. Put another way, ERICν (λ)
282 is more ‘aggressive’ at shrinking coefficients to zero. Conversely, RVPERICν < RVPBIC
283 for underfitted models, meaning that this aggressive shrinkage comes at the risk of
284 underfitting. In Section 5, we shall see empirically that the dynamic variance penalty
285 of ERICν (λ) can lead to substantial improvements in variable selection.

286 4.1 Hybrid Estimation Approach

287 While ERICν (λ) performs strongly in variable selection, it does have a tendency to
288 over-shrink correctly selected coefficients. This is not surprising given its dynamic
289 variance penalty makes it a more aggressive criterion in terms of shrinkage and se-
290 lection, as discussed in the previous section. On the other hand, BIC(λ) and its
291 modifications are more conservative criteria in that, for any active set αλ , they al-
292 ways choose the smallest possible λ. This is illustrated in Figure 1 where, for datasets
293 whereby both ERICν (λ) and BIC(λ) select the true model, almost all tuning param-
294 eters chosen by ERICν (λ) lie above the median tuning parameter value chosen by
295 BIC(λ). Note however that the variability in λ values chosen by ERICν (λ) is sig-
296 nificantly less compared to BIC(λ). While some shrinkage is useful for improving
297 predictive accuracy citephoerl70, over shrinkage of correctly selected covariates can
298 be detrimental to prediction since it introduces an excessive amount of bias.
299 To resolve this problem, we adopt the hybrid estimation approach of Efron et al.
300 (2004), and use the unpenalized MLEs of the selected covariates as the final estimates.
301 That is, we use ERICν (λ) to select the model but not to estimate the coefficients.
302 Several authors have suggested this approach in various applications of penalized

14
 Figure 1: Boxplots of λ values chosen by BIC(λ) and ERIC1 (λ), in datasets where
both criteria selected the true model. The boxplots are based on datasets of with
n = 200 simulated from Model 2 (see Section 5.2).

0.04 ●

Values of λ chosen
0.03 ●
●
●
●
●
●
●
●

0.02

0.01

0.00
BIC ERIC

303 likelihood methods (e.g., Meier et al., 2008; Schelldorfer et al., 2013). Since the
304 motivation behind ERICν (λ) was to develop an information criterion for consistent
305 variable selection, as opposed to efficient prediction, then it seems sensible to use
306 maximum likelihood estimation to obtain asymptotically unbiased final estimates.
307 The use of a hybrid estimator with ERICν (λ) also echoes the underlying aim of the
308 adaptive lasso, namely to achieve the oracle property i.e., asymptotically unbiased
309 and as efficient as the MLEs from the true model (Zou, 2006).

310 5 Simulation Study

311 We conducted a simulation study to compare the performance of ERICν (λ), with
312 ν = 0.5 and 1, against several BIC-type criteria used to select λ in adaptive lasso
313 GLMs. These included BIC(λ), BICwll (λ) = −2`(y|β̂λ ) + |αλ | log(n) log(log(pn ))

15
314 (Wang et al., 2009), BICft (λ) = −2`(y|β̂λ ) + |αλ | log(log(n)) log(pn ) (Fan and Tang,
315 2012), and BICwz (λ) = −2`(y|β̂λ ) + |αλ | log(pn ) (Wang and Zhu, 2011). We also
316 included 10-fold cross validation (CV) as a method of selecting λ, as this is the
317 standard choice in many software packages for performing adaptive lasso regression
318 (e.g., the glmnet package, Friedman et al., 2010).
319 Note that all information criteria except for ERICν (λ) have a fixed variance penalty.
320 That is, the same amount is penalized for each covariate included in the model. In
321 other words, for any model αλ , these criteria will always choose the smallest possible
322 λ attaining that model. Also, note these criteria and CV use the penalized estimator
323 based on maximizing (1), whereas we apply ERICν (λ) with the hybrid estimator (see
324 Section 4.1).
325 We generated 1000 datasets for each simulation model, and assessed performance by
326 looking at the proportion of true models selected, the false positive and false negative
327 rates (which are presented in the Supplementary Material B), and the Kullback-
328 Leibler (KL) distance between the selected and true models based on a validation
329 dataset of n = 5, 000 observations. A smaller KL distance implies better predictive
330 performance.

331 5.1 Model 1

332 We used a similar setup to Wang et al. (2009), considering a linear model with
333 pn = b7n1/4 c where b·c denotes the floor function. Covariates {xi ; i = 1, . . . , n} were
334 generated independently from an AR1 multivariate Gaussian distribution with ρ =
335 0.5. The first bpn /3c coefficients were generated randomly from a uniform distribution
336 on [0.5, 1.5], and the remaining coefficients set to zero. We considered sample sizes
337 n = 100, 200, 400, 800, 1600 and error variances σ 2 = 4, 16, and used γ = 1 in the

16
338 adaptive lasso weights for both ν = 0.5 and 1.
339 Both ERICν (λ) with ν = 0.5 and 1 significantly outperformed the other criteria and
340 10-fold CV in terms of variable selection and predictive performance, irrespective
341 of n and σ 2 (Table 1). The improvement was driven largely by a reduction in the
342 false positive rates for ERICν (λ), with little compromise in the false negative rates
343 (see Supplementary Material B). This is consistent with the discussion at the end
344 of Section 4 regarding the tendency for ERICν (λ) to choose larger values of λ and
345 thus smaller models compared to the BIC-type criteria. Of the four BIC-type criteria
346 tested, BICwll (λ) was the best performer, although this was somewhat expected given
347 our simulation design was based on Wang et al. (2009), from which the criteria origi-
348 nates. Its performance though was behind ERIC0.5 (λ) and ERIC1 (λ) both in variable
349 selection and prediction. Finally, 10-fold CV performed very poorly with regards
350 to selecting the true model, although its predictive performance was similar to the
351 BIC-type criteria. This result was not surprising – in the fixed p setting at least, it
352 has been shown known that using cross-validation to choose λ is asymptotically loss
353 efficient rather than consistent (Wang et al., 2007; Zhang et al., 2010).

354 5.2 Model 2

355 We considered a logistic regression model with the rate of divergence of pn and covari-
356 ates generated in the same manner as Model 1. The first five coefficients were equal
357 to (−3, 1.5, 0, 0, 2). Afterward, every fifth coefficient took alternating values of ±2
358 while the rest were set equal to zero. An unpenalized intercept of 1 was also included
359 in the true model. We used γ = 1 for the adaptive lasso weights.
360 ERICν (λ) performed best in this setting, with improvements in both the proportion of
361 correct models and KL distance over all four BIC-type criteria as sample size increased

17
 Table 1: Proportion of true models selected and KL distance (×10, in parentheses)
for Model 1.
σ2 = 4
n ERIC0.5 (λ) ERIC1 (λ) BIC(λ) BICwll (λ) BICft (λ) BICwz (λ) CV
100 0.70(0.76) 0.86(0.66) 0.55(1.07) 0.58(1.07) 0.55(1.07) 0.36(1.09) 0.24(1.14)
200 0.92(0.39) 0.99(0.36) 0.78(0.55) 0.83(0.56) 0.78(0.55) 0.55(0.57) 0.32(0.62)
400 0.97(0.25) 1(0.24) 0.85(0.33) 0.90(0.33) 0.86(0.33) 0.66(0.33) 0.36(0.35)
800 0.99(0.15) 1(0.14) 0.90(0.23) 0.94(0.23) 0.90(0.23) 0.82(0.21) 0.34(0.21)
1600 0.99(0.08) 1(0.08) 0.95(0.08) 0.99(0.08) 0.97(0.08) 0.90(0.08) 0.36(0.10)

σ 2 = 16
n ERIC0.5 (λ) ERIC1 (λ) BIC(λ) BICwll (λ) BICft (λ) BICwz (λ) CV
100 0.21(1.34) 0.26(1.31) 0.16(1.47) 0.17(1.49) 0.16(1.47) 0.10(1.45) 0.04(1.51)
200 0.35(0.66) 0.40(0.68) 0.27(0.92) 0.31(0.96) 0.27(0.92) 0.14(0.83) 0.09(0.82)
400 0.72(0.27) 0.86(0.24) 0.54(0.40) 0.61(0.41) 0.55(0.41) 0.34(0.39) 0.15(0.39)
800 0.93(0.19) 0.94(0.21) 0.78(0.28) 0.85(0.31) 0.79(0.28) 0.54(0.25) 0.29(0.25)
1600 0.99(0.09) 0.99(0.09) 0.90(0.17) 0.97(0.19) 0.90(0.18) 0.85(0.14) 0.28(0.14)

362 (Table 2). As in Model 1, the improvement could be attributed to a reduction in the
363 false positive rates for ERICν (λ), resulting in less overfitting compared to the BIC-
364 type criteria (see Supplementary Material B). Using ν = 1 generally lead stronger
365 performance compared to ν = 0.5, although the KL distance for ERIC1 (λ) was larger
366 than ERIC0.5 (λ) at n = 100. This was due to ERIC1 (λ) being overly aggressive in
367 shrinking coefficients to zero, and thus underfitting more compared to ERIC0.5 (λ).
368 Both BICwz (λ) and 10-fold CV consistently overfitted for the five sample sizes tested
369 (see also the higher false positive rates in Supplementary Material B), although this
370 overfitting did lead to lower KL distance compared to the other methods.

18
 Table 2: Proportion of true models selected and KL distance (×10, in parentheses)
for Model 2.
n ERIC0.5 (λ) ERIC1 (λ) BIC(λ) BICwll (λ) BICft (λ) BICwz (λ) CV
100 0.22(0.71) 0.30(0.76) 0.16(0.82) 0.19(0.86) 0.16(0.83) 0.09(0.77) 0.09(0.68)
200 0.28(0.40) 0.59(0.30) 0.20(0.40) 0.28(0.42) 0.22(0.41) 0.08(0.39) 0.06(0.33)
400 0.39(0.23) 0.70(0.13) 0.29(0.21) 0.36(0.22) 0.30(0.21) 0.10(0.19) 0.08(0.15)
800 0.45(0.15) 0.80(0.07) 0.32(0.13) 0.45(0.15) 0.34(0.13) 0.11(0.11) 0.09(0.09)
1600 0.59(0.05) 0.87(0.04) 0.49(0.05) 0.59(0.05) 0.50(0.05) 0.24(0.04) 0.08(0.04)

371 5.3 Model 3

372 We considered a linear model with pn = d4n1/2 e−5 and σ 2 = 1 and 16. The covariates
373 and coefficients were generated in the same manner as Model 2, although the rate of
374 divergence is now greater. An unpenalized intercept of -1 was also included in the
375 model. We used γ = 3 in the adaptive lasso weights to ensure selection consistency
376 for both ν = 0.5 and 1. Results where similar to those obtained for Models 1 and
377 2, with ERIC0.5 (λ) and ERIC1 (λ) outperforming the four BIC-type criteria and 10-
378 fold CV in model selection and prediction (Table 3). This is especially the case at
379 σ 2 = 16, where ERICν (λ) had significantly higher proportions of true model selected
380 and lower KL distance for n = 200, 400, 800. Again, these improvements were driven
381 by a significant reduction in the false positive rate at the expense of a very small rise
382 in the false negative rate (see Supplementary Material B).

383 5.4 Additional Simulations

384 In the results above from the three models, one might wonder if the improvements
385 in prediction are mostly a result of using a hybrid estimator in conjunction with
386 ERICν (λ). To test this, we also assessed predictive performance using the hybrid
387 estimator in conjunction with all four BIC-type criteria considered above. Results

19
 Table 3: Proportion of true models selected and KL distance (×10, in parentheses)
for Model 3.
σ2 = 1
n ERIC0.5 (λ) ERIC1 (λ) BIC(λ) BICwll (λ) BICft (λ) BICwz (λ) CV
100 0.99(0.95) 1(0.95) 0.56(1.11) 0.86(1.03) 0.82(1.06) 0.54(1.29) 0.25(1.70)
200 1(0.58) 1(0.58) 0.74(0.63) 0.92(0.61) 0.89(0.61) 0.65(0.69) 0.14(1.06)
400 1(0.41) 1(0.41) 0.92(0.51) 0.98(0.41) 0.96(0.41) 0.82(0.43) 0.10(0.73)
800 1(0.27) 1(0.27) 0.99(0.27) 0.99(0.27) 0.99(0.27) 0.96(0.27) 0.05(0.47)
1600 1(0.21) 1(0.21) 0.99(0.21) 1(0.21) 1(0.21) 0.98(0.21) 0.05(0.34)

σ 2 = 16
n ERIC0.5 (λ) ERIC1 (λ) BIC(λ) BICwll (λ) BICft (λ) BICwz (λ) CV
100 0.18(1.89) 0.20(2.01) 0.13(1.98) 0.18(1.98) 0.17(1.97) 0.10(2.03) 0.04(2.12)
200 0.60(0.73) 0.64(0.71) 0.35(0.97) 0.47(1.00) 0.43(0.97) 0.27(0.98) 0.08(1.25)
400 0.96(0.43) 0.94(0.43) 0.71(0.52) 0.84(0.52) 0.80(0.52) 0.53(0.54) 0.08(0.81)
800 0.99(0.28) 0.99(0.28) 0.87(0.31) 0.94(0.31) 0.93(0.31) 0.81(0.31) 0.04(0.51)
1600 1(0.21) 1(0.21) 0.98(0.22) 0.99(0.22) 0.99(0.22) 0.92(0.22) 0.02(0.36)

388 presented in the Supplementary Material C show that, while the differences in KL
389 distance between the various criteria were smaller when the hybrid estimator was
390 used for all criteria, using the hybrid estimator in conjunction with ERICν (λ) still
391 produced the lowest or equal lowest KL distances in all settings. This suggested that
392 the improvement in prediction made by ERICν (λ) was a result of both the use of a
393 hybrid estimator and its improvement in variable selection (due to better choice of
394 tuning parameter).
395 As pointed out by a reviewer, given ERICν (λ) tends to choose larger tuning param-
396 eters compared to BIC-type criteria, one might hypothesize its performance could
397 deteriorate when the true model is not as sparse. To test this, we performed an addi-
398 tional simulation with modified Models 1 and 2 to include more non-zero coefficients.
399 Specifically, we considered a modified Model 1 where the first b2pn /3c coefficients

20
400 were non-zero, and a modified Model 2 where every third coefficient took alternat-
401 ing values of ±2. Results are presented in Supplementary Material D, and show
402 that ERICν (λ) can still outperform BIC-type criteria and 10-fold CV even if the true
403 model is relatively large i.e., over 50% of the coefficients are truly non-zero in the case
404 of modified Model 1. ERICν (λ) continued to have the lowest false positive rates, but
405 not surprisingly a larger true model also meant it had slightly higher false negative
406 rates at the smaller sample sizes compared to the other methods.
407 Finally, while the theory and simulations above were focused towards cases where p
408 grows with n, we also performed simulations for fixed p settings. Results presented in
409 Supplementary Material E again showed the strong performance ERICν (λ), whereas
410 the three BIC-type criteria tailored made for high-dimensional settings performed
411 worse than BIC(λ) when the number of covariates was fixed.

412 6 Ultra High-Dimensional Data

413 For settings where p > n, often referred in the literature as ultra high-dimensional
414 data (Fan and Lv, 010a), adaptive lasso weights based on the MLEs of the full model
415 cannot be calculated. To overcome this problem, we adapt the two stage approach
416 of Zou (2006); Wang and Zhu (2011), and apply Sure Independence Screening (SIS,
417 Fan and Lv, 2008; Fan et al., 010b) at a first stage to reduce the the dimensionality
418 of the problem from p to dn = O(nκ ) where κ ∈ (0, 1). Afterward, adaptive Lasso
419 GLMs with ERICν (λ) can be applied on the dn selected covariates. This two-stage
420 approach, which we call SIS-ERICν (λ), is selection consistent. Let αλ̂ be the model
421 selected from SIS-ERICν (λ). Then we have the following result,

422 Theorem 2. Suppose the conditions in Theorem 8 of Fan et al. (010b) hold. Under

21
423 conditions (A1)-(A5), then P (αλ̂ = α0 ) → 1.

424 The above result follows the sure independence screening property of Fan et al. (010b)
425 i.e., the remaining dn covariates will contain the important covariates with probability
426 tending to one exponentially fast, and Theorem 1 in Section 4. Therefore, the proof
427 is omitted. The result is similar to Theorem 5.1 in Zou and Zhang (2009), with
428 two key differences. First, the result in Zou and Zhang (2009) assumed the tuning
429 parameter can be chosen appropriately for adaptive elastic net, whereas we focus
430 on how to actually choose the tuning parameter via ERICν (λ). Second, the theory
431 of Zou and Zhang (2009) was developed for normal responses, whereas we consider
432 GLMs. Theorem 2 is also similar to the result in Wang and Zhu (2011) where SIS
433 was combined with the adaptive elastic net, and the tuning parameters chosen using
434 BICwz (λ), although the development there was restricted to linear models.
435 We conducted a small simulation study comparing the performance of SIS-ERICν (λ)
436 to combining SIS with the BIC-type criteria introduced in Section 5. Two models
437 were considered. The first was a logistic regression model with n = 80, 300, 500 and
438 p = 2000. Covariates {xi ; i = 1, . . . , n} were generated from an AR1 multivariate
439 Gaussian distribution with ρ = 0.6. The first twelve coefficients took alternating
440 values of ±3, with the remaining coefficients set to zero. We used SIS to reduce
441 the number of covariates from p = 2000 to dn = b3n0.5 c, and then applied the
442 adaptive lasso logistic regression with γ = 3 in the weights. 1000 datasets were
443 generated for each sample size. The second model considered was a linear model with
444 n = 80, 300, 500 and p = 5000. The covariates, coefficients, the choice of dn were the
445 same as in Model 1 immediately above. Given the high-dimensionality of the problem
446 in both models, we assessed performance using the mean positive selection rate (PSR)
447 and false discovery rate (FDR), as in done in Chen and Chen (2008, 2012).

22
 Table 4: Postive selection rates/false discovery rates for the ultra high-dimensional
data simulations. Results are presented to three decimal places to better differentiate
between the methods.
n ERIC0.5 (λ) ERIC1 (λ) BIC(λ) BICwll (λ) BICft (λ) BICwz (λ)
Model 1 (logistic regression)
80 0.136/0.861 0.131/0.853 0.143/0.870 0.139/0.869 0.141/0.869 0.144/0.870
300 0.385/0.670 0.348/0.405 0.427/0.712 0.380/0.525 0.403/0.623 0.442/0.756
500 0.630/0.224 0.608/0.062 0.635/0.290 0.620/0.156 0.626/0.210 0.645/0.437

Model 2 (linear models)

80 0.159/0.843 0.147/0.783 0.163/0.861 0.157/0.857 0.159/0.858 0.168/0.867
300 0.502/0.455 0.485/0.145 0.538/0.688 0.500/0.420 0.519/0.557 0.553/0.763
500 0.758/0.044 0.748/0.002 0.778/0.284 0.759/0.065 0.763/0.106 0.787/0.505

448 In both models, SIS-ERICν (λ) had slightly lower PSRs compared to combining SIS
449 with the four BIC-type criteria i.e., it was marginally less successful at detecting
450 true non-zeros. On the other hand, SIS-ERICν (λ) (with ν = 1 in particular) had
451 significantly smaller FDRs i.e., it was substantially better at removing true zeros
452 from the model, meaning it tended choose smaller models compared to the BIC-type
453 criteria. Both these result again underline the aggressive nature of ERICν (λ) – it
454 applies greater shrinkage to dramatically reduce the number of false positives, at the
455 risk of missing some truly informative coefficients. The PSRs of all four BIC-type
456 criteria (when used in conjunction with SIS) were similar regardless of sample size.
457 BICwll (λ) and BICft (λ) i.e., the two BIC-type criteria which penalize more severely
458 for model complexity, had lower FDRs than BIC(λ) and BICwz (λ).

23
459 7 Discussion

460 Like all penalized likelihood methods, choosing an appropriate tuning parameter for
461 adaptive lasso regression is critical to ensuring its good performance. We have pro-
462 posed a new information criterion, ERICν (λ), which extends BIC to account for the
463 effect of applying the adaptive lasso on the bias-variance tradeoff. We showed that
464 ERICν (λ) is selection consistent when the number of covariates increases with sample
465 size, and that this consistency holds in a wider range of cases compared to BIC(λ).
466 Simulations showed ERICν (λ), in conjunction with a hybrid estimator, can dramat-
467 ically outperform several BIC-type criteria for choosing the tuning parameter. For
468 ultra high-dimensional data, we proposed combining ERICν (λ) with the SIS proce-
469 dure, leading to a two-stage approach that is selection consistent and has competitive
470 empirical performance.
471 While it was proposed as extension of BIC to adaptive lasso regression only, an obvious
472 question is whether ERICν (λ) (or some appropriately modified form of it) could
473 be applied to other penalties that are selection consistent under certain regularity
474 conditions e.g., bridge regression. One approach might be to apply the local linear
475 approximation (Zou and Li, 2008) to ‘convert’ them to adaptive lasso penalties and
476 assess the performance of ERICν (λ) in such case. More generally, the principle of
477 formulating an information criterion which accounts for the effect of prior information
478 on the bias-variance tradeoff should, in principle, be applicable to any penalty that
479 has a proper prior distribution, such as the Bayesian elastic net (Hans, 2011). On
480 the other hand, given penalties such as SCAD and MCP (Zhang, 2010) can not be
481 written as a proper prior distribution since p0λ (|β|) = 0 for sufficiently large |β|, then
482 we conjecture ERICν (λ) may not be straightforwardly extendable to these penalties.

24
483 While the results in Section 6 showed promise, they are relatively brief, and con-
484 siderably more research needs to be conducted into the use of ERICν (λ) in ultra
485 high-dimensional data settings. As one reviewer pointed out, the two-stage approach
486 combining SIS with a penalized regression may not seem appealing, given that penal-
487 ties such as SCAD could be used directly with selection consistency guaranteed by
488 using BICft (λ) (for instance) to select the tuning parameter (Fan and Tang, 2012).
489 With the adaptive lasso, a more direct approach to handling p > n settings is to
490 construct weights based on fitting univariate GLMs to each covariate (Huang et al.,
491 2008). Given the strong performance of ERICν (λ) in the simulations presented earlier,
492 such a direct approach might also perform very well.

493 Acknowledgements

494 FKCH is supported by a Research Excellence Award from the University of New
495 South Wales and a CSIRO PhD Scholarship. DIW is supported by Australian Re-
496 search Council Discovery Projects and Future Fellow funding schemes (project num-
497 ber DP130102131 and FT120100501). SDF was supported in part by the Marine Bio-
498 diversity Hub, a collaborative partnership supported through funding from the Aus-
499 tralian Government’s National Environmental Research Program (NERP). Thanks
500 to Berwin Turlach for useful discussions and Eric Foster for inspiration.

501 References

502 Akaike, H. (1974). A new look at the statistical model identification. IEEE Transac-
503 tions on Automatic Control, 19:716–723.

25
504 Bühlmann, P. L. and van de Geer, S. A. (2011). Statistics for high-dimensional data.
505 Springer.

506 Chen, J. and Chen, Z. (2008). Extended Bayesian information criteria for model
507 selection with large model spaces. Biometrika, 95:759–771.

508 Chen, J. and Chen, Z. (2012). Extended BIC for small-n-large-P sparse GLM. Sta-
509 tistica Sinica, 22:555–574.

510 Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004). Least angle regression
511 (with discussion). The Annals of Statistics, 32:407–499.

512 Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and
513 its oracle properties. Journal of the American Statistical Association, 96:1348–1360.

514 Fan, J. and Lv, J. (2008). Sure independence screening for ultrahigh dimensional fea-
515 ture space. Journal of the Royal Statistical Society: Series B (Statistical Method-
516 ology), 70:849–911.

517 Fan, J. and Lv, J. (2010a). A selective overview of variable selection in high dimen-
518 sional feature space. Statistica Sinica, 20:101–148.

519 Fan, J. and Peng, H. (2004). Nonconcave penalized likelihood with a diverging number
520 of parameters . Annals of Statistics, 32:928–961.

521 Fan, J., Song, R., et al. (2010b). Sure independence screening in generalized linear
522 models with NP-dimensionality. The Annals of Statistics, 38:3567–3604.

523 Fan, Y. and Tang, C. Y. (2012). Tuning parameter selection in high dimensional
524 penalized likelihood. Journal of the Royal Statistical Society: Series B, 75:531–
525 552.

26
526 Frank, I. E. and Friedman, J. H. (1993). A statistical view of some chemometrics
527 regression tools (with discussion). Technometrics, 35:109–148.

528 Friedman, J., Hastie, T., and Tibshirani, R. (2010). Regularization paths for general-
529 ized linear models via coordinate descent. Journal of Statistical Software, 33:1–22.

530 Hans, C. (2011). Elastic net regression modeling with the orthant normal prior.
531 Journal of the American Statistical Association, 106:1383–1393.

532 Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learn-
533 ing - Data Mining, Inference, and Prediction. Second Edition. Springer-Verlag.

534 Hoerl, A. E. and Kennard, R. W. (1970). Ridge regression: Biased estimation for
535 nonorthogonal problems. Technometrics, 12:55–67.

536 Huang, J., Ma, S., and Zhang, C.-H. (2008). Adaptive lasso for sparse high-
537 dimensional regression models. Statistica Sinica, 18:1603.

538 Kawano, S. (2012). Selection of tuning parameters in bridge regression models via
539 Bayesian information criterion. arXiv preprint arXiv:1203.4326.

540 Kim, Y., Kwon, S., and Choi, H. (2012). Consistent model selection criteria on high
541 dimensions. The Journal of Machine Learning Research, 13:1037–1057.

542 Konishi, S., Ando, T., and Imoto, S. (2004). Bayesian information criteria and
543 smoothing parameter selection in radial basis function networks. Biometrika, 91:27–
544 43.

545 Luo, S. and Chen, Z. (2011). Selection consistency of EBIC for GLIM with
546 non-canonical links and diverging number of parameters. arXiv preprint
547 arXiv:1112.2815.

27
548 Luo, S. and Chen, Z. (2013). Extended BIC for linear regression models with diverg-
549 ing number of relevant features and high or ultra-high feature spaces. Journal of
550 Statistical Planning and Inference, 143:494–504.

551 McCullagh, P. and Nelder, J. A. (1989). Generalized Linear Models. Chapman &
552 Hall.

553 Meier, L., Van De Geer, S., and Bühlmann, P. (2008). The group lasso for logistic
554 regression. Journal of the Royal Statistical Society: Series B, 70:53–71.

555 Park, T. and Casella, G. (2008). The Bayesian lasso. Journal of the American
556 Statistical Association, 103:681–686.

557 Schelldorfer, J., Meier, L., Bühlmann, P., Winterthur, A., and Zürich, E. (2013).
558 Glmmlasso: An algorithm for high-dimensional generalized linear mixed models
559 using `1-penalization. Journal of Computational and Graphical Statistics, just ac-
560 cepted.

561 Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics,

562 6:461–464.

563 Sun, W., Wang, J., and Fang, Y. (2013). Consistent selection of tuning parameters via
564 variable selection stability. The Journal of Machine Learning Research, 14:3419–
565 3440.

566 Wang, H., Li, B., and Leng, C. (2009). Shrinkage tuning parameter selection with
567 a diverging number of parameters. Journal of the Royal Statistical Society: Series
568 B, 71:671–683.

28
 569 Wang, H., Li, R., and Tsai, C. (2007). Tuning parameter selectors for the smoothly
570 clipped absolute deviation method. Biometrika, 94:553–568.

571 Wang, T. and Zhu, L. (2011). Consistent tuning parameter selection in high dimen-
572 sional sparse linear regression. Journal of Multivariate Analysis, 102:1141–1151.

573 Zhang, C. (2010). Nearly unbiased variable selection under minimax concave penalty.
574 The Annals of Statistics, 38:894–942.

575 Zhang, Y., Li, R., and Tsai, C. (2010). Regularization parameter selections via
576 generalized information criterion. Journal of the American Statistical Association,
577 105:312–323.

578 Zou, H. (2006). The adaptive lasso and its oracle properties. Journal of the American
579 Statistical Association, 101:1418–1429.

580 Zou, H. and Li, R. (2008). One-step sparse estimates in nonconcave penalized likeli-
581 hood models. The Annals of Statistics, 36:1509–1533.

582 Zou, H. and Zhang, H. H. (2009). On the adaptive elastic-net with a diverging number
583 of parameters. The Annals of Statistics, 37:1733–1751.

29

View publication stats