Scribd
Upload
32 views6 pages

Optimal Ridge Parameter Selection

This document proposes a new method for selecting the optimal ridge parameter value in linear regression models. The goal is to achieve smaller variance without substantially increasing bias. The proposed ridge estimators will be evaluated using mean squared error and compared to ordinary least squares and existing ridge estimators using simulations and real data examples. Key areas of focus include addressing multicollinearity issues and developing estimators that are less sensitive to signal-to-noise ratios.

Uploaded by

Ayesha Shahbaz
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
32 views6 pages

Optimal Ridge Parameter Selection

This document proposes a new method for selecting the optimal ridge parameter value in linear regression models. The goal is to achieve smaller variance without substantially increasing bias. The proposed ridge estimators will be evaluated using mean squared error and compared to ordinary least squares and existing ridge estimators using simulations and real data examples. Key areas of focus include addressing multicollinearity issues and developing estimators that are less sensitive to signal-to-noise ratios.

Uploaded by

Ayesha Shahbaz
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Optimal Selection of Ridge Biasing Parameter for Linear Regression Model

.
Summary

The choice of ridge parameter plays a pivotal role in achieving the desirable property i.e. the
smaller variance. Generally, this is achieved at the cost of introducing a bias in the estimates.
In this work, we will propose a new method for selection of optimal value of ridge parameter.
The main objective in this new proposal is to achieve the smaller variance without a
substantial increase in the bias. The proposed estimators will be compared with the ordinary
least square and existing ridge estimators. Extensive simulations will be performed to
evaluate the performance for small to large samples. Real-life examples will be provided to
illustrate the use of proposed methods.

Keywords: Multicollinearity, linear regression, Monte Carlo simulation, ridge estimators,


MSE

1. Introduction

The multiple linear regression (MLR) model is defined as:

where represents vector of dependent variable vector, represents the design


matrix of fixed independent variables, denotes the vector of unknown regression
coefficients and represents the vector of error term given that . The
ordinary least square (OLS) estimator of regression coefficient ) is given as:

and

( ̂)

The OLS estimator is unbiased and may have smaller variance under usual conditions
but its covariance matrix is heavily dependent upon the characteristics of matrix.
When the independent variables are correlated, the OLS remains unbiased but it leads to a
substantial increase in the variance.

2. Background of the Proposed Research Project

1
To overcome the problem of multicollinearity, Hoerl and Kennard (1970) proposed ridge
regression (RR) estimator by introducing a biasing parameter, , in the objective function of
the Lagrange multiplier as follows:

where represents a constant value, i.e., . The normal equations for regression
parameter are obtained as:

( )

The solution of equation (5) provides the RR estimator of as:

̂ ( )

where represent the biasing parameter, denotes the number of independent variables and
denotes the identity matrix of order . The biasing parameter plays a vital role in
tradeoff between the bias and consistency of the estimator. When ̂ approaches to
OLS estimator and we obtain the unbiased estimator but inconsistent estimator of . Further,
when , ̂ we obtain consistent, stable but biased estimator of For positive
value of biasing parameter ( ), the estimator provides smaller mean squared error compared
to the OLS estimator (Muniz & Kibria, 2009).

Multicollinearity is a common practical problem that requires an optimal solution. In


ridge regression, the key idea is to introduce a biased estimator that decreases the overall
variance. Thus, the optimal value of biasing parameter which can yield the best results is still
an open issue in the literature. The ridge regression (RR) estimator is a function of biasing
parameter and many researchers have proposed different methods for its estimation (see
e.g., Kibria 2003; Muniz & Kibria, 2009; Khalaf et al., 2013; Dorugade 2014; Suhail, et al.,
2020; Kibria & Lukman, 2020).

Consider the canonical form of the MLR model in (1) as follows:

where , ( ) and represents orthogonal matrix given

that and  where  ( ) consist the eigen

2
values obtained from the matrix. The OLS estimator ( ̂), and the ridge regression
estimator ̂ in the canonical form are defined as follows:

̂ 

where ( ) and , for The MSE of the OLS estimator


and ridge regression estimator is defined respectively as:

̂ ∑( ⁄ )

̂ ∑ ∑

To deal with the problem of multicollinearity, Hoerl and Kennard (1970) introduced an
alternative technique to OLS using (4). To use ridge regression, a researcher needs to
determine the value of biasing parameter ( ). Thus, for the estimation of many methods
have been recommended by the investigators in terms of minimum MSE of ridge estimator
which depends on parameter vector of regression coefficient ( ) and the variance of MLR
model ( ). Since and are unknown, the researcher needs to employ their estimates. In
past twenty years, the researchers have been focusing on the estimation of biasing parameter
( ) by several ways and under different situations and then compare the results with the OLS
estimator. Much discussion concerning ridge regression go around finding an optimal value
of biasing parameter .

To estimate the biasing parameter ( ), Hoerl and Kennard (1970) defined the ridge
estimator as follows:

̂ ̂ ( )
̂

where ̂ represents the residual mean square estimate and can be computed using ̂
̂
∑ , ̂ are the OLS residuals and ̂ (̂ ̂ ).

Hoerl, et al., (1975) suggested a new ridge estimator as:

3
̂
̂
∑ ̂

Some new methods to estimate were suggested by Kibria (2003) by taking the
arithmetic mean, geometric mean and median of ̂ ⁄ ̂ respectively are defined as:

̂
̂ ∑( )
̂

̂
̂
√∏ ̂

̂
̂ ( )
̂

3. Justification / Rationale of the Project:

In general, biasing parameter ( ) in ridge regression leads to reduced variance but the cost
may be high in terms of bias. To deal with this problem, we will propose some new
estimators that may achieve smaller MSE with no or very little bias.

4. Research Objectives:

i. To propose a new ridge estimator to address the problem of multicollinearity with smaller
variance and bias.

ii. To address the estimation issue specifically for ill-conditioned problems.

iii. To make the estimator least affected by the signal-to-noise ratio.

5. Methodology:

To evaluate the performance of the proposed estimators in this study, the MSE criterions will
be used. Many researchers e.g., Kibria (2003), Suhail et al., (2020) have used the MSE
criterion for performance analysis of estimators. The MSE is described as follows:

̂ ̂ ̂ ∑ ̂

6. Proposed Analysis:

4
Following Kibria (2003) and Suhail et al., (2021), we have generated the independent
variables equation as follows:

where represents the specified degree of correlation between the independent variables
and is generated using the standard normal distribution. The response variable is obtained
by:

where represents the pseudo-random numbers i.e., and denotes the


number of observations.

Algorithm: MSE Computation

i. Generate the independent variables ( ) matrix using (18).


ii. Obtain the eigen values (i.e., ), and eigen vectors ( ) of the matrix
given that ∑
iii. Obtain the regression parameters in canonical form ( ) by using where
denotes the eigen vector of the maximum eigen value ( ) and
.
iv. Generate the random error term ( ).
v. Obtain the dependent variables ( ) by using the model in (19).
vi. Compute the OLS and RR estimates using (8) and (9) respectively.
vii. Repeat the above steps (ii) to (vi) for each of Monte Carlo runs.
viii. Calculate the MSE using:

̂ ∑ ̂ ̂

References
Dorugade, A., 2014. New ridge parameters for ridge regression. Journal of the Association of
Arab Universities for Basic and Applied Sciences, 15(1), pp. 94-99.

Hoerl, A. E. & Kennard, R. W., 1970. Ridge regression: Biased estimation for nonorthogonal
problems. Technometrics, 12(1), pp. 55-67.

5
Hoerl, A. E., Kennard, R. W. & Baldwin, K. F., 1975. Ridge regression: Some simulations.
Communications in Statistics-Theory and Methods, 4(2), pp. 105-123.

Khalaf, G., Mansson, K. & Shukur, G., 2013. Modified ridge regression estimators.
Communications in Statistics-Theory and Methods, 42(8), pp. 1476-1487.

Kibria, B. G., 2003. Performance of some new ridge regression estimators. Communications
in Statistics-Simulation and Computation, 32(2), pp. 419-435.

Kibria, B. & Lukman, A., 2020. A new ridge-type estimator for the linear regression model:
Simulations and applications. Scientifica, Volume 2020, pp. 1-16.

Muniz, G. & Kibria, B. G., 2009. On some ridge regression estimators: An empirical
comparison. Communications in Statistics - Simulation and Computation, 38(3), pp. 621-630.

Suhail, M., Chand, S. & Aslam, M., 2021. New quantile based ridge M-estimator for linear
regression models with multicollinearity and outliers. Communications in Statistics-
Simulation and Computation, pp. 1-25.

Suhail, M., Chand, S. & Kibria, . B. M. G., 2020. Quantile based estimation of biasing
parameters in ridge regression model. Communications in Statistics-Simulation and
Computation, 49(10), pp. 2732-2744.

Suhail, M., Chand, S. & Kibria, G. B., 2019. Quantile-based robust ridge m-estimator for
linear regression model in presence of multicollinearity and outliers. Communications in
Statistics-Simulation and Computation, pp. 1-13.

Common questions

Powered by AI

Multicollinearity affects the validity of regression models by inflating the variance of coefficient estimates, leading to less reliable and presumably misleading results, despite the estimators being unbiased . Ridge Regression is justified in these contexts because it mitigates high variance through introducing a biasing parameter that stabilizes the estimates and results in more reliable predictions . This trade-off is crucial in contexts where multicollinearity undermines model effectiveness, thus necessitating techniques like Ridge Regression to preserve the model's utility .

Multicollinearity occurs when independent variables in a regression model are correlated, which leads to a significant increase in the variance of the OLS estimates. Although the OLS estimator remains unbiased, the high variance can render the model's predictions unreliable . Ridge Regression addresses this problem by introducing a biasing parameter to the regression equation, which shrinks the coefficients and thus reduces variance at the cost of introducing some bias. This trade-off results in a more stable and consistent estimator, effectively tackling the issues caused by multicollinearity .

In canonical form, ridge regression estimators are derived by transforming the original model into one where the independent variables are orthogonal, meaning uncorrelated. This involves using eigenvalues and eigenvectors of the matrix formed by the independent variables. In this setting, the ridge estimator takes the specific form that adjusts each component of the canonical form by the biasing parameter, depending on the magnitude of its corresponding eigenvalue, thereby selectively shrinking the coefficients based on this variance decomposition . This approach helps explicitly address multicollinearity while controlling the degree of bias introduced .

The Mean Squared Error (MSE) is employed as a metric to evaluate ridge regression estimators' performance because it incorporates both variance and bias components of the estimator. A lower MSE indicates a better trade-off between the bias and variance, providing a more consistent and reliable estimate . Choosing the optimal biasing parameter involves minimizing the MSE, ensuring that the estimator performs well under different conditions of multicollinearity without overly increasing the bias .

Researchers have proposed various methodologies to estimate the optimal biasing parameter in ridge regression, mainly focusing on minimizing the MSE of the ridge estimator. These include taking the arithmetic mean, geometric mean, and median of estimates derived from the OLS residuals . Methods are evaluated based on how well they address multicollinearity and their computational efficiency. These approaches differ in complexity and accuracy. While some are simple heuristics (e.g., arithmetic mean), others use more sophisticated statistical measures like quantile-based estimations, which may offer more robustness in the presence of outliers or extreme values .

Simulations help in evaluating proposed methods for ridge regression by allowing researchers to systematically control factors like multicollinearity level, sample size, and variance elements, providing a robust platform to analyze performance under various hypothetical scenarios . Through simulations, researchers can rigorously compare the effectiveness of different biasing parameter estimation methods by assessing their impact on mean squared error and bias-variance tradeoff across diverse conditions, thereby offering insights into their practical utility and robustness .

The problem of finding the optimal value of the biasing parameter remains an open issue in ridge regression due to the balance required between bias introduction and variance reduction. The optimal biasing parameter must adapt to varying levels of multicollinearity and dataset conditions, yet no single methodology perfectly accommodates all scenarios. Moreover, computational constraints and the practical applicability of theoretical solutions contribute to this ongoing challenge . The complex nature of data structures and differing objectives in practical applications further exacerbate the difficulty of convergence on a universally optimal solution .

In real-life applications, ridge regression offers significant benefits in scenarios where multicollinearity is present, as it provides more stable and reliable estimates compared to the classical OLS method. The method's capacity to reduce variance and control bias ensures that the predictive power of models is maintained even when independent variables are highly correlated. This is crucial in fields such as econometrics, genetics, and any area where data characteristics often include multicollinear predictors. Adopting ridge regression can lead to better decision-making processes based on more robust statistical inference, as demonstrated through numerous empirical studies and simulations .

The introduction of bias in ridge regression is justified by the substantial reduction it offers in estimator variance, a critical factor in improving model reliability when multicollinearity is present. Although biased estimators often bear negative connotations due to their departure from the true parameter values, in ridge regression, this bias often results in a lower Mean Squared Error (MSE) compared to unbiased estimators like the OLS, making the model more useful in prediction tasks . The rationale is to achieve more consistent outcomes even at the cost of some bias .

The proposed new ridge estimator aims to achieve a more precise balance between variance reduction and bias introduction by optimizing the selection of the biasing parameter to minimize the increase in bias while achieving significant variance reductions . By focusing on optimizing the mean squared error (MSE) criterion, this new approach intends to provide more consistent estimates even under high multicollinearity, and potentially offer computational advantages in real-life applications .

You might also like