A Simple Approach in Regression Variable Selection

Regression variable subset selection is one of the most important aspects in linear model theory. If the selected subset is consistent when the sample size tends to infinity, and the prediction mean square error is small, then the selection method is preferred. The BIC criterion can give consistent subset, but as the number of variables get large, it involves too much computation. The adaptive lasso has better computational efficiency, while keeping consistency. In this paper we propose a new approach for multiple linear regression variable selection, which is much simpler than the other variable selection methods, while it gives consistent subset. The new method only compute two passes of ordinary least squares regressions, the first pass computes a complete set regression, selects a variable subset based on the regression coefficient estimates, then the second pass regresses on the selected variables.
Consider the following regression model: where the indexes of the non-zero elements of are denoted by , and suppose the new method gives a regression variable subset indexed by , and is the regression coefficient estimate using our new method, in which the coefficients of the dropped out variables are defined to be zero. We proved that under suitable conditions where denotes the vector composed of the elements of indexed by , is the error variance, are matrix and constant relying on the limit of.
Simulation result and application examples show that the new approach have good small to medium sample performance, which is comparable to the other methods such as BIC, adaptive lasso.