jmlr jmlr2013 jmlr2013-27 jmlr2013-27-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Wei Sun, Junhui Wang, Yixin Fang
Abstract: Penalized regression models are popularly used in high-dimensional data analysis to conduct variable selection and model fitting simultaneously. Whereas success has been widely reported in literature, their performances largely depend on the tuning parameters that balance the trade-off between model fitting and model sparsity. Existing tuning criteria mainly follow the route of minimizing the estimated prediction error or maximizing the posterior model probability, such as cross validation, AIC and BIC. This article introduces a general tuning parameter selection criterion based on variable selection stability. The key idea is to select the tuning parameters so that the resultant penalized regression model is stable in variable selection. The asymptotic selection consistency is established for both fixed and diverging dimensions. Its effectiveness is also demonstrated in a variety of simulated examples as well as an application to the prostate cancer data. Keywords: kappa coefficient, penalized regression, selection consistency, stability, tuning
H. Akaike. A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716-723, 1974. A. Ben-Hur, A. Elisseeff and I. Guyon. A stability based method for discovering structure in clustered data. Pacific Symposium on Biocomputing, 6-17, 2002. P. Breheny and J. Huang. Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. Annals of Applied Statistics, 5, 232-253, 2011. J. Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46, 1960. P. Craven and G. Wahba. Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross-validation. Numerische Mathematik, 31, 317403, 1979. B. Efron, T. Hastie, I. Johnstone and R. Tibshirani. Least angle regression. Annals of Statistics, 32, 407-451, 2004. 3438 C ONSISTENT S ELECTION OF T UNING PARAMETERS VIA VARIABLE S ELECTION S TABILITY J. Fan and R. Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96, 1348-1360, 2001. J. Fan and R. Li. Statistical Challenges with high dimensionality: Feature selection in knowledge discovery. In Proceedings of the International Congress of Mathematicians, 3, 595-622, 2006. J. Fan and H. Peng. Nonconcave penalized likelihood with a diverging number of parameters. Annals of Statistics, 32, 928-961, 2004. W. Hoeffding. The strong law of large numbers for U-statistics. Institute of Statistical Mimeograph Series, No. 302, University of North Carolina, 1961. J. Huang, S. Ma and C.H. Zhang. Adaptive Lasso for sparse high-dimensional regression models. Statistica Sinica, 18, 1603-1618, 2008. J. Huang and H. Xie. Asymptotic oracle properties of SCAD-penalized least squares estimators. IMS Lecture Notes - Monograph Series, Asymptotics: Particles, Processes and Inverse Problems, 55, 149-166, 2007. C. Mallows. Some comments on Cp. Technometrics, 15, 661-675, 1973. N. Meinshausen and P. B¨ hlmann. Stability selection. Journal of the Royal Statistical Society, Series u B, 72, 414-473, 2010. G.E. Schwarz. Estimating the dimension of a model. Annals of Statistics, 6, 461-464, 1978. X. Shen, W. Pan and Y. Zhu. Likelihood-based selection and sharp parameter estimation. Journal of the American Statistical Association, 107, 223-232, 2012. T.A. Stamey, J.N. Kabalin, J.E. McNeal, I.M. Johnstone, F. Freiha, E.A. Redwine and N. Yang. Prostate specific antigen in the diagnosis and treatment of adenocarcinoma of the prostate: II. Radical prostatectomy treated patients. Journal of Urology, 141, 1076-1083, 1989. M. Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society, Series B, 36, 111-147, 1974. W. Sun, J. Wang and Y. Fang. Regularized K-means clustering of high-dimensional data and its asymptotic consistency, Electronic Journal of Statistics, 6, 148-167, 2012. R. Tibshirani. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, Series B, 58, 267-288, 1996. H. Wang and C. Leng. A note on adaptive group Lasso. Computational Statistics and Data Analysis, 52, 5277-5286, 2008. H. Wang, R. Li and C.L. Tsai. Tuning parameter selectors for the smoothly clipped absolute deviation method. Biometrika, 94, 553-568, 2007. H. Wang, B. Li and C. Leng. Shrinkage tuning parameter selection with a diverging number of parameters. Journal of the Royal Statistical Society, Series B, 71, 671-683, 2009. 3439 S UN , WANG AND FANG J. Wang. Consistent selection of the number of clusters via cross validation. Biometrika, 97, 893904, 2010. L. Xue, A. Qu and J. Zhou. Consistent model selection for marginal generalized additive model for correlated data. Journal of the American Statistical Association, 105, 1518-1530, 2010. M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society, Series B, 68, 49-67, 2006. Y. Zhang, R. Li and C.L. Tsai. Regularization parameter selections via generalized information criterion. Journal of the American Statistical Association, 105, 312-323, 2010. P. Zhao and B. Yu. On model selection consistency of Lasso. Journal of Machine Learning Research, 7, 2541-2563, 2006. H. Zou. The adaptive Lasso and its oracle properties. Journal of the American Statistical Association, 101, 1418-1429, 2006. H. Zou and T. Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, Series B, 67, 301-320, 2005. H. Zou, T. Hastie and R. Tibshirani. On the “Degree of Freedom” of the Lasso. Annals of Statistics, 35, 2173-2192, 2007. H. Zou and H. Zhang. On the adaptive elastic-net with a diverging number of parameters. Annals of Statistics, 37, 1733-1751, 2009. 3440