nips nips2009 nips2009-173 nips2009-173-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Han Liu, Xi Chen
Abstract: This paper studies the forward greedy strategy in sparse nonparametric regression. For additive models, we propose an algorithm called additive forward regression; for general multivariate models, we propose an algorithm called generalized forward regression. Both algorithms simultaneously conduct estimation and variable selection in nonparametric settings for the high dimensional sparse learning problem. Our main emphasis is empirical: on both simulated and real data, these two simple greedy methods can clearly outperform several state-ofthe-art competitors, including LASSO, a nonparametric version of LASSO called the sparse additive model (SpAM) and a recently proposed adaptive parametric forward-backward algorithm called Foba. We also provide some theoretical justifications of specific versions of the additive forward regression. 1
[1] Francis Bach. Consistency of the group lasso and multiple kernel learning. Journal of Machine Learning Research, 8:1179–1225, 2008.
[2] Francis Bach. Exploring large feature spaces with hierarchical multiple kernel learning. In Advances in Neural Information Processing Systems 21. MIT Press, 2008.
[3] Andrew R. Barron, Albert Cohen, Wolfgang Dahmen, and Ronald A. DeVore. Approximation and learning by greedy algorithms. The Annals of Statistics, 36:64–94, 2008.
[4] Peter B¨ hlmann and Bin Yu. Sparse boosting. Journal of Machine Learning Research, 7:1001– u 1024, 2006.
[5] Andreas Buja, Trevor Hastie, and Robert Tibshirani. Linear smoothers and additive models. The Annals of Statistics, 17:453–510, 1989.
[6] Emmanuel Candes and Terence Tao. The dantzig selector: statistical estimation when p is much larger than n. The Annals of Statistics, 35:2313–2351, 2007.
[7] Scott Shaobing Chen, David L. Donoho, and Michael A. Saunders. Atomic decomposition by basis pursuit. SIAM Journal on Scientific and Statistical Computing, 20:33–61, 1998.
[8] Jianqing Fan and Ir` ne Gijbels. Local polynomial modelling and its applications. Chapman e and Hall, 1996.
[9] Jerome H. Friedman. Multivariate adaptive regression splines. The Annals of Statistics, 19:1– 67, 1991.
[10] Trevor Hastie and Robert Tibshirani. Generalized additive models. Chapman & Hall Ltd., 1999.
[11] John Lafferty and Larry Wasserman. Rodeo: Sparse, greedy nonparametric regression. The Annals of Statistics, 36(1):28–63, 2008.
[12] Yi Lin and Hao Helen Zhang. Component selection and smoothing in multivariate nonparametric regression. The Annals of Statistics., 34(5):2272–2297, 2006.
[13] Han Liu and Jian Zhang. On the estimation consistency of the group lasso and its applications. Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, 2009.
[14] S. Mallat and Z. Zhang. Matching pursuit with time-frequency dictionaries. IEEE Transactions on Signal Processing, 41:3397–3415, 1993.
[15] Lukas Meier, Sara van de Geer, and Peter B¨ hlmann. High-dimensional additive modelling. u The Annals of Statistics (to appear), 2009.
[16] Pradeep Ravikumar, John Lafferty, Han Liu, and Larry Wasserman. Sparse additive models. Journal of the Royal Statistical Society, Series B, Methodological, 2009. To appear.
[17] Pradeep Ravikumar, Han Liu, John Lafferty, and Larry Wasserman. Spam: Sparse additive models. In Advances in Neural Information Processing Systems 20, 2007.
[18] B. W. Silverman. Spline smoothing: The equivalent variable kernel method. The Annals of Statistics, 12:898–916, 1984.
[19] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, Methodological, 58:267–288, 1996.
[20] Joel A. Tropp. Greed is good: Algorithmic results for sparse approximation. IEEE Trans. Inform. Theory, 50(10):2231–2241, October 2004.
[21] Grace Wahba. Spline models for observational data. SIAM [Society for Industrial and Applied Mathematics], 1990.
[22] Tong Zhang. Adaptive forward-backward greedy algorithm for learning sparse representations. Technical report, Rutgers University, 2008.
[23] Tong Zhang. On the consistency of feature selection usinggreedy least squares regression. Journal of Machine Learning Research, 10:555–568, 2009. 9