jmlr jmlr2008 jmlr2008-33 jmlr2008-33-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: David Mease, Abraham Wyner
Abstract: The statistical perspective on boosting algorithms focuses on optimization, drawing parallels with maximum likelihood estimation for logistic regression. In this paper we present empirical evidence that raises questions about this view. Although the statistical perspective provides a theoretical framework within which it is possible to derive theorems and create new algorithms in general contexts, we show that there remain many unanswered important questions. Furthermore, we provide examples that reveal crucial flaws in the many practical suggestions and new methods that are derived from the statistical view. We perform carefully designed experiments using simple simulation models to illustrate some of these flaws and their practical consequences. Keywords: boosting algorithms, LogitBoost, AdaBoost
L. Breiman. Discussion of additive logistic regression: A statistical view of boosting. Annals of Statistics, 28:374–377, 2000. L. Breiman. Bagging predictors. Machine Learning, 24:123–140, 1996. L. Breiman. Random forests. Machine Learning, 45:5–32, 2001. P. Buhlmann and B. Yu. Boosting with the L2 loss: Regression and classification. Journal of the American Statistical Association, 98:324–339, 2003. A. Buja. Discussion of additive logistic regression: A statistical view of boosting. Annals of Statistics, 28:387–391, 2000. A. Buja, W. Stuetzle, and Y. Shen. Loss functions for binary class probability estimation and classification: Structure and applications. 2006. M. Collins, R. E. Schapire, and Y. Singer. Logistic regression, adaboost and bregman distances. In Computational Learing Theory, pages 158–169, 2000. A. Cutler and G. Zhao. Pert: Perfect random tree ensembles. Computing Science and Statistics, 33: 490–497, 2001. M. Dettling and P. Buhlmann. Boosting for tumor classification with gene expression data. Bioinformatics, 19:1061–1069, 2003. Y. Freund and R. E. Schapire. Experiments with a new boosting algorithm. In Machine Learning: Proceedings of the Thirteenth International Conference, pages 148–156, 1996. Y. Freund and R. E. Schapire. Discussion of additive logistic regression: A statistical view of boosting. Annals of Statistics, 28:391–393, 2000. J. Friedman, T. Hastie, and R. Tibshirani. Additive logistic regression: A statistical view of boosting. Annals of Statistics, 28:337–374, 2000a. J. Friedman, T. Hastie, and R. Tibshirani. Rejoiner for additive logistic regression: A statistical view of boosting. Annals of Statistics, 28:400–407, 2000b. 155 M EASE AND W YNER A. J. Grove and D. Schuurmans. Boosting in the limit: Maximizing the margin of learned ensembles. In Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI-98), pages 692–699, 1998. T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer, 2001. W. Jiang. Does boosting overfit: Views from an exact solution. Technical Report 00-03, Department of Statistics, Northwestern University, 2000. W. Jiang. Is regularization unnecessary for boosting? In Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics, pages 57–64, 2001. W. Jiang. On weak base hypotheses and their implications for boosting regression and classification. Annals of Statistics, 30:51–73, 2002. A. Krieger, C. Long, and A. J. Wyner. Boosting noisy data. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 274–281, 2001. G. Lugosi and N. Vayatis. On the bayes-risk consistency of regularized boosting methods. Annals of Statistics, 32:30–55, 2004. D. Mease, A. Wyner, and A. Buja. Boosted classification trees and class probability/quantile estimation. Journal of Machine Learning Research, 8:409–439, 2007. G. Ridgeway. Discussion of additive logistic regression: A statistical view of boosting. Annals of Statistics, 28:393–400, 2000. G. Ridgeway. Generalized boosted models: A guide to the gbm package. 2005. R. E. Schapire, Y. Freund, P. L. Bartlett, and W. S. Lee. Boosting the margin: A new explanation for the effectiveness of voting methods. Annals of Statistics, 26:1651–1686, 1998. T. Zhang and B. Yu. Boosting with early stopping: Convergence and consistency. Annals of Statistics, 33:1538–1579, 2005. 156