nips nips2003 nips2003-122 nips2003-122-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Saharon Rosset, Ji Zhu, Trevor J. Hastie
Abstract: Margin maximizing properties play an important role in the analysis of classi£cation models, such as boosting and support vector machines. Margin maximization is theoretically interesting because it facilitates generalization error analysis, and practically interesting because it presents a clear geometric interpretation of the models being built. We formulate and prove a suf£cient condition for the solutions of regularized loss functions to converge to margin maximizing separators, as the regularization vanishes. This condition covers the hinge loss of SVM, the exponential loss of AdaBoost and logistic regression loss. We also generalize it to multi-class classi£cation problems, and present margin maximizing multiclass versions of logistic regression and support vector machines. 1
[1] Bartlett, P., Jordan, M. & McAuliffe, J. (2003). Convexity, Classi£cation and Risk Bounds. Technical reports, dept. of Statistics, UC Berkeley.
[2] Breiman, L. (1999). Prediction games and arcing algorithms. Neural Computation 7:1493-1517.
[3] Freund, Y. & Scahpire, R.E. (1995). A decision theoretic generalization of on-line learning and an application to boosting. Proc. of 2nd Eurpoean Conf. on Computational Learning Theory.
[4] Friedman, J. H., Hastie, T. & Tibshirani, R. (2000). Additive logistic regression: a statistical view of boosting. Annals of Statistics 28, pp. 337-407.
[5] Grove, A.J. & Schuurmans, D. (1998). Boosting in the limit: Maximizing the margin of learned ensembles. Proc. of 15th National Conf. on AI.
[6] Hastie, T., Tibshirani, R. & Friedman, J. (2001). Elements of Stat. Learning. Springer-Verlag.
[7] Mangasarian, O.L. (1999). Arbitrary-norm separating plane. Operations Research Letters, Vol. 24 1-2:15-23
[8] Rosset, R., Zhu, J & Hastie, T. (2003). Boosting as a regularized path to a maximum margin classi£er. Technical report, Dept. of Statistics, Stanford Univ.
[9] Scahpire, R.E., Freund, Y., Bartlett, P. & Lee, W.S. (1998). Boosting the margin: a new explanation for the effectiveness of voting methods. Annals of Statistics 26(5):1651-1686
[10] Vapnik, V. (1995). The Nature of Statistical Learning Theory. Springer.
[11] Weston, J. & Watkins, C. (1998). Multi-class support vector machines. Technical report CSDTR-98-04, dept of CS, Royal Holloway, University of London.