jmlr jmlr2006 jmlr2006-65 jmlr2006-65-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Ichiro Takeuchi, Quoc V. Le, Timothy D. Sears, Alexander J. Smola
Abstract: In regression, the desired estimate of y|x is not always given by a conditional mean, although this is most common. Sometimes one wants to obtain a good estimate that satisfies the property that a proportion, τ, of y|x, will be below the estimate. For τ = 0.5 this is an estimate of the median. What might be called median regression, is subsumed under the term quantile regression. We present a nonparametric version of a quantile estimator, which can be obtained by solving a simple quadratic programming problem and provide uniform convergence statements and bounds on the quantile property of our estimator. Experimental results show the feasibility of the approach and competitiveness of our method with existing ones. We discuss several types of extensions including an approach to solve the quantile crossing problems, as well as a method to incorporate prior qualitative knowledge such as monotonicity constraints. Keywords: support vector machines, kernel methods, quantile estimation, nonparametric techniques, estimation with constraints
L. K. Bachrach, T. Hastie, M. C. Wang, B. Narashimhan, and R. Marcus. Bone mineral acquisition in healthy asian, hispanic, black and caucasian youth, a longitudinal study. Journal of Clinical Endocrinal Metabolism, 84:4702– 4712, 1999. P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463–482, 2002. P. L. Bartlett, O. Bousquet, and S. Mendelson. Localized rademacher averages. In Proc. Annual Conf. Computational Learning Theory, pages 44–58, 2002. P. J. Bickel, C. A. J. Klaassen, Y. Ritov, and J. A. Wellner. Efficient and adaptive estimation for semiparametric models. J. Hopkins Press, Baltimore, ML, 1994. R. J. Bosch, Y. Ye, and G. G.Woodworth. A convergent algorithm for quantile regression with smoothing splines. Computational Statistics and Data Analysis, 19:613–630, 1995. D. D. Cox. Approximation of method of regularization estimators. Annals of Statistics, 16:694–713, 1988. G. Fung, O. L. Mangasarian, and A. J. Smola. Minimal kernel classifiers. Journal of Machine Learning Research, 3:303–321, 2002. C. Gu and G. Wahba. Semiparametric analysis of variance with tensor product thin plate splines. Journal of the Royal Statistical Society B, 55:353–368, 1993. P. Hall and N. Tajvidi. Distribution and dependence-function estimation for bivariate extreme-value distributions. Bernoulli, 2000. P. Hall, J. S. Marron, and A. Neeman. Geometric representation of high dimension low sample size data. Journal of the Royal Statistical Society - Series B, 2005. forthcoming. T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer, New York, 2001. X. He. Quantile curves without crossing. The American Statistician, 51(2):186–192, may 1997. W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58:13–30, 1963. A. E. Hoerl and R. W. Kennard. Ridge regression: biased estimation for nonorthogonal problems. Technometrics, 12:55–67, 1970. P. J. Huber. Robust Statistics. John Wiley and Sons, New York, 1981. R. Koenker. Quantile Regression. Cambridge University Press, 2005. R. Koenker and G. Bassett. Regression quantiles. Econometrica, 46(1):33–50, 1978. R. Koenker, P. Ng, and S. Portnoy. Quantile smoothing splines. Biometrika, 81:673–680, 1994. 1263 TAKEUCHI , L E , S EARS AND S MOLA Q. V. Le, A. J. Smola, and T. G¨ rtner. Simpler knowledge-based support vector machines. In Proc. a Intl. Conf. Machine Learning, 2006. E. Mammen, J. S. Marron, B. A. Turlach, and M. P. Wand. A general projection framework for constrained smoothing. Statistical Science, 16(3):232–248, August 2001. S. Mendelson. A few notes on statistical learning theory. In S. Mendelson and A. J. Smola, editors, Advanced Lectures on Machine Learning, number 2600 in LNAI, pages 1–40. Springer, 2003. D. Ruppert and R. J. Carroll. Semiparametric Regression. Wiley, 2003. B. Sch¨ lkopf and A. Smola. Learning with Kernels. MIT Press, Cambridge, MA, 2002. o B. Sch¨ lkopf, R. C. Williamson, A. J. Smola, and J. Shawe-Taylor. Single-class support vector o machines. In J. Buhmann, W. Maass, H. Ritter, and N. Tishby, editors, Unsupervised Learning, Dagstuhl-Seminar-Report 235, pages 19–20, 1999. B. Sch¨ lkopf, A. J. Smola, R. C. Williamson, and P. L. Bartlett. New support vector algorithms. o Neural Computation, 12:1207–1245, 2000. A. J. Smola and B. Sch¨ lkopf. On a kernel-based method for pattern recognition, regression, apo proximation and operator inversion. Algorithmica, 22:211–231, 1998. A. J. Smola, T. Frieß, and B. Sch¨ lkopf. Semiparametric support vector and linear programming o machines. In M. S. Kearns, S. A. Solla, and D. A. Cohn, editors, Advances in Neural Information Processing Systems 11, pages 585–591, Cambridge, MA, 1999. MIT Press. I. Takeuchi and T. Furuhashi. Non-crossing quantile regressions by SVM. In Proc. International Joint Conference on Neural Networks, 2004. M. Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research, 1:211–244, 2001. V. Vapnik. The Nature of Statistical Learning Theory. Springer, New York, 1995. V. Vapnik, S. Golowich, and A. J. Smola. Support vector method for function approximation, regression estimation, and signal processing. In M. C. Mozer, M. I. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems 9, pages 281–287, Cambridge, MA, 1997. MIT Press. V. N. Vapnik. Estimation of Dependences Based on Empirical Data. Springer, Berlin, 1982. S. V. N. Vishwanathan, A. J. Smola, and M. N. Murty. SimpleSVM. In Tom Fawcett and Nina Mishra, editors, Proc. Intl. Conf. Machine Learning, Washington DC, 2003. AAAI press. G. Wahba. Spline Models for Observational Data, volume 59 of CBMS-NSF Regional Conference Series in Applied Mathematics. SIAM, Philadelphia, 1990. R. C. Williamson, A. J. Smola, and B. Sch¨ lkopf. Generalization bounds for regularization networks o and support vector machines via entropy numbers of compact operators. IEEE Transaction on Information Theory, 47(6):2516–2532, 2001. 1264