jmlr jmlr2006 jmlr2006-58 jmlr2006-58-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Guillaume Lecué
Abstract: In this paper we prove the optimality of an aggregation procedure. We prove lower bounds for aggregation of model selection type of M density estimators for the Kullback-Leibler divergence (KL), the Hellinger’s distance and the L1 -distance. The lower bound, with respect to the KL distance, can be achieved by the on-line type estimate suggested, among others, by Yang (2000a). Combining these results, we state that log M/n is an optimal rate of aggregation in the sense of Tsybakov (2003), where n is the sample size. Keywords: aggregation, optimal rates, Kullback-Leibler divergence
N. H. Augustin, S. T. Buckland, and K. P. Burnham. Model selection: An integral part of inference. Biometrics, 53:603–618, 1997. A. Barron and G. Leung. Information theory and mixing least-square regressions. 2004. manuscript. L. Birg´ . e Model selection via testing: an alternative to (penalized) maximum likelihood estimators. To appear in Annales of IHP, 2004. Available at http://www.proba.jussieu.fr/mathdoc/textes/PMA-862.pdf. F. Bunea and A. Nobel. Online prediction algorithms for aggregation of arbitrary estimators of a conditional mean. 2005. Submitted to IEEE Transactions in Information Theory. O. Catoni. A mixture approach to universal model selection. 1997. preprint LMENS-97-30, available at http://www.dma.ens.fr/EDITION/preprints/. O. Catoni. Statistical Learning Theory and Stochastic Optimization. Ecole d’´ t´ de Probabilit´ s de ee e Saint-Flour 2001, Lecture Notes in Mathematics. Springer, N.Y., 2004. L. Devroye and G. Lugosi. Combinatorial methods in density estimation. 2001. Springer, NewYork. J.A. Hartigan. Bayesian regression using akaike priors. 2002. Yale University, New Haven, Preprint. I.A. Ibragimov and R.Z. Hasminskii. An estimate of density of a distribution. Studies in mathematical stat. IV. Zap. Nauchn. Semin., LOMI, 98(1980),61–85. A. Juditsky, A. Nazin, A.B. Tsybakov and N. Vayatis. Online aggregation with mirror-descent algorithm. 2005. Preprint n.987, Laboratoire de Probabilit´ s et Mod` le al´ atoires, Universit´ s e e e e Paris 6 and Paris 7 (available at http://www.proba.jussieu.fr/mathdoc/preprints/index.html#2005). A. Juditsky and A. Nemirovski. Functionnal aggregation for nonparametric estimation. Ann. of Statist., 28:681–712, 2000. 980 L OWER B OUNDS AND AGGREGATION IN D ENSITY E STIMATION G. Lecu´ . Simultaneous adaptation to the marge and to complexity in classification. 2005. Available e at http://hal.ccsd.cnrs.fr/ccsd-00009241/en/. A. Nemirovski. Topics in Non-parametric Statistics. Springer, N.Y., 2000. P. Rigollet and A. B. Tsybakov. Linear and convex aggregation of density estimators. 2004. Manuscript. A.B. Tsybakov. Optimal rates of aggregation. Computational Learning Theory and Kernel Machines. B.Sch¨ lkopf and M.Warmuth, eds. Lecture Notes in Artificial Intelligence, 2777:303–313, o 2003. Springer, Heidelberg. A.B. Tsybakov. Introduction a l’estimation non-param´ trique. Springer, 2004. e ` Y. Yang. Mixing strategies for density estimation. Ann. Statist., 28(1):75–87, 2000a. Y. Yang. Combining Different Procedures for Adaptive Regression. Journal of Multivariate Analysis, 74:135–161, 2000b. Y. Yang. Adaptive regression by mixing. Journal of American Statistical Association, 96:574–588, 2001. Y. Yang. Aggregating regression procedures to impose performance. Bernoulli, 10(1):25–47, 2004. T. Zhang. From epsilon-entropy to KL-complexity: analysis of minimum information complexity density estimation. 2003. Tech. Report RC22980, IBM T.J.Watson Research Center. 981