nips nips2002 nips2002-138 nips2002-138-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Pascal Vincent, Yoshua Bengio
Abstract: The similarity between objects is a fundamental element of many learning algorithms. Most non-parametric methods take this similarity to be fixed, but much recent work has shown the advantages of learning it, in particular to exploit the local invariances in the data or to capture the possibly non-linear manifold on which most of the data lies. We propose a new non-parametric kernel density estimation method which captures the local structure of an underlying manifold through the leading eigenvectors of regularized local covariance matrices. Experiments in density estimation show significant improvements with respect to Parzen density estimators. The density estimators can also be used within Bayes classifiers, yielding classification rates similar to SVMs and much superior to the Parzen classifier.
[1] P. Vincent and Y. Bengio. K-local hyperplane and convex distance nearest neighbor algorithms. In T.G. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems, volume 14. The MIT Press, 2002.
[2] E. Parzen. On the estimation of a probability density function and mode. Annals of Mathematical Statistics, 33:1064–1076, 1962.
[3] G.E. Hinton, M. Revow, and P. Dayan. Recognizing handwritten digits using mixtures of linear models. In G. Tesauro, D.S. Touretzky, and T.K. Leen, editors, Advances in Neural Information Processing Systems 7, pages 1015–1022. MIT Press, Cambridge, MA, 1995.
[4] M.E. Tipping and C.M. Bishop. Mixtures of probabilistic principal component analysers. Neural Computation, 11(2):443–482, 1999.
[5] Z. Ghahramani and G.E. Hinton. The EM algorithm for mixtures of factor analyzers. Technical Report CRG-TR-96-1, Dpt. of Comp. Sci., Univ. of Toronto, 21 1996.
[6] Z. Ghahramani and M. J. Beal. Variational inference for Bayesian mixtures of factor analysers. In Advances in Neural Information Processing Systems 12, Cambridge, MA, 2000. MIT Press.
[7] P. Y. Simard, Y. A. LeCun, J. S. Denker, and B. Victorri. Transformation invariance in pattern recognition — tangent distance and tangent propagation. Lecture Notes in Computer Science, 1524, 1998.
[8] D. Keysers, J. Dahmen, and H. Ney. A probabilistic view on tangent distance. In 22nd Symposium of the German Association for Pattern Recognition, Kiel, Germany, 2000.
[9] J. Dahmen, D. Keysers, M. Pitz, and H. Ney. Structured covariance matrices for statistical image object recognition. In 22nd Symposium of the German Association for Pattern Recognition, Kiel, Germany, 2000.
[10] S. Roweis and L. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2323–2326, Dec. 2000.
[11] Y. Whye Teh and S. Roweis. Automatic alignment of local representations. In S. Becker, S. Thrun, and K. Obermayer, editors, Advances in Neural Information Processing Systems, volume 15. The MIT Press, 2003.
[12] V. de Silva and J.B. Tenenbaum. Global versus local approaches to nonlinear dimensionality reduction. In S. Becker, S. Thrun, and K. Obermayer, editors, Advances in Neural Information Processing Systems, volume 15. The MIT Press, 2003.
[13] M. Brand. Charting a manifold. In S. Becker, S. Thrun, and K. Obermayer, editors, Advances in Neural Information Processing Systems, volume 15. The MIT Press, 2003.
[14] R. D. Short and K. Fukunaga. The optimal distance measure for nearest neighbor classification. IEEE Transactions on Information Theory, 27:622–627, 1981.
[15] J. Myles and D. Hand. The multi-class measure problem in nearest neighbour discrimination rules. Pattern Recognition, 23:1291–1297, 1990.
[16] J. Friedman. Flexible metric nearest neighbor classification. Technical Report 113, Stanford University Statistics Department, 1994.
[17] T. Hastie and R. Tibshirani. Discriminant adaptive nearest neighbor classification and regression. In David S. Touretzky, Michael C. Mozer, and Michael E. Hasselmo, editors, Advances in Neural Information Processing Systems, volume 8, pages 409–415. The MIT Press, 1996.
[18] A.J. Inzenman. Recent developments in nonparametric density estimation. Journal of the American Statistical Association, 86(413):205–224, 1991.