nips nips2007 nips2007-195 nips2007-195-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Andrew Naish-guzman, Sean Holden
Abstract: We present an efficient generalization of the sparse pseudo-input Gaussian process (SPGP) model developed by Snelson and Ghahramani [1], applying it to binary classification problems. By taking advantage of the SPGP prior covariance structure, we derive a numerically stable algorithm with O(N M 2 ) training complexity—asymptotically the same as related sparse methods such as the informative vector machine [2], but which more faithfully represents the posterior. We present experimental results for several benchmark problems showing that in many cases this allows an exceptional degree of sparsity without compromising accuracy. Following [1], we locate pseudo-inputs by gradient ascent on the marginal likelihood, but exhibit occasions when this is likely to fail, for which we suggest alternative solutions.
[1] Edward Snelson and Zoubin Ghahramani. Sparse Gaussian processes using pseudo-inputs. In Advances in Neural Information Processing Systems 18. MIT Press, 2005.
[2] Neil Lawrence, Matthias Seeger, and Ralf Herbrich. Fast sparse Gaussian process methods: the informative vector machine. In Advances in Neural Information Processing Systems 15. MIT Press, 2003.
[3] Manfred Opper and Ole Winther. Gaussian processes for classification: mean field methods. Neural Computation, 12(11):2655–2684, 2000.
[4] Volker Tresp. A Bayesian committee machine. Neural Computation, 12(11):2719–2741, 2000.
[5] Alex Smola and Peter Bartlett. Sparse greedy Gaussian process regression. In Advances in Neural Information Processing Systems 13. MIT Press, 2001.
[6] Lehel Csat´ . Gaussian processes: iterative sparse approximations. PhD thesis, Aston University, 2002. o
[7] Matthias Seeger. Bayesian Gaussian process models: PAC-Bayesian generalisation error bounds and sparse approximations. PhD thesis, University of Edinburgh, 2003.
[8] Joaquin Qui˜ onero-Candela and Carl Edward Rasmussen. A unifying view of sparse approximate Gausn sian process regression. Journal of Machine Learning Research, 6(12):1939–1959, 2005.
[9] Carl Rasmussen and Christopher Williams. Gaussian processes for machine learning. MIT Press, 2006.
[10] Malte Kuss and Carl Edward Rasmussen. Assessing approximations for Gaussian process classification. In Advances in Neural Information Processing Systems 18. MIT Press, 2005.
[11] Thomas Minka. A family of algorithms for approximate Bayesian inference. PhD thesis, Massachusetts Institute of Technology, 2001.
[12] Matthias Seeger. Expectation propagation for exponential families, 2005. Available from http://www.cs.berkeley.edu/˜mseeger/papers/epexpfam.ps.gz.
[13] Matthias Seeger, Christopher Williams, and Neil Lawrence. Fast forward selection to speed up sparse Gaussian process regression. In Proceedings of the 9th International Workshop on AI Stats. Society for Artificial Intelligence and Statistics, 2003.
[14] Brian Ripley. Pattern recognition and neural networks. Cambridge University Press, 1996.
[15] Michael E. Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research, 1:211–244, 2001.
[16] Carl Edward Rasmussen and Joaquin Qui˜ onero-Candela. Healing the relevance vector machine through n augmentation. In Proceedings of 22nd ICML. ACM Press, 2005.
[17] Edward Snelson and Zoubin Ghahramani. Variable noise and dimensionality reduction for sparse Gaussian processes. In Proceedings of the 22nd Annual Conference on Uncertainty in AI. AUAI Press, 2006. 8