nips nips2009 nips2009-59 nips2009-59-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Peter Orbanz
Abstract: We consider the general problem of constructing nonparametric Bayesian models on infinite-dimensional random objects, such as functions, infinite graphs or infinite permutations. The problem has generated much interest in machine learning, where it is treated heuristically, but has not been studied in full generality in nonparametric Bayesian statistics, which tends to focus on models over probability distributions. Our approach applies a standard tool of stochastic process theory, the construction of stochastic processes from their finite-dimensional marginal distributions. The main contribution of the paper is a generalization of the classic Kolmogorov extension theorem to conditional probabilities. This extension allows a rigorous construction of nonparametric Bayesian models from systems of finitedimensional, parametric Bayes equations. Using this approach, we show (i) how existence of a conjugate posterior for the nonparametric model can be guaranteed by choosing conjugate finite-dimensional models in the construction, (ii) how the mapping to the posterior parameters of the nonparametric model can be explicitly determined, and (iii) that the construction of conjugate models in essence requires the finite-dimensional models to be in the exponential family. As an application of our constructive framework, we derive a model on infinite permutations, the nonparametric Bayesian analogue of a model recently proposed for the analysis of rank data. 1
[1] H. Bauer. Probability Theory. W. de Gruyter, 1996.
[2] M. J. Beal, Z. Ghahramani, and C. E. Rasmussen. The infinite hidden Markov model. In Advances in Neural Information Processing Systems, 2001.
[3] P. Billingsley. Probability and measure, 1995.
[4] S. R. Dalal and W. J. Hall. Approximating priors by mixtures of natural conjugate priors. Annals of Statistics, 45(2):278–286, 1983.
[5] T. S. Ferguson. A Bayesian analysis of some nonparametric problems. Annals of Statistics, 1(2), 1973.
[6] M. A. Fligner and J. S. Verducci. Distance based ranking models. Journal of the Royal Statistical Society B, 48(3):359–369, 1986.
[7] J. K. Ghosh and R. V. Ramamoorthi. Bayesian Nonparametrics. Springer, 2002.
[8] T. L. Griffiths and Z. Ghahramani. Infinite latent feature models and the Indian buffet process. In Advances in Neural Information Processing Systems, 2005.
[9] M. Meil˘ and L. Bao. Estimation and clustering with infinite rankings. In Uncertainty in a Artificial Intelligence, 2008.
[10] M. M. Rao. Conditional Measures and Applications. Chapman & Hall, second edition, 2005.
[11] C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
[12] C. P. Robert. The Bayesian Choice. Springer, 1994.
[13] D. M. Roy and Y. W. Teh. The Mondrian process. In Advances in Neural Information Processing Systems, 2009.
[14] M. J. Schervish. Theory of Statistics. Springer, 1995.
[15] Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei. Hierarchical Dirichlet processes. Journal of the American Statistical Association, (476):1566–1581, 2006.
[16] S. G. Walker, P. Damien, P. W. Laud, and A. F. M. Smith. Bayesian nonparametric inference for random distributions and related functions. Journal of the Royal Statistical Society B, 61(3):485–527, 1999.
[17] L. Wasserman. All of Nonparametric Statistics. Springer, 2006. 9