nips nips2005 nips2005-201 nips2005-201-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Kazuho Watanabe, Sumio Watanabe
Abstract: The Variational Bayesian framework has been widely used to approximate the Bayesian learning. In various applications, it has provided computational tractability and good generalization performance. In this paper, we discuss the Variational Bayesian learning of the mixture of exponential families and provide some additional theoretical support by deriving the asymptotic form of the stochastic complexity. The stochastic complexity, which corresponds to the minimum free energy and a lower bound of the marginal likelihood, is a key quantity for model selection. It also enables us to discuss the effect of hyperparameters and the accuracy of the Variational Bayesian approach as an approximation of the true Bayesian learning. 1
[1] H.Akaike, “Likelihood and Bayes procedure,” Bayesian Statistics, (Bernald J.M. eds.) University Press, Valencia, Spain, pp.143-166, 1980.
[2] H.Attias, ”Inferring parameters and structure of latent variable models by variational bayes,” Proc. of UAI, 1999.
[3] L.D.Brown, “Fundamentals of statistical exponential families,” IMS Lecture NotesMonograph Series, 1986.
[4] Z.Ghahramani, M.J.Beal, “Graphical models and variational methods,” Advanced Mean Field Methods , MIT Press, 2000.
[5] J.A.Hartigan, “A Failure of likelihood asymptotics for normal mixtures,” Proc. of the Berkeley Conference in Honor of J.Neyman and J.Kiefer, Vol.2, 807-810, 1985.
[6] D.J. Mackay, “Bayesian interpolation,” Neural Computation, 4(2), pp.415-447, 1992.
[7] G.McLachlan, D.Peel,”Finite mixture models,” Wiley, 2000.
[8] M.Sato, “Online model selection based on the variational bayes,” Neural Computation, 13(7), pp.1649-1681, 2001.
[9] G.Schwarz, “Estimating the dimension of a model,” Annals of Statistics, 6(2), pp.461464, 1978.
[10] K.Watanabe, S.Watanabe, ”Lower bounds of stochastic complexities in variational bayes learning of gaussian mixture models,” Proc. of IEEE CIS04, pp.99-104, 2004.
[11] K.Watanabe, S.Watanabe, ”Stochastic complexity for mixture of exponential families in variational bayes,” Proc. of ALT05, pp.107-121, 2005.
[12] S.Watanabe,“Algebraic analysis for non-identifiable learning machines,” Neural Computation, 13(4), pp.899-933, 2001.
[13] K.Yamazaki, S.Watanabe, ”Singularities in mixture models and upper bounds of stochastic complexity,” Neural Networks, 16, pp.1029-1038, 2003.