jmlr jmlr2006 jmlr2006-87 jmlr2006-87-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Kazuho Watanabe, Sumio Watanabe
Abstract: Bayesian learning has been widely used and proved to be effective in many data modeling problems. However, computations involved in it require huge costs and generally cannot be performed exactly. The variational Bayesian approach, proposed as an approximation of Bayesian learning, has provided computational tractability and good generalization performance in many applications. The properties and capabilities of variational Bayesian learning itself have not been clarified yet. It is still unknown how good approximation the variational Bayesian approach can achieve. In this paper, we discuss variational Bayesian learning of Gaussian mixture models and derive upper and lower bounds of variational stochastic complexities. The variational stochastic complexity, which corresponds to the minimum variational free energy and a lower bound of the Bayesian evidence, not only becomes important in addressing the model selection problem, but also enables us to discuss the accuracy of the variational Bayesian approach as an approximation of true Bayesian learning. Keywords: Gaussian mixture model, variational Bayesian learning, stochastic complexity
H. Akaike. Likelihood and bayes procedure. In Bayesian Statistics, pages 143–166, Valencia, Spain, 1980. (Bernald J.M. eds.), University Press. H. Alzer. On some inequalities for the Gamma and Psi functions. Mathematics of computation, 66 (217):373–389, 1997. H. Attias. Inferring parameters and structure of latent variable models by variational bayes. In Proceedings of Uncertainty in Artificial Intelligence(UAI’99), 1999. M. J. Beal. Variational Algorithms for approximate Bayesian inference. PhD thesis, University College London, 2003. D. Dacunha-Castelle and E. Gassiat. Testing in locally conic models, and application to mixture models. Probability and Statistics, 1:285–317, 1997. Z. Ghahramani and M. J. Beal. Graphical models and variational methods. Advanced Mean Field Methods – Theory and Practice, eds. D. Saad and M. Opper, MIT Press, 2000. J. A. Hartigan. A failure of likelihood asymptotics for normal mixtures. In Proceedings of the Berkeley Conference in Honor of J.Neyman and J.Kiefer, pages 807–810, 1985. 642 S TOCHASTIC C OMPLEXITIES OF G AUSSIAN M IXTURES E. Levin, N. Tishby, and S. A. Solla. A statistical approaches to learning and generalization in layered neural networks. Proc. of IEEE, 78(10):1568–1674, 1990. D. J. Mackay. Bayesian interpolation. Neural Computation, 4(2):415–447, 1992. J. Rissanen. Stochastic complexity and modeling. Annals of Statistics, 14(3):1080–1100, 1986. M. Sato. Online model selection based on the variational bayes. Neural Computation, 13(7):1649– 1681, 2001. G. Schwarz. Estimating the dimension of a model. Annals of Statistics, 6(2):461–464, 1978. K. Watanabe and S. Watanabe. Lower bounds of stochastic complexities in variational bayes learning of gaussian mixture models. In Proceedings of IEEE conference on Cybernetics and Intelligent Systems (CIS04), pages 99–104, Singapore, 2004. S. Watanabe. Algebraic analysis for non-identifiable learning machines. Neural Computation, 13 (4):899–933, 2001. S. Watanabe, K. Yamazaki, and M. Aoyagi. Kullback information of normal mixture is not an analytic function. Technical Report of IEICE (in Japanese), NC2004-50:41–46, 2004. K. Yamazaki and S. Watanabe. Singularities in mixture models and upper bounds of stochastic complexity. International Journal of Neural Networks, 16:1023–1038, 2003a. K. Yamazaki and S. Watanabe. Stochastic complexity of bayesian networks. In Proceedings of Uncertainty in Artificial Intelligence(UAI’03), 2003b. 643