nips nips2004 nips2004-63 nips2004-63-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Manfred Opper, Ole Winther
Abstract: We propose a novel a framework for deriving approximations for intractable probabilistic models. This framework is based on a free energy (negative log marginal likelihood) and can be seen as a generalization of adaptive TAP [1, 2, 3] and expectation propagation (EP) [4, 5]. The free energy is constructed from two approximating distributions which encode different aspects of the intractable model such a single node constraints and couplings and are by construction consistent on a chosen set of moments. We test the framework on a difficult benchmark problem with binary variables on fully connected graphs and 2D grid graphs. We find good performance using sets of moments which either specify factorized nodes or a spanning tree on the nodes (structured approximation). Surprisingly, the Bethe approximation gives very inferior results even on grids. 1
[1] M. Opper and O. Winther, “Gaussian processes for classification: Mean field algorithms,” Neural Computation, vol. 12, pp. 2655–2684, 2000.
[2] M. Opper and O. Winther, “Tractable approximations for probabilistic models: The adaptive Thouless-Anderson-Palmer mean field approach,” Phys. Rev. Lett., vol. 86, pp. 3695, 2001.
[3] M. Opper and O. Winther, “Adaptive and self-averaging Thouless-Anderson-Palmer mean field theory for probabilistic modeling,” Phys. Rev. E, vol. 64, pp. 056131, 2001.
[4] T. P. Minka, “Expectation propagation for approximate Bayesian inference,” in UAI 2001, 2001, pp. 362–369.
[5] T. Minka and Y. Qi, “Tree-structured approximations by expectation propagation,” in NIPS 16, S. Thrun, L. Saul, and B. Sch¨ lkopf, Eds. MIT Press, Cambridge, MA, 2004. o
[6] Christopher M. Bishop, David Spiegelhalter, and John Winn, “Vibes: A variational inference engine for bayesian networks,” in Advances in Neural Information Processing Systems 15, S. Thrun S. Becker and K. Obermayer, Eds., pp. 777–784. MIT Press, Cambridge, MA, 2003.
[7] H. Attias, “A variational Bayesian framework for graphical models,” in Advances in Neural Information Processing Systems 12, T. Leen et al., Ed. 2000, MIT Press, Cambridge.
[8] J. S. Yedidia, W. T. Freeman, and Y. Weiss, “Generalized belief propagation,” in Advances in Neural Information Processing Systems 13, T. K. Leen, T. G. Dietterich, and V. Tresp, Eds., 2001, pp. 689–695.
[9] A. L. Yuille, “CCCP algorithms to minimize the Bethe and Kikuchi free energies: convergent alternatives to belief propagation,” Neural Comput., vol. 14, no. 7, pp. 1691–1722, 2002.
[10] T. Heskes, K. Albers, and H. Kappen, “Approximate inference and constrained optimization,” in UAI-03, San Francisco, CA, 2003, pp. 313–320, Morgan Kaufmann Publishers.
[11] S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004.
[12] M. J. Wainwright and M. I. Jordan, “Semidefinite methods for approximate inference on graphs with cycles,” Tech. Rep. UCB/CSD-03-1226, UC Berkeley CS Division, 2003.