nips nips2000 nips2000-114 nips2000-114-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Hilbert J. Kappen, Wim Wiegerinck
Abstract: In this paper, we derive a second order mean field theory for directed graphical probability models. By using an information theoretic argument it is shown how this can be done in the absense of a partition function. This method is a direct generalisation of the well-known TAP approximation for Boltzmann Machines. In a numerical example, it is shown that the method greatly improves the first order mean field approximation. For a restricted class of graphical models, so-called single overlap graphs, the second order method has comparable complexity to the first order method. For sigmoid belief networks, the method is shown to be particularly fast and effective.
Barber, D. and Wiegerinck, W. (1999). Tractable variational structures for approximating graphical models. In Kearns, M., Solla, S ., and Cohn, D., editors, Advances in Neural Information Processing Systems, volume 11 of Advances in Neural Information Processing Systems, pages 183- 189. MIT Press. Kappen, H. and Rodriguez, F. (1998). Efficient learning in Boltzmann Machines using linear response theory. Neural Computation, 10:1137-1156. Kappen, H. and Spanjers, J. (1999). Mean field theory for asymmetric neural networks. Physical Review E, 61 :5658-5663. Kappen, H. and Wiegerinck, W. (2001). Mean field theory for graphical models. In Saad, D. and Opper, M., editors, Advanced mean field theory. MIT Press. Lauritzen, S. and Spiegelhalter, D. (1988). Local computations with probabilties on graphical structures and their application to expert systems. J. Royal Statistical society B, 50: 154-227. Plefka, T. (1982). Convergence condition of the TAP equation for the infinite-range Ising spin glass model. Journal of Physics A, 15:1971- 1978. Saul, L., Jaakkola, T., and Jordan, M. (1996) . Mean field theory for sigmoid belief networks. Journal of anificial intelligence research, 4:61-76. Thouless, D., Anderson, P., and Palmer, R. (1977). Solution of 'Solvable Model of a Spin Glass'. Phil. Mag., 35:593- 601. Wiegerinck, W. and Kappen, H. (1999) . Approximations of bayesian networks through kl minimisation. New Generation Computing, 18:167- 175.