nips nips2000 nips2000-13 nips2000-13-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Martijn A. R. Leisink, Hilbert J. Kappen
Abstract: We present a method to bound the partition function of a Boltzmann machine neural network with any odd order polynomial. This is a direct extension of the mean field bound, which is first order. We show that the third order bound is strictly better than mean field. Additionally we show the rough outline how this bound is applicable to sigmoid belief networks. Numerical experiments indicate that an error reduction of a factor two is easily reached in the region where expansion based approximations are useful. 1
[1) J. Pearl. Probabilistic Reasoning in Intelligent Systems, chapter 8.2.1, pages 387- 390. Morgan Kaufmann, San Francisco, 1988. [2) S.K. Saul, T.S. Jaakkola, and M.l. Jordan. Mean field theory for sigmoid belief networks. Technical R eport 1, Computational Cognitive Science, 1995. [3) R. Neil. Connectionist learning of belief networks. Artificial intelligence, 56:71- 113, 1992. [4) D. Ackley, G. Hinton, and T. Sejnowski. A learning algorithm for Boltzmann machines. Cognitive Science, 9:147-169, 1985. [5) C. P eterson and J . Anderson. A mean field theory learning algorithm for neural networks. Complex systems, 1:995- 1019, 1987. [6) D.J. Thouless, P.W. Andersson, and R.G. Palmer. Solution of 'solvable model of a spin glass'. Philisophi cal Magazine, 35(3):593-601, 1977. [7) H.J . Kappen and F .B. Rodriguez. Boltzmann machine learning using mean field theory and linear response correction. In M.S. Kearns, S.A. Solla, and D .A . Cohn, editors, Advances in Neural Tnformation Processing Systems, volume 11, pages 280286. MIT Press, 1999. [8) D. Sherrington and S. Kirkpatrick. Solvable model of a spin-glass. Physical R eview Letters, 35(26):1793- 1796,121975. [9) M.A.R. Leisink and H.J. Kappen. Validity of TAP equations in neural networks. In ICANN 99, volume 1, pages 425- 430, ISBN 0852967217, 1999. Insti tution of Electrical Engineers, London. [10) D. Barber and W. Wiegerinck. Tractable variational structures for approximating graphical models. In M .S. Kearns, S.A. Solla, and D.A . Cohn, editors, Advances in Neural Information Processin g Systems, volume 11, pages 183- 189. MIT Press, 1999.