nips nips2000 nips2000-64 nips2000-64-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Oliver B. Downs
Abstract: Recent work has exploited boundedness of data in the unsupervised learning of new types of generative model. For nonnegative data it was recently shown that the maximum-entropy generative model is a Nonnegative Boltzmann Distribution not a Gaussian distribution, when the model is constrained to match the first and second order statistics of the data. Learning for practical sized problems is made difficult by the need to compute expectations under the model distribution. The computational cost of Markov chain Monte Carlo methods and low fidelity of naive mean field techniques has led to increasing interest in advanced mean field theories and variational methods. Here I present a secondorder mean-field approximation for the Nonnegative Boltzmann Machine model, obtained using a
[1] Downs, DB, MacKay, DJC, & Lee, DD (2000). The Nonnegative Boltzmann Machine. Advances in Neural Information Processing Systems 12, 428-434.
[2] Lee, DD, and Seung, HS (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401,788-791.
[3] Socci, ND, Lee, DD, and Seung, HS (1998). The rectified Gaussian distribution. Advances in Neural Information Processing Systems 10, 350-356.
[4] Georges, A, & Yedidia, JS (1991). How to expand around mean-field theory using hightemperature expansions. Journal of Physics A 24, 2173- 2192.
[5] Neal, RM (1997). Markov chain Monte Carlo methods based on 'slicing' the density function. Technical Report 9722, Dept. of Statistics, University of Toronto.
[6] Kappen, HJ & Rodriguez, FB (1998). Efficient learning in Boltzmann Machines using linear response theory. Neural Computation 10, 1137-1156.
[7] Ben-Yishai, R, Bar-Or, RL, & Sompolinsky, H (1995). Theory of orientation tuning in visual cortex. Proc. Nat. Acad. Sci. USA,92(9):3844-3848.
[8] Yedidia, JS , Freeman, WT, & Weiss, Y (2000). Generalized Belief Propagation. Mitsubishi Electric Research Laboratory Technical Report, TR-2000-26.