nips nips2008 nips2008-237 nips2008-237-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Ilya Sutskever, Geoffrey E. Hinton, Graham W. Taylor
Abstract: The Temporal Restricted Boltzmann Machine (TRBM) is a probabilistic model for sequences that is able to successfully model (i.e., generate nice-looking samples of) several very high dimensional sequences, such as motion capture data and the pixels of low resolution videos of balls bouncing in a box. The major disadvantage of the TRBM is that exact inference is extremely hard, since even computing a Gibbs update for a single variable of the posterior is exponentially expensive. This difficulty has necessitated the use of a heuristic inference procedure, that nonetheless was accurate enough for successful learning. In this paper we introduce the Recurrent TRBM, which is a very slight modification of the TRBM for which exact inference is very easy and exact gradient learning is almost tractable. We demonstrate that the RTRBM is better than an analogous TRBM at generating motion capture and videos of bouncing balls. 1
[1] A.J. Bell and T.J. Sejnowski. An Information-Maximization Approach to Blind Separation and Blind Deconvolution. Neural Computation, 7(6):1129–1159, 1995.
[2] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166, 1994.
[3] G.E. Hinton. Training Products of Experts by Minimizing Contrastive Divergence. Neural Computation, 14(8):1771–1800, 2002.
[4] G.E. Hinton, S. Osindero, and Y.W. Teh. A Fast Learning Algorithm for Deep Belief Nets. Neural Computation, 18(7):1527–1554, 2006.
[5] G.E. Hinton and R.R. Salakhutdinov. Reducing the Dimensionality of Data with Neural Networks. Science, 313(5786):504–507, 2006.
[6] S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, 1997.
[7] S. Osindero and G. Hinton. Modeling image patches with a directed hierarchy of Markov random fields. Advances Neural Information Processing Systems, 2008.
[8] C. Peterson and J.R. Anderson. A mean field theory learning algorithm for neural networks. Complex Systems, 1(5):995–1019, 1987.
[9] L.R. Rabiner. A tutorial on hidden Markov models and selected applications inspeech recognition. Proceedings of the IEEE, 77(2):257–286, 1989.
[10] D.E. Rumelhart, G.E. Hinton, and R.J. Williams. Learning representations by back-propagating errors. Nature, 323(6088):533–536, 1986.
[11] R. Salakhutdinov and I. Murray. On the quantitative analysis of deep belief networks. In Proceedings of the International Conference on Machine Learning, volume 25, 2008.
[12] D. Sontag and T. Jaakkola. New Outer Bounds on the Marginal Polytope. Advances in Neural Information Processing Systems, 2008.
[13] I. Sutskever and G.E. Hinton. Learning multilevel distributed representations for high-dimensional sequences. Proceeding of the Eleventh International Conference on Artificial Intelligence and Statistics, pages 544–551, 2007.
[14] G.W. Taylor, G.E. Hinton, and S. Roweis. Modeling human motion using binary latent variables. Advances in Neural Information Processing Systems, 19:1345–1352, 2007.
[15] T. Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the International Conference on Machine Learning, volume 25, 2008.
[16] M.J. Wainwright, T.S. Jaakkola, and A.S. Willsky. A new class of upper bounds on the log partition function. IEEE Transactions on Information Theory, 51(7):2313–2335, 2005.
[17] M.J. Wainwright and M.I. Jordan. Graphical models, exponential families, and variational inference. UC Berkeley, Dept. of Statistics, Technical Report, 649, 2003.
[18] M. Welling, M. Rosen-Zvi, and G. Hinton. Exponential family harmoniums with an application to information retrieval. Advances in Neural Information Processing Systems, 17:1481–1488, 2005.
[19] J.S. Yedidia, W.T. Freeman, and Y. Weiss. Understanding belief propagation and its generalizations. Exploring Artificial Intelligence in the New Millennium, pages 239–236, 2003.