nips nips2006 nips2006-57 nips2006-57-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Peter Carbonetto, Nando D. Freitas
Abstract: Despite all the attention paid to variational methods based on sum-product message passing (loopy belief propagation, tree-reweighted sum-product), these methods are still bound to inference on a small set of probabilistic models. Mean field approximations have been applied to a broader set of problems, but the solutions are often poor. We propose a new class of conditionally-specified variational approximations based on mean field theory. While not usable on their own, combined with sequential Monte Carlo they produce guaranteed improvements over conventional mean field. Moreover, experiments on a well-studied problem— inferring the stable configurations of the Ising spin glass—show that the solutions can be significantly better than those obtained using sum-product-based methods. 1
[1] S. M. Aji and R. J. McEliece. The Generalized distributive law and free energy minimization. In Proceedings of the 39th Allerton Conference, pages 672–681, 2001.
[2] B. Arnold, E. Castillo, and J.-M. Sarabia. Conditional Specification of Statistical Models. Springer, 1999.
[3] J. Besag. Spatial interaction and the statistical analysis of lattice systems. J. Roy. Statist. Soc., Ser. B, 36:192–236, 1974.
[4] J. Besag. Comment to “Conditionally specified distributions”. Statist. Sci., 16:265–267, 2001.
[5] W. Buntine and A. Jakulin. Applying discrete PCA in data analysis. In Uncertainty in Artificial Intelligence, volume 20, pages 59–66, 2004.
[6] N. de Freitas, P. Højen-Sørensen, M. I. Jordan, and S. Russell. Variational MCMC. In Uncertainty in Artificial Intelligence, volume 17, pages 120–127, 2001.
[7] P. del Moral, A. Doucet, and A. Jasra. Sequential Monte Carlo samplers. J. Roy. Statist. Soc., Ser. B, 68:411–436, 2006.
[8] Z. Ghahramani and M. J. Beal. Variational inference for Bayesian mixtures of factor analysers. In Advances in Neural Information Processing Systems, volume 12, pages 449–455, 1999.
[9] F. Hamze and N. de Freitas. Hot Coupling: a particle approach to inference and normalization on pairwise undirected graphs. Advances in Neural Information Processing Systems, 18:491–498, 2005.
[10] T. Heskes, K. Albers, and B. Kappen. Approximate inference and constrained optimization. In Uncertainty in Artificial Intelligence, volume 19, pages 313–320, 2003.
[11] C. Jarzynski. Nonequilibrium equality for free energy differences. Phys. Rev. Lett., 78:2690–2693, 1997.
[12] M. Jerrum and A. Sinclair. The Markov chain Monte Carlo method: an approach to approximate counting and integration. In Approximation Algorithms for NP-hard Problems, pages 482–520. PWS Pubs., 1996.
[13] G. Kitagawa. Monte Carlo filter and smoother for non-Gaussian nonlinear state space models. J. Comput. Graph. Statist., 5:1–25, 1996.
[14] P. Muyan and N. de Freitas. A blessing of dimensionality: measure concentration and probabilistic inference. In Proceedings of the 19th Workshop on Artificial Intelligence and Statistics, 2003.
[15] R. M. Neal. Annealed importance sampling. Statist. and Comput., 11:125–139, 2001.
[16] M. Newman and G. Barkema. Monte Carlo Methods in Statistical Physics. Oxford Univ. Press, 1999.
[17] M. Opper and D. Saad, editors. Advanced Mean Field Methods, Theory and Practice. MIT Press, 2001.
[18] C. P. Robert and G. Casella. Monte Carlo Statistical Methods. Springer, 2nd edition, 2004.
[19] M. N. Rosenbluth and A. W. Rosenbluth. Monte Carlo calculation of the average extension of molecular chains. J. Chem. Phys., 23:356–359, 1955.
[20] J. S. Sadowsky and J. A. Bucklew. On large deviations theory and asymptotically efficient Monte Carlo estimation. IEEE Trans. Inform. Theory, 36:579–588, 1990.
[21] L. K. Saul, T. Jaakola, and M. I. Jordan. Mean field theory for sigmoid belief networks. J. Artificial Intelligence Res., 4:61–76, 1996.
[22] L. K. Saul and M. I. Jordan. Exploiting tractable structures in intractable networks. In Advances in Neural Information Processing Systems, volume 8, pages 486–492, 1995.
[23] E. B. Sudderth, A. T. Ihler, W. T. Freeman, and A. S. Willsky. Nonparametric belief propagation. In Computer Vision and Pattern Recognition,, volume I, pages 605–612, 2003.
[24] M. J. Wainwright, T. S. Jaakkola, and A. S. Willsky. A new class of upper bounds on the log partition function. IEEE Trans. Inform. Theory, 51:2313–2335, 2005.
[25] M. J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational inference. Technical report, EECS Dept., University of California, Berkeley, 2003.
[26] W. Wiegerinck. Variational approximations between mean field theory and the junction tree algorithm. In Uncertainty in Artificial Intelligence, volume 16, pages 626–633, 2000.