nips nips2013 nips2013-317 nips2013-317-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Tamara Broderick, Nicholas Boyd, Andre Wibisono, Ashia C. Wilson, Michael Jordan
Abstract: We present SDA-Bayes, a framework for (S)treaming, (D)istributed, (A)synchronous computation of a Bayesian posterior. The framework makes streaming updates to the estimated posterior according to a user-specified approximation batch primitive. We demonstrate the usefulness of our framework, with variational Bayes (VB) as the primitive, by fitting the latent Dirichlet allocation model to two large-scale document collections. We demonstrate the advantages of our algorithm over stochastic variational inference (SVI) by comparing the two after a single pass through a known amount of data—a case where SVI may be applied—and in the streaming setting, where SVI does not apply. 1
[1] F. Niu, B. Recht, C. R´ , and S. J. Wright. Hogwild!: A lock-free approach to parallelizing stochastic e gradient descent. In Neural Information Processing Systems, 2011.
[2] A. Kleiner, A. Talwalkar, P. Sarkar, and M. Jordan. The big data bootstrap. In International Conference on Machine Learning, 2012.
[3] M. Hoffman, D. M. Blei, and F. Bach. Online learning for latent Dirichlet allocation. In Neural Information Processing Systems, volume 23, pages 856–864, 2010.
[4] M. Hoffman, D. M. Blei, J. Paisley, and C. Wang. Stochastic variational inference. Journal of Machine Learning Research, 14:1303–1347.
[5] C. Wang, J. Paisley, and D. M. Blei. Online variational inference for the hierarchical Dirichlet process. In Artificial Intelligence and Statistics, 2011.
[6] M. J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2):1–305, 2008.
[7] T. P. Minka. Expectation propagation for approximate Bayesian inference. In Uncertainty in Artificial Intelligence, pages 362–369. Morgan Kaufmann, 2001.
[8] T. P. Minka. A family of algorithms for approximate Bayesian inference. PhD thesis, Massachusetts Institute of Technology, 2001.
[9] M. Opper. A Bayesian approach to on-line learning.
[10] K. R Canini, L. Shi, and T. L Griffiths. Online inference of topics with latent Dirichlet allocation. In Artificial Intelligence and Statistics, volume 5, 2009.
[11] A. Honkela and H. Valpola. On-line variational Bayesian learning. In International Symposium on Independent Component Analysis and Blind Signal Separation, pages 803–808, 2003.
[12] J. Luts, T. Broderick, and M. P. Wand. Real-time semiparametric regression. Journal of Computational and Graphical Statistics, to appear. Preprint arXiv:1209.3550.
[13] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022, 2003.
[14] T. Minka and J. Lafferty. Expectation-propagation for the generative aspect model. In Uncertainty in Artificial Intelligence, pages 352–359. Morgan Kaufmann, 2002.
[15] Y. Teh, D. Newman, and M. Welling. A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation. In Neural Information Processing Systems, 2006.
[16] A. Asuncion, M. Welling, P. Smyth, and Y. Teh. On smoothing and inference for topic models. In Uncertainty in Artificial Intelligence, 2009.
[17] M. Hoffman. Online inference for LDA (Python code) at http://www.cs.princeton.edu/˜blei/downloads/onlineldavb.tar, 2010.
[18] R. Ranganath, C. Wang, D. M. Blei, and E. P. Xing. An adaptive learning rate for stochastic variational inference. In International Conference on Machine Learning, 2013.
[19] W. L. Buntine and A. Jakulin. Applying discrete PCA in data analysis. In Uncertainty in Artificial Intelligence.
[20] M. Seeger. Expectation propagation for exponential families. Technical report, University of California at Berkeley, 2005. 9