nips nips2010 nips2010-284 nips2010-284-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Mohammad E. Khan, Guillaume Bouchard, Kevin P. Murphy, Benjamin M. Marlin
Abstract: We propose a new variational EM algorithm for fitting factor analysis models with mixed continuous and categorical observations. The algorithm is based on a simple quadratic bound to the log-sum-exp function. In the special case of fully observed binary data, the bound we propose is significantly faster than previous variational methods. We show that EM is significantly more robust in the presence of missing data compared to treating the latent factors as parameters, which is the approach used by exponential family PCA and other related matrix-factorization methods. A further benefit of the variational approach is that it can easily be extended to the case of mixtures of factor analyzers, as we show. We present results on synthetic and real data sets demonstrating several desirable properties of our proposed method. 1
[AX07] A. Ahmed and E. Xing. On tight approximate inference of the logistic-normal topic admixture model. In AI/Statistics, 2007. + [BDdF 03] Kobus Barnard, Pinar Duygulu, Nando de Freitas, David Forsyth, David Blei, and Michael I. Jordan. Matching words and pictures. J. of Machine Learning Research, 3:1107–1135, 2003. [BJ04] W. Buntine and A. Jakulin. Applying Discrete PCA in Data Analysis. In UAI, 2004. [BL06] D. Blei and J. Lafferty. Correlated topic models. In NIPS, 2006. [BNJ03] D. Blei, A. Ng, and M. Jordan. Latent dirichlet allocation. J. of Machine Learning Research, 3:993–1022, 2003. [Boh92] D. Bohning. Multinomial logistic regression algorithm. Annals of the Inst. of Statistical Math., 44:197–200, 1992. [Bou07] G. Bouchard. Efficient bounds for the softmax and applications to approximate inference in hybrid models. In NIPS 2007 Workshop on Approximate Inference in Hybrid Models, 2007. [CDS02] M. Collins, S. Dasgupta, and R. E. Schapire. A generalization of principal components analysis to the exponential family. In NIPS-14, 2002. [Dun07] D. Dunson. Bayesian methods for latent trait modelling of longitudinal data. Stat. Methods Med. Res., 16(5):399–415, Oct 2007. [GCC09] M. Girolami, B. Calderhead, and S.A. Chin. Riemannian manifold hamiltonian monte carlo. Arxiv preprint arXiv:0907.1100, 2009. [GH96] Z. Ghahramani and G. Hinton. The EM algorithm for mixtures of factor analyzers. Technical report, Dept. of Comp. Sci., Uni. Toronto, 1996. [HSC09] P. R. Hahn, J. Scott, and C. Carvahlo. Sparse Factor-Analytic Probit Models. Technical report, Duke, 2009. [JJ96] T. Jaakkola and M. Jordan. A variational approach to Bayesian logistic regression problems and their extensions. In AI/Statistics, 1996. [KPBSK10] S. Koyama, L. Perez-Bolde, C. Shalizi, and R. Kass. Approximate methods for statespace models. Technical report, CMU, 2010. [LT10] J. Li and D. Tao. Simple exponential family PCA. In AI/Statistics, 2010. [MHG08] S. Mohamed, K. Heller, and Z. Ghahramani. Bayesian Exponential Family PCA. In NIPS, 2008. [RMC09] H. Rue, S. Martino, and N. Chopin. Approximate Bayesian Inference for Latent Gaussian Models Using Integrated Nested Laplace Approximations. J. of Royal Stat. Soc. Series B, 71:319–392, 2009. [Row97] S. Roweis. EM algorithms for PCA and SPCA. In NIPS, 1997. [SH03] V. Siivola and A. Honkela. A state-space method for language modeling. In Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 548–553, 2003. [TB99] M. Tipping and C. Bishop. Probabilistic principal component analysis. J. of Royal Stat. Soc. Series B, 21(3):611–622, 1999. [Tip98] M. Tipping. Probabilistic visualization of high-dimensional binary data. In NIPS, 1998. [WCS08] Max Welling, Chaitanya Chemudugunta, and Nathan Sutter. Deterministic latent variable models and their pitfalls. In Intl. Conf. on Data Mining, 2008. [WK01] Michel Wedel and Wagner Kamakura. Factor analysis with (mixed) observed and latent variables in the exponential family. Psychometrika, 66(4):515–530, December 2001. [YT04] K. Yu and V. Tresp. Heterogenous data fusion via a probabilistic latent-variable model. In Organic and Pervasive Computing (ARCS 2004), 2004. 9