nips nips2012 nips2012-192 nips2012-192-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Yunlong He, Yanjun Qi, Koray Kavukcuoglu, Haesun Park
Abstract: In this paper, we study latent factor models with dependency structure in the latent space. We propose a general learning framework which induces sparsity on the undirected graphical model imposed on the vector of latent factors. A novel latent factor model SLFA is then proposed as a matrix factorization problem with a special regularization term that encourages collaborative reconstruction. The main benefit (novelty) of the model is that we can simultaneously learn the lowerdimensional representation for data and model the pairwise relationships between latent factors explicitly. An on-line learning algorithm is devised to make the model feasible for large-scale learning problems. Experimental results on two synthetic data and two real-world data sets demonstrate that pairwise relationships and latent factors learned by our model provide a more structured way of exploring high-dimensional data, and the learned representations achieve the state-of-the-art classification performance. 1
[1] Bertsekas, D.: Nonlinear programming. Athena Scientific Belmont, MA (1999)
[2] Blei, D., Lafferty, J.: Correlated topic models. Advances in Neural Information Processing Systems (2006)
[3] Chandrasekaran, V., Parrilo, P., Willsky, A.: Latent variable graphical model selection via convex optimization. Arxiv preprint arXiv:1008.1290 (2010)
[4] Collins, M., Dasgupta, S., Schapire, R.: A generalization of principal component analysis to the exponential family. Advances in neural information processing systems (2002)
[5] Fan, R., Chang, K., Hsieh, C., Wang, X., Lin, C.: Liblinear: A library for large linear classification. JMLR (2008)
[6] Goldfarb, D., Ma, S., Scheinberg, K.: Fast alternating linearization methods for minimizing the sum of two convex functions. Arxiv preprint arXiv:0912.4571 (2009)
[7] Gregor, K., Szlam, A., LeCun, Y.: Structured sparse coding via lateral inhibition. Advances in Neural Information Processing Systems 24 (2011)
[8] Hinton, G., Osindero, S., Bao, K.: Learning causally linked markov random fields. In: AI & Statistics (2005)
[9] Hsieh, C., Sustik, M., Ravikumar, P., Dhillon, I.: Sparse inverse covariance matrix estimation using quadratic approximation. Advances in Neural Information Processing Systems (NIPS) 24 (2011)
[10] Hyv¨ rinen, A., Hurri, J., Hoyer, P.: Independent component analysis. Natural Image Statistics (2009) a
[11] Jacob, L., Obozinski, G., Vert, J.: Group lasso with overlap and graph lasso. Proceedings of the 26th Annual International Conference on Machine Learning (2009)
[12] Jenatton, R., Mairal, J., Obozinski, G., Bach, F.: Proximal methods for sparse hierarchical dictionary learning. Proceedings of the International Conference on Machine Learning (2010)
[13] Karklin, Y., Lewicki, M.S.: Emergence of complex cell properties by learning to generalize in natural scenes. Nature (2009)
[14] Lee, D., Seung, H.: Learning the parts of objects by non-negative matrix factorization. Nature (1999)
[15] Lee, H., Battle, A., Raina, R., Ng, A.: Efficient sparse coding algorithms. Advances in neural information processing systems (2007)
[16] Lee, H., Raina, R., Teichman, A., Ng, A.: Exponential family sparse coding with applications to selftaught learning. Proceedings of the 21st international jont conference on Artifical intelligence (2009)
[17] Lyu, S., Simoncelli, E.: Nonlinear extraction of independent components of natural images using radial gaussianization. Neural computation (2009)
[18] Murray, I., Adams, R.: Slice sampling covariance hyperparameters of latent gaussian models. Arxiv preprint arXiv:1006.0868 (2010)
[19] Olshausen, B., et al.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature (1996)
[20] Rendl, F., Rinaldi, G., Wiegele, A.: Solving Max-Cut to optimality by intersecting semidefinite and polyhedral relaxations. Math. Programming 121(2), 307 (2010)
[21] Schmidt, M.: Graphical model structure learning with l1-regularization. Ph.D. thesis, UNIVERSITY OF BRITISH COLUMBIA (2010)
[22] Schmidt, M., Van Den Berg, E., Friedlander, M., Murphy, K.: Optimizing costly functions with simple constraints: A limited-memory projected quasi-newton algorithm. In: AI & Statistics (2009)
[23] Silva, R., Scheine, R., Glymour, C., Spirtes, P.: Learning the structure of linear latent variable models. The Journal of Machine Learning Research 7, 191–246 (2006)
[24] Stegle, O., Lippert, C., Mooij, J., Lawrence, N., Borgwardt, K.: Efficient inference in matrix-variate gaussian models with iid observation noise. Advances in Neural Information Processing Systems (2011)
[25] Teh, Y., Seeger, M., Jordan, M.: Semiparametric latent factor models. In: AI & Statistics (2005)
[26] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological) (1996)
[27] Wainwright, M., Simoncelli, E.: Scale mixtures of gaussians and the statistics of natural images. Advances in neural information processing systems (2000)
[28] Yuan, M., Lin, Y.: Model selection and estimation in the gaussian graphical model. Biometrika (2007) 9