nips nips2012 nips2012-365 nips2012-365-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Philip Sterne, Joerg Bornschein, Abdul-saboor Sheikh, Joerg Luecke, Jacquelyn A. Shelton
Abstract: Modelling natural images with sparse coding (SC) has faced two main challenges: flexibly representing varying pixel intensities and realistically representing lowlevel image components. This paper proposes a novel multiple-cause generative model of low-level image statistics that generalizes the standard SC model in two crucial points: (1) it uses a spike-and-slab prior distribution for a more realistic representation of component absence/intensity, and (2) the model uses the highly nonlinear combination rule of maximal causes analysis (MCA) instead of a linear combination. The major challenge is parameter optimization because a model with either (1) or (2) results in strongly multimodal posteriors. We show for the first time that a model combining both improvements can be trained efficiently while retaining the rich structure of the posteriors. We design an exact piecewise Gibbs sampling method and combine this with a variational method based on preselection of latent dimensions. This combined training scheme tackles both analytical and computational intractability and enables application of the model to a large number of observed and hidden dimensions. Applying the model to image patches we study the optimal encoding of images by simple cells in V1 and compare the model’s predictions with in vivo neural recordings. In contrast to standard SC, we find that the optimal prior favors asymmetric and bimodal activity of simple cells. Testing our model for consistency we find that the average posterior is approximately equal to the prior. Furthermore, we find that the model predicts a high percentage of globular receptive fields alongside Gabor-like fields. Similarly high percentages are observed in vivo. Our results thus argue in favor of improvements of the standard sparse coding model for simple cells by using flexible priors and nonlinear combinations. 1
[1] B. Olshausen and D. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381:607–9, 1996.
[2] D. H. Hubel and T. N. Wiesel. Receptive fields of single neurones in the cat’s striate cortex. The Journal of Physiology, 1959. 8
[3] M. Seeger. Bayesian inference and optimal design for the sparse linear model. Journal of Machine Learning Research, 9:759–813, June 2008.
[4] H. Lee, A. Battle, R. Raina, and A. Ng. Efficient sparse coding algorithms. In Advances in Neural Information Processing Systems, volume 20, pages 801–08, 2007.
[5] M. Titsias and M. L´ zaro-Gredilla. Spike and slab variational inference for multi-task and multiple kernel a learning. In Advances in Neural Information Processing Systems, 2011.
[6] S. Mohamed, K. Heller, and Z. Ghahramani. Evaluating Bayesian and L1 approaches for sparse unsupervised learning. In ICML, 2012.
[7] I. Goodfellow, A. Courville, and Y. Bengio. Large-scale feature learning with spike-and-slab sparse coding. In ICML, 2012.
[8] J¨ rg L¨ cke and Abdul-Saboor Sheikh. Closed-form EM for sparse coding and its application to source o u separation. In LVA/ICA, LNCS, pages 213–221. Springer, 2012.
[9] E. Saund. A multiple cause mixture model for unsupervised learning. Neural Computation, 1995.
[10] P. Dayan and R. S. Zemel. Competition and multiple cause models. Neural Computation, 1995.
[11] J. L¨ cke and M. Sahani. Maximal causes for non-linear component extraction. Journal of Machine u Learning Research, 9:1227–67, 2008.
[12] G. Puertas, J. Bornschein, and J. L¨ cke. The maximal causes of natural scenes are edge filters. In u Advances in Neural Information Processing Systems, volume 23, pages 1939–47. 2010.
[13] I. Goodfellow, A. Courville, and Y. Bengio. Spike-and-slab sparse coding for unsupervised feature discovery. In NIPS Workshop on Challenges in Learning Hierarchical Models. 2011.
[14] J¨ rg L¨ cke and Julian Eggert. Expectation truncation and the benefits of preselection in training generative o u models. Journal of Machine Learning Research, 11:2855–900, 2010.
[15] J. Shelton, J. Bornschein, A.-S. Sheikh, P. Berkes, and J. L¨ cke. Select and sample - a model of efficient u neural inference and learning. Advances in Neural Information Processing Systems, 24, 2011.
[16] R. Neal and G. Hinton. A view of the EM algorithm that justifies incremental, sparse, and other variants. In M. I. Jordan, editor, Learning in Graphical Models. Kluwer, 1998.
[17] B. Olshausen and K. Millman. Learning sparse codes with a mixture-of-Gaussians prior. Advances in Neural Information Processing Systems, 12:841–847, 2000.
[18] X. Tan, J. Li, and P. Stoica. Efficient sparse Bayesian learning via Gibbs sampling. In ICASSP, pages 3634–3637, 2010.
[19] J. Bornschein, Z. Dai, and J. L¨ cke. Approximate EM learning on large computer clusters. In NIPS u Workshop: LCCC. 2010.
[20] J. H. van Hateren and A. van der Schaaf. Independent component filters of natural images compared with simple cells in primary visual cortex. Proceedings of the Royal Society of London B, 265:359–66, 1998.
[21] D. Ringach. Spatial structure and symmetry of simple-cell receptive fields in macaque primary visual cortex. Journal of Neurophysiology, 88:455–63, 2002.
[22] W. M. Usrey, M. P. Sceniak, and B. Chapman. Receptive Fields and Response Properties of Neurons in Layer 4 of Ferret Visual Cortex. Journal of Neurophysiology, 89:1003–1015, 2003.
[23] C. Niell and M. Stryker. Highly Selective Receptive Fields in Mouse Visual Cortex. The Journal of Neuroscience, 28(30):7520–7536, 2008.
[24] P. Berkes, G. Orban, M. Lengyel, and J. Fiser. Spontaneous Cortical Activity Reveals Hallmarks of an Optimal Internal Model of the Environment. Science, 331(6013):83–87, January 2011.
[25] B. Olshausen and D. Field. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research, 37(23):3311–3325, December 1997.
[26] P. Berkes, R. Turner, and M. Sahani. On sparsity and overcompleteness in image models. Advances in Neural Information Processing Systems, 21, 2008.
[27] M. Rehn and F. Sommer. A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive fields. Journal of Computational Neuroscience, 22(2):135–46, 2007.
[28] J. L¨ cke. A dynamical model for receptive field self-organization in V1 cortical columns. In Proc. u International Conference on Artificial Neural Networks, LNCS 4669, pages 389 – 398. Springer, 2007.
[29] B. A. Olshausen, C. Cadieu, and D.K. Warland. Learning real and complex overcomplete representations from the statistics of natural images. Proc. SPIE, (7446), 2009.
[30] J. L¨ cke. Receptive field self-organization in a model of the fine-structure in V1 cortical columns. Neural u Computation, 21(10):2805–45, 2009.
[31] M. Henniges, G. Puertas, J. Bornschein, J. Eggert, and J. L¨ cke. Binary Sparse Coding. In Proceedings u LVA/ICA, LNCS 6365, pages 450–57. Springer, 2010.
[32] J. Zylberberg, J. Murphy, and M. Deweese. A Sparse Coding Model with Synaptically Local Plasticity and Spiking Neurons Can Account for the Diverse Shapes of V1 Simple Cell Receptive Fields. PLoS Computational Biology, 7(10):e1002250, 2011.
[33] M. Zhou, H. Chen, J. Paisley, L. Ren, G. Sapiro, and L. Carin. Non-parametric Bayesian dictionary learning for sparse image representations 1. In NIPS Workshop. 2009. 9