nips nips2012 nips2012-79 nips2012-79-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Xaq Pitkow
Abstract: This paper shows how sparse, high-dimensional probability distributions could be represented by neurons with exponential compression. The representation is a novel application of compressive sensing to sparse probability distributions rather than to the usual sparse signals. The compressive measurements correspond to expected values of nonlinear functions of the probabilistically distributed variables. When these expected values are estimated by sampling, the quality of the compressed representation is limited only by the quality of sampling. Since the compression preserves the geometric structure of the space of sparse probability distributions, probabilistic computation can be performed in the compressed domain. Interestingly, functions satisfying the requirements of compressive sensing can be implemented as simple perceptrons. If we use perceptrons as a simple model of feedforward computation by neurons, these results show that the mean activity of a relatively small number of neurons can accurately represent a highdimensional joint distribution implicitly, even without accounting for any noise correlations. This comprises a novel hypothesis for how neurons could encode probabilities in the brain. 1
[1] Ma W, Beck J, Latham P, Pouget A (2006) Bayesian inference with probabilistic population codes. Nat Neurosci 9: 1432–8.
[2] Berkes P, Orb´ n G, Lengyel M, Fiser J (2011) Spontaneous cortical activity reveals hallmarks of an a optimal internal model of the environment. Science 331: 83–7.
[3] Cand` s E, Romberg J, Tao T (2006) Robust uncertainty principles: Exact signal reconstruction from e highly incomplete frequency information. IEEE Transactions on Information Theory 52: 489–509.
[4] Cand` s E, Tao T (2006) Near-optimal signal recovery from random projections: Universal encoding e strategies? IEEE Transactions on Information Theory 52: 5406–5425.
[5] Donoho D (2006) Compressed sensing. IEEE Transactions on Information Theory 52: 1289–1306.
[6] Cand` s E, Plan Y (2011) A probabilistic and RIPless theory of compressed sensing. IEEE Transactions e on Information Theory 57: 7235–7254.
[7] Donoho DL, Maleki A, Montanari A (2009) Message-passing algorithms for compressed sensing. Proc Natl Acad Sci USA 106: 18914–9.
[8] Cand` s E, Wakin M (2008) An introduction to compressive sampling. Signal Processing Magazine 25: e 21–30.
[9] Kueng R, Gross D (2012) RIPless compressed sensing from anisotropic measurements. Arxiv preprint arXiv:12051423 .
[10] Calderbank R, Howard S, Jafarpour S (2010) Construction of a large class of deterministic sensing matrices that satisfy a statistical isometry property. Selected Topics in Signal Processing 4: 358–374.
[11] Gurevich S, Hadani R (2009) Statistical rip and semi-circle distribution of incoherent dictionaries. arXiv cs.IT.
[12] Mendelson S, Pajor A, Tomczak-Jaegermann N (2006) Uniform uncertainty principle for Bernoulli and subgaussian ensembles. arXiv math.ST.
[13] Irmatov A (2009) Bounds for the number of threshold functions. Discrete Mathematics and Applications 6: 569–583.
[14] Ackley D, Hinton G, Sejnowski T (1985) A learning algorithm for Boltzmann machines. Cognitive Science 9: 147–169.
[15] Glauber RJ (1963) Time-dependent statistics of the Ising model. Journal of Mathematical Physics 4: 294–307.
[16] Yang J, Zhang Y (2011) Alternating direction algorithms for L1 problems in compressive sensing. SIAM Journal on Scientific Computing 33: 250–278.
[17] Zhang Y, Yang J, Yin W (2010) YALL1: Your ALgorithms for L1. CAAM Technical Report : TR09-17.
[18] Baraniuk R, Cevher V, Wakin MB (2010) Low-dimensional models for dimensionality reduction and signal recovery: A geometric perspective. Proceedings of the IEEE 98: 959–971.
[19] der Maaten LV, Hinton G (2008) Visualizing high-dimensional data using t-SNE. Journal of Machine Learning Research 9: 2579–2605.
[20] Carter KM, Raich R, Finn WG, Hero AO (2011) Information-geometric dimensionality reduction. IEEE Signal Process Mag 28: 89–99.
[21] Olshausen BA, Field DJ (1996) Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381: 607–9.
[22] Stephens GJ, Mora T, Tkacik G, Bialek W (2008) Thermodynamics of natural images. arXiv q-bio.NC.
[23] Isely G, Hillar CJ, Sommer FT (2010) Deciphering subsampled data: adaptive compressive sampling as a principle of brain communication. arXiv q-bio.NC. 9