nips nips2008 nips2008-192 nips2008-192-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Siwei Lyu, Eero P. Simoncelli
Abstract: We consider the problem of transforming a signal to a representation in which the components are statistically independent. When the signal is generated as a linear transformation of independent Gaussian or non-Gaussian sources, the solution may be computed using a linear transformation (PCA or ICA, respectively). Here, we consider a complementary case, in which the source is non-Gaussian but elliptically symmetric. Such a source cannot be decomposed into independent components using a linear transform, but we show that a simple nonlinear transformation, which we call radial Gaussianization (RG), is able to remove all dependencies. We apply this methodology to natural signals, demonstrating that the joint distributions of nearby bandpass filter responses, for both sounds and images, are closer to being elliptically symmetric than linearly transformed factorial sources. Consistent with this, we demonstrate that the reduction in dependency achieved by applying RG to either pairs or blocks of bandpass filter responses is significantly greater than that achieved by PCA or ICA.
[1] F Attneave. Some informational aspects of visual perception. Psych. Rev., 61:183–193, 1954. 1 Similar results for the comparison of ICA to PCA were obtained with a slightly different method of removing the mean values of each block [8]. 7
[2] H B Barlow. Possible principles underlying the transformation of sensory messages. In W A Rosenblith, editor, Sensory Communication, pages 217–234. MIT Press, Cambridge, MA, 1961.
[3] B A Olshausen and D J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381:607–609, 1996.
[4] A van der Schaaf and J H van Hateren. Modelling the power spectra of natural images: Statistics and information. Vision Research, 28(17):2759–2770, 1996.
[5] A J Bell and T J Sejnowski. The ’independent components’ of natural scenes are edge filters. Vision Research, 37(23):3327–3338, 1997.
[6] M S Lewicki. Efficient coding of natural sounds. Nature Neuroscience, 5(4):356–363, 2002.
[7] R. Baddeley. Searching for filters with “interesting” output distributions: an uninteresting direction to explore. Network, 7:409–421, 1996.
[8] Matthias Bethge. Factorial coding of natural images: how effective are linear models in removing higherorder dependencies? J. Opt. Soc. Am. A, 23(6):1253–1268, 2006.
[9] B Wegmann and C Zetzsche. Statistical dependence between orientation filter outputs used in an human vision based image code. In Proc Visual Comm. and Image Processing, volume 1360, pages 909–922, Lausanne, Switzerland, 1990.
[10] E P Simoncelli. Statistical models for images: Compression, restoration and synthesis. In Proc 31st Asilomar Conf on Signals, Systems and Computers, volume 1, pages 673–678, Pacific Grove, CA, November 2-5 1997. IEEE Computer Society.
[11] M J Wainwright and E P Simoncelli. Scale mixtures of Gaussians and the statistics of natural images. In S. A. Solla, T. K. Leen, and K.-R. M¨ ller, editors, Adv. Neural Information Processing Systems u (NIPS*99), volume 12, pages 855–861, Cambridge, MA, May 2000. MIT Press.
[12] K.T. Fang, S. Kotz, and K.W. Ng. Symmetric Multivariate and Related Distributions. Chapman and Hall, London, 1990.
[13] S. Lyu and E. P. Simoncelli. Nonlinear extraction of “independent components” of elliptically symmetric densities using radial Gaussianization. Technical Report TR2008-911, Computer Science Technical Report, Courant Inst. of Mathematical Sciences, New York University, April 2008.
[14] D. Nash and M. S. Klamkin. A spherical characterization of the normal distribution. Journal of Multivariate Analysis, 55:56–158, 1976.
[15] O Schwartz and E P Simoncelli. Natural signal statistics and sensory gain control. Nature Neuroscience, 4(8):819–825, August 2001.
[16] William Feller. An Introduction to Probability Theory and Its Applications, volume 1. Wiley, January 1968.
[17] C Zetzsche and G Krieger. The atoms of vision: Cartesian or polar? J. Opt. Soc. Am. A, 16(7), July 1999.
[18] J. Huang and D. Mumford. Statistics of natural images and models. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 1999.
[19] A Srivastava, X Liu, and U Grenander. Universal analytical forms for modeling image probability. IEEE Pat. Anal. Mach. Intell., 24(9):1200–1214, Sep 2002.
[20] Y. Teh, M. Welling, and S. Osindero. Energy-based models for sparse overcomplete representations. Journal of Machine Learning Research, 4:1235–1260, 2003.
[21] P I M Johannesma. The pre-response stimulus ensemble of neurons in the cochlear nucleus. In Symposium on Hearing Theory (IPO), pages 58–69, Eindhoven, Holland, 1972.
[22] M. Studeny and J. Vejnarova. The multiinformation function as a tool for measuring stochastic dependence. In M. I. Jordan, editor, Learning in Graphical Models, pages 261–297. Dordrecht: Kluwer., 1998.
[23] T. Cover and J. Thomas. Elements of Information Theory. Wiley-Interscience, 2nd edition, 2006.
[24] A. Kraskov, H. St¨ gbauer, and P. Grassberger. Estimating mutual information. Phys. Rev. E, 69(6):66–82, o Jun 2004.
[25] E. G. Learned-Miller and J. W. Fisher. ICA using spacings estimates of entropy. Journal of Machine Learning Research, 4(1):1271–1295, 2000.
[26] J H van Hateren and A van der Schaaf. Independent component filters of natural images compared with simple cells in primary visual cortex. Proc. R. Soc. Lond. B, 265:359–366, 1998.
[27] A. Hyv¨ rinen. Fast and robust fixed-point algorithms for independent component analysis. IEEE Transa actions on Neural Networks, 10(3):626–634, 1999.
[28] Scott Saobing Chen and Ramesh A. Gopinath. Gaussianization. In Advances in Neural Computation Systems (NIPS), pages 423–429, 2000.
[29] S. Roth and M. Black. Fields of experts: A framework for learning image priors. In IEEE Conference on Computer Vision and Patten Recognition (CVPR), volume 2, pages 860–867, 2005.
[30] S Lyu and E P Simoncelli. Statistical modeling of images with fields of Gaussian scale mixtures. In B Sch¨ lkopf, J Platt, and T Hofmann, editors, Adv. Neural Information Processing Systems 19, volume 19, o Cambridge, MA, May 2007. MIT Press. 8