nips nips2009 nips2009-219 nips2009-219-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Yoshua Bengio, James S. Bergstra
Abstract: We introduce a new type of neural network activation function based on recent physiological rate models for complex cells in visual area V1. A single-hiddenlayer neural network of this kind of model achieves 1.50% error on MNIST. We also introduce an existing criterion for learning slow, decorrelated features as a pretraining strategy for image models. This pretraining strategy results in orientation-selective features, similar to the receptive fields of complex cells. With this pretraining, the same single-hidden-layer model achieves 1.34% error, even though the pretraining sample distribution is very different from the fine-tuning distribution. To implement this pretraining strategy, we derive a fast algorithm for online learning of decorrelated features such that each iteration of the algorithm runs in linear time with respect to the number of features. 1
Adelson, E. H., & Bergen, J. R. (1985). Spatiotemporal energy models for the perception of motion. Journal of the Optical Society of America, 2, 284–99. Becker, S., & Hinton, G. E. (1993). Learning mixture models of spatial coherence. Neural Computation, 5, 267–277. Bengio, Y. (2009). Learning deep architectures for AI. Foundations and Trends in Machine Learning, to appear. Berkes, P., & Wiskott, L. (2005). Slow feature analysis yields a rich repertoire of complex cell properties. Journal of Vision, 5, 579–602. Cadieu, C., & Olshausen, B. (2009). Learning transformational invariants from natural movies. In Advances in neural information processing systems 21 (nips’08), 209–216. MIT Press. Dayan, P., & Abbott, L. F. (2001). Theoretical neuroscience. The MIT Press. Decoste, D., & Sch¨ lkopf, B. (2002). Training invariant support vector machines. Machine Learno ing, 46, 161–190. Erhan, D., Manzagol, P.-A., Bengio, Y., Bengio, S., & Vincent, P. (2009). The difficulty of training deep architectures and the effect of unsupervised pre-training. AISTATS’2009 (pp. 153–160). Clearwater (Florida), USA. Finn, I., & Ferster, D. (2007). Computational diversity in complex cells of cat primary visual cortex. Journal of Neuroscience, 27, 9638–48. Hinton, G. E., Osindero, S., & Teh, Y. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527–1554. Hurri, J., & Hyv¨ rinen, A. (2003). Temporal coherence, natural image sequences, and the visual a cortex. Advances in Neural Information Processing Systems 15 (NIPS’02) (pp. 141–148). K¨ rding, K. P., Kayser, C., Einh¨ user, W., & K¨ nig, P. (2004). How are complex cell properties o a o adapted to the statistics of natural stimuli? Journal of Neurophysiology, 91, 206–212. Kouh, M. M., & Poggio, T. T. (2008). A canonical neural circuit for cortical nonlinear operations. Neural Computation, 20, 1427–1451. Larochelle, H., Erhan, D., Courville, A., Bergstra, J., & Bengio, Y. (2007). An empirical evaluation of deep architectures on problems with many factors of variation. ICML 2007 (pp. 473–480). Corvallis, OR: ACM. Lee, H., Ekanadham, C., & Ng, A. (2008). Sparse deep belief net model for visual area V2. In Advances in neural information processing systems 20 (nips’07). Cambridge, MA: MIT Press. Loosli, G., Canu, S., & Bottou, L. (2007). Training invariant support vector machines using selective sampling. In L. Bottou, O. Chapelle, D. DeCoste and J. Weston (Eds.), Large scale kernel machines, 301–320. Cambridge, MA.: MIT Press. Mobahi, H., Collobert, R., & Weston, J. (2009). Deep learning from temporal coherence in video. ICML 2009. ACM. To appear. Ranzato, M., Boureau, Y., & LeCun, Y. (2008). Sparse feature learning for deep belief networks. NIPS 20. Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007). Efficient learning of sparse representations with an energy-based model. NIPS 19. Riesenhuber, M., & Poggio, T. (1999). Hierarchical models of object recognition in cortex. Nature Neuroscience, 2, 1019–1025. Rust, N., Schwartz, O., Movshon, J. A., & Simoncelli, E. (2005). Spatiotemporal elements of macaque V1 receptive fields. Neuron, 46, 945–956. Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. ICML 2008 (pp. 1096–1103). ACM. Weston, J., Ratle, F., & Collobert, R. (2008). Deep learning via semi-supervised embedding. ICML 2008 (pp. 1168–1175). New York, NY, USA: ACM. Wiskott, L., & Sejnowski, T. (2002). Slow feature analysis: Unsupervised learning of invariances. Neural Computation, 14, 715–770. 9