jmlr jmlr2005 jmlr2005-25 jmlr2005-25-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Jaakko Särelä, Harri Valpola
Abstract: A new algorithmic framework called denoising source separation (DSS) is introduced. The main benefit of this framework is that it allows for the easy development of new source separation algorithms which can be optimised for specific problems. In this framework, source separation algorithms are constructed around denoising procedures. The resulting algorithms can range from almost blind to highly specialised source separation algorithms. Both simple linear and more complex nonlinear or adaptive denoising schemes are considered. Some existing independent component analysis algorithms are reinterpreted within the DSS framework and new, robust blind source separation algorithms are suggested. The framework is derived as a one-unit equivalent to an EM algorithm for source separation. However, in the DSS framework it is easy to utilise various kinds of denoising procedures which need not be based on generative models. In the experimental section, various DSS schemes are applied extensively to artificial data, to real magnetoencephalograms and to simulated CDMA mobile network signals. Finally, various extensions to the proposed DSS algorithms are considered. These include nonlinear observation mappings, hierarchical models and over-complete, nonorthogonal feature spaces. With these extensions, DSS appears to have relevance to many existing models of neural information processing. Keywords: blind source separation, BSS, prior information, denoising, denoising source separation, DSS, independent component analysis, ICA, magnetoencephalograms, MEG, CDMA
B. D. Anderson and J. B. Moore. Optimal filtering. Prentice-Hall, 1979. H. Attias. Independent factor analysis. Neural Computation, 11(4):803–851, 1999. A. Belouchrani, K. Abed Meraim, J.-F. Cardoso, and E. Moulines. A blind source separation technique based on second order statistics. IEEE Transactions on Signal Processing, 45(2):434–44, 1997. O. Bermond and J.-F. Cardoso. Approximate likelihood for noisy mixtures. In Proceedings of the First International Workshop on Independent Component Analysis and Signal Separation (ICA’99), pages 325–330, Aussois, France, Jan. 11-15, 1999. J.-F. Cardoso. High-order contrasts for independent component analysis. Neural Computation, 11 (1):157 – 192, 1999. K.-L. Chan, T.-W. Lee, and T. J. Sejnowski. Variational Bayesian learning of ICA with missing data. Neural Computation, 15 (8):1991–2011, 2003. R. A. Choudrey and S. J. Roberts. Flexible Bayesian independent component analysis for blind source separation. In Proceedings of the Third International Conference on Independent Component Analysis and Signal Separation (ICA2001), pages 90–95, San Diego, USA, 2001. P. A. d. F. R. Højen-Sørensen, O. Winther, and L. K. Hansen. Mean-field approaches to independent component analysis. Neural Computation, 14(4):889–918, 2002. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B (Methodological), 39(1):1–38, 1977. D. L. Donoho, I. M. Johnstone, G. Kerkyacharian, and D. Picard. Wavelet shrinkage: asymptopia? Journal of the Royal Statistical Society, Series B (Methodological), 57:301–337, 1995. S. C. Douglas and A. Cichocki. Neural networks for blind decorrelation of signals. IEEE Transactions on Signal Processing, 45(11):2829 – 2842, 1997. 269 ¨ ¨ S AREL A AND VALPOLA DSS. The DSS MATLAB package. 2004. Available at http://www.cis.hut.fi/projects/ dss/. FastICA. The FastICA MATLAB package. projects/ica/fastica/. 1998. Available at http://www.cis.hut.fi/ P. F¨ ldi´ k. Forming sparse representations by local anti-hebbian learning. Biological Cybernetics, o a 64:165 – 170, 1990. P. F¨ ldi´ k. Learning invariance from transformation sequences. Neural Computation, 3:194–200, o a 1991. M. Funaro, E. Oja, and H. Valpola. Independent component analysis for artefact separation in astrophysical images. Neural Networks, 16(3 – 4):469 – 478, 2003. M. S. Gazzaniga, editor. The New Cognitive Neurosciences. A Bradford book/MIT Press, 2nd edition, 2000. X. Giannakopoulos, J. Karhunen, and E. Oja. Experimental comparison of neural algorithms for independent component analysis and blind separation. International Journal of Neural Systems, 9(2):651–656, 1999. M. H¨ m¨ l¨ inen, R. Hari, R. Ilmoniemi, J. Knuutila, and O. V. Lounasmaa. a aa Magnetoencephalography—theory, instrumentation, and applications to noninvasive studies of the working human brain. Reviews of Modern Physics, 65:413–497, 1993. J. Himberg and A. Hyv¨ rinen. Icasso: software for investigating the reliability of ica estimates a by clustering and visualization. In Proceedings of the IEEE Workshop on Neural Networks for Signal Processing (NNSP’2003), pages 259–268, Toulouse, France, 2003. A. Hyv¨ rinen. New approximations of differential entropy for independent component analysis and a projection pursuit. In Advances in Neural Information Processing 10 (NIPS’98), pages 273–279. MIT Press, 1998. A. Hyv¨ rinen. Fast and robust fixed-point algorithms for independent component analysis. IEEE a Transactions on Neural Networks, 10(3):626–634, 1999. A. Hyv¨ rinen, P. Hoyer, and M. Inki. Topographic independent component analysis. Neural Coma putation, 13(7):1525–1558, 2001a. A. Hyv¨ rinen, J. Karhunen, and E. Oja. Independent component analysis. Wiley, 2001b. a JADE. The JADE MATLAB package. icacentral/Algos/cardoso/. 1999. Available at http://www.tsi.enst.fr/ K. H. Knuth. Bayesian source separation and localization. In A. Mohammad-Djafari, editor, SPIE’98 Proceedings: Bayesian Inference for Inverse Problems, pages 147–158, San Diego, USA, 1998. P. Kuosmanen and J. T. Astola. Fundamentals of nonlinear digital filtering. CRC press, 1997. 270 D ENOISING S OURCE S EPARATION K. P. K¨ rding and P. K¨ nig. Neurons with two sites of synaptic integration learn invariant represeno o tations. Neural Computation, 13:2823 – 2849, 2001. H. Lappalainen. Ensemble learning for independent component analysis. In Proceedings of the First International Workshop on Independent Component Analysis and Signal Separation, (ICA’99), pages 7–12, Aussois, France, 1999. D. G. Luenberger. Optimization by Vector Space Methods. John Wiley & Sons, 1969. W. Maass, T. Natschl¨ ger, and H. Markram. Real-time computing without stable states: A new a framework for neural computation based on perturbations. Neural Computation, 14(11):2531 – 2560, 2002. F. Meinecke, A. Ziehe, M. Kawanabe, and K.-R. M¨ ller. A resampling approach to estimate the stau bility of one- and multidimensional independent components. IEEE Transactions on Biomedical Engineering, 49(12):1514 – 1525, 2002. J. Miskin and David J. C. MacKay. Ensemble learning for blind source separation. In S. Roberts and R. Everson, editors, Independent Component Analysis: Principles and Practice, pages 209–233. Cambridge University Press, 2001. J. Molgedey and H. G. Schuster. Separation of a mixture of independent signals using time delayed correlations. Physical Review Letters, 72:541–557, 1994. E. Niedermeyer and F. Lopes da Silva, editors. Electroencephalography. Basic principles, clinical applications, and related fields. Baltimore: Williams & Wilkins, 1993. E. Oja. Principal components, minor components, and linear neural networks. Neural Networks, 5: 927–935, 1992. B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381:607–609, 1996. D. B. Percival and W. T. Walden. Spectral Analysis for Physical Applications: Multitaper and Conventional Univariate Techniques. Cambridge University Press, Cambridge, UK, 1993. D.-T. Pham and J.-F. Cardoso. Blind separation of instantaneous mixtures of non stationary sources. IEEE Transactions on Signal Processing, 49:1837–1848, 2001. K. Raju and T. Ristaniemi. ICA-RAKE switching for jammer cancellation in DS-CDMA array systems. In Proceedings of the IEEE International Symposium on Spread Spectrum Techniques and Applications (ISSSTA), pages 638 – 642, Prague, September 2002. R. M. Rangayyan. Biomedical signal analysis: A case-study approach. IEEE Press Series in Biomedical Engineering, 2002. J. S¨ rel¨ , H. Valpola, R. Vig´ rio, and E. Oja. Dynamical factor analysis of rhythmic magnetoena a a cephalographic activity. In Proceedings of the Third International Conference on Independent Component Analysis and Signal Separation (ICA2001), pages 451–456, San Diego, USA, 2001. 271 ¨ ¨ S AREL A AND VALPOLA J. S¨ rel¨ and R. Vig´ rio. Overlearning in marginal distribution-based ICA: analysis and solutions. a a a Journal of Machine Learning Research, 4 (Dec):1447–1469, 2003. B. Sch¨ lkopf, S. Mika, A. Smola, Gunnar R¨ tsch, and K.-R. M¨ ller. Kernel PCA pattern recono a u struction via approximate pre-images. In Proceedings of the 8th International Conference on Artificial Neural Networks (ICANN’98), pages 147 – 152, Sk¨ vde, 1998. o O. Schwartz and E. P. Simoncelli. Natural signal statistics and sensory gain control. Nature Neuroscience, 4(8):819 – 825, 2001. L. Tong, V. Soo, R. Liu, and Y. Huang. Indeterminacy and identifiability of blind identification. IEEE Transactions on Circuits and Systems, 38:499–509, 1991. H. Valpola and P. Pajunen. Fast algorithms for Bayesian independent component analysis. In Proceedings of the Second International Workshop on Independent Component Analysis and Signal Separation (ICA2000), pages 233–237, Helsinki, Finland, 2000. H. Valpola, T. Raiko, and J. Karhunen. Building blocks for hierarchical latent variable models. In Proceedings of the Third International Conference on Independent Component Analysis and Signal Separation (ICA2001), pages 710–715, San Diego, USA, 2001. H. Valpola and J. S¨ rel¨ . Accurate, fast and stable denoising source separation algorithms. In Proa a ceedings of the Fifth International Conference on Independent Component Analysis and Signal Separation (ICA2004), pages 64 – 71, Granada, Spain, 2004. M. Vetterli and J. Kovacevic. Wavelets and subband coding. Prentice-Hall, 1995. R. Vig´ rio, J. S¨ rel¨ , V. Jousm¨ ki, M. H¨ m¨ l¨ inen, and E. Oja. Independent component approach a a a a a aa to the analysis of EEG and MEG recordings. IEEE Transactions on Biomedical Engineering, 47 (5):589–593, 2000. V. Vigneron, A. Paraschiv-Ionescu, A. Azancot, O. Sibony, and C. Jutten. Fetal electrocardiogram extraction based on non-stationary ICA and wavelet denoising. In Proceedings of the Seventh International Symposium on Signal Processing and its Applications (ISSPA2003), Paris, France, July 2003. A. J. Viterbi. CDMA : Principles of Spread Spectrum Communication. Wireless Info Networks Series. Addison-Wesley, 1995. J. H. Wilkinson. The algebraic eigenvalue problem. Monographs on numerical analysis. Clarendon press, London, 1965. L. Wiskott and T. Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural Computation, 14:715 – 770, 2002. A. Ziehe and K.-R. M¨ ller. TDSEP — an effective algorithm for blind separation using time u structure. In Proceedings of the 8th International Conference on Artificial Neural Networks (ICANN’98), pages 675–680, Sk¨ vde, Sweden, 1998. o 272