nips nips2000 nips2000-33 knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Un-Min Bae, Soo-Young Lee
Abstract: We present an algorithm which compensates for the mismatches between characteristics of real-world problems and assumptions of independent component analysis algorithm. To provide additional information to the ICA network, we incorporate top-down selective attention. An MLP classifier is added to the separated signal channel and the error of the classifier is backpropagated to the ICA network. This backpropagation process results in estimation of expected ICA output signal for the top-down attention. Then, the unmixing matrix is retrained according to a new cost function representing the backpropagated error as well as independence. It modifies the density of recovered signals to the density appropriate for classification. For noisy speech signal recorded in real environments, the algorithm improved the recognition performance and showed robustness against parametric changes. 1
Reference: text
sentIndex sentText sentNum sentScore
1 kr Abstract We present an algorithm which compensates for the mismatches between characteristics of real-world problems and assumptions of independent component analysis algorithm. [sent-7, score-0.359]
2 To provide additional information to the ICA network, we incorporate top-down selective attention. [sent-8, score-0.255]
3 An MLP classifier is added to the separated signal channel and the error of the classifier is backpropagated to the ICA network. [sent-9, score-0.827]
4 This backpropagation process results in estimation of expected ICA output signal for the top-down attention. [sent-10, score-0.197]
5 Then, the unmixing matrix is retrained according to a new cost function representing the backpropagated error as well as independence. [sent-11, score-0.705]
6 It modifies the density of recovered signals to the density appropriate for classification. [sent-12, score-0.177]
7 For noisy speech signal recorded in real environments, the algorithm improved the recognition performance and showed robustness against parametric changes. [sent-13, score-0.488]
8 1 Introduction Independent Component Analysis (ICA) is a method for blind signal separation. [sent-14, score-0.163]
9 ICA linearly transforms data to be statistically as independent from each other as possible [1,2,5]. [sent-15, score-0.109]
10 ICA depends on several assumptions such as linear mixing and source independence which may not be satisfied in many real-world applications. [sent-16, score-0.39]
11 In order to apply ICA to most real-world problems, it is necessary either to release of all assumptions or to compensate for the mismatches with another method. [sent-17, score-0.432]
12 In this paper, we present a complementary approach to compensate for the mismatches. [sent-18, score-0.139]
13 The top-down selective attention from a classifier to the ICA network provides additional information of the signal-mixing environment. [sent-19, score-0.637]
14 A new cost function is defined to retrain the unmixing matrix of the ICA network considering the propagated information. [sent-20, score-0.559]
15 Under a stationary mixing environment, the averaged adaptation by iterative feedback operations can adjust the feature space to be more helpful to classification performance. [sent-21, score-0.619]
16 This process can be regarded as a selective attention model in which input patterns are adapted according to top-down infor- mation. [sent-22, score-0.499]
17 The proposed algorithm was applied to noisy speech recognition in real environments and showed the effectiveness of the feedback operations. [sent-23, score-0.55]
18 1 The proposed algorithm Feedback operations based on selective attention As previously mentioned, ICA supposes several assumptions. [sent-25, score-0.427]
19 For example, one assumption is a linearly mixing condition, but in general, there is inevitable nonlinearity of microphones to record input signals. [sent-26, score-0.496]
20 Such mismatches between the assumptions of ICA and real mixing conditions cause unsuccessful separation of sources. [sent-27, score-0.61]
21 To overcome this problem, a method to supply valuable information to the rcA network was proposed. [sent-28, score-0.184]
22 In the learning phase of ICA, the unmixing matrix is subject to the signal-mixing matrix, not the input patterns. [sent-29, score-0.414]
23 Under stationary mixing environment where the mixing matrix is fixed, iteratively providing additional information of the mixing matrix can contribute to improving blind signal separation performance. [sent-30, score-1.281]
24 The algorithm performs feedback operations from a classifier to the ICA network in the test phase, which adapts the unmixing matrices of ICA according to a newly defined measure considering both independence and classification error. [sent-31, score-1.149]
25 This can result in adaptation of input space of the classifier and so improve recognition performance. [sent-32, score-0.379]
26 This process is inspired from the selective attention model [9,10] which calculates expected input signals according to top-down information. [sent-33, score-0.528]
27 In the test phase, as shown in Figure 1, ICA separates signal and noise, and Melfrequency cepstral coefficients (MFCCs) extracted as a feature vector are delivered to a classifier, multi-layer perceptron (MLP). [sent-34, score-0.289]
28 After classification, the error function of the classifier is defined as E m1p 1~ = 2" L. [sent-35, score-0.294]
29 (tmIP,i - 2 (1) Ymlp,i) , i where tmlp,i is target value of the output neuron Ymlp,i. [sent-39, score-0.095]
30 In general, the target values are not known and should be determined from the outputs Ymlp. [sent-40, score-0.094]
31 Only the target value of the highest output is set to 1, and the others are set to -1 when the nonlinear function of the classifier is the bipolar sigmoid function. [sent-41, score-0.423]
32 To reduce the error, it computes the required changes of the input values of the classifier and finally those of the unmixed signals of the ICA network. [sent-43, score-0.353]
33 Then, the leaning rule of the ICA algorithm should be changed considering these variations. [sent-44, score-0.176]
34 The newly defined cost function of the ICA network includes the error backpropagated term as well as the joint entropy H (Yica) of the outputs Yica. [sent-45, score-0.536]
35 2"~u~u , H (2) where u are the estimate recovered sources and 'Y is a coefficient which represents the relative importance of two terms. [sent-48, score-0.092]
36 The learning rule derived using gradient descent on the cost function in Eq. [sent-49, score-0.131]
wordName wordTfidf (topN-words)
[('ica', 0.615), ('unmixing', 0.229), ('mixing', 0.22), ('classifier', 0.213), ('backpropagated', 0.2), ('mismatches', 0.2), ('yica', 0.2), ('selective', 0.173), ('attention', 0.137), ('korea', 0.133), ('utarget', 0.133), ('feedback', 0.13), ('mlp', 0.104), ('compensate', 0.096), ('assumptions', 0.091), ('newly', 0.09), ('signal', 0.085), ('operations', 0.08), ('phase', 0.078), ('blind', 0.078), ('speech', 0.074), ('environments', 0.072), ('considering', 0.068), ('cost', 0.067), ('signals', 0.064), ('recognition', 0.062), ('network', 0.062), ('matrix', 0.061), ('recovered', 0.061), ('environment', 0.059), ('separation', 0.058), ('adaptation', 0.058), ('target', 0.058), ('retrained', 0.057), ('bae', 0.057), ('inevitable', 0.057), ('stationary', 0.056), ('additional', 0.052), ('showed', 0.052), ('bipolar', 0.052), ('cepstral', 0.052), ('modifies', 0.052), ('supply', 0.052), ('linearly', 0.048), ('error', 0.048), ('rca', 0.048), ('input', 0.046), ('noisy', 0.045), ('microphones', 0.045), ('release', 0.045), ('separates', 0.045), ('independence', 0.043), ('according', 0.043), ('backpropagation', 0.043), ('adapts', 0.043), ('delivered', 0.043), ('complementary', 0.043), ('performs', 0.042), ('real', 0.041), ('record', 0.041), ('valuable', 0.039), ('propagated', 0.039), ('nonlinearity', 0.039), ('changed', 0.039), ('adjust', 0.039), ('algorithm', 0.037), ('output', 0.037), ('regarded', 0.037), ('iteratively', 0.037), ('effectiveness', 0.037), ('improving', 0.037), ('contribute', 0.037), ('separated', 0.037), ('classification', 0.036), ('advanced', 0.036), ('satisfied', 0.036), ('outputs', 0.036), ('perceptron', 0.033), ('sigmoid', 0.033), ('inspired', 0.033), ('defined', 0.033), ('descent', 0.032), ('robustness', 0.032), ('process', 0.032), ('rule', 0.032), ('transforms', 0.031), ('coefficient', 0.031), ('adapted', 0.031), ('characteristics', 0.031), ('extracted', 0.031), ('electrical', 0.031), ('overcome', 0.031), ('channel', 0.031), ('calculation', 0.03), ('statistically', 0.03), ('incorporate', 0.03), ('parametric', 0.03), ('highest', 0.03), ('computes', 0.03), ('recorded', 0.03)]
simIndex simValue paperId paperTitle
same-paper 1 1.0 33 nips-2000-Combining ICA and Top-Down Attention for Robust Speech Recognition
Author: Un-Min Bae, Soo-Young Lee
Abstract: We present an algorithm which compensates for the mismatches between characteristics of real-world problems and assumptions of independent component analysis algorithm. To provide additional information to the ICA network, we incorporate top-down selective attention. An MLP classifier is added to the separated signal channel and the error of the classifier is backpropagated to the ICA network. This backpropagation process results in estimation of expected ICA output signal for the top-down attention. Then, the unmixing matrix is retrained according to a new cost function representing the backpropagated error as well as independence. It modifies the density of recovered signals to the density appropriate for classification. For noisy speech signal recorded in real environments, the algorithm improved the recognition performance and showed robustness against parametric changes. 1
2 0.23349746 36 nips-2000-Constrained Independent Component Analysis
Author: Wei Lu, Jagath C. Rajapakse
Abstract: The paper presents a novel technique of constrained independent component analysis (CICA) to introduce constraints into the classical ICA and solve the constrained optimization problem by using Lagrange multiplier methods. This paper shows that CICA can be used to order the resulted independent components in a specific manner and normalize the demixing matrix in the signal separation procedure. It can systematically eliminate the ICA's indeterminacy on permutation and dilation. The experiments demonstrate the use of CICA in ordering of independent components while providing normalized demixing processes. Keywords: Independent component analysis, constrained independent component analysis, constrained optimization, Lagrange multiplier methods 1
3 0.1192287 2 nips-2000-A Comparison of Image Processing Techniques for Visual Speech Recognition Applications
Author: Michael S. Gray, Terrence J. Sejnowski, Javier R. Movellan
Abstract: We examine eight different techniques for developing visual representations in machine vision tasks. In particular we compare different versions of principal component and independent component analysis in combination with stepwise regression methods for variable selection. We found that local methods, based on the statistics of image patches, consistently outperformed global methods based on the statistics of entire images. This result is consistent with previous work on emotion and facial expression recognition. In addition, the use of a stepwise regression technique for selecting variables and regions of interest substantially boosted performance. 1
4 0.11905964 96 nips-2000-One Microphone Source Separation
Author: Sam T. Roweis
Abstract: Source separation, or computational auditory scene analysis , attempts to extract individual acoustic objects from input which contains a mixture of sounds from different sources, altered by the acoustic environment. Unmixing algorithms such as lCA and its extensions recover sources by reweighting multiple observation sequences, and thus cannot operate when only a single observation signal is available. I present a technique called refiltering which recovers sources by a nonstationary reweighting (
5 0.099587902 123 nips-2000-Speech Denoising and Dereverberation Using Probabilistic Models
Author: Hagai Attias, John C. Platt, Alex Acero, Li Deng
Abstract: This paper presents a unified probabilistic framework for denoising and dereverberation of speech signals. The framework transforms the denoising and dereverberation problems into Bayes-optimal signal estimation. The key idea is to use a strong speech model that is pre-trained on a large data set of clean speech. Computational efficiency is achieved by using variational EM, working in the frequency domain, and employing conjugate priors. The framework covers both single and multiple microphones. We apply this approach to noisy reverberant speech signals and get results substantially better than standard methods.
6 0.099233523 24 nips-2000-An Information Maximization Approach to Overcomplete and Recurrent Representations
7 0.08713115 65 nips-2000-Higher-Order Statistical Properties Arising from the Non-Stationarity of Natural Signals
9 0.068860762 99 nips-2000-Periodic Component Analysis: An Eigenvalue Method for Representing Periodic Structure in Speech
10 0.066069633 46 nips-2000-Ensemble Learning and Linear Response Theory for ICA
11 0.065279871 91 nips-2000-Noise Suppression Based on Neurophysiologically-motivated SNR Estimation for Robust Speech Recognition
12 0.061540294 32 nips-2000-Color Opponency Constitutes a Sparse Representation for the Chromatic Structure of Natural Scenes
13 0.057450209 145 nips-2000-Weak Learners and Improved Rates of Convergence in Boosting
14 0.056170054 58 nips-2000-From Margin to Sparsity
15 0.05411287 84 nips-2000-Minimum Bayes Error Feature Selection for Continuous Speech Recognition
16 0.052888982 90 nips-2000-New Approaches Towards Robust and Adaptive Speech Recognition
17 0.051583666 51 nips-2000-Factored Semi-Tied Covariance Matrices
18 0.051099014 40 nips-2000-Dendritic Compartmentalization Could Underlie Competition and Attentional Biasing of Simultaneous Visual Stimuli
19 0.047388479 9 nips-2000-A PAC-Bayesian Margin Bound for Linear Classifiers: Why SVMs work
20 0.047229759 68 nips-2000-Improved Output Coding for Classification Using Continuous Relaxation
topicId topicWeight
[(0, 0.167), (1, -0.046), (2, 0.036), (3, 0.117), (4, -0.058), (5, -0.126), (6, -0.178), (7, -0.106), (8, 0.015), (9, 0.026), (10, -0.215), (11, -0.069), (12, 0.011), (13, -0.071), (14, -0.143), (15, -0.06), (16, -0.224), (17, -0.06), (18, 0.148), (19, 0.151), (20, -0.13), (21, -0.132), (22, 0.063), (23, -0.126), (24, 0.098), (25, -0.07), (26, -0.155), (27, -0.242), (28, -0.046), (29, 0.052), (30, -0.191), (31, -0.09), (32, -0.181), (33, -0.133), (34, 0.067), (35, -0.135), (36, 0.003), (37, 0.055), (38, 0.025), (39, 0.032), (40, -0.024), (41, 0.081), (42, 0.035), (43, 0.092), (44, -0.001), (45, 0.073), (46, 0.032), (47, 0.058), (48, -0.005), (49, 0.049)]
simIndex simValue paperId paperTitle
same-paper 1 0.97922212 33 nips-2000-Combining ICA and Top-Down Attention for Robust Speech Recognition
Author: Un-Min Bae, Soo-Young Lee
Abstract: We present an algorithm which compensates for the mismatches between characteristics of real-world problems and assumptions of independent component analysis algorithm. To provide additional information to the ICA network, we incorporate top-down selective attention. An MLP classifier is added to the separated signal channel and the error of the classifier is backpropagated to the ICA network. This backpropagation process results in estimation of expected ICA output signal for the top-down attention. Then, the unmixing matrix is retrained according to a new cost function representing the backpropagated error as well as independence. It modifies the density of recovered signals to the density appropriate for classification. For noisy speech signal recorded in real environments, the algorithm improved the recognition performance and showed robustness against parametric changes. 1
2 0.85281521 36 nips-2000-Constrained Independent Component Analysis
Author: Wei Lu, Jagath C. Rajapakse
Abstract: The paper presents a novel technique of constrained independent component analysis (CICA) to introduce constraints into the classical ICA and solve the constrained optimization problem by using Lagrange multiplier methods. This paper shows that CICA can be used to order the resulted independent components in a specific manner and normalize the demixing matrix in the signal separation procedure. It can systematically eliminate the ICA's indeterminacy on permutation and dilation. The experiments demonstrate the use of CICA in ordering of independent components while providing normalized demixing processes. Keywords: Independent component analysis, constrained independent component analysis, constrained optimization, Lagrange multiplier methods 1
3 0.34300476 96 nips-2000-One Microphone Source Separation
Author: Sam T. Roweis
Abstract: Source separation, or computational auditory scene analysis , attempts to extract individual acoustic objects from input which contains a mixture of sounds from different sources, altered by the acoustic environment. Unmixing algorithms such as lCA and its extensions recover sources by reweighting multiple observation sequences, and thus cannot operate when only a single observation signal is available. I present a technique called refiltering which recovers sources by a nonstationary reweighting (
4 0.31101939 2 nips-2000-A Comparison of Image Processing Techniques for Visual Speech Recognition Applications
Author: Michael S. Gray, Terrence J. Sejnowski, Javier R. Movellan
Abstract: We examine eight different techniques for developing visual representations in machine vision tasks. In particular we compare different versions of principal component and independent component analysis in combination with stepwise regression methods for variable selection. We found that local methods, based on the statistics of image patches, consistently outperformed global methods based on the statistics of entire images. This result is consistent with previous work on emotion and facial expression recognition. In addition, the use of a stepwise regression technique for selecting variables and regions of interest substantially boosted performance. 1
5 0.3071045 24 nips-2000-An Information Maximization Approach to Overcomplete and Recurrent Representations
Author: Oren Shriki, Haim Sompolinsky, Daniel D. Lee
Abstract: The principle of maximizing mutual information is applied to learning overcomplete and recurrent representations. The underlying model consists of a network of input units driving a larger number of output units with recurrent interactions. In the limit of zero noise, the network is deterministic and the mutual information can be related to the entropy of the output units. Maximizing this entropy with respect to both the feedforward connections as well as the recurrent interactions results in simple learning rules for both sets of parameters. The conventional independent components (ICA) learning algorithm can be recovered as a special case where there is an equal number of output units and no recurrent connections. The application of these new learning rules is illustrated on a simple two-dimensional input example.
6 0.30688503 65 nips-2000-Higher-Order Statistical Properties Arising from the Non-Stationarity of Natural Signals
9 0.25546172 46 nips-2000-Ensemble Learning and Linear Response Theory for ICA
11 0.24201791 32 nips-2000-Color Opponency Constitutes a Sparse Representation for the Chromatic Structure of Natural Scenes
12 0.24110159 123 nips-2000-Speech Denoising and Dereverberation Using Probabilistic Models
13 0.22913894 29 nips-2000-Bayes Networks on Ice: Robotic Search for Antarctic Meteorites
14 0.22304018 138 nips-2000-The Use of Classifiers in Sequential Inference
15 0.18251342 126 nips-2000-Stagewise Processing in Error-correcting Codes and Image Restoration
16 0.17612334 40 nips-2000-Dendritic Compartmentalization Could Underlie Competition and Attentional Biasing of Simultaneous Visual Stimuli
17 0.17461886 90 nips-2000-New Approaches Towards Robust and Adaptive Speech Recognition
18 0.17140321 28 nips-2000-Balancing Multiple Sources of Reward in Reinforcement Learning
19 0.16724199 91 nips-2000-Noise Suppression Based on Neurophysiologically-motivated SNR Estimation for Robust Speech Recognition
20 0.16667603 147 nips-2000-Who Does What? A Novel Algorithm to Determine Function Localization
topicId topicWeight
[(10, 0.014), (17, 0.074), (32, 0.011), (33, 0.05), (54, 0.012), (62, 0.029), (65, 0.587), (67, 0.048), (76, 0.02), (81, 0.011), (90, 0.023), (91, 0.01), (93, 0.013)]
simIndex simValue paperId paperTitle
same-paper 1 0.94308412 33 nips-2000-Combining ICA and Top-Down Attention for Robust Speech Recognition
Author: Un-Min Bae, Soo-Young Lee
Abstract: We present an algorithm which compensates for the mismatches between characteristics of real-world problems and assumptions of independent component analysis algorithm. To provide additional information to the ICA network, we incorporate top-down selective attention. An MLP classifier is added to the separated signal channel and the error of the classifier is backpropagated to the ICA network. This backpropagation process results in estimation of expected ICA output signal for the top-down attention. Then, the unmixing matrix is retrained according to a new cost function representing the backpropagated error as well as independence. It modifies the density of recovered signals to the density appropriate for classification. For noisy speech signal recorded in real environments, the algorithm improved the recognition performance and showed robustness against parametric changes. 1
2 0.84845394 15 nips-2000-Accumulator Networks: Suitors of Local Probability Propagation
Author: Brendan J. Frey, Anitha Kannan
Abstract: One way to approximate inference in richly-connected graphical models is to apply the sum-product algorithm (a.k.a. probability propagation algorithm), while ignoring the fact that the graph has cycles. The sum-product algorithm can be directly applied in Gaussian networks and in graphs for coding, but for many conditional probability functions - including the sigmoid function - direct application of the sum-product algorithm is not possible. We introduce
3 0.76526713 68 nips-2000-Improved Output Coding for Classification Using Continuous Relaxation
Author: Koby Crammer, Yoram Singer
Abstract: Output coding is a general method for solving multiclass problems by reducing them to multiple binary classification problems. Previous research on output coding has employed, almost solely, predefined discrete codes. We describe an algorithm that improves the performance of output codes by relaxing them to continuous codes. The relaxation procedure is cast as an optimization problem and is reminiscent of the quadratic program for support vector machines. We describe experiments with the proposed algorithm, comparing it to standard discrete output codes. The experimental results indicate that continuous relaxations of output codes often improve the generalization performance, especially for short codes.
4 0.38006866 47 nips-2000-Error-correcting Codes on a Bethe-like Lattice
Author: Renato Vicente, David Saad, Yoshiyuki Kabashima
Abstract: We analyze Gallager codes by employing a simple mean-field approximation that distorts the model geometry and preserves important interactions between sites. The method naturally recovers the probability propagation decoding algorithm as an extremization of a proper free-energy. We find a thermodynamic phase transition that coincides with information theoretical upper-bounds and explain the practical code performance in terms of the free-energy landscape.
5 0.32934582 62 nips-2000-Generalized Belief Propagation
Author: Jonathan S. Yedidia, William T. Freeman, Yair Weiss
Abstract: Belief propagation (BP) was only supposed to work for tree-like networks but works surprisingly well in many applications involving networks with loops, including turbo codes. However, there has been little understanding of the algorithm or the nature of the solutions it finds for general graphs. We show that BP can only converge to a stationary point of an approximate free energy, known as the Bethe free energy in statistical physics. This result characterizes BP fixed-points and makes connections with variational approaches to approximate inference. More importantly, our analysis lets us build on the progress made in statistical physics since Bethe's approximation was introduced in 1935. Kikuchi and others have shown how to construct more accurate free energy approximations, of which Bethe's approximation is the simplest. Exploiting the insights from our analysis, we derive generalized belief propagation (GBP) versions ofthese Kikuchi approximations. These new message passing algorithms can be significantly more accurate than ordinary BP, at an adjustable increase in complexity. We illustrate such a new GBP algorithm on a grid Markov network and show that it gives much more accurate marginal probabilities than those found using ordinary BP. 1
6 0.32686183 36 nips-2000-Constrained Independent Component Analysis
7 0.32065502 118 nips-2000-Smart Vision Chip Fabricated Using Three Dimensional Integration Technology
8 0.30258998 80 nips-2000-Learning Switching Linear Models of Human Motion
9 0.29057541 45 nips-2000-Emergence of Movement Sensitive Neurons' Properties by Learning a Sparse Code for Natural Moving Images
10 0.2897191 115 nips-2000-Sequentially Fitting ``Inclusive'' Trees for Inference in Noisy-OR Networks
11 0.28390744 99 nips-2000-Periodic Component Analysis: An Eigenvalue Method for Representing Periodic Structure in Speech
12 0.28117785 108 nips-2000-Recognizing Hand-written Digits Using Hierarchical Products of Experts
13 0.27611163 8 nips-2000-A New Model of Spatial Representation in Multimodal Brain Areas
14 0.27508911 127 nips-2000-Structure Learning in Human Causal Induction
15 0.27274284 14 nips-2000-A Variational Mean-Field Theory for Sigmoidal Belief Networks
16 0.26950583 126 nips-2000-Stagewise Processing in Error-correcting Codes and Image Restoration
17 0.26724321 138 nips-2000-The Use of Classifiers in Sequential Inference
18 0.26390803 7 nips-2000-A New Approximate Maximal Margin Classification Algorithm
19 0.25505024 106 nips-2000-Propagation Algorithms for Variational Bayesian Learning
20 0.25264037 69 nips-2000-Incorporating Second-Order Functional Knowledge for Better Option Pricing