jmlr jmlr2005 jmlr2005-65 knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Luís B. Almeida
Abstract: When acquiring an image of a paper document, the image printed on the back page sometimes shows through. The mixture of the front- and back-page images thus obtained is markedly nonlinear, and thus constitutes a good real-life test case for nonlinear blind source separation. This paper addresses a difficult version of this problem, corresponding to the use of “onion skin” paper, which results in a relatively strong nonlinearity of the mixture, which becomes close to singular in the lighter regions of the images. The separation is achieved through the MISEP technique, which is an extension of the well known INFOMAX method. The separation results are assessed with objective quality measures. They show an improvement over the results obtained with linear separation, but have room for further improvement. Keywords: ICA, blind source separation, nonlinear mixtures, nonlinear separation, image mixture, image separation
Reference: text
sentIndex sentText sentNum sentScore
1 The mixture of the front- and back-page images thus obtained is markedly nonlinear, and thus constitutes a good real-life test case for nonlinear blind source separation. [sent-7, score-0.854]
2 The separation results are assessed with objective quality measures. [sent-10, score-0.49]
3 Keywords: ICA, blind source separation, nonlinear mixtures, nonlinear separation, image mixture, image separation 1. [sent-12, score-1.515]
4 This is a situation that seems suited for handling by blind source separation (BSS) techniques. [sent-18, score-0.711]
5 The main difficulty is that the images that are acquired are nonlinear mixtures of the original images printed on each of the sides of the paper. [sent-19, score-0.984]
6 For separation we use MISEP, which is a nonlinear independent component analysis (ICA) technique (Almeida, 2003b). [sent-23, score-0.714]
7 ı A LMEIDA Besides the separation itself, an important practical issue in this specific situation is the alignment of the two mixture images. [sent-26, score-0.679]
8 It was found, however, that scanners normally introduce slight geometrical distortions that make it necessary to use local alignment techniques to obtain an image alignment that is adequate for separation. [sent-28, score-0.578]
9 To the author’s knowledge, and apart from an earlier version of the present work (Almeida and Faria, 2004), the only published report of blind source separation of a real-life nonlinear mixture in which the recovery of the original sources can be confirmed is (Haritopoulos et al. [sent-31, score-1.132]
10 This manuscript’s structure is as follows: Section 2 provides a brief overview of nonlinear separation methods. [sent-36, score-0.703]
11 However, the paper is freely available online, and in the electronic online version one can zoom in on the images (scatter plots and images of sources, mixtures and separated components) to better view the details. [sent-42, score-0.612]
12 The source and mixture images used in this paper are available online at http://www. [sent-55, score-0.523]
13 The essential uniqueness of the solution of linear ICA (Comon, 1994), together with the greater simplicity of linear separation and with the fact that many naturally occurring mixtures are essentially linear, led to a quick development of linear ICA. [sent-68, score-0.482]
14 The work on nonlinear ICA probably was 1200 S EPARATING A R EAL -L IFE N ONLINEAR I MAGE M IXTURE slowed down mostly by its inherent ill-posedness and by its greater complexity, but development of nonlinear methods has continued steadily (e. [sent-69, score-0.492]
15 Block F performs the separation proper, the separated components being yi . [sent-106, score-0.473]
16 An issue that has frequently been discussed is whether nonlinear blind source separation, based on ICA, is feasible in practice. [sent-130, score-0.531]
17 This author has argued that in the nonlinear case, when the mixture is not too strongly nonlinear, adequate regularization should allow the handling of the ill-posedness of nonlinear ICA, still allowing the approximate recovery of the sources. [sent-135, score-0.62]
18 As we shall see below, approximate source recovery was possible, and the indetermination of nonlinear ICA didn’t lead to inadequate separation. [sent-137, score-0.446]
19 The corresponding pairs of source images are shown in Figs. [sent-141, score-0.457]
20 Thus, by construction, the intensities of the two images are independent, and each of the images has an intensity distribution which is close to uniform. [sent-146, score-0.7]
21 The images have been cropped, and one image in each pair has been horizontally flipped, to correspond to its position in the acquired images. [sent-152, score-0.57]
22 Since the text has many large changes of intensity in very small areas, a good “mixing” of the intensities from both images takes place, and the two sources are approximately independent. [sent-160, score-0.552]
23 The large, relatively uniform areas of the images make imperfect separation easier to notice visually than in case 2 above. [sent-170, score-0.686]
24 First of all we should note that, for the joint distributions of the two sources of each pair to be meaningful, the source images had to be adjusted in resolution and aligned, so as to be in the same relative position as in the acquired mixtures. [sent-174, score-0.705]
25 For that purpose each source image was reduced in resolution to the same size as the corresponding acquired mixture images, and was then aligned with the corresponding separated component from nonlinear separation (see Section 4. [sent-175, score-1.409]
26 2 and 3, which show the source images after resizing and alignment. [sent-180, score-0.474]
27 Some more comments are useful for a better understanding of the source distributions: • The “grid” look of the first scatter plot reflects the fact that each of the source images had only 25 equally spaced intensities. [sent-181, score-0.896]
28 Some intermediate intensities also appear in the plot due to the intensity interpolation performed in the resizing and alignment processes. [sent-182, score-0.537]
29 The plot shows some evidence of saturation in the lightest intensities of the righthand source image (vertical axis of the scatter plot). [sent-184, score-0.81]
30 Since this saturation is in the source image, before printing, it should have no significant influence on the mixture and separation processes. [sent-185, score-0.752]
31 • The third and fourth scatter plots also show that the corresponding source pairs are approximately independent. [sent-186, score-0.534]
32 From left to right: source images, acquired images, linear separation and nonlinear separation. [sent-191, score-0.976]
33 In each scatter plot, the horizontal axis corresponds to intensities from the left-hand image and the vertical axis to intensities from the right-hand image. [sent-194, score-0.681]
34 From left to right: source images, acquired images, linear separation and nonlinear separation. [sent-198, score-0.976]
35 The plot shows some evidence of intensity quantization in the darkest levels of the left-hand source image (horizontal axis of the scatter plot), and of saturation in the lightest intensities of the same image. [sent-202, score-0.994]
36 Since the quantization and saturation are in the source image, before printing, they should have no significant influence on the mixture and separation processes. [sent-203, score-0.787]
37 2 The Mixture Process: Printing and Acquisition The images from each pair were printed on opposite faces of a sheet of onion skin paper. [sent-205, score-0.471]
38 We tried to keep the printing and acquisition processes as symmetrical as possible: the two source images in each pair were handled in an identical way, and the two acquired mixture images in each pair were also handled in an identical way. [sent-210, score-1.122]
39 3 Preprocessing In the preprocessing stage, in each pair of acquired images one of them was first horizontally flipped, so that both images would have the same orientation. [sent-227, score-0.644]
40 In the alignment procedure that was finally adopted, the first step consisted just of a manual displacement of one of the images by an integer number of pixels in each direction, so that the two images would be coarsely aligned with each other. [sent-233, score-0.769]
41 There is a large variety of image alignment methods described in the literature, varying due to such aspects as the kinds of images to be aligned, the purpose of the alignment, etc. [sent-241, score-0.546]
42 1209 A LMEIDA As a final preprocessing step, the intensity range of each pair of images was normalized to the interval [0, 1], 0 corresponding to the darkest pixel in the image pair and 1 to the lightest one. [sent-244, score-0.667]
43 The separation results that we present ahead, based on a symmetry constraint, seem to confirm that the mixture process was kept very close to symmetrical, despite the asymmetry in the alignment procedure. [sent-249, score-0.706]
44 Separation Results One of the main purposes of the work reported in this paper was to assess the viability and the advantage of performing nonlinear source separation, in a real-life nonlinear mixture problem, by means of an ICA-based separation system. [sent-251, score-1.211]
45 Therefore we used source separation by linear ICA as a baseline for comparison. [sent-252, score-0.626]
46 The next sections present the results of separation by linear and nonlinear ICA, followed by an assessment of the results with objective quality measures. [sent-253, score-0.736]
47 The mixture process that we used was as symmetrical as possible, so that an exchange of the source images should result just in a corresponding exchange of the mixture images (apart from noise). [sent-254, score-0.893]
48 Therefore we applied symmetry constraints to the separation systems, as detailed ahead. [sent-255, score-0.453]
49 For each image pair, ten runs of the separation were made. [sent-264, score-0.582]
50 We see that a reasonable degree of separation was achieved in all cases, but some interference remained. [sent-269, score-0.456]
51 4 and 5 (third column) show that, although a certain amount of separation was achieved, the nonlinear character of the mixture could not be undone by linear ICA, as expected. [sent-271, score-0.802]
52 All images of mixtures and of separation results displayed in this paper were adjusted in brightness and contrast so as to saturate the 1% brightest and 1% darkest pixels. [sent-274, score-0.778]
53 This adjustment was performed for image display only: not for image separation and also not for the computation of quality measures. [sent-276, score-0.802]
54 2 Nonlinear Separation For nonlinear separation we used MISEP with a nonlinear F block. [sent-282, score-0.918]
55 Since the output units were linear, the block could implement linear separation exactly, by setting the weights of the hidden layer’s connections to zero. [sent-286, score-0.488]
56 The ψ blocks had the same structure as in the linear separation case. [sent-293, score-0.455]
57 Figures 12 and 13 show the worst separation results that were obtained (“worst” again according to Q2 ). [sent-300, score-0.453]
58 3 Measures of Separation Quality The images shown in the previous section give an idea of the separation quality, but their evaluation is rather subjective. [sent-302, score-0.656]
59 Experience with objective quality measures for nonlinear source separation is still very limited. [sent-306, score-0.936]
60 3 We should note that, in a nonlinear separation context, the SNR, besides being sensitive to incomplete source separation and to noise, is also sensitive to any nonlinear transformation of the intensity scale that may be caused by the mixture and separation processes. [sent-309, score-2.179]
61 1219 A LMEIDA source, including any nonlinear transformation of the intensity scale, besides including incomplete separation and noise. [sent-318, score-0.788]
62 It was given by variance of S , (4) Q2 = variance of N where S was the source image and N was the noise that was present in the extracted component. [sent-321, score-0.499]
63 In other terms, we chose a nonlinear, monotonic transformation of the intensity scale of the extracted component that made it become as close as possible to the corresponding source in SNR terms, and then used its SNR as the quality measure. [sent-323, score-0.53]
64 As a reference for assessing the amount of separation achieved by the various methods, we show in Table 1 the values of the quality measures for the mixture components after preprocessing, without any separation. [sent-338, score-0.62]
65 1220 S EPARATING A R EAL -L IFE N ONLINEAR I MAGE M IXTURE Image pair 1 2 3 4 5 Quality measure Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 No separation source 1 source 2 1. [sent-341, score-0.873]
66 1221 A LMEIDA Image pair 1 2 3 4 5 Quality measure Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Linear separation source 1 source 2 9. [sent-383, score-0.873]
67 The cases in which the difference between linear and nonlinear separation was significant at the 95% confidence level are shown in bold in the table. [sent-467, score-0.672]
68 The measure that seemed to correlate best with our subjective evaluation of separation quality was Q2 , and this is why we chose it for the selection of the “best” and “worst” examples shown in Sections 5. [sent-468, score-0.545]
69 4 Assessment of the Results For the first three image pairs, both the objective quality measures and our subjective evaluation showed a clear advantage of nonlinear separation over linear separation. [sent-474, score-0.947]
70 Even the worst results of nonlinear separation seemed to be better, in general, than the best results of linear separation. [sent-475, score-0.699]
71 The source scatter plot is dominated by two lines of points, located on the top and right-hand edges of the plot. [sent-484, score-0.466]
72 But this represented a rather small percentage of pixels, and had little impact on the overall separation quality. [sent-487, score-0.453]
73 We also see, from the rightmost scatter plot, that nonlinear separation also left the lower-left area unfilled. [sent-488, score-0.887]
74 The results for the fifth image pair show that one of the sources was best separated by the linear method, while the other was best separated by the nonlinear one. [sent-493, score-0.625]
75 Nonlinear separation apparently suffered a negative impact from the fact that the sources were not independent from each other and we were using independence as the separation criterion. [sent-495, score-0.934]
76 The nonlinear separation network had many more degrees of freedom than the linear one, and used them to try to make the extracted components more independent from each other. [sent-496, score-0.78]
77 In doing so it impaired the separation of one of the sources, instead of improving it, since the actual sources were not independent. [sent-497, score-0.508]
78 This is clear from the separation images that were shown (which were only normalized in brightness and contrast, as mentioned above) and from the values of the Q1 measure. [sent-499, score-0.689]
79 This figure shows a scatter plot of the first extracted component versus the corresponding source, for the “average” case of the first image pair (the “average” case was chosen as the one whose value of Q2 was closest to the average for the ten runs). [sent-502, score-0.619]
80 In previous tests in which these networks had only 6 hidden units, the separation results, as measured by Q2 , Q3 or Q4 were not very different from those presented here, but there often was a significant amount of nonlinearity introduced in the extracted components. [sent-506, score-0.601]
81 One of them has to do with the amount of noise introduced 1223 A LMEIDA Figure 14: Scatter plot of the first extracted component versus the corresponding source, in an “average” run of nonlinear separation of the first image pair. [sent-509, score-1.101]
82 We can take advantage of the fact that the source images that contain text have a large percentage of purely white pixels, which show up as strong, very thin lines in the corresponding scatter plots in the first column of Figs. [sent-512, score-0.747]
83 After the mixture, and also after linear or nonlinear separation, these lines appear broadened in the scatter plots, looking like fuzzy dark bands. [sent-514, score-0.494]
84 In the separation results the noise represents a significant percentage of the whole intensity range. [sent-516, score-0.604]
85 We were surprised by the relatively low values of mutual information between source and extracted component, even when the images looked well separated and Q2 indicated relatively high SNR values after compensation of nonlinearities. [sent-520, score-0.735]
86 For natural scene images, the mutual information between source and extracted component was roughly around 2 bits, while for text images it was below 1 bit. [sent-521, score-0.67]
87 The intensity of each source image at each point appeared to affect the observed mixture intensities in a small neighborhood of that point. [sent-528, score-0.689]
88 This is especially noticeable by closely examining the separation results in the cases in which the image to be suppressed was a text image. [sent-529, score-0.612]
89 This allowed us to use a symmetry constraint in the separation networks. [sent-536, score-0.453]
90 We had no access to images from such scanners, and therefore couldn’t assess what degree of separation would be achievable with them. [sent-541, score-0.656]
91 Conclusion We showed an application of ICA to nonlinear source separation in a real-life problem of practical interest. [sent-554, score-0.872]
92 We should say, however, that it took quite a bit of experimentation to find a set of conditions that could be used for all image pairs, yielding a good separation with relatively little variability in the separation results. [sent-557, score-1.038]
93 We presented comparisons of MISEP-based nonlinear ICA with linear ICA, one of the main purposes being to demonstrate the feasibility and the advantage of nonlinear source separation through ICA in a practical situation. [sent-559, score-1.118]
94 It would also be very interesting to compare the nonlinear separation results presented here with those obtained with other nonlinear separation methods, such as ensemble learning (Lappalainen and Honkela, 2000), kernel-based nonlinear ICA (Harmeling et al. [sent-560, score-1.59]
95 Future work will address several different issues, among which we can mention: • The development of separation criteria that are more adequate for this problem than statistical independence. [sent-569, score-0.461]
96 In such a case the quality of separation suffers. [sent-571, score-0.49]
97 A more adequate separation criterion would not cause such degradation and might also be able to overcome much of the ill-posedness of nonlinear ICA, decreasing the dependence on regularization. [sent-572, score-0.707]
98 • The use of the spatial redundancy of images to reduce the ill-posedness of the problem, hopefully achieving separation with less dependence on regularization. [sent-573, score-0.656]
99 An information-maximization approach to blind separation and blind deconvolution. [sent-633, score-0.596]
100 Advances in blind source separation (BSS) and independent component analysis (ICA) for nonlinear mixtures. [sent-700, score-0.999]
wordName wordTfidf (topN-words)
[('separation', 0.426), ('nonlinear', 0.246), ('ica', 0.237), ('images', 0.23), ('scatter', 0.215), ('source', 0.2), ('misep', 0.178), ('ife', 0.166), ('ixture', 0.166), ('lmeida', 0.166), ('alignment', 0.16), ('image', 0.156), ('almeida', 0.144), ('onlinear', 0.14), ('eal', 0.14), ('mage', 0.14), ('eparating', 0.14), ('intensities', 0.124), ('intensity', 0.116), ('extracted', 0.108), ('acquired', 0.104), ('mixture', 0.093), ('infomax', 0.09), ('mutual', 0.09), ('url', 0.087), ('blind', 0.085), ('printed', 0.084), ('printing', 0.083), ('sources', 0.082), ('scanner', 0.078), ('scanners', 0.067), ('quality', 0.064), ('pixels', 0.063), ('block', 0.062), ('mixtures', 0.056), ('halftoning', 0.055), ('onion', 0.055), ('skin', 0.055), ('subjective', 0.055), ('aligned', 0.053), ('snr', 0.053), ('plot', 0.051), ('jutten', 0.05), ('plots', 0.049), ('separated', 0.047), ('fth', 0.047), ('pair', 0.047), ('symmetrical', 0.047), ('resizing', 0.044), ('fourth', 0.043), ('component', 0.042), ('resolution', 0.042), ('interpolation', 0.042), ('acquisition', 0.041), ('scanning', 0.039), ('pixel', 0.038), ('epochs', 0.038), ('lappalainen', 0.037), ('amount', 0.037), ('hyv', 0.036), ('rinen', 0.036), ('noise', 0.035), ('bell', 0.035), ('adequate', 0.035), ('quantization', 0.035), ('sides', 0.034), ('bicubic', 0.033), ('brightness', 0.033), ('cdf', 0.033), ('darkest', 0.033), ('faria', 0.033), ('horizontally', 0.033), ('marques', 0.033), ('nonlinearities', 0.033), ('scanned', 0.033), ('ipped', 0.033), ('dark', 0.033), ('harmeling', 0.033), ('saturation', 0.033), ('displacement', 0.033), ('experience', 0.032), ('overview', 0.031), ('http', 0.031), ('axis', 0.031), ('noticeable', 0.03), ('bss', 0.03), ('interference', 0.03), ('nonlinearity', 0.03), ('relatively', 0.03), ('blocks', 0.029), ('printer', 0.028), ('multilayer', 0.028), ('honkela', 0.028), ('worst', 0.027), ('symmetry', 0.027), ('pairs', 0.027), ('percentage', 0.027), ('sejnowski', 0.026), ('white', 0.026), ('signal', 0.025)]
simIndex simValue paperId paperTitle
same-paper 1 1.0000014 65 jmlr-2005-Separating a Real-Life Nonlinear Image Mixture
Author: Luís B. Almeida
Abstract: When acquiring an image of a paper document, the image printed on the back page sometimes shows through. The mixture of the front- and back-page images thus obtained is markedly nonlinear, and thus constitutes a good real-life test case for nonlinear blind source separation. This paper addresses a difficult version of this problem, corresponding to the use of “onion skin” paper, which results in a relatively strong nonlinearity of the mixture, which becomes close to singular in the lighter regions of the images. The separation is achieved through the MISEP technique, which is an extension of the well known INFOMAX method. The separation results are assessed with objective quality measures. They show an improvement over the results obtained with linear separation, but have room for further improvement. Keywords: ICA, blind source separation, nonlinear mixtures, nonlinear separation, image mixture, image separation
2 0.21109863 30 jmlr-2005-Estimating Functions for Blind Separation When Sources Have Variance Dependencies
Author: Motoaki Kawanabe, Klaus-Robert Müller
Abstract: A blind separation problem where the sources are not independent, but have variance dependencies is discussed. For this scenario Hyv¨ rinen and Hurri (2004) proposed an algorithm which requires a no assumption on distributions of sources and no parametric model of dependencies between components. In this paper, we extend the semiparametric approach of Amari and Cardoso (1997) to variance dependencies and study estimating functions for blind separation of such dependent sources. In particular, we show that many ICA algorithms are applicable to the variance-dependent model as well under mild conditions, although they should in principle not. Our results indicate that separation can be done based only on normalized sources which are adjusted to have stationary variances and is not affected by the dependent activity levels. We also study the asymptotic distribution of the quasi maximum likelihood method and the stability of the natural gradient learning in detail. Simulation results of artificial and realistic examples match well with our theoretical findings. Keywords: blind source separation, variance dependencies, independent component analysis, semiparametric statistical models, estimating functions
3 0.15800884 25 jmlr-2005-Denoising Source Separation
Author: Jaakko Särelä, Harri Valpola
Abstract: A new algorithmic framework called denoising source separation (DSS) is introduced. The main benefit of this framework is that it allows for the easy development of new source separation algorithms which can be optimised for specific problems. In this framework, source separation algorithms are constructed around denoising procedures. The resulting algorithms can range from almost blind to highly specialised source separation algorithms. Both simple linear and more complex nonlinear or adaptive denoising schemes are considered. Some existing independent component analysis algorithms are reinterpreted within the DSS framework and new, robust blind source separation algorithms are suggested. The framework is derived as a one-unit equivalent to an EM algorithm for source separation. However, in the DSS framework it is easy to utilise various kinds of denoising procedures which need not be based on generative models. In the experimental section, various DSS schemes are applied extensively to artificial data, to real magnetoencephalograms and to simulated CDMA mobile network signals. Finally, various extensions to the proposed DSS algorithms are considered. These include nonlinear observation mappings, hierarchical models and over-complete, nonorthogonal feature spaces. With these extensions, DSS appears to have relevance to many existing models of neural information processing. Keywords: blind source separation, BSS, prior information, denoising, denoising source separation, DSS, independent component analysis, ICA, magnetoencephalograms, MEG, CDMA
4 0.13844617 63 jmlr-2005-Quasi-Geodesic Neural Learning Algorithms Over the Orthogonal Group: A Tutorial
Author: Simone Fiori
Abstract: The aim of this contribution is to present a tutorial on learning algorithms for a single neural layer whose connection matrix belongs to the orthogonal group. The algorithms exploit geodesics appropriately connected as piece-wise approximate integrals of the exact differential learning equation. The considered learning equations essentially arise from the Riemannian-gradient-based optimization theory with deterministic and diffusion-type gradient. The paper aims specifically at reviewing the relevant mathematics (and at presenting it in as much transparent way as possible in order to make it accessible to readers that do not possess a background in differential geometry), at bringing together modern optimization methods on manifolds and at comparing the different algorithms on a common machine learning problem. As a numerical case-study, we consider an application to non-negative independent component analysis, although it should be recognized that Riemannian gradient methods give rise to general-purpose algorithms, by no means limited to ICA-related applications. Keywords: differential geometry, diffusion-type gradient, Lie groups, non-negative independent component analysis, Riemannian gradient
5 0.088465601 9 jmlr-2005-Active Learning to Recognize Multiple Types of Plankton
Author: Tong Luo, Kurt Kramer, Dmitry B. Goldgof, Lawrence O. Hall, Scott Samson, Andrew Remsen, Thomas Hopkins
Abstract: This paper presents an active learning method which reduces the labeling effort of domain experts in multi-class classification problems. Active learning is applied in conjunction with support vector machines to recognize underwater zooplankton from higher-resolution, new generation SIPPER II images. Most previous work on active learning with support vector machines only deals with two class problems. In this paper, we propose an active learning approach “breaking ties” for multiclass support vector machines using the one-vs-one approach with a probability approximation. Experimental results indicate that our approach often requires significantly less labeled images to reach a given accuracy than the approach of labeling the least certain test example and random sampling. It can also be applied in batch mode resulting in an accuracy comparable to labeling one image at a time and retraining. Keywords: active learning, support vector machine, plankton recognition, probabilistic output, multi-class support vector machine
6 0.082570426 41 jmlr-2005-Kernel Methods for Measuring Independence
7 0.077875182 31 jmlr-2005-Estimation of Non-Normalized Statistical Models by Score Matching
8 0.055426732 18 jmlr-2005-Characterization of a Family of Algorithms for Generalized Discriminant Analysis on Undersampled Problems
9 0.046510667 64 jmlr-2005-Semigroup Kernels on Measures
10 0.045812629 42 jmlr-2005-Large Margin Methods for Structured and Interdependent Output Variables
11 0.041917302 62 jmlr-2005-Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models
12 0.040504109 49 jmlr-2005-Learning the Kernel with Hyperkernels (Kernel Machines Section)
13 0.038629498 34 jmlr-2005-Feature Selection for Unsupervised and Supervised Inference: The Emergence of Sparsity in a Weight-Based Approach
14 0.035541851 2 jmlr-2005-A Bayesian Model for Supervised Clustering with the Dirichlet Process Prior
15 0.030949719 19 jmlr-2005-Clustering on the Unit Hypersphere using von Mises-Fisher Distributions
16 0.030877266 27 jmlr-2005-Dimension Reduction in Text Classification with Support Vector Machines
17 0.026949733 46 jmlr-2005-Learning a Mahalanobis Metric from Equivalence Constraints
18 0.025832191 43 jmlr-2005-Learning Hidden Variable Networks: The Information Bottleneck Approach
19 0.025022531 4 jmlr-2005-A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data
20 0.024561044 54 jmlr-2005-Managing Diversity in Regression Ensembles
topicId topicWeight
[(0, 0.203), (1, 0.416), (2, -0.349), (3, -0.014), (4, -0.048), (5, -0.145), (6, 0.07), (7, 0.188), (8, -0.001), (9, -0.079), (10, 0.136), (11, 0.092), (12, -0.122), (13, -0.033), (14, 0.033), (15, -0.113), (16, -0.01), (17, -0.058), (18, -0.035), (19, 0.052), (20, 0.001), (21, -0.018), (22, 0.054), (23, -0.055), (24, 0.011), (25, -0.035), (26, 0.035), (27, -0.009), (28, 0.087), (29, -0.017), (30, -0.0), (31, -0.016), (32, -0.012), (33, 0.017), (34, -0.026), (35, -0.095), (36, -0.021), (37, -0.007), (38, 0.009), (39, -0.076), (40, 0.083), (41, -0.015), (42, 0.131), (43, 0.021), (44, -0.019), (45, -0.063), (46, -0.035), (47, -0.052), (48, 0.029), (49, 0.074)]
simIndex simValue paperId paperTitle
same-paper 1 0.99076349 65 jmlr-2005-Separating a Real-Life Nonlinear Image Mixture
Author: Luís B. Almeida
Abstract: When acquiring an image of a paper document, the image printed on the back page sometimes shows through. The mixture of the front- and back-page images thus obtained is markedly nonlinear, and thus constitutes a good real-life test case for nonlinear blind source separation. This paper addresses a difficult version of this problem, corresponding to the use of “onion skin” paper, which results in a relatively strong nonlinearity of the mixture, which becomes close to singular in the lighter regions of the images. The separation is achieved through the MISEP technique, which is an extension of the well known INFOMAX method. The separation results are assessed with objective quality measures. They show an improvement over the results obtained with linear separation, but have room for further improvement. Keywords: ICA, blind source separation, nonlinear mixtures, nonlinear separation, image mixture, image separation
2 0.79188657 30 jmlr-2005-Estimating Functions for Blind Separation When Sources Have Variance Dependencies
Author: Motoaki Kawanabe, Klaus-Robert Müller
Abstract: A blind separation problem where the sources are not independent, but have variance dependencies is discussed. For this scenario Hyv¨ rinen and Hurri (2004) proposed an algorithm which requires a no assumption on distributions of sources and no parametric model of dependencies between components. In this paper, we extend the semiparametric approach of Amari and Cardoso (1997) to variance dependencies and study estimating functions for blind separation of such dependent sources. In particular, we show that many ICA algorithms are applicable to the variance-dependent model as well under mild conditions, although they should in principle not. Our results indicate that separation can be done based only on normalized sources which are adjusted to have stationary variances and is not affected by the dependent activity levels. We also study the asymptotic distribution of the quasi maximum likelihood method and the stability of the natural gradient learning in detail. Simulation results of artificial and realistic examples match well with our theoretical findings. Keywords: blind source separation, variance dependencies, independent component analysis, semiparametric statistical models, estimating functions
3 0.73420739 25 jmlr-2005-Denoising Source Separation
Author: Jaakko Särelä, Harri Valpola
Abstract: A new algorithmic framework called denoising source separation (DSS) is introduced. The main benefit of this framework is that it allows for the easy development of new source separation algorithms which can be optimised for specific problems. In this framework, source separation algorithms are constructed around denoising procedures. The resulting algorithms can range from almost blind to highly specialised source separation algorithms. Both simple linear and more complex nonlinear or adaptive denoising schemes are considered. Some existing independent component analysis algorithms are reinterpreted within the DSS framework and new, robust blind source separation algorithms are suggested. The framework is derived as a one-unit equivalent to an EM algorithm for source separation. However, in the DSS framework it is easy to utilise various kinds of denoising procedures which need not be based on generative models. In the experimental section, various DSS schemes are applied extensively to artificial data, to real magnetoencephalograms and to simulated CDMA mobile network signals. Finally, various extensions to the proposed DSS algorithms are considered. These include nonlinear observation mappings, hierarchical models and over-complete, nonorthogonal feature spaces. With these extensions, DSS appears to have relevance to many existing models of neural information processing. Keywords: blind source separation, BSS, prior information, denoising, denoising source separation, DSS, independent component analysis, ICA, magnetoencephalograms, MEG, CDMA
4 0.41629878 63 jmlr-2005-Quasi-Geodesic Neural Learning Algorithms Over the Orthogonal Group: A Tutorial
Author: Simone Fiori
Abstract: The aim of this contribution is to present a tutorial on learning algorithms for a single neural layer whose connection matrix belongs to the orthogonal group. The algorithms exploit geodesics appropriately connected as piece-wise approximate integrals of the exact differential learning equation. The considered learning equations essentially arise from the Riemannian-gradient-based optimization theory with deterministic and diffusion-type gradient. The paper aims specifically at reviewing the relevant mathematics (and at presenting it in as much transparent way as possible in order to make it accessible to readers that do not possess a background in differential geometry), at bringing together modern optimization methods on manifolds and at comparing the different algorithms on a common machine learning problem. As a numerical case-study, we consider an application to non-negative independent component analysis, although it should be recognized that Riemannian gradient methods give rise to general-purpose algorithms, by no means limited to ICA-related applications. Keywords: differential geometry, diffusion-type gradient, Lie groups, non-negative independent component analysis, Riemannian gradient
5 0.34606844 9 jmlr-2005-Active Learning to Recognize Multiple Types of Plankton
Author: Tong Luo, Kurt Kramer, Dmitry B. Goldgof, Lawrence O. Hall, Scott Samson, Andrew Remsen, Thomas Hopkins
Abstract: This paper presents an active learning method which reduces the labeling effort of domain experts in multi-class classification problems. Active learning is applied in conjunction with support vector machines to recognize underwater zooplankton from higher-resolution, new generation SIPPER II images. Most previous work on active learning with support vector machines only deals with two class problems. In this paper, we propose an active learning approach “breaking ties” for multiclass support vector machines using the one-vs-one approach with a probability approximation. Experimental results indicate that our approach often requires significantly less labeled images to reach a given accuracy than the approach of labeling the least certain test example and random sampling. It can also be applied in batch mode resulting in an accuracy comparable to labeling one image at a time and retraining. Keywords: active learning, support vector machine, plankton recognition, probabilistic output, multi-class support vector machine
6 0.26960191 41 jmlr-2005-Kernel Methods for Measuring Independence
7 0.22024803 31 jmlr-2005-Estimation of Non-Normalized Statistical Models by Score Matching
9 0.19473405 64 jmlr-2005-Semigroup Kernels on Measures
10 0.17631891 42 jmlr-2005-Large Margin Methods for Structured and Interdependent Output Variables
11 0.1627098 49 jmlr-2005-Learning the Kernel with Hyperkernels (Kernel Machines Section)
12 0.14856674 19 jmlr-2005-Clustering on the Unit Hypersphere using von Mises-Fisher Distributions
13 0.14603204 27 jmlr-2005-Dimension Reduction in Text Classification with Support Vector Machines
14 0.14080264 46 jmlr-2005-Learning a Mahalanobis Metric from Equivalence Constraints
15 0.14057973 62 jmlr-2005-Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models
16 0.13398165 2 jmlr-2005-A Bayesian Model for Supervised Clustering with the Dirichlet Process Prior
17 0.12659506 43 jmlr-2005-Learning Hidden Variable Networks: The Information Bottleneck Approach
18 0.12228352 33 jmlr-2005-Fast Kernel Classifiers with Online and Active Learning
19 0.11815846 4 jmlr-2005-A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data
20 0.11516636 58 jmlr-2005-Multiclass Classification with Multi-Prototype Support Vector Machines
topicId topicWeight
[(1, 0.051), (13, 0.025), (17, 0.018), (19, 0.018), (20, 0.389), (36, 0.047), (37, 0.038), (42, 0.015), (43, 0.036), (47, 0.013), (52, 0.081), (59, 0.024), (70, 0.036), (80, 0.038), (88, 0.045), (90, 0.024), (94, 0.028)]
simIndex simValue paperId paperTitle
same-paper 1 0.78164935 65 jmlr-2005-Separating a Real-Life Nonlinear Image Mixture
Author: Luís B. Almeida
Abstract: When acquiring an image of a paper document, the image printed on the back page sometimes shows through. The mixture of the front- and back-page images thus obtained is markedly nonlinear, and thus constitutes a good real-life test case for nonlinear blind source separation. This paper addresses a difficult version of this problem, corresponding to the use of “onion skin” paper, which results in a relatively strong nonlinearity of the mixture, which becomes close to singular in the lighter regions of the images. The separation is achieved through the MISEP technique, which is an extension of the well known INFOMAX method. The separation results are assessed with objective quality measures. They show an improvement over the results obtained with linear separation, but have room for further improvement. Keywords: ICA, blind source separation, nonlinear mixtures, nonlinear separation, image mixture, image separation
2 0.31128094 25 jmlr-2005-Denoising Source Separation
Author: Jaakko Särelä, Harri Valpola
Abstract: A new algorithmic framework called denoising source separation (DSS) is introduced. The main benefit of this framework is that it allows for the easy development of new source separation algorithms which can be optimised for specific problems. In this framework, source separation algorithms are constructed around denoising procedures. The resulting algorithms can range from almost blind to highly specialised source separation algorithms. Both simple linear and more complex nonlinear or adaptive denoising schemes are considered. Some existing independent component analysis algorithms are reinterpreted within the DSS framework and new, robust blind source separation algorithms are suggested. The framework is derived as a one-unit equivalent to an EM algorithm for source separation. However, in the DSS framework it is easy to utilise various kinds of denoising procedures which need not be based on generative models. In the experimental section, various DSS schemes are applied extensively to artificial data, to real magnetoencephalograms and to simulated CDMA mobile network signals. Finally, various extensions to the proposed DSS algorithms are considered. These include nonlinear observation mappings, hierarchical models and over-complete, nonorthogonal feature spaces. With these extensions, DSS appears to have relevance to many existing models of neural information processing. Keywords: blind source separation, BSS, prior information, denoising, denoising source separation, DSS, independent component analysis, ICA, magnetoencephalograms, MEG, CDMA
3 0.29937676 30 jmlr-2005-Estimating Functions for Blind Separation When Sources Have Variance Dependencies
Author: Motoaki Kawanabe, Klaus-Robert Müller
Abstract: A blind separation problem where the sources are not independent, but have variance dependencies is discussed. For this scenario Hyv¨ rinen and Hurri (2004) proposed an algorithm which requires a no assumption on distributions of sources and no parametric model of dependencies between components. In this paper, we extend the semiparametric approach of Amari and Cardoso (1997) to variance dependencies and study estimating functions for blind separation of such dependent sources. In particular, we show that many ICA algorithms are applicable to the variance-dependent model as well under mild conditions, although they should in principle not. Our results indicate that separation can be done based only on normalized sources which are adjusted to have stationary variances and is not affected by the dependent activity levels. We also study the asymptotic distribution of the quasi maximum likelihood method and the stability of the natural gradient learning in detail. Simulation results of artificial and realistic examples match well with our theoretical findings. Keywords: blind source separation, variance dependencies, independent component analysis, semiparametric statistical models, estimating functions
4 0.2873233 32 jmlr-2005-Expectation Consistent Approximate Inference
Author: Manfred Opper, Ole Winther
Abstract: We propose a novel framework for approximations to intractable probabilistic models which is based on a free energy formulation. The approximation can be understood as replacing an average over the original intractable distribution with a tractable one. It requires two tractable probability distributions which are made consistent on a set of moments and encode different features of the original intractable distribution. In this way we are able to use Gaussian approximations for models with discrete or bounded variables which allow us to include non-trivial correlations. These are neglected in many other methods. We test the framework on toy benchmark problems for binary variables on fully connected graphs and 2D grids and compare with other methods, such as loopy belief propagation. Good performance is already achieved by using single nodes as tractable substructures. Significant improvements are obtained when a spanning tree is used instead.
5 0.28424925 33 jmlr-2005-Fast Kernel Classifiers with Online and Active Learning
Author: Antoine Bordes, Seyda Ertekin, Jason Weston, Léon Bottou
Abstract: Very high dimensional learning systems become theoretically possible when training examples are abundant. The computing cost then becomes the limiting factor. Any efficient learning algorithm should at least take a brief look at each example. But should all examples be given equal attention? This contribution proposes an empirical answer. We first present an online SVM algorithm based on this premise. LASVM yields competitive misclassification rates after a single pass over the training examples, outspeeding state-of-the-art SVM solvers. Then we show how active example selection can yield faster training, higher accuracies, and simpler models, using only a fraction of the training example labels.
6 0.27790362 36 jmlr-2005-Gaussian Processes for Ordinal Regression
7 0.27759567 64 jmlr-2005-Semigroup Kernels on Measures
8 0.27755937 46 jmlr-2005-Learning a Mahalanobis Metric from Equivalence Constraints
9 0.27679318 49 jmlr-2005-Learning the Kernel with Hyperkernels (Kernel Machines Section)
10 0.27192339 9 jmlr-2005-Active Learning to Recognize Multiple Types of Plankton
11 0.27103472 31 jmlr-2005-Estimation of Non-Normalized Statistical Models by Score Matching
12 0.27029416 24 jmlr-2005-Core Vector Machines: Fast SVM Training on Very Large Data Sets
13 0.26944029 71 jmlr-2005-Variational Message Passing
14 0.26796803 19 jmlr-2005-Clustering on the Unit Hypersphere using von Mises-Fisher Distributions
15 0.26515591 63 jmlr-2005-Quasi-Geodesic Neural Learning Algorithms Over the Orthogonal Group: A Tutorial
16 0.26474234 45 jmlr-2005-Learning Multiple Tasks with Kernel Methods
17 0.26415384 54 jmlr-2005-Managing Diversity in Regression Ensembles
18 0.26287898 3 jmlr-2005-A Classification Framework for Anomaly Detection
19 0.26214442 39 jmlr-2005-Information Bottleneck for Gaussian Variables
20 0.26168522 35 jmlr-2005-Frames, Reproducing Kernels, Regularization and Learning