nips nips2008 nips2008-160 knowledge-graph by maker-knowledge-mining

160 nips-2008-On Computational Power and the Order-Chaos Phase Transition in Reservoir Computing

Source: pdf

Author: Benjamin Schrauwen, Lars Buesing, Robert A. Legenstein

Abstract: Randomly connected recurrent neural circuits have proven to be very powerful models for online computations when a trained memoryless readout function is appended. Such Reservoir Computing (RC) systems are commonly used in two ﬂavors: with analog or binary (spiking) neurons in the recurrent circuits. Previous work showed a fundamental difference between these two incarnations of the RC idea. The performance of a RC system built from binary neurons seems to depend strongly on the network connectivity structure. In networks of analog neurons such dependency has not been observed. In this article we investigate this apparent dichotomy in terms of the in-degree of the circuit nodes. Our analyses based amongst others on the Lyapunov exponent reveal that the phase transition between ordered and chaotic network behavior of binary circuits qualitatively differs from the one in analog circuits. This explains the observed decreased computational performance of binary circuits of high node in-degree. Furthermore, a novel mean-ﬁeld predictor for computational performance is introduced and shown to accurately predict the numerically obtained results. 1

Reference: text

Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 at Abstract Randomly connected recurrent neural circuits have proven to be very powerful models for online computations when a trained memoryless readout function is appended. [sent-5, score-0.355]

2 Such Reservoir Computing (RC) systems are commonly used in two ﬂavors: with analog or binary (spiking) neurons in the recurrent circuits. [sent-6, score-0.327]

3 The performance of a RC system built from binary neurons seems to depend strongly on the network connectivity structure. [sent-8, score-0.408]

4 In networks of analog neurons such dependency has not been observed. [sent-9, score-0.293]

5 Our analyses based amongst others on the Lyapunov exponent reveal that the phase transition between ordered and chaotic network behavior of binary circuits qualitatively differs from the one in analog circuits. [sent-11, score-0.858]

6 This explains the observed decreased computational performance of binary circuits of high node in-degree. [sent-12, score-0.309]

7 Furthermore, a novel mean-ﬁeld predictor for computational performance is introduced and shown to accurately predict the numerically obtained results. [sent-13, score-0.316]

8 1 Introduction In 2001, Jaeger [1] and Maass [2] independently introduced the idea of using a ﬁxed, randomly connected recurrent neural network of simple units as a set of basis ﬁlters (operating at the edge-ofstability where the system has fading memory). [sent-14, score-0.358]

9 A memoryless readout is then trained on these basis ﬁlters in order to approximate a given time-invariant target operator with fading memory [2]. [sent-15, score-0.261]

10 Jaeger used analog sigmoidal neurons as network units and named the model Echo State Network (ESN). [sent-16, score-0.406]

11 Maass termed the idea Liquid State Machine (LSM) and most of the related literature focuses on networks of spiking neurons or threshold units. [sent-17, score-0.286]

12 a network of interacting optical ampliﬁers [3]) – the so-called reservoirs – in conjunction with trained memoryless readout functions as computational devices. [sent-20, score-0.556]

13 largely independent of the sparsity of the 1 network [8] or the exact network topology such as small-world or scale-free connectivity graphs1 . [sent-25, score-0.421]

14 In the results of [10] it can be observed (although not speciﬁcally stated there) that for networks of threshold units with a simple connectivity topology of ﬁxed in-degree per neuron, an increase in performance can be found for decreasing in-degree. [sent-30, score-0.29]

15 The reservoir of a quantized ESN is deﬁned as a network of discrete units, where the number of admissible states of a single unit is controlled by a parameter called quantization level. [sent-34, score-1.044]

16 LSMs and ESNs can be interpreted as the two limiting cases of quantized ESNs for low and high quantization level respectively. [sent-35, score-0.57]

17 We numerically study the inﬂuence of the network topology in terms of the in-degree of the network units on the computational performance of quantized ESNs for different quantization levels. [sent-36, score-1.07]

18 3 the empirical results are analyzed by studying the Lyapunov exponent of quantized ESNs, which exhibits a clear relation to the computational performance [11]. [sent-39, score-0.327]

19 It is shown that for ESNs with low quantization level, the chaos-order phase transition is signiﬁcantly more gradual when the networks are sparsely connected. [sent-40, score-0.681]

20 It is exactly in this transition regime that the computational power of a Reservoir Computing system is found to be optimal [11]. [sent-41, score-0.256]

21 This effect disappears for ESNs with high quantization level. [sent-42, score-0.407]

22 A clear explanation of the inﬂuence of the in-degree on the computational performance can be found by investigating the rank measure presented in [11]. [sent-43, score-0.212]

23 This measure characterizes the computational capabilities of a network as a trade-off between the so-called kernel quality and the generalization ability. [sent-44, score-0.312]

24 We show that for highly connected reservoirs with a low quantization level the region of an efﬁcient trade-off implying high performance is narrow. [sent-45, score-0.669]

25 Consistently for high quantization levels the region is found to be independent of the interconnection degree. [sent-47, score-0.511]

26 4 we present a novel mean-ﬁeld predictor for computational power which is able to reproduce the inﬂuence of the topology on the quantized ESN model. [sent-49, score-0.479]

27 It is related to the predictor introduced in [10], but it can be calculated for all quantization levels, and can be determined with a signiﬁcantly reduced computation time. [sent-50, score-0.507]

28 2 Online Computations with Quantized ESNs We consider networks of N neurons with the state variable x(t) = (x1 (t), . [sent-52, score-0.231]

29 The network state is updated according to:   N xi (t + 1) = (ψm ◦ g)  j=1 wij xj (t) + u(t) , where g = tanh is the usual hyperbolic tangent nonlinearity and u denotes the input common to all units. [sent-60, score-0.267]

30 The function ψm (·) is called quantization function for m bits as it maps from (−1, 1) to its discrete range Sm of cardinality 2m : 2⌊2m−1 (x + 1)⌋ + 1 ψm : (−1, 1) → Sm , ψm (x) := − 1. [sent-62, score-0.435]

31 2 A m=1 B m=3 C m=6 Figure 1: The performance pexp (C, PAR5 ) for three different quantization levels m = 1, 3, 6 is plotted as a function of the network in-degree K and the weight STD σ. [sent-69, score-0.809]

32 The networks size is N = 150, the results have been averaged over 10 circuits C, initial conditions and randomly drawn input time series of length 104 time steps. [sent-70, score-0.272]

33 We consider in this article tasks where the binary target output at time t depends solely on the n input bits u(t − τ − 1), . [sent-73, score-0.348]

34 In order to approximate the target output, a linear classiﬁer of N the form sign( i=1 αi xi (t) + b) is applied to the instantaneous network state x(t). [sent-82, score-0.242]

35 The RC system consisting of the network and the linear classiﬁer is called a quantized ESN of quantization level m in the remainder of this paper. [sent-84, score-0.681]

36 We assessed the computational capabilities of a given network based on the numerically determined performance on an example task, which was chosen to be the τ -delayed parity function of n bits n PARn,τ , i. [sent-85, score-0.472]

37 We deﬁne pexp quantifying the performance of a given circuit C on the PARn task as: ∞ pexp (C, PARn ) := κ(C, PARn,τ ), (1) τ =0 where κ(C, PARn,τ ) denotes the performance of circuit C on the PARn,τ task measured in terms of Cohen’s kappa coefﬁcient2 . [sent-89, score-0.642]

38 The performance results for PARn can be considered representative for the general computational capabilities of a circuit C as qualitatively very similar results were obtained for the ANDn task of n bits and random Boolean functions of n bit (results not shown). [sent-90, score-0.396]

39 1 the performance pexp (C, PAR5 ) is shown averaged over 10 circuits C for three different quantization levels m = 1, 3, 6. [sent-92, score-0.687]

40 pexp (C, PAR5 ) is plotted as a function of the network in-degree K and the logarithm3 of the weight STD σ. [sent-93, score-0.408]

41 One can see that for ESNs with low quantization levels (m = 1, 3), networks with a small in-degree K reach a signiﬁcantly better peak performance than those with 2 κ is deﬁned as (c − cl )/(1 − cl ) where c is the fraction of correct trials and cl is the chance level. [sent-101, score-0.612]

42 (1) was truncated at τ = 8, as the performance was negligible for higher delays τ > 8 for the network size N = 150. [sent-103, score-0.206]

43 3 quantization m=1bit 0 B1 λ λ A1 quantization m=6bit 0 K=3 K=12 K=24 −1 −0. [sent-112, score-0.636]

44 1 log(σ)−log(σ0) Figure 2: Phase transitions in binary networks (m = 1) differ from phase transition in high resolution networks (m = 6). [sent-116, score-0.671]

45 The transition sharpens with increasing K for binary reservoirs (A), whereas it is virtually independent of K for high resolution reservoirs (B). [sent-121, score-0.767]

46 The effect disappears for a high quantization level (m = 6). [sent-123, score-0.407]

47 This phenomenon is consistent with the observation that network connectivity structure is in general an important issue if the reservoir is composed of binary or spiking neurons but less important if analog neurons are employed. [sent-124, score-0.921]

48 3 Phase Transitions in Binary and High Resolution Networks Where does the difference between binary and high resolution reservoirs shown in Fig. [sent-126, score-0.439]

49 It was often hypothesized that high computational power in recurrent networks is located in a parameter regime near the critical line, i. [sent-128, score-0.433]

50 , near the phase transition between ordered and chaotic behavior (see, e. [sent-130, score-0.323]

51 Starting from this hypothesis, we investigated whether the network dynamics of binary networks near this transition differs qualitatively from the one of high resolution networks. [sent-134, score-0.676]

52 We estimated the network properties by empirically measuring the Lyapunov exponent λ with the same procedure as in the estimation of the critical line in Fig. [sent-135, score-0.323]

53 For binary networks the transition becomes much sharper with increasing K which is not the case for high resolution networks. [sent-145, score-0.468]

54 How can this sharp transition explain the reduced computational performance of binary ESNs with high in-degree K? [sent-146, score-0.331]

55 Hence, the network dynamics has to be located in a regime where memory about recent inputs is available and past input bits do not interfere with that memory. [sent-148, score-0.521]

56 Intuitively, an effect of the sharper phase transition could be stated in the following way. [sent-149, score-0.218]

57 We estimated two measures of the reservoir, the so called “kernelquality” and the “generalization rank”, both being the rank of a matrix consisting of certain state vectors of the reservoir. [sent-155, score-0.206]

58 −1 0 log(σ) 1 Figure 3: Kernel-quality and generalization rank of quantized ESNs of size N = 150. [sent-161, score-0.373]

59 Upper plots are for binary reservoirs (m = 1bit), lower plots for high resolution reservoirs (m = 6 bit). [sent-162, score-0.731]

60 A) The difference between the kernel-quality and the generalization rank as a function of the log STD of weights and the in-degree K. [sent-163, score-0.212]

61 5 Intuitively, this rank measures how well the reservoir represents different input streams. [sent-170, score-0.535]

62 , uN (·) such that the last three ˜ ˜ input bits in all these input streams were identical. [sent-176, score-0.288]

63 6 The generalization rank is then given by the rank of the N × N matrix whose columns are the circuit states resulting from these input streams. [sent-177, score-0.525]

64 Intuitively, the generalization rank with this input distribution measures how strongly the reservoir state at time t is sensitive to inputs older than three time steps. [sent-178, score-0.67]

65 The rank measures calculated here will thus have predictive power for computations which require memory of the last three time steps (see [11] for a theoretical justiﬁcation of the measures). [sent-179, score-0.291]

66 In general, a high kernel-quality and a low generalization rank (corresponding to a high ability of the network to generalize) are desirable. [sent-180, score-0.456]

67 3A and D show the difference between the two measures as a function of log(σ) and the indegree K for binary networks and high resolution networks respectively. [sent-182, score-0.487]

68 The plots show that the peak value of this difference is decreasing with K in binary networks, whereas it is independent of K in high resolution reservoirs, reproducing the observations in the plots for the computational performance. [sent-183, score-0.363]

69 For K = 24, the reservoir increases its separation power very fast as log(σ) increases. [sent-188, score-0.421]

70 In comparison, the corresponding plots for high resolution reservoirs (Figs. [sent-191, score-0.411]

71 The rank of the matrix was estimated by singular value decomposition on the network states after 15 time steps of simulation. [sent-195, score-0.356]

72 ˜ ˜ 5 A m=1 m=3 B m=6 C Figure 4: Mean-ﬁeld predictor p∞ for computational power for different quantization levels m as a function of the STD σ of the weights and in-degree K. [sent-210, score-0.605]

73 Compare this result to the numerically determined performance pexp plotted in Fig. [sent-214, score-0.328]

74 1C, one sees that the rank measure does not accurately predict the whole region of good performance for high resolution reservoirs. [sent-218, score-0.387]

75 It also does not predict the observed bifurcation in the zones of optimal performance, a phenomenon that is reproduced by the mean-ﬁeld predictor introduced in the following section. [sent-219, score-0.32]

76 There, the computational performance of networks of randomly connected threshold gates was linked to their separation property (for a formal deﬁnition see [2]): It was shown that only networks which exhibit sufﬁciently different network states for different instances of the input stream, i. [sent-221, score-0.652]

77 Furthermore, the authors introduced an accurate predictor for the computational capabilities for the considered type of networks based on the separation capability which was quantiﬁed via a simple mean-ﬁeld approximation of the Hamming distance between different network states. [sent-224, score-0.637]

78 Here we aim at extending this approach to a larger class of networks, the class of quantized ESNs introduced above. [sent-225, score-0.225]

79 However a severe problem arises when directly applying the mean-ﬁeld theory developed in [10] to quantized ESNs with a quantization level m > 1: Calculation of the important quantities becomes computationally infeasible as the state space of a network grows exponentially with m. [sent-226, score-0.72]

80 Suppose the target output of the network at time t is a function fT ∈ F = {f |f : {−1, 1}n → {−1, 1}} of the n bits u(t − τ − 1), . [sent-228, score-0.32]

81 In order to exhibit good performance on an arbitrary fT ∈ F , pairs of inputs that differ in at least one of the n bits have to be mapped by the network to different states at time t. [sent-233, score-0.431]

82 In order to quantify this so-called separation property of a given network, we introduce the normalized distance d(k): It measures the average distance between two networks states x1 (t) = (x1 (t), . [sent-235, score-0.27]

83 , x2 (t)) arising from applying to the 1 1 N N same network two input streams u1 (·) and u2 (·) which only differ in the single bit at time t − k, i. [sent-241, score-0.277]

84 is taken over all inputs u1 (·), u2 (·) from the ensemble deﬁned above, all initial conditions of the network and all circuits C. [sent-246, score-0.321]

85 d(k) ≫ 0, τ < k ≤ n + τ , is a necessary but not a sufﬁcient condition for the ability of the network to calculate the target function. [sent-249, score-0.203]

86 target function) irrelevant bits u(t − k), k > n + τ of the input sufﬁciently fast, i. [sent-258, score-0.214]

87 We use the limit d(∞) = limk→∞ d(k) to quantify this irrelevant separation which signiﬁes sensitivity to initial conditions (making the reservoir not time invariant). [sent-261, score-0.378]

88 4 the predictor p∞ is plotted as a function of the STD σ of the weight distribution and the in-degree K for three different values of the quantization level m ∈ {1, 3, 6}. [sent-271, score-0.535]

89 When comparing these results with the actual network performance pexp (PAR) on the PAR-task plotted in Fig. [sent-272, score-0.447]

90 1 one can see that p∞ serves as a reliable predictor for pexp of a network for sufﬁciently small m. [sent-273, score-0.511]

91 The dominant effect of the quantization level m on the performance discussed in Sec. [sent-275, score-0.357]

92 The interplay between the two contributions d(2) and d(∞) of p∞ delivers insight into the dependence of pexp on the network parameters. [sent-280, score-0.351]

93 A high value of d(2) corresponds to a good separation of inputs on short time scales relevant for the target task, a property that is found predominantly in networks that are not strongly input driven. [sent-281, score-0.385]

94 A small value of d(∞) guarantees that inputs on which the target function assumes the same value are mapped to nearby network states and thus a linear readout is able to assign them to the same class irrespectively of their irrelevant remote history. [sent-282, score-0.413]

95 The importance of a gradual order-chaos phase transition could explain why ESNs are more often used for applications than LSMs. [sent-292, score-0.254]

96 It should be noted that the effect of quantization cannot just be emulated by additive or multiplicative iid. [sent-295, score-0.318]

97 The noise degrades performance homogeneously and the differences in the inﬂuence of the in-degree observed for varying quantization levels cannot be reproduced. [sent-297, score-0.401]

98 The ﬁnding that binary reservoirs have superior performance for low in-degree stands in stark contrast to the fact that cortical neurons have very high in-degrees of over 104 . [sent-298, score-0.486]

99 This raises the interesting question which properties and mechanisms of cortical circuits not accounted for in this article contribute to their computational power. [sent-299, score-0.236]

100 In view of the results presented in this article, such mechanisms should tend to soften the phase transition between order and chaos. [sent-300, score-0.218]

similar papers computed by tfidf model

tfidf for this paper:

wordName wordTfidf (topN-words)

[('esns', 0.388), ('quantization', 0.318), ('reservoir', 0.307), ('reservoirs', 0.204), ('quantized', 0.196), ('pexp', 0.184), ('network', 0.167), ('parn', 0.163), ('predictor', 0.16), ('std', 0.153), ('rank', 0.133), ('esn', 0.125), ('lyapunov', 0.125), ('transition', 0.124), ('lsms', 0.123), ('bits', 0.117), ('networks', 0.109), ('resolution', 0.107), ('readout', 0.102), ('circuits', 0.102), ('analog', 0.101), ('circuit', 0.098), ('phase', 0.094), ('rc', 0.089), ('neurons', 0.083), ('schrauwen', 0.082), ('binary', 0.072), ('chaotic', 0.071), ('jaeger', 0.071), ('recurrent', 0.071), ('separation', 0.071), ('sm', 0.069), ('critical', 0.065), ('article', 0.062), ('echo', 0.061), ('legenstein', 0.061), ('lsm', 0.061), ('verstraeten', 0.061), ('capabilities', 0.061), ('spiking', 0.061), ('input', 0.061), ('drew', 0.058), ('plotted', 0.057), ('stream', 0.056), ('states', 0.056), ('high', 0.056), ('units', 0.055), ('bifurcation', 0.054), ('eld', 0.053), ('inputs', 0.052), ('region', 0.052), ('exponent', 0.052), ('ft', 0.049), ('streams', 0.049), ('regime', 0.049), ('numerically', 0.048), ('connectivity', 0.047), ('reproduced', 0.046), ('memory', 0.044), ('levels', 0.044), ('plots', 0.044), ('generalization', 0.044), ('memoryless', 0.043), ('maass', 0.043), ('power', 0.043), ('qualitatively', 0.041), ('ghent', 0.041), ('interconnection', 0.041), ('microcircuit', 0.041), ('topology', 0.04), ('computational', 0.04), ('state', 0.039), ('ui', 0.039), ('performance', 0.039), ('line', 0.039), ('delay', 0.038), ('computations', 0.037), ('aa', 0.037), ('target', 0.036), ('dichotomy', 0.036), ('liquid', 0.036), ('gradual', 0.036), ('fading', 0.036), ('natschl', 0.036), ('dynamical', 0.035), ('uence', 0.035), ('log', 0.035), ('measures', 0.034), ('ordered', 0.034), ('dashed', 0.034), ('cl', 0.034), ('termed', 0.033), ('disappears', 0.033), ('cortical', 0.032), ('annealed', 0.031), ('interfere', 0.031), ('chaos', 0.031), ('zones', 0.031), ('graz', 0.031), ('introduced', 0.029)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.99999934 160 nips-2008-On Computational Power and the Order-Chaos Phase Transition in Reservoir Computing

Author: Benjamin Schrauwen, Lars Buesing, Robert A. Legenstein

2 0.1242158 218 nips-2008-Spectral Clustering with Perturbed Data

Author: Ling Huang, Donghui Yan, Nina Taft, Michael I. Jordan

Abstract: Spectral clustering is useful for a wide-ranging set of applications in areas such as biological data analysis, image processing and data mining. However, the computational and/or communication resources required by the method in processing large-scale data are often prohibitively high, and practitioners are often required to perturb the original data in various ways (quantization, downsampling, etc) before invoking a spectral algorithm. In this paper, we use stochastic perturbation theory to study the effects of data perturbation on the performance of spectral clustering. We show that the error under perturbation of spectral clustering is closely related to the perturbation of the eigenvectors of the Laplacian matrix. From this result we derive approximate upper bounds on the clustering error. We show that this bound is tight empirically across a wide range of problems, suggesting that it can be used in practical settings to determine the amount of data reduction allowed in order to meet a speciﬁcation of permitted loss in clustering performance. 1

3 0.087052092 152 nips-2008-Non-stationary dynamic Bayesian networks

Author: Joshua W. Robinson, Alexander J. Hartemink

Abstract: A principled mechanism for identifying conditional dependencies in time-series data is provided through structure learning of dynamic Bayesian networks (DBNs). An important assumption of DBN structure learning is that the data are generated by a stationary process—an assumption that is not true in many important settings. In this paper, we introduce a new class of graphical models called non-stationary dynamic Bayesian networks, in which the conditional dependence structure of the underlying data-generation process is permitted to change over time. Non-stationary dynamic Bayesian networks represent a new framework for studying problems in which the structure of a network is evolving over time. We deﬁne the non-stationary DBN model, present an MCMC sampling algorithm for learning the structure of the model from time-series data under different assumptions, and demonstrate the effectiveness of the algorithm on both simulated and biological data. 1

4 0.082889885 209 nips-2008-Short-Term Depression in VLSI Stochastic Synapse

Author: Peng Xu, Timothy K. Horiuchi, Pamela A. Abshire

Abstract: We report a compact realization of short-term depression (STD) in a VLSI stochastic synapse. The behavior of the circuit is based on a subtractive single release model of STD. Experimental results agree well with simulation and exhibit expected STD behavior: the transmitted spike train has negative autocorrelation and lower power spectral density at low frequencies which can remove redundancy in the input spike train, and the mean transmission probability is inversely proportional to the input spike rate which has been suggested as an automatic gain control mechanism in neural systems. The dynamic stochastic synapse could potentially be a powerful addition to existing deterministic VLSI spiking neural systems. 1

5 0.082834899 204 nips-2008-Self-organization using synaptic plasticity

Author: Vicençc Gómez, Andreas Kaltenbrunner, Vicente López, Hilbert J. Kappen

Abstract: Large networks of spiking neurons show abrupt changes in their collective dynamics resembling phase transitions studied in statistical physics. An example of this phenomenon is the transition from irregular, noise-driven dynamics to regular, self-sustained behavior observed in networks of integrate-and-ﬁre neurons as the interaction strength between the neurons increases. In this work we show how a network of spiking neurons is able to self-organize towards a critical state for which the range of possible inter-spike-intervals (dynamic range) is maximized. Self-organization occurs via synaptic dynamics that we analytically derive. The resulting plasticity rule is deﬁned locally so that global homeostasis near the critical state is achieved by local regulation of individual synapses. 1

6 0.081025593 58 nips-2008-Dependence of Orientation Tuning on Recurrent Excitation and Inhibition in a Network Model of V1

7 0.07576783 158 nips-2008-Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks

8 0.068243325 43 nips-2008-Cell Assemblies in Large Sparse Inhibitory Networks of Biologically Realistic Spiking Neurons

9 0.06692151 240 nips-2008-Tracking Changing Stimuli in Continuous Attractor Neural Networks

10 0.065568052 230 nips-2008-Temporal Difference Based Actor Critic Learning - Convergence and Neural Implementation

11 0.064167902 219 nips-2008-Spectral Hashing

12 0.059639778 118 nips-2008-Learning Transformational Invariants from Natural Movies

13 0.056425005 166 nips-2008-On the asymptotic equivalence between differential Hebbian and temporal difference learning using a local third factor

14 0.056192469 148 nips-2008-Natural Image Denoising with Convolutional Networks

15 0.055749252 72 nips-2008-Empirical performance maximization for linear rank statistics

16 0.051997613 194 nips-2008-Regularized Learning with Networks of Features

17 0.050739724 121 nips-2008-Learning to Use Working Memory in Partially Observable Environments through Dopaminergic Reinforcement

18 0.05021058 21 nips-2008-An Homotopy Algorithm for the Lasso with Online Observations

19 0.049328335 3 nips-2008-A Massively Parallel Digital Learning Processor

20 0.049167462 231 nips-2008-Temporal Dynamics of Cognitive Control

similar papers computed by lsi model

lsi for this paper:

topicId topicWeight

[(0, -0.159), (1, 0.052), (2, 0.083), (3, 0.075), (4, -0.001), (5, -0.017), (6, 0.014), (7, -0.025), (8, 0.036), (9, 0.005), (10, -0.016), (11, 0.098), (12, -0.088), (13, 0.055), (14, -0.02), (15, -0.082), (16, 0.035), (17, -0.017), (18, 0.035), (19, -0.167), (20, -0.087), (21, 0.026), (22, -0.017), (23, 0.072), (24, 0.065), (25, 0.063), (26, -0.043), (27, -0.027), (28, 0.06), (29, -0.014), (30, 0.034), (31, -0.011), (32, -0.043), (33, 0.04), (34, 0.0), (35, 0.021), (36, 0.096), (37, -0.048), (38, 0.138), (39, -0.094), (40, -0.045), (41, 0.103), (42, 0.049), (43, 0.087), (44, 0.117), (45, -0.08), (46, 0.026), (47, -0.09), (48, -0.059), (49, -0.01)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.94570011 160 nips-2008-On Computational Power and the Order-Chaos Phase Transition in Reservoir Computing

Author: Benjamin Schrauwen, Lars Buesing, Robert A. Legenstein

2 0.67693669 240 nips-2008-Tracking Changing Stimuli in Continuous Attractor Neural Networks

Author: K. Wong, Si Wu, Chi Fung

Abstract: Continuous attractor neural networks (CANNs) are emerging as promising models for describing the encoding of continuous stimuli in neural systems. Due to the translational invariance of their neuronal interactions, CANNs can hold a continuous family of neutrally stable states. In this study, we systematically explore how neutral stability of a CANN facilitates its tracking performance, a capacity believed to have wide applications in brain functions. We develop a perturbative approach that utilizes the dominant movement of the network stationary states in the state space. We quantify the distortions of the bump shape during tracking, and study their effects on the tracking performance. Results are obtained on the maximum speed for a moving stimulus to be trackable, and the reaction time to catch up an abrupt change in stimulus. 1

3 0.66291404 204 nips-2008-Self-organization using synaptic plasticity

Author: Vicençc Gómez, Andreas Kaltenbrunner, Vicente López, Hilbert J. Kappen

4 0.65948528 158 nips-2008-Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks

Author: Alex Graves, Jürgen Schmidhuber

Abstract: Ofﬂine handwriting recognition—the automatic transcription of images of handwritten text—is a challenging task that combines computer vision with sequence learning. In most systems the two elements are handled separately, with sophisticated preprocessing techniques used to extract the image features and sequential models such as HMMs used to provide the transcriptions. By combining two recent innovations in neural networks—multidimensional recurrent neural networks and connectionist temporal classiﬁcation—this paper introduces a globally trained ofﬂine handwriting recogniser that takes raw pixel data as input. Unlike competing systems, it does not require any alphabet speciﬁc preprocessing, and can therefore be used unchanged for any language. Evidence of its generality and power is provided by data from a recent international Arabic recognition competition, where it outperformed all entries (91.4% accuracy compared to 87.2% for the competition winner) despite the fact that neither author understands a word of Arabic. 1

5 0.58099103 152 nips-2008-Non-stationary dynamic Bayesian networks

Author: Joshua W. Robinson, Alexander J. Hartemink

6 0.5735392 58 nips-2008-Dependence of Orientation Tuning on Recurrent Excitation and Inhibition in a Network Model of V1

7 0.57122481 27 nips-2008-Artificial Olfactory Brain for Mixture Identification

8 0.56043941 3 nips-2008-A Massively Parallel Digital Learning Processor

9 0.47829229 148 nips-2008-Natural Image Denoising with Convolutional Networks

10 0.46961451 218 nips-2008-Spectral Clustering with Perturbed Data

11 0.44872332 230 nips-2008-Temporal Difference Based Actor Critic Learning - Convergence and Neural Implementation

12 0.42642531 43 nips-2008-Cell Assemblies in Large Sparse Inhibitory Networks of Biologically Realistic Spiking Neurons

13 0.42493048 209 nips-2008-Short-Term Depression in VLSI Stochastic Synapse

14 0.41637626 219 nips-2008-Spectral Hashing

15 0.37649021 96 nips-2008-Hebbian Learning of Bayes Optimal Decisions

16 0.36726201 222 nips-2008-Stress, noradrenaline, and realistic prediction of mouse behaviour using reinforcement learning

17 0.36391243 166 nips-2008-On the asymptotic equivalence between differential Hebbian and temporal difference learning using a local third factor

18 0.35525653 121 nips-2008-Learning to Use Working Memory in Partially Observable Environments through Dopaminergic Reinforcement

19 0.34665218 169 nips-2008-Online Models for Content Optimization

20 0.34293905 34 nips-2008-Bayesian Network Score Approximation using a Metagraph Kernel

similar papers computed by lda model

lda for this paper:

topicId topicWeight

[(4, 0.019), (6, 0.055), (7, 0.061), (12, 0.024), (28, 0.172), (47, 0.342), (57, 0.08), (59, 0.012), (63, 0.014), (71, 0.022), (77, 0.058), (78, 0.022), (83, 0.037)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.74193192 160 nips-2008-On Computational Power and the Order-Chaos Phase Transition in Reservoir Computing

Author: Benjamin Schrauwen, Lars Buesing, Robert A. Legenstein

2 0.72519553 146 nips-2008-Multi-task Gaussian Process Learning of Robot Inverse Dynamics

Author: Christopher Williams, Stefan Klanke, Sethu Vijayakumar, Kian M. Chai

Abstract: The inverse dynamics problem for a robotic manipulator is to compute the torques needed at the joints to drive it along a given trajectory; it is beneﬁcial to be able to learn this function for adaptive control. A robotic manipulator will often need to be controlled while holding different loads in its end effector, giving rise to a multi-task learning problem. By placing independent Gaussian process priors over the latent functions of the inverse dynamics, we obtain a multi-task Gaussian process prior for handling multiple loads, where the inter-task similarity depends on the underlying inertial parameters. Experiments demonstrate that this multi-task formulation is effective in sharing information among the various loads, and generally improves performance over either learning only on single tasks or pooling the data over all tasks. 1

3 0.61415505 159 nips-2008-On Bootstrapping the ROC Curve

Author: Patrice Bertail, Stéphan J. Clémençcon, Nicolas Vayatis

Abstract: This paper is devoted to thoroughly investigating how to bootstrap the ROC curve, a widely used visual tool for evaluating the accuracy of test/scoring statistics in the bipartite setup. The issue of conﬁdence bands for the ROC curve is considered and a resampling procedure based on a smooth version of the empirical distribution called the ”smoothed bootstrap” is introduced. Theoretical arguments and simulation results are presented to show that the ”smoothed bootstrap” is preferable to a ”naive” bootstrap in order to construct accurate conﬁdence bands. 1

4 0.53090096 118 nips-2008-Learning Transformational Invariants from Natural Movies

Author: Charles Cadieu, Bruno A. Olshausen

Abstract: We describe a hierarchical, probabilistic model that learns to extract complex motion from movies of the natural environment. The model consists of two hidden layers: the ﬁrst layer produces a sparse representation of the image that is expressed in terms of local amplitude and phase variables. The second layer learns the higher-order structure among the time-varying phase variables. After training on natural movies, the top layer units discover the structure of phase-shifts within the ﬁrst layer. We show that the top layer units encode transformational invariants: they are selective for the speed and direction of a moving pattern, but are invariant to its spatial structure (orientation/spatial-frequency). The diversity of units in both the intermediate and top layers of the model provides a set of testable predictions for representations that might be found in V1 and MT. In addition, the model demonstrates how feedback from higher levels can inﬂuence representations at lower levels as a by-product of inference in a graphical model. 1

5 0.52940875 200 nips-2008-Robust Kernel Principal Component Analysis

Author: Minh H. Nguyen, Fernando Torre

Abstract: Kernel Principal Component Analysis (KPCA) is a popular generalization of linear PCA that allows non-linear feature extraction. In KPCA, data in the input space is mapped to higher (usually) dimensional feature space where the data can be linearly modeled. The feature space is typically induced implicitly by a kernel function, and linear PCA in the feature space is performed via the kernel trick. However, due to the implicitness of the feature space, some extensions of PCA such as robust PCA cannot be directly generalized to KPCA. This paper presents a technique to overcome this problem, and extends it to a uniﬁed framework for treating noise, missing data, and outliers in KPCA. Our method is based on a novel cost function to perform inference in KPCA. Extensive experiments, in both synthetic and real data, show that our algorithm outperforms existing methods. 1

6 0.52892685 231 nips-2008-Temporal Dynamics of Cognitive Control

7 0.5287233 216 nips-2008-Sparse probabilistic projections

8 0.52864641 138 nips-2008-Modeling human function learning with Gaussian processes

9 0.52522987 176 nips-2008-Partially Observed Maximum Entropy Discrimination Markov Networks

10 0.52519846 197 nips-2008-Relative Performance Guarantees for Approximate Inference in Latent Dirichlet Allocation

11 0.5250631 79 nips-2008-Exploring Large Feature Spaces with Hierarchical Multiple Kernel Learning

12 0.52499151 235 nips-2008-The Infinite Hierarchical Factor Regression Model

13 0.52461118 135 nips-2008-Model Selection in Gaussian Graphical Models: High-Dimensional Consistency of \boldmath$\ell 1$-regularized MLE

14 0.5242523 62 nips-2008-Differentiable Sparse Coding

15 0.52390093 31 nips-2008-Bayesian Exponential Family PCA

16 0.52380055 195 nips-2008-Regularized Policy Iteration

17 0.5236308 4 nips-2008-A Scalable Hierarchical Distributed Language Model

18 0.52346236 49 nips-2008-Clusters and Coarse Partitions in LP Relaxations

19 0.52216476 129 nips-2008-MAS: a multiplicative approximation scheme for probabilistic inference

20 0.5218488 50 nips-2008-Continuously-adaptive discretization for message-passing algorithms