nips nips2009 nips2009-254 knowledge-graph by maker-knowledge-mining

254 nips-2009-Variational Gaussian-process factor analysis for modeling spatio-temporal data

Source: pdf

Author: Jaakko Luttinen, Alexander T. Ihler

Abstract: We present a probabilistic factor analysis model which can be used for studying spatio-temporal datasets. The spatial and temporal structure is modeled by using Gaussian process priors both for the loading matrix and the factors. The posterior distributions are approximated using the variational Bayesian framework. High computational cost of Gaussian process modeling is reduced by using sparse approximations. The model is used to compute the reconstructions of the global sea surface temperatures from a historical dataset. The results suggest that the proposed model can outperform the state-of-the-art reconstruction systems.

Reference: text

Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 The spatial and temporal structure is modeled by using Gaussian process priors both for the loading matrix and the factors. [sent-6, score-0.476]

2 The posterior distributions are approximated using the variational Bayesian framework. [sent-7, score-0.267]

3 The model is used to compute the reconstructions of the global sea surface temperatures from a historical dataset. [sent-9, score-0.399]

4 1 Introduction Factor analysis and principal component analysis (PCA) are widely used linear techniques for ﬁnding dominant patterns in multivariate datasets. [sent-11, score-0.149]

5 The found principal patterns can also give an insight into the observed data variability. [sent-13, score-0.149]

6 For example, taking into account the temporal information typically leads to more accurate modeling of time series. [sent-15, score-0.164]

7 In this work, we present a factor analysis model which makes use of both temporal and spatial information for a set of collected data. [sent-16, score-0.34]

8 The method is based on the standard factor analysis model D w:d xT + noise , d: Y = WX + noise = (1) d=1 where Y is a matrix of spatio-temporal data in which each row contains measurements in one spatial location and each column corresponds to one time instance. [sent-17, score-0.352]

9 Thus, each xd: represents the time series of one of the D factors whereas w:d is a vector of loadings which are spatially distributed. [sent-19, score-0.186]

10 1 We assume that both the factors xd: and the corresponding loadings w:d have prominent structures. [sent-21, score-0.175]

11 The advantage of the proposed approach is that we perform GP modeling only either in the spatial or temporal domain at a time. [sent-29, score-0.327]

12 Also, good interpretability of the model makes it easy to explore the results in the spatial and temporal domain and to set priors reﬂecting our modeling assumptions. [sent-31, score-0.405]

13 Therefore we use an approximation based on the variational Bayesian methodology. [sent-41, score-0.15]

14 We also show how to use sparse variational approximations to reduce the computational load. [sent-42, score-0.277]

15 The nonnegative matrix factorization model in [11] uses point-estimates for the unknown parameters, thus ignoring posterior uncertainties. [sent-45, score-0.205]

16 In our method, we take into account posterior uncertainties, which helps reduce overﬁtting and facilitates learning a more accurate model. [sent-46, score-0.147]

17 In the experimental part, we use the model to compute reconstruction of missing values in a realworld spatio-temporal dataset. [sent-47, score-0.176]

18 We use a historical sea surface temperature dataset which contains monthly anomalies in the 1856-1991 period to reconstruct the global sea surface temperatures. [sent-48, score-0.771]

19 The same dataset was used in designing the state-of-the-art reconstruction methodology [5]. [sent-49, score-0.139]

20 Since reconstruction of missing values can be an important application for the method, we give all the formulas assuming missing values in the data matrix Y. [sent-51, score-0.302]

21 2 Factor analysis model with Gaussian process priors We use the factor analysis model (1) in which Y has dimensionality M × N and the number of factors D is much smaller than the number of spatial locations M and the number of time instances N . [sent-52, score-0.438]

22 The m-th row of Y corresponds to a spatial location lm (e. [sent-53, score-0.192]

23 The ij-th element of Kd is computed using the covariance function ψd with the kernel hyperparameters θd . [sent-58, score-0.185]

24 The priors for W are deﬁned similarly assuming that each spatial pattern w:d contains measurements of a function ω(l) at different spatial locations lm : D N (w:d |0, Kw ) , d p(W) = [Kw ]ij = ϕd (li , lj ; φd ) , d (3) d=1 where ϕd is a covariance function with hyperparameters φd . [sent-59, score-0.703]

25 The noise term in (1) is modeled with a Gaussian distribution, resulting in a likelihood function T 2 N ymn wm: x:n , σmn , p(Y|W, X, σ) = mn∈O 2 (4) where the product is evaluated over the observed elements in Y whose indices are included in the set O. [sent-63, score-0.213]

26 One can also use spatially and temporally varying 2 noise level σmn if this variability can be estimated somehow. [sent-66, score-0.153]

27 There are two main difﬁculties which should be addressed when learning the model: 1) The posterior p(W, X|Y) is intractable and 2) the computational load for dealing with GPs can be too large for real-world datasets. [sent-67, score-0.169]

28 We use the variational Bayesian framework to cope with the ﬁrst difﬁculty and we also adopt the variational approach when computing sparse approximations for the GP posterior. [sent-68, score-0.397]

29 3 Learning algorithm In the variational Bayesian framework, the true posterior is approximated using some restricted class of possible distributions. [sent-69, score-0.267]

30 Omitting the derivations here, this boils down to the following update rule: q(X) = N X: K−1 + U x −1 Z: , K−1 + U x −1 , (6) where Z: is a DN × 1 vector formed by concatenation of vectors −2 σmn wm: ymn . [sent-78, score-0.154]

31 z:n = (7) m∈On The summation in (7) is over a set On of indices m for which ymn is observed. [sent-79, score-0.122]

32 (8) m∈On Note that the form of the approximate posterior (6) is similar to the regular GP regression: One can interpret U−1 z:n as noisy observations with the corresponding noise covariance matrices U−1 . [sent-84, score-0.318]

33 n n Then, q(X) in (6) is simply the posterior distribution of the latent functions values χd (tn ). [sent-85, score-0.154]

34 The variational EM algorithm for learning the model consists of alternate updates of q(W) and q(X) until convergence. [sent-87, score-0.15]

35 The noise level can be estimated by using a point estimate or adding a factor factor q(σmn ) to the approximate posterior distribution. [sent-88, score-0.272]

36 1 Component-wise factorization In practice, one may need to factorize further the posterior approximation in order to reduce the computational burden. [sent-91, score-0.206]

37 This can be done in two ways: by neglecting the posterior correlations between different factors xd: (and between spatial patterns w:d , respectively) or by neglecting the posterior correlations between different time instances x:n (and between spatial locations wm: , respectively). [sent-92, score-0.922]

38 This yields a posterior approximation q(X) = which can be updated as follows: q(xd: ) = N xd: K−1 + Vd d −1 cd , K−1 + Vd d −1 , D d=1 q(xd: ) d = 1, . [sent-94, score-0.157]

39 , D , (9) where cd is an N × 1 vector whose n-th component is −2 σmn wmd [cd ]n = ymn − m∈On wmj xjn (10) j=d −2 2 and Vd is an N ×N diagonal matrix whose n-th diagonal element is [Vd ]nn = m∈On σmn wmd . [sent-97, score-0.289]

40 The component-wise factorization may provide a meaningful representation of data because the model is biased in favor of solutions with dynamically and spatially decoupled components. [sent-100, score-0.12]

41 When the factors are modeled using rather general covariance functions, the proposed method is somewhat related to the blind source separation techniques using time structure (e. [sent-101, score-0.206]

42 The advantage here is that the method can handle more sophisticated temporal correlations and it is easily applicable to incomplete data. [sent-104, score-0.166]

43 In addition, one can use the method in semi-blind settings when prior knowledge is used to extract components with speciﬁc types of temporal or spatial features [9]. [sent-105, score-0.37]

44 Although the variational learning of the GPFA model works only in either spatial or temporal domain at a time, the size of the data may still be too large in practice. [sent-109, score-0.44]

45 A common way to reduce the computational cost is to use sparse approximations [7]. [sent-110, score-0.127]

46 In this work, we follow the variational formulation of sparse approximations presented in [15]. [sent-111, score-0.247]

47 The main idea is to introduce a set of auxiliary variables {w, x} which contain the values of the d latent functions ωd (l), χd (t) in some locations {l = λd |m = 1, . [sent-112, score-0.135]

48 Optimal q(w), q(x) can be computed by maximizing the variational lower bound of the marginal log-likelihood similar to (5). [sent-120, score-0.15]

49 q(x) yields the following update rule: q(x) = N x ΣK−1 Kxx Z: , Σ , x Σ = K−1 + K−1 Kxx UKxx K−1 x x x −1 , (12) where x is the vector of concatenated auxiliary variables for all factors, Kx is the GP prior covariance matrix of x and Kxx is the covariance between x and X: . [sent-124, score-0.324]

50 The advantage here is that the number of inducing inputs is smaller than then the number of data samples, that is, Md < M and Nd < N , and therefore the required computational load can be reduced (see more details in [15]). [sent-127, score-0.241]

51 (12) can be quite easily adapted to the component-wise factorization of the posterior in order to reduce the computational load of (9). [sent-129, score-0.258]

52 3 Update of GP hyperparameters The hyperparameters of the GP priors can be updated quite similarly to the standard GP regression by maximizing the lower bound of the marginal log-likelihood. [sent-132, score-0.23]

53 A symmetrical equation can be derived for the hyperparameters of the spatial functions ϕd (t). [sent-136, score-0.303]

54 The inducing inputs can also be treated as variational parameters and they can be changed to optimize the lower bound (13). [sent-138, score-0.339]

55 1 Artiﬁcial example We generated a dataset with M = 30 sensors (two-dimensional spatial locations) and N = 200 time instances using the generative model (1) with a moderate amount of observation noise, assuming σmn = σ. [sent-140, score-0.277]

56 The loadings were generated from GPs over the two-dimensional space using the squared exponential covariance function (14) with an additional scale parameter θ2 : 2 2 k(r; θ1 , θ2 ) = θ2 exp −r2 /(2θ1 ) . [sent-142, score-0.209]

57 The hyperparameters of the Gaussian processes were initialized randomly close to the values used for data generation, assuming that a good guess about the hidden signals can be obtained by exploratory analysis of data. [sent-148, score-0.192]

58 The posterior predictive distributions of the missing values presented in Fig. [sent-152, score-0.183]

59 1a show that the method was able to capture temporal correlations on different timescales. [sent-153, score-0.166]

60 Note also that although some of the sensors contain very few observations, the missing values are reconstructed pretty well. [sent-154, score-0.118]

61 (a) Posterior predictive distribution for four randomly selected locations with the observations shown as crosses, the gap with no training observations marked with vertical lines and some test values shown as circles. [sent-157, score-0.16]

62 2 Reconstruction of global SST using the MOHSST5 dataset We demonstrate how the presented model can be used to reconstruct global sea surface temperatures (SST) from historical measurements. [sent-161, score-0.428]

63 Meteorological Ofﬁce historical SST data set (MOHSST5) [6] that contain monthly SST anomalies in the 1856-1991 period for 5◦ ×5◦ longitudelatitude bins. [sent-164, score-0.2]

64 The dataset contains in total approximately 1600 time instances and 1700 spatial locations. [sent-165, score-0.225]

65 We used ﬁve time signals xd: with the squared exponential function (14) to describe climate trends. [sent-169, score-0.154]

66 Another ﬁve temporal components were modeled with the quasi-periodic covariance function (15) to capture periodic signals (e. [sent-170, score-0.469]

67 We also used ﬁve components with the squared exponential function to model prominent interannual phenomena such as El Ni˜ o. [sent-173, score-0.277]

68 The covariance function for each spatial pattern w:d was the scaled squared exponential (17). [sent-176, score-0.308]

69 The distance r between the locations li and lj was measured on the surface of the Earth using the spherical law of cosines. [sent-177, score-0.188]

70 We also introduced 500 inducing inputs for each spatial function ωd (l) in order to use sparse variational approximations. [sent-181, score-0.557]

71 Similar sparse approximations were used for the 15 temporal functions χ(t) which modeled slow climate variability: the slowest, quasi-periodic and interannual components had 80, 300 and 300 inducing inputs, respectively. [sent-182, score-0.613]

72 The inducing inputs were initialized by taking a random subset from the original inputs and then kept ﬁxed throughout learning because their optimization would have increased the computational burden substantially. [sent-183, score-0.304]

73 For the rest of the temporal phenomena, we used the piecewise polynomial functions (16) that produce priors with a sparse covariance matrix and therefore allow efﬁcient computations. [sent-184, score-0.432]

74 The dataset was preprocessed by weighting the data points by the square root of the corresponding latitudes in order to diminish the effect of denser sampling in the polar regions, then the same noise level was assumed for all measurements (σmn = σ). [sent-185, score-0.133]

75 Preprocessing by weighting data points ymn with weights sm is essentially equivalent to assuming spatially varying noise level σmn = σ/sm . [sent-186, score-0.238]

76 The GP hyperparameters were initialized taking into account the assumed smoothness of the spatial patterns and the variability timescale of the temporal factors. [sent-187, score-0.517]

77 The spatial and temporal patterns of the four most dominating principal components for GPFA (above) and VBPCA (below). [sent-205, score-0.519]

78 The uncertainties of the spatial patterns are not shown, and we saturated the visualizations of the VBPCA spatial components to reduce the effect of the uncertain pole regions. [sent-207, score-0.542]

79 The variational EM-algorithm of GPFA was run for 200 iterations. [sent-209, score-0.15]

80 We also applied the variational Bayesian PCA (VBPCA) [2] to the same dataset for comparison. [sent-210, score-0.179]

81 Finally, we rotated the GPFA components such that the orthogonal basis in the factor analysis subspace was ordered according to the amount of explained data variance (where the variance was computed by averaging over time). [sent-212, score-0.13]

82 Thus, “GPFA principal components” are mixtures of the original factors found by the algorithm. [sent-213, score-0.143]

83 2 shows the spatial and temporal patterns of the four most dominant principal components for both models. [sent-216, score-0.519]

84 The GPFA principal components and the corresponding spatial patterns are generally smoother, especially in the data-sparse regions, for example, in the period before 1875. [sent-217, score-0.423]

85 The ﬁrst and the second principal components of GPFA as well as the ﬁrst and the third components of VBPCA are related to El Ni˜ o. [sent-218, score-0.242]

86 We should make a note here that the rotation within the principal subspace n may be affected by noise and therefore the components may not be directly comparable. [sent-219, score-0.217]

87 Another observation was that the model efﬁciently used only some of the 15 slow components: about three very slow and two interannual components had relatively large weights in the loading matrix W. [sent-220, score-0.285]

88 None 7 of the periodic components had large weights, which suggests that the fourth VBPCA component might contain artifacts. [sent-222, score-0.128]

89 The model is based on using GP priors for both spatial patterns and time signals corresponding to the hidden factors. [sent-229, score-0.377]

90 The method can be seen as a combination of temporal smoothing, empirical orthogonal functions (EOF) analysis and kriging. [sent-230, score-0.127]

91 The proposed model was applied to the problem of reconstruction of historical global sea surface temperatures. [sent-235, score-0.454]

92 EOF) analysis with smoothness assumptions for the spatial and temporal patterns. [sent-238, score-0.29]

93 The improvement is possible because the method is able to model temporal and spatial phenomena on different scales by using properly selected GPs. [sent-243, score-0.328]

94 a hyperparameter (or inducing input) θ of any covariance function is given by 1 ∂Kxx 1 ∂Kx ∂Kxx ∂Kx − tr UKxx A−1 + − bT tr K−1 − A−1 b + bT (Z: − UKxx b) x 2 ∂θ ∂θ 2 ∂θ ∂θ where A = Kx + Kxx UKxx , b = A−1 Kxx Z: . [sent-247, score-0.318]

95 Without the sparse approximation, it holds that Kx = Kx = Kxx = Kxx and the equation simpliﬁes to the regular gradient in GP regression for projected observations U−1 Z: with the noise covariance U−1 . [sent-249, score-0.256]

96 The second part of (13) results in the extra terms tr ∂Kx U + tr ∂θ ∂Kx −1 Kx Kxx UKxx K−1 x ∂θ − 2 tr ∂Kxx −1 Kx Kxx U ∂θ . [sent-250, score-0.16]

97 We would like to thank Alexey Kaplan for fruitful discussions and providing his expertise on the problem of sea surface temperature reconstruction. [sent-255, score-0.293]

98 Bayesian PCA for reconstruction of historical sea surface temperatures. [sent-277, score-0.454]

99 Interdecadal changes of surface temperature since the late nineteenth century. [sent-296, score-0.147]

100 Variational learning of inducing variables in sparse Gaussian processes. [sent-359, score-0.176]

similar papers computed by tfidf model

tfidf for this paper:

wordName wordTfidf (topN-words)

[('gpfa', 0.39), ('kxx', 0.39), ('gp', 0.243), ('kx', 0.225), ('xd', 0.196), ('mn', 0.175), ('vbpca', 0.17), ('spatial', 0.163), ('variational', 0.15), ('sea', 0.146), ('temporal', 0.127), ('sst', 0.122), ('ukxx', 0.122), ('ymn', 0.122), ('inducing', 0.121), ('posterior', 0.117), ('reconstruction', 0.11), ('covariance', 0.109), ('surface', 0.103), ('historical', 0.095), ('wm', 0.083), ('principal', 0.082), ('components', 0.08), ('priors', 0.078), ('kd', 0.076), ('hyperparameters', 0.076), ('ilin', 0.073), ('interannual', 0.073), ('signals', 0.069), ('inputs', 0.068), ('patterns', 0.067), ('gps', 0.066), ('vd', 0.066), ('missing', 0.066), ('symmetrical', 0.064), ('loadings', 0.064), ('factors', 0.061), ('spatially', 0.061), ('factorization', 0.059), ('tj', 0.059), ('noise', 0.055), ('sparse', 0.055), ('temperatures', 0.055), ('locations', 0.053), ('finland', 0.052), ('sensors', 0.052), ('load', 0.052), ('prominent', 0.05), ('factor', 0.05), ('climate', 0.049), ('diminish', 0.049), ('eof', 0.049), ('geophysical', 0.049), ('wmd', 0.049), ('periodic', 0.048), ('initialized', 0.047), ('dn', 0.046), ('auxiliary', 0.045), ('tr', 0.044), ('temperature', 0.044), ('tn', 0.044), ('ti', 0.043), ('pca', 0.043), ('kaplan', 0.043), ('loading', 0.043), ('approximations', 0.042), ('cd', 0.04), ('correlations', 0.039), ('md', 0.039), ('anomalies', 0.039), ('uncertainties', 0.039), ('phenomena', 0.038), ('observations', 0.037), ('modeling', 0.037), ('variability', 0.037), ('aistats', 0.037), ('kw', 0.037), ('latent', 0.037), ('squared', 0.036), ('modeled', 0.036), ('gaussian', 0.035), ('monthly', 0.035), ('neglecting', 0.035), ('seeger', 0.035), ('piecewise', 0.034), ('marked', 0.033), ('helsinki', 0.033), ('instances', 0.033), ('omitting', 0.032), ('lj', 0.032), ('update', 0.032), ('period', 0.031), ('formulas', 0.031), ('informatics', 0.031), ('reduce', 0.03), ('slow', 0.03), ('lm', 0.029), ('dataset', 0.029), ('matrix', 0.029), ('extra', 0.028)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.9999994 254 nips-2009-Variational Gaussian-process factor analysis for modeling spatio-temporal data

Author: Jaakko Luttinen, Alexander T. Ihler

2 0.1994119 117 nips-2009-Inter-domain Gaussian Processes for Sparse Inference using Inducing Features

Author: Anibal Figueiras-vidal, Miguel Lázaro-gredilla

Abstract: We present a general inference framework for inter-domain Gaussian Processes (GPs) and focus on its usefulness to build sparse GP models. The state-of-the-art sparse GP model introduced by Snelson and Ghahramani in [1] relies on ﬁnding a small, representative pseudo data set of m elements (from the same domain as the n available data elements) which is able to explain existing data well, and then uses it to perform inference. This reduces inference and model selection computation time from O(n3 ) to O(m2 n), where m n. Inter-domain GPs can be used to ﬁnd a (possibly more compact) representative set of features lying in a different domain, at the same computational cost. Being able to specify a different domain for the representative features allows to incorporate prior knowledge about relevant characteristics of data and detaches the functional form of the covariance and basis functions. We will show how previously existing models ﬁt into this framework and will use it to develop two new sparse GP models. Tests on large, representative regression data sets suggest that signiﬁcant improvement can be achieved, while retaining computational efﬁciency. 1 Introduction and previous work Along the past decade there has been a growing interest in the application of Gaussian Processes (GPs) to machine learning tasks. GPs are probabilistic non-parametric Bayesian models that combine a number of attractive characteristics: They achieve state-of-the-art performance on supervised learning tasks, provide probabilistic predictions, have a simple and well-founded model selection scheme, present no overﬁtting (since parameters are integrated out), etc. Unfortunately, the direct application of GPs to regression problems (with which we will be concerned here) is limited due to their training time being O(n3 ). To overcome this limitation, several sparse approximations have been proposed [2, 3, 4, 5, 6]. In most of them, sparsity is achieved by projecting all available data onto a smaller subset of size m n (the active set), which is selected according to some speciﬁc criterion. This reduces computation time to O(m2 n). However, active set selection interferes with hyperparameter learning, due to its non-smooth nature (see [1, 3]). These proposals have been superseded by the Sparse Pseudo-inputs GP (SPGP) model, introduced in [1]. In this model, the constraint that the samples of the active set (which are called pseudoinputs) must be selected among training data is relaxed, allowing them to lie anywhere in the input space. This allows both pseudo-inputs and hyperparameters to be selected in a joint continuous optimisation and increases ﬂexibility, resulting in much superior performance. In this work we introduce Inter-Domain GPs (IDGPs) as a general tool to perform inference across domains. This allows to remove the constraint that the pseudo-inputs must remain within the same domain as input data. This added ﬂexibility results in an increased performance and allows to encode prior knowledge about other domains where data can be represented more compactly. 1 2 Review of GPs for regression We will brieﬂy state here the main deﬁnitions and results for regression with GPs. See [7] for a comprehensive review. Assume we are given a training set with n samples D ≡ {xj , yj }n , where each D-dimensional j=1 input xj is associated to a scalar output yj . The regression task goal is, given a new input x∗ , predict the corresponding output y∗ based on D. The GP regression model assumes that the outputs can be expressed as some noiseless latent function plus independent noise, y = f (x)+ε, and then sets a zero-mean1 GP prior on f (x), with covariance k(x, x ), and a zero-mean Gaussian prior on ε, with variance σ 2 (the noise power hyperparameter). The covariance function encodes prior knowledge about the smoothness of f (x). The most common choice for it is the Automatic Relevance Determination Squared Exponential (ARD SE): 2 k(x, x ) = σ0 exp − 1 2 D d=1 (xd − xd )2 2 d , (1) 2 with hyperparameters σ0 (the latent function power) and { d }D (the length-scales, deﬁning how d=1 rapidly the covariance decays along each dimension). It is referred to as ARD SE because, when coupled with a model selection method, non-informative input dimensions can be removed automatically by growing the corresponding length-scale. The set of hyperparameters that deﬁne the GP are 2 θ = {σ 2 , σ0 , { d }D }. We will omit the dependence on θ for the sake of clarity. d=1 If we evaluate the latent function at X = {xj }n , we obtain a set of latent variables following a j=1 joint Gaussian distribution p(f |X) = N (f |0, Kﬀ ), where [Kﬀ ]ij = k(xi , xj ). Using this model it is possible to express the joint distribution of training and test cases and then condition on the observed outputs to obtain the predictive distribution for any test case pGP (y∗ |x∗ , D) = N (y∗ |kf ∗ (Kﬀ + σ 2 In )−1 y, σ 2 + k∗∗ − kf ∗ (Kﬀ + σ 2 In )−1 kf ∗ ), (2) where y = [y1 , . . . , yn ] , kf ∗ = [k(x1 , x∗ ), . . . , k(xn , x∗ )] , and k∗∗ = k(x∗ , x∗ ). In is used to denote the identity matrix of size n. The O(n3 ) cost of these equations arises from the inversion of the n × n covariance matrix. Predictive distributions for additional test cases take O(n2 ) time each. These costs make standard GPs impractical for large data sets. To select hyperparameters θ, Type-II Maximum Likelihood (ML-II) is commonly used. This amounts to selecting the hyperparameters that correspond to a (possibly local) maximum of the log-marginal likelihood, also called log-evidence. 3 Inter-domain GPs In this section we will introduce Inter-Domain GPs (IDGPs) and show how they can be used as a framework for computationally efﬁcient inference. Then we will use this framework to express two previous relevant models and develop two new ones. 3.1 Deﬁnition Consider a real-valued GP f (x) with x ∈ RD and some deterministic real function g(x, z), with z ∈ RH . We deﬁne the following transformation: u(z) = f (x)g(x, z)dx. (3) RD There are many examples of transformations that take on this form, the Fourier transform being one of the best known. We will discuss possible choices for g(x, z) in Section 3.3; for the moment we will deal with the general form. Since u(z) is obtained by a linear transformation of GP f (x), 1 We follow the common approach of subtracting the sample mean from the outputs and then assume a zero-mean model. 2 it is also a GP. This new GP may lie in a different domain of possibly different dimension. This transformation is not invertible in general, its properties being deﬁned by g(x, z). IDGPs arise when we jointly consider f (x) and u(z) as a single, “extended” GP. The mean and covariance function of this extended GP are overloaded to accept arguments from both the input and transformed domains and treat them accordingly. We refer to each version of an overloaded function as an instance, which will accept a different type of arguments. If the distribution of the original GP is f (x) ∼ GP(m(x), k(x, x )), then it is possible to compute the remaining instances that deﬁne the distribution of the extended GP over both domains. The transformed-domain instance of the mean is m(z) = E[u(z)] = E[f (x)]g(x, z)dx = m(x)g(x, z)dx. RD RD The inter-domain and transformed-domain instances of the covariance function are: k(x, z ) = E[f (x)u(z )] = E f (x) f (x )g(x , z )dx = RD k(z, z ) = E[u(z)u(z )] = E f (x)g(x, z)dx RD = k(x, x )g(x , z )dx f (x )g(x , z )dx RD k(x, x )g(x, z)g(x , z )dxdx . RD (4) RD (5) RD Mean m(·) and covariance function k(·, ·) are therefore deﬁned both by the values and domains of their arguments. This can be seen as if each argument had an additional domain indicator used to select the instance. Apart from that, they deﬁne a regular GP, and all standard properties hold. In particular k(a, b) = k(b, a). This approach is related to [8], but here the latent space is deﬁned as a transformation of the input space, and not the other way around. This allows to pre-specify the desired input-domain covariance. The transformation is also more general: Any g(x, z) can be used. We can sample an IDGP at n input-domain points f = [f1 , f2 , . . . , fn ] (with fj = f (xj )) and m transformed-domain points u = [u1 , u2 , . . . , um ] (with ui = u(zi )). With the usual assumption of f (x) being a zero mean GP and deﬁning Z = {zi }m , the joint distribution of these samples is: i=1 Kﬀ Kfu f f 0, X, Z = N p , (6) u u Kfu Kuu with [Kﬀ ]pq = k(xp , xq ), [Kfu ]pq = k(xp , zq ), [Kuu ]pq = k(zp , zq ), which allows to perform inference across domains. We will only be concerned with one input domain and one transformed domain, but IDGPs can be deﬁned for any number of domains. 3.2 Sparse regression using inducing features In the standard regression setting, we are asked to perform inference about the latent function f (x) from a data set D lying in the input domain. Using IDGPs, we can use data from any domain to perform inference in the input domain. Some latent functions might be better deﬁned by a set of data lying in some transformed space rather than in the input space. This idea is used for sparse inference. Following [1] we introduce a pseudo data set, but here we place it in the transformed domain: D = {Z, u}. The following derivation is analogous to that of SPGP. We will refer to Z as the inducing features and u as the inducing variables. The key approximation leading to sparsity is to set m n and assume that f (x) is well-described by the pseudo data set D, so that any two samples (either from the training or test set) fp and fq with p = q will be independent given xp , xq and D. With this simplifying assumption2 , the prior over f can be factorised as a product of marginals: n p(f |X, Z, u) ≈ p(fj |xj , Z, u). (7) j=1 2 Alternatively, (7) can be obtained by proposing a generic factorised form for the approximate conn ditional p(f |X, Z, u) ≈ q(f |X, Z, u) = q (f |xj , Z, u) and then choosing the set of funcj=1 j j tions {qj (·)}n so as to minimise the Kullback-Leibler (KL) divergence from the exact joint prior j=1 KL(p(f |X, Z, u)p(u|Z)||q(f |X, Z, u)p(u|Z)), as noted in [9], Section 2.3.6. 3 Marginals are in turn obtained from (6): p(fj |xj , Z, u) = N (fj |kj K−1 u, λj ), where kj is the j-th uu row of Kfu and λj is the j-th element of the diagonal of matrix Λf = diag(Kf f − Kfu K−1 Kuf ). uu Operator diag(·) sets all off-diagonal elements to zero, so that Λf is a diagonal matrix. Since p(u|Z) is readily available and also Gaussian, the inducing variables can be integrated out from (7), yielding a new, approximate prior over f (x): n p(f |X, Z) = p(fj |xj , Z, u)p(u|Z)du = N (f |0, Kfu K−1 Kuf + Λf ) uu p(f , u|X, Z)du ≈ j=1 Using this approximate prior, the posterior distribution for a test case is: pIDGP (y∗ |x∗ , D, Z) = N (y∗ |ku∗ Q−1 Kfu Λ−1 y, σ 2 + k∗∗ + ku∗ (Q−1 − K−1 )ku∗ ), y uu (8) Kfu Λ−1 Kfu y where we have deﬁned Q = Kuu + and Λy = Λf + σ 2 In . The distribution (2) is approximated by (8) with the information available in the pseudo data set. After O(m2 n) time precomputations, predictive means and variances can be computed in O(m) and O(m2 ) time per test case, respectively. This model is, in general, non-stationary, even when it is approximating a stationary input-domain covariance and can be interpreted as a degenerate GP plus heteroscedastic white noise. The log-marginal likelihood (or log-evidence) of the model, explicitly including the conditioning on kernel hyperparameters θ can be expressed as 1 log p(y|X, Z, θ) = − [y Λ−1 y−y Λ−1 Kfu Q−1 Kfu Λ−1 y+log(|Q||Λy |/|Kuu |)+n log(2π)] y y y 2 which is also computable in O(m2 n) time. Model selection will be performed by jointly optimising the evidence with respect to the hyperparameters and the inducing features. If analytical derivatives of the covariance function are available, conjugate gradient optimisation can be used with O(m2 n) cost per step. 3.3 On the choice of g(x, z) The feature extraction function g(x, z) deﬁnes the transformed domain in which the pseudo data set lies. According to (3), the inducing variables can be seen as projections of the target function f (x) on the feature extraction function over the whole input space. Therefore, each of them summarises information about the behaviour of f (x) everywhere. The inducing features Z deﬁne the concrete set of functions over which the target function will be projected. It is desirable that this set captures the most signiﬁcant characteristics of the function. This can be achieved either using prior knowledge about data to select {g(x, zi )}m or using a very general family of functions and letting model i=1 selection automatically choose the appropriate set. Another way to choose g(x, z) relies on the form of the posterior. The posterior mean of a GP is often thought of as a linear combination of “basis functions”. For full GPs and other approximations such as [1, 2, 3, 4, 5, 6], basis functions must have the form of the input-domain covariance function. When using IDGPs, basis functions have the form of the inter-domain instance of the covariance function, and can therefore be adjusted by choosing g(x, z), independently of the input-domain covariance function. If two feature extraction functions g(·, ·) and h(·, ·) can be related by g(x, z) = h(x, z)r(z) for any function r(·), then both yield the same sparse GP model. This property can be used to simplify the expressions of the instances of the covariance function. In this work we use the same functional form for every feature, i.e. our function set is {g(x, zi )}m , i=1 but it is also possible to use sets with different functional forms for each inducing feature, i.e. {gi (x, zi )}m where each zi may even have a different size (dimension). In the sections below i=1 we will discuss different possible choices for g(x, z). 3.3.1 Relation with Sparse GPs using pseudo-inputs The sparse GP using pseudo-inputs (SPGP) was introduced in [1] and was later renamed to Fully Independent Training Conditional (FITC) model to ﬁt in the systematic framework of [10]. Since 4 the sparse model introduced in Section 3.2 also uses a fully independent training conditional, we will stick to the ﬁrst name to avoid possible confusion. IDGP innovation with respect to SPGP consists in letting the pseudo data set lie in a different domain. If we set gSPGP (x, z) ≡ δ(x − z) where δ(·) is a Dirac delta, we force the pseudo data set to lie in the input domain. Thus there is no longer a transformed space and the original SPGP model is retrieved. In this setting, the inducing features of IDGP play the role of SPGP’s pseudo-inputs. 3.3.2 Relation with Sparse Multiscale GPs Sparse Multiscale GPs (SMGPs) are presented in [11]. Seeking to generalise the SPGP model with ARD SE covariance function, they propose to use a different set of length-scales for each basis function. The resulting model presents a defective variance that is healed by adding heteroscedastic white noise. SMGPs, including the variance improvement, can be derived in a principled way as IDGPs: D 1 gSMGP (x, z) ≡ D d=1 2π(c2 d D kSMGP (x, z ) = exp − d=1 D kSMGP (z, z ) = exp − d=1 − 2) d exp − d=1 (xd − µd )2 2cd2 D µ c with z = 2 d cd2 d=1 (µd − µd )2 2(c2 + cd2 − 2 ) d d (xd − µd )2 2(c2 − 2 ) d d (9) (10) D c2 + d d=1 2 d cd2 − 2. d (11) With this approximation, each basis function has its own centre µ = [µ1 , µ2 , . . . , µd ] and its own length-scales c = [c1 , c2 , . . . , cd ] , whereas global length-scales { d }D are shared by all d=1 inducing features. Equations (10) and (11) are derived from (4) and (5) using (1) and (9). The integrals deﬁning kSMGP (·, ·) converge if and only if c2 ≥ 2 , ∀d , which suggests that other values, d d even if permitted in [11], should be avoided for the model to remain well deﬁned. 3.3.3 Frequency Inducing Features GP If the target function can be described more compactly in the frequency domain than in the input domain, it can be advantageous to let the pseudo data set lie in the former domain. We will pursue that possibility for the case where the input domain covariance is the ARD SE. We will call the resulting sparse model Frequency Inducing Features GP (FIFGP). Directly applying the Fourier transform is not possible because the target function is not square 2 integrable (it has constant power σ0 everywhere, so (5) does not converge). We will workaround this by windowing the target function in the region of interest. It is possible to use a square window, but this results in the covariance being deﬁned in terms of the complex error function, which is very slow to evaluate. Instead, we will use a Gaussian window3 . Since multiplying by a Gaussian in the input domain is equivalent to convolving with a Gaussian in the frequency domain, we will be working with a blurred version of the frequency space. This model is deﬁned by: gFIF (x, z) ≡ D 1 D d=1 2πc2 d D kFIF (x, z ) = exp − d=1 D kFIF (z, z ) = exp − d=1 exp − d=1 x2 d cos ω0 + 2c2 d x2 + c2 ωd2 d d cos ω0 + 2(c2 + 2 ) d d D d=1 exp − d=1 D + exp 3 c4 (ωd + ωd )2 d − 2(2c2 + 2 ) d d d=1 x d ωd with z = ω D 2 d d=1 c2 + d 2 d (13) c4 (ωd − ωd )2 d cos(ω0 − ω0 ) 2(2c2 + 2 ) d d D cos(ω0 + ω0 ) d=1 2 d 2c2 + d 2. d A mixture of m Gaussians could also be used as window without increasing the complexity order. 5 (12) d=1 c2 ωd xd d c2 + 2 d d D 2 c2 (ωd + ωd2 ) d 2(2c2 + 2 ) d d D (14) The inducing features are ω = [ω0 , ω1 , . . . , ωd ] , where ω0 is the phase and the remaining components are frequencies along each dimension. In this model, both global length-scales { d }D and d=1 window length-scales {cd }D are shared, thus cd = cd . Instances (13) and (14) are induced by (12) d=1 using (4) and (5). 3.3.4 Time-Frequency Inducing Features GP Instead of using a single window to select the region of interest, it is possible to use a different window for each feature. We will use windows of the same size but different centres. The resulting model combines SPGP and FIFGP, so we will call it Time-Frequency Inducing Features GP (TFIFGP). It is deﬁned by gTFIF (x, z) ≡ gFIF (x − µ, ω), with z = [µ ω ] . The implied inter-domain and transformed-domain instances of the covariance function are: D kTFIF (x, z ) = kFIF (x − µ , ω ) , kTFIF (z, z ) = kFIF (z, z ) exp − d=1 (µd − µd )2 2(2c2 + 2 ) d d FIFGP is trivially obtained by setting every centre to zero {µi = 0}m , whereas SPGP is obtained i=1 by setting window length-scales c, frequencies and phases {ω i }m to zero. If the window lengthi=1 scales were individually adjusted, SMGP would be obtained. While FIFGP has the modelling power of both FIFGP and SPGP, it might perform worse in practice due to it having roughly twice as many hyperparameters, thus making the optimisation problem harder. The same problem also exists in SMGP. A possible workaround is to initialise the hyperparameters using a simpler model, as done in [11] for SMGP, though we will not do this here. 4 Experiments In this section we will compare the proposed approximations FIFGP and TFIFGP with the current state of the art, SPGP on some large data sets, for the same number of inducing features/inputs and therefore, roughly equal computational cost. Additionally, we provide results using a full GP, which is expected to provide top performance (though requiring an impractically big amount of computation). In all cases, the (input-domain) covariance function is the ARD SE (1). We use four large data sets: Kin-40k, Pumadyn-32nm4 (describing the dynamics of a robot arm, used with SPGP in [1]), Elevators and Pole Telecomm5 (related to the control of the elevators of an F16 aircraft and a telecommunications problem, and used in [12, 13, 14]). Input dimensions that remained constant throughout the training set were removed. Input data was additionally centred for use with FIFGP (the remaining methods are translation invariant). Pole Telecomm outputs actually take discrete values in the 0-100 range, in multiples of 10. This was taken into account by using the corresponding quantization noise variance (102 /12) as lower bound for the noise hyperparameter6 . n 1 2 2 2 Hyperparameters are initialised as follows: σ0 = n j=1 yj , σ 2 = σ0 /4, { d }D to one half of d=1 the range spanned by training data along each dimension. For SPGP, pseudo-inputs are initialised to a random subset of the training data, for FIFGP window size c is initialised to the standard deviation of input data, frequencies are randomly chosen from a zero-mean −2 -variance Gaussian d distribution, and phases are obtained from a uniform distribution in [0 . . . 2π). TFIFGP uses the same initialisation as FIFGP, with window centres set to zero. Final values are selected by evidence maximisation. Denoting the output average over the training set as y and the predictive mean and variance for test sample y∗l as µ∗l and σ∗l respectively, we deﬁne the following quality measures: Normalized Mean Square Error (NMSE) (y∗l − µ∗l )2 / (y∗l − y)2 and Mean Negative Log-Probability (MNLP) 1 2 2 2 2 (y∗l − µ∗l ) /σ∗l + log σ∗l + log 2π , where · averages over the test set. 4 Kin-40k: 8 input dimensions, 10000/30000 samples for train/test, Pumadyn-32nm: 32 input dimensions, 7168/1024 samples for train/test, using exactly the same preprocessing and train/test splits as [1, 3]. Note that their error measure is actually one half of the Normalized Mean Square Error deﬁned here. 5 Pole Telecomm: 26 non-constant input dimensions, 10000/5000 samples for train/test. Elevators: 17 non-constant input dimensions, 8752/7847 samples for train/test. Both have been downloaded from http://www.liaad.up.pt/∼ltorgo/Regression/datasets.html 6 If unconstrained, similar plots are obtained; in particular, no overﬁtting is observed. 6 For Kin-40k (Fig. 1, top), all three sparse methods perform similarly, though for high sparseness (the most useful case) FIFGP and TFIFGP are slightly superior. In Pumadyn-32nm (Fig. 1, bottom), only 4 out the 32 input dimensions are relevant to the regression task, so it can be used as an ARD capabilities test. We follow [1] and use a full GP on a small subset of the training data (1024 data points) to obtain the initial length-scales. This allows better minima to be found during optimisation. Though all methods are able to properly ﬁnd a good solution, FIFGP and especially TFIFGP are better in the sparser regime. Roughly the same considerations can be made about Pole Telecomm and Elevators (Fig. 2), but in these data sets the superiority of FIFGP and TFIFGP is more dramatic. Though not shown here, we have additionally tested these models on smaller, overﬁtting-prone data sets, and have found no noticeable overﬁtting even using m > n, despite the relatively high number of parameters being adjusted. This is in line with the results and discussion of [1]. Mean Negative Log−Probability Normalized Mean Squared Error 2.5 0.5 0.1 0.05 0.01 0.005 SPGP FIFGP TFIFGP Full GP on 10000 data points 0.001 25 50 100 200 300 500 750 SPGP FIFGP TFIFGP Full GP on 10000 data points 2 1.5 1 0.5 0 −0.5 −1 −1.5 1250 25 50 100 200 300 500 750 1250 Inducing features / pseudo−inputs (b) Kin-40k MNLP (semilog plot) Inducing features / pseudo−inputs (a) Kin-40k NMSE (log-log plot) 0.05 0.04 10 25 50 75 Mean Negative Log−Probability Normalized Mean Squared Error 0.2 SPGP FIFGP TFIFGP Full GP on 7168 data points 0.1 0.1 0.05 0 −0.05 −0.1 −0.15 −0.2 100 Inducing features / pseudo−inputs (c) Pumadyn-32nm NMSE (log-log plot) SPGP FIFGP TFIFGP Full GP on 7168 data points 0.15 10 25 50 75 100 Inducing features / pseudo−inputs (d) Pumadyn-32nm MNLP (semilog plot) Figure 1: Performance of the compared methods on Kin-40k and Pumadyn-32nm. 5 Conclusions and extensions In this work we have introduced IDGPs, which are able combine representations of a GP in different domains, and have used them to extend SPGP to handle inducing features lying in a different domain. This provides a general framework for sparse models, which are deﬁned by a feature extraction function. Using this framework, SMGPs can be reinterpreted as fully principled models using a transformed space of local features, without any need for post-hoc variance improvements. Furthermore, it is possible to develop new sparse models of practical use, such as the proposed FIFGP and TFIFGP, which are able to outperform the state-of-the-art SPGP on some large data sets, especially for high sparsity regimes. 7 0.25 0.2 0.15 0.1 10 25 50 100 250 Mean Negative Log−Probability Normalized Mean Squared Error −3.8 SPGP FIFGP TFIFGP Full GP on 8752 data points SPGP FIFGP TFIFGP Full GP on 8752 data points −4 −4.2 −4.4 −4.6 −4.8 500 750 1000 10 Inducing features / pseudo−inputs (a) Elevators NMSE (log-log plot) 25 50 100 250 500 750 1000 Inducing features / pseudo−inputs (b) Elevators MNLP (semilog plot) 0.2 0.15 0.1 0.05 0.04 0.03 0.02 0.01 10 25 50 100 250 500 Mean Negative Log−Probability Normalized Mean Squared Error 5.5 SPGP FIFGP TFIFGP Full GP on 10000 data points 4.5 4 3.5 3 2.5 1000 SPGP FIFGP TFIFGP Full GP on 10000 data points 5 10 25 50 100 250 500 1000 Inducing features / pseudo−inputs (d) Pole Telecomm MNLP (semilog plot) Inducing features / pseudo−inputs (c) Pole Telecomm NMSE (log-log plot) Figure 2: Performance of the compared methods on Elevators and Pole Telecomm. Choosing a transformed space for the inducing features enables to use domains where the target function can be expressed more compactly, or where the evidence (which is a function of the features) is easier to optimise. This added ﬂexibility translates as a detaching of the functional form of the input-domain covariance and the set of basis functions used to express the posterior mean. IDGPs approximate full GPs optimally in the KL sense noted in Section 3.2, for a given set of inducing features. Using ML-II to select the inducing features means that models providing a good ﬁt to data are given preference over models that might approximate the full GP more closely. This, though rarely, might lead to harmful overﬁtting. To more faithfully approximate the full GP and avoid overﬁtting altogether, our proposal can be combined with the variational approach from [15], in which the inducing features would be regarded as variational parameters. This would result in more constrained models, which would be closer to the full GP but might show reduced performance. We have explored the case of regression with Gaussian noise, which is analytically tractable, but it is straightforward to apply the same model to other tasks such as robust regression or classiﬁcation, using approximate inference (see [16]). Also, IDGPs as a general tool can be used for other purposes, such as modelling noise in the frequency domain, aggregating data from different domains or even imposing constraints on the target function. Acknowledgments We would like to thank the anonymous referees for helpful comments and suggestions. This work has been partly supported by the Spanish government under grant TEC2008- 02473/TEC, and by the Madrid Community under grant S-505/TIC/0223. 8 References [1] E. Snelson and Z. Ghahramani. Sparse Gaussian processes using pseudo-inputs. In Advances in Neural Information Processing Systems 18, pages 1259–1266. MIT Press, 2006. [2] A. J. Smola and P. Bartlett. Sparse greedy Gaussian process regression. In Advances in Neural Information Processing Systems 13, pages 619–625. MIT Press, 2001. [3] M. Seeger, C. K. I. Williams, and N. D. Lawrence. Fast forward selection to speed up sparse Gaussian process regression. In Proceedings of the 9th International Workshop on AI Stats, 2003. [4] V. Tresp. A Bayesian committee machine. Neural Computation, 12:2719–2741, 2000. [5] L. Csat´ and M. Opper. Sparse online Gaussian processes. Neural Computation, 14(3):641–669, 2002. o [6] C. K. I. Williams and M. Seeger. Using the Nystr¨ m method to speed up kernel machines. In Advances o in Neural Information Processing Systems 13, pages 682–688. MIT Press, 2001. [7] C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. MIT Press, 2006. [8] M. Alvarez and N. D. Lawrence. Sparse convolved Gaussian processes for multi-output regression. In Advances in Neural Information Processing Systems 21, pages 57–64, 2009. [9] Ed. Snelson. Flexible and efﬁcient Gaussian process models for machine learning. PhD thesis, University of Cambridge, 2007. [10] J. Qui˜ onero-Candela and C. E. Rasmussen. A unifying view of sparse approximate Gaussian process n regression. Journal of Machine Learning Research, 6:1939–1959, 2005. [11] C. Walder, K. I. Kim, and B. Sch¨ lkopf. Sparse multiscale Gaussian process regression. In 25th Internao tional Conference on Machine Learning. ACM Press, New York, 2008. [12] G. Potgietera and A. P. Engelbrecht. Evolving model trees for mining data sets with continuous-valued classes. Expert Systems with Applications, 35:1513–1532, 2007. [13] L. Torgo and J. Pinto da Costa. Clustered partial linear regression. In Proceedings of the 11th European Conference on Machine Learning, pages 426–436. Springer, 2000. [14] G. Potgietera and A. P. Engelbrecht. Pairwise classiﬁcation as an ensemble technique. In Proceedings of the 13th European Conference on Machine Learning, pages 97–110. Springer-Verlag, 2002. [15] M. K. Titsias. Variational learning of inducing variables in sparse Gaussian processes. In Proceedings of the 12th International Workshop on AI Stats, 2009. [16] A. Naish-Guzman and S. Holden. The generalized FITC approximation. In Advances in Neural Information Processing Systems 20, pages 1057–1064. MIT Press, 2008. 9

3 0.19461085 43 nips-2009-Bayesian estimation of orientation preference maps

Author: Sebastian Gerwinn, Leonard White, Matthias Kaschube, Matthias Bethge, Jakob H. Macke

Abstract: Imaging techniques such as optical imaging of intrinsic signals, 2-photon calcium imaging and voltage sensitive dye imaging can be used to measure the functional organization of visual cortex across different spatial and temporal scales. Here, we present Bayesian methods based on Gaussian processes for extracting topographic maps from functional imaging data. In particular, we focus on the estimation of orientation preference maps (OPMs) from intrinsic signal imaging data. We model the underlying map as a bivariate Gaussian process, with a prior covariance function that reﬂects known properties of OPMs, and a noise covariance adjusted to the data. The posterior mean can be interpreted as an optimally smoothed estimate of the map, and can be used for model based interpolations of the map from sparse measurements. By sampling from the posterior distribution, we can get error bars on statistical properties such as preferred orientations, pinwheel locations or pinwheel counts. Finally, the use of an explicit probabilistic model facilitates interpretation of parameters and quantitative model comparisons. We demonstrate our model both on simulated data and on intrinsic signaling data from ferret visual cortex. 1

4 0.12216941 100 nips-2009-Gaussian process regression with Student-t likelihood

Author: Jarno Vanhatalo, Pasi Jylänki, Aki Vehtari

Abstract: In the Gaussian process regression the observation model is commonly assumed to be Gaussian, which is convenient in computational perspective. However, the drawback is that the predictive accuracy of the model can be signiﬁcantly compromised if the observations are contaminated by outliers. A robust observation model, such as the Student-t distribution, reduces the inﬂuence of outlying observations and improves the predictions. The problem, however, is the analytically intractable inference. In this work, we discuss the properties of a Gaussian process regression model with the Student-t likelihood and utilize the Laplace approximation for approximate inference. We compare our approach to a variational approximation and a Markov chain Monte Carlo scheme, which utilize the commonly used scale mixture representation of the Student-t distribution. 1

5 0.12201885 101 nips-2009-Generalization Errors and Learning Curves for Regression with Multi-task Gaussian Processes

Author: Kian M. Chai

Abstract: We provide some insights into how task correlations in multi-task Gaussian process (GP) regression affect the generalization error and the learning curve. We analyze the asymmetric two-tasks case, where a secondary task is to help the learning of a primary task. Within this setting, we give bounds on the generalization error and the learning curve of the primary task. Our approach admits intuitive understandings of the multi-task GP by relating it to single-task GPs. For the case of one-dimensional input-space under optimal sampling with data only for the secondary task, the limitations of multi-task GP can be quantiﬁed explicitly. 1

6 0.10622972 255 nips-2009-Variational Inference for the Nested Chinese Restaurant Process

7 0.086927481 120 nips-2009-Kernels and learning curves for Gaussian process regression on random graphs

8 0.086335033 18 nips-2009-A Stochastic approximation method for inference in probabilistic graphical models

9 0.085267954 224 nips-2009-Sparse and Locally Constant Gaussian Graphical Models

10 0.081389964 228 nips-2009-Speeding up Magnetic Resonance Image Acquisition by Bayesian Multi-Slice Adaptive Compressed Sensing

11 0.080892608 147 nips-2009-Matrix Completion from Noisy Entries

12 0.076216139 47 nips-2009-Boosting with Spatial Regularization

13 0.07613761 41 nips-2009-Bayesian Source Localization with the Multivariate Laplace Prior

14 0.074758731 140 nips-2009-Linearly constrained Bayesian matrix factorization for blind source separation

15 0.067832761 133 nips-2009-Learning models of object structure

16 0.06360092 240 nips-2009-Sufficient Conditions for Agnostic Active Learnable

17 0.062564895 137 nips-2009-Learning transport operators for image manifolds

18 0.057308607 19 nips-2009-A joint maximum-entropy model for binary neural population patterns and continuous signals

19 0.057183273 157 nips-2009-Multi-Label Prediction via Compressed Sensing

20 0.056014035 208 nips-2009-Robust Principal Component Analysis: Exact Recovery of Corrupted Low-Rank Matrices via Convex Optimization

similar papers computed by lsi model

lsi for this paper:

topicId topicWeight

[(0, -0.205), (1, -0.061), (2, 0.025), (3, 0.029), (4, 0.071), (5, -0.086), (6, 0.162), (7, -0.059), (8, 0.004), (9, -0.049), (10, 0.003), (11, -0.026), (12, -0.009), (13, 0.067), (14, -0.033), (15, -0.012), (16, -0.064), (17, 0.234), (18, -0.046), (19, -0.2), (20, -0.071), (21, -0.088), (22, -0.035), (23, -0.078), (24, 0.091), (25, -0.022), (26, -0.128), (27, -0.105), (28, 0.155), (29, 0.031), (30, -0.047), (31, 0.101), (32, -0.025), (33, 0.148), (34, 0.028), (35, 0.005), (36, -0.072), (37, 0.008), (38, 0.108), (39, 0.023), (40, -0.032), (41, -0.121), (42, -0.036), (43, 0.023), (44, 0.088), (45, -0.015), (46, -0.011), (47, -0.019), (48, 0.021), (49, -0.051)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.93851161 254 nips-2009-Variational Gaussian-process factor analysis for modeling spatio-temporal data

Author: Jaakko Luttinen, Alexander T. Ihler

2 0.87257707 117 nips-2009-Inter-domain Gaussian Processes for Sparse Inference using Inducing Features

Author: Anibal Figueiras-vidal, Miguel Lázaro-gredilla

3 0.81054795 43 nips-2009-Bayesian estimation of orientation preference maps

Author: Sebastian Gerwinn, Leonard White, Matthias Kaschube, Matthias Bethge, Jakob H. Macke

4 0.70247102 100 nips-2009-Gaussian process regression with Student-t likelihood

Author: Jarno Vanhatalo, Pasi Jylänki, Aki Vehtari

5 0.52765334 228 nips-2009-Speeding up Magnetic Resonance Image Acquisition by Bayesian Multi-Slice Adaptive Compressed Sensing

Author: Matthias Seeger

Abstract: We show how to sequentially optimize magnetic resonance imaging measurement designs over stacks of neighbouring image slices, by performing convex variational inference on a large scale non-Gaussian linear dynamical system, tracking dominating directions of posterior covariance without imposing any factorization constraints. Our approach can be scaled up to high-resolution images by reductions to numerical mathematics primitives and parallelization on several levels. In a ﬁrst study, designs are found that improve signiﬁcantly on others chosen independently for each slice or drawn at random. 1

6 0.49334118 101 nips-2009-Generalization Errors and Learning Curves for Regression with Multi-task Gaussian Processes

7 0.44806024 255 nips-2009-Variational Inference for the Nested Chinese Restaurant Process

8 0.44772366 224 nips-2009-Sparse and Locally Constant Gaussian Graphical Models

9 0.40561783 41 nips-2009-Bayesian Source Localization with the Multivariate Laplace Prior

10 0.40167025 140 nips-2009-Linearly constrained Bayesian matrix factorization for blind source separation

11 0.3911944 120 nips-2009-Kernels and learning curves for Gaussian process regression on random graphs

12 0.37583795 231 nips-2009-Statistical Models of Linear and Nonlinear Contextual Interactions in Early Visual Processing

13 0.36867937 90 nips-2009-Factor Modeling for Advertisement Targeting

14 0.3562451 108 nips-2009-Heterogeneous multitask learning with joint sparsity constraints

15 0.35558751 192 nips-2009-Posterior vs Parameter Sparsity in Latent Variable Models

16 0.32715568 36 nips-2009-Asymptotic Analysis of MAP Estimation via the Replica Method and Compressed Sensing

17 0.32686365 18 nips-2009-A Stochastic approximation method for inference in probabilistic graphical models

18 0.32612416 109 nips-2009-Hierarchical Learning of Dimensional Biases in Human Categorization

19 0.32609096 222 nips-2009-Sparse Estimation Using General Likelihoods and Non-Factorial Priors

20 0.32222533 155 nips-2009-Modelling Relational Data using Bayesian Clustered Tensor Factorization

similar papers computed by lda model

lda for this paper:

topicId topicWeight

[(7, 0.017), (21, 0.012), (24, 0.046), (25, 0.066), (32, 0.177), (35, 0.07), (36, 0.112), (39, 0.036), (48, 0.028), (58, 0.138), (61, 0.01), (71, 0.058), (81, 0.044), (86, 0.071), (91, 0.012), (92, 0.013)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.86338258 254 nips-2009-Variational Gaussian-process factor analysis for modeling spatio-temporal data

Author: Jaakko Luttinen, Alexander T. Ihler

2 0.86143374 85 nips-2009-Explaining human multiple object tracking as resource-constrained approximate inference in a dynamic probabilistic model

Author: Ed Vul, George Alvarez, Joshua B. Tenenbaum, Michael J. Black

Abstract: Multiple object tracking is a task commonly used to investigate the architecture of human visual attention. Human participants show a distinctive pattern of successes and failures in tracking experiments that is often attributed to limits on an object system, a tracking module, or other specialized cognitive structures. Here we use a computational analysis of the task of object tracking to ask which human failures arise from cognitive limitations and which are consequences of inevitable perceptual uncertainty in the tracking task. We ﬁnd that many human performance phenomena, measured through novel behavioral experiments, are naturally produced by the operation of our ideal observer model (a Rao-Blackwelized particle ﬁlter). The tradeoff between the speed and number of objects being tracked, however, can only arise from the allocation of a ﬂexible cognitive resource, which can be formalized as either memory or attention. 1

3 0.81461042 107 nips-2009-Help or Hinder: Bayesian Models of Social Goal Inference

Author: Tomer Ullman, Chris Baker, Owen Macindoe, Owain Evans, Noah Goodman, Joshua B. Tenenbaum

Abstract: Everyday social interactions are heavily inﬂuenced by our snap judgments about others’ goals. Even young infants can infer the goals of intentional agents from observing how they interact with objects and other agents in their environment: e.g., that one agent is ‘helping’ or ‘hindering’ another’s attempt to get up a hill or open a box. We propose a model for how people can infer these social goals from actions, based on inverse planning in multiagent Markov decision problems (MDPs). The model infers the goal most likely to be driving an agent’s behavior by assuming the agent acts approximately rationally given environmental constraints and its model of other agents present. We also present behavioral evidence in support of this model over a simpler, perceptual cue-based alternative. 1

4 0.76099104 158 nips-2009-Multi-Label Prediction via Sparse Infinite CCA

Author: Piyush Rai, Hal Daume

Abstract: Canonical Correlation Analysis (CCA) is a useful technique for modeling dependencies between two (or more) sets of variables. Building upon the recently suggested probabilistic interpretation of CCA, we propose a nonparametric, fully Bayesian framework that can automatically select the number of correlation components, and effectively capture the sparsity underlying the projections. In addition, given (partially) labeled data, our algorithm can also be used as a (semi)supervised dimensionality reduction technique, and can be applied to learn useful predictive features in the context of learning a set of related tasks. Experimental results demonstrate the efﬁcacy of the proposed approach for both CCA as a stand-alone problem, and when applied to multi-label prediction. 1

5 0.75788575 224 nips-2009-Sparse and Locally Constant Gaussian Graphical Models

Author: Jean Honorio, Dimitris Samaras, Nikos Paragios, Rita Goldstein, Luis E. Ortiz

Abstract: Locality information is crucial in datasets where each variable corresponds to a measurement in a manifold (silhouettes, motion trajectories, 2D and 3D images). Although these datasets are typically under-sampled and high-dimensional, they often need to be represented with low-complexity statistical models, which are comprised of only the important probabilistic dependencies in the datasets. Most methods attempt to reduce model complexity by enforcing structure sparseness. However, sparseness cannot describe inherent regularities in the structure. Hence, in this paper we ﬁrst propose a new class of Gaussian graphical models which, together with sparseness, imposes local constancy through 1 -norm penalization. Second, we propose an efﬁcient algorithm which decomposes the strictly convex maximum likelihood estimation into a sequence of problems with closed form solutions. Through synthetic experiments, we evaluate the closeness of the recovered models to the ground truth. We also test the generalization performance of our method in a wide range of complex real-world datasets and demonstrate that it captures useful structures such as the rotation and shrinking of a beating heart, motion correlations between body parts during walking and functional interactions of brain regions. Our method outperforms the state-of-the-art structure learning techniques for Gaussian graphical models both for small and large datasets. 1

6 0.75777662 162 nips-2009-Neural Implementation of Hierarchical Bayesian Inference by Importance Sampling

7 0.75661808 100 nips-2009-Gaussian process regression with Student-t likelihood

8 0.75653559 62 nips-2009-Correlation Coefficients are Insufficient for Analyzing Spike Count Dependencies

9 0.75604934 35 nips-2009-Approximating MAP by Compensating for Structural Relaxations

10 0.75458586 41 nips-2009-Bayesian Source Localization with the Multivariate Laplace Prior

11 0.75431627 19 nips-2009-A joint maximum-entropy model for binary neural population patterns and continuous signals

12 0.75376523 228 nips-2009-Speeding up Magnetic Resonance Image Acquisition by Bayesian Multi-Slice Adaptive Compressed Sensing

13 0.75271165 61 nips-2009-Convex Relaxation of Mixture Regression with Efficient Algorithms

14 0.75242585 215 nips-2009-Sensitivity analysis in HMMs with application to likelihood maximization

15 0.75059652 30 nips-2009-An Integer Projected Fixed Point Method for Graph Matching and MAP Inference

16 0.74948961 191 nips-2009-Positive Semidefinite Metric Learning with Boosting

17 0.74869162 104 nips-2009-Group Sparse Coding

18 0.74823701 97 nips-2009-Free energy score space

19 0.746629 70 nips-2009-Discriminative Network Models of Schizophrenia

20 0.74545062 36 nips-2009-Asymptotic Analysis of MAP Estimation via the Replica Method and Compressed Sensing