jmlr jmlr2010 jmlr2010-41 knowledge-graph by maker-knowledge-mining

41 jmlr-2010-Gaussian Processes for Machine Learning (GPML) Toolbox

Source: pdf

Author: Carl Edward Rasmussen, Hannes Nickisch

Abstract: The GPML toolbox provides a wide range of functionality for Gaussian process (GP) inference and prediction. GPs are speciﬁed by mean and covariance functions; we offer a library of simple mean and covariance functions and mechanisms to compose more complex ones. Several likelihood functions are supported including Gaussian and heavy-tailed for regression as well as others suitable for classiﬁcation. Finally, a range of inference methods is provided, including exact and variational inference, Expectation Propagation, and Laplace’s method dealing with non-Gaussian likelihoods and FITC for dealing with large regression tasks. Keywords: Gaussian processes, nonparametric Bayes, probabilistic regression and classiﬁcation Gaussian processes (GPs) (Rasmussen and Williams, 2006) have convenient properties for many modelling tasks in machine learning and statistics. They can be used to specify distributions over functions without having to commit to a speciﬁc functional form. Applications range from regression over classiﬁcation to reinforcement learning, spatial models, survival and other time series1 models. Predictions of GP models come with a natural conﬁdence measure: predictive error-bars. Although the implementation of the basic principles in the simplest case is straight forward, various complicating features are often desired in practice. For example, a GP is determined by a mean function and a covariance function, but these functions are mostly difﬁcult to specify fully a priori, and typically they are given in terms of hyperparameters, that is, parameters which have to be inferred. Another source of difﬁculty is the likelihood function. For Gaussian likelihoods, inference is analytically tractable; however, in many tasks, Gaussian likelihoods are not appropriate, and approximate inference methods such as Expectation Propagation (EP) (Minka, 2001), Laplace’s approximation (LA) (Williams and Barber, 1998) and variational bounds (VB) (Gibbs and MacKay, 2000) become necessary (Nickisch and Rasmussen, 2008). In case of large training data, approximations (Candela and Rasmussen, 2005) like FITC (Snelson and Ghahramani, 2006) are needed. The GPML toolbox is designed to overcome these hurdles with its variety of mean, covariance and likelihood functions as well as inference methods, while being simple to use and easy to extend. ∗. Also at Max Planck Institute for Biological Cybernetics, Spemannstraße 38, 72076 T¨ bingen, Germany. u 1. Note, that here we typically think of GPs with a more general index set than time. ©2010 Carl Edward Rasmussen and Hannes Nickisch. R ASMUSSEN AND N ICKISCH 1. Implementation The GPML toolbox can be obtained from http://gaussianprocess.org/gpml/code/matlab/ and also http://mloss.org/software/view/263/ under the FreeBSD license. Based on simple interfaces for covariance, mean, likelihood functions as well as inference methods, we offer full compatibility to both Matlab 7.x2 and GNU Octave 3.2.x.3 Special attention has been given to properly disentangle covariance, likelihood and mean hyperparameters. Also, care has been taken to avoid numerical inaccuracies, for example, safe likelihood evaluations for extreme inputs and stable matrix operations. For example, the covariance matrix K can become numerically close to singular making its naive inversion numerically unsafe. We handle these situations in a principled way4 such that Cholesky decompositions are computed of well-conditioned matrices only. As a result, our code shows a high level of robustness along the full spectrum of possible hyperparameters. The focus of the toolbox is on approximate inference using dense matrix algebra. We currently do not support covariance matrix approximation techniques to deal with large numbers of training examples n. Looking at the (growing) body of literature on sparse approximations, this knowledge is still somewhat in ﬂux, and consensus on the best approaches has not yet been reached. We provide stable and modular code checked by an exhaustive suite of test cases. A single function gp.m serves as main interface to the user—it can make inference and predictions and allows the mean, covariance and likelihood function as well as the inference methods to be speciﬁed freely. Furthermore, gp.m enables convenient learning of the hyperparameters by maximising the log marginal likelihood ln Z. One of the particularly appealing properties of GP models is that principled and practical approaches exist for learning the parameters of mean, covariance and likelihood functions. Good adaptation of such parameters can be essential to obtain both high quality predictions and insights into the properties of the data. The GPML toolbox is particularly ﬂexible, including a large library of different covariance and mean functions, and ﬂexible ways to combine these into more expressive, specialised functions. The user can choose between two gradient-based optimisers: one uses conjugate gradients (CG)5 and the other one relies on a quasi-Newton scheme.6 ∂ Computing the derivatives w.r.t. hyperparameters ∂θi ln Z with gp.m does not need any extra programming effort; every inference method automatically collects the respective derivatives from the mean, covariance and likelihood functions and passes them to gp.m. Our documentation comes in two pieces: a hypertext user documentation7 doc/index.html with examples and code browsing and a technical documentation8 doc/manual.pdf focusing on the interfaces and more technical issues. A casual user will use the hypertext document to quickly get his data analysed, however a power user will consult the pdf document once he wants to include his own mean, covariance, likelihood and inference routines or learn about implementation details. 2. 3. 4. 5. 6. 7. 8. Matlab is available from MathWorks, http://www.mathworks.com/. Octave is available from the Free Software Foundation, http://www.gnu.org/software/octave/. We do not consider the “blind” addition of a “small ridge” to K a principled way. Carl Rasmussen’s code is available at http://www.kyb.tuebingen.mpg.de/bs/people/carl/code/minimize/. Peter Carbonetto’s wrapper can be found at http://www.cs.ubc.ca/˜pcarbo/lbfgsb-for-matlab.html. Documentation can be found at http://www.gaussianprocess.org/gpml/code/matlab/doc/index.html. Technical docs are available at http://www.gaussianprocess.org/gpml/code/matlab/doc/manual.pdf. 3012 G AUSSIAN P ROCESSES FOR M ACHINE L EARNING T OOLBOX 2. The GPML Toolbox We illustrate the modular structure of the GPML toolbox by means of a simple code example. GPs are used to formalise and update knowledge about distributions over functions. A GP prior distribution on an unknown latent function f ∼ GP (mφ (x), kψ (x, x′ )), consists of a mean function m(x) = E[ f (x)], and a covariance function k(x, x) = E[( f (x) − m(x))( f (x′ ) − m(x′ ))], both of which typically contain hyperparameters φ and ψ, which we want to ﬁt in the light of data. We generally assume independent observations, that is, input/output pairs (xi , yi ) of f with joint likelihood Pρ (y|f) = ∏n Pρ (yi | f (xi )) factorising over cases. Finally, after speciﬁcation of the prior and i=1 ﬁtting of the hyperparameters θ = {φ, ψ, ρ}, we wish to compute predictive distributions for test cases. 1 2 3 4 5 6 7 % 1) SET UP THE GP : COVARIANCE ; MEAN , LIKELIHOOD , INFERENCE METHOD mf = { ’ meanSum ’ ,{ ’ meanLinear ’, @meanConst }}; a = 2; b = 1; % m(x) = a*x+b cf = { ’ covSEiso ’}; sf = 1; ell = 0.7; % squared exponential covariance funct lf = ’ likLaplace ’; sn = 0.2; % assume Laplace noise with variance sn ˆ2 hyp0 . mean = [a;b ]; hyp0 . cov = log([ ell ; sf ]); hyp0 . lik = log( sn ); % hypers inf = ’ infEP ’; % specify expectation propagation as inference method % 2) MINIMISE NEGATIVE LOG MARGINAL LIKELIHOOD nlZ wrt . hyp ; do 50 CG steps Ncg = 50; [hyp , nlZ ] = minimize ( hyp0 , ’gp ’, -Ncg , inf , mf , cf , lf , X , y ); % 3) PREDICT AT UNKNOWN TEST INPUTS [ymu , ys2 ] = gp (hyp , inf , mf , cf , lf , X , y , Xs ); % test input Xs In line 1, we specify the mean mφ (x) = a⊤ x + b of the GP with hyperparameters φ = {a, b}. First, the functional form of the mean function is given and its parameters are initialised. The desired mean function, happens not to exist in the library of mean functions; instead we have to make a composite mean function from simple constituents. This is done using a nested cell array containing the algebraic expression for m(x): As the sum of a linear (mean/meanLinear.m) and a constant mean function (mean/meanConst.m) it is an afﬁne function. In addition to linear and constant mean functions, the toolbox offers m(x) = 0 and m(x) = 1. These simple mean functions can be combined by composite mean functions to obtain sums (mean/meanSum.m) m(x) = ∑ j m j (x), products m(x) = ∏ j m j (x), scaled versions m(x) = αm0 (x) and powers m(x) = m0 (x)d . This ﬂexible mechanism is used for convenient speciﬁcation of an extensible algebra of mean functions. Note that functions are referred to either as name strings ’meanConst’ or alternatively function handles @meanConst. The order of components of the hyperparameters φ is the same as in the speciﬁcation of the cell array. Every mean function implements its evaluation m = mφ (X) and ﬁrst derivative ∂ computation mi = ∂φi mφ (X) on a data set X. In the same spirit, the squared exponential covariance kψ (x, x′ ) = σ f ² exp(− x − x′ 2 /2ℓ2 ) (cov/covSEiso.m) with hyperparameters ψ = {ln ℓ, ln σ f } is set up in line 2. Note, that the hyperparameters are represented by the logarithms, as these parameters are naturally positive. Many other simple covariance functions are contained in the toolbox. Among others, we offer linear, constant, Mat´ rn, rational quadratic, polynomial, periodic, neural network and ﬁnite support coe variance functions. Composite covariance functions allow for sums k(x, x′ ) = ∑ j k j (x, x′ ), products k(x, x′ ) = ∏ j k j (x, x′ ), positive scaling k(x, x′ ) = σ2 k0 (x, x′ ) and masking of components f k(x, x′ ) = k0 (xI , x′ ) with I ⊆ [1, 2, .., D], x ∈ RD . Again, the interface is simple since only the I ∂ evaluation of the covariance matrix K = kψ (X) and its derivatives ∂i K = ∂ψi kψ (X) on a data set X are required. Furthermore, we need cross terms k∗ = kψ (X, x∗ ) and k∗∗ = kψ (x∗ , x∗ ) for prediction. There are no restrictions on the composition of both mean and covariance functions—any combination is allowed including nested composition. 3013 R ASMUSSEN AND N ICKISCH √ √ The Laplace (lik/likLaplace.m) likelihood Pρ (y| f ) = exp(− 2/σn |y − f |)/ 2σn with hyperparameters ρ = {ln σn } is speciﬁed in line 3. There are only simple likelihood functions: Gaussian, Sech-squared, Laplacian and Student’s t for ordinary and sparse regression as well as the error and the logistic function for classiﬁcation. Again, the same inference code is used for any likelihood function. Although the speciﬁcation of likelihood functions is simple for the user, writing new likelihood functions is slightly more involved as different inference methods require access to different properties; for example, LA requires second derivatives and EP requires derivatives of moments. All hyperparameters θ = {φ, ψ, ρ} are stored in a struct hyp.{mean,cov,lik}, which is initialised in line 4; we select the approximate inference algorithm EP (inf/infEP.m) in line 5. We optimise the hyperparameters θ ≡ hyp by calling the CG optimiser (util/minimize.m) with initial value θ0 ≡ hyp0 in line 6 allowing at most N = 50 evaluations of the EP approximation to the marginal likelihood ZEP (θ) as done by gp.m. Here, D = (X, y) ≡ (X,y) is the training data where X = {x1 , .., xn } and y ∈ Rn . Under the hood, gp.m computes in every step a Gaussian ∂ posterior approximation and the derivatives ∂θ ln ZEP (θ) of the marginal likelihood by calling EP. Predictions with optimised hyperparameters are done in line 7, where we call gp.m with the unseen test inputs X∗ ≡ Xs as additional argument. As a result, we obtain the approximate marginal predictive mean E[P(y∗ |D , X∗ )] ≡ ymu and the predictive variance V[P(y∗ |D , X∗ )] ≡ ys2. Likelihood \ Inference Gaussian Sech-squared Laplacian Student’s t Error function Logistic function Exact FITC EP Laplace VB Type, Output Domain regression, R regression, R regression, R regression, R classiﬁcation, {±1} classiﬁcation, {±1} Alternate Name logistic distribution double exponential probit regression logit regression Table 1: Likelihood ↔ inference compatibility in the GPML toolbox Table 1 gives the legal likelihood/inference combinations. Exact inference and the FITC approximation support the Gaussian likelihood only. Variational Bayesian (VB) inference is applicable to all likelihoods. Expectation propagation (EP) for the Student’s t likelihood is inherently unstable due to its non-log-concavity. The Laplace approximation (LA) for Laplace likelihoods is not sensible due to the non-differentiable peak of the Laplace likelihood. Special care has been taken for the non-convex optimisation problem imposed by the combination Student’s t likelihood and LA. If the number of training examples is larger than a few thousand, dense matrix computations become too slow. We provide the FITC approximation for regression with Gaussian likelihood where ˜ instead of the exact covariance matrix K, a low-rank plus diagonal matrix K = Q + diag(K − Q) ⊤ K−1 K is used. The matrices K and K contain covariances and cross-covariances where Q = Ku uu u uu u of and between inducing inputs ui and data points x j . Using inf/infFITC.m together with any covariance function wrapped into cov/covFITC.m makes the computations feasible for large n. Acknowledgments Thanks to Ed Snelson for assisting with the FITC approximation. 3014 G AUSSIAN P ROCESSES FOR M ACHINE L EARNING T OOLBOX References Joaquin Qui˜ onero Candela and Carl E. Rasmussen. A unifying view of sparse approximate Gausn sian process regression. Journal of Machine Learning Research, 6(6):1935–1959, 2005. Mark N. Gibbs and David J. C. MacKay. Variational Gaussian process classiﬁers. IEEE Transactions on Neural Networks, 11(6):1458–1464, 2000. Thomas P. Minka. Expectation propagation for approximate Bayesian inference. In UAI, pages 362–369. Morgan Kaufmann, 2001. Hannes Nickisch and Carl E. Rasmussen. Approximations for binary Gaussian process classiﬁcation. Journal of Machine Learning Research, 9:2035–2078, 10 2008. Carl E. Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, Cambridge, MA, 2006. Ed Snelson and Zoubin Ghahramani. Sparse Gaussian processes using pseudo-inputs. In Advances in Neural Information Processing Systems 18, 2006. Christopher K. I. Williams and D. Barber. Bayesian classiﬁcation with Gaussian processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(20):1342–1351, 1998. 3015

Reference: text

Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 DE Max Planck Institute for Biological Cybernetics Spemannstraße 38 72076 T¨ bingen, Germany u Editor: S¨ ren Sonnenburg o Abstract The GPML toolbox provides a wide range of functionality for Gaussian process (GP) inference and prediction. [sent-5, score-0.294]

2 GPs are speciﬁed by mean and covariance functions; we offer a library of simple mean and covariance functions and mechanisms to compose more complex ones. [sent-6, score-0.71]

3 Several likelihood functions are supported including Gaussian and heavy-tailed for regression as well as others suitable for classiﬁcation. [sent-7, score-0.234]

4 Finally, a range of inference methods is provided, including exact and variational inference, Expectation Propagation, and Laplace’s method dealing with non-Gaussian likelihoods and FITC for dealing with large regression tasks. [sent-8, score-0.333]

5 Keywords: Gaussian processes, nonparametric Bayes, probabilistic regression and classiﬁcation Gaussian processes (GPs) (Rasmussen and Williams, 2006) have convenient properties for many modelling tasks in machine learning and statistics. [sent-9, score-0.128]

6 They can be used to specify distributions over functions without having to commit to a speciﬁc functional form. [sent-10, score-0.049]

7 Applications range from regression over classiﬁcation to reinforcement learning, spatial models, survival and other time series1 models. [sent-11, score-0.051]

8 Predictions of GP models come with a natural conﬁdence measure: predictive error-bars. [sent-12, score-0.045]

9 For example, a GP is determined by a mean function and a covariance function, but these functions are mostly difﬁcult to specify fully a priori, and typically they are given in terms of hyperparameters, that is, parameters which have to be inferred. [sent-14, score-0.337]

10 The GPML toolbox is designed to overcome these hurdles with its variety of mean, covariance and likelihood functions as well as inference methods, while being simple to use and easy to extend. [sent-18, score-0.679]

11 Implementation The GPML toolbox can be obtained from http://gaussianprocess. [sent-25, score-0.171]

12 Based on simple interfaces for covariance, mean, likelihood functions as well as inference methods, we offer full compatibility to both Matlab 7. [sent-28, score-0.479]

13 3 Special attention has been given to properly disentangle covariance, likelihood and mean hyperparameters. [sent-32, score-0.269]

14 Also, care has been taken to avoid numerical inaccuracies, for example, safe likelihood evaluations for extreme inputs and stable matrix operations. [sent-33, score-0.272]

15 For example, the covariance matrix K can become numerically close to singular making its naive inversion numerically unsafe. [sent-34, score-0.202]

16 We handle these situations in a principled way4 such that Cholesky decompositions are computed of well-conditioned matrices only. [sent-35, score-0.046]

17 As a result, our code shows a high level of robustness along the full spectrum of possible hyperparameters. [sent-36, score-0.048]

18 The focus of the toolbox is on approximate inference using dense matrix algebra. [sent-37, score-0.294]

19 We currently do not support covariance matrix approximation techniques to deal with large numbers of training examples n. [sent-38, score-0.202]

20 We provide stable and modular code checked by an exhaustive suite of test cases. [sent-40, score-0.102]

21 m serves as main interface to the user—it can make inference and predictions and allows the mean, covariance and likelihood function as well as the inference methods to be speciﬁed freely. [sent-42, score-0.709]

22 m enables convenient learning of the hyperparameters by maximising the log marginal likelihood ln Z. [sent-44, score-0.551]

23 One of the particularly appealing properties of GP models is that principled and practical approaches exist for learning the parameters of mean, covariance and likelihood functions. [sent-45, score-0.431]

24 Good adaptation of such parameters can be essential to obtain both high quality predictions and insights into the properties of the data. [sent-46, score-0.04]

25 The GPML toolbox is particularly ﬂexible, including a large library of different covariance and mean functions, and ﬂexible ways to combine these into more expressive, specialised functions. [sent-47, score-0.542]

26 The user can choose between two gradient-based optimisers: one uses conjugate gradients (CG)5 and the other one relies on a quasi-Newton scheme. [sent-48, score-0.07]

27 m does not need any extra programming effort; every inference method automatically collects the respective derivatives from the mean, covariance and likelihood functions and passes them to gp. [sent-53, score-0.591]

28 Our documentation comes in two pieces: a hypertext user documentation7 doc/index. [sent-55, score-0.179]

29 html with examples and code browsing and a technical documentation8 doc/manual. [sent-56, score-0.048]

30 A casual user will use the hypertext document to quickly get his data analysed, however a power user will consult the pdf document once he wants to include his own mean, covariance, likelihood and inference routines or learn about implementation details. [sent-58, score-0.582]

31 We do not consider the “blind” addition of a “small ridge” to K a principled way. [sent-72, score-0.046]

32 The GPML Toolbox We illustrate the modular structure of the GPML toolbox by means of a simple code example. [sent-92, score-0.273]

33 GPs are used to formalise and update knowledge about distributions over functions. [sent-93, score-0.033]

34 We generally assume independent observations, that is, input/output pairs (xi , yi ) of f with joint likelihood Pρ (y|f) = ∏n Pρ (yi | f (xi )) factorising over cases. [sent-95, score-0.22]

35 Finally, after speciﬁcation of the prior and i=1 ﬁtting of the hyperparameters θ = {φ, ψ, ρ}, we wish to compute predictive distributions for test cases. [sent-96, score-0.274]

36 1 2 3 4 5 6 7 % 1) SET UP THE GP : COVARIANCE ; MEAN , LIKELIHOOD , INFERENCE METHOD mf = { ’ meanSum ’ ,{ ’ meanLinear ’, @meanConst }}; a = 2; b = 1; % m(x) = a*x+b cf = { ’ covSEiso ’}; sf = 1; ell = 0. [sent-97, score-0.36]

37 7; % squared exponential covariance funct lf = ’ likLaplace ’; sn = 0. [sent-98, score-0.342]

38 2; % assume Laplace noise with variance sn ˆ2 hyp0 . [sent-99, score-0.048]

39 lik = log( sn ); % hypers inf = ’ infEP ’; % specify expectation propagation as inference method % 2) MINIMISE NEGATIVE LOG MARGINAL LIKELIHOOD nlZ wrt . [sent-102, score-0.342]

40 First, the functional form of the mean function is given and its parameters are initialised. [sent-104, score-0.086]

41 The desired mean function, happens not to exist in the library of mean functions; instead we have to make a composite mean function from simple constituents. [sent-105, score-0.362]

42 This is done using a nested cell array containing the algebraic expression for m(x): As the sum of a linear (mean/meanLinear. [sent-106, score-0.035]

43 In addition to linear and constant mean functions, the toolbox offers m(x) = 0 and m(x) = 1. [sent-109, score-0.257]

44 These simple mean functions can be combined by composite mean functions to obtain sums (mean/meanSum. [sent-110, score-0.23]

45 This ﬂexible mechanism is used for convenient speciﬁcation of an extensible algebra of mean functions. [sent-112, score-0.122]

46 The order of components of the hyperparameters φ is the same as in the speciﬁcation of the cell array. [sent-114, score-0.229]

47 Every mean function implements its evaluation m = mφ (X) and ﬁrst derivative ∂ computation mi = ∂φi mφ (X) on a data set X. [sent-115, score-0.086]

48 In the same spirit, the squared exponential covariance kψ (x, x′ ) = σ f ² exp(− x − x′ 2 /2ℓ2 ) (cov/covSEiso. [sent-116, score-0.202]

49 m) with hyperparameters ψ = {ln ℓ, ln σ f } is set up in line 2. [sent-117, score-0.292]

50 Note, that the hyperparameters are represented by the logarithms, as these parameters are naturally positive. [sent-118, score-0.229]

51 Many other simple covariance functions are contained in the toolbox. [sent-119, score-0.202]

52 Among others, we offer linear, constant, Mat´ rn, rational quadratic, polynomial, periodic, neural network and ﬁnite support coe variance functions. [sent-120, score-0.092]

53 Composite covariance functions allow for sums k(x, x′ ) = ∑ j k j (x, x′ ), products k(x, x′ ) = ∏ j k j (x, x′ ), positive scaling k(x, x′ ) = σ2 k0 (x, x′ ) and masking of components f k(x, x′ ) = k0 (xI , x′ ) with I ⊆ [1, 2, . [sent-121, score-0.239]

54 Again, the interface is simple since only the I ∂ evaluation of the covariance matrix K = kψ (X) and its derivatives ∂i K = ∂ψi kψ (X) on a data set X are required. [sent-124, score-0.323]

55 There are no restrictions on the composition of both mean and covariance functions—any combination is allowed including nested composition. [sent-126, score-0.323]

56 m) likelihood Pρ (y| f ) = exp(− 2/σn |y − f |)/ 2σn with hyperparameters ρ = {ln σn } is speciﬁed in line 3. [sent-128, score-0.412]

57 There are only simple likelihood functions: Gaussian, Sech-squared, Laplacian and Student’s t for ordinary and sparse regression as well as the error and the logistic function for classiﬁcation. [sent-129, score-0.234]

58 Again, the same inference code is used for any likelihood function. [sent-130, score-0.354]

59 Although the speciﬁcation of likelihood functions is simple for the user, writing new likelihood functions is slightly more involved as different inference methods require access to different properties; for example, LA requires second derivatives and EP requires derivatives of moments. [sent-131, score-0.655]

60 All hyperparameters θ = {φ, ψ, ρ} are stored in a struct hyp. [sent-132, score-0.229]

61 {mean,cov,lik}, which is initialised in line 4; we select the approximate inference algorithm EP (inf/infEP. [sent-133, score-0.16]

62 We optimise the hyperparameters θ ≡ hyp by calling the CG optimiser (util/minimize. [sent-135, score-0.502]

63 m) with initial value θ0 ≡ hyp0 in line 6 allowing at most N = 50 evaluations of the EP approximation to the marginal likelihood ZEP (θ) as done by gp. [sent-136, score-0.223]

64 m computes in every step a Gaussian ∂ posterior approximation and the derivatives ∂θ ln ZEP (θ) of the marginal likelihood by calling EP. [sent-142, score-0.431]

65 Predictions with optimised hyperparameters are done in line 7, where we call gp. [sent-143, score-0.229]

66 m with the unseen test inputs X∗ ≡ Xs as additional argument. [sent-144, score-0.055]

67 As a result, we obtain the approximate marginal predictive mean E[P(y∗ |D , X∗ )] ≡ ymu and the predictive variance V[P(y∗ |D , X∗ )] ≡ ys2. [sent-145, score-0.303]

68 Exact inference and the FITC approximation support the Gaussian likelihood only. [sent-147, score-0.306]

69 Variational Bayesian (VB) inference is applicable to all likelihoods. [sent-148, score-0.123]

70 Expectation propagation (EP) for the Student’s t likelihood is inherently unstable due to its non-log-concavity. [sent-149, score-0.266]

71 The Laplace approximation (LA) for Laplace likelihoods is not sensible due to the non-differentiable peak of the Laplace likelihood. [sent-150, score-0.108]

72 Special care has been taken for the non-convex optimisation problem imposed by the combination Student’s t likelihood and LA. [sent-151, score-0.217]

73 We provide the FITC approximation for regression with Gaussian likelihood where ˜ instead of the exact covariance matrix K, a low-rank plus diagonal matrix K = Q + diag(K − Q) ⊤ K−1 K is used. [sent-153, score-0.436]

74 The matrices K and K contain covariances and cross-covariances where Q = Ku uu u uu u of and between inducing inputs ui and data points x j . [sent-154, score-0.171]

75 3014 G AUSSIAN P ROCESSES FOR M ACHINE L EARNING T OOLBOX References Joaquin Qui˜ onero Candela and Carl E. [sent-159, score-0.037]

76 A unifying view of sparse approximate Gausn sian process regression. [sent-161, score-0.037]

similar papers computed by tfidf model

tfidf for this paper:

wordName wordTfidf (topN-words)

[('gpml', 0.347), ('gp', 0.257), ('hyperparameters', 0.229), ('fitc', 0.223), ('covariance', 0.202), ('laplace', 0.196), ('likelihood', 0.183), ('hyp', 0.174), ('carl', 0.173), ('toolbox', 0.171), ('rasmussen', 0.157), ('ep', 0.135), ('gps', 0.134), ('hannes', 0.13), ('nickisch', 0.13), ('inference', 0.123), ('cf', 0.111), ('snelson', 0.111), ('likelihoods', 0.108), ('mf', 0.1), ('student', 0.098), ('lf', 0.092), ('candela', 0.087), ('ell', 0.087), ('ickisch', 0.087), ('meanconst', 0.087), ('nlz', 0.087), ('ymu', 0.087), ('zep', 0.087), ('mean', 0.086), ('propagation', 0.083), ('gaussian', 0.083), ('derivatives', 0.083), ('asmussen', 0.074), ('cg', 0.07), ('user', 0.07), ('vb', 0.067), ('compatibility', 0.067), ('oolbox', 0.067), ('ln', 0.063), ('achine', 0.062), ('hypertext', 0.062), ('calling', 0.062), ('rocesses', 0.062), ('sf', 0.062), ('xs', 0.06), ('composite', 0.058), ('octave', 0.058), ('aussian', 0.058), ('uu', 0.058), ('la', 0.056), ('inputs', 0.055), ('offer', 0.055), ('modular', 0.054), ('spemannstra', 0.054), ('christopher', 0.051), ('edward', 0.051), ('interfaces', 0.051), ('variational', 0.051), ('regression', 0.051), ('planck', 0.049), ('specify', 0.049), ('sn', 0.048), ('code', 0.048), ('williams', 0.047), ('documentation', 0.047), ('library', 0.046), ('principled', 0.046), ('predictive', 0.045), ('exible', 0.043), ('bingen', 0.043), ('processes', 0.041), ('predictions', 0.04), ('marginal', 0.04), ('inf', 0.039), ('interface', 0.038), ('laplacian', 0.038), ('factorising', 0.037), ('optimiser', 0.037), ('casual', 0.037), ('cer', 0.037), ('coe', 0.037), ('consult', 0.037), ('initialised', 0.037), ('joaquin', 0.037), ('masking', 0.037), ('onero', 0.037), ('qui', 0.037), ('sian', 0.037), ('specialised', 0.037), ('trumpington', 0.037), ('wrapped', 0.037), ('convenient', 0.036), ('nested', 0.035), ('care', 0.034), ('barber', 0.033), ('carbonetto', 0.033), ('analysed', 0.033), ('compose', 0.033), ('formalise', 0.033)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 1.0000006 41 jmlr-2010-Gaussian Processes for Machine Learning (GPML) Toolbox

Author: Carl Edward Rasmussen, Hannes Nickisch

2 0.24435462 104 jmlr-2010-Sparse Spectrum Gaussian Process Regression

Author: Miguel Lázaro-Gredilla, Joaquin Quiñonero-Candela, Carl Edward Rasmussen, Aníbal R. Figueiras-Vidal

Abstract: We present a new sparse Gaussian Process (GP) model for regression. The key novel idea is to sparsify the spectral representation of the GP. This leads to a simple, practical algorithm for regression tasks. We compare the achievable trade-offs between predictive accuracy and computational requirements, and show that these are typically superior to existing state-of-the-art sparse approximations. We discuss both the weight space and function space representations, and note that the new construction implies priors over functions which are always stationary, and can approximate any covariance function in this class. Keywords: Gaussian process, probabilistic regression, sparse approximation, power spectrum, computational efﬁciency

3 0.1004642 46 jmlr-2010-High Dimensional Inverse Covariance Matrix Estimation via Linear Programming

Author: Ming Yuan

Abstract: This paper considers the problem of estimating a high dimensional inverse covariance matrix that can be well approximated by “sparse” matrices. Taking advantage of the connection between multivariate linear regression and entries of the inverse covariance matrix, we propose an estimating procedure that can effectively exploit such “sparsity”. The proposed method can be computed using linear programming and therefore has the potential to be used in very high dimensional problems. Oracle inequalities are established for the estimation error in terms of several operator norms, showing that the method is adaptive to different types of sparsity of the problem. Keywords: covariance selection, Dantzig selector, Gaussian graphical model, inverse covariance matrix, Lasso, linear programming, oracle inequality, sparsity

4 0.074562021 111 jmlr-2010-Topology Selection in Graphical Models of Autoregressive Processes

Author: Jitkomut Songsiri, Lieven Vandenberghe

Abstract: An algorithm is presented for topology selection in graphical models of autoregressive Gaussian time series. The graph topology of the model represents the sparsity pattern of the inverse spectrum of the time series and characterizes conditional independence relations between the variables. The method proposed in the paper is based on an ℓ1 -type nonsmooth regularization of the conditional maximum likelihood estimation problem. We show that this reduces to a convex optimization problem and describe a large-scale algorithm that solves the dual problem via the gradient projection method. Results of experiments with randomly generated and real data sets are also included. Keywords: graphical models, time series, topology selection, convex optimization

5 0.072054908 39 jmlr-2010-FastInf: An Efficient Approximate Inference Library

Author: Ariel Jaimovich, Ofer Meshi, Ian McGraw, Gal Elidan

Abstract: The FastInf C++ library is designed to perform memory and time efﬁcient approximate inference in large-scale discrete undirected graphical models. The focus of the library is propagation based approximate inference methods, ranging from the basic loopy belief propagation algorithm to propagation based on convex free energies. Various message scheduling schemes that improve on the standard synchronous or asynchronous approaches are included. Also implemented are a clique tree based exact inference, Gibbs sampling, and the mean ﬁeld algorithm. In addition to inference, FastInf provides parameter estimation capabilities as well as representation and learning of shared parameters. It offers a rich interface that facilitates extension of the basic classes to other inference and learning methods. Keywords: graphical models, Markov random ﬁeld, loopy belief propagation, approximate inference

6 0.067140475 14 jmlr-2010-Approximate Riemannian Conjugate Gradient Learning for Fixed-Form Variational Bayes

7 0.065889992 110 jmlr-2010-The SHOGUN Machine Learning Toolbox

8 0.060904864 99 jmlr-2010-Restricted Eigenvalue Properties for Correlated Gaussian Designs

9 0.059143703 118 jmlr-2010-libDAI: A Free and Open Source C++ Library for Discrete Approximate Inference in Graphical Models

10 0.05586892 75 jmlr-2010-Mean Field Variational Approximation for Continuous-Time Bayesian Networks

11 0.055469289 109 jmlr-2010-Stochastic Composite Likelihood

12 0.054311432 51 jmlr-2010-Importance Sampling for Continuous Time Bayesian Networks

13 0.052715961 29 jmlr-2010-Covariance in Unsupervised Learning of Probabilistic Grammars

14 0.052063145 95 jmlr-2010-Rademacher Complexities and Bounding the Excess Risk in Active Learning

15 0.051642425 78 jmlr-2010-Model Selection: Beyond the Bayesian Frequentist Divide

16 0.049707878 8 jmlr-2010-A Surrogate Modeling and Adaptive Sampling Toolbox for Computer Based Design

17 0.047275528 76 jmlr-2010-Message-passing for Graph-structured Linear Programs: Proximal Methods and Rounding Schemes

18 0.046790771 45 jmlr-2010-High-dimensional Variable Selection with Sparse Random Projections: Measurement Sparsity and Statistical Efficiency

19 0.044419494 83 jmlr-2010-On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation

20 0.044254109 89 jmlr-2010-PAC-Bayesian Analysis of Co-clustering and Beyond

similar papers computed by lsi model

lsi for this paper:

topicId topicWeight

[(0, -0.194), (1, -0.02), (2, -0.056), (3, -0.105), (4, -0.142), (5, 0.002), (6, 0.168), (7, -0.154), (8, -0.222), (9, 0.078), (10, -0.023), (11, -0.125), (12, -0.261), (13, -0.061), (14, 0.116), (15, 0.265), (16, -0.06), (17, 0.05), (18, 0.328), (19, -0.209), (20, -0.039), (21, 0.117), (22, 0.112), (23, -0.062), (24, 0.144), (25, -0.059), (26, 0.094), (27, -0.119), (28, 0.014), (29, 0.079), (30, -0.003), (31, -0.036), (32, -0.015), (33, -0.051), (34, -0.021), (35, 0.029), (36, -0.0), (37, -0.01), (38, 0.078), (39, -0.013), (40, -0.012), (41, 0.021), (42, -0.019), (43, 0.04), (44, -0.032), (45, -0.001), (46, 0.004), (47, 0.022), (48, -0.007), (49, 0.012)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.96008837 41 jmlr-2010-Gaussian Processes for Machine Learning (GPML) Toolbox

Author: Carl Edward Rasmussen, Hannes Nickisch

2 0.91421276 104 jmlr-2010-Sparse Spectrum Gaussian Process Regression

Author: Miguel Lázaro-Gredilla, Joaquin Quiñonero-Candela, Carl Edward Rasmussen, Aníbal R. Figueiras-Vidal

3 0.31019223 46 jmlr-2010-High Dimensional Inverse Covariance Matrix Estimation via Linear Programming

Author: Ming Yuan

4 0.29674718 14 jmlr-2010-Approximate Riemannian Conjugate Gradient Learning for Fixed-Form Variational Bayes

Author: Antti Honkela, Tapani Raiko, Mikael Kuusela, Matti Tornio, Juha Karhunen

Abstract: Variational Bayesian (VB) methods are typically only applied to models in the conjugate-exponential family using the variational Bayesian expectation maximisation (VB EM) algorithm or one of its variants. In this paper we present an efﬁcient algorithm for applying VB to more general models. The method is based on specifying the functional form of the approximation, such as multivariate Gaussian. The parameters of the approximation are optimised using a conjugate gradient algorithm that utilises the Riemannian geometry of the space of the approximations. This leads to a very efﬁcient algorithm for suitably structured approximations. It is shown empirically that the proposed method is comparable or superior in efﬁciency to the VB EM in a case where both are applicable. We also apply the algorithm to learning a nonlinear state-space model and a nonlinear factor analysis model for which the VB EM is not applicable. For these models, the proposed algorithm outperforms alternative gradient-based methods by a signiﬁcant margin. Keywords: variational inference, approximate Riemannian conjugate gradient, ﬁxed-form approximation, Gaussian approximation

5 0.2742295 39 jmlr-2010-FastInf: An Efficient Approximate Inference Library

Author: Ariel Jaimovich, Ofer Meshi, Ian McGraw, Gal Elidan

6 0.27172503 110 jmlr-2010-The SHOGUN Machine Learning Toolbox

7 0.2504788 8 jmlr-2010-A Surrogate Modeling and Adaptive Sampling Toolbox for Computer Based Design

8 0.2481496 109 jmlr-2010-Stochastic Composite Likelihood

9 0.23825815 99 jmlr-2010-Restricted Eigenvalue Properties for Correlated Gaussian Designs

10 0.23743556 118 jmlr-2010-libDAI: A Free and Open Source C++ Library for Discrete Approximate Inference in Graphical Models

11 0.23714872 111 jmlr-2010-Topology Selection in Graphical Models of Autoregressive Processes

12 0.22436452 29 jmlr-2010-Covariance in Unsupervised Learning of Probabilistic Grammars

13 0.21864238 38 jmlr-2010-Expectation Truncation and the Benefits of Preselection In Training Generative Models

14 0.2066631 75 jmlr-2010-Mean Field Variational Approximation for Continuous-Time Bayesian Networks

15 0.18980531 17 jmlr-2010-Bayesian Learning in Sparse Graphical Factor Models via Variational Mean-Field Annealing

16 0.18183051 78 jmlr-2010-Model Selection: Beyond the Bayesian Frequentist Divide

17 0.17181978 92 jmlr-2010-Practical Approaches to Principal Component Analysis in the Presence of Missing Values

18 0.171133 101 jmlr-2010-Second-Order Bilinear Discriminant Analysis

19 0.16851737 62 jmlr-2010-Learning Gradients: Predictive Models that Infer Geometry and Statistical Dependence

20 0.1618496 52 jmlr-2010-Incremental Sigmoid Belief Networks for Grammar Learning

similar papers computed by lda model

lda for this paper:

topicId topicWeight

[(32, 0.03), (36, 0.016), (37, 0.749), (75, 0.076), (85, 0.032), (96, 0.015)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.96225893 41 jmlr-2010-Gaussian Processes for Machine Learning (GPML) Toolbox

Author: Carl Edward Rasmussen, Hannes Nickisch

2 0.87285548 99 jmlr-2010-Restricted Eigenvalue Properties for Correlated Gaussian Designs

Author: Garvesh Raskutti, Martin J. Wainwright, Bin Yu

Abstract: Methods based on ℓ1 -relaxation, such as basis pursuit and the Lasso, are very popular for sparse regression in high dimensions. The conditions for success of these methods are now well-understood: (1) exact recovery in the noiseless setting is possible if and only if the design matrix X satisﬁes the restricted nullspace property, and (2) the squared ℓ2 -error of a Lasso estimate decays at the minimax optimal rate k log p , where k is the sparsity of the p-dimensional regression problem with additive n Gaussian noise, whenever the design satisﬁes a restricted eigenvalue condition. The key issue is thus to determine when the design matrix X satisﬁes these desirable properties. Thus far, there have been numerous results showing that the restricted isometry property, which implies both the restricted nullspace and eigenvalue conditions, is satisﬁed when all entries of X are independent and identically distributed (i.i.d.), or the rows are unitary. This paper proves directly that the restricted nullspace and eigenvalue conditions hold with high probability for quite general classes of Gaussian matrices for which the predictors may be highly dependent, and hence restricted isometry conditions can be violated with high probability. In this way, our results extend the attractive theoretical guarantees on ℓ1 -relaxations to a much broader class of problems than the case of completely independent or unitary designs. Keywords: Lasso, basis pursuit, random matrix theory, Gaussian comparison inequality, concentration of measure

3 0.80409533 110 jmlr-2010-The SHOGUN Machine Learning Toolbox

Author: Sören Sonnenburg, Gunnar Rätsch, Sebastian Henschel, Christian Widmer, Jonas Behr, Alexander Zien, Fabio de Bona, Alexander Binder, Christian Gehl, Vojtěch Franc

Abstract: We have developed a machine learning toolbox, called SHOGUN, which is designed for uniﬁed large-scale learning for a broad range of feature types and learning settings. It offers a considerable number of machine learning models such as support vector machines, hidden Markov models, multiple kernel learning, linear discriminant analysis, and more. Most of the speciﬁc algorithms are able to deal with several different data classes. We have used this toolbox in several applications from computational biology, some of them coming with no less than 50 million training examples and others with 7 billion test examples. With more than a thousand installations worldwide, SHOGUN is already widely adopted in the machine learning community and beyond. SHOGUN is , implemented in C++ and interfaces to MATLABTM R, Octave, Python, and has a stand-alone command line interface. The source code is freely available under the GNU General Public License, Version 3 at http://www.shogun-toolbox.org. Keywords: support vector machines, kernels, large-scale learning, Python, Octave, R

4 0.76595837 42 jmlr-2010-Generalized Expectation Criteria for Semi-Supervised Learning with Weakly Labeled Data

Author: Gideon S. Mann, Andrew McCallum

Abstract: In this paper, we present an overview of generalized expectation criteria (GE), a simple, robust, scalable method for semi-supervised training using weakly-labeled data. GE ﬁts model parameters by favoring models that match certain expectation constraints, such as marginal label distributions, on the unlabeled data. This paper shows how to apply generalized expectation criteria to two classes of parametric models: maximum entropy models and conditional random ﬁelds. Experimental results demonstrate accuracy improvements over supervised training and a number of other stateof-the-art semi-supervised learning methods for these models. Keywords: generalized expectation criteria, semi-supervised learning, logistic regression, conditional random ﬁelds

5 0.75253206 57 jmlr-2010-Iterative Scaling and Coordinate Descent Methods for Maximum Entropy Models

Author: Fang-Lan Huang, Cho-Jui Hsieh, Kai-Wei Chang, Chih-Jen Lin

Abstract: Maximum entropy (Maxent) is useful in natural language processing and many other areas. Iterative scaling (IS) methods are one of the most popular approaches to solve Maxent. With many variants of IS methods, it is difﬁcult to understand them and see the differences. In this paper, we create a general and uniﬁed framework for iterative scaling methods. This framework also connects iterative scaling and coordinate descent methods. We prove general convergence results for IS methods and analyze their computational complexity. Based on the proposed framework, we extend a coordinate descent method for linear SVM to Maxent. Results show that it is faster than existing iterative scaling methods. Keywords: maximum entropy, iterative scaling, coordinate descent, natural language processing, optimization

6 0.56242245 104 jmlr-2010-Sparse Spectrum Gaussian Process Regression

7 0.54437113 118 jmlr-2010-libDAI: A Free and Open Source C++ Library for Discrete Approximate Inference in Graphical Models

8 0.4599936 116 jmlr-2010-WEKA−Experiences with a Java Open-Source Project

9 0.4093405 96 jmlr-2010-Rate Minimaxity of the Lasso and Dantzig Selector for thelqLoss inlrBalls

10 0.40707645 46 jmlr-2010-High Dimensional Inverse Covariance Matrix Estimation via Linear Programming

11 0.40691125 103 jmlr-2010-Sparse Semi-supervised Learning Using Conjugate Functions

12 0.40467715 24 jmlr-2010-Collective Inference for Extraction MRFs Coupled with Symmetric Clique Potentials

13 0.37124813 97 jmlr-2010-Regret Bounds and Minimax Policies under Partial Monitoring

14 0.36591262 95 jmlr-2010-Rademacher Complexities and Bounding the Excess Risk in Active Learning

15 0.36258954 109 jmlr-2010-Stochastic Composite Likelihood

16 0.35426763 100 jmlr-2010-SFO: A Toolbox for Submodular Function Optimization

17 0.34864238 15 jmlr-2010-Approximate Tree Kernels

18 0.34725362 45 jmlr-2010-High-dimensional Variable Selection with Sparse Random Projections: Measurement Sparsity and Statistical Efficiency

19 0.34714448 39 jmlr-2010-FastInf: An Efficient Approximate Inference Library

20 0.34207296 54 jmlr-2010-Information Retrieval Perspective to Nonlinear Dimensionality Reduction for Data Visualization