nips nips2010 nips2010-113 knowledge-graph by maker-knowledge-mining

113 nips-2010-Heavy-Tailed Process Priors for Selective Shrinkage

Source: pdf

Author: Fabian L. Wauthier, Michael I. Jordan

Abstract: Heavy-tailed distributions are often used to enhance the robustness of regression and classiﬁcation methods to outliers in output space. Often, however, we are confronted with “outliers” in input space, which are isolated observations in sparsely populated regions. We show that heavy-tailed stochastic processes (which we construct from Gaussian processes via a copula), can be used to improve robustness of regression and classiﬁcation estimators to such outliers by selectively shrinking them more strongly in sparse regions than in dense regions. We carry out a theoretical analysis to show that selective shrinkage occurs when the marginals of the heavy-tailed process have sufﬁciently heavy tails. The analysis is complemented by experiments on biological data which indicate signiﬁcant improvements of estimates in sparse regions while producing competitive results in dense regions. 1

Reference: text

Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 edu Abstract Heavy-tailed distributions are often used to enhance the robustness of regression and classiﬁcation methods to outliers in output space. [sent-7, score-0.162]

2 We show that heavy-tailed stochastic processes (which we construct from Gaussian processes via a copula), can be used to improve robustness of regression and classiﬁcation estimators to such outliers by selectively shrinking them more strongly in sparse regions than in dense regions. [sent-9, score-0.655]

3 We carry out a theoretical analysis to show that selective shrinkage occurs when the marginals of the heavy-tailed process have sufﬁciently heavy tails. [sent-10, score-0.541]

4 The analysis is complemented by experiments on biological data which indicate signiﬁcant improvements of estimates in sparse regions while producing competitive results in dense regions. [sent-11, score-0.226]

5 1 Introduction Gaussian process classiﬁers (GPCs) [12] provide a Bayesian approach to nonparametric classiﬁcation with the key advantage of producing predictive class probabilities. [sent-12, score-0.121]

6 In particular, while Nature provides samples of protein conﬁgurations near the global minima of free energy functions, protein-folding algorithms, which imitate Nature by minimizing an estimated energy function, necessarily explore regions far from the minimum. [sent-15, score-0.143]

7 If the estimate of free energy is poor in those sparsely-sampled regions then the algorithm has a poor guide towards the minimum. [sent-16, score-0.082]

8 In this paper we investigate a GPC-based approach that addresses overﬁtting by shrinking predictive class probabilities towards conservative values. [sent-18, score-0.122]

9 For an unevenly sampled input space it is natural to consider a selective shrinkage strategy: we wish to shrink probability estimates more strongly in sparse regions than in dense regions. [sent-19, score-0.664]

10 If sparse regions can be readily identiﬁed, selective shrinkage could be induced by tailoring the Gaussian process (GP) kernel to reﬂect that information. [sent-21, score-0.625]

11 In the absence of such knowledge, Goldberg and Williams [5] showed that Gaussian process regression (GPR) can be augmented with a GP on the log noise level. [sent-22, score-0.09]

12 More recent work has focused on partitioning input space into discrete regions and deﬁning different kernel functions on each. [sent-23, score-0.145]

13 Treed Gaussian process regression [6] and Treed Gaussian process classiﬁcation [1] represent advanced variations of this theme that deﬁne a prior distribution over partitions and their respective kernel hyperparameters. [sent-24, score-0.209]

14 Another line of research which could be adapted to this problem posits that the covariate space is a nonlinear deformation of another space on which a Gaussian process prior is placed [3, 13]. [sent-25, score-0.098]

15 Instead of directly modifying the kernel matrix, the observed non-uniformity of measurements is interpreted as being caused by the spatial deformation. [sent-26, score-0.148]

16 1 This paper shows that selective shrinkage can be more elegantly introduced by replacing the Gaussian process underlying GPC with a stochastic process that has heavy-tailed marginals (e. [sent-28, score-0.567]

17 While heavy-tailed marginals are generally viewed as providing robustness to outliers in the output space (i. [sent-31, score-0.184]

18 , the response space), selective shrinkage can be viewed as a form of robustness to outliers in the input space (i. [sent-33, score-0.461]

19 Indeed, selective shrinkage means the data points that are far from other data points in the input space are regularized more strongly. [sent-36, score-0.366]

20 We provide a theoretical analysis and empirical results to show that inference based on stochastic processes with heavy-tailed marginals yields precisely this kind of shrinkage. [sent-37, score-0.177]

21 The paper is structured as follows: Section 2 provides background on GPCs and highlights how selective shrinkage can arise. [sent-38, score-0.366]

22 We present a construction of heavy-tailed processes in Section 3 and show that inference reduces to standard computations in a Gaussian process. [sent-39, score-0.127]

23 Experiments on biological data in Section 6 demonstrate that heavy-tailed process classiﬁcation substantially outperforms GPC in sparse regions while performing competitively in dense regions. [sent-41, score-0.322]

24 2 Gaussian process classiﬁcation and shrinkage A Gaussian process (GP) [12] is a prior on functions z : X → R deﬁned through a mean function (usually identically zero) and a symmetric positive semideﬁnite kernel k(·, ·). [sent-43, score-0.363]

25 The predictive distribution p(z(x∗ )|X, y, x∗ ) represents a regression on z(x∗ ) with a complicated observation model y|z. [sent-50, score-0.099]

26 (1) is that we could selectively shrink the prediction p(y∗ = 1|X, y, x∗ ) towards a conservative value 1/2 by selectively shrinking p(z(x∗ )|X, y, x∗ ) closer to a point mass at zero. [sent-52, score-0.214]

27 3 Heavy-tailed process priors via the Gaussian copula In this section we construct the heavy-tailed stochastic process by transforming a GP. [sent-53, score-0.275]

28 We deﬁne the heavy-tailed process f (X) with marginal c. [sent-56, score-0.098]

29 of a heavy-tailed density gb with scale parameter b that is symmetric about the origin. [sent-67, score-0.191]

30 Examples include the Laplace, hyperbolic secant and Student-t distribution. [sent-68, score-0.278]

31 The process u(X) has the density of a Gaussian copula [10, 16] and is critical in transferring the correlation structure encoded by K(X, X) from z(X) to f (X). [sent-71, score-0.219]

32 2 z(f (X)) = Φ−1 2 (Gb (f (X))), it is well known [7, 9, 11, 15, 16] that the density of f (X) satisﬁes 0,σ p(f (X)) = 1 i=1 gb (f (xi )) exp − z(f (X)) 2 |K(X, X)/σ 2 |1/2 K(X, X)−1 − I z(f (X)) . [sent-74, score-0.191]

33 σ2 (4) Observe that if K(X, X) = σ 2 I then p(f (X)) = i=1 gb (f (xi )). [sent-75, score-0.191]

34 The predictive distribution p(f (x∗ )|X, f (X), x∗ ) can be interpreted as a Heavy-tailed process regression (HPR). [sent-77, score-0.155]

35 Having deﬁned the heavy-tailed stochastic process in general we now turn to an analysis of its shrinkage properties. [sent-81, score-0.244]

36 4 Selective shrinkage By “selective shrinkage” we mean that the degree of shrinkage applied to a collection of estimators varies across estimators. [sent-82, score-0.407]

37 As motivated in Section 2, we are speciﬁcally interested in selectively shrinking posterior distributions near isolated observations more strongly than in dense regions. [sent-83, score-0.381]

38 This section shows that we can achieve this by changing the form of prior marginals (heavy-tailed instead of Gaussian) and that this induces stronger selective shrinkage than any GPR could induce. [sent-84, score-0.455]

39 Since HPR uses a GP in its construction, which can induce some selective shrinkage on its own, care must be taken to investigate only the additional beneﬁts the transformation G−1 (Φ0,σ2 (·)) has on b shrinkage. [sent-85, score-0.366]

40 For this reason we assume a particular GP prior which leads to a special type of shrinkage in GPR and then check how an HPR model built on top of that GP changes the observed behavior. [sent-86, score-0.188]

41 In this section we provide an idealized analysis that allows us to compare the selective shrinkage obtained by GPR and HPR. [sent-87, score-0.396]

42 Assume that xd = xd , ∀d, d ∈ D, so that we may let (without loss of generality) K(xd , xd ) = ˜ ˜ 1, ∀d = d ∈ D. [sent-96, score-0.573]

43 We also assert that xd = xs ∀d ∈ D and let K(xd , xs ) = K(xs , xd ) = 0 ∀d ∈ D. [sent-97, score-0.652]

44 Denote any distributions computed under the GPR model by pgp (·) and those computed in HPR by php (·). [sent-100, score-0.132]

45 Let y denote a vector of real-valued measurements for a regression task. [sent-103, score-0.119]

46 The posterior distribution of z(xi ) given y, with xi ∈ X, is derived by standard Gaussian computations as 2 pgp (z(xi )|X, y) = N µi , σi ˜ µi = K(xi , X)K(X, X)−1 y 2 ˜ ˜ σi = K(xi , xi ) − K(xi , X)K(X, X)−1 K(X, xi ). [sent-104, score-0.117]

47 To ensure that the posterior distributions agree at the two locations we require µd = µs , which holds if measurements y satisfy y ∈ Ygp ˜ ˜ y| K(xd , X) − K(xs , X) K(X, X)−1 y = 0 = y yd = ys . [sent-106, score-0.231]

48 (5)–(7) HPR inference leads to identical distributions php (z(xd )|X, y ) = php (z(xs )|X, y ) with d ∈ D if measurements y in f -space satisfy y ∈ Yhp ˜ ˜ y | K(xd , X) − K(xs , X) K(X, X)−1 Φ−1 2 (Gb (y )) = 0 0,σ = y = G−1 (Φ0,σ2 (y))|y ∈ Ygp . [sent-109, score-0.266]

49 b 3 −5 −10 (a) 0 x 10 |x| 1 gb (x) = 2b exp − b 0 0 b G−1(Φ(x)) b G−1(Φ(x)) b 0 5 G−1(Φ(x)) 5 5 −5 −10 (b) 0 x 1 gb (x) = 2b sech 10 πx 2b −5 −10 (c) gb (x) = 0 x 10 1 3/2 b 2+(x/b)2 Figure 1: Illustration of G−1 (Φ0,σ2 (x)), for σ 2 = 1. [sent-110, score-0.573]

50 of (a) the Laplace disb tribution (b) the hyperbolic secant distribution (c) a Student-t inspired distribution, all with scale parameter b. [sent-114, score-0.278]

51 b To compare the shrinkage properties of GPR and HPR we analyze select pairs of measurements in Ygp and Yhp . [sent-117, score-0.273]

52 The derivation requires that G−1 (Φ0,σ2 (·)) is strongly concave on (−∞, 0], b strongly convex on [0, +∞) and has gradient > 1 on R. [sent-118, score-0.096]

53 Analyzing such y is relevant, as we are most interested in comparing how multiple reinforcing observations at clustered locations and a single isolated observation are absorbed during inference. [sent-125, score-0.105]

54 y(xd∗ ) y(xd∗ ) (8) Thus HPR inference leads to identical predictive distributions in f -space at the two locations even though the isolated observation y (xs ) has disproportionately larger magnitude than y (xd∗ ), relative to the GPR measurements y(xs ) and y(xd∗ ). [sent-128, score-0.318]

55 As this statement holds for any y ∈ Ygp satisfying our earlier sign requirement, it indicates that HPR systematically shrinks isolated observations more strongly than GPR. [sent-129, score-0.103]

56 Since the second derivative of G−1 (Φ0,σ2 (·)) scales linearly with scale b > 0, b an intuitive connection suggests itself when looking at inequality (8): the heavier the marginal tails, the stronger the inequality and thus the stronger the selective shrinkage effect. [sent-130, score-0.408]

57 The previous derivation exempliﬁes in an idealized setting that HPR leads to improved shrinkage of predictive distributions near isolated observations. [sent-131, score-0.419]

58 More generally, because GPR transforms measurements only linearly, while HPR additionally pre-transforms measurements nonlinearly, our analysis suggests that for any GPR we can ﬁnd an HPR model which leads to stronger selective shrinkage. [sent-132, score-0.348]

59 The result has intuitive parallels to the parametric case: just as 1 -regularization improves shrinkage of parametric estimators, heavy-tailed processes improve shrinkage of nonparametric estimators. [sent-133, score-0.434]

60 We note that although our analysis kept K(X, X) ﬁxed for GPR and HPR, in practice we are free to tune the kernel to yield a desired scale of predictive distributions. [sent-134, score-0.128]

61 5 Heavy-tailed process classiﬁcation The derivation of heavy-tailed process classiﬁcation (HPC) is similar to that of standard multiclass GPC with Laplace approximation in Rasmussen and Williams [12]. [sent-136, score-0.144]

62 For a C-class classiﬁcation problem with n training points we 4 1 1 2 2 C C introduce a vector of nC latent function measurements (f1 , . [sent-139, score-0.085]

63 Each kernel matrix Kc is deﬁned by a (possibly different) symmetric positive semideﬁnite kernel with its own set of parameters. [sent-161, score-0.126]

64 The previous section motivates that improved selective shrinkage will occur in p(f∗ |X, y, x∗ ), provided the prior marginals have sufﬁciently heavy tails. [sent-172, score-0.485]

65 With W = diag (π) − ΠΠ , the gradients are J(z) = diag 2 J(z) = diag df (y − π) − K −1 z dz d2 f diag (y − π) − diag dz 2 df dz W diag df dz − K −1 . [sent-180, score-0.909]

67 5 pi r=1 r=2 r=3 Residue ue id Res Re sid ue pi/2 Rotamer r ∈ {1, 2, 3} O Ψ C Cα N Φ Ψ H 0 N C −pi/2 H H O −pi −pi −pi/2 0 Φ pi/2 pi (b) (a) Figure 2: (a) Schematic of a protein segment. [sent-188, score-0.125]

68 ” (b) Ramachandran plot of 400 (Φ, Ψ) measurements and corresponding rotamers (by shapes/colors) for amino-acid arginine (arg). [sent-191, score-0.164]

69 The dark shading indicates the sparse region we considered in producing results in Figure 3. [sent-192, score-0.098]

70 Progressively lighter shadings indicate how the sparse region was grown to produce Figure 4. [sent-193, score-0.138]

71 Differentiating the marginal likelihood we have z d log q(y|x) = (y − π ) diag ˆ dθ 1 dK tr K −1 2 dθ ˆ df dˆ z dˆ dˆ −1 z z 1 dK −1 − K z + z K −1 ˆ ˆ K z− ˆ dθ dθ 2 dθ 1 − tr 2 2 J(ˆ)−1 z d 2 J(ˆ) z dθ . [sent-201, score-0.185]

72 In addition to optimizing the kernel parameters, it may also be of interest to optimize the scale parameter b of marginals Gb . [sent-203, score-0.152]

73 Suppressing the detailed computations, the derivative of the ˆ marginal log likelihood with respect to b is ˆ dˆ d log q(y|x) df z 1 = (y − π ) ˆ − K −1 z − tr ˆ db db db 2 6 2 J(ˆ)−1 z d 2 J(ˆ) z db . [sent-207, score-0.297]

74 2 0 trp tyr ser phe glu asn leu thr his asp arg cys lys met gln ile val trp tyr ser phe glu asn leu thr his asp arg cys lys met gln ile val (a) (b) Figure 3: Rotamer prediction rates in percent in (a) sparse and (b) dense regions. [sent-220, score-1.329]

75 Both ﬂavors of HPC (hyperbolic secant and Laplace marginals) signiﬁcantly outperform GPC in sparse regions while performing competitively in dense regions. [sent-221, score-0.405]

76 There is a strong dependence between backbone angles (Φ, Ψ) and rotamer values; this is illustrated in the “Ramachandran plot” shown in Figure 2(b), which plots the backbone angles for each rotamer (indicated by the shapes/colors). [sent-224, score-0.68]

77 The dependence is exploited in computational approaches to protein structure prediction, where estimates of rotamer probabilities given backbone angles are used as one term in an energy function that models native protein states as minima of the energy. [sent-225, score-0.462]

78 Poor estimates of rotamer probabilities in sparse regions can derail the prediction procedure. [sent-226, score-0.367]

79 Indeed, sparsity has been a serious problem in state-of-the-art rotamer models based on kernel density estimates (Roland Dunbrack, personal communication). [sent-227, score-0.261]

80 To evaluate our algorithm we consider rotamer-prediction tasks on the 17 amino-acids (out of 20) that have three rotamers at the ﬁrst dihedral angle along the sidechain2 . [sent-229, score-0.119]

81 To ﬁnd good GPC kernel parameters we optimize an 2 -regularized version of the Laplace approximation to the log marginal likelihood reported in Eq. [sent-232, score-0.105]

82 For HPC we let Gb be either the centered Laplace distribution or the hyperbolic secant distribution with scale parameter b. [sent-235, score-0.278]

83 Since good regularization parameters for the objectives are not known a priori we train with and test them on a grid for each of the 17 rotameric residues in ten-fold cross-validation. [sent-239, score-0.128]

84 We evaluate the algorithms on predeﬁned sparse and dense regions in the Ramachandran plot, as indicated by the background shading in Figure 2(b). [sent-242, score-0.266]

85 Across 17 residues the sparse regions usually contained more than 70 measurements (and often more than 150), each of which appears in one of the 10 cross validations. [sent-243, score-0.353]

86 Figure 3 compares the label prediction rates on the dense and sparse 2 Residues alanine and glycine are non-discrete while proline has two rotamers at the ﬁrst dihedral angle. [sent-244, score-0.292]

87 45 155 246 390 618 980 ’Density of test data’ 1554 2463 3906 Figure 4: Average rotamer prediction rate in the sparse region for two ﬂavors of HPC, standard GPC well as CTGP [1] as a function of the average number of points per residue in the sparse region. [sent-258, score-0.391]

88 Averaged over all 17 residues HPC outperforms GPC by 5. [sent-260, score-0.128]

89 With Laplace marginals HPC underperforms GPC on only two residues in sparse regions: by 8. [sent-263, score-0.275]

90 Using hyperbolic secant marginals HPC often improves GPC by more than 10% on sparse regions and degrades by more than 5% only on cysteine (cys) and his. [sent-269, score-0.507]

91 In Figure 4 we show how the average rotamer prediction rate across 17 residues changes for HPC, GPC, as well as CTGP [1] as we grow the sparse region to include more measurements from dense regions. [sent-272, score-0.584]

92 The growth of the sparse region is indicated by progressively lighter shadings in Figure 2(b). [sent-273, score-0.138]

93 Early work by Song [16] used the Gaussian copula to generate complex multivariate distributions by complementing a simple copula form with marginal distributions of choice. [sent-278, score-0.434]

94 Popularity of the Gaussian copula in the ﬁnancial literature is generally credited to Li [8] who used it to model correlation structure for pairs of random variables with known marginals. [sent-279, score-0.163]

95 They demonstrate that posterior distributions can better approximate the true noise distribution if the transformation deﬁning the warped process is learned. [sent-282, score-0.127]

96 Bayesian approaches focusing on estimation of the Gaussian copula covariance matrix for a given dataset are given in [4, 11]. [sent-285, score-0.163]

97 We illustrated heavy-tailed processes as a straightforward extension of GPs and an economical way to improve the robustness of estimators in sparse regions beyond those of GP-based methods. [sent-288, score-0.263]

98 Importantly, because these processes are based on a GP, they inherit many of its favorable computational properties; predictive inference in regression, for instance, is straightforward. [sent-289, score-0.153]

99 In this way the beneﬁts of heavy-tailed processes extend to any GP-based model that struggles with covariate shift. [sent-291, score-0.1]

100 Acknowledgements We thank Roland Dunbrack for helpful discussions and providing access to the rotamer datasets. [sent-292, score-0.198]

similar papers computed by tfidf model

tfidf for this paper:

wordName wordTfidf (topN-words)

[('hpc', 0.396), ('gpc', 0.357), ('hpr', 0.258), ('gpr', 0.198), ('rotamer', 0.198), ('gb', 0.191), ('xd', 0.191), ('shrinkage', 0.188), ('selective', 0.178), ('copula', 0.163), ('laplace', 0.155), ('hyperbolic', 0.139), ('secant', 0.139), ('xs', 0.135), ('residues', 0.128), ('ygp', 0.119), ('gp', 0.091), ('marginals', 0.089), ('dense', 0.086), ('measurements', 0.085), ('regions', 0.082), ('backbone', 0.08), ('rotamers', 0.079), ('gaussian', 0.077), ('diag', 0.076), ('isolated', 0.071), ('treed', 0.07), ('df', 0.067), ('predictive', 0.065), ('selectively', 0.064), ('kernel', 0.063), ('dz', 0.063), ('angles', 0.062), ('protein', 0.061), ('outliers', 0.061), ('ctgp', 0.059), ('cys', 0.059), ('fic', 0.059), ('gln', 0.059), ('gpcs', 0.059), ('php', 0.059), ('yhp', 0.059), ('sparse', 0.058), ('processes', 0.058), ('shrinking', 0.057), ('process', 0.056), ('snelson', 0.052), ('ramachandran', 0.048), ('residue', 0.048), ('db', 0.047), ('covariate', 0.042), ('marginal', 0.042), ('rasmussen', 0.041), ('kc', 0.041), ('yd', 0.041), ('asn', 0.04), ('asp', 0.04), ('competitively', 0.04), ('dihedral', 0.04), ('dunbrack', 0.04), ('glu', 0.04), ('ile', 0.04), ('leu', 0.04), ('lighter', 0.04), ('lys', 0.04), ('pgp', 0.04), ('phe', 0.04), ('shading', 0.04), ('shadings', 0.04), ('thr', 0.04), ('tyr', 0.04), ('unevenly', 0.04), ('nc', 0.039), ('cos', 0.039), ('computations', 0.039), ('posterior', 0.038), ('mode', 0.038), ('jaimungal', 0.035), ('val', 0.035), ('avors', 0.035), ('trp', 0.035), ('robustness', 0.034), ('locations', 0.034), ('regression', 0.034), ('classi', 0.033), ('distributions', 0.033), ('strongly', 0.032), ('ue', 0.032), ('derivation', 0.032), ('estimators', 0.031), ('roland', 0.03), ('ser', 0.03), ('perturbing', 0.03), ('idealized', 0.03), ('heavy', 0.03), ('inference', 0.03), ('williams', 0.029), ('prediction', 0.029), ('fn', 0.029), ('goldberg', 0.028)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.99999958 113 nips-2010-Heavy-Tailed Process Priors for Selective Shrinkage

Author: Fabian L. Wauthier, Michael I. Jordan

2 0.14497584 53 nips-2010-Copula Bayesian Networks

Author: Gal Elidan

Abstract: We present the Copula Bayesian Network model for representing multivariate continuous distributions, while taking advantage of the relative ease of estimating univariate distributions. Using a novel copula-based reparameterization of a conditional density, joined with a graph that encodes independencies, our model offers great ﬂexibility in modeling high-dimensional densities, while maintaining control over the form of the univariate marginals. We demonstrate the advantage of our framework for generalization over standard Bayesian networks as well as tree structured copula models for varied real-life domains that are of substantially higher dimension than those typically considered in the copula literature. 1

3 0.12763101 54 nips-2010-Copula Processes

Author: Andrew Wilson, Zoubin Ghahramani

Abstract: We deﬁne a copula process which describes the dependencies between arbitrarily many random variables independently of their marginal distributions. As an example, we develop a stochastic volatility model, Gaussian Copula Process Volatility (GCPV), to predict the latent standard deviations of a sequence of random variables. To make predictions we use Bayesian inference, with the Laplace approximation, and with Markov chain Monte Carlo as an alternative. We ﬁnd our model can outperform GARCH on simulated and ﬁnancial data. And unlike GARCH, GCPV can easily handle missing data, incorporate covariates other than time, and model a rich class of covariance structures. Imagine measuring the distance of a rocket as it leaves Earth, and wanting to know how these measurements correlate with one another. How much does the value of the measurement at ﬁfteen minutes depend on the measurement at ﬁve minutes? Once we’ve learned this correlation structure, suppose we want to compare it to the dependence between measurements of the rocket’s velocity. To do this, it is convenient to separate dependence from the marginal distributions of our measurements. At any given time, a rocket’s distance from Earth could have a Gamma distribution, while its velocity could have a Gaussian distribution. And separating dependence from marginal distributions is precisely what a copula function does. While copulas have recently become popular, especially in ﬁnancial applications [1, 2], as Nelsen [3] writes, “the study of copulas and the role they play in probability, statistics, and stochastic processes is a subject still in its infancy. There are many open problems. . . ” Typically only bivariate (and recently trivariate) copulas are being used and studied. In our introductory example, we are interested in learning the correlations in different stochastic processes, and comparing them. It would therefore be useful to have a copula process, which can describe the dependencies between arbitrarily many random variables independently of their marginal distributions. We deﬁne such a process. And as an example, we develop a stochastic volatility model, Gaussian Copula Process Volatility (GCPV). In doing so, we provide a Bayesian framework for the learning the marginal distributions and dependency structure of what we call a Gaussian copula process. The volatility of a random variable is its standard deviation. Stochastic volatility models are used to predict the volatilities of a heteroscedastic sequence – a sequence of random variables with different variances, like distance measurements of a rocket as it leaves the Earth. As the rocket gets further away, the variance on the measurements increases. Heteroscedasticity is especially important in econometrics; the returns on equity indices, like the S&P; 500, or on currency exchanges, are heteroscedastic. Indeed, in 2003, Robert Engle won the Nobel Prize in economics “for methods of analyzing economic time series with time-varying volatility”. GARCH [4], a generalized version of Engle’s ARCH, is arguably unsurpassed for predicting the volatility of returns on equity indices and currency exchanges [5, 6, 7]. GCPV can outperform GARCH, and is competitive on ﬁnancial data that especially suits GARCH [8, 9, 10]. Before discussing GCPV, we ﬁrst introduce copulas and the copula process. For a review of Gaussian processes, see Rasmussen and Williams [11]. ∗ † http://mlg.eng.cam.ac.uk/andrew Also at the machine learning department at Carnegie Mellon University. 1 1 Copulas Copulas are important because they separate the dependency structure between random variables from their marginal distributions. Intuitively, we can describe the dependency structure of any multivariate joint distribution H(x1 , . . . , xn ) = P (X1 ≤ x1 , . . . Xn ≤ xn ) through a two step process. First we take each univariate random variable Xi and transform it through its cumulative distribution function (cdf) Fi to get Ui = Fi (Xi ), a uniform random variable. We then express the dependencies between these transformed variables through the n-copula C(u1 , . . . , un ). Formally, an n-copula C : [0, 1]n → [0, 1] is a multivariate cdf with uniform univariate marginals: C(u1 , u2 , . . . , un ) = P (U1 ≤ u1 , U2 ≤ u2 , . . . , Un ≤ un ), where U1 , U2 , . . . , Un are standard uniform random variables. Sklar [12] precisely expressed our intuition in the theorem below. Theorem 1.1. Sklar’s theorem Let H be an n-dimensional distribution function with marginal distribution functions F1 , F2 , . . . , Fn . Then there exists an n-copula C such that for all (x1 , x2 , . . . , xn ) ∈ [−∞, ∞]n , H(x1 , x2 , . . . , xn ) = C(F1 (x1 ), F2 (x2 ), . . . , Fn (xn )) = C(u1 , u2 , . . . , un ). (1) If F1 , F2 , . . . , Fn are all continuous then C is unique; otherwise C is uniquely determined on Range F1 × Range F2 × · · · × Range Fn . Conversely, if C is an n-copula and F1 , F2 , . . . , Fn are distribution functions, then the function H is an n-dimensional distribution function with marginal distribution functions F1 , F2 , . . . , Fn . (−1) As a corollary, if Fi (u) = inf{x : F (x) ≥ u}, the quasi-inverse of Fi , then for all u1 , u2 , . . . , un ∈ [0, 1]n , (−1) C(u1 , u2 , . . . , un ) = H(F1 (−1) (u1 ), F2 (−1) (u2 ), . . . , Fn (un )). (2) In other words, (2) can be used to construct a copula. For example, the bivariate Gaussian copula is deﬁned as C(u, v) = Φρ (Φ−1 (u), Φ−1 (v)), (3) where Φρ is a bivariate Gaussian cdf with correlation coefﬁcient ρ, and Φ is the standard univariate Gaussian cdf. Li [2] popularised the bivariate Gaussian copula, by showing how it could be used to study ﬁnancial risk and default correlation, using credit derivatives as an example. By substituting F (x) for u and G(y) for v in equation (3), we have a bivariate distribution H(x, y), with a Gaussian dependency structure, and marginals F and G. Regardless of F and G, the resulting H(x, y) can still be uniquely expressed as a Gaussian copula, so long as F and G are continuous. It is then a copula itself that captures the underlying dependencies between random variables, regardless of their marginal distributions. For this reason, copulas have been called dependence functions [13, 14]. Nelsen [3] contains an extensive discussion of copulas. 2 Copula Processes Imagine choosing a covariance function, and then drawing a sample function at some ﬁnite number of points from a Gaussian process. The result is a sample from a collection of Gaussian random variables, with a dependency structure encoded by the speciﬁed covariance function. Now, suppose we transform each of these values through a univariate Gaussian cdf, such that we have a sample from a collection of uniform random variables. These uniform random variables also have this underlying Gaussian process dependency structure. One might call the resulting values a draw from a Gaussian-Uniform Process. We could subsequently put these values through an inverse beta cdf, to obtain a draw from what could be called a Gaussian-Beta Process: the values would be a sample from beta random variables, again with an underlying Gaussian process dependency structure. We could also transform the uniform values with different inverse cdfs, which would give a sample from different random variables, with dependencies encoded by the Gaussian process. The above procedure is a means to generate samples from arbitrarily many random variables, with arbitrary marginal distributions, and desired dependencies. It is an example of how to use what we call a copula process – in this case, a Gaussian copula process, since a Gaussian copula describes the dependency structure of a ﬁnite number of samples. We now formally deﬁne a copula process. 2 Deﬁnition 2.1. Copula Process Let {Wt } be a collection of random variables indexed by t ∈ T , with marginal distribution functions Ft , and let Qt = Ft (Wt ). Further, let µ be a stochastic process measure with marginal distribution functions Gt , and joint distribution function H. Then Wt is copula process distributed with base measure µ, or Wt ∼ CP(µ), if and only if for all n ∈ N, ai ∈ R, n P( (−1) {Gti (Qti ) ≤ ai }) = Ht1 ,t2 ,...,tn (a1 , a2 , . . . , an ). (4) i=1 (−1) Each Qti ∼ Uniform(0, 1), and Gti is the quasi-inverse of Gti , as previously deﬁned. Deﬁnition 2.2. Gaussian Copula Process Wt is Gaussian copula process distributed if it is copula process distributed and the base measure µ is a Gaussian process. If there is a mapping Ψ such that Ψ(Wt ) ∼ GP(m(t), k(t, t )), then we write Wt ∼ GCP(Ψ, m(t), k(t, t )). For example, if we have Wt ∼ GCP with m(t) = 0 and k(t, t) = 1, then in the deﬁnition of a copula process, Gt = Φ, the standard univariate Gaussian cdf, and H is the usual GP joint distribution function. Supposing this GCP is a Gaussian-Beta process, then Ψ = Φ−1 ◦ FB , where FB is a univariate Beta cdf. One could similarly deﬁne other copula processes. We described generally how a copula process can be used to generate samples of arbitrarily many random variables with desired marginals and dependencies. We now develop a speciﬁc and practical application of this framework. We introduce a stochastic volatility model, Gaussian Copula Process Volatility (GCPV), as an example of how to learn the joint distribution of arbitrarily many random variables, the marginals of these random variables, and to make predictions. To do this, we ﬁt a Gaussian copula process by using a type of Warped Gaussian Process [15]. However, our methodology varies substantially from Snelson et al. [15], since we are doing inference on latent variables as opposed to observations, which is a much greater undertaking that involves approximations, and we are doing so in a different context. 3 Gaussian Copula Process Volatility Assume we have a sequence of observations y = (y1 , . . . , yn ) at times t = (t1 , . . . , tn ) . The observations are random variables with different latent standard deviations. We therefore have n unobserved standard deviations, σ1 , . . . , σn , and want to learn the correlation structure between these standard deviations, and also to predict the distribution of σ∗ at some unrealised time t∗ . We model the standard deviation function as a Gaussian copula process: σt ∼ GCP(g −1 , 0, k(t, t )). (5) f (t) ∼ GP(m(t) = 0, k(t, t )) σ(t) = g(f (t), ω) (6) (7) y(t) ∼ N (0, σ 2 (t)), (8) Speciﬁcally, where g is a monotonic warping function, parametrized by ω. For each of the observations y = (y1 , . . . , yn ) we have corresponding GP latent function values f = (f1 , . . . , fn ) , where σ(ti ) = g(fi , ω), using the shorthand fi to mean f (ti ). σt ∼ GCP, because any ﬁnite sequence (σ1 , . . . , σp ) is distributed as a Gaussian copula: P (σ1 ≤ a1 , . . . , σp ≤ ap ) = P (g −1 (σ1 ) ≤ g −1 (a1 ), . . . , g −1 (σp ) ≤ g −1 (ap )) = ΦΓ (g −1 (a1 ), . . . , g −1 −1 (ap )) = ΦΓ (Φ = ΦΓ (Φ −1 (F (a1 )), . . . , Φ (u1 ), . . . , Φ −1 −1 (9) (F (ap ))) (up )) = C(u1 , . . . , up ), where Φ is the standard univariate Gaussian cdf (supposing k(t, t) = 1), ΦΓ is a multivariate Gaussian cdf with covariance matrix Γij = cov(g −1 (σi ), g −1 (σj )), and F is the marginal distribution of 3 each σi . In (5), we have Ψ = g −1 , because it is g −1 which maps σt to a GP. The speciﬁcation in (5) is equivalently expressed by (6) and (7). With GCPV, the form of g is learned so that g −1 (σt ) is best modelled by a GP. By learning g, we learn the marginal of each σ: F (a) = Φ(g −1 (a)) for a ∈ R. Recently, a different sort of ‘kernel copula process’ has been used, where the marginals of the variables being modelled are not learned [16].1 Further, we also consider a more subtle and ﬂexible form of our model, where the function g itself is indexed by time: g = gt (f (t), ω). We only assume that the marginal distributions of σt are stationary over ‘small’ time periods, and for each of these time periods (5)-(7) hold true. We return to this in the ﬁnal discussion section. Here we have assumed that each observation, conditioned on knowing its variance, is normally distributed with zero mean. This is a common assumption in heteroscedastic models. The zero mean and normality assumptions can be relaxed and are not central to this paper. 4 Predictions with GCPV Ultimately, we wish to infer p(σ(t∗ )|y, z), where z = {θ, ω}, and θ are the hyperparameters of the GP covariance function. To do this, we sample from p(f∗ |y, z) = p(f∗ |f , θ)p(f |y, z)df (10) and then transform these samples by g. Letting (Cf )ij = δij g(fi , ω)2 , where δij is the Kronecker delta, Kij = k(ti , tj ), (k∗ )i = k(t∗ , ti ), we have p(f |y, z) = N (f ; 0, K)N (y; 0, Cf )/p(y|z), p(f∗ |f , θ) = N (k∗ K −1 f , k(t∗ , t∗ ) − k∗ K −1 (11) k∗ ). (12) We also wish to learn z, which we can do by ﬁnding the z that maximizes the marginal likelihood, ˆ p(y|z) = p(y|f , ω)p(f |θ)df . (13) Unfortunately, for many functions g, (10) and (13) are intractable. Our methods of dealing with this can be used in very general circumstances, where one has a Gaussian process prior, but an (optionally parametrized) non-Gaussian likelihood. We use the Laplace approximation to estimate p(f |y, z) as a Gaussian. Then we can integrate (10) for a Gaussian approximation to p(f∗ |y, z), which we sample from to make predictions of σ∗ . Using Laplace, we can also ﬁnd an expression for an approximate marginal likelihood, which we maximize to determine z. Once we have found z with Laplace, we use Markov chain Monte Carlo to sample from p(f∗ |y, z), and compare that to using Laplace to sample from p(f∗ |y, z). In the supplement we relate this discussion to (9). 4.1 Laplace Approximation The goal is to approximate (11) with a Gaussian, so that we can evaluate (10) and (13) and make predictions. In doing so, we follow Rasmussen and Williams [11] in their treatment of Gaussian process classiﬁcation, except we use a parametrized likelihood, and modify Newton’s method. First, consider as an objective function the logarithm of an unnormalized (11): s(f |y, z) = log p(y|f , ω) + log p(f |θ). (14) ˆ The Laplace approximation uses a second order Taylor expansion about the f which maximizes ˆ, for which we use (14), to ﬁnd an approximate objective s(f |y, z). So the ﬁrst step is to ﬁnd f ˜ Newton’s method. The Newton update is f new = f − ( s(f ))−1 s(f ). Differentiating (14), s(f |y, z) = s(f |y, z) = where W is the diagonal matrix − 1 log p(y|f , ω) − K −1 f log p(y|f , ω) − K −1 = −W − K −1 , log p(y|f , ω). Note added in proof : Also, for a very recent related model, see Rodr´guez et al. [17]. ı 4 (15) (16) If the likelihood function p(y|f , ω) is not log concave, then W may have negative entries. Vanhatalo et al. [18] found this to be problematic when doing Gaussian process regression with a Student-t ˆ likelihood. They instead use an expectation-maximization (EM) algorithm for ﬁnding f , and iterate ordered rank one Cholesky updates to evaluate the Laplace approximate marginal likelihood. But EM can converge slowly, especially near a local optimum, and each of the rank one updates is vulnerable to numerical instability. With a small modiﬁcation of Newton’s method, we often get close to ˆ quadratic convergence for ﬁnding f , and can evaluate the Laplace approximate marginal likelihood in a numerically stable fashion, with no approximate Cholesky factors, and optimal computational requirements. Some comments are in the supplementary material but, in short, we use an approximate negative Hessian, − s ≈ M + K −1 , which is guaranteed to be positive deﬁnite, since M is formed on each iteration by zeroing the negative entries of W . For stability, we reformulate our 1 1 1 1 optimization in terms of B = I + M 2 KM 2 , and let Q = M 2 B −1 M 2 , b = M f + log p(y|f ), a = b − QKb. Since (K −1 + M )−1 = K − KQK, the Newton update becomes f new = Ka. ˆ With these updates we ﬁnd f and get an expression for s which we use to approximate (13) and ˜ (11). The approximate marginal likelihood q(y|z) is given by exp(˜)df . Taking its logarithm, s 1ˆ 1 ˆ log q(y|z) = − f af + log p(y|f ) − log |Bf |, (17) ˆ ˆ 2 2 ˆ ˆ where Bf is B evaluated at f , and af is a numerically stable evaluation of K −1 f . ˆ ˆ To learn the parameters z, we use conjugate gradient descent to maximize (17) with respect to z. ˆ ˆ Since f is a function of z, we initialize z, and update f every time we vary z. Once we have found an optimum z , we can make predictions. By exponentiating s, we ﬁnd a Gaussian approximation to ˆ ˜ ˆ the posterior (11), q(f |y, z) = N (f , K − KQK). The product of this approximate posterior with p(f∗ |f ) is Gaussian. Integrating this product, we approximate p(f∗ |y, z) as ˆ q(f∗ |y, z) = N (k∗ log p(y|f ), k(t∗ , t∗ ) − k∗ Qk∗ ). (18) Given n training observations, the cost of each Newton iteration is dominated by computing the cholesky decomposition of B, which takes O(n3 ) operations. The objective function typically changes by less than 10−6 after 3 iterations. Once Newton’s method has converged, it takes only O(1) operations to draw from q(f∗ |y, z) and make predictions. 4.2 Markov chain Monte Carlo We use Markov chain Monte Carlo (MCMC) to sample from (11), so that we can later sample from p(σ∗ |y, z) to make predictions. Sampling from (11) is difﬁcult, because the variables f are strongly coupled by a Gaussian process prior. We use a new technique, Elliptical Slice Sampling [19], and ﬁnd it extremely effective for this purpose. It was speciﬁcally designed to sample from posteriors with correlated Gaussian priors. It has no free parameters, and jointly updates every element of f . For our setting, it is over 100 times as fast as axis aligned slice sampling with univariate updates. To make predictions, we take J samples of p(f |y, z), {f 1 , . . . , f J }, and then approximate (10) as a mixture of J Gaussians: J 1 p(f∗ |f i , θ). (19) p(f∗ |y, z) ≈ J i=1 Each of the Gaussians in this mixture have equal weight. So for each sample of f∗ |y, we uniformly choose a random p(f∗ |f i , θ) and draw a sample. In the limit J → ∞, we are sampling from the exact p(f∗ |y, z). Mapping these samples through g gives samples from p(σ∗ |y, z). After one O(n3 ) and one O(J) operation, a draw from (19) takes O(1) operations. 4.3 Warping Function The warping function, g, maps fi , a GP function value, to σi , a standard deviation. Since fi can take any value in R, and σi can take any non-negative real value, g : R → R+ . For each fi to correspond to a unique deviation, g must also be one-to-one. We use K g(x, ω) = aj log[exp[bj (x + cj )] + 1], j=1 5 aj , bj > 0. (20) This is monotonic, positive, inﬁnitely differentiable, asymptotic towards zero as x → −∞, and K tends to ( j=1 aj bj )x as x → ∞. In practice, it is useful to add a small constant to (20), to avoid rare situations where the parameters ω are trained to make g extremely small for certain inputs, at the expense of a good overall ﬁt; this can happen when the parameters ω are learned by optimizing a likelihood. A suitable constant could be one tenth the absolute value of the smallest nonzero observation. By inferring the parameters of the warping function, or distributions of these parameters, we are learning a transformation which will best model σt with a Gaussian process. The more ﬂexible the warping function, the more potential there is to improve the GCPV ﬁt – in other words, the better we can estimate the ‘perfect’ transformation. To test the importance of this ﬂexibility, we also try a simple unparametrized warping function, g(x) = ex . In related work, Goldberg et al. [20] place a GP prior on the log noise level in a standard GP regression model on observations, except for inference they use Gibbs sampling, and a high level of ‘jitter’ for conditioning. Once g is trained, we can infer the marginal distribution of each σ: F (a) = Φ(g −1 (a)), for a ∈ R. This suggests an alternate way to initialize g: we can initialize F as a mixture of Gaussians, and then map through Φ−1 to ﬁnd g −1 . Since mixtures of Gaussians are dense in the set of probability distributions, we could in principle ﬁnd the ‘perfect’ g using an inﬁnite mixture of Gaussians [21]. 5 Experiments In our experiments, we predict the latent standard deviations σ of observations y at times t, and also σ∗ at unobserved times t∗ . To do this, we use two versions of GCPV. The ﬁrst variant, which we simply refer to as GCPV, uses the warping function (20) with K = 1, and squared exponential covariance function, k(t, t ) = A exp(−(t−t )2 /l2 ), with A = 1. The second variant, which we call GP-EXP, uses the unparametrized warping function ex , and the same covariance function, except the amplitude A is a trained hyperparameter. The other hyperparameter l is called the lengthscale of the covariance function. The greater l, the greater the covariance between σt and σt+a for a ∈ R. We train hyperparameters by maximizing the Laplace approximate log marginal likelihood (17). We then sample from p(f∗ |y) using the Laplace approximation (18). We also do this using MCMC (19) with J = 10000, after discarding a previous 10000 samples of p(f |y) as burn-in. We pass 2 these samples of f∗ |y through g and g 2 to draw from p(σ∗ |y) and p(σ∗ |y), and compute the sample mean and variance of σ∗ |y. We use the sample mean as a point predictor, and the sample variance for error bounds on these predictions, and we use 10000 samples to compute these quantities. For GCPV we use Laplace and MCMC for inference, but for GP-EXP we only use Laplace. We compare predictions to GARCH(1,1), which has been shown in extensive and recent reviews to be competitive with other GARCH variants, and more sophisticated models [5, 6, 7]. GARCH(p,q) speciﬁes y(t) ∼ p 2 2 N (0, σ 2 (t)), and lets the variance be a deterministic function of the past: σt = a0 + i=1 ai yt−i + q 2 j=1 bj σt−j . We use the Matlab Econometrics Toolbox implementation of GARCH, where the parameters a0 , ai and bj are estimated using a constrained maximum likelihood. We make forecasts of volatility, and we predict historical volatility. By ‘historical volatility’ we mean the volatility at observed time points, or between these points. Uncovering historical volatility is important. It could, for instance, be used to study what causes ﬂuctuations in the stock market, or to understand physical systems. To evaluate our model, we use the Mean Squared Error (MSE) between the true variance, or proxy for the truth, and the predicted variance. Although likelihood has advantages, we are limited in space, and we wish to harmonize with the econometrics literature, and other assessments of volatility models, where MSE is the standard. In a similar assessment of volatility models, Brownlees et al. [7] found that MSE and quasi-likelihood rankings were comparable. When the true variance is unknown we follow Brownlees et al. [7] and use squared observations as a proxy for the truth, to compare our model to GARCH.2 The more observations, the more reliable these performance estimates will be. However, not many observations (e.g. 100) are needed for a stable ranking of competing models; in Brownlees et al. [7], the rankings derived from high frequency squared observations are similar to those derived using daily squared observations. 2 Since each observation y is assumed to have zero mean and variance σ 2 , E[y 2 ] = σ 2 . 6 5.1 Simulations We simulate observations from N (0, σ 2 (t)), using σ(t) = sin(t) cos(t2 ) + 1, at t = (0, 0.02, 0.04, . . . , 4) . We call this data set TRIG. We also simulate using a standard deviation that jumps from 0.1 to 7 and back, at times t = (0, 0.1, 0.2, . . . , 6) . We call this data set JUMP. To forecast, we use all observations up until the current time point, and make 1, 7, and 30 step ahead predictions. So, for example, in TRIG we start by observing t = 0, and make forecasts at t = 0.02, 0.14, 0.60. Then we observe t = 0, 0.02 and make forecasts at t = 0.04, 0.16, 0.62, and so on, until all data points have been observed. For historical volatility, we predict the latent σt at the observation times, which is safe since we are comparing to the true volatility, which is not used in training; the results are similar if we interpolate. Figure 1 panels a) and b) show the true volatility for TRIG and JUMP respectively, alongside GCPV Laplace, GCPV MCMC, GP-EXP Laplace, and GARCH(1,1) predictions of historical volatility. Table 1 shows the results for forecasting and historical volatility. In panel a) we see that GCPV more accurately captures the dependencies between σ at different times points than GARCH: if we manually decrease the lengthscale in the GCPV covariance function, we can replicate the erratic GARCH behaviour, which inaccurately suggests that the covariance between σt and σt+a decreases quickly with increases in a. We also see that GCPV with an unparametrized exponential warping function tends to overestimates peaks and underestimate troughs. In panel b), the volatility is extremely difﬁcult to reconstruct or forecast – with no warning it will immediately and dramatically increase or decrease. This behaviour is not suited to a smooth squared exponential covariance function. Nevertheless, GCPV outperforms GARCH, especially in regions of low volatility. We also see this in panel a) for t ∈ (1.5, 2). GARCH is known to respond slowly to large returns, and to overpredict volatility [22]. In JUMP, the greater the peaks, and the smaller the troughs, the more GARCH suffers, while GCPV is mostly robust to these changes. 5.2 Financial Data The returns on the daily exchange rate between the Deutschmark (DM) and the Great Britain Pound (GBP) from 1984 to 1992 have become a benchmark for assessing the performance of GARCH models [8, 9, 10]. This exchange data, which we refer to as DMGBP, can be obtained from www.datastream.com, and the returns are calculated as rt = log(Pt+1 /Pt ), where Pt is the number of DM to GBP on day t. The returns are assumed to have a zero mean function. We use a rolling window of the previous 120 days of returns to make 1, 7, and 30 day ahead volatility forecasts, starting at the beginning of January 1988, and ending at the beginning of January 1992 (659 trading days). Every 7 days, we retrain the parameters of GCPV and GARCH. Every time we retrain parameters, we predict historical volatility over the past 120 days. The average MSE for these historical predictions is given in Table 1, although they should be observed with caution; unlike with the simulations, the DMGBP historical predictions are trained using the same data they are assessed on. In Figure 1c), we see that the GARCH one day ahead forecasts are lifted above the GCPV forecasts, but unlike in the simulations, they are now operating on a similar lengthscale. This suggests that GARCH could still be overpredicting volatility, but that GCPV has adapted its estimation of how σt and σt+a correlate with one another. Since GARCH is suited to this ﬁnancial data set, it is reassuring that GCPV predictions have a similar time varying structure. Overall, GCPV and GARCH are competitive with one another for forecasting currency exchange returns, as seen in Table 1. Moreover, a learned warping function g outperforms an unparametrized one, and a full Laplace solution is comparable to using MCMC for inference, in accuracy and speed. This is also true for the simulations. Therefore we recommend whichever is more convenient to implement. 6 Discussion We deﬁned a copula process, and as an example, developed a stochastic volatility model, GCPV, which can outperform GARCH. With GCPV, the volatility σt is distributed as a Gaussian Copula Process, which separates the modelling of the dependencies between volatilities at different times from their marginal distributions – arguably the most useful property of a copula. Further, GCPV ﬁts the marginals in the Gaussian copula process by learning a warping function. If we had simply chosen an unparametrized exponential warping function, we would incorrectly be assuming that the log 7 Table 1: MSE for predicting volatility. Data set Model Historical 1 step 7 step 30 step TRIG GCPV (LA) GCPV (MCMC) GP-EXP GARCH 0.0953 0.0760 0.193 0.938 0.588 0.622 0.646 1.04 0.951 0.979 1.36 1.79 1.71 1.76 1.15 5.12 JUMP GCPV (LA) GCPV (MCMC) GP-EXP GARCH 0.588 1.21 1.43 1.88 0.891 0.951 1.76 1.58 1.38 1.37 6.95 3.43 1.35 1.35 14.7 5.65 GCPV (LA) GCPV (MCMC) GP-EXP GARCH 2.43 2.39 2.52 2.83 3.00 3.00 3.20 3.03 3.08 3.08 3.46 3.12 3.17 3.17 5.14 3.32 ×103 DMGBP ×10−9 TRIG JUMP DMGBP 20 DMGBP 0.015 600 Probability Density 3 1 Volatility Volatility Volatility 15 2 10 0.01 0.005 5 0 0 1 2 Time (a) 3 4 0 0 2 4 0 6 Time (b) 0 200 400 Days (c) 600 400 200 0 0 0.005 σ (d) 0.01 Figure 1: Predicting volatility and learning its marginal pdf. For a) and b), the true volatility, and GCPV (MCMC), GCPV (LA), GP-EXP, and GARCH predictions, are shown respectively by a thick green line, a dashed thick blue line, a dashed black line, a cyan line, and a red line. a) shows predictions of historical volatility for TRIG, where the shade is a 95% conﬁdence interval about GCPV (MCMC) predictions. b) shows predictions of historical volatility for JUMP. In c), a black line and a dashed red line respectively show GCPV (LA) and GARCH one day ahead volatility forecasts for DMGBP. In d), a black line and a dashed blue line respectively show the GCPV learned marginal pdf of σt in DMGBP and a Gamma(4.15,0.00045) pdf. volatilities are marginally Gaussian distributed. Indeed, for the DMGBP data, we trained the warping function g over a 120 day period, and mapped its inverse through the univariate standard Gaussian cdf Φ, and differenced, to estimate the marginal probability density function (pdf) of σt over this period. The learned marginal pdf, shown in Figure 1d), is similar to a Gamma(4.15,0.00045) distribution. However, in using a rolling window to retrain the parameters of g, we do not assume that the marginals of σt are stationary; we have a time changing warping function. While GARCH is successful, and its simplicity is attractive, our model is also simple and has a number of advantages. We can effortlessly handle missing data, we can easily incorporate covariates other than time (like interest rates) in our covariance function, and we can choose from a rich class of covariance functions – squared exponential, Brownian motion, Mat´ rn, periodic, etc. In fact, the e volatility of high frequency intradaily returns on equity indices and currency exchanges is cyclical [23], and GCPV with a periodic covariance function is uniquely well suited to this data. And the parameters of GCPV, like the covariance function lengthscale, or the learned warping function, provide insight into the underlying source of volatility, unlike the parameters of GARCH. Finally, copulas are rapidly becoming popular in applications, but often only bivariate copulas are being used. With our copula process one can learn the dependencies between arbitrarily many random variables independently of their marginal distributions. We hope the Gaussian Copula Process Volatility model will encourage other applications of copula processes. More generally, we hope our work will help bring together the machine learning and econometrics communities. Acknowledgments: Thanks to Carl Edward Rasmussen and Ferenc Husz´ r for helpful conversaa tions. AGW is supported by an NSERC grant. 8 References [1] Paul Embrechts, Alexander McNeil, and Daniel Straumann. Correlation and dependence in risk management: Properties and pitfalls. In Risk Management: Value at risk and beyond, pages 176–223. Cambridge University Press, 1999. [2] David X. Li. On default correlation: A copula function approach. Journal of Fixed Income, 9(4):43–54, 2000. [3] Roger B. Nelsen. An Introduction to Copulas. Springer Series in Statistics, second edition, 2006. [4] Tim Bollerslev. Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics, 31 (3):307–327, 1986. [5] Ser-Huang Poon and Clive W.J. Granger. Practical issues in forecasting volatility. Financial Analysts Journal, 61(1):45–56, 2005. [6] Peter Reinhard Hansen and Asger Lunde. A forecast comparison of volatility models: Does anything beat a GARCH(1,1). Journal of Applied Econometrics, 20(7):873–889, 2005. [7] Christian T. Brownlees, Robert F. Engle, and Bryan T. Kelly. A practical guide to volatility forecasting through calm and storm, 2009. Available at SSRN: http://ssrn.com/abstract=1502915. [8] T. Bollerslev and E. Ghysels. Periodic autoregressive conditional heteroscedasticity. Journal of Business and Economic Statistics, 14:139–151, 1996. [9] B.D. McCullough and C.G. Renfro. Benchmarks and software standards: A case study of GARCH procedures. Journal of Economic and Social Measurement, 25:59–71, 1998. [10] C. Brooks, S.P. Burke, and G. Persand. Benchmarks and the accuracy of GARCH model estimation. International Journal of Forecasting, 17:45–56, 2001. [11] Carl Edward Rasmussen and Christopher K.I. Williams. Gaussian processes for Machine Learning. The MIT Press, 2006. ` [12] Abe Sklar. Fonctions de r´ partition a n dimensions et leurs marges. Publ. Inst. Statist. Univ. Paris, 8: e 229–231, 1959. [13] P Deheuvels. Caract´ isation compl` te des lois extrˆ mes multivari´ s et de la convergence des types e e e e extrˆ mes. Publications de l’Institut de Statistique de l’Universit´ de Paris, 23:1–36, 1978. e e [14] G Kimeldorf and A Sampson. Uniform representations of bivariate distributions. Communications in Statistics, 4:617–627, 1982. [15] Edward Snelson, Carl Edward Rasmussen, and Zoubin Ghahramani. Warped Gaussian Processes. In NIPS, 2003. [16] Sebastian Jaimungal and Eddie K.H. Ng. Kernel-based Copula processes. In ECML PKDD, 2009. [17] A. Rodr´guez, D.B. Dunson, and A.E. Gelfand. Latent stick-breaking processes. Journal of the American ı Statistical Association, 105(490):647–659, 2010. [18] Jarno Vanhatalo, Pasi Jylanki, and Aki Vehtari. Gaussian process regression with Student-t likelihood. In NIPS, 2009. [19] Iain Murray, Ryan Prescott Adams, and David J.C. MacKay. Elliptical Slice Sampling. In AISTATS, 2010. [20] Paul W. Goldberg, Christopher K.I. Williams, and Christopher M. Bishop. Regression with inputdependent noise: A Gaussian process treatment. In NIPS, 1998. [21] Carl Edward Rasmussen. The Inﬁnite Gaussian Mixture Model. In NIPS, 2000. [22] Ruey S. Tsay. Analysis of Financial Time Series. John Wiley & Sons, 2002. [23] Torben G. Andersen and Tim Bollerslev. Intraday periodicity and volatility persistence in ﬁnancial markets. Journal of Empirical Finance, 4(2-3):115–158, 1997. 9

4 0.086280085 101 nips-2010-Gaussian sampling by local perturbations

Author: George Papandreou, Alan L. Yuille

Abstract: We present a technique for exact simulation of Gaussian Markov random ﬁelds (GMRFs), which can be interpreted as locally injecting noise to each Gaussian factor independently, followed by computing the mean/mode of the perturbed GMRF. Coupled with standard iterative techniques for the solution of symmetric positive deﬁnite systems, this yields a very efﬁcient sampling algorithm with essentially linear complexity in terms of speed and memory requirements, well suited to extremely large scale probabilistic models. Apart from synthesizing data under a Gaussian model, the proposed technique directly leads to an efﬁcient unbiased estimator of marginal variances. Beyond Gaussian models, the proposed algorithm is also very useful for handling highly non-Gaussian continuously-valued MRFs such as those arising in statistical image modeling or in the ﬁrst layer of deep belief networks describing real-valued data, where the non-quadratic potentials coupling different sites can be represented as ﬁnite or inﬁnite mixtures of Gaussians with the help of local or distributed latent mixture assignment variables. The Bayesian treatment of such models most naturally involves a block Gibbs sampler which alternately draws samples of the conditionally independent latent mixture assignments and the conditionally multivariate Gaussian continuous vector and we show that it can directly beneﬁt from the proposed methods. 1

5 0.081179924 33 nips-2010-Approximate inference in continuous time Gaussian-Jump processes

Author: Manfred Opper, Andreas Ruttor, Guido Sanguinetti

Abstract: We present a novel approach to inference in conditionally Gaussian continuous time stochastic processes, where the latent process is a Markovian jump process. We ﬁrst consider the case of jump-diffusion processes, where the drift of a linear stochastic differential equation can jump at arbitrary time points. We derive partial differential equations for exact inference and present a very efﬁcient mean ﬁeld approximation. By introducing a novel lower bound on the free energy, we then generalise our approach to Gaussian processes with arbitrary covariance, such as the non-Markovian RBF covariance. We present results on both simulated and real data, showing that the approach is very accurate in capturing latent dynamics and can be useful in a number of real data modelling tasks.

6 0.073256925 80 nips-2010-Estimation of Renyi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs

7 0.060620811 233 nips-2010-Scrambled Objects for Least-Squares Regression

8 0.059237514 242 nips-2010-Slice sampling covariance hyperparameters of latent Gaussian models

9 0.055495545 85 nips-2010-Exact learning curves for Gaussian process regression on large random graphs

10 0.053727999 118 nips-2010-Implicit Differentiation by Perturbation

11 0.051780976 225 nips-2010-Relaxed Clipping: A Global Training Method for Robust Regression and Classification

12 0.051225018 100 nips-2010-Gaussian Process Preference Elicitation

13 0.048403174 133 nips-2010-Kernel Descriptors for Visual Recognition

14 0.046786655 280 nips-2010-Unsupervised Kernel Dimension Reduction

15 0.046484016 218 nips-2010-Probabilistic latent variable models for distinguishing between cause and effect

16 0.046357218 288 nips-2010-Worst-case bounds on the quality of max-product fixed-points

17 0.044327099 49 nips-2010-Computing Marginal Distributions over Continuous Markov Networks for Statistical Relational Learning

18 0.044243954 276 nips-2010-Tree-Structured Stick Breaking for Hierarchical Data

19 0.044221282 174 nips-2010-Multi-label Multiple Kernel Learning by Stochastic Approximation: Application to Visual Object Recognition

20 0.04365417 249 nips-2010-Spatial and anatomical regularization of SVM for brain image analysis

similar papers computed by lsi model

lsi for this paper:

topicId topicWeight

[(0, 0.15), (1, 0.041), (2, 0.002), (3, 0.038), (4, -0.007), (5, -0.002), (6, 0.067), (7, 0.014), (8, -0.03), (9, -0.025), (10, -0.116), (11, -0.042), (12, -0.023), (13, -0.046), (14, -0.037), (15, 0.043), (16, 0.0), (17, 0.075), (18, 0.098), (19, 0.063), (20, -0.104), (21, -0.021), (22, -0.064), (23, -0.068), (24, -0.071), (25, -0.12), (26, 0.13), (27, -0.01), (28, 0.045), (29, 0.01), (30, 0.047), (31, -0.015), (32, 0.092), (33, 0.147), (34, -0.02), (35, -0.041), (36, 0.03), (37, 0.086), (38, 0.095), (39, 0.01), (40, 0.09), (41, 0.013), (42, -0.054), (43, -0.027), (44, -0.05), (45, -0.059), (46, 0.119), (47, -0.035), (48, 0.014), (49, 0.039)]

similar papers list:

simIndex simValue paperId paperTitle

1 0.89060009 54 nips-2010-Copula Processes

Author: Andrew Wilson, Zoubin Ghahramani

same-paper 2 0.88395435 113 nips-2010-Heavy-Tailed Process Priors for Selective Shrinkage

Author: Fabian L. Wauthier, Michael I. Jordan

3 0.83760267 53 nips-2010-Copula Bayesian Networks

Author: Gal Elidan

4 0.4985289 80 nips-2010-Estimation of Renyi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs

Author: Barnabás Póczos, Csaba Szepesvári, David Tax

Abstract: We present simple and computationally efﬁcient nonparametric estimators of R´ nyi entropy and mutual information based on an i.i.d. sample drawn from an e unknown, absolutely continuous distribution over Rd . The estimators are calculated as the sum of p-th powers of the Euclidean lengths of the edges of the ‘generalized nearest-neighbor’ graph of the sample and the empirical copula of the sample respectively. For the ﬁrst time, we prove the almost sure consistency of these estimators and upper bounds on their rates of convergence, the latter of which under the assumption that the density underlying the sample is Lipschitz continuous. Experiments demonstrate their usefulness in independent subspace analysis. 1

5 0.48830098 154 nips-2010-Learning sparse dynamic linear systems using stable spline kernels and exponential hyperpriors

Author: Alessandro Chiuso, Gianluigi Pillonetto

Abstract: We introduce a new Bayesian nonparametric approach to identiﬁcation of sparse dynamic linear systems. The impulse responses are modeled as Gaussian processes whose autocovariances encode the BIBO stability constraint, as deﬁned by the recently introduced “Stable Spline kernel”. Sparse solutions are obtained by placing exponential hyperpriors on the scale factors of such kernels. Numerical experiments regarding estimation of ARMAX models show that this technique provides a deﬁnite advantage over a group LAR algorithm and state-of-the-art parametric identiﬁcation techniques based on prediction error minimization. 1

6 0.46683109 82 nips-2010-Evaluation of Rarity of Fingerprints in Forensics

7 0.46581769 233 nips-2010-Scrambled Objects for Least-Squares Regression

8 0.46321991 126 nips-2010-Inference with Multivariate Heavy-Tails in Linear Models

9 0.44531199 84 nips-2010-Exact inference and learning for cumulative distribution functions on loopy graphs

10 0.44377905 33 nips-2010-Approximate inference in continuous time Gaussian-Jump processes

11 0.43976581 242 nips-2010-Slice sampling covariance hyperparameters of latent Gaussian models

12 0.43834811 101 nips-2010-Gaussian sampling by local perturbations

13 0.4382239 262 nips-2010-Switched Latent Force Models for Movement Segmentation

14 0.41122782 118 nips-2010-Implicit Differentiation by Perturbation

15 0.38617182 100 nips-2010-Gaussian Process Preference Elicitation

16 0.37851572 49 nips-2010-Computing Marginal Distributions over Continuous Markov Networks for Statistical Relational Learning

17 0.37814793 85 nips-2010-Exact learning curves for Gaussian process regression on large random graphs

18 0.36608618 213 nips-2010-Predictive Subspace Learning for Multi-view Data: a Large Margin Approach

19 0.34283185 211 nips-2010-Predicting Execution Time of Computer Programs Using Sparse Polynomial Regression

20 0.32122576 89 nips-2010-Factorized Latent Spaces with Structured Sparsity

similar papers computed by lda model

lda for this paper:

topicId topicWeight

[(13, 0.018), (17, 0.044), (27, 0.055), (30, 0.044), (45, 0.185), (50, 0.088), (52, 0.041), (60, 0.029), (77, 0.042), (78, 0.01), (90, 0.027), (97, 0.344)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.75072551 113 nips-2010-Heavy-Tailed Process Priors for Selective Shrinkage

Author: Fabian L. Wauthier, Michael I. Jordan

2 0.69689 198 nips-2010-Optimal Web-Scale Tiering as a Flow Problem

Author: Gilbert Leung, Novi Quadrianto, Kostas Tsioutsiouliklis, Alex J. Smola

Abstract: We present a fast online solver for large scale parametric max-ﬂow problems as they occur in portfolio optimization, inventory management, computer vision, and logistics. Our algorithm solves an integer linear program in an online fashion. It exploits total unimodularity of the constraint matrix and a Lagrangian relaxation to solve the problem as a convex online game. The algorithm generates approximate solutions of max-ﬂow problems by performing stochastic gradient descent on a set of ﬂows. We apply the algorithm to optimize tier arrangement of over 84 million web pages on a layered set of caches to serve an incoming query stream optimally. 1

3 0.61711758 21 nips-2010-Accounting for network effects in neuronal responses using L1 regularized point process models

Author: Ryan Kelly, Matthew Smith, Robert Kass, Tai S. Lee

Abstract: Activity of a neuron, even in the early sensory areas, is not simply a function of its local receptive ﬁeld or tuning properties, but depends on global context of the stimulus, as well as the neural context. This suggests the activity of the surrounding neurons and global brain states can exert considerable inﬂuence on the activity of a neuron. In this paper we implemented an L1 regularized point process model to assess the contribution of multiple factors to the ﬁring rate of many individual units recorded simultaneously from V1 with a 96-electrode “Utah” array. We found that the spikes of surrounding neurons indeed provide strong predictions of a neuron’s response, in addition to the neuron’s receptive ﬁeld transfer function. We also found that the same spikes could be accounted for with the local ﬁeld potentials, a surrogate measure of global network states. This work shows that accounting for network ﬂuctuations can improve estimates of single trial ﬁring rate and stimulus-response transfer functions. 1

4 0.59314305 225 nips-2010-Relaxed Clipping: A Global Training Method for Robust Regression and Classification

Author: Min Yang, Linli Xu, Martha White, Dale Schuurmans, Yao-liang Yu

Abstract: Robust regression and classiﬁcation are often thought to require non-convex loss functions that prevent scalable, global training. However, such a view neglects the possibility of reformulated training methods that can yield practically solvable alternatives. A natural way to make a loss function more robust to outliers is to truncate loss values that exceed a maximum threshold. We demonstrate that a relaxation of this form of “loss clipping” can be made globally solvable and applicable to any standard loss while guaranteeing robustness against outliers. We present a generic procedure that can be applied to standard loss functions and demonstrate improved robustness in regression and classiﬁcation problems. 1

5 0.54125118 96 nips-2010-Fractionally Predictive Spiking Neurons

Author: Jaldert Rombouts, Sander M. Bohte

Abstract: Recent experimental work has suggested that the neural ﬁring rate can be interpreted as a fractional derivative, at least when signal variation induces neural adaptation. Here, we show that the actual neural spike-train itself can be considered as the fractional derivative, provided that the neural signal is approximated by a sum of power-law kernels. A simple standard thresholding spiking neuron sufﬁces to carry out such an approximation, given a suitable refractory response. Empirically, we ﬁnd that the online approximation of signals with a sum of powerlaw kernels is beneﬁcial for encoding signals with slowly varying components, like long-memory self-similar signals. For such signals, the online power-law kernel approximation typically required less than half the number of spikes for similar SNR as compared to sums of similar but exponentially decaying kernels. As power-law kernels can be accurately approximated using sums or cascades of weighted exponentials, we demonstrate that the corresponding decoding of spiketrains by a receiving neuron allows for natural and transparent temporal signal ﬁltering by tuning the weights of the decoding kernel. 1

6 0.54046893 238 nips-2010-Short-term memory in neuronal networks through dynamical compressed sensing

7 0.54037452 51 nips-2010-Construction of Dependent Dirichlet Processes based on Poisson Processes

8 0.53664309 33 nips-2010-Approximate inference in continuous time Gaussian-Jump processes

9 0.53529435 54 nips-2010-Copula Processes

10 0.5345419 49 nips-2010-Computing Marginal Distributions over Continuous Markov Networks for Statistical Relational Learning

11 0.53239822 87 nips-2010-Extended Bayesian Information Criteria for Gaussian Graphical Models

12 0.53145504 109 nips-2010-Group Sparse Coding with a Laplacian Scale Mixture Prior

13 0.53128266 241 nips-2010-Size Matters: Metric Visual Search Constraints from Monocular Metadata

14 0.53123897 158 nips-2010-Learning via Gaussian Herding

15 0.53108096 155 nips-2010-Learning the context of a category

16 0.53058803 103 nips-2010-Generating more realistic images using gated MRF's

17 0.53033769 257 nips-2010-Structured Determinantal Point Processes

18 0.530303 239 nips-2010-Sidestepping Intractable Inference with Structured Ensemble Cascades

19 0.52975273 63 nips-2010-Distributed Dual Averaging In Networks

20 0.52926487 85 nips-2010-Exact learning curves for Gaussian process regression on large random graphs