nips nips2007 nips2007-97 knowledge-graph by maker-knowledge-mining

97 nips-2007-Hidden Common Cause Relations in Relational Learning

Source: pdf

Author: Ricardo Silva, Wei Chu, Zoubin Ghahramani

Abstract: When predicting class labels for objects within a relational database, it is often helpful to consider a model for relationships: this allows for information between class labels to be shared and to improve prediction performance. However, there are different ways by which objects can be related within a relational database. One traditional way corresponds to a Markov network structure: each existing relation is represented by an undirected edge. This encodes that, conditioned on input features, each object label is independent of other object labels given its neighbors in the graph. However, there is no reason why Markov networks should be the only representation of choice for symmetric dependence structures. Here we discuss the case when relationships are postulated to exist due to hidden common causes. We discuss how the resulting graphical model differs from Markov networks, and how it describes different types of real-world relational processes. A Bayesian nonparametric classiﬁcation model is built upon this graphical representation and evaluated with several empirical studies. 1 Contribution Prediction problems, such as classiﬁcation, can be easier when class labels share a sort of relational dependency that is not accounted by the input features [10]. If the variables to be predicted are attributes of objects in a relational database, such dependencies are often postulated from the relations that exist in the database. This paper proposes and evaluates a new method for building classiﬁers that uses information concerning the relational structure of the problem. Consider the following standard example, adapted from [3]. There are different webpages, each one labeled according to some class (e.g., “student page” or “not a student page”). Features such as the word distribution within the body of each page can be used to predict each webpage’s class. However, webpages do not exist in isolation: there are links connecting them. Two pages having a common set of links is evidence for similarity between such pages. For instance, if W1 and W3 both link to W2 , this is commonly considered to be evidence for W1 and W3 having the same class. One way of expressing this dependency is through the following Markov network [5]: ∗ Now at the Statistical Laboratory, University of Cambridge. E-mail: silva@statslab.cam.ac.uk F1 F2 F 3 C1 C2 C3 Here Fi are the features of page Wi , and Ci is its respective page label. Other edges linking F variables to C variables (e.g., F1 −C2 ) can be added without affecting the main arguments presented in this section. The semantics of the graph, for a ﬁxed input feature set {F1 , F2 , F3 }, are as follows: C1 is marginally dependent on C3 , but conditionally independent given C2 . Depending on the domain, this might be either a suitable or unsuitable representation of relations. For instance, in some domains it could be the case that the most sensible model would state that C1 is only informative about C3 once we know what C2 is: that is, C1 and C3 are marginally independent, but dependent given C2 . This can happen if the existence of a relation (Ci , Cj ) corresponds to the existence of hidden common causes generating this pair of random variables. Consider the following example, loosely based on a problem described by [12]. We have three objects, Microsoft (M ), Sony (S) and Philips (P ). The task is a regression task where we want to predict the stock market price of each company given its proﬁtability from last year. The given relationships are that M and S are direct competitors (due to the videogame console market), as well S and P (due to the TV set market). M.Profit S.Profit P.Profit M.Profit S.Profit P.Profit M.Profit S.Profit P.Profit M.Stock S.Stock P.Stock M.Stock S.Stock P.Stock M.Stock S.Stock P.Stock εm εs εp εm εs εp (a) (b) (c) Figure 1: (a) Assumptions that relate Microsoft, Sony and Philips stock prices through hidden common cause mechanisms, depicted as unlabeled gray vertices; (b) A graphical representation for generic hidden common causes relationships by using bi-directed edges; (c) A depiction of the same relationship skeleton by a Markov network model, which has different probabilistic semantics. It is expected that several market factors that affect stock prices are unaccounted by the predictor variable Past Year Proﬁt. For example, a shortage of Microsoft consoles is a hidden common factor for both Microsoft’s and Sony’s stock. Another hidden common cause would be a high price for Sony’s consoles. Assume here that these factors have no effect on Philips’ stock value. A depiction of several hidden common causes that correpond to the relations Competitor(M, S) and Competitor(S, P ) is given in Figure 1(a) as unlabeled gray vertices. Consider a linear regression model for this setup. We assume that for each object Oi ∈ {M, S, P }, the stock price Oi .Stock, centered at the mean, is given by Oi .Stock = β × Oi .P rof it + where each i i (1) is a Gaussian random variable. The fact that there are several hidden common causes between M and S can be modeled by the covariance of m and s , σms . That is, unlike in standard directed Gaussian models, σms is allowed to be non-zero. The same holds for σsp . Covariances of error terms of unrelated objects should be zero (σmp = 0). This setup is very closely related to the classic seemingly unrelated regression model popular in economics [12]. A graphical representation for this type of model is the directed mixed graph (DMG) [9, 11], with bi-directed edges representing the relationship of having hidden common causes between a pair of vertices. This is shown in Figure 1(b). Contrast this to the Markov network representation in Figure 1(c). The undirected representation encodes that m and p are marginally dependent, which does not correspond to our assumptions1 . Moreover, the model in Figure 1(b) states that once we observe Sony’s stock price, Philip’s stocks (and proﬁt) should have a non-zero association with Microsoft’s proﬁt: this follows from a extension of d-separation to DMGs [9]. This is expected from the assumptions (Philip’s stocks should tell us something about Microsoft’s once we know Sony’s, but not before it), but does not hold in the graphical model in Figure 1(c). While it is tempting to use Markov networks to represent relational models (free of concerns raised by cyclic directed representations), it is clear that there are problems for which they are not a sensible choice. This is not to say that Markov networks are not the best representation for large classes of relational problems. Conditional random ﬁelds [4] are well-motivated Markov network models for sequence learning. The temporal relationship is closed under marginalization: if we do not measure some steps in the sequence, we will still link the corresponding remaining vertices accordingly, as illustrated in Figure 2. Directed mixed graphs are not a good representation for this sequence structure. X1 X2 X3 X4 X5 Y1 Y2 Y3 Y4 Y5 X1 X2 X3 X4 X5 Y1 Y2 Y3 Y4 Y5 (a) (b) X1 X3 X5 Y1 Y3 Y5 (c) Figure 2: (a) A conditional random ﬁeld (CRF) graph for sequence data; (b) A hypothetical scenario where two of the time slices are not measured, as indicated by dashed boxes; (c) The resulting CRF graph for the remaining variables, which corresponds to the same criteria for construction of (a). To summarize, the decision between using a Markov network or a DMG reduces to the following modeling issue: if two unlinked object labels yi , yj are statistically associated when some chain of relationships exists between yi and yj , then the Markov network semantics should apply (as in the case for temporal relationships). However, if the association arises only given the values of the other objects in the chain, then this is accounted by the dependence semantics of the directed mixed graph representation. The DMG representation propagates training data information through other training points. The Markov network representation propagates training data information through test points. Propagation through training points is relevant in real problems. For instance, in a webpage domain where each webpage has links to pages of several kinds (e.g., [3]), a chain of intermediated points between two classes labels yi and yj is likely to be more informative if we know the values of the labels in this chain. The respective Markov network would ignore all training points in this chain besides the endpoints. In this paper, we introduce a non-parametric classiﬁcation model for relational data that factorizes according to a directed mixed graph. Sections 2 and 3 describes the model and contrasts it to a closely related approach which bears a strong analogy to the Markov network formulation. Experiments in text classiﬁcation are described in Section 4. 2 Model Chu et al. [2] describe an approach for Gaussian process classiﬁcation using relational information, which we review and compare to our proposed model. Previous approach: relational Gaussian processes through indicators − For each point x in the input space X , there is a corresponding function value fx . Given observed input points x1 , x2 , . . . , xn , a Gaussian process prior over f = [f1 , f2 , . . . , fn ]T has the shape P(f ) = 1 (2π)n/2 |Σ|1/2 1 exp − f T Σ−1 f 2 (2) 1 For Gaussian models, the absence of an edge in the undirected representation (i.e., Gaussian Markov random ﬁelds) corresponds to a zero entry in the inverse covariance matrix, where in the DMG it corresponds to a zero in the covariance matrix [9]. X1 X2 f1 X3 X2 X3 X1 X2 X3 f3 f2 ξ 12 X1 f1 f2 f3 f1 f2 f3 Y2 Y3 ξ 23 Y1 Y2 Y3 Y1 Y2 Y3 Y1 ε1 ε2 ε3 ε1 ε2 ε3 ε1 (a) ζ1 (b) ζ2 ε2 ζ3 ε3 (c) Figure 3: (a) A prediction problem where y3 is unknown and the training set is composed of other two datapoints. Dependencies between f1 , f2 and f3 are given by a Gaussian process prior and not represented in the picture. Indicators ξij are known and set to 1; (b) The extra associations that arise by conditioning on ξ = 1 can be factorized as the Markov network model here depicted, in the spirit of [9]; (c) Our proposed model, which ties the error terms and has origins in known statistical models such as seemingly unrelated regression and structural equation models [11]. where the ijth entry of Σ is given by a Mercer kernel function K(xi , xj ) [8]. The idea is to start from a standard Gaussian process prior, and add relational information by conditioning on relational indicators. Let ξij be an indicator that assumes different values, e.g., 1 or 0. The indicator values are observed for each pair of data points (xi , xj ): they are an encoding of the given relational structure. A model for P (ξij = 1|fi , fj ) is deﬁned. This evidence is incorporated into the Gaussian process by conditioning on all indicators ξij that are positive. Essentially, the idea boils down to using P(f |ξ = 1) as the prior for a Gaussian process classiﬁer. Figure 3(a) illustrates a problem with datapoints {(x1 , y1 ), (x2 , y2 ), (x3 , y3 )}. Gray vertices represent unobserved variables. Each yi is a binary random variable, with conditional probability given by P(yi = 1|fi ) = Φ(fi /σ) (3) where Φ(·) is the standard normal cumulative function and σ is a hyperparameter. This can be interpreted as the cumulative distribution of fi + i , where fi is given and i is a normal random variable with zero mean and variance σ 2 . In the example of Figure 3(a), one has two relations: (x1 , x2 ), (x2 , x3 ). This information is incorporated by conditioning on the evidence (ξ12 = 1, ξ23 = 1). Observed points (x1 , y1 ), (x2 , y2 ) form the training set. The prediction task is to estimate y3 . Notice that ξ12 is not used to predict y3 : the Markov blanket for f3 includes (f1 , f2 , ξ23 , y3 , 3 ) and the input features. Essentially, conditioning on ξ = 1 corresponds to a pairwise Markov network structure, as depicted in Figure 3(b) [9]2 . Our approach: mixed graph relational model − Figure 3(c) illustrates our proposed setup. For reasons that will become clear in the sequel, we parameterize the conditional probability of yi as √ (4) P(yi = 1|gi , vi ) = Φ(gi / vi ) where gi = fi + ζi . As before, Equation (4) can be interpreted as the cumulative distribution of 2 gi + i , with i as a normal random variable with zero mean and variance vi = σ 2 − σζi , the last term being the variance of ζi . That is, we break the original error term as i = ζi + i , where i and j are independent for all i = j. Random vector ζ is a multivariate normal with zero mean and covariance matrix Σζ . The key aspect in our model is that the covariance of ζi and ζj is non-zero only if objects i and j are related (that is, bi-directed edge yi ↔ yj is in the relational graph). Parameterizing Σζ for relational problems is non-trivial and discussed in the next section. In the example of Figure 3, one noticeable difference of our model 3(c) to a standard Markov network models 3(b) is that now the Markov blanket for f3 includes error terms for all variables (both and ζ terms), following the motivation presented in Section 1. 2 In the ﬁgure, we are not representing explicitly that f1 , f2 and f3 are not independent (the prior covariance matrix Σ is complete). The ﬁgure is meant as a representation of the extra associations that arise when conditioning on ξ = 1, and the way such associations factorize. As before, the prior for f in our setup is the Gaussian process prior (2). This means that g has the following Gaussian process prior (implicitly conditioned on x): P(g) = 1 (2π)n/2 |R|1/2 1 exp − g R−1 g 2 (5) where R = K + Σζ is the covariance matrix of g = f + ζ, with Kij = K(xi , xj ). 3 Parametrizing a mixed graph model for relational classiﬁcation For simplicity, in this paper we will consider only relationships that induce positive associations between labels. Ideally, the parameterization of Σζ has to fulﬁll two desiderata: (i). it should respect the marginal independence constraints as encoded by the graphical model (i.e., zero covariance for vertices that are not adjacent), and be positive deﬁnite; (ii). it has to be parsimonious in order to facilitate hyperparameter selection, both computationally and statistically. Unlike the multivariate analysis problems in [11], the size of our covariance matrix grows with the number of data points. As shown by [11], exact inference in models with covariance matrices with zero-entry constraints is computationally demanding. We provide two alternative parameterizations that are not as ﬂexible, but which lead to covariance matrices that are simple to compute and easy to implement. We will work under the transductive scenario, where training and all test points are given in advance. The corresponding graph thus contain unobserved and observed label nodes. 3.1 Method I The ﬁrst method is an automated method to relax some of the independence constraints, while guaranteeing positive-deﬁniteness, and a parameterization that depends on a single scalar ρ. This allows for more efﬁcient inference and is done as follows: 1. Let Gζ be the corresponding bi-directed subgraph of our original mixed graph, and let U0 be a matrix with n × n entries, n being the number of nodes in Gζ 2. Set U0 to be the number of cliques in Gζ where yi and yj appear together; ij 3. Set U0 to be the number of cliques containing yi , plus a small constant ∆; ii 4. Set U to be the corresponding correlation matrix obtained by intepreting U0 as a covariance matrix and rescaling it; Finally, set Σζ = ρU, where ρ ∈ [0, 1] is a given hyperparameter. Matrix U is always guaranteed to be positive deﬁnite: it is equivalent to obtaining the covariance matrix of y from a linear latent variable model, where there is an independent standard Gaussian latent variable as a common parent to every clique, and every observed node yi is given by the sum of its parents plus an independent error term of variance ∆. Marginal independencies are respected, since independent random variables will never be in a same clique in Gζ . In practice, this method cannot be used as is since the number of cliques will in general grow at an exponential rate as a function of n. Instead, we ﬁrst triangulate the graph: in this case, extracting cliques can be done in polynomial time. This is a relaxation of the original goal, since some of the original marginal independence constraints will not be enforced due to the triangulation3. 3.2 Method II The method suggested in the previous section is appealing under the assumption that vertices that appear in many common cliques are more likely to have more hidden common causes, and hence should have stronger associations. However, sometimes the triangulation introduces bad artifacts, with lots of marginal independence constraints being violated. In this case, this will often result in a poor prediction performance. A cheap alternative approach is not generating cliques, and instead 3 The need for an approximation is not a shortcoming only of the DMG approach. Notice that the relational Gaussian process of [2] also requires an approximation of its relational kernel. 10 10 10 20 20 20 30 30 30 40 40 40 50 50 50 60 60 60 70 70 70 80 80 80 90 90 90 100 100 100 10 20 30 40 50 60 (a) 70 80 90 100 10 20 30 40 50 60 70 80 90 (b) 100 10 20 30 40 50 60 70 80 90 100 (c) Figure 4: (a) The link matrix for the political books dataset. (b) The relational kernel matrix obtained with the approximated Method I. (c) The kernel matrix obtained with Method II, which tends to produce much weaker associations but does not introduce spurious relations. getting a marginal covariance matrix from a different latent variable model. In this model, we create an independent standard Gaussian variable for each edge yi ↔ yj instead of each clique. No triangulation will be necessary, and all marginal independence constraints will be respected. This, however, has shortcomings of its own: for all pairs (yi , yj ) connected by an edge, it will be the case that U0 = 1, while U0 can be as large as n. This means that the resulting correlation in Uij can be ij ii close to zero even if yi and yj are always in the same cliques. In Section 4, we will choose between Methods I and II according to the marginal likelihood of the model. 3.3 Algorithm Recall that our model is a Gaussian process classiﬁer with error terms i of variance σ such that i = ζi + i . Without loss of generality, we will assume that σ = 1. This results in the following parameterization of the full error covariance matrix: Σ = (1 − ρ)I + ρU (6) where I is an n × n identity matrix. Matrix (1 − ρ)I corresponds to the covariance matrix Σ . The usefulness of separating as and ζ becomes evident when we use an expectation-propagation (EP) algorithm [7] to perform inference in our relational classiﬁer. Instead of approximating the posterior of f , we approximate the posterior density P(g|D), D = {(x1 , y1 ), . . . , (xn , yn )} being ˜ the given training data. The approximate posterior has the form Q(g) ∝ P(g) i ti (gi ) where P(g) is the Gaussian process prior with kernel matrix R = K + Σζ as deﬁned in the previous section. Since the covariance matrix Σ is diagonal, the true likelihood of y given g factorizes n over each datapoint: P(y|g) = i=1 P(yi |gi ), and standard EP algorithms for Gaussian process classiﬁcation can be used [8] (with the variance given by Σ instead of Σ , and kernel matrix R instead of K). The ﬁnal algorithm deﬁnes a whole new class of relational models, depends on a single hyperparameter ρ which can be optimized by grid search in [0, 1], and requires virtually no modiﬁcation of code written for EP-based Gaussian process classiﬁers4 . 4 Results We now compare three different methods in relational classiﬁcation tasks. We will compare a standard Gaussian process classiﬁer (GPC), the relational Gaussian process (RGP) of [2] and our method, the mixed graph Gaussian process (XGP). A linear kernel K(x, z) = x · z is used, as described by [2]. We set ∆ = 10−4 and the hyperparameter ρ is found by a grid search in the space {0.1, 0.2, 0.3, . . . , 1.0} maximizing the approximate EP marginal likelihood5. 4 We provide MATLAB/Octave code for our method in http://www.statslab.cam.ac.uk/∼silva. For triangulation, we used the MATLAB implementation of the Reverse Cuthill McKee vertex ordering available at http://people.scs.fsu.edu/∼burkardt/m src/rcm/rcm.html 5 Table 1: The averaged AUC scores of citation prediction on test cases of the Cora database are recorded along with standard deviation over 100 trials. “n” denotes the number of papers in one class. “Citations” denotes the citation count within the two paper classes. Group n Citations GPC GPC with Citations XGP 5vs1 346/488 2466 0.905 ± 0.031 0.891 ± 0.022 0.945 ± 0.053 5vs2 346/619 3417 0.900 ± 0.032 0.905 ± 0.044 0.933 ± 0.059 5vs3 346/1376 3905 0.863 ± 0.040 0.893 ± 0.017 0.883 ± 0.013 5vs4 346/646 2858 0.916 ± 0.030 0.887 ± 0.018 0.951 ± 0.042 5vs6 346/281 1968 0.887 ± 0.054 0.843 ± 0.076 0.955 ± 0.041 5vs7 346/529 2948 0.869 ± 0.045 0.867 ± 0.041 0.926 ± 0.076 4.1 Political books We consider ﬁrst a simple classiﬁcation problem where the goal is to classify whether a particular book is of liberal political inclination or not. The features of each book are given by the words in the Amazon.com front page for that particular book. The choice of books, labels, and relationships are given in the data collected by Valdis Krebs and available at http://www-personal.umich.edu/ mejn/netdata. The data containing book features can be found at http://www.statslab.cam.ac.uk/∼silva. There are 105 books, 43 of which are labeled as liberal books. The relationships are pairs of books which are frequently purchased together by a same customer. Notice this is an easy problem, where labels are strongly associated if they share a relationship. We performed evaluation by sampling 100 times from the original pool of books, assigning half of them as trainining data. The evaluation criterion was the area under the curve (AUC) for this binary problem. This is a problem where Method I is suboptimal. Figure 4(a) shows the original binary link matrix. Figure 4(b) depicts the corresponding U0 matrix obtained with Method I, where entries closer to red correspond to stronger correlations. Method II gives a better performance here (Method I was better in the next two experiments). The AUC result for GPC was of 0.92, while both RGP and XGP achieved 0.98 (the difference between XGP and GPC having a std. deviation of 0.02). 4.2 Cora The Cora collection [6] contains over 50,000 computer science research papers including bibliographic citations. We used a subset in our experiment. The subset consists of 4,285 machine learning papers categorized into 7 classes. The second column of Table 1 shows the class sizes. Each paper was preprocessed as a bag-of-words, a vector of “term frequency” components scaled by “inverse document frequency”, and then normalized to unity length. This follows the pre-processing used in [2]. There is a total of 20,082 features. For each class, we randomly selected 1% of the labelled samples for training and tested on the remainder. The partition was repeated 100 times. We used the fact that the database is composed of fairly specialized papers as an illustration of when XGP might not be as optimal as RGP (whose AUC curves are very close to 1), since the population of links tends to be better separated between different classes (but this is also means that the task is fairly easy, and differences disappear very rapidly with increasing sample sizes). The fact there is very little training data also favors RGP, since XGP propagates information through training points. Still, XGP does better than the non-relational GPC. Notice that adding the citation adjacency matrix as a binary input feature for each paper does not improve the performance of the GPC, as shown in Table 1. Results for other classes are of similar qualitative nature and not displayed here. 4.3 WebKB The WebKB dataset consists of homepages from 4 different universities: Cornell, Texas, Washington and Wisconsin [3]. Each webpage belongs to one out of 7 categories: student, professor, course, project, staff, department and “other”. The relations come from actual links in the webpages. There is relatively high heterogeneity of types of links in each page: in terms of mixed graph modeling, this linkage mechanism is explained by a hidden common cause (e.g., a student and a course page are associated because that person’s interest in enrolling as a student also creates demand for a course). The heterogeneity also suggests that two unlinked pages should not, on average, have an association if they link to a common page W . However, observing the type of page W might create Table 2: Comparison of the three algorithms on the task “other” vs. “not-other” in the WebKB domain. Results for GPC and RGP taken from [2]. The same partitions for training and test are used to generate the results for XGP. Mean and standard deviation of AUC results are reported. University Numbers Other or Not Other All Link GPC RGP XGP Cornell 617 865 13177 0.708 ± 0.021 0.884 ± 0.025 0.917 ± 0.022 Texas 571 827 16090 0.799 ± 0.021 0.906 ± 0.026 0.949 ± 0.015 Washington 939 1205 15388 0.782 ± 0.023 0.877 ± 0.024 0.923 ± 0.016 Wisconsin 942 1263 21594 0.839 ± 0.014 0.899 ± 0.015 0.941 ± 0.018 the association. We compare how the three algorithms perform when trying to predict if a webpage is of class “other” or not (the other classiﬁcations are easier, with smaller differences. Results are omitted for space purposes). The proportion of “other” to non-“other” is about 4:1, which makes the area under the curve (AUC) a more suitable measure of success. We used the same 100 subsamples from [2], where 10% of the whole data is sampled from the pool for a speciﬁc university, and the remaining is used for test. We also used the same features as in [2], pre-processed as described in the previous section. The results are shown in Table 2. Both relational Gaussian processes are far better than the non-relational GPC. XGP gives signiﬁcant improvements over RGP in all four universities. 5 Conclusion We introduced a new family of relational classiﬁers by extending a classical statistical model [12] to non-parametric relational classiﬁcation. This is inspired by recent advances in relational Gaussian processes [2] and Bayesian inference for mixed graph models [11]. We showed empirically that modeling the type of latent phenomena that our approach postulates can sometimes improve prediction performance in problems traditionally approached by Markov network structures. Several interesting problems can be treated in the future. It is clear that there are many different ways by which the relational covariance matrix can be parameterized. Intermediate solutions between Methods I and II, approximations through matrix factorizations and graph cuts are only a few among many alternatives that can be explored. Moreover, there is a relationship between our model and multiple kernel learning [1], where one of the kernels comes from error covariances. This might provide alternative ways of learning our models, including multiple types of relationships. Acknowledgements: We thank Vikas Sindhwani for the preprocessed Cora database. References [1] F. Bach, G. Lanckriet, and M. Jordan. Multiple kernel learning, conic duality, and the SMO algorithm. 21st International Conference on Machine Learning, 2004. [2] W. Chu, V. Sindhwani, Z. Ghahramani, and S. Keerthi. Relational learning with Gaussian processes. Neural Information Processing Systems, 2006. [3] M. Craven, D. DiPasquo, D. Freitag, A. McCallum, T. Mitchell, K. Nigam, and S. Slattery. Learning to extract symbolic knowledge from the World Wide Web. Proceedings of AAAI’98, pages 509–516, 1998. [4] J. Lafferty, A. McCallum, and F. Pereira. Conditional random ﬁelds: Probabilistic models for segmenting and labeling sequence data. 18th International Conference on Machine Learning, 2001. [5] S. Lauritzen. Graphical Models. Oxford University Press, 1996. [6] A. McCallum, K. Nigam, J. Rennie, and K. Seymore. Automating the construction of Internet portals with machine learning. Information Retrieval Journal, 3:127–163, 2000. [7] T. Minka. A family of algorithms for approximate Bayesian inference. PhD Thesis, MIT, 2001. [8] C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006. [9] T. Richardson and P. Spirtes. Ancestral graph Markov models. Annals of Statistics, 30:962–1030, 2002. [10] P. Sen and L. Getoor. Link-based classiﬁcation. Report CS-TR-4858, University of Maryland, 2007. [11] R. Silva and Z. Ghahramani. Bayesian inference for Gaussian mixed graph models. UAI, 2006. [12] A. Zellner. An efﬁcient method of estimating seemingly unrelated regression equations and tests for aggregation bias. Journal of the American Statistical Association, 1962.

Reference: text

Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 uk Abstract When predicting class labels for objects within a relational database, it is often helpful to consider a model for relationships: this allows for information between class labels to be shared and to improve prediction performance. [sent-9, score-0.704]

2 However, there are different ways by which objects can be related within a relational database. [sent-10, score-0.547]

3 Here we discuss the case when relationships are postulated to exist due to hidden common causes. [sent-14, score-0.297]

4 We discuss how the resulting graphical model differs from Markov networks, and how it describes different types of real-world relational processes. [sent-15, score-0.528]

5 1 Contribution Prediction problems, such as classiﬁcation, can be easier when class labels share a sort of relational dependency that is not accounted by the input features [10]. [sent-17, score-0.547]

6 If the variables to be predicted are attributes of objects in a relational database, such dependencies are often postulated from the relations that exist in the database. [sent-18, score-0.656]

7 This paper proposes and evaluates a new method for building classiﬁers that uses information concerning the relational structure of the problem. [sent-19, score-0.486]

8 Features such as the word distribution within the body of each page can be used to predict each webpage’s class. [sent-24, score-0.125]

9 However, webpages do not exist in isolation: there are links connecting them. [sent-25, score-0.134]

10 Two pages having a common set of links is evidence for similarity between such pages. [sent-26, score-0.158]

11 uk F1 F2 F 3 C1 C2 C3 Here Fi are the features of page Wi , and Ci is its respective page label. [sent-32, score-0.25]

12 This can happen if the existence of a relation (Ci , Cj ) corresponds to the existence of hidden common causes generating this pair of random variables. [sent-39, score-0.215]

13 The task is a regression task where we want to predict the stock market price of each company given its proﬁtability from last year. [sent-42, score-0.255]

14 It is expected that several market factors that affect stock prices are unaccounted by the predictor variable Past Year Proﬁt. [sent-63, score-0.234]

15 For example, a shortage of Microsoft consoles is a hidden common factor for both Microsoft’s and Sony’s stock. [sent-64, score-0.148]

16 Another hidden common cause would be a high price for Sony’s consoles. [sent-65, score-0.255]

17 A depiction of several hidden common causes that correpond to the relations Competitor(M, S) and Competitor(S, P ) is given in Figure 1(a) as unlabeled gray vertices. [sent-67, score-0.32]

18 We assume that for each object Oi ∈ {M, S, P }, the stock price Oi . [sent-69, score-0.165]

19 The fact that there are several hidden common causes between M and S can be modeled by the covariance of m and s , σms . [sent-73, score-0.33]

20 Covariances of error terms of unrelated objects should be zero (σmp = 0). [sent-76, score-0.13]

21 This setup is very closely related to the classic seemingly unrelated regression model popular in economics [12]. [sent-77, score-0.125]

22 A graphical representation for this type of model is the directed mixed graph (DMG) [9, 11], with bi-directed edges representing the relationship of having hidden common causes between a pair of vertices. [sent-78, score-0.623]

23 Contrast this to the Markov network representation in Figure 1(c). [sent-80, score-0.118]

24 Moreover, the model in Figure 1(b) states that once we observe Sony’s stock price, Philip’s stocks (and proﬁt) should have a non-zero association with Microsoft’s proﬁt: this follows from a extension of d-separation to DMGs [9]. [sent-82, score-0.186]

25 While it is tempting to use Markov networks to represent relational models (free of concerns raised by cyclic directed representations), it is clear that there are problems for which they are not a sensible choice. [sent-84, score-0.564]

26 This is not to say that Markov networks are not the best representation for large classes of relational problems. [sent-85, score-0.531]

27 The temporal relationship is closed under marginalization: if we do not measure some steps in the sequence, we will still link the corresponding remaining vertices accordingly, as illustrated in Figure 2. [sent-87, score-0.134]

28 Directed mixed graphs are not a good representation for this sequence structure. [sent-88, score-0.182]

29 However, if the association arises only given the values of the other objects in the chain, then this is accounted by the dependence semantics of the directed mixed graph representation. [sent-91, score-0.476]

30 The DMG representation propagates training data information through other training points. [sent-92, score-0.182]

31 The Markov network representation propagates training data information through test points. [sent-93, score-0.22]

32 For instance, in a webpage domain where each webpage has links to pages of several kinds (e. [sent-95, score-0.335]

33 , [3]), a chain of intermediated points between two classes labels yi and yj is likely to be more informative if we know the values of the labels in this chain. [sent-97, score-0.349]

34 The respective Markov network would ignore all training points in this chain besides the endpoints. [sent-98, score-0.146]

35 In this paper, we introduce a non-parametric classiﬁcation model for relational data that factorizes according to a directed mixed graph. [sent-99, score-0.701]

36 [2] describe an approach for Gaussian process classiﬁcation using relational information, which we review and compare to our proposed model. [sent-103, score-0.531]

37 Previous approach: relational Gaussian processes through indicators − For each point x in the input space X , there is a corresponding function value fx . [sent-104, score-0.545]

38 , Gaussian Markov random ﬁelds) corresponds to a zero entry in the inverse covariance matrix, where in the DMG it corresponds to a zero in the covariance matrix [9]. [sent-113, score-0.297]

39 The idea is to start from a standard Gaussian process prior, and add relational information by conditioning on relational indicators. [sent-118, score-1.089]

40 The indicator values are observed for each pair of data points (xi , xj ): they are an encoding of the given relational structure. [sent-122, score-0.486]

41 This evidence is incorporated into the Gaussian process by conditioning on all indicators ξij that are positive. [sent-124, score-0.176]

42 Each yi is a binary random variable, with conditional probability given by P(yi = 1|fi ) = Φ(fi /σ) (3) where Φ(·) is the standard normal cumulative function and σ is a hyperparameter. [sent-128, score-0.132]

43 This can be interpreted as the cumulative distribution of fi + i , where fi is given and i is a normal random variable with zero mean and variance σ 2 . [sent-129, score-0.197]

44 Essentially, conditioning on ξ = 1 corresponds to a pairwise Markov network structure, as depicted in Figure 3(b) [9]2 . [sent-135, score-0.182]

45 Our approach: mixed graph relational model − Figure 3(c) illustrates our proposed setup. [sent-136, score-0.729]

46 For reasons that will become clear in the sequel, we parameterize the conditional probability of yi as √ (4) P(yi = 1|gi , vi ) = Φ(gi / vi ) where gi = fi + ζi . [sent-137, score-0.349]

47 As before, Equation (4) can be interpreted as the cumulative distribution of 2 gi + i , with i as a normal random variable with zero mean and variance vi = σ 2 − σζi , the last term being the variance of ζi . [sent-138, score-0.169]

48 Random vector ζ is a multivariate normal with zero mean and covariance matrix Σζ . [sent-140, score-0.217]

49 The key aspect in our model is that the covariance of ζi and ζj is non-zero only if objects i and j are related (that is, bi-directed edge yi ↔ yj is in the relational graph). [sent-141, score-0.851]

50 Parameterizing Σζ for relational problems is non-trivial and discussed in the next section. [sent-142, score-0.486]

51 In the example of Figure 3, one noticeable difference of our model 3(c) to a standard Markov network models 3(b) is that now the Markov blanket for f3 includes error terms for all variables (both and ζ terms), following the motivation presented in Section 1. [sent-143, score-0.118]

52 2 In the ﬁgure, we are not representing explicitly that f1 , f2 and f3 are not independent (the prior covariance matrix Σ is complete). [sent-144, score-0.182]

53 The ﬁgure is meant as a representation of the extra associations that arise when conditioning on ξ = 1, and the way such associations factorize. [sent-145, score-0.305]

54 This means that g has the following Gaussian process prior (implicitly conditioned on x): P(g) = 1 (2π)n/2 |R|1/2 1 exp − g R−1 g 2 (5) where R = K + Σζ is the covariance matrix of g = f + ζ, with Kij = K(xi , xj ). [sent-147, score-0.227]

55 3 Parametrizing a mixed graph model for relational classiﬁcation For simplicity, in this paper we will consider only relationships that induce positive associations between labels. [sent-148, score-0.923]

56 it should respect the marginal independence constraints as encoded by the graphical model (i. [sent-150, score-0.147]

57 , zero covariance for vertices that are not adjacent), and be positive deﬁnite; (ii). [sent-152, score-0.177]

58 Unlike the multivariate analysis problems in [11], the size of our covariance matrix grows with the number of data points. [sent-154, score-0.182]

59 As shown by [11], exact inference in models with covariance matrices with zero-entry constraints is computationally demanding. [sent-155, score-0.115]

60 We provide two alternative parameterizations that are not as ﬂexible, but which lead to covariance matrices that are simple to compute and easy to implement. [sent-156, score-0.115]

61 Let Gζ be the corresponding bi-directed subgraph of our original mixed graph, and let U0 be a matrix with n × n entries, n being the number of nodes in Gζ 2. [sent-162, score-0.204]

62 Set U0 to be the number of cliques in Gζ where yi and yj appear together; ij 3. [sent-163, score-0.355]

63 Set U0 to be the number of cliques containing yi , plus a small constant ∆; ii 4. [sent-164, score-0.271]

64 Set U to be the corresponding correlation matrix obtained by intepreting U0 as a covariance matrix and rescaling it; Finally, set Σζ = ρU, where ρ ∈ [0, 1] is a given hyperparameter. [sent-165, score-0.249]

65 In practice, this method cannot be used as is since the number of cliques will in general grow at an exponential rate as a function of n. [sent-168, score-0.113]

66 2 Method II The method suggested in the previous section is appealing under the assumption that vertices that appear in many common cliques are more likely to have more hidden common causes, and hence should have stronger associations. [sent-172, score-0.392]

67 However, sometimes the triangulation introduces bad artifacts, with lots of marginal independence constraints being violated. [sent-173, score-0.172]

68 Notice that the relational Gaussian process of [2] also requires an approximation of its relational kernel. [sent-176, score-1.017]

69 (b) The relational kernel matrix obtained with the approximated Method I. [sent-178, score-0.597]

70 (c) The kernel matrix obtained with Method II, which tends to produce much weaker associations but does not introduce spurious relations. [sent-179, score-0.205]

71 getting a marginal covariance matrix from a different latent variable model. [sent-180, score-0.234]

72 In this model, we create an independent standard Gaussian variable for each edge yi ↔ yj instead of each clique. [sent-181, score-0.189]

73 No triangulation will be necessary, and all marginal independence constraints will be respected. [sent-182, score-0.172]

74 This means that the resulting correlation in Uij can be ij ii close to zero even if yi and yj are always in the same cliques. [sent-184, score-0.303]

75 This results in the following parameterization of the full error covariance matrix: Σ = (1 − ρ)I + ρU (6) where I is an n × n identity matrix. [sent-189, score-0.162]

76 Matrix (1 − ρ)I corresponds to the covariance matrix Σ . [sent-190, score-0.182]

77 The usefulness of separating as and ζ becomes evident when we use an expectation-propagation (EP) algorithm [7] to perform inference in our relational classiﬁer. [sent-191, score-0.486]

78 The approximate posterior has the form Q(g) ∝ P(g) i ti (gi ) where P(g) is the Gaussian process prior with kernel matrix R = K + Σζ as deﬁned in the previous section. [sent-196, score-0.156]

79 Since the covariance matrix Σ is diagonal, the true likelihood of y given g factorizes n over each datapoint: P(y|g) = i=1 P(yi |gi ), and standard EP algorithms for Gaussian process classiﬁcation can be used [8] (with the variance given by Σ instead of Σ , and kernel matrix R instead of K). [sent-197, score-0.338]

80 The ﬁnal algorithm deﬁnes a whole new class of relational models, depends on a single hyperparameter ρ which can be optimized by grid search in [0, 1], and requires virtually no modiﬁcation of code written for EP-based Gaussian process classiﬁers4 . [sent-198, score-0.568]

81 4 Results We now compare three different methods in relational classiﬁcation tasks. [sent-199, score-0.486]

82 We will compare a standard Gaussian process classiﬁer (GPC), the relational Gaussian process (RGP) of [2] and our method, the mixed graph Gaussian process (XGP). [sent-200, score-0.864]

83 html 5 Table 1: The averaged AUC scores of citation prediction on test cases of the Cora database are recorded along with standard deviation over 100 trials. [sent-219, score-0.135]

84 1 Political books We consider ﬁrst a simple classiﬁcation problem where the goal is to classify whether a particular book is of liberal political inclination or not. [sent-259, score-0.286]

85 The relationships are pairs of books which are frequently purchased together by a same customer. [sent-271, score-0.204]

86 The fact there is very little training data also favors RGP, since XGP propagates information through training points. [sent-295, score-0.137]

87 Notice that adding the citation adjacency matrix as a binary input feature for each paper does not improve the performance of the GPC, as shown in Table 1. [sent-297, score-0.13]

88 Each webpage belongs to one out of 7 categories: student, professor, course, project, staff, department and “other”. [sent-301, score-0.123]

89 The relations come from actual links in the webpages. [sent-302, score-0.149]

90 There is relatively high heterogeneity of types of links in each page: in terms of mixed graph modeling, this linkage mechanism is explained by a hidden common cause (e. [sent-303, score-0.566]

91 , a student and a course page are associated because that person’s interest in enrolling as a student also creates demand for a course). [sent-305, score-0.281]

92 The heterogeneity also suggests that two unlinked pages should not, on average, have an association if they link to a common page W . [sent-306, score-0.404]

93 However, observing the type of page W might create Table 2: Comparison of the three algorithms on the task “other” vs. [sent-307, score-0.125]

94 We compare how the three algorithms perform when trying to predict if a webpage is of class “other” or not (the other classiﬁcations are easier, with smaller differences. [sent-337, score-0.123]

95 Both relational Gaussian processes are far better than the non-relational GPC. [sent-343, score-0.486]

96 5 Conclusion We introduced a new family of relational classiﬁers by extending a classical statistical model [12] to non-parametric relational classiﬁcation. [sent-345, score-0.972]

97 This is inspired by recent advances in relational Gaussian processes [2] and Bayesian inference for mixed graph models [11]. [sent-346, score-0.729]

98 It is clear that there are many different ways by which the relational covariance matrix can be parameterized. [sent-349, score-0.668]

99 Intermediate solutions between Methods I and II, approximations through matrix factorizations and graph cuts are only a few among many alternatives that can be explored. [sent-350, score-0.173]

100 An efﬁcient method of estimating seemingly unrelated regression equations and tests for aggregation bias. [sent-420, score-0.125]

similar papers computed by tfidf model

tfidf for this paper:

wordName wordTfidf (topN-words)

[('relational', 0.486), ('xgp', 0.254), ('rgp', 0.198), ('gpc', 0.18), ('sony', 0.169), ('dmg', 0.141), ('mixed', 0.137), ('page', 0.125), ('webpage', 0.123), ('covariance', 0.115), ('cliques', 0.113), ('markov', 0.11), ('graph', 0.106), ('books', 0.104), ('stock', 0.104), ('relationships', 0.1), ('cora', 0.098), ('yi', 0.097), ('gi', 0.097), ('associations', 0.094), ('yj', 0.092), ('auc', 0.091), ('market', 0.09), ('oi', 0.09), ('links', 0.089), ('citations', 0.085), ('philips', 0.085), ('microsoft', 0.082), ('gaussian', 0.081), ('fi', 0.081), ('hidden', 0.079), ('directed', 0.078), ('student', 0.078), ('classi', 0.074), ('political', 0.074), ('network', 0.073), ('conditioning', 0.072), ('link', 0.072), ('unrelated', 0.069), ('common', 0.069), ('triangulation', 0.067), ('webkb', 0.067), ('propagates', 0.067), ('matrix', 0.067), ('causes', 0.067), ('citation', 0.063), ('vertices', 0.062), ('labels', 0.061), ('objects', 0.061), ('price', 0.061), ('ii', 0.061), ('relations', 0.06), ('indicators', 0.059), ('liberal', 0.056), ('unlinked', 0.056), ('seemingly', 0.056), ('silva', 0.054), ('papers', 0.053), ('ij', 0.053), ('independence', 0.053), ('marginal', 0.052), ('book', 0.052), ('chu', 0.052), ('semantics', 0.052), ('marginally', 0.05), ('competitor', 0.049), ('postulated', 0.049), ('parameterization', 0.047), ('cause', 0.046), ('pro', 0.046), ('representation', 0.045), ('process', 0.045), ('depiction', 0.045), ('webpages', 0.045), ('blanket', 0.045), ('nigam', 0.045), ('mccallum', 0.044), ('kernel', 0.044), ('graphical', 0.042), ('association', 0.042), ('clique', 0.042), ('zoubin', 0.04), ('heterogeneity', 0.04), ('prices', 0.04), ('stocks', 0.04), ('philip', 0.04), ('texas', 0.04), ('chain', 0.038), ('cornell', 0.038), ('crf', 0.038), ('sindhwani', 0.038), ('wisconsin', 0.038), ('vi', 0.037), ('hyperparameter', 0.037), ('ep', 0.037), ('database', 0.037), ('depicted', 0.037), ('prediction', 0.035), ('training', 0.035), ('normal', 0.035)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.99999952 97 nips-2007-Hidden Common Cause Relations in Relational Learning

Author: Ricardo Silva, Wei Chu, Zoubin Ghahramani

2 0.23592907 94 nips-2007-Gaussian Process Models for Link Analysis and Transfer Learning

Author: Kai Yu, Wei Chu

Abstract: This paper aims to model relational data on edges of networks. We describe appropriate Gaussian Processes (GPs) for directed, undirected, and bipartite networks. The inter-dependencies of edges can be effectively modeled by adapting the GP hyper-parameters. The framework suggests an intimate connection between link prediction and transfer learning, which were traditionally two separate research topics. We develop an efﬁcient learning algorithm that can handle a large number of observations. The experimental results on several real-world data sets verify superior learning capacity. 1

3 0.12814078 131 nips-2007-Modeling homophily and stochastic equivalence in symmetric relational data

Author: Peter Hoff

Abstract: This article discusses a latent variable model for inference and prediction of symmetric relational data. The model, based on the idea of the eigenvalue decomposition, represents the relationship between two nodes as the weighted inner-product of node-speciﬁc vectors of latent characteristics. This “eigenmodel” generalizes other popular latent variable models, such as latent class and distance models: It is shown mathematically that any latent class or distance model has a representation as an eigenmodel, but not vice-versa. The practical implications of this are examined in the context of three real datasets, for which the eigenmodel has as good or better out-of-sample predictive performance than the other two models. 1

4 0.098604344 187 nips-2007-Structured Learning with Approximate Inference

Author: Alex Kulesza, Fernando Pereira

Abstract: In many structured prediction problems, the highest-scoring labeling is hard to compute exactly, leading to the use of approximate inference methods. However, when inference is used in a learning algorithm, a good approximation of the score may not be sufﬁcient. We show in particular that learning can fail even with an approximate inference method with rigorous approximation guarantees. There are two reasons for this. First, approximate methods can effectively reduce the expressivity of an underlying model by making it impossible to choose parameters that reliably give good predictions. Second, approximations can respond to parameter changes in such a way that standard learning algorithms are misled. In contrast, we give two positive results in the form of learning bounds for the use of LP-relaxed inference in structured perceptron and empirical risk minimization settings. We argue that without understanding combinations of inference and learning, such as these, that are appropriately compatible, learning performance under approximate inference cannot be guaranteed. 1

5 0.09429127 32 nips-2007-Bayesian Co-Training

Author: Shipeng Yu, Balaji Krishnapuram, Harald Steck, R. B. Rao, Rómer Rosales

Abstract: We propose a Bayesian undirected graphical model for co-training, or more generally for semi-supervised multi-view learning. This makes explicit the previously unstated assumptions of a large class of co-training type algorithms, and also clariﬁes the circumstances under which these assumptions fail. Building upon new insights from this model, we propose an improved method for co-training, which is a novel co-training kernel for Gaussian process classiﬁers. The resulting approach is convex and avoids local-maxima problems, unlike some previous multi-view learning methods. Furthermore, it can automatically estimate how much each view should be trusted, and thus accommodate noisy or unreliable views. Experiments on toy data and real world data sets illustrate the beneﬁts of this approach. 1

6 0.089931972 135 nips-2007-Multi-task Gaussian Process Prediction

7 0.087666504 212 nips-2007-Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes

8 0.084829755 105 nips-2007-Infinite State Bayes-Nets for Structured Domains

9 0.081166178 156 nips-2007-Predictive Matrix-Variate t Models

10 0.075418383 63 nips-2007-Convex Relaxations of Latent Variable Training

11 0.071658321 79 nips-2007-Efficient multiple hyperparameter learning for log-linear models

12 0.071569957 103 nips-2007-Inferring Elapsed Time from Stochastic Neural Processes

13 0.070439026 143 nips-2007-Object Recognition by Scene Alignment

14 0.067215703 75 nips-2007-Efficient Bayesian Inference for Dynamically Changing Graphs

15 0.06580957 195 nips-2007-The Generalized FITC Approximation

16 0.065160669 170 nips-2007-Robust Regression with Twinned Gaussian Processes

17 0.065005027 121 nips-2007-Local Algorithms for Approximate Inference in Minor-Excluded Graphs

18 0.063092545 39 nips-2007-Boosting the Area under the ROC Curve

19 0.062797941 49 nips-2007-Colored Maximum Variance Unfolding

20 0.062243577 132 nips-2007-Modeling image patches with a directed hierarchy of Markov random fields

similar papers computed by lsi model

lsi for this paper:

topicId topicWeight

[(0, -0.253), (1, 0.079), (2, -0.09), (3, 0.013), (4, -0.028), (5, -0.042), (6, -0.086), (7, -0.07), (8, 0.036), (9, -0.062), (10, -0.106), (11, -0.081), (12, -0.049), (13, 0.036), (14, -0.062), (15, -0.056), (16, -0.052), (17, -0.053), (18, 0.026), (19, 0.008), (20, 0.089), (21, 0.135), (22, -0.017), (23, 0.035), (24, 0.018), (25, 0.043), (26, 0.033), (27, -0.011), (28, 0.042), (29, -0.018), (30, 0.005), (31, 0.011), (32, 0.012), (33, 0.006), (34, -0.139), (35, -0.082), (36, 0.023), (37, 0.156), (38, 0.107), (39, -0.004), (40, -0.082), (41, 0.082), (42, -0.013), (43, -0.048), (44, -0.095), (45, -0.113), (46, -0.093), (47, 0.023), (48, 0.104), (49, -0.042)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.93851292 97 nips-2007-Hidden Common Cause Relations in Relational Learning

Author: Ricardo Silva, Wei Chu, Zoubin Ghahramani

2 0.71639097 131 nips-2007-Modeling homophily and stochastic equivalence in symmetric relational data

Author: Peter Hoff

3 0.6915096 94 nips-2007-Gaussian Process Models for Link Analysis and Transfer Learning

Author: Kai Yu, Wei Chu

4 0.54241198 156 nips-2007-Predictive Matrix-Variate t Models

Author: Shenghuo Zhu, Kai Yu, Yihong Gong

Abstract: It is becoming increasingly important to learn from a partially-observed random matrix and predict its missing elements. We assume that the entire matrix is a single sample drawn from a matrix-variate t distribution and suggest a matrixvariate t model (MVTM) to predict those missing elements. We show that MVTM generalizes a range of known probabilistic models, and automatically performs model selection to encourage sparse predictive models. Due to the non-conjugacy of its prior, it is difﬁcult to make predictions by computing the mode or mean of the posterior distribution. We suggest an optimization method that sequentially minimizes a convex upper-bound of the log-likelihood, which is very efﬁcient and scalable. The experiments on a toy data and EachMovie dataset show a good predictive accuracy of the model. 1

5 0.54010653 32 nips-2007-Bayesian Co-Training

Author: Shipeng Yu, Balaji Krishnapuram, Harald Steck, R. B. Rao, Rómer Rosales

6 0.53031808 187 nips-2007-Structured Learning with Approximate Inference

7 0.52624083 79 nips-2007-Efficient multiple hyperparameter learning for log-linear models

8 0.45800912 49 nips-2007-Colored Maximum Variance Unfolding

9 0.4547075 88 nips-2007-Fast and Scalable Training of Semi-Supervised CRFs with Application to Activity Recognition

10 0.45260426 135 nips-2007-Multi-task Gaussian Process Prediction

11 0.45248702 105 nips-2007-Infinite State Bayes-Nets for Structured Domains

12 0.45242715 172 nips-2007-Scene Segmentation with CRFs Learned from Partially Labeled Images

13 0.44917059 28 nips-2007-Augmented Functional Time Series Representation and Forecasting with Gaussian Processes

14 0.44183454 93 nips-2007-GRIFT: A graphical model for inferring visual classification features from human data

15 0.44090983 68 nips-2007-Discovering Weakly-Interacting Factors in a Complex Stochastic Process

16 0.43275744 197 nips-2007-The Infinite Markov Model

17 0.42951843 174 nips-2007-Selecting Observations against Adversarial Objectives

18 0.42431 158 nips-2007-Probabilistic Matrix Factorization

19 0.42188674 24 nips-2007-An Analysis of Inference with the Universum

20 0.41459391 212 nips-2007-Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes

similar papers computed by lda model

lda for this paper:

topicId topicWeight

[(5, 0.04), (11, 0.281), (13, 0.033), (16, 0.032), (18, 0.014), (21, 0.095), (31, 0.029), (34, 0.04), (35, 0.041), (47, 0.077), (49, 0.013), (83, 0.119), (85, 0.044), (87, 0.033), (90, 0.053)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.77647138 97 nips-2007-Hidden Common Cause Relations in Relational Learning

Author: Ricardo Silva, Wei Chu, Zoubin Ghahramani

2 0.68390423 69 nips-2007-Discriminative Batch Mode Active Learning

Author: Yuhong Guo, Dale Schuurmans

Abstract: Active learning sequentially selects unlabeled instances to label with the goal of reducing the effort needed to learn a good classiﬁer. Most previous studies in active learning have focused on selecting one unlabeled instance to label at a time while retraining in each iteration. Recently a few batch mode active learning approaches have been proposed that select a set of most informative unlabeled instances in each iteration under the guidance of heuristic scores. In this paper, we propose a discriminative batch mode active learning approach that formulates the instance selection task as a continuous optimization problem over auxiliary instance selection variables. The optimization is formulated to maximize the discriminative classiﬁcation performance of the target classiﬁer, while also taking the unlabeled data into account. Although the objective is not convex, we can manipulate a quasi-Newton method to obtain a good local solution. Our empirical studies on UCI datasets show that the proposed active learning is more effective than current state-of-the art batch mode active learning algorithms. 1

3 0.59097737 94 nips-2007-Gaussian Process Models for Link Analysis and Transfer Learning

Author: Kai Yu, Wei Chu

4 0.55502319 138 nips-2007-Near-Maximum Entropy Models for Binary Neural Representations of Natural Images

Author: Matthias Bethge, Philipp Berens

Abstract: Maximum entropy analysis of binary variables provides an elegant way for studying the role of pairwise correlations in neural populations. Unfortunately, these approaches suffer from their poor scalability to high dimensions. In sensory coding, however, high-dimensional data is ubiquitous. Here, we introduce a new approach using a near-maximum entropy model, that makes this type of analysis feasible for very high-dimensional data—the model parameters can be derived in closed form and sampling is easy. Therefore, our NearMaxEnt approach can serve as a tool for testing predictions from a pairwise maximum entropy model not only for low-dimensional marginals, but also for high dimensional measurements of more than thousand units. We demonstrate its usefulness by studying natural images with dichotomized pixel intensities. Our results indicate that the statistics of such higher-dimensional measurements exhibit additional structure that are not predicted by pairwise correlations, despite the fact that pairwise correlations explain the lower-dimensional marginal statistics surprisingly well up to the limit of dimensionality where estimation of the full joint distribution is feasible. 1

5 0.55155295 93 nips-2007-GRIFT: A graphical model for inferring visual classification features from human data

Author: Michael Ross, Andrew Cohen

Abstract: This paper describes a new model for human visual classiﬁcation that enables the recovery of image features that explain human subjects’ performance on different visual classiﬁcation tasks. Unlike previous methods, this algorithm does not model their performance with a single linear classiﬁer operating on raw image pixels. Instead, it represents classiﬁcation as the combination of multiple feature detectors. This approach extracts more information about human visual classiﬁcation than previous methods and provides a foundation for further exploration. 1

6 0.55117375 172 nips-2007-Scene Segmentation with CRFs Learned from Partially Labeled Images

7 0.5511573 156 nips-2007-Predictive Matrix-Variate t Models

8 0.55050218 18 nips-2007-A probabilistic model for generating realistic lip movements from speech

9 0.54838508 209 nips-2007-Ultrafast Monte Carlo for Statistical Summations

10 0.5474745 63 nips-2007-Convex Relaxations of Latent Variable Training

11 0.54699153 153 nips-2007-People Tracking with the Laplacian Eigenmaps Latent Variable Model

12 0.54695964 73 nips-2007-Distributed Inference for Latent Dirichlet Allocation

13 0.54374772 175 nips-2007-Semi-Supervised Multitask Learning

14 0.54330885 79 nips-2007-Efficient multiple hyperparameter learning for log-linear models

15 0.54313421 86 nips-2007-Exponential Family Predictive Representations of State

16 0.54217964 34 nips-2007-Bayesian Policy Learning with Trans-Dimensional MCMC

17 0.54077327 174 nips-2007-Selecting Observations against Adversarial Objectives

18 0.54020745 104 nips-2007-Inferring Neural Firing Rates from Spike Trains Using Gaussian Processes

19 0.53990656 16 nips-2007-A learning framework for nearest neighbor search

20 0.53970021 56 nips-2007-Configuration Estimates Improve Pedestrian Finding