nips nips2013 nips2013-88 knowledge-graph by maker-knowledge-mining

88 nips-2013-Designed Measurements for Vector Count Data

Source: pdf

Author: Liming Wang, David Carlson, Miguel Rodrigues, David Wilcox, Robert Calderbank, Lawrence Carin

Abstract: We consider design of linear projection measurements for a vector Poisson signal model. The projections are performed on the vector Poisson rate, X ∈ Rn , and the + observed data are a vector of counts, Y ∈ Zm . The projection matrix is designed + by maximizing mutual information between Y and X, I(Y ; X). When there is a latent class label C ∈ {1, . . . , L} associated with X, we consider the mutual information with respect to Y and C, I(Y ; C). New analytic expressions for the gradient of I(Y ; X) and I(Y ; C) are presented, with gradient performed with respect to the measurement matrix. Connections are made to the more widely studied Gaussian measurement model. Example results are presented for compressive topic modeling of a document corpora (word counting), and hyperspectral compressive sensing for chemical classiﬁcation (photon counting). 1

Reference: text

Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 edu Abstract We consider design of linear projection measurements for a vector Poisson signal model. [sent-9, score-0.256]

2 The projection matrix is designed + by maximizing mutual information between Y and X, I(Y ; X). [sent-11, score-0.291]

3 , L} associated with X, we consider the mutual information with respect to Y and C, I(Y ; C). [sent-15, score-0.224]

4 New analytic expressions for the gradient of I(Y ; X) and I(Y ; C) are presented, with gradient performed with respect to the measurement matrix. [sent-16, score-0.32]

5 Connections are made to the more widely studied Gaussian measurement model. [sent-17, score-0.192]

6 Example results are presented for compressive topic modeling of a document corpora (word counting), and hyperspectral compressive sensing for chemical classiﬁcation (photon counting). [sent-18, score-0.938]

7 For example, mutual information and conditional mean estimation have been discovered to possess close interrelationships. [sent-20, score-0.224]

8 The derivative of mutual information in a scalar Gaussian channel [11] has been expressed in terms of the minimum mean-squared error (MMSE). [sent-21, score-0.517]

9 The connections have also been extended from the scalar Gaussian to the scalar Poisson channel model [12]. [sent-22, score-0.428]

10 The gradient of mutual information in a vector Gaussian channel [17] has been expressed in terms of the MMSE matrix. [sent-23, score-0.484]

11 Recently, parallel results for scalar binomial and negative binomial channels have been established [22, 10]. [sent-25, score-0.229]

12 Inspired by the Lipster-Shiryaev formula [16], it has been demonstrated that for certain channels (or measurement models), investigation of the gradient of mutual information can often lead to a relatively simple formulation, relative to computing mutual information itself. [sent-26, score-0.704]

13 Further, it has been shown that the derivative of mutual information with respect to key system parameters also relates to the conditional mean estimates in other channel settings beyond Gaussian and Poisson models [18]. [sent-27, score-0.42]

14 This paper pursues this overarching theme for a vector Poisson measurement model. [sent-28, score-0.192]

15 Results for scalar Poisson signal models have been developed recently [12, 1] for signal recovery; the vector results presented here are new, with known scalar results recovered as a special case. [sent-29, score-0.27]

16 Further, we consider the gradient of mutual information for Poisson data in the context of classiﬁcation, for which there are no previous results, even in the scalar case. [sent-30, score-0.385]

17 The results we present for optimizing mutual information in vector Poisson measurement models are general, and may be applied to optical communication systems [15, 13]. [sent-31, score-0.461]

18 The speciﬁc applications that motivate this study are compressive measurements for vector Poisson data. [sent-32, score-0.384]

19 Direct observation of long vectors of counts may be computationally or experimentally expensive, and therefore it is of interest to design compressive Poisson measurements. [sent-33, score-0.37]

20 Almost all existing results for compres1 sive sensing (CS) directly or implicitly assume a Gaussian measurement model [6], and extension to Poisson measurements represents an important contribution of this paper. [sent-34, score-0.46]

21 To the authors knowledge, the only previous examination of CS with Poisson data was considered in [20], and that paper considered a single special (random) measurement matrix, it did not consider design of measurement matrices, and the classiﬁcation problems was not addressed. [sent-35, score-0.452]

22 It has been demonstrated in the context of Gaussian measurements that designed measurement matrices, using information-theoretic metrics, may yield substantially improved performance relative to randomly constituted measurement matrices [7, 8, 21]. [sent-36, score-0.647]

23 In this paper we extend these ideas to vector Poisson measurement systems, for both signal recovery and classiﬁcation, and make connections to the Gaussian measurement model. [sent-37, score-0.496]

24 The theory is demonstrated by considering compressive topic modeling of a document corpora, and chemical classiﬁcation with a compressive photon-counting hyperspectral camera [25]. [sent-38, score-0.896]

25 Concerning PY |X (Y |X), in this paper we focus on Poisson measurement models, but we also make connections to the much more widely considered Gaussian case. [sent-41, score-0.23]

26 For the Poisson and Gaussian measurement models the mean of PY |X (Y |X) is ΦX, where Φ ∈ Rm×n is the measurement matrix. [sent-42, score-0.384]

27 When the interest is in recovering X, we design Φ with the goal of maximizing mutual information I(X; Y ), while when interested in inferring C we design Φ with the goal of maximizing I(C; Y ). [sent-50, score-0.36]

28 To motivate use of the mutual information as the design metric, we note several results from the literature. [sent-51, score-0.292]

29 2 Existing results for Gaussian measurements There are recent results for the gradient of mutual information for vector Gaussian measurements, which we summarize here. [sent-57, score-0.438]

30 Note that PC and PX|C are arbitrary, while PY |X = N (Y ; ΦX, Λ−1 ) corresponds to a Gaussian measurement with mean ΦX. [sent-59, score-0.192]

31 It has been established that the gradient of mutual information between the input and the output of the vector Gaussian channel model obeys [17] (3) Φ I(X; Y ) = ΛΦE, 2 where E = E (X − E(X|Y ))(X − E(X|Y ))T denotes the MMSE matrix. [sent-60, score-0.484]

32 The gradient of mutual information between the class label and the output for the vector Gaussian channel is [8] Φ I(C; Y ˜ ) = ΛΦE, (4) ˜ where E = E (E(X|Y, C) − E(X|Y ))(E(X|Y, C) − E(X|Y ))T denotes the equivalent MMSE matrix. [sent-61, score-0.484]

33 These results highlight the connection between the gradient of mutual information with respect to the measurement matrix Φ and conditional-mean estimation, constituted by E(X|Y ) and E(X|Y, C). [sent-64, score-0.526]

34 1 Vector Poisson Data Model The vector Poisson channel model is deﬁned as m Pois(Y ; ΦX + λ) = PY |X (Y |X) = m PYi |X (Yi |X) = i=1 Pois (Yi ; (ΦX)i + λi ) (5) i=1 where the random vector X = (X1 , X2 , . [sent-67, score-0.196]

35 , Xn ) ∈ Rn represents the channel input, the random + vector Y = (Y1 , Y2 , . [sent-70, score-0.232]

36 , Ym ) ∈ Zm represents the channel output, Φ ∈ Rm×n represents a mea+ + surement matrix, and the vector λ = (λ1 , λ2 , . [sent-73, score-0.268]

37 + The vector Poisson channel model associated with arbitrary m and n is a generalization of the scalar Poisson model, for which m = n = 1 [12, 1]. [sent-77, score-0.293]

38 In the scalar case PY |X (Y |X) = Pois(Y ; φX + λ), where here scalar random variables X ∈ R+ and Y ∈ Z+ are associated with the input and output of the scalar channel, respectively, φ ∈ R+ is a scaling factor, and λ ∈ R+ is associated with the dark current. [sent-78, score-0.411]

39 The goal is to design Φ to maximize the mutual information between X and Y . [sent-79, score-0.292]

40 Toward that end, we consider the gradient of mutual information with respect to Φ: Φ I(X; Y ) = [ Φ I(X; Y )ij ], where Φ I(X; Y )ij represents the (i, j)-th entry of the matrix Φ I(X; Y ). [sent-80, score-0.324]

41 We also consider the gradient with respect to the vector dark current, λ I(X; Y ) = [ λ I(X; Y )i ], where λ I(X; Y )i represents the i-th entry of the vector λ I(X; Y ). [sent-81, score-0.22]

42 Consider the vector Poisson channel model in (5) and mixture signal model. [sent-104, score-0.234]

43 The gradient with respect to Φ of mutual information between the class label and output of the channel is E[(ΦX)i + λi |Y, C] [ Φ I(C; Y )ij ] = E E[Xj |Y, C] log , (8) E[(ΦX)i + λi |Y ] and with respect to the dark current is given by ( λ I(C; Y ))i =E log E[(ΦX)i + λi |Y, C] . [sent-105, score-0.604]

45 (10) (11) While the scalar result in [12] for signal recovery is obtained as a special case of our Theorem 1, for recovery of the class label C there are no previous results for our Theorem 2, even in the scalar case. [sent-113, score-0.304]

46 The gradient with respect to the dark current λ g ˜ has no analog for the Gaussian case, but similarly we have λ I(X; Y ) = E[f1 (X, E(X|Y ))] and g λ I(C; Y ) = E[˜1 (E(X|Y, C), E(X|Y ))]. [sent-117, score-0.184]

47 ∂ For the scalar Poisson channel in Corollary 1, it has been shown in [1] that ∂φ I(X; Y ) = E[ (X, E(X|Y ))], where (X, E(X|Y )) is deﬁned by the right side of (10), and is related to the Bregman divergence [5, 2]. [sent-118, score-0.363]

48 Nevertheless, these theoretical results, through generalized Bregman divergence, underscore the primacy the conditional mean estimators E(X|Y ) and E(X|Y, C) within the gradient of mutual information with respect to Φ, for both the Gaussian and Poisson vector measurement models. [sent-122, score-0.48]

49 The count for the number of times each of the n words is manifested in document d may often be modeled as Yd |Sd ∼ Pois(Yd ; ΨSd ); see [26] and the extensive set of references therein. [sent-126, score-0.231]

50 represents the number of times words are manifested in a document in m distinct sets. [sent-149, score-0.229]

51 Our goal is to use the theory developed above to design the binary Φ such that the compressive Yd |Xd ∼ Pois(Yd ; ΦXd ) is as informative as possible. [sent-150, score-0.302]

52 For the case in which we are interested in inferring Sd from the compressive measurements, i. [sent-155, score-0.234]

53 This is done for all document classes, and we design a compressive matrix Φ ∈ {0, 1}m×n , with gradient performed using Theorem 2. [sent-163, score-0.446]

54 In the testing phase, using held-out documents, we employ the matrix Φ to group the counts of words in document d into counts on m sets of words, with sets deﬁned by the rows of Φ. [sent-164, score-0.252]

55 2 Model for Chemical Sensing The model employed for the chemical sensing [25] considered below is very similar in form to that used for topic modeling, so we reuse notation. [sent-168, score-0.329]

56 Assume that there are T fundamental (building-block) chemicals of interest, and that the hyperspectral sensor performs measurements at n wavelengths. [sent-169, score-0.353]

57 For the compressive chemical-sensing system discussed in Section 4. [sent-197, score-0.234]

58 5, the measurement matrix is again binary, Φ ∈ {0, 1}m×n . [sent-198, score-0.192]

59 Through calibrations and known properties of chemicals and characteristics of the camera, one may readily constitute Ψ and λ, and a model similar to that employed for topic modeling is utilized to model Sd ; here λ is a characteristic of the camera, and is not optimized. [sent-199, score-0.193]

60 For the chemical sensing application, the goal is to classify the chemical sample under test, and therefore Φ is deﬁned based on optimization using the Theorem 2 gradient. [sent-201, score-0.404]

61 The “clean” forms of the gradients in Theorems 1 and 2 signiﬁcantly simpliﬁed design implementation within the below experiments, with the added value of allowing connections to be made to the Gaussian measurement model. [sent-213, score-0.298]

62 (c) The confusion matrix on the “comp” subgroup for 150 compressive measurements. [sent-304, score-0.275]

63 For the KL divergence, we compare the topic mixture learned from the projection measurements to the topic mixture learned from the case where each word is observed (no compressive measurement). [sent-312, score-0.607]

64 We calculate DKL (Sd,p ||Sd,f ) = K k=1 Sdk,p log(Sdk,p /Sdk,f ), where Sdk,p is the relative weight on document d, topic k for the full set of words, and Sdk,p is the same for the compressive topic model. [sent-314, score-0.486]

65 Because different projection metrics are in different dimensions, we use 75% of a document’s words to get the projection measurements Yd and use the remaining 25% as the original word tokens Wd . [sent-316, score-0.237]

66 At very low numbers of compressive measurements we get similar PLL between the designed matrix and the random methods. [sent-321, score-0.451]

67 As in the 20 Newsgroups results, the predictive log-likelihood and KL divergence of the random and designed measurements are similar when the number of projections are low. [sent-327, score-0.416]

68 The compressive measurements give near the same performance with half the per-document processing time. [sent-330, score-0.384]

69 Classiﬁcation versus number of prod jections for random projections and designed projections are shown in Figure 3(a). [sent-336, score-0.211]

70 5 Poisson Compressive Sensing for Chemical Classiﬁcation We consider chemical sensing based on the wavelength-dependent signature of chemicals, at optical frequencies (here we consider a 850-1000 nm laser system). [sent-352, score-0.394]

71 In Figure 4(a) the measurement system is summarized; details of this system are described in [25]. [sent-353, score-0.192]

72 In Part 1 of Figure 4(a) multi-wavelength photons are scattered off a chemical sample. [sent-354, score-0.345]

73 In Part 2 of this ﬁgure a volume holographic grating (VHG) is employed to diffract the photons in a wavelength-dependent manner, and therefore photons are distributed spatially across a digital mirror microdevice (DMD); distinct wavelengths are associated with each micromirror. [sent-355, score-0.686]

74 Each mirror approximately samples a single wavelength, as a result of the VHG, and the photon counter counts all photons at wavelengths for which the mirrors direct light to the sensor. [sent-358, score-0.709]

75 Hence, the sensor counts all photons at a subset of the wavelengths, those for which the mirror is at the appropriate angle. [sent-359, score-0.354]

76 The measurement may be represented Y |Sd ∼ Pois[Φ(ΨSd + λ0 )], Classification of 10 Chemicals where λ0 ∈ Rn is known from cali+ bration. [sent-360, score-0.192]

77 5 per bin, and the cumulative dark current Φλ0 can provide in excess of 50% of the signal energy, depending on the measurement (very noisy measurements). [sent-363, score-0.35]

78 Design of Φ was based on Theorem 2, and λ0 here is treated as the sigNumber of Measurements nature of an additional chemical (ac(a) (b) tually associated with measurement in unison. [sent-364, score-0.353]

79 The −12 designed 4: the pure of ∼1 below the incident light in component emission rates are normalized to the same value, graphic grating, thatandspatially order to spatially separate the spreads photons in a wavelengthsurement dark current. [sent-366, score-0.413]

80 Schematic of the DMD-based near infrared digital compressive detection instrument. [sent-381, score-0.272]

81 ◦ ◦ i = (8) dependent manner across the digital mirror microdevice (DMD), j in a second pass through the holographic grating, and focused onto a ﬁber optic cable that is connected to a photodiode photon counting module (PerkinElmer, SPCMCD2969PE). [sent-382, score-0.362]

82 The photon counting module has a dark count rate of ∼200 photons s−1 and no read noise. [sent-383, score-0.547]

83 A TTL pulse is output by the photon counter as each photon is detected, and the pulses are counted in a USB data acquisition (DAQ) card (National Instruments, USB-6212BNC). [sent-384, score-0.35]

84 vi) that sets blocks of mirrors on the DMD array corresponding to different wavelengths to the appropriate ±12◦ position. [sent-391, score-0.192]

85 Labview scripts were used to sequentially apply the ﬁlters and integrate for the corresponding times, to store the raw photon counts, and to calculate the photon rates. [sent-392, score-0.312]

86 i and j, design ﬁlters F for The ten chemicals considered in allthisi. [sent-397, score-0.175]

87 (b) Perimize the error in estimating a the DMD is mixture of photons emitted by all chemical species are the same. [sent-400, score-0.345]

88 Seeof compressive measurements; ten chemicals are of the number www. [sent-407, score-0.341]

89 Experimental apparatus from Figure 4 that after only ﬁve compressive measurements excellent chemical classiﬁcation is The detection shown Fig. [sent-416, score-0.545]

90 The excitation source is a 785 nm tional measurement of these data, mode laser (Innovative Photonic Solutions). [sent-420, score-0.298]

91 Note thatlaser Rayleigh scattering withprojection measurements perform markedly better than the designed a dichroic mirror (Semrock, LPD01-785RS-25) random, where here the probability of and aone in the random design was 10% (this yielded best random a 785 nm notch ﬁlter (Semrock, NF03-785E25). [sent-425, score-0.482]

92 The window of the dispersed light is ∼200–1700 cm−1 with a spectral resolution of 30 cm−1 (this resolution is limited by the beam quality and hence the image of the diode laser focal spot size, which spans approximately 15 mirrors on the surface of the DMD). [sent-428, score-0.193]

93 , if we want to divide the energy of the photons into 128 “bins”, then groups of 15 adjacent columns 3. [sent-434, score-0.184]

94 Constructing ﬁlters Generating accurate ﬁlters a given application requires high New results are presented for the gradient of mutual informationforofwithcomponents of inter- to the measurement respect signal-to-noise training spectra each of the est. [sent-436, score-0.528]

95 Measuring full spectra with the DMD is achieved matrix and a dark current, within the context of a Poisson modelsequentiallyvectorby notch for directing one mirror (or count data. [sent-437, score-0.36]

96 For the all other mir- we recover known former rors directed away) and counting the number of photons detected at each notch position. [sent-440, score-0.32]

97 Notch scanning measurements were perscalar results as a special case, and the latter results for classiﬁcationspectra with a signal-to-noisebeen addressed in any have not formed using 1 s per notch to obtain ratio of ∼500:1. [sent-441, score-0.237]

98 The ﬁrst method involves measuring the background (with no hyperspectral chemical been demonstrated for compressive topic modeling, and for compressive sample) sensing (with demonstration on a real compressive camera). [sent-446, score-1.092]

99 Gradient of mutual information in linear vector Gaussian channels. [sent-567, score-0.224]

100 Certain relations between mutual information and ﬁdelity of statistical estimation. [sent-579, score-0.224]

similar papers computed by tfidf model

tfidf for this paper:

wordName wordTfidf (topN-words)

[('poisson', 0.248), ('compressive', 0.234), ('mutual', 0.224), ('pois', 0.209), ('yd', 0.198), ('channel', 0.196), ('measurement', 0.192), ('sd', 0.192), ('dmd', 0.191), ('photons', 0.184), ('chemical', 0.161), ('photon', 0.156), ('measurements', 0.15), ('nnmf', 0.139), ('py', 0.127), ('ortho', 0.122), ('dark', 0.12), ('chemicals', 0.107), ('nytimes', 0.107), ('pll', 0.107), ('wavelengths', 0.104), ('scalar', 0.097), ('mmse', 0.093), ('qy', 0.092), ('px', 0.09), ('mirrors', 0.088), ('notch', 0.087), ('verd', 0.087), ('ydk', 0.087), ('xd', 0.087), ('topic', 0.086), ('co', 0.085), ('newsgroups', 0.085), ('sensing', 0.082), ('document', 0.08), ('manifested', 0.077), ('pe', 0.073), ('projections', 0.072), ('divergence', 0.07), ('design', 0.068), ('counts', 0.068), ('mirror', 0.067), ('designed', 0.067), ('binomial', 0.066), ('gradient', 0.064), ('laser', 0.063), ('hyperspectral', 0.061), ('lda', 0.059), ('predictive', 0.057), ('grating', 0.057), ('bregman', 0.054), ('comp', 0.053), ('calderbank', 0.052), ('holographic', 0.052), ('labview', 0.052), ('rodrigues', 0.052), ('semrock', 0.052), ('vhg', 0.052), ('wilcox', 0.052), ('xdi', 0.052), ('word', 0.051), ('kl', 0.051), ('optimized', 0.051), ('lters', 0.051), ('instruments', 0.051), ('classi', 0.049), ('counting', 0.049), ('spectra', 0.048), ('fy', 0.048), ('documents', 0.048), ('constituted', 0.046), ('mismatched', 0.046), ('optical', 0.045), ('regularity', 0.044), ('nm', 0.043), ('rand', 0.042), ('light', 0.042), ('confusion', 0.041), ('ij', 0.041), ('topics', 0.04), ('rn', 0.04), ('camera', 0.04), ('gaussian', 0.039), ('connections', 0.038), ('signal', 0.038), ('card', 0.038), ('sim', 0.038), ('raman', 0.038), ('count', 0.038), ('digital', 0.038), ('theorems', 0.036), ('irrespective', 0.036), ('represents', 0.036), ('words', 0.036), ('recovery', 0.036), ('sensor', 0.035), ('aluminum', 0.035), ('analytica', 0.035), ('carson', 0.035), ('chimica', 0.035)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 1.0000005 88 nips-2013-Designed Measurements for Vector Count Data

Author: Liming Wang, David Carlson, Miguel Rodrigues, David Wilcox, Robert Calderbank, Lawrence Carin

2 0.14023842 304 nips-2013-Sparse nonnegative deconvolution for compressive calcium imaging: algorithms and phase transitions

Author: Eftychios A. Pnevmatikakis, Liam Paninski

Abstract: We propose a compressed sensing (CS) calcium imaging framework for monitoring large neuronal populations, where we image randomized projections of the spatial calcium concentration at each timestep, instead of measuring the concentration at individual locations. We develop scalable nonnegative deconvolution methods for extracting the neuronal spike time series from such observations. We also address the problem of demixing the spatial locations of the neurons using rank-penalized matrix factorization methods. By exploiting the sparsity of neural spiking we demonstrate that the number of measurements needed per timestep is signiﬁcantly smaller than the total number of neurons, a result that can potentially enable imaging of larger populations at considerably faster rates compared to traditional raster-scanning techniques. Unlike traditional CS setups, our problem involves a block-diagonal sensing matrix and a non-orthogonal sparse basis that spans multiple timesteps. We provide tight approximations to the number of measurements needed for perfect deconvolution for certain classes of spiking processes, and show that this number undergoes a “phase transition,” which we characterize using modern tools relating conic geometry to compressed sensing. 1

3 0.13235639 265 nips-2013-Reconciling "priors" & "priors" without prejudice?

Author: Remi Gribonval, Pierre Machart

Abstract: There are two major routes to address linear inverse problems. Whereas regularization-based approaches build estimators as solutions of penalized regression optimization problems, Bayesian estimators rely on the posterior distribution of the unknown, given some assumed family of priors. While these may seem radically different approaches, recent results have shown that, in the context of additive white Gaussian denoising, the Bayesian conditional mean estimator is always the solution of a penalized regression problem. The contribution of this paper is twofold. First, we extend the additive white Gaussian denoising results to general linear inverse problems with colored Gaussian noise. Second, we characterize conditions under which the penalty function associated to the conditional mean estimator can satisfy certain popular properties such as convexity, separability, and smoothness. This sheds light on some tradeoff between computational efﬁciency and estimation accuracy in sparse regularization, and draws some connections between Bayesian estimation and proximal optimization. 1

4 0.12710811 217 nips-2013-On Poisson Graphical Models

Author: Eunho Yang, Pradeep Ravikumar, Genevera I. Allen, Zhandong Liu

Abstract: Undirected graphical models, such as Gaussian graphical models, Ising, and multinomial/categorical graphical models, are widely used in a variety of applications for modeling distributions over a large number of variables. These standard instances, however, are ill-suited to modeling count data, which are increasingly ubiquitous in big-data settings such as genomic sequencing data, user-ratings data, spatial incidence data, climate studies, and site visits. Existing classes of Poisson graphical models, which arise as the joint distributions that correspond to Poisson distributed node-conditional distributions, have a major drawback: they can only model negative conditional dependencies for reasons of normalizability given its inﬁnite domain. In this paper, our objective is to modify the Poisson graphical model distribution so that it can capture a rich dependence structure between count-valued variables. We begin by discussing two strategies for truncating the Poisson distribution and show that only one of these leads to a valid joint distribution. While this model can accommodate a wider range of conditional dependencies, some limitations still remain. To address this, we investigate two additional novel variants of the Poisson distribution and their corresponding joint graphical model distributions. Our three novel approaches provide classes of Poisson-like graphical models that can capture both positive and negative conditional dependencies between count-valued variables. One can learn the graph structure of our models via penalized neighborhood selection, and we demonstrate the performance of our methods by learning simulated networks as well as a network from microRNA-sequencing data. 1

5 0.12624699 174 nips-2013-Lexical and Hierarchical Topic Regression

Author: Viet-An Nguyen, Jordan Boyd-Graber, Philip Resnik

Abstract: Inspired by a two-level theory from political science that uniﬁes agenda setting and ideological framing, we propose supervised hierarchical latent Dirichlet allocation (S H L DA), which jointly captures documents’ multi-level topic structure and their polar response variables. Our model extends the nested Chinese restaurant processes to discover tree-structured topic hierarchies and uses both per-topic hierarchical and per-word lexical regression parameters to model response variables. S H L DA improves prediction on political afﬁliation and sentiment tasks in addition to providing insight into how topics under discussion are framed. 1 Introduction: Agenda Setting and Framing in Hierarchical Models How do liberal-leaning bloggers talk about immigration in the US? What do conservative politicians have to say about education? How do Fox News and MSNBC differ in their language about the gun debate? Such questions concern not only what, but how things are talked about. In political communication, the question of “what” falls under the heading of agenda setting theory, which concerns the issues introduced into political discourse (e.g., by the mass media) and their inﬂuence over public priorities [1]. The question of “how” concerns framing: the way the presentation of an issue reﬂects or encourages a particular perspective or interpretation [2]. For example, the rise of the “innocence frame” in the death penalty debate, emphasizing the irreversible consequence of mistaken convictions, has led to a sharp decline in the use of capital punishment in the US [3]. In its concern with the subjects or issues under discussion in political discourse, agenda setting maps neatly to topic modeling [4] as a means of discovering and characterizing those issues [5]. Interestingly, one line of communication theory seeks to unify agenda setting and framing by viewing frames as a second-level kind of agenda [1]: just as agenda setting is about which objects of discussion are salient, framing is about the salience of attributes of those objects. The key is that what communications theorists consider an attribute in a discussion can itself be an object, as well. For example, “mistaken convictions” is one attribute of the death penalty discussion, but it can also be viewed as an object of discussion in its own right. This two-level view leads naturally to the idea of using a hierarchical topic model to formalize both agendas and frames within a uniform setting. In this paper, we introduce a new model to do exactly that. The model is predictive: it represents the idea of alternative or competing perspectives via a continuous-valued response variable. Although inspired by the study of political discourse, associating texts with “perspectives” is more general and has been studied in sentiment analysis, discovery of regional variation, and value-sensitive design. We show experimentally that the model’s hierarchical structure improves prediction of perspective in both a political domain and on sentiment analysis tasks, and we argue that the topic hierarchies exposed by the model are indeed capturing structure in line with the theory that motivated the work. 1 ߨ ݉ ߠௗ ߙ ߰ௗ ߛ ‫ݐ‬ௗ௦ ‫ݖ‬ௗ௦௡ ‫ݓ‬ௗ௦௡ ܿௗ௧ ܰௗ௦ ∞ ߩ ܵௗ ‫ݕ‬ௗ ‫ܦ‬ ߱ ߟ௞ ߬௩ ܸ 1. For each node k ∈ [1, ∞) in the tree (a) Draw topic φk ∼ Dir(βk ) (b) Draw regression parameter ηk ∼ N (µ, σ) 2. For each word v ∈ [1, V ], draw τv ∼ Laplace(0, ω) 3. For each document d ∈ [1, D] (a) Draw level distribution θd ∼ GEM(m, π) (b) Draw table distribution ψd ∼ GEM(α) (c) For each table t ∈ [1, ∞), draw a path cd,t ∼ nCRP(γ) (d) For each sentence s ∈ [1, Sd ], draw a table indicator td,s ∼ Mult(ψd ) i. For each token n ∈ [1, Nd,s ] A. Draw level zd,s,n ∼ Mult(θd ) B. Draw word wd,s,n ∼ Mult(φcd,td,s ,zd,s,n ) ¯ ¯ (e) Draw response yd ∼ N (η T zd + τ T wd , ρ): ߶௞ ∞ ߤ i. zd,k = ¯ ߪ ߚ ii. wd,v = ¯ 1 Nd,· 1 Nd,· Sd s=1 Sd s=1 Nd,s n=1 I [kd,s,n = k] Nd,s n=1 I [wd,s,n = v] Figure 1: S H L DA’s generative process and plate diagram. Words w are explained by topic hierarchy φ, and response variables y are explained by per-topic regression coefﬁcients η and global lexical coefﬁcients τ . 2 S H L DA: Combining Supervision and Hierarchical Topic Structure Jointly capturing supervision and hierarchical topic structure falls under a class of models called supervised hierarchical latent Dirichlet allocation. These models take as input a set of D documents, each of which is associated with a response variable yd , and output a hierarchy of topics which is informed by yd . Zhang et al. [6] introduce the S H L DA family, focusing on a categorical response. In contrast, our novel model (which we call S H L DA for brevity), uses continuous responses. At its core, S H L DA’s document generative process resembles a combination of hierarchical latent Dirichlet allocation [7, HLDA] and the hierarchical Dirichlet process [8, HDP]. HLDA uses the nested Chinese restaurant process (nCRP(γ)), combined with an appropriate base distribution, to induce an unbounded tree-structured hierarchy of topics: general topics at the top, speciﬁc at the bottom. A document is generated by traversing this tree, at each level creating a new child (hence a new path) with probability proportional to γ or otherwise respecting the “rich-get-richer” property of a CRP. A drawback of HLDA, however, is that each document is restricted to only a single path in the tree. Recent work relaxes this restriction through different priors: nested HDP [9], nested Chinese franchises [10] or recursive CRPs [11]. In this paper, we address this problem by allowing documents to have multiple paths through the tree by leveraging information at the sentence level using the twolevel structure used in HDP. More speciﬁcally, in the HDP’s Chinese restaurant franchise metaphor, customers (i.e., tokens) are grouped by sitting at tables and each table takes a dish (i.e., topic) from a ﬂat global menu. In our S H L DA, dishes are organized in a tree-structured global menu by using the nCRP as prior. Each path in the tree is a collection of L dishes (one for each level) and is called a combo. S H L DA groups sentences of a document by assigning them to tables and associates each table with a combo, and thus, models each document as a distribution over combos.1 In S H L DA’s metaphor, customers come in a restaurant and sit at a table in groups, where each group is a sentence. A sentence wd,s enters restaurant d and selects a table t (and its associated combo) with probability proportional to the number of sentences Sd,t at that table; or, it sits at a new table with probability proportional to α. After choosing the table (indexed by td,s ), if the table is new, the group will select a combo of dishes (i.e., a path, indexed by cd,t ) from the tree menu. Once a combo is in place, each token in the sentence chooses a “level” (indexed by zd,s,n ) in the combo, which speciﬁes the topic (φkd,s,n ≡ φcd,td,s ,zd,s,n ) producing the associated observation (Figure 2). S H L DA also draws on supervised LDA [12, SLDA] associating each document d with an observable continuous response variable yd that represents the author’s perspective toward a topic, e.g., positive vs. negative sentiment, conservative vs. liberal ideology, etc. This lets us infer a multi-level topic structure informed by how topics are “framed” with respect to positions along the yd continuum. 1 We emphasize that, unlike in HDP where each table is assigned to a single dish, each table in our metaphor is associated with a combo–a collection of L dishes. We also use combo and path interchangeably. 2 Sd Sd,t ߶ଵ ߟଵ dish ߶ଵଵ ߟଵଵ ߶ଵଶ ߟଵଶ ߶ଵଵଵ ߟଵଵଵ ߶ଵଵଶ ߟଵଵଶ ߶ଵଶଵ ߟଵଶଵ ߶ଵଶଶ ߟଵଶଶ table ܿௗ௧ ‫1=ݐ‬ ‫2=ݐ‬ ‫1=ݐ‬ ‫2=ݐ‬ ‫3=ݐ‬ ‫1=ݐ‬ ‫2=ݐ‬ ‫ݐ‬ௗ௦ ‫2=ݏ 1=ݏ‬ ‫ܵ = ݏ‬ଵ ‫3=ݏ 2=ݏ 1=ݏ‬ ݀=1 ݇ௗ௦௡ ‫ܵ = ݏ‬ଶ ‫ܵ = ݏ‬஽ ݀=2 ߶ଵ ߟଵ ݀=‫ܦ‬ customer group (token) (sentence) restaurant (document) ߶ଵଵ ߟଵଵ ݀=1 ‫1=ݏ‬ ߶ଵଵଵ ߟଵଵଵ combo (path) Nd,s Nd,·,l Nd,·,>l Nd,·,≥l Mc,l Cc,l,v Cd,x,l,v φk ηk τv cd,t td,s zd,s,n kd,s,n L C+ Figure 2: S H L DA’s restaurant franchise metaphor. # sentences in document d # groups (i.e. sentences) sitting at table t in restaurant d # tokens wd,s # tokens in wd assigned to level l # tokens in wd assigned to level > l ≡ Nd,·,l + Nd,·,>l # tables at level l on path c # word type v assigned to level l on path c # word type v in vd,x assigned to level l Topic at node k Regression parameter at node k Regression parameter of word type v Path assignment for table t in restaurant d Table assignment for group wd,s Level assignment for wd,s,n Node assignment for wd,s,n (i.e., node at level zd,s,n on path cd,td,s ) Height of the tree Set of all possible paths (including new ones) of the tree Table 1: Notation used in this paper Unlike SLDA, we model the response variables using a normal linear regression that contains both pertopic hierarchical and per-word lexical regression parameters. The hierarchical regression parameters are just like topics’ regression parameters in SLDA: each topic k (here, a tree node) has a parameter ηk , and the model uses the empirical distribution over the nodes that generated a document as the regressors. However, the hierarchy in S H L DA makes it possible to discover relationships between topics and the response variable that SLDA’s simple latent space obscures. Consider, for example, a topic model trained on Congressional debates. Vanilla LDA would likely discover a healthcare category. SLDA [12] could discover a pro-Obamacare topic and an anti-Obamacare topic. S H L DA could do that and capture the fact that there are alternative perspectives, i.e., that the healthcare issue is being discussed from two ideological perspectives, along with characterizing how the higher level topic is discussed by those on both sides of that ideological debate. Sometimes, of course, words are strongly associated with extremes on the response variable continuum regardless of underlying topic structure. Therefore, in addition to hierarchical regression parameters, we include global lexical regression parameters to model the interaction between speciﬁc words and response variables. We denote the regression parameter associated with a word type v in the vocabulary as τv , and use the normalized frequency of v in the documents to be its regressor. Including both hierarchical and lexical parameters is important. For detecting ideology in the US, “liberty” is an effective indicator of conservative speakers regardless of context; however, “cost” is a conservative-leaning indicator in discussions about environmental policy but liberal-leaning in debates about foreign policy. For sentiment, “wonderful” is globally a positive word; however, “unexpected” is a positive descriptor of books but a negative one of a car’s steering. S H L DA captures these properties in a single model. 3 Posterior Inference and Optimization Given documents with observed words w = {wd,s,n } and response variables y = {yd }, the inference task is to ﬁnd the posterior distribution over: the tree structure including topic φk and regression parameter ηk for each node k, combo assignment cd,t for each table t in document d, table assignment td,s for each sentence s in a document d, and level assignment zd,s,n for each token wd,s,n . We approximate S H L DA’s posterior using stochastic EM, which alternates between a Gibbs sampling E-step and an optimization M-step. More speciﬁcally, in the E-step, we integrate out ψ, θ and φ to construct a Markov chain over (t, c, z) and alternate sampling each of them from their conditional distributions. In the M-step, we optimize the regression parameters η and τ using L-BFGS [13]. Before describing each step in detail, let us deﬁne the following probabilities. For more thorough derivations, please see the supplement. 3 • First, deﬁne vd,x as a set of tokens (e.g., a token, a sentence or a set of sentences) in document d. The conditional density of vd,x being assigned to path c given all other assignments is −d,x Γ(Cc,l,· + V βl ) L −d,x fc (vd,x ) = l=1 −d,x Γ(Cc,l,v + Cd,x,l,v + βl ) V −d,x Γ(Cc,l,· + Cd,x,l,· + V βl ) (1) −d,x Γ(Cc,l,v + βl ) v=1 where superscript −d,x denotes the same count excluding assignments of vd,x ; marginal counts −d,x are represented by ·’s. For a new path cnew , if the node does not exist, Ccnew ,l,v = 0 for all word types v. • Second, deﬁne the conditional density of the response variable yd of document d given vd,x being −d,x assigned to path c and all other assignments as gc (yd ) =  1 N Nd,· ηc,l · Cd,x,l,· + ηcd,td,s ,zd,s,n + wd,s,n ∈{wd \vd,x }  Sd Nd,s L τwd,s,n , ρ (2) s=1 n=1 l=1 where Nd,· is the total number of tokens in document d. For a new node at level l on a new path cnew , we integrate over all possible values of ηcnew ,l . Sampling t: For each group wd,s we need to sample a table td,s . The conditional distribution of a table t given wd,s and other assignments is proportional to the number of sentences sitting at t times the probability of wd,s and yd being observed under this assignment. This is P (td,s = t | rest) ∝ P (td,s = t | t−s ) · P (wd,s , yd | td,s = t, w−d,s , t−d,s , z, c, η) d ∝ −d,s −d,s −d,s Sd,t · fcd,t (wd,s ) · gcd,t (yd ), for existing table t; (3) −d,s −d,s α · c∈C + P (cd,tnew = c | c−d,s ) · fc (wd,s ) · gc (yd ), for new table tnew . For a new table tnew , we need to sum over all possible paths C + of the tree, including new ones. For example, the set C + for the tree shown in Figure 2 consists of four existing paths (ending at one of the four leaf nodes) and three possible new paths (a new leaf off of one of the three internal nodes). The prior probability of path c is: P (cd,tnew = c | c−d,s ) ∝       L l=2 −d,s Mc,l −d,s Mc,l−1 + γl−1  γl∗    −d,s M ∗ cnew ,l∗ + γl , l∗ l=2 for an existing path c; (4) −d,s Mcnew ,l , for a new path cnew which consists of an existing path −d,s Mcnew ,l−1 + γl−1 from the root to a node at level l∗ and a new node. Sampling z: After assigning a sentence wd,s to a table, we assign each token wd,s,n to a level to choose a dish from the combo. The probability of assigning wd,s,n to level l is −s,n P (zd,s,n = l | rest) ∝ P (zd,s,n = l | zd )P (wd,s,n , yd | zd,s,n = l, w−d,s,n , z −d,s,n , t, c, η) (5) The ﬁrst factor captures the probability that a customer in restaurant d is assigned to level l, conditioned on the level assignments of all other customers in restaurant d, and is equal to P (zd,s,n = −s,n l | zd ) = −d,s,n mπ + Nd,·,l −d,s,n π + Nd,·,≥l l−1 −d,s,n (1 − m)π + Nd,·,>j −d,s,n π + Nd,·,≥j j=1 , The second factor is the probability of observing wd,s,n and yd , given that wd,s,n is assigned to level −d,s,n −d,s,n l: P (wd,s,n , yd | zd,s,n = l, w−d,s,n , z −d,s,n , t, c, η) = fcd,t (wd,s,n ) · gcd,t (yd ). d,s d,s Sampling c: After assigning customers to tables and levels, we also sample path assignments for all tables. This is important since it can change the assignments of all customers sitting at a table, which leads to a well-mixed Markov chain and faster convergence. The probability of assigning table t in restaurant d to a path c is P (cd,t = c | rest) ∝ P (cd,t = c | c−d,t ) · P (wd,t , yd | cd,t = c, w−d,t , c−d,t , t, z, η) (6) where we slightly abuse the notation by using wd,t ≡ ∪{s|td,s =t} wd,s to denote the set of customers in all the groups sitting at table t in restaurant d. The ﬁrst factor is the prior probability of a path given all tables’ path assignments c−d,t , excluding table t in restaurant d and is given in Equation 4. The second factor in Equation 6 is the probability of observing wd,t and yd given the new path −d,t −d,t assignments, P (wd,t , yd | cd,t = c, w−d,t , c−d,t , t, z, η) = fc (wd,t ) · gc (yd ). 4 Optimizing η and τ : We optimize the regression parameters η and τ via the likelihood, 1 L(η, τ ) = − 2ρ D 1 ¯ ¯ (yd − η zd − τ wd ) − 2σ T d=1 T K+ 2 (ηk − µ)2 − k=1 1 ω V |τv |, (7) v=1 where K + is the number of nodes in the tree.2 This maximization is performed using L-BFGS [13]. 4 Data: Congress, Products, Films We conduct our experiments using three datasets: Congressional ﬂoor debates, Amazon product reviews, and movie reviews. For all datasets, we remove stopwords, add bigrams to the vocabulary, and ﬁlter the vocabulary using tf-idf.3 • U.S Congressional ﬂoor debates: We downloaded debates of the 109th US Congress from GovTrack4 and preprocessed them as in Thomas et al. [14]. To remove uninterestingly non-polarized debates, we ignore bills with less than 20% “Yea” votes or less than 20% “Nay” votes. Each document d is a turn (a continuous utterance by a single speaker, i.e. speech segment [14]), and its response variable yd is the ﬁrst dimension of the speaker’s DW- NOMINATE score [15], which captures the traditional left-right political distinction.5 After processing, our corpus contains 5,201 turns in the House, 3,060 turns in the Senate, and 5,000 words in the vocabulary.6 • Amazon product reviews: From a set of Amazon reviews of manufactured products such as computers, MP 3 players, GPS devices, etc. [16], we focused on the 50 most frequently reviewed products. After ﬁltering, this corpus contains 37,191 reviews with a vocabulary of 5,000 words. We use the rating associated with each review as the response variable yd .7 • Movie reviews: Our third corpus is a set of 5,006 reviews of movies [17], again using review ratings as the response variable yd , although in this corpus the ratings are normalized to the range from 0 to 1. After preprocessing, the vocabulary contains 5,000 words. 5 Evaluating Prediction S H L DA’s response variable predictions provide a formally rigorous way to assess whether it is an improvement over prior methods. We evaluate effectiveness in predicting values of the response variables for unseen documents in the three datasets. For comparison we consider these baselines: • Multiple linear regression (MLR) models the response variable as a linear function of multiple features (or regressors). Here, we consider two types of features: topic-based features and lexicallybased features. Topic-based MLR, denoted by MLR - LDA, uses the topic distributions learned by vanilla LDA as features [12], while lexically-based MLR, denoted by MLR - VOC, uses the frequencies of words in the vocabulary as features. MLR - LDA - VOC uses both features. • Support vector regression (SVM) is a discriminative method [18] that uses LDA topic distributions (SVM - LDA), word frequencies (SVM - VOC), and both (SVM - LDA - VOC) as features.8 • Supervised topic model (SLDA): we implemented SLDA using Gibbs sampling. The version of SLDA we use is slightly different from the original SLDA described in [12], in that we place a Gaussian prior N (0, 1) over the regression parameters to perform L2-norm regularization.9 For parametric models (LDA and SLDA), which require the number of topics K to be speciﬁed beforehand, we use K ∈ {10, 30, 50}. We use symmetric Dirichlet priors in both LDA and SLDA, initialize The superscript + is to denote that this number is unbounded and varies during the sampling process. To ﬁnd bigrams, we begin with bigram candidates that occur at least 10 times in the corpus and use Pearson’s χ2 -test to ﬁlter out those that have χ2 -value less than 5, which corresponds to a signiﬁcance level of 0.025. We then treat selected bigrams as single word types and add them to the vocabulary. 2 3 4 http://www.govtrack.us/data/us/109/ 5 Scores were downloaded from http://voteview.com/dwnomin_joint_house_and_senate.htm 6 Data will be available after blind review. 7 The ratings can range from 1 to 5, but skew positive. 8 9 http://svmlight.joachims.org/ This performs better than unregularized SLDA in our experiments. 5 Floor Debates House-Senate Senate-House PCC ↑ MSE ↓ PCC ↑ MSE ↓ Amazon Reviews PCC ↑ MSE ↓ Movie Reviews PCC ↑ MSE ↓ SVM - LDA 10 SVM - LDA 30 SVM - LDA 50 SVM - VOC SVM - LDA - VOC 0.173 0.172 0.169 0.336 0.256 0.861 0.840 0.832 1.549 0.784 0.08 0.155 0.215 0.131 0.246 1.247 1.183 1.135 1.467 1.101 0.157 0.277 0.245 0.373 0.371 1.241 1.091 1.130 0.972 0.965 0.327 0.365 0.395 0.584 0.585 0.970 0.938 0.906 0.681 0.678 MLR - LDA 10 MLR - LDA 30 MLR - LDA 50 MLR - VOC MLR - LDA - VOC 0.163 0.160 0.150 0.322 0.319 0.735 0.737 0.741 0.889 0.873 0.068 0.162 0.248 0.191 0.194 1.151 1.125 1.081 1.124 1.120 0.143 0.258 0.234 0.408 0.410 1.034 1.065 1.114 0.869 0.860 0.328 0.367 0.389 0.568 0.581 0.957 0.936 0.914 0.721 0.702 SLDA 10 SLDA 30 SLDA 50 0.154 0.174 0.254 0.729 0.793 0.897 0.090 0.128 0.245 1.145 1.188 1.184 0.270 0.357 0.241 1.113 1.146 1.939 0.383 0.433 0.503 0.953 0.852 0.772 S H L DA 0.356 0.753 0.303 1.076 0.413 0.891 0.597 0.673 Models Table 2: Regression results for Pearson’s correlation coefﬁcient (PCC, higher is better (↑)) and mean squared error (MSE, lower is better (↓)). Results on Amazon product reviews and movie reviews are averaged over 5 folds. Subscripts denote the number of topics for parametric models. For SVM - LDA - VOC and MLR - LDA - VOC, only best results across K ∈ {10, 30, 50} are reported. Best results are in bold. the Dirichlet hyperparameters to 0.5, and use slice sampling [19] for updating hyperparameters. For SLDA , the variance of the regression is set to 0.5. For S H L DA , we use trees with maximum depth of three. We slice sample m, π, β and γ, and ﬁx µ = 0, σ = 0.5, ω = 0.5 and ρ = 0.5. We found that the following set of initial hyperparameters works reasonably well for all the datasets in our experiments: m = 0.5, π = 100, β = (1.0, 0.5, 0.25), γ = (1, 1), α = 1. We also set the regression parameter of the root node to zero, which speeds inference (since it is associated with every document) and because it is reasonable to assume that it would not change the response variable. To compare the performance of different methods, we compute Pearson’s correlation coefﬁcient (PCC) and mean squared error (MSE) between the true and predicted values of the response variables and average over 5 folds. For the Congressional debate corpus, following Yu et al. [20], we use documents in the House to train and test on documents in the Senate and vice versa. Results and analysis Table 2 shows the performance of all models on our three datasets. Methods that only use topic-based features such as SVM - LDA and MLR - LDA do poorly. Methods only based on lexical features like SVM - VOC and MLR - VOC outperform methods that are based only on topic features signiﬁcantly for the two review datasets, but are comparable or worse on congressional debates. This suggests that reviews have more highly discriminative words than political speeches (Table 3). Combining topic-based and lexically-based features improves performance, which supports our choice of incorporating both per-topic and per-word regression parameters in S H L DA. In all cases, S H L DA achieves strong performance results. For the two cases where S H L DA was second best in MSE score (Amazon reviews and House-Senate), it outperforms other methods in PCC. Doing well in PCC for these two datasets is important since achieving low MSE is relatively easier due to the response variables’ bimodal distribution in the ﬂoor debates and positively-skewed distribution in Amazon reviews. For the ﬂoor debate dataset, the results of the House-Senate experiment are generally better than those of the Senate-House experiment, which is consistent with previous results [20] and is explained by the greater number of debates in the House. 6 Qualitative Analysis: Agendas and Framing/Perspective Although a formal coherence evaluation [21] remains a goal for future work, a qualitative look at the topic hierarchy uncovered by the model suggests that it is indeed capturing agenda/framing structure as discussed in Section 1. In Figure 3, a portion of the topic hierarchy induced from the Congressional debate corpus, Nodes A and B illustrate agendas—issues introduced into political discourse—associated with a particular ideology: Node A focuses on the hardships of the poorer victims of hurricane Katrina and is associated with Democrats, and text associated with Node E discusses a proposed constitutional amendment to ban ﬂag burning and is associated with Republicans. Nodes C and D, children of a neutral “tax” topic, reveal how parties frame taxes as gains in terms of new social services (Democrats) and losses for job creators (Republicans). 6 E ﬂag constitution freedom supreme_court elections rights continuity american_ﬂag constitutional_amendm ent gses credit_rating fannie_mae regulator freddie_mac market ﬁnancial_services agencies competition investors fannie bill speaker time amendment chairman people gentleman legislation congress support R:1.1 R:0 A minimum_wage commission independent_commissio n investigate hurricane_katrina increase investigation R:1.0 B percent tax economy estate_tax capital_gains money taxes businesses families tax_cuts pay tax_relief social_security affordable_housing housing manager fund activities funds organizations voter_registration faithbased nonproﬁts R:0.4 D:1.7 C death_tax jobs businesses business family_businesses equipment productivity repeal_permanency employees capital farms D REPUBLICAN billion budget children cuts debt tax_cuts child_support deﬁcit education students health_care republicans national_debt R:4.3 D:2.2 DEMOCRAT D:4.5 Figure 3: Topics discovered from Congressional ﬂoor debates. Many ﬁrst-level topics are bipartisan (purple), while lower level topics are associated with speciﬁc ideologies (Democrats blue, Republicans red). For example, the “tax” topic (B) is bipartisan, but its Democratic-leaning child (D) focuses on social goals supported by taxes (“children”, “education”, “health care”), while its Republican-leaning child (C) focuses on business implications (“death tax”, “jobs”, “businesses”). The number below each topic denotes the magnitude of the learned regression parameter associated with that topic. Colors and the numbers beneath each topic show the regression parameter η associated with the topic. Figure 4 shows the topic structure discovered by S H L DA in the review corpus. Nodes at higher levels are relatively neutral, with relatively small regression parameters.10 These nodes have general topics with no speciﬁc polarity. However, the bottom level clearly illustrates polarized positive/negative perspective. For example, Node A concerns washbasins for infants, and has two polarized children nodes: reviewers take a positive perspective when their children enjoy the product (Node B: “loves”, “splash”, “play”) but have negative reactions when it leaks (Node C: “leak(s/ed/ing)”). transmitter ipod car frequency iriver product transmitters live station presets itrip iriver_aft charges international_mode driving P:6.6 tried waste batteries tunecast rabbit_ears weak terrible antenna hear returned refund returning item junk return A D router setup network expander set signal wireless connect linksys connection house wireless_router laptop computer wre54g N:2.2 N:1.0 tivo adapter series adapters phone_line tivo_wireless transfer plugged wireless_adapter tivos plug dvr tivo_series tivo_box tivo_unit P:5.1 tub baby water bath sling son daughter sit bathtub sink newborn months bath_tub bathe bottom N:8.0 months loves hammock splash love baby drain eurobath hot ﬁts wash play infant secure slip P:7.5 NEGATIVE N:0 N:2.7 B POSITIVE time bought product easy buy love using price lot able set found purchased money months transmitter car static ipod radio mp3_player signal station sound music sound_quality volume stations frequency frequencies C leaks leaked leak leaking hard waste snap suction_cups lock tabs difﬁcult bottom tub_leaks properly ring N:8.9 monitor radio weather_radio night baby range alerts sound sony house interference channels receiver static alarm N:1.7 hear feature static monitors set live warning volume counties noise outside alert breathing rechargeable_battery alerts P:6.2 version hours phone F ﬁrmware told spent linksys tech_support technical_supportcusto mer_service range_expander support return N:10.6 E router ﬁrmware ddwrt wrt54gl version wrt54g tomato linksys linux routers ﬂash versions browser dlink stable P:4.8 z22 palm pda palm_z22 calendar software screen contacts computer device sync information outlook data programs N:1.9 headphones sound pair bass headset sound_quality ear ears cord earbuds comfortable hear head earphones ﬁt N:1.3 appointments organized phone lists handheld organizer photos etc pictures memos track bells books purse whistles P:5.8 noise_canceling noise sony exposed noise_cancellation stopped wires warranty noise_cancelling bud pay white_noise disappointed N:7.6 bottles bottle baby leak nipples nipple avent avent_bottles leaking son daughter formula leaks gas milk comfortable sound phones sennheiser bass px100 px100s phone headset highs portapros portapro price wear koss N:2.0 leak formula bottles_leak feeding leaked brown frustrating started clothes waste newborn playtex_ventaire soaked matter N:7.9 P:5.7 nipple breast nipples dishwasher ring sippy_cups tried breastfeed screwed breastfeeding nipple_confusion avent_system bottle P:6.4 Figure 4: Topics discovered from Amazon reviews. Higher topics are general, while lower topics are more speciﬁc. The polarity of the review is encoded in the color: red (negative) to blue (positive). Many of the ﬁrstlevel topics have no speciﬁc polarity and are associated with a broad class of products such as “routers” (Node D). However, the lowest topics in the hierarchy are often polarized; one child topic of “router” focuses on upgradable ﬁrmware such as “tomato” and “ddwrt” (Node E, positive) while another focuses on poor “tech support” and “customer service” (Node F, negative). The number below each topic is the regression parameter learned with that topic. In addition to the per-topic regression parameters, S H L DA also associates each word with a lexical regression parameter τ . Table 3 shows the top ten words with highest and lowest τ . The results are unsuprising, although the lexical regression for the Congressional debates is less clear-cut than other 10 All of the nodes at the second level have slightly negative values for the regression parameters mainly due to the very skewed distribution of the review ratings in Amazon. 7 datasets. As we saw in Section 5, for similar datasets, S H L DA’s context-speciﬁc regression is more useful when global lexical weights do not readily differentiate documents. Dataset Floor Debates Amazon Reviews Movie Reviews Top 10 words with positive weights bringing, private property, illegally, tax relief, regulation, mandates, constitutional, committee report, illegal alien highly recommend, pleased, love, loves, perfect, easy, excellent, amazing, glad, happy hilarious, fast, schindler, excellent, motion pictures, academy award, perfect, journey, fortunately, ability Top 10 words with negative weights bush administration, strong opposition, ranking, republicans, republican leadership, secret, discriminate, majority, undermine waste, returned, return, stopped, leak, junk, useless, returning, refund, terrible bad, unfortunately, supposed, waste, mess, worst, acceptable, awful, suppose, boring Table 3: Top words based on the global lexical regression coefﬁcient, τ . For the ﬂoor debates, positive τ ’s are Republican-leaning while negative τ ’s are Democrat-leaning. 7 Related Work S H L DA joins a family of LDA extensions that introduce hierarchical topics, supervision, or both. Owing to limited space, we focus here on related work that combines the two. Petinot et al. [22] propose hierarchical Labeled LDA (hLLDA), which leverages an observed document ontology to learn topics in a tree structure; however, hLLDA assumes that the underlying tree structure is known a priori. SSHLDA [23] generalizes hLLDA by allowing the document hierarchy labels to be partially observed, with unobserved labels and topic tree structure then inferred from the data. Boyd-Graber and Resnik [24] used hierarchical distributions within topics to learn topics across languages. In addition to these “upstream” models [25], Perotte et al. [26] propose a “downstream” model called HSLDA , which jointly models documents’ hierarchy of labels and topics. HSLDA ’s topic structure is ﬂat, however, and the response variable is a hierarchy of labels associated with each document, unlike S H L DA’s continuous response variable. Finally, another body related body of work includes models that jointly capture topics and other facets such as ideologies/perspectives [27, 28] and sentiments/opinions [29], albeit with discrete rather than continuously valued responses. Computational modeling of sentiment polarity is a voluminous ﬁeld [30], and many computational political science models describe agendas [5] and ideology [31]. Looking at framing or bias at the sentence level, Greene and Resnik [32] investigate the role of syntactic structure in framing, Yano et al. [33] look at lexical indications of sentence-level bias, and Recasens et al. [34] develop linguistically informed sentence-level features for identifying bias-inducing words. 8 Conclusion We have introduced S H L DA, a model that associates a continuously valued response variable with hierarchical topics to capture both the issues under discussion and alternative perspectives on those issues. The two-level structure improves predictive performance over existing models on multiple datasets, while also adding potentially insightful hierarchical structure to the topic analysis. Based on a preliminary qualitative analysis, the topic hierarchy exposed by the model plausibly captures the idea of agenda setting, which is related to the issues that get discussed, and framing, which is related to authors’ perspectives on those issues. We plan to analyze the topic structure produced by S H L DA with political science collaborators and more generally to study how S H L DA and related models can help analyze and discover useful insights from political discourse. Acknowledgments This research was supported in part by NSF under grant #1211153 (Resnik) and #1018625 (BoydGraber and Resnik). Any opinions, ﬁndings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reﬂect the view of the sponsor. 8 References [1] McCombs, M. The agenda-setting role of the mass media in the shaping of public opinion. North, 2009(05-12):21, 2002. [2] McCombs, M., S. Ghanem. The convergence of agenda setting and framing. In Framing public life. 2001. [3] Baumgartner, F. R., S. L. De Boef, A. E. Boydstun. The decline of the death penalty and the discovery of innocence. Cambridge University Press, 2008. [4] Blei, D. M., A. Ng, M. Jordan. Latent Dirichlet allocation. JMLR, 3, 2003. [5] Grimmer, J. A Bayesian hierarchical topic model for political texts: Measuring expressed agendas in Senate press releases. Political Analysis, 18(1):1–35, 2010. [6] Zhang, J. Explore objects and categories in unexplored environments based on multimodal data. Ph.D. thesis, University of Hamburg, 2012. [7] Blei, D. M., T. L. Grifﬁths, M. I. Jordan. The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. J. ACM, 57(2), 2010. [8] Teh, Y. W., M. I. Jordan, M. J. Beal, et al. Hierarchical Dirichlet processes. JASA, 101(476), 2006. [9] Paisley, J. W., C. Wang, D. M. Blei, et al. Nested hierarchical Dirichlet processes. arXiv:1210.6738, 2012. [10] Ahmed, A., L. Hong, A. Smola. The nested Chinese restaurant franchise process: User tracking and document modeling. In ICML. 2013. [11] Kim, J. H., D. Kim, S. Kim, et al. Modeling topic hierarchies with the recursive Chinese restaurant process. In CIKM, pages 783–792. 2012. [12] Blei, D. M., J. D. McAuliffe. Supervised topic models. In NIPS. 2007. [13] Liu, D., J. Nocedal. On the limited memory BFGS method for large scale optimization. Math. Prog., 1989. [14] Thomas, M., B. Pang, L. Lee. Get out the vote: Determining support or opposition from Congressional ﬂoor-debate transcripts. In EMNLP. 2006. [15] Lewis, J. B., K. T. Poole. Measuring bias and uncertainty in ideal point estimates via the parametric bootstrap. Political Analysis, 12(2), 2004. [16] Jindal, N., B. Liu. Opinion spam and analysis. In WSDM. 2008. [17] Pang, B., L. Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL. 2005. [18] Joachims, T. Making large-scale SVM learning practical. In Adv. in Kernel Methods - SVM. 1999. [19] Neal, R. M. Slice sampling. Annals of Statistics, 31:705–767, 2003. [20] Yu, B., D. Diermeier, S. Kaufmann. Classifying party afﬁliation from political speech. JITP, 2008. [21] Chang, J., J. Boyd-Graber, C. Wang, et al. Reading tea leaves: How humans interpret topic models. In NIPS. 2009. [22] Petinot, Y., K. McKeown, K. Thadani. A hierarchical model of web summaries. In HLT. 2011. [23] Mao, X., Z. Ming, T.-S. Chua, et al. SSHLDA: A semi-supervised hierarchical topic model. In EMNLP. 2012. [24] Boyd-Graber, J., P. Resnik. Holistic sentiment analysis across languages: Multilingual supervised latent Dirichlet allocation. In EMNLP. 2010. [25] Mimno, D. M., A. McCallum. Topic models conditioned on arbitrary features with Dirichlet-multinomial regression. In UAI. 2008. [26] Perotte, A. J., F. Wood, N. Elhadad, et al. Hierarchically supervised latent Dirichlet allocation. In NIPS. 2011. [27] Ahmed, A., E. P. Xing. Staying informed: Supervised and semi-supervised multi-view topical analysis of ideological perspective. In EMNLP. 2010. [28] Eisenstein, J., A. Ahmed, E. P. Xing. Sparse additive generative models of text. In ICML. 2011. [29] Jo, Y., A. H. Oh. Aspect and sentiment uniﬁcation model for online review analysis. In WSDM. 2011. [30] Pang, B., L. Lee. Opinion Mining and Sentiment Analysis. Now Publishers Inc, 2008. [31] Monroe, B. L., M. P. Colaresi, K. M. Quinn. Fightin’words: Lexical feature selection and evaluation for identifying the content of political conﬂict. Political Analysis, 16(4):372–403, 2008. [32] Greene, S., P. Resnik. More than words: Syntactic packaging and implicit sentiment. In NAACL. 2009. [33] Yano, T., P. Resnik, N. A. Smith. Shedding (a thousand points of) light on biased language. In NAACL-HLT Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. 2010. [34] Recasens, M., C. Danescu-Niculescu-Mizil, D. Jurafsky. Linguistic models for analyzing and detecting biased language. In ACL. 2013. 9

6 0.11748625 173 nips-2013-Least Informative Dimensions

7 0.10366781 301 nips-2013-Sparse Additive Text Models with Low Rank Background

8 0.097602457 237 nips-2013-Optimal integration of visual speed across different spatiotemporal frequency channels

9 0.092453912 247 nips-2013-Phase Retrieval using Alternating Minimization

10 0.09175837 345 nips-2013-Variance Reduction for Stochastic Gradient Optimization

11 0.091490813 262 nips-2013-Real-Time Inference for a Gamma Process Model of Neural Spiking

12 0.086409599 59 nips-2013-Blind Calibration in Compressed Sensing using Message Passing Algorithms

13 0.08356902 65 nips-2013-Compressive Feature Learning

14 0.072364584 98 nips-2013-Documents as multiple overlapping windows into grids of counts

15 0.072255582 9 nips-2013-A Kernel Test for Three-Variable Interactions

16 0.069379114 4 nips-2013-A Comparative Framework for Preconditioned Lasso Algorithms

17 0.069346577 236 nips-2013-Optimal Neural Population Codes for High-dimensional Stimulus Variables

18 0.064976886 51 nips-2013-Bayesian entropy estimation for binary spike train data using parametric prior knowledge

19 0.064546905 353 nips-2013-When are Overcomplete Topic Models Identifiable? Uniqueness of Tensor Tucker Decompositions with Structured Sparsity

20 0.062716477 5 nips-2013-A Deep Architecture for Matching Short Texts

similar papers computed by lsi model

lsi for this paper:

topicId topicWeight

[(0, 0.181), (1, 0.084), (2, -0.029), (3, 0.014), (4, -0.052), (5, 0.029), (6, 0.034), (7, 0.023), (8, 0.021), (9, 0.034), (10, -0.026), (11, 0.066), (12, -0.081), (13, -0.078), (14, -0.012), (15, -0.102), (16, 0.182), (17, 0.052), (18, -0.085), (19, -0.015), (20, -0.103), (21, -0.003), (22, -0.025), (23, -0.052), (24, 0.018), (25, -0.12), (26, 0.026), (27, 0.031), (28, -0.019), (29, -0.047), (30, 0.086), (31, 0.024), (32, -0.056), (33, 0.083), (34, 0.08), (35, -0.043), (36, 0.13), (37, -0.023), (38, -0.069), (39, -0.069), (40, 0.026), (41, -0.014), (42, -0.047), (43, -0.09), (44, 0.08), (45, -0.02), (46, 0.127), (47, -0.085), (48, -0.058), (49, -0.142)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.92877859 88 nips-2013-Designed Measurements for Vector Count Data

Author: Liming Wang, David Carlson, Miguel Rodrigues, David Wilcox, Robert Calderbank, Lawrence Carin

2 0.59915036 174 nips-2013-Lexical and Hierarchical Topic Regression

Author: Viet-An Nguyen, Jordan Boyd-Graber, Philip Resnik

3 0.56549764 65 nips-2013-Compressive Feature Learning

Author: Hristo S. Paskov, Robert West, John C. Mitchell, Trevor Hastie

Abstract: This paper addresses the problem of unsupervised feature learning for text data. Our method is grounded in the principle of minimum description length and uses a dictionary-based compression scheme to extract a succinct feature set. Specifically, our method ﬁnds a set of word k-grams that minimizes the cost of reconstructing the text losslessly. We formulate document compression as a binary optimization task and show how to solve it approximately via a sequence of reweighted linear programs that are efﬁcient to solve and parallelizable. As our method is unsupervised, features may be extracted once and subsequently used in a variety of tasks. We demonstrate the performance of these features over a range of scenarios including unsupervised exploratory analysis and supervised text categorization. Our compressed feature space is two orders of magnitude smaller than the full k-gram space and matches the text categorization accuracy achieved in the full feature space. This dimensionality reduction not only results in faster training times, but it can also help elucidate structure in unsupervised learning tasks and reduce the amount of training data necessary for supervised learning. 1

4 0.55504477 70 nips-2013-Contrastive Learning Using Spectral Methods

Author: James Y. Zou, Daniel Hsu, David C. Parkes, Ryan P. Adams

Abstract: In many natural settings, the analysis goal is not to characterize a single data set in isolation, but rather to understand the difference between one set of observations and another. For example, given a background corpus of news articles together with writings of a particular author, one may want a topic model that explains word patterns and themes speciﬁc to the author. Another example comes from genomics, in which biological signals may be collected from different regions of a genome, and one wants a model that captures the differential statistics observed in these regions. This paper formalizes this notion of contrastive learning for mixture models, and develops spectral algorithms for inferring mixture components speciﬁc to a foreground data set when contrasted with a background data set. The method builds on recent moment-based estimators and tensor decompositions for latent variable models, and has the intuitive feature of using background data statistics to appropriately modify moments estimated from foreground data. A key advantage of the method is that the background data need only be coarsely modeled, which is important when the background is too complex, noisy, or not of interest. The method is demonstrated on applications in contrastive topic modeling and genomic sequence analysis. 1

5 0.54012382 98 nips-2013-Documents as multiple overlapping windows into grids of counts

Author: Alessandro Perina, Nebojsa Jojic, Manuele Bicego, Andrzej Truski

Abstract: In text analysis documents are often represented as disorganized bags of words; models of such count features are typically based on mixing a small number of topics [1, 2]. Recently, it has been observed that for many text corpora documents evolve into one another in a smooth way, with some features dropping and new ones being introduced. The counting grid [3] models this spatial metaphor literally: it is a grid of word distributions learned in such a way that a document’s own distribution of features can be modeled as the sum of the histograms found in a window into the grid. The major drawback of this method is that it is essentially a mixture and all the content must be generated by a single contiguous area on the grid. This may be problematic especially for lower dimensional grids. In this paper, we overcome this issue by introducing the Componential Counting Grid which brings the componential nature of topic models to the basic counting grid. We evaluated our approach on document classiﬁcation and multimodal retrieval obtaining state of the art results on standard benchmarks. 1

6 0.53922373 301 nips-2013-Sparse Additive Text Models with Low Rank Background

7 0.53160751 287 nips-2013-Scalable Inference for Logistic-Normal Topic Models

8 0.52071553 217 nips-2013-On Poisson Graphical Models

9 0.5176385 59 nips-2013-Blind Calibration in Compressed Sensing using Message Passing Algorithms

10 0.51447338 304 nips-2013-Sparse nonnegative deconvolution for compressive calcium imaging: algorithms and phase transitions

11 0.50159454 247 nips-2013-Phase Retrieval using Alternating Minimization

12 0.47798285 353 nips-2013-When are Overcomplete Topic Models Identifiable? Uniqueness of Tensor Tucker Decompositions with Structured Sparsity

13 0.45794719 274 nips-2013-Relevance Topic Model for Unstructured Social Group Activity Recognition

14 0.44627094 317 nips-2013-Streaming Variational Bayes

15 0.44276664 305 nips-2013-Spectral methods for neural characterization using generalized quadratic models

16 0.43378726 265 nips-2013-Reconciling "priors" & "priors" without prejudice?

17 0.42797253 354 nips-2013-When in Doubt, SWAP: High-Dimensional Sparse Recovery from Correlated Measurements

18 0.41964248 345 nips-2013-Variance Reduction for Stochastic Gradient Optimization

19 0.41673434 312 nips-2013-Stochastic Gradient Riemannian Langevin Dynamics on the Probability Simplex

20 0.4153125 173 nips-2013-Least Informative Dimensions

similar papers computed by lda model

lda for this paper:

topicId topicWeight

[(16, 0.029), (33, 0.602), (34, 0.064), (41, 0.026), (49, 0.034), (56, 0.048), (70, 0.012), (85, 0.021), (86, 0.025), (89, 0.029), (93, 0.042)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.99280965 88 nips-2013-Designed Measurements for Vector Count Data

Author: Liming Wang, David Carlson, Miguel Rodrigues, David Wilcox, Robert Calderbank, Lawrence Carin

2 0.98934251 81 nips-2013-DeViSE: A Deep Visual-Semantic Embedding Model

Author: Andrea Frome, Greg S. Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc'Aurelio Ranzato, Tomas Mikolov

Abstract: Modern visual recognition systems are often limited in their ability to scale to large numbers of object categories. This limitation is in part due to the increasing difﬁculty of acquiring sufﬁcient training data in the form of labeled images as the number of object categories grows. One remedy is to leverage data from other sources – such as text data – both to train visual models and to constrain their predictions. In this paper we present a new deep visual-semantic embedding model trained to identify visual objects using both labeled image data as well as semantic information gleaned from unannotated text. We demonstrate that this model matches state-of-the-art performance on the 1000-class ImageNet object recognition challenge while making more semantically reasonable errors, and also show that the semantic information can be exploited to make predictions about tens of thousands of image labels not observed during training. Semantic knowledge improves such zero-shot predictions achieving hit rates of up to 18% across thousands of novel labels never seen by the visual model. 1

3 0.98772317 217 nips-2013-On Poisson Graphical Models

Author: Eunho Yang, Pradeep Ravikumar, Genevera I. Allen, Zhandong Liu

4 0.98223156 306 nips-2013-Speeding up Permutation Testing in Neuroimaging

Author: Chris Hinrichs, Vamsi Ithapu, Qinyuan Sun, Sterling C. Johnson, Vikas Singh

Abstract: Multiple hypothesis testing is a signiﬁcant problem in nearly all neuroimaging studies. In order to correct for this phenomena, we require a reliable estimate of the Family-Wise Error Rate (FWER). The well known Bonferroni correction method, while simple to implement, is quite conservative, and can substantially under-power a study because it ignores dependencies between test statistics. Permutation testing, on the other hand, is an exact, non-parametric method of estimating the FWER for a given α-threshold, but for acceptably low thresholds the computational burden can be prohibitive. In this paper, we show that permutation testing in fact amounts to populating the columns of a very large matrix P. By analyzing the spectrum of this matrix, under certain conditions, we see that P has a low-rank plus a low-variance residual decomposition which makes it suitable for highly sub–sampled — on the order of 0.5% — matrix completion methods. Based on this observation, we propose a novel permutation testing methodology which offers a large speedup, without sacriﬁcing the ﬁdelity of the estimated FWER. Our evaluations on four different neuroimaging datasets show that a computational speedup factor of roughly 50× can be achieved while recovering the FWER distribution up to very high accuracy. Further, we show that the estimated α-threshold is also recovered faithfully, and is stable. 1

5 0.97688252 160 nips-2013-Learning Stochastic Feedforward Neural Networks

Author: Yichuan Tang, Ruslan Salakhutdinov

Abstract: Multilayer perceptrons (MLPs) or neural networks are popular models used for nonlinear regression and classiﬁcation tasks. As regressors, MLPs model the conditional distribution of the predictor variables Y given the input variables X. However, this predictive distribution is assumed to be unimodal (e.g. Gaussian). For tasks involving structured prediction, the conditional distribution should be multi-modal, resulting in one-to-many mappings. By using stochastic hidden variables rather than deterministic ones, Sigmoid Belief Nets (SBNs) can induce a rich multimodal distribution in the output space. However, previously proposed learning algorithms for SBNs are not efﬁcient and unsuitable for modeling real-valued data. In this paper, we propose a stochastic feedforward network with hidden layers composed of both deterministic and stochastic variables. A new Generalized EM training procedure using importance sampling allows us to efﬁciently learn complicated conditional distributions. Our model achieves superior performance on synthetic and facial expressions datasets compared to conditional Restricted Boltzmann Machines and Mixture Density Networks. In addition, the latent features of our model improves classiﬁcation and can learn to generate colorful textures of objects. 1

6 0.96902597 46 nips-2013-Bayesian Estimation of Latently-grouped Parameters in Undirected Graphical Models

7 0.96459311 212 nips-2013-Non-Uniform Camera Shake Removal Using a Spatially-Adaptive Sparse Penalty

8 0.95377105 253 nips-2013-Prior-free and prior-dependent regret bounds for Thompson Sampling

9 0.9491623 222 nips-2013-On the Linear Convergence of the Proximal Gradient Method for Trace Norm Regularization

10 0.92956245 356 nips-2013-Zero-Shot Learning Through Cross-Modal Transfer

11 0.91147155 93 nips-2013-Discriminative Transfer Learning with Tree-based Priors

12 0.89110982 200 nips-2013-Multi-Prediction Deep Boltzmann Machines

13 0.88761961 334 nips-2013-Training and Analysing Deep Recurrent Neural Networks

14 0.88678253 67 nips-2013-Conditional Random Fields via Univariate Exponential Families

15 0.88637465 260 nips-2013-RNADE: The real-valued neural autoregressive density-estimator

16 0.88482404 254 nips-2013-Probabilistic Low-Rank Matrix Completion with Adaptive Spectral Regularization Algorithms

17 0.88099289 349 nips-2013-Visual Concept Learning: Combining Machine Vision and Bayesian Generalization on Concept Hierarchies

18 0.87709349 315 nips-2013-Stochastic Ratio Matching of RBMs for Sparse High-Dimensional Inputs

19 0.87540984 331 nips-2013-Top-Down Regularization of Deep Belief Networks

20 0.87372428 335 nips-2013-Transfer Learning in a Transductive Setting