acl acl2013 acl2013-28 knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Reut Tsarfaty
Abstract: Stanford Dependencies (SD) provide a functional characterization of the grammatical relations in syntactic parse-trees. The SD representation is useful for parser evaluation, for downstream applications, and, ultimately, for natural language understanding, however, the design of SD focuses on structurally-marked relations and under-represents morphosyntactic realization patterns observed in Morphologically Rich Languages (MRLs). We present a novel extension of SD, called Unified-SD (U-SD), which unifies the annotation of structurally- and morphologically-marked relations via an inheritance hierarchy. We create a new resource composed of U-SDannotated constituency and dependency treebanks for the MRL Modern Hebrew, and present two systems that can automatically predict U-SD annotations, for gold segmented input as well as raw texts, with high baseline accuracy.
Reference: text
sentIndex sentText sentNum sentScore
1 Abstract Stanford Dependencies (SD) provide a functional characterization of the grammatical relations in syntactic parse-trees. [sent-2, score-0.332]
2 We present a novel extension of SD, called Unified-SD (U-SD), which unifies the annotation of structurally- and morphologically-marked relations via an inheritance hierarchy. [sent-4, score-0.26]
3 We create a new resource composed of U-SDannotated constituency and dependency treebanks for the MRL Modern Hebrew, and present two systems that can automatically predict U-SD annotations, for gold segmented input as well as raw texts, with high baseline accuracy. [sent-5, score-0.26]
4 1 Introduction Stanford Dependencies (SD) provide a functional characterization of the grammatical relations in syntactic trees, capturing the predicate-argument structure of natural language sentences (de Marneffe et al. [sent-6, score-0.332]
5 , 2007), and in recent years SD structures have also become a defacto standard for parser evaluation in English (de Marneffe and Manning, 2008a; Cer et al. [sent-10, score-0.096]
6 Efforts now commence towards extending SD for cross-lingual annotation l ingfi l . [sent-13, score-0.037]
7 However, the original SD design emphasizes word-tokens and configurational structures, and consequently, these schemes overlook properties and realization patterns observed in a range of languages known as Morphologically Rich Languages (MRLs) (Tsarfaty et al. [sent-20, score-0.142]
8 MRLs use word-level affixes to express grammatical relations that are typically indicated by structural positions in English. [sent-22, score-0.197]
9 By virtue of word-level morphological marking, word-order in MRLs may be flexible. [sent-23, score-0.118]
10 1 Here we argue that the SD hierarchy and design principles similarly emphasize English-like structures and underrepresent morphosyntactic argument-marking alternatives. [sent-25, score-0.194]
11 We define an extension of SD, called Unified-SD (U-SD), which unifies the annotation of structurally and morphologically marked relations via an inheritance hierarchy. [sent-26, score-0.34]
12 We extend SD with a functional branch, and provide a principled treatment of morpho-syntactic argument marking. [sent-27, score-0.138]
13 Based on the U-SD scheme we create a new parallel resource for the MRL Modern Hebrew, whereby aligned constituency and dependency trees reflect equivalent U-SD annotations (cf. [sent-28, score-0.41]
14 We present two systems that can automatically learn U-SD annotations, from the dependency and the constituency versions respectively, delivering high baseline accuracy on the prediction task. [sent-30, score-0.168]
15 At the same time, the scheme is designed with end-users in mind, allow- ing them to utilize parser output in a form which is intuitively interpretable and easily processed. [sent-42, score-0.155]
16 SD basic trees represent sentences as binary relations between word tokens. [sent-43, score-0.165]
17 These relations are labeled using traditional grammatical concepts (subject, object, modifier) that are arranged into an inheritance hierarchy (de Marneffe and Manning, 2008a, Sec. [sent-44, score-0.291]
18 The SD scheme defines a core set of labels and principles which are assumed to be useful for different languages. [sent-47, score-0.202]
19 However, a close examination of the SD label-set and inheritance hierarchy reveals that some of its design principles are geared towards English-like (that is, configurational) phenomena, and conflict with basic properties of MRLs. [sent-48, score-0.267]
20 Let us list three such design principles and outline the challenges that they pose. [sent-49, score-0.043]
21 One or more morphemes may be appended to a content word, and several morphemes may be contained in a single space-delimited token. [sent-54, score-0.243]
22 For example, the Hebrew token wkfraiti2 in (1) includes the morphemes w (and), kf (when) and raiti (saw); the latter segment is a content word, and the former two are functional morphemes. [sent-55, score-0.279]
23 S ”NJoP-hsnb”j Vlo-vpersd”VPN”MP-aorby”jN”DdPa -n s”bj Vol h-pver vds”A CeNtC-Pd-Daonba nj”a Figure 1: English (a) and Hebrew (b) PS trees decorated with function labels as dash features. [sent-67, score-0.206]
24 Na¨ ıvely taking input tokens as words fails to capture meaningful relations between morphological segments internal to space-delimited tokens. [sent-68, score-0.269]
25 Configurational languages like English use function words such as prepositions and auxiliaries to indicate relations between content words and to mark properties of complete structures. [sent-72, score-0.243]
26 In MRLs, such relations and properties may be indicated by word-level morphological marking such as case (Blake, 1994) and agreement (Corbett, 2006). [sent-73, score-0.342]
27 In (1), for instance, the case marker at indicates an accusative object relation between “see” and “movie”, to be distinguished from, e. [sent-74, score-0.193]
28 While the original SD scheme label-set covers function words (e. [sent-77, score-0.115]
29 auxpass, expl, prep), it misses labels for bound morphemes that mark grammatical relations across languages (such as accusative, dative or genitive). [sent-79, score-0.349]
30 Explicit labeling of such relational morphemes will allow us to benefit from the information that they provide. [sent-80, score-0.102]
31 SD relations are extracted from different types of trees for the purpose of, e. [sent-84, score-0.165]
32 Insofar, recovering SD relations from phrase-structure (PS) trees have used a range of structural cues such as positions and phrase-labels (see, for instance, the software of de Marneffe and Manning (2008a)). [sent-88, score-0.252]
33 In MRLs, positions and phrase types may not suffice for recovering SD relations: an NP under S in Hebrew, for instance, may be a subject or an object, as shown in Figure 1, and morphological information then determines the function of these constituents. [sent-89, score-0.118]
34 Automatically inferring predicate-argument structures across treebanks thus must rely on both structural and morphological marking, calling for a single annotation scheme that inter-relate the marking alternatives. [sent-90, score-0.47]
35 /bVDEjF S-root root root Figure 2: Sample U-SD Trees for sentence (1). [sent-96, score-0.162]
36 (a) a phrase-structure tree decorated with U-SD labels, (b) a basic U-SD tree, and (c) a collapsed USD tree, where functional nodes are consumed. [sent-97, score-0.239]
37 3 The Proposal: Unified-SD (U-SD) To address these challenges, we propose an extension of SD called Unified-SD (U-SD) which annotates relations between morphological segments and reflects different types of argument-marking patterns. [sent-98, score-0.269]
38 The SD ontology is re-organized and extended to allow us to annotate morphologicallyand structurally-marked relations alike. [sent-99, score-0.143]
39 sm is a morphological analysis (fwunction that identifies all morphological segments of a sentence S = w1. [sent-108, score-0.302]
40 The U-SD scheme provides the syntactic representation of S by means of a set of triplets (l, si, sj) consisting of a label l, a head si and a dependent sj (i j). [sent-112, score-0.173]
41 Everything in the ontology is of type gf (grammatical function). [sent-119, score-0.095]
42 The dep branch is used for dependent types, and it retains much of the structure in the original SD scheme (separating sbj types, obj types, mod types, etc. [sent-122, score-0.51]
43 The new func branch contains argument-marking elements, that is, function words and morphemes that play a role in indicating properties or grammatical relations in the syntactic representation. [sent-124, score-0.642]
44 These functional elements may be of types marker (prepositions and case), aux (auxiliary verbs and copular elements) and sub (subordination/conjunction markers). [sent-125, score-0.422]
45 All inherited func elements may be consumed (henceforth, collapsed) in order to infer grammatical properties and relations between content words. [sent-126, score-0.566]
46 Head types are implicit in dependency triplets, however, when decorating PS trees with dependency labels as dash features or edge features (as in TigerXML formats (Brants et al. [sent-127, score-0.317]
47 , 2002) or via unificationbased formalisms) both heads and dependents are labeled with their grammatical types (see Figure 2(a)). [sent-128, score-0.063]
48 The hd branch extends the scheme with an inventory of argument-taking elements, to be used when employing SD inside constituency treebanks. [sent-129, score-0.419]
49 The punct branch is reserved for punctuation, prosody and other non-verbal speech acts. [sent-130, score-0.177]
50 Anderson (1992) delineates three kinds of properties that are realized by morphology: structural, inherent, and agreement properties. [sent-133, score-0.081]
51 , case) are marked on a content word to indicate its rela580 SegmentsFunctions Gold:DRERP11. [sent-136, score-0.039]
52 87181370 Table 1: Inferring U-SD trees using different frameworks. [sent-148, score-0.08]
53 Agreement properties indicate the semantic properties of nominals on top of other elements (verbs, adjectives, etc. [sent-153, score-0.26]
54 We define annotation guidelines that reflect these different properties. [sent-155, score-0.037]
55 Structural morphemes (case) connect words in the arc-structure, linking a head to its semantic dependent, like the case marker “at”-ACC in Figure 2(b). [sent-156, score-0.239]
56 Inherent / agreement properties are annotated as dependents of the content word they add properties to, for instance, the prefixes def in Figure 2(b) hang under the modified noun and adjective. [sent-157, score-0.238]
57 Collapsed U-SD structures interpret func elements in order to refine the representation of relations between content words. [sent-158, score-0.478]
58 Case markers can be used for refining the relation between the content words they connect by labeling their direct relation, much like prep in the original SD scheme (see, e. [sent-159, score-0.262]
59 Inherent/agreement features are in fact features of their respective head word (as the X. [sent-162, score-0.058]
60 The revised USD ontology provides a typological inventory of labels that describe different types of argu- ments (dep), argument-taking elements (hd), and argument-marking elements (func) in the grammar of different languages. [sent-166, score-0.298]
61 , morphological markers of particular types, are daughters within more specific branches. [sent-169, score-0.165]
62 4 Automatic Annotation of U-SD Trees Can U-SD structures be automatically predicted? [sent-173, score-0.056]
63 For MRLs, this requires disambiguating both morphological and syntactic information. [sent-174, score-0.118]
64 Here we employ the U-SD scheme for annotating morphosyntactic structures in Modern Hebrew, and use these resources to train two systems that pre- dict U-SD annotations for raw texts. [sent-175, score-0.31]
65 , 2001), a corpus of 6220 sentences morphologically segmented and syntactically analyzed as PS trees. [sent-178, score-0.08]
66 We infer the function label of each node in the PS trees based on the morphological features, syntactic environment, and dash-feature (if exist), using deterministic grammar rules (Glinert, 1989). [sent-179, score-0.198]
67 Specifically, we compare each edge with a set of templates, and, once finding a template that fits the morphological and syntactic profile of an edge, we assign functions to all daughters. [sent-180, score-0.118]
68 This delivers PS trees where each node is annotated with a U-SD label (Figure 2a). [sent-181, score-0.08]
69 At a second stage we project the inferred labels onto the arcs of the unlabeled dependency trees of Goldberg (201 1), using the tree unification operation of Tsarfaty et al. [sent-182, score-0.2]
70 The result is a dependency tree aligned with the constituency tree where dependency arcs are labeled with the same function as the respective span in the PS tree. [sent-184, score-0.244]
71 We present two systems that predict U-SD labels along with morphological and syntactic information, using [DEP], a dependency parser (Nivre et al. [sent-186, score-0.278]
72 , 2007), and [RR], a RelationalRealizational (RR) constituency parser (Tsarfaty and Sima’an, 2008). [sent-187, score-0.132]
73 DEP is trained directly on the dependency version of the U-SD resource. [sent-188, score-0.076]
74 Since it cannot predict its own segmentation, automatic segments and tags are predicted using the system of Adler and Elhadad (2006). [sent-189, score-0.066]
75 The constituency4Despite significant advances in parsing Hebrew, as of yet there has been no functional evaluation of Hebrew parsers. [sent-190, score-0.138]
76 This is largely due to the lack of standard resources and guidelines for annotating functional structures in such a language. [sent-194, score-0.194]
77 581 based model is trained on U-SD-labeled RR trees using Petrov et al. [sent-198, score-0.08]
78 We use the lattice-based extension of Goldberg and Elhadad (201 1) to perform joint segmentation and parsing. [sent-200, score-0.043]
79 We evaluate three input scenarios: [Gold] gold segmentation and gold tags, [Predicted] gold segments, and [Raw] raw words. [sent-201, score-0.215]
80 The gold input scenarios obtain higher accuracy on function labels in all cases, since gold morphological analysis delivers disambiguated functions almost for free. [sent-208, score-0.242]
81 Constituencybased RR structures obtain better accuracy on USD annotations than the respective dependency parser. [sent-209, score-0.179]
82 All in all, the U-SD seed we created allows us to infer rich interpretable annotations automatically for raw text, using either a dependency parser or a constituency parser, in good accuracy. [sent-210, score-0.307]
83 We finally present two systems that automatically predict U-SD annotations for raw texts. [sent-214, score-0.099]
84 We further intend to use this scheme and computational frameworks to serve a wide cross-parser investigation on inferring functional structures across languages. [sent-216, score-0.346]
85 The hierarchy employs and extends the SD label set of de Marneffe et al. [sent-218, score-0.093]
86 Daniel Cer, Marie-Catherine de Marneffe, Daniel Jurafsky, and Christopher D. [sent-246, score-0.038]
87 Parsing to stanford dependencies: Trade-offs between speed and accuracy. [sent-249, score-0.046]
88 Joint Hebrew segmentation and parsing using a PCFGLA lattice parser. [sent-295, score-0.043]
89 Statistical parsing for morphologically rich language (SPMRL): What, how and whither. [sent-348, score-0.08]
wordName wordTfidf (topN-words)
[('sd', 0.446), ('mrls', 0.275), ('tsarfaty', 0.211), ('func', 0.2), ('hebrew', 0.195), ('functional', 0.138), ('dep', 0.128), ('marneffe', 0.12), ('modifier', 0.118), ('morphological', 0.118), ('scheme', 0.115), ('reut', 0.115), ('goldberg', 0.112), ('branch', 0.111), ('sbj', 0.111), ('sima', 0.111), ('morphemes', 0.102), ('hd', 0.101), ('elements', 0.098), ('constituency', 0.092), ('inheritance', 0.088), ('relations', 0.085), ('root', 0.081), ('properties', 0.081), ('trees', 0.08), ('nivre', 0.08), ('morphologically', 0.08), ('marker', 0.079), ('comp', 0.077), ('ps', 0.076), ('dependency', 0.076), ('segmentsfunctions', 0.075), ('spmrl', 0.075), ('usd', 0.075), ('yoav', 0.075), ('joakim', 0.068), ('modern', 0.067), ('aux', 0.066), ('mrl', 0.066), ('punct', 0.066), ('segments', 0.066), ('genitive', 0.065), ('grammatical', 0.063), ('prep', 0.061), ('configurational', 0.061), ('collapsed', 0.06), ('head', 0.058), ('rr', 0.058), ('accusative', 0.058), ('marking', 0.058), ('ontology', 0.058), ('structures', 0.056), ('object', 0.056), ('hierarchy', 0.055), ('dative', 0.055), ('dependencies', 0.054), ('raw', 0.052), ('auxpass', 0.05), ('evelina', 0.05), ('fundel', 0.05), ('khalil', 0.05), ('relationalrealizational', 0.05), ('seddah', 0.05), ('tedeval', 0.05), ('unifies', 0.05), ('structural', 0.049), ('markers', 0.047), ('annotations', 0.047), ('stanford', 0.046), ('characterization', 0.046), ('mod', 0.045), ('adler', 0.044), ('expl', 0.044), ('parc', 0.044), ('postal', 0.044), ('labels', 0.044), ('inherent', 0.044), ('principles', 0.043), ('cer', 0.043), ('passive', 0.043), ('segmentation', 0.043), ('auxiliary', 0.041), ('copular', 0.041), ('dash', 0.041), ('decorated', 0.041), ('haverinen', 0.041), ('clause', 0.04), ('morphosyntactic', 0.04), ('parser', 0.04), ('gold', 0.04), ('elhadad', 0.039), ('content', 0.039), ('auxiliaries', 0.038), ('de', 0.038), ('universal', 0.038), ('annotation', 0.037), ('inferring', 0.037), ('gf', 0.037), ('clausal', 0.037), ('def', 0.037)]
simIndex simValue paperId paperTitle
same-paper 1 0.99999964 28 acl-2013-A Unified Morpho-Syntactic Scheme of Stanford Dependencies
Author: Reut Tsarfaty
Abstract: Stanford Dependencies (SD) provide a functional characterization of the grammatical relations in syntactic parse-trees. The SD representation is useful for parser evaluation, for downstream applications, and, ultimately, for natural language understanding, however, the design of SD focuses on structurally-marked relations and under-represents morphosyntactic realization patterns observed in Morphologically Rich Languages (MRLs). We present a novel extension of SD, called Unified-SD (U-SD), which unifies the annotation of structurally- and morphologically-marked relations via an inheritance hierarchy. We create a new resource composed of U-SDannotated constituency and dependency treebanks for the MRL Modern Hebrew, and present two systems that can automatically predict U-SD annotations, for gold segmented input as well as raw texts, with high baseline accuracy.
2 0.21274625 368 acl-2013-Universal Dependency Annotation for Multilingual Parsing
Author: Ryan McDonald ; Joakim Nivre ; Yvonne Quirmbach-Brundage ; Yoav Goldberg ; Dipanjan Das ; Kuzman Ganchev ; Keith Hall ; Slav Petrov ; Hao Zhang ; Oscar Tackstrom ; Claudia Bedini ; Nuria Bertomeu Castello ; Jungmee Lee
Abstract: We present a new collection of treebanks with homogeneous syntactic dependency annotation for six languages: German, English, Swedish, Spanish, French and Korean. To show the usefulness of such a resource, we present a case study of crosslingual transfer parsing with more reliable evaluation than has been possible before. This ‘universal’ treebank is made freely available in order to facilitate research on multilingual dependency parsing.1
3 0.10128368 80 acl-2013-Chinese Parsing Exploiting Characters
Author: Meishan Zhang ; Yue Zhang ; Wanxiang Che ; Ting Liu
Abstract: Characters play an important role in the Chinese language, yet computational processing of Chinese has been dominated by word-based approaches, with leaves in syntax trees being words. We investigate Chinese parsing from the character-level, extending the notion of phrase-structure trees by annotating internal structures of words. We demonstrate the importance of character-level information to Chinese processing by building a joint segmentation, part-of-speech (POS) tagging and phrase-structure parsing system that integrates character-structure features. Our joint system significantly outperforms a state-of-the-art word-based baseline on the standard CTB5 test, and gives the best published results for Chinese parsing.
4 0.10004249 358 acl-2013-Transition-based Dependency Parsing with Selectional Branching
Author: Jinho D. Choi ; Andrew McCallum
Abstract: We present a novel approach, called selectional branching, which uses confidence estimates to decide when to employ a beam, providing the accuracy of beam search at speeds close to a greedy transition-based dependency parsing approach. Selectional branching is guaranteed to perform a fewer number of transitions than beam search yet performs as accurately. We also present a new transition-based dependency parsing algorithm that gives a complexity of O(n) for projective parsing and an expected linear time speed for non-projective parsing. With the standard setup, our parser shows an unlabeled attachment score of 92.96% and a parsing speed of 9 milliseconds per sentence, which is faster and more accurate than the current state-of-the-art transitionbased parser that uses beam search.
5 0.097515255 343 acl-2013-The Effect of Higher-Order Dependency Features in Discriminative Phrase-Structure Parsing
Author: Greg Coppola ; Mark Steedman
Abstract: Higher-order dependency features are known to improve dependency parser accuracy. We investigate the incorporation of such features into a cube decoding phrase-structure parser. We find considerable gains in accuracy on the range of standard metrics. What is especially interesting is that we find strong, statistically significant gains on dependency recovery on out-of-domain tests (Brown vs. WSJ). This suggests that higher-order dependency features are not simply overfitting the training material.
6 0.096271247 26 acl-2013-A Transition-Based Dependency Parser Using a Dynamic Parsing Strategy
7 0.091191433 41 acl-2013-Aggregated Word Pair Features for Implicit Discourse Relation Disambiguation
8 0.083908908 222 acl-2013-Learning Semantic Textual Similarity with Structural Representations
9 0.083237633 270 acl-2013-ParGramBank: The ParGram Parallel Treebank
10 0.078819804 98 acl-2013-Cross-lingual Transfer of Semantic Role Labeling Models
11 0.078120776 123 acl-2013-Discriminative Learning with Natural Annotations: Word Segmentation as a Case Study
12 0.07735455 132 acl-2013-Easy-First POS Tagging and Dependency Parsing with Beam Search
13 0.076932624 303 acl-2013-Robust multilingual statistical morphological generation models
14 0.076569356 208 acl-2013-Joint Inference for Heterogeneous Dependency Parsing
15 0.072578199 155 acl-2013-Fast and Accurate Shift-Reduce Constituent Parsing
16 0.071946025 378 acl-2013-Using subcategorization knowledge to improve case prediction for translation to German
17 0.071405366 295 acl-2013-Real-World Semi-Supervised Learning of POS-Taggers for Low-Resource Languages
18 0.071357921 94 acl-2013-Coordination Structures in Dependency Treebanks
19 0.069300391 99 acl-2013-Crowd Prefers the Middle Path: A New IAA Metric for Crowdsourcing Reveals Turker Biases in Query Segmentation
20 0.067660831 7 acl-2013-A Lattice-based Framework for Joint Chinese Word Segmentation, POS Tagging and Parsing
topicId topicWeight
[(0, 0.176), (1, -0.052), (2, -0.165), (3, -0.021), (4, -0.104), (5, -0.006), (6, -0.009), (7, 0.001), (8, 0.057), (9, -0.026), (10, -0.017), (11, 0.038), (12, 0.003), (13, 0.066), (14, -0.061), (15, -0.022), (16, -0.017), (17, -0.077), (18, -0.019), (19, -0.01), (20, -0.045), (21, 0.074), (22, -0.023), (23, 0.007), (24, -0.055), (25, 0.013), (26, -0.06), (27, -0.044), (28, 0.032), (29, -0.015), (30, -0.008), (31, 0.02), (32, 0.014), (33, 0.044), (34, 0.018), (35, -0.046), (36, -0.05), (37, -0.064), (38, 0.059), (39, -0.02), (40, 0.066), (41, -0.059), (42, 0.051), (43, 0.073), (44, 0.037), (45, -0.024), (46, 0.065), (47, 0.011), (48, -0.057), (49, 0.05)]
simIndex simValue paperId paperTitle
same-paper 1 0.95323491 28 acl-2013-A Unified Morpho-Syntactic Scheme of Stanford Dependencies
Author: Reut Tsarfaty
Abstract: Stanford Dependencies (SD) provide a functional characterization of the grammatical relations in syntactic parse-trees. The SD representation is useful for parser evaluation, for downstream applications, and, ultimately, for natural language understanding, however, the design of SD focuses on structurally-marked relations and under-represents morphosyntactic realization patterns observed in Morphologically Rich Languages (MRLs). We present a novel extension of SD, called Unified-SD (U-SD), which unifies the annotation of structurally- and morphologically-marked relations via an inheritance hierarchy. We create a new resource composed of U-SDannotated constituency and dependency treebanks for the MRL Modern Hebrew, and present two systems that can automatically predict U-SD annotations, for gold segmented input as well as raw texts, with high baseline accuracy.
2 0.76559746 368 acl-2013-Universal Dependency Annotation for Multilingual Parsing
Author: Ryan McDonald ; Joakim Nivre ; Yvonne Quirmbach-Brundage ; Yoav Goldberg ; Dipanjan Das ; Kuzman Ganchev ; Keith Hall ; Slav Petrov ; Hao Zhang ; Oscar Tackstrom ; Claudia Bedini ; Nuria Bertomeu Castello ; Jungmee Lee
Abstract: We present a new collection of treebanks with homogeneous syntactic dependency annotation for six languages: German, English, Swedish, Spanish, French and Korean. To show the usefulness of such a resource, we present a case study of crosslingual transfer parsing with more reliable evaluation than has been possible before. This ‘universal’ treebank is made freely available in order to facilitate research on multilingual dependency parsing.1
3 0.7421394 94 acl-2013-Coordination Structures in Dependency Treebanks
Author: Martin Popel ; David Marecek ; Jan StÄłpanek ; Daniel Zeman ; ZdÄłnÄłk Zabokrtsky
Abstract: Paratactic syntactic structures are notoriously difficult to represent in dependency formalisms. This has painful consequences such as high frequency of parsing errors related to coordination. In other words, coordination is a pending problem in dependency analysis of natural languages. This paper tries to shed some light on this area by bringing a systematizing view of various formal means developed for encoding coordination structures. We introduce a novel taxonomy of such approaches and apply it to treebanks across a typologically diverse range of 26 languages. In addition, empirical observations on convertibility between selected styles of representations are shown too.
4 0.67917228 270 acl-2013-ParGramBank: The ParGram Parallel Treebank
Author: Sebastian Sulger ; Miriam Butt ; Tracy Holloway King ; Paul Meurer ; Tibor Laczko ; Gyorgy Rakosi ; Cheikh Bamba Dione ; Helge Dyvik ; Victoria Rosen ; Koenraad De Smedt ; Agnieszka Patejuk ; Ozlem Cetinoglu ; I Wayan Arka ; Meladel Mistica
Abstract: This paper discusses the construction of a parallel treebank currently involving ten languages from six language families. The treebank is based on deep LFG (LexicalFunctional Grammar) grammars that were developed within the framework of the ParGram (Parallel Grammar) effort. The grammars produce output that is maximally parallelized across languages and language families. This output forms the basis of a parallel treebank covering a diverse set of phenomena. The treebank is publicly available via the INESS treebanking environment, which also allows for the alignment of language pairs. We thus present a unique, multilayered parallel treebank that represents more and different types of languages than are avail- able in other treebanks, that represents me ladel .mi st ica@ gmai l com . deep linguistic knowledge and that allows for the alignment of sentences at several levels: dependency structures, constituency structures and POS information.
5 0.6651001 335 acl-2013-Survey on parsing three dependency representations for English
Author: Angelina Ivanova ; Stephan Oepen ; Lilja vrelid
Abstract: In this paper we focus on practical issues of data representation for dependency parsing. We carry out an experimental comparison of (a) three syntactic dependency schemes; (b) three data-driven dependency parsers; and (c) the influence of two different approaches to lexical category disambiguation (aka tagging) prior to parsing. Comparing parsing accuracies in various setups, we study the interactions of these three aspects and analyze which configurations are easier to learn for a dependency parser.
6 0.64187747 367 acl-2013-Universal Conceptual Cognitive Annotation (UCCA)
7 0.61181718 204 acl-2013-Iterative Transformation of Annotation Guidelines for Constituency Parsing
8 0.61133558 331 acl-2013-Stop-probability estimates computed on a large corpus improve Unsupervised Dependency Parsing
9 0.58509982 208 acl-2013-Joint Inference for Heterogeneous Dependency Parsing
10 0.57000047 343 acl-2013-The Effect of Higher-Order Dependency Features in Discriminative Phrase-Structure Parsing
11 0.56772339 280 acl-2013-Plurality, Negation, and Quantification:Towards Comprehensive Quantifier Scope Disambiguation
12 0.56325418 26 acl-2013-A Transition-Based Dependency Parser Using a Dynamic Parsing Strategy
13 0.54986948 186 acl-2013-Identifying English and Hungarian Light Verb Constructions: A Contrastive Approach
14 0.5456633 295 acl-2013-Real-World Semi-Supervised Learning of POS-Taggers for Low-Resource Languages
15 0.54437935 78 acl-2013-Categorization of Turkish News Documents with Morphological Analysis
16 0.54340535 358 acl-2013-Transition-based Dependency Parsing with Selectional Branching
17 0.54267973 288 acl-2013-Punctuation Prediction with Transition-based Parsing
18 0.53357947 227 acl-2013-Learning to lemmatise Polish noun phrases
19 0.53245109 155 acl-2013-Fast and Accurate Shift-Reduce Constituent Parsing
20 0.53162229 303 acl-2013-Robust multilingual statistical morphological generation models
topicId topicWeight
[(0, 0.037), (6, 0.016), (11, 0.066), (14, 0.02), (24, 0.027), (26, 0.473), (35, 0.045), (42, 0.05), (48, 0.024), (61, 0.031), (70, 0.04), (88, 0.027), (95, 0.049)]
simIndex simValue paperId paperTitle
same-paper 1 0.95903862 28 acl-2013-A Unified Morpho-Syntactic Scheme of Stanford Dependencies
Author: Reut Tsarfaty
Abstract: Stanford Dependencies (SD) provide a functional characterization of the grammatical relations in syntactic parse-trees. The SD representation is useful for parser evaluation, for downstream applications, and, ultimately, for natural language understanding, however, the design of SD focuses on structurally-marked relations and under-represents morphosyntactic realization patterns observed in Morphologically Rich Languages (MRLs). We present a novel extension of SD, called Unified-SD (U-SD), which unifies the annotation of structurally- and morphologically-marked relations via an inheritance hierarchy. We create a new resource composed of U-SDannotated constituency and dependency treebanks for the MRL Modern Hebrew, and present two systems that can automatically predict U-SD annotations, for gold segmented input as well as raw texts, with high baseline accuracy.
2 0.95885772 115 acl-2013-Detecting Event-Related Links and Sentiments from Social Media Texts
Author: Alexandra Balahur ; Hristo Tanev
Abstract: Nowadays, the importance of Social Media is constantly growing, as people often use such platforms to share mainstream media news and comment on the events that they relate to. As such, people no loger remain mere spectators to the events that happen in the world, but become part of them, commenting on their developments and the entities involved, sharing their opinions and distributing related content. This paper describes a system that links the main events detected from clusters of newspaper articles to tweets related to them, detects complementary information sources from the links they contain and subsequently applies sentiment analysis to classify them into positive, negative and neutral. In this manner, readers can follow the main events happening in the world, both from the perspective of mainstream as well as social media and the public’s perception on them. This system will be part of the EMM media monitoring framework working live and it will be demonstrated using Google Earth.
3 0.94333547 323 acl-2013-Simpler unsupervised POS tagging with bilingual projections
Author: Long Duong ; Paul Cook ; Steven Bird ; Pavel Pecina
Abstract: We present an unsupervised approach to part-of-speech tagging based on projections of tags in a word-aligned bilingual parallel corpus. In contrast to the existing state-of-the-art approach of Das and Petrov, we have developed a substantially simpler method by automatically identifying “good” training sentences from the parallel corpus and applying self-training. In experimental results on eight languages, our method achieves state-of-the-art results. 1 Unsupervised part-of-speech tagging Currently, part-of-speech (POS) taggers are available for many highly spoken and well-resourced languages such as English, French, German, Italian, and Arabic. For example, Petrov et al. (2012) build supervised POS taggers for 22 languages using the TNT tagger (Brants, 2000), with an average accuracy of 95.2%. However, many widelyspoken languages including Bengali, Javanese, and Lahnda have little data manually labelled for POS, limiting supervised approaches to POS tagging for these languages. However, with the growing quantity of text available online, and in particular, multilingual parallel texts from sources such as multilingual websites, government documents and large archives ofhuman translations ofbooks, news, and so forth, unannotated parallel data is becoming more widely available. This parallel data can be exploited to bridge languages, and in particular, transfer information from a highly-resourced language to a lesser-resourced language, to build unsupervised POS taggers. In this paper, we propose an unsupervised approach to POS tagging in a similar vein to the work of Das and Petrov (201 1). In this approach, — — pecina@ ufal .mff .cuni . c z a parallel corpus for a more-resourced language having a POS tagger, and a lesser-resourced language, is word-aligned. These alignments are exploited to infer an unsupervised tagger for the target language (i.e., a tagger not requiring manuallylabelled data in the target language). Our approach is substantially simpler than that of Das and Petrov, the current state-of-the art, yet performs comparably well. 2 Related work There is a wealth of prior research on building unsupervised POS taggers. Some approaches have exploited similarities between typologically similar languages (e.g., Czech and Russian, or Telugu and Kannada) to estimate the transition probabilities for an HMM tagger for one language based on a corpus for another language (e.g., Hana et al., 2004; Feldman et al., 2006; Reddy and Sharoff, 2011). Other approaches have simultaneously tagged two languages based on alignments in a parallel corpus (e.g., Snyder et al., 2008). A number of studies have used tag projection to copy tag information from a resource-rich to a resource-poor language, based on word alignments in a parallel corpus. After alignment, the resource-rich language is tagged, and tags are projected from the source language to the target language based on the alignment (e.g., Yarowsky and Ngai, 2001 ; Das and Petrov, 2011). Das and Petrov (201 1) achieved the current state-of-the-art for unsupervised tagging by exploiting high confidence alignments to copy tags from the source language to the target language. Graph-based label propagation was used to automatically produce more labelled training data. First, a graph was constructed in which each vertex corresponds to a unique trigram, and edge weights represent the syntactic similarity between vertices. Labels were then propagated by optimizing a convex function to favor the same tags for closely related nodes 634 Proce dingSsof oifa, th Beu 5l1gsarti Aan,An u aglu Mste 4e-ti9n2g 0 o1f3 t.he ?c A2s0s1o3ci Aatsiosonc fioartio Cno fmorpu Ctoamtiopnuatalt Lioin gauli Lsitnicgsu,i psatgices 634–639, ModelCoverageAccuracy Many-to-1 alignments88%68% 1-to-1 alignments 68% 78% 1-to-1 alignments: Top 60k sents 91% 80% Table 1: Token coverage and accuracy of manyto-one and 1-to-1 alignments, as well as the top 60k sentences based on alignment score for 1-to-1 alignments, using directly-projected labels only. while keeping a uniform tag distribution for unrelated nodes. A tag dictionary was then extracted from the automatically labelled data, and this was used to constrain a feature-based HMM tagger. The method we propose here is simpler to that of Das and Petrov in that it does not require convex optimization for label propagation or a feature based HMM, yet it achieves comparable results. 3 Tagset Our tagger exploits the idea ofprojecting tag information from a resource-rich to resource-poor language. To facilitate this mapping, we adopt Petrov et al.’s (2012) twelve universal tags: NOUN, VERB, ADJ, ADV, PRON (pronouns), DET (de- terminers and articles), ADP (prepositions and postpositions), NUM (numerals), CONJ (conjunctions), PRT (particles), “.” (punctuation), and X (all other categories, e.g., foreign words, abbreviations). These twelve basic tags are common across taggers for most languages. Adopting a universal tagset avoids the need to map between a variety of different, languagespecific tagsets. Furthermore, it makes it possible to apply unsupervised tagging methods to languages for which no tagset is available, such as Telugu and Vietnamese. 4 A Simpler Unsupervised POS Tagger Here we describe our proposed tagger. The key idea is to maximize the amount of information gleaned from the source language, while limiting the amount of noise. We describe the seed model and then explain how it is successively refined through self-training and revision. 4.1 Seed Model The first step is to construct a seed tagger from directly-projected labels. Given a parallel corpus for a source and target language, Algorithm 1provides a method for building an unsupervised tagger for the target language. In typical applications, the source language would be a better-resourced language having a tagger, while the target language would be lesser-resourced, lacking a tagger and large amounts of manually POS-labelled data. Algorithm 1 Build seed model Algorithm 1Build seed model 1:Tag source side. 2: Word align the corpus with Giza++ and remove the many-to-one mappings. 3: Project tags from source to target using the remaining 1-to-1 alignments. 4: Select the top n sentences based on sentence alignment score. 5: Estimate emission and transition probabilities. 6: Build seed tagger T. We eliminate many-to-one alignments (Step 2). Keeping these would give more POS-tagged tokens for the target side, but also introduce noise. For example, suppose English and French were the source and target language, respectively. In this case alignments such as English laws (NNS) to French les (DT) lois (NNS) would be expected (Yarowsky and Ngai, 2001). However, in Step 3, where tags are projected from the source to target language, this would incorrectly tag French les as NN. We build a French tagger based on English– French data from the Europarl Corpus (Koehn, 2005). We also compare the accuracy and coverage of the tags obtained through direct projection using the French Melt POS tagger (Denis and Sagot, 2009). Table 1confirms that the one-to-one alignments indeed give higher accuracy but lower coverage than the many-to-one alignments. At this stage of the model we hypothesize that highconfidence tags are important, and hence eliminate the many-to-one alignments. In Step 4, in an effort to again obtain higher quality target language tags from direct projection, we eliminate all but the top n sentences based on their alignment scores, as provided by the aligner via IBM model 3. We heuristically set this cutoff × to 60k to balance the accuracy and size of the seed model.1 Returning to our preliminary English– French experiments in Table 1, this process gives improvements in both accuracy and coverage.2 1We considered values in the range 60–90k, but this choice had little impact on the accuracy of the model. 2We also considered using all projected labels for the top 60k sentences, not just 1-to-1 alignments, but in preliminary experiments this did not perform as well, possibly due to the previously-observed problems with many-to-one alignments. 635 The number of parameters for the emission probability is |V | |T| where V is the vocabulary and aTb iilsi ttyh eis tag |s e×t. TTh| ew htrearnesi Vtio ins probability, on atnhed other hand, has only |T|3 parameters for the trigram hmaondde,l we use. TB|ecause of this difference in number of parameters, in step 5, we use different strategies to estimate the emission and transition probabilities. The emission probability is estimated from all 60k selected sentences. However, for the transition probability, which has less parameters, we again focus on “better” sentences, by estimating this probability from only those sen- tences that have (1) token coverage > 90% (based on direct projection of tags from the source language), and (2) length > 4 tokens. These criteria aim to identify longer, mostly-tagged sentences, which we hypothesize are particularly useful as training data. In the case of our preliminary English–French experiments, roughly 62% of the 60k selected sentences meet these criteria and are used to estimate the transition probability. For unaligned words, we simply assign a random POS and very low probability, which does not substantially affect transition probability estimates. In Step 6 we build a tagger by feeding the estimated emission and transition probabilities into the TNT tagger (Brants, 2000), an implementation of a trigram HMM tagger. 4.2 Self training and revision For self training and revision, we use the seed model, along with the large number of target language sentences available that have been partially tagged through direct projection, in order to build a more accurate tagger. Algorithm 2 describes this process of self training and revision, and assumes that the parallel source–target corpus has been word aligned, with many-to-one alignments removed, and that the sentences are sorted by alignment score. In contrast to Algorithm 1, all sentences are used, not just the 60k sentences with the highest alignment scores. We believe that sentence alignment score might correspond to difficulty to tag. By sorting the sentences by alignment score, sentences which are more difficult to tag are tagged using a more mature model. Following Algorithm 1, we divide sentences into blocks of 60k. In step 3 the tagged block is revised by comparing the tags from the tagger with those obtained through direct projection. Suppose source Algorithm 2 Self training and revision 1:Divide target language sentences into blocks of n sentences. 2: Tag the first block with the seed tagger. 3: Revise the tagged block. 4: Train a new tagger on the tagged block. 5: Add the previous tagger’s lexicon to the new tagger. 6: Use the new tagger to tag the next block. 7: Goto 3 and repeat until all blocks are tagged. language word wis is aligned with target language word wjt with probability p(wjt |wsi), Tis is the tag for wis using the tagger availa|bwle for the source language, and Tjt is the tag for wjt using the tagger learned for the > S, where S is a threshold which we heuristically set to 0.7, we replace Tjt by Tis. Self-training can suffer from over-fitting, in which errors in the original model are repeated and amplified in the new model (McClosky et al., 2006). To avoid this, we remove the tag of any token that the model is uncertain of, i.e., if p(wjt |wsi) < S and Tjt Tis then Tjt = Null. So, on th|ew target side, aligned words have a tag from direct projection or no tag, and unaligned words have a tag assigned by our model. Step 4 estimates the emission and transition target language. If p(wtj|wis) = probabilities as in Algorithm 1. In Step 5, emission probabilities for lexical items in the previous model, but missing from the current model, are added to the current model. Later models therefore take advantage of information from earlier models, and have wider coverage. 5 Experimental Results Using parallel data from Europarl (Koehn, 2005) we apply our method to build taggers for the same eight target languages as Das and Petrov (201 1) Danish, Dutch, German, Greek, Italian, Portuguese, Spanish and Swedish with English as the source language. Our training data (Europarl) is a subset of the training data of Das and Petrov (who also used the ODS United Nations dataset which we were unable to obtain). The evaluation metric and test data are the same as that used by Das and Petrov. Our results are comparable to theirs, although our system is penalized by having less training data. We tag the source language with the Stanford POS tagger (Toutanova et al., 2003). — — 636 DanishDutchGermanGreekItalianPortugueseSpanishSwedishAverage Seed model83.781.183.677.878.684.981.478.981.3 Self training + revision 85.6 84.0 85.4 80.4 81.4 86.3 83.3 81.0 83.4 Das and Petrov (2011) 83.2 79.5 82.8 82.5 86.8 87.9 84.2 80.5 83.4 Table 2: Token-level POS tagging accuracy for our seed model, self training and revision, and the method of Das and Petrov (201 1). The best results on each language, and on average, are shown in bold. 1 1 Iteration 2 2 3 1 1 2 2 3 Iteration Figure 1: Overall accuracy, accuracy on known tokens, accuracy on unknown tokens, and proportion of known tokens for Italian (left) and Dutch (right). Table 2 shows results for our seed model, self training and revision, and the results reported by Das and Petrov. Self training and revision improve the accuracy for every language over the seed model, and gives an average improvement of roughly two percentage points. The average accuracy of self training and revision is on par with that reported by Das and Petrov. On individual languages, self training and revision and the method of Das and Petrov are split each performs better on half of the cases. Interestingly, our method achieves higher accuracies on Germanic languages the family of our source language, English while Das and Petrov perform better on Romance languages. This might be because our model relies on alignments, which might be more accurate for more-related languages, whereas Das and Petrov additionally rely on label propagation. Compared to Das and Petrov, our model performs poorest on Italian, in terms of percentage point difference in accuracy. Figure 1 (left panel) shows accuracy, accuracy on known words, accuracy on unknown words, and proportion of known tokens for each iteration of our model for Italian; iteration 0 is the seed model, and iteration 3 1 is the final model. Our model performs poorly on unknown words as indicated by the low accuracy on unknown words, and high accuracy on known — — — words compared to the overall accuracy. The poor performance on unknown words is expected because we do not use any language-specific rules to handle this case. Moreover, on average for the final model, approximately 10% of the test data tokens are unknown. One way to improve the performance of our tagger might be to reduce the proportion of unknown words by using a larger training corpus, as Das and Petrov did. We examine the impact of self-training and revision over training iterations. We find that for all languages, accuracy rises quickly in the first 5–6 iterations, and then subsequently improves only slightly. We exemplify this in Figure 1 (right panel) for Dutch. (Findings are similar for other languages.) Although accuracy does not increase much in later iterations, they may still have some benefit as the vocabulary size continues to grow. 6 Conclusion We have proposed a method for unsupervised POS tagging that performs on par with the current state- of-the-art (Das and Petrov, 2011), but is substantially less-sophisticated (specifically not requiring convex optimization or a feature-based HMM). The complexity of our algorithm is O(nlogn) compared to O(n2) for that of Das and Petrov 637 (201 1) where n is the size of training data.3 We made our code are available for download.4 In future work we intend to consider using a larger training corpus to reduce the proportion of unknown tokens and improve accuracy. Given the improvements of our model over that of Das and Petrov on languages from the same family as our source language, and the observation of Snyder et al. (2008) that a better tagger can be learned from a more-closely related language, we also plan to consider strategies for selecting an appropriate source language for a given target language. Using our final model with unsupervised HMM methods might improve the final performance too, i.e. use our final model as the initial state for HMM, then experiment with differ- ent inference algorithms such as Expectation Maximization (EM), Variational Bayers (VB) or Gibbs sampling (GS).5 Gao and Johnson (2008) compare EM, VB and GS for unsupervised English POS tagging. In many cases, GS outperformed other methods, thus we would like to try GS first for our model. 7 Acknowledgements This work is funded by Erasmus Mundus European Masters Program in Language and Communication Technologies (EM-LCT) and by the Czech Science Foundation (grant no. P103/12/G084). We would like to thank Prokopis Prokopidis for providing us the Greek Treebank and Antonia Marti for the Spanish CoNLL 06 dataset. Finally, we thank Siva Reddy and Spandana Gella for many discussions and suggestions. References Thorsten Brants. 2000. TnT: A statistical part-ofspeech tagger. In Proceedings of the sixth conference on Applied natural language processing (ANLP ’00), pages 224–231 . Seattle, Washington, USA. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proceedings of 3We re-implemented label propagation from Das and Petrov (2011). It took over a day to complete this step on an eight core Intel Xeon 3.16GHz CPU with 32 Gb Ram, but only 15 minutes for our model. 4https://code.google.com/p/universal-tagger/ 5We in fact have tried EM, but it did not help. The overall performance dropped slightly. This might be because selftraining with revision already found the local maximal point. the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1 (ACL 2011), pages 600–609. Portland, Oregon, USA. Pascal Denis and Beno ıˆt Sagot. 2009. Coupling an annotated corpus and a morphosyntactic lexicon for state-of-the-art POS tagging with less human effort. In Proceedings of the 23rd PacificAsia Conference on Language, Information and Computation, pages 721–736. Hong Kong, China. Anna Feldman, Jirka Hana, and Chris Brew. 2006. A cross-language approach to rapid creation of new morpho-syntactically annotated resources. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’06), pages 549–554. Genoa, Italy. Jianfeng Gao and Mark Johnson. 2008. A comparison of bayesian estimators for unsupervised hidden markov model pos taggers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 344–352. Association for Computational Linguistics, Stroudsburg, PA, USA. Jiri Hana, Anna Feldman, and Chris Brew. 2004. A resource-light approach to Russian morphology: Tagging Russian using Czech resources. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP ’04), pages 222–229. Barcelona, Spain. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proceedings of the Tenth Machine Translation Summit (MT Summit X), pages 79–86. AAMT, Phuket, Thailand. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the main conference on Human Language Technology Conference ofthe North American Chapter of the Association of Computational Linguistics (HLT-NAACL ’06), pages 152–159. New York, USA. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 2089–2096. Istanbul, Turkey. Siva Reddy and Serge Sharoff. 2011. Cross language POS Taggers (and other tools) for Indian 638 languages: An experiment with Kannada using Telugu resources. In Proceedings of the IJCNLP 2011 workshop on Cross Lingual Information Access: Computational Linguistics and the Information Need of Multilingual Societies (CLIA 2011). Chiang Mai, Thailand. Benjamin Snyder, Tahira Naseem, Jacob Eisenstein, and Regina Barzilay. 2008. Unsupervised multilingual learning for POS tagging. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP ’08), pages 1041–1050. Honolulu, Hawaii. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Featurerich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Vol- ume 1 (NAACL ’03), pages 173–180. Edmonton, Canada. David Yarowsky and Grace Ngai. 2001 . Inducing multilingual POS taggers and NP bracketers via robust projection across aligned corpora. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies (NAACL ’01), pages 1–8. Pittsburgh, Pennsylvania, USA. 639
4 0.93500292 221 acl-2013-Learning Non-linear Features for Machine Translation Using Gradient Boosting Machines
Author: Kristina Toutanova ; Byung-Gyu Ahn
Abstract: In this paper we show how to automatically induce non-linear features for machine translation. The new features are selected to approximately maximize a BLEU-related objective and decompose on the level of local phrases, which guarantees that the asymptotic complexity of machine translation decoding does not increase. We achieve this by applying gradient boosting machines (Friedman, 2000) to learn new weak learners (features) in the form of regression trees, using a differentiable loss function related to BLEU. Our results indicate that small gains in perfor- mance can be achieved using this method but we do not see the dramatic gains observed using feature induction for other important machine learning tasks.
5 0.90711015 257 acl-2013-Natural Language Models for Predicting Programming Comments
Author: Dana Movshovitz-Attias ; William W. Cohen
Abstract: Statistical language models have successfully been used to describe and analyze natural language documents. Recent work applying language models to programming languages is focused on the task of predicting code, while mainly ignoring the prediction of programmer comments. In this work, we predict comments from JAVA source files of open source projects, using topic models and n-grams, and we analyze the performance of the models given varying amounts of background data on the project being predicted. We evaluate models on their comment-completion capability in a setting similar to codecompletion tools built into standard code editors, and show that using a comment completion tool can save up to 47% of the comment typing. 1 Introduction and Related Work Statistical language models have traditionally been used to describe and analyze natural language documents. Recently, software engineering researchers have adopted the use of language models for modeling software code. Hindle et al. (2012) observe that, as code is created by humans it is likely to be repetitive and predictable, similar to natural language. NLP models have thus been used for a variety of software development tasks such as code token completion (Han et al., 2009; Jacob and Tairas, 2010), analysis of names in code (Lawrie et al., 2006; Binkley et al., 2011) and mining software repositories (Gabel and Su, 2008). An important part of software programming and maintenance lies in documentation, which may come in the form of tutorials describing the code, or inline comments provided by the programmer. The documentation provides a high level description of the task performed by the code, and may William W. Cohen Computer Science Department Carnegie Mellon University wcohen @ c s .cmu .edu include examples of use-cases for specific code segments or identifiers such as classes, methods and variables. Well documented code is easier to read and maintain in the long-run but writing comments is a laborious task that is often overlooked or at least postponed by many programmers. Code commenting not only provides a summarization of the conceptual idea behind the code (Sridhara et al., 2010), but can also be viewed as a form of document expansion where the comment contains significant terms relevant to the described code. Accurately predicted comment words can therefore be used for a variety of linguistic uses including improved search over code bases using natural language queries, code categorization, and locating parts of the code that are relevant to a specific topic or idea (Tseng and Juang, 2003; Wan et al., 2007; Kumar and Carterette, 2013; Shepherd et al., 2007; Rastkar et al., 2011). A related and well studied NLP task is that of predicting natural language caption and commentary for images and videos (Blei and Jordan, 2003; Feng and Lapata, 2010; Feng and Lapata, 2013; Wu and Li, 2011). In this work, our goal is to apply statistical language models for predicting class comments. We show that n-gram models are extremely successful in this task, and can lead to a saving of up to 47% in comment typing. This is expected as n-grams have been shown as a strong model for language and speech prediction that is hard to improve upon (Rosenfeld, 2000). In some cases however, for example in a document expansion task, we wish to extract important terms relevant to the code regardless of local syntactic dependencies. We hence also evaluate the use of LDA (Blei et al., 2003) and link-LDA (Erosheva et al., 2004) topic models, which are more relevant for the term ex- traction scenario. We find that the topic model performance can be improved by distinguishing code and text tokens in the code. 35 Proce dinSgosfi oa,f tB huel 5g1arsita, An Anu gauls Mt 4e-e9ti n2g01 o3f. th ?c e2 A0s1s3oc Aiastsio cnia fotiron C fo mrp Cuotmatpiounta tlio Lninaglu Li sntgicusi,s ptaicgses 35–40, 2 Method 2.1 Models We train n-gram models (n = 1, 2, 3) over source code documents containing sequences of combined code and text tokens from multiple training datasets (described below). We use the Berkeley Language Model package (Pauls and Klein, 2011) with absolute discounting (Kneser-Ney smoothing; (1995)) which includes a backoff strategy to lower-order n-grams. Next, we use LDA topic models (Blei et al., 2003) trained on the same data, with 1, 5, 10 and 20 topics. The joint distribution of a topic mixture θ, and a set of N topics z, for a single source code document with N observed word tokens, d = {wi}iN=1, given the Dirichlet parameters α sa,n dd β, {isw th}erefore p(θ, z, w|α, β) = p(θ|α) Yp(z|θ)p(w|z, (1) β) Yw Under the models described so far, there is no distinction between text and code tokens. Finally, we consider documents as having a mixed membership of two entity types, code and text tokens, d = where tthexet text ws,o drd =s are tok}ens f,r{owm comment and string literals, and the code words include the programming language syntax tokens (e.g., publ ic, private, for, etc’ ) and all identifiers. In this case, we train link-LDA models (Erosheva et al., 2004) with 1, 5, 10 and 20 topics. Under the linkLDA model, the mixed-membership joint distribution of a topic mixture, words and topics is then ({wciode}iC=n1, {witext}iT=n1), p(θ, z, w|α, β) = p(θ|α) Y wYtext · p(ztext|θ)p(wtext|ztext,β)· (2) Y p(zcode|θ)p(wcode|zcode,β) wYcode where θ is the joint topic distribution, w is the set of observed document words, ztext is a topic associated with a text word, and zcode a topic associated with a code word. The LDA and link-LDA models use Gibbs sampling (Griffiths and Steyvers, 2004) for topic inference, based on the implementation of Balasubramanyan and Cohen (201 1) with single or multiple entities per document, respectively. 2.2 Testing Methodology Our goal is to predict the tokens of the JAVA class comment (the one preceding the class definition) in each of the test files. Each of the models described above assigns a probability to the next comment token. In the case of n-grams, the probability of a token word wi is given by considering previous words p(wi |wi−1 , . . . , w0). This probability is estimated given the previous n 1tokens as p(wi|wi−1, wi−(n−1)). For t|hwe topic models, we separate the docu- ..., − ment tokens into the class definition and the comment we wish to predict. The set of tokens of the class comment are all considered as text tokens. The rest of the tokens in the document are considered to be the class definition, and they may contain both code and text tokens (from string literals and other comments in the source file). We then compute the posterior probability of document topics by solving the following inference problem conditioned on the tokens wc, wr, wr p(θ,zr|wr,α,β) =p(θp,(zwr,rw|αr,|αβ),β) (3) This gives us an estimate of the document distribution, θ, with which we infer the probability of the comment tokens as p(wc|θ,β) = Xp(wc|z,β)p(z|θ) (4) Xz Following Blei et al. (2003), for the case of a single entity LDA, the inference problem from equation (3) can be solved by considering p(θ, z, w|α, β), as in equation (1), and by taking tph(eθ marginal )di,s atrsib iunti eoqnu aotfio othne ( 1d)o,c aunmde bnyt t toakkeinngs as a continuous mixture distribution for the set w = by integrating over θ and summing over the set of topics z wr, p(w|α,β) =Zp(θ|α)· (5) YwXzp(z|θ)p(w|z,β)!dθ For the case of link-LDA where the document is comprised of two entities, in our case code tokens and text tokens, we can consider the mixedmembership joint distribution θ, as in equation (2), and similarly the marginal distribution p(w|α, β) over bimoithla rclyod teh ean mda tregxint tlok deisntsri bfruotmion w pr(.w |Sαi,nβce) comment words in are all considered as text tokens they are sampled using text topics, namely ztext, in equation (4). wc 36 3 Experimental Settings 3.1 Data and Training Methodology We use source code from nine open source JAVA projects: Ant, Cassandra, Log4j, Maven, MinorThird, Batik, Lucene, Xalan and Xerces. For each project, we divide the source files into a training and testing dataset. Then, for each project in turn, we consider the following three main training scenarios, leading to using three training datasets. To emulate a scenario in which we are predicting comments in the middle of project development, we can use data (documented code) from the same project. In this case, we use the in-project training dataset (IN). Alternatively, if we train a comment prediction model at the beginning of the development, we need to use source files from other, possibly related projects. To analyze this scenario, for each of the projects above we train models using an out-of-project dataset (OUT) containing data from the other eight projects. Typically, source code files contain a greater amount ofcode versus comment text. Since we are interested in predicting comments, we consider a third training data source which contains more English text as well as some code segments. We use data from the popular Q&A; website StackOverflow (SO) where users ask and answer technical questions about software development, tools, algorithms, etc’ . We downloaded a dataset of all actions performed on the site since it was launched in August 2008 until August 2012. The data includes 3,453,742 questions and 6,858,133 answers posted by 1,295,620 users. We used only posts that are tagged as JAVA related questions and answers. All the models for each project are then tested on the testing set of that project. We report results averaged over all projects in Table 1. Source files were tokenized using the Eclipse JDT compiler tools, separating code tokens and identifiers. Identifier names (of classes, methods and variables), were further tokenized by camel case notation (e.g., ’minMargin’ was converted to ’min margin’). Non alpha-numeric tokens (e.g., dot, semicolon) were discarded from the code, as well as numeric and single character literals. Text from comments or any string literals within the code were further tokenized with the Mallet statistical natural language processing package (Mc- Callum, 2002). Posts from SO were parsed using the Apache Tika toolkit1 and then tokenized with the Mallet package. We considered as raw code tokens anything labeled using a markup (as indicated by the SO users who wrote the post). 3.2 Evaluation Since our models are trained using various data sources the vocabularies used by each of them are different, making the comment likelihood given by each model incomparable due to different sets of out-of-vocabulary tokens. We thus evaluate models using a character saving metric which aims at quantifying the percentage of characters that can be saved by using the model in a word-completion settings, similar to standard code completion tools built into code editors. For a comment word with n characters, w = w1, . . . , wn, we predict the two most likely words given each model filtered by the first 0, . . . , n characters ofw. Let k be the minimal ki for which w is in the top two predicted word tokens where tokens are filtered by the first ki characters. Then, the number of saved characters for w is n k. In Table 1we report the average percentage o−f ksa.v Iend T Tcahbalera 1cte wrse per ocrotm thmee avnet using eearcchen not-f the above models. The final results are also averaged over the nine input projects. As an example, in the predicted comment shown in Table 2, taken from the project Minor-Third, the token entity is the most likely token according to the model SO trigram, out of tokens starting with the prefix ’en’ . The saved characters in this case are ’tity’ . − 4 Results Table 1 displays the average percentage of characters saved per class comment using each of the models. Models trained on in-project data (IN) perform significantly better than those trained on another data source, regardless of the model type, with an average saving of 47. 1% characters using a trigram model. This is expected, as files from the same project are likely to contain similar comments, and identifier names that appear in the comment of one class may appear in the code of another class in the same project. Clearly, in-project data should be used when available as it improves comment prediction leading to an average increase of between 6% for the worst model (26.6 for OUT unigram versus 33.05 for IN) and 14% for the best (32.96 for OUT trigram versus 47. 1for IN). 1http://tika.apache.org/ 37 Model n / topics n-gram LDA Link-LDA 1 2 3 20 10 5 1 20 10 5 1 IN 33.05 (3.62) 43.27 (5.79) 47.1 (6.87) 34.20 (3.63) 33.93 (3.67) 33.63 (3.67) 33.05 (3.62) 35.76 (3.95) 35.81 (4.12) 35.37 (3.98) 34.59 (3.92) OUT 26.6 (3.37) 31.52 (4.17) 32.96 (4.33) 26.79 (3.26) 26.8 (3.36) 26.86 (3.44) 26.6 (3.37) 28.03 (3.60) 28 (3.56) 28 (3.67) 27.82 (3.62) SO 27.8 (3.51) 33.29 (4.40) 34.56 (4.78) 27.25 (3.67) 27.22 (3.44) 27.34 (3.55) 27.8 (3.51) 28.08 (3.48) 28.12 (3.58) 27.94 (3.56) 27.9 (3.45) Table 1: Average percentage of characters saved per comment using n-gram, LDA and link-LDA models trained on three training sets: IN, OUT, and SO. The results are averaged over nine JAVA projects (with standard deviations in parenthesis). Model Predicted Comment trigram IN link-LDA OUT trigram SO trigram “Train “Train “Train “Train IN named-entity a named-entity a named-entity a named-entity a extractor“ extractor“ extractor“ extractor“ Table 2: Sample comment from the Minor-Third project predicted using IN, OUT and SO based models. Saved characters are underlined. Of the out-of-project data sources, models using a greater amount of text (SO) mostly outperformed models based on more code (OUT). This increase in performance, however, comes at a cost of greater run-time due to the larger word dictionary associated with the SO data. Note that in the scope of this work we did not investigate the contribution of each of the background projects used in OUT, and how their relevance to the target prediction project effects their performance. The trigram model shows the best performance across all training data sources (47% for IN, 32% for OUT and 34% for SO). Amongst the tested topic models, link-LDA models which distinguish code and text tokens perform consistently better than simple LDA models in which all tokens are considered as text. We did not however find a correlation between the number of latent topics learned by a topic model and its performance. In fact, for each of the data sources, a different num- ber of topics gave the optimal character saving results. Note that in this work, all topic models are based on unigram tokens, therefore their results are most comparable with that of the unigram in Dataset n-gram link-LDA IN 2778.35 574.34 OUT 1865.67 670.34 SO 1898.43 638.55 Table 3: Average words per project for which each tested model completes the word better than the other. This indicates that each of the models is better at predicting a different set of comment words. Table 1, which does not benefit from the backoff strategy used by the bigram and trigram models. By this comparison, the link-LDA topic model proves more successful in the comment prediction task than the simpler models which do not distin- guish code and text tokens. Using n-grams without backoff leads to results significantly worse than any of the presented models (not shown). Table 2 shows a sample comment segment for which words were predicted using trigram models from all training sources and an in-project linkLDA. The comment is taken from the TrainExtractor class in the Minor-Third project, a machine learning library for annotating and categorizing text. Both IN models show a clear advantage in completing the project-specific word Train, compared to models based on out-of-project data (OUT and SO). Interestingly, in this example the trigram is better at completing the term namedentity given the prefix named. However, the topic model is better at completing the word extractor which refers to the target class. This example indicates that each model type may be more successful in predicting different comment words, and that combining multiple models may be advantageous. 38 This can also be seen by the analysis in Table 3 where we compare the average number of words completed better by either the best n-gram or topic model given each training dataset. Again, while n-grams generally complete more words better, a considerable portion of the words is better completed using a topic model, further motivating a hybrid solution. 5 Conclusions We analyze the use of language models for predicting class comments for source file documents containing a mixture of code and text tokens. Our experiments demonstrate the effectiveness of using language models for comment completion, showing a saving of up to 47% of the comment characters. When available, using in-project training data proves significantly more successful than using out-of-project data. However, we find that when using out-of-project data, a dataset based on more words than code performs consistently better. The results also show that different models are better at predicting different comment words, which motivates a hybrid solution combining the advantages of multiple models. Acknowledgments This research was supported by the NSF under grant CCF-1247088. References Ramnath Balasubramanyan and William W Cohen. 2011. Block-lda: Jointly modeling entity-annotated text and entity-entity links. In Proceedings ofthe 7th SIAM International Conference on Data Mining. Dave Binkley, Matthew Hearn, and Dawn Lawrie. 2011. Improving identifier informativeness using part of speech information. In Proc. of the Working Conference on Mining Software Repositories. ACM. David M Blei and Michael I Jordan. 2003. Modeling annotated data. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval. ACM. David M Blei, Andrew Y Ng, and Michael IJordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research. Elena Erosheva, Stephen Fienberg, and John Lafferty. 2004. Mixed-membership models of scientific publications. Proceedings of the National Academy of Sciences of the United States of America. Yansong Feng and Mirella Lapata. 2010. How many words is a picture worth? automatic caption generation for news images. In Proc. of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. Yansong Feng and Mirella Lapata. 2013. Automatic caption generation for news images. IEEE transactions on pattern analysis and machine intelligence. Mark Gabel and Zhendong Su. 2008. Javert: fully automatic mining of general temporal properties from dynamic traces. In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, pages 339–349. ACM. Thomas L Griffiths and Mark Steyvers. 2004. Finding scientific topics. Proc. of the National Academy of Sciences of the United States of America. Sangmok Han, David R Wallace, and Robert C Miller. 2009. Code completion from abbreviated input. In Automated Software Engineering, 2009. ASE’09. 24th IEEE/ACM International Conference on, pages 332–343. IEEE. Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. 2012. On the naturalness of software. In Software Engineering (ICSE), 2012 34th International Conference on. IEEE. Ferosh Jacob and Robert Tairas. 2010. Code template inference using language models. In Proceedings of the 48th Annual Southeast Regional Conference. ACM. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In Acoustics, Speech, and Signal Processing, 1995. ICASSP-95., volume 1, pages 181–184. IEEE. Naveen Kumar and Benjamin Carterette. 2013. Time based feedback and query expansion for twitter search. In Advances in Information Retrieval, pages 734–737. Springer. Dawn Lawrie, Christopher Morrell, Henry Feild, and David Binkley. 2006. Whats in a name? a study of identifiers. In Program Comprehension, 2006. ICPC 2006. 14th IEEE International Conference on, pages 3–12. IEEE. Andrew Kachites McCallum. 2002. Mallet: A machine learning for language toolkit. Adam Pauls and Dan Klein. 2011. Faster and smaller language models. In Proceedings of the 49th annual meeting of the Association for Computational Linguistics: Human Language Technologies, volume 1, pages 258–267. n-gram Sarah Rastkar, Gail C Murphy, and Alexander WJ Bradley. 2011. Generating natural language summaries for crosscutting source code concerns. In Software Maintenance (ICSM), 2011 27th IEEE International Conference on, pages 103–1 12. IEEE. 39 Ronald Rosenfeld. 2000. Two decades of statistical language modeling: Where do we go from here? Proceedings of the IEEE, 88(8): 1270–1278. David Shepherd, Zachary P Fry, Emily Hill, Lori Pollock, and K Vijay-Shanker. 2007. Using natural language program analysis to locate and understand action-oriented concerns. In Proceedings of the 6th international conference on Aspect-oriented software development, pages 212–224. ACM. Giriprasad Sridhara, Emily Hill, Divya Muppaneni, Lori Pollock, and K Vijay-Shanker. 2010. Towards automatically generating summary comments for java methods. In Proceedings of the IEEE/ACM international conference on Automated software engineering, pages 43–52. ACM. Yuen-Hsien Tseng and Da-Wei Juang. 2003. Document-self expansion for text categorization. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pages 399–400. ACM. Xiaojun Wan, Jianwu Yang, and Jianguo Xiao. 2007. Single document summarization with document expansion. In Proc. of the National Conference on Artificial Intelligence. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999. Roung-Shiunn Wu and Po-Chun Li. 2011. Video annotation using hierarchical dirichlet process mixture model. Expert Systems with Applications, 38(4):3040–3048. 40
6 0.83949125 305 acl-2013-SORT: An Interactive Source-Rewriting Tool for Improved Translation
8 0.66858935 209 acl-2013-Joint Modeling of News Readerâ•Žs and Comment Writerâ•Žs Emotions
9 0.66448301 368 acl-2013-Universal Dependency Annotation for Multilingual Parsing
10 0.65803379 163 acl-2013-From Natural Language Specifications to Program Input Parsers
11 0.64468026 95 acl-2013-Crawling microblogging services to gather language-classified URLs. Workflow and case study
12 0.64212453 369 acl-2013-Unsupervised Consonant-Vowel Prediction over Hundreds of Languages
13 0.6346451 148 acl-2013-Exploring Sentiment in Social Media: Bootstrapping Subjectivity Clues from Multilingual Twitter Streams
14 0.62471879 295 acl-2013-Real-World Semi-Supervised Learning of POS-Taggers for Low-Resource Languages
15 0.62057185 144 acl-2013-Explicit and Implicit Syntactic Features for Text Classification
16 0.61610508 342 acl-2013-Text Classification from Positive and Unlabeled Data using Misclassified Data Correction
17 0.6152699 216 acl-2013-Large tagset labeling using Feed Forward Neural Networks. Case study on Romanian Language
18 0.61443853 7 acl-2013-A Lattice-based Framework for Joint Chinese Word Segmentation, POS Tagging and Parsing
19 0.6068033 147 acl-2013-Exploiting Topic based Twitter Sentiment for Stock Prediction
20 0.6023432 131 acl-2013-Dual Training and Dual Prediction for Polarity Classification