acl acl2013 acl2013-97 knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Mo Yu ; Tiejun Zhao ; Yalong Bai ; Hao Tian ; Dianhai Yu
Abstract: Cross-lingual projection methods can benefit from resource-rich languages to improve performances of NLP tasks in resources-scarce languages. However, these methods confronted the difficulty of syntactic differences between languages especially when the pair of languages varies greatly. To make the projection method well-generalize to diverse languages pairs, we enhance the projection method based on word alignments by introducing target-language word representations as features and proposing a novel noise removing method based on these word representations. Experiments showed that our methods improve the performances greatly on projections between English and Chinese.
Reference: text
sentIndex sentText sentNum sentScore
1 com Abstract Cross-lingual projection methods can benefit from resource-rich languages to improve performances of NLP tasks in resources-scarce languages. [sent-8, score-0.907]
2 However, these methods confronted the difficulty of syntactic differences between languages especially when the pair of languages varies greatly. [sent-9, score-0.286]
3 To make the projection method well-generalize to diverse languages pairs, we enhance the projection method based on word alignments by introducing target-language word representations as features and proposing a novel noise removing method based on these word representations. [sent-10, score-2.193]
4 Experiments showed that our methods improve the performances greatly on projections between English and Chinese. [sent-11, score-0.223]
5 To address NLP tasks in resource-scarce languages, cross-lingual projection methods were proposed, which make use of existing resources in resource-rich language (also called source language) to help NLP tasks in resource-scarce language (also named as target language). [sent-15, score-0.827]
6 One intuitive and effective method is to build a common feature space for all languages, so that the model trained on one language could be directly used on other languages (McDonald et al. [sent-17, score-0.255]
7 Otherwise the performance will degrade especially when the orders of phrases between source and target languages differ a lot. [sent-22, score-0.287]
8 Another common type of projection methods map labels from resource-rich language sentences to resource-scarce ones in a parallel corpus using word alignment information (Yarowsky et al. [sent-23, score-0.881]
9 We refer them as projection based on word alignments in this paper. [sent-26, score-0.823]
10 Compared to other types of projection methods, this type of methods is more robust to syntactic differences between languages since it trained models on the target side thus following the topology of the target language. [sent-27, score-1.112]
11 This paper aims to build an accurate projection method with strong generality to various pairs of languages, even when the languages are from different families and are typologically divergent. [sent-28, score-0.967]
12 We adopted the projection method based on word alignments since it is less affected by language differences. [sent-31, score-0.954]
13 Firstly, the models trained on projected data could only cover words and cases appeared in the target side of parallel corpus, making it difficult to generalize to test data in broader domains. [sent-33, score-0.611]
14 Secondly, the performances of these methods are limited by the accuracy of word alignments, especially when words between two languages are not one-one aligned. [sent-34, score-0.29]
15 This paper aims to build an accurate projection method with strong generality to various pairs of languages. [sent-36, score-0.799]
16 We built the method on top of projection method based on word alignments because of its advantage of being less affected by syntactic differences, and proposed two solutions to solve the above two difficulties of this type of methods. [sent-37, score-1.021]
17 c A2s0s1o3ci Aatsiosonc fioartio Cno fmorpu Ctoamtiopnuatalt Lioin gauli Lsitnicgsu,i psatgices 312–317, Firstly, we introduce Brown clusters of target language to make the projection models cover broader cases. [sent-40, score-1.068]
18 They can be efficiently learned on large-scale unlabeled data in target language, which is much easier to acquire even when the scales of parallel corpora of minor languages are limited. [sent-42, score-0.364]
19 Brown clusters have been first introduced to the field of cross-lingual projections in (T ¨ackstr o¨m et al. [sent-43, score-0.331]
20 , 2012) and have achieved great improvements on projection between European languages. [sent-44, score-0.775]
21 However, their work was based on the direct projection methods so that it do not work very well between languages from different families as will be shown in Section 3. [sent-45, score-0.894]
22 Secondly, to reduce the noises in projection, we propose a noise removing method to detect and correct noisy projected labels. [sent-46, score-0.736]
23 The method was also built on Brown clusters, based on the assumption that instances with similar representations of Brown clusters tend to have similar labels. [sent-47, score-0.391]
24 As far as we know, no one has done any research on removing noises based on the space of word representations in the field of NLP. [sent-48, score-0.512]
25 Using above techniques, we achieved a projection method that adapts well on different language pairs even when the two languages differ enormously. [sent-49, score-0.901]
26 Experiments of NER and POS tagging projection from English to Chinese proved the effectiveness of our methods. [sent-50, score-0.703]
27 In the rest of our paper, Section 2 describes the proposed cross-lingual projection method. [sent-51, score-0.666]
28 2 Proposed Cross-lingual Projection Methods In this section, we first briefly introduce the crosslingual projection method based on word alignments. [sent-54, score-0.755]
29 Then we describe how the word representations (Brown clusters) were used in the projection method. [sent-55, score-0.803]
30 1 Projection based on word alignments In this paper we consider cross-lingual projection based on word alignment, because we want to build projection methods that can be used between language pairs with large differences. [sent-59, score-1.57]
31 Fig- ure 1 shows the procedure of cross-lingual projection methods, taking projection of NER from English to Chinese as an example. [sent-60, score-1.332]
32 First, sentences from the source side of the parallel corpus are labeled by an accurate model in English (e. [sent-62, score-0.25]
33 , ”Rongji Zhu” and ”Gan Luo” were labeled as ”PER”), since the source language has rich resources to build accurate NER models. [sent-64, score-0.197]
34 Then word alignments are generated from the parallel corpus and serve as a bridge, so that unlabeled words in the target language will get the same labels with words aligning to them in the source language, e. [sent-65, score-0.441]
35 the first word ‘朱(金容)基’ in Chinese gets the projected label ‘PER’, since it is aligned to “Rongji” and “Zhu”. [sent-67, score-0.383]
36 In this way, labels in source language sentences are projected to the target sentences. [sent-68, score-0.426]
37 Labels of Chinese sentence (right) in brackets are projected from the source sentence. [sent-81, score-0.285]
38 From the projection procedure we can see that a labeled dataset of target language is built based on the projected labels from source sentences. [sent-82, score-1.178]
39 The projected dataset has a large size, but with a lot of noises. [sent-83, score-0.243]
40 Since the models are trained on the target language, this projection approach is less affected by language differences, comparing with direct projection methods. [sent-86, score-1.58]
41 For example in Figure 1, some Chinese politicians in 1990’s will be learned as person names, but some names of recent politi- cians such as “Obama”, which did not appeared in the parallel corpus, would not be recognized. [sent-91, score-0.294]
42 To broader the coverage of the projected data, we introduced word representations as features. [sent-92, score-0.43]
43 Same or similar word representations will be assigned to words appearing in similar contexts, such as person names. [sent-93, score-0.21]
44 Since word representations are trained on large-scale unlabeled sentences in target language, they cover much more words than the parallel corpus does. [sent-94, score-0.39]
45 So the information of a word in projected labeled data will apply to other words with the same or similar representations, even if they did not appear in the parallel data. [sent-95, score-0.425]
46 In this work we use Brown clusters as word representations on target languages. [sent-96, score-0.462]
47 Brown clustering assigns words to hierarchical clusters according to the distributions of words before and after them. [sent-97, score-0.238]
48 The cluster id of the word to predict (c0) and those of context words (ci, i∈ {−2, −1, 1, 2}), as well as the conjunctions ,oif ∈the {s−e 2cl,−ust1e,r1s were auss ewde as afesa tthuer ceos ninCRF models in the same way the traditional word features were used. [sent-99, score-0.235]
49 Since Brown clusters are hi- erarchical, the cluster for each word can be represented as a binary string. [sent-100, score-0.424]
50 So we also use prefix of cluster IDs as features, in order to compensate for clusters containing small number ofwords. [sent-101, score-0.375]
51 3 Noise Removing in Word Representation Space Another disadvantage of the projection method is that the accuracy of projected labels is badly affected by non-literate translation and word alignment errors, making the data contain many noises. [sent-105, score-1.24]
52 For example in Figure 1, the word “吴仪(Wu Yi)” was not labeled as a named entity since it was not aligned to any words in English due to the alignment errors. [sent-106, score-0.242]
53 A more accurate model will be trained if such noises can be reduced. [sent-107, score-0.248]
54 A direct way to remove the noises is to modify the label of a word to make it consistent with the majority of labels assigned to the same word in the parallel corpus. [sent-108, score-0.491]
55 The method is limited when a word with low frequency has many of its appearances incorrectly labeled because of alignment errors. [sent-109, score-0.19]
56 In this situation the noises are impossible to remove according to the word itself. [sent-110, score-0.255]
57 Such difficulties can be easily solved when we turned to the space of Brown clusters, based on the observation that words in a same cluster tend to have same labels. [sent-112, score-0.191]
58 Having observed that a large portion of words in this cluster are person names, it is reasonable to modified the label of “吴仪(Wu Yi)” to “PER”. [sent-114, score-0.241]
59 The space of clusters is also less sparse so it is also possible to use combination of the clusters to help noise removing, in order to utilize the context information of data instances. [sent-115, score-0.626]
60 For example, we could represent a instance as bigram of the cluster of target word and that of the previous word. [sent-116, score-0.308]
61 The whole noise removing method can be represented as following: Suppose a target word wi was assigned label yi during projection with probability of alignment pi. [sent-118, score-1.24]
62 From the whole projected labeled data, we can get the distribution pw (y) for the word wi, the distribution pc(y) for its cluster ci and the distribution pb(y) for the bigram ci−1ci. [sent-119, score-0.586]
63 With the noise removing method, we can build a more accurate labeled dataset based on the projected data and then use it for training models. [sent-123, score-0.647]
64 1 Data Preparation We took English as resource-rich language and used Chinese to imitate resource-scarce languages, since the two languages differ a lot. [sent-125, score-0.213]
65 For evaluation of projection of POS tagging, we used the test set of CTB. [sent-132, score-0.666]
66 Since English and Chinese have different annotation standards, labels in the two languages were converted to the universal POS tag set (Petrov et al. [sent-133, score-0.21]
67 , 2011; Das and Petrov, 2011) so that the labels between the source and target languages were consistent. [sent-134, score-0.304]
68 The Brown clusters were trained on Chinese Wikipedia. [sent-136, score-0.276]
69 The bodies of all articles are retained to induce 1000 clusters using the algorithm in (Liang, 2005) . [sent-137, score-0.271]
70 When English Brown clusters were in need, we trained the word clusters on the tokenized English Wikipedia. [sent-140, score-0.563]
71 It is easier to obtain similar amount of parallel sentences between English and minor languages, making the conclusions more general for problems of projection in real applications. [sent-143, score-0.819]
72 We re-implemented the direct projection method with projected clusters in (T ¨ackstr o¨m et al. [sent-146, score-1.247]
73 Although their method was proven to work well on European language pairs, the results showed that projection based on word alignments (WA) worked much better since the source and tar- get languages are from different families. [sent-148, score-1.087]
74 After we add the clusters trained on Chinese Wikipedia as features as in Section 2. [sent-149, score-0.276]
75 The performances of all three categories of named entities were improved greatly after adding word representation features. [sent-155, score-0.269]
76 Since in test set, most entities that are not covered are person names, Brown clusters helped to increase the recall greatly. [sent-160, score-0.345]
77 , 2012), Brown clusters trained on the source side were projected to the target side based on word alignments. [sent-162, score-0.779]
78 , 2012), we tried to use the projected clusters as features in projection based on word alignments. [sent-164, score-1.221]
79 In the experiments, we tried to project clusters trained on English Wikipedia to Chinese words. [sent-166, score-0.301]
80 1% and the result was about 20% higher than that achieved by the direct projection method, showing that even using exactly the same resources, the proposed method outperformed that in (T ¨ackstr o¨m et al. [sent-168, score-0.803]
81 Next we studied the effects of noise removing methods. [sent-170, score-0.277]
82 Firstly, we removed noises according to Eq(1), which yielded another huge improvement of about 6% against the best results based on cluster features. [sent-171, score-0.336]
83 The results show that both the noise removing methods based on words and on clusters achieved improvements between 1. [sent-173, score-0.579]
84 This is because a great proportion of the vocabulary was made up of person names, some of which are mixed in clusters with common nouns. [sent-178, score-0.382]
85 315 While noise removing method based on clusters failed to recognize them as name entities, cluster bigrams will make use of context information to help the discrimination of these mixed clusters. [sent-179, score-0.718]
86 3 Performances of POS Projection In this section we test our method on projection of POS tagging from English to Chinese, to show that our methods can well extend to other NLP tasks. [sent-185, score-0.743]
87 When one target word is aligned to more than one words with different POS tags on the source side, it is hard to decide which POS tag to choose. [sent-187, score-0.21]
88 So we only retained the data labeled by 1-to-1 alignments, which also contain less noises as pointed out by (Hu et al. [sent-188, score-0.27]
89 Because of the great differences between English and Chinese, projection based on word alignments worked better than direct projection did. [sent-192, score-1.671]
90 After adding word cluster features and removing noises, an error reduction of 12. [sent-193, score-0.337]
91 POS tagging projection can benefit more from our noise removing methods than NER projection could, i. [sent-195, score-1.646]
92 noise removing gave rise to a higher improvement (2. [sent-197, score-0.3]
93 One possible reason is that our noise removing methods assume that labels are associated with single words, which is more suitable for POS tagging. [sent-200, score-0.331]
94 4 Conclusion and perspectives In this paper we introduced Brown clusters of target languages to cross-lingual projection and proposed methods for removing noises on projected labels. [sent-203, score-1.682]
95 Experiments showed that both the two techniques could greatly improve the performances and could help the projection method well generalize to languages differ a lot. [sent-204, score-1.018]
96 Note that although projection methods based on word alignments are less affected by syntactic differences, the topological differences between languages still remain an importance reason for the limitation of performances of cross-lingual projection. [sent-205, score-1.197]
97 In the future we will try to make use of representations of sub-structures to deal with syntactic differences in more complex tasks such as projection of dependency parsing. [sent-206, score-0.798]
98 Future improvements also include combining the direct projection methods based on joint feature representations with the proposed method as well as making use of projected data from multiple languages. [sent-207, score-1.15]
99 Cross-lingual word clusters for direct transfer of linguistic structure. [sent-293, score-0.382]
100 Inducing multilingual text analysis tools via robust projection across aligned corpora. [sent-327, score-0.698]
wordName wordTfidf (topN-words)
[('projection', 0.666), ('projected', 0.243), ('clusters', 0.238), ('noises', 0.176), ('brown', 0.164), ('ackstr', 0.159), ('removing', 0.151), ('chinese', 0.15), ('ner', 0.148), ('cluster', 0.137), ('noise', 0.126), ('languages', 0.121), ('performances', 0.12), ('alignments', 0.108), ('rongji', 0.093), ('representations', 0.088), ('target', 0.087), ('pos', 0.077), ('person', 0.073), ('names', 0.072), ('parallel', 0.072), ('projections', 0.069), ('affected', 0.063), ('labeled', 0.061), ('ci', 0.061), ('direct', 0.06), ('petrov', 0.058), ('labels', 0.054), ('politicians', 0.05), ('yi', 0.05), ('broader', 0.05), ('word', 0.049), ('das', 0.047), ('families', 0.047), ('great', 0.045), ('differences', 0.044), ('xia', 0.043), ('tseng', 0.043), ('source', 0.042), ('mcdonald', 0.041), ('zhu', 0.041), ('side', 0.041), ('sighan', 0.04), ('alignment', 0.04), ('method', 0.04), ('gan', 0.039), ('trained', 0.038), ('differ', 0.037), ('achieved', 0.037), ('harbin', 0.037), ('tagging', 0.037), ('wu', 0.037), ('luo', 0.036), ('firstly', 0.035), ('transfer', 0.035), ('hwa', 0.035), ('universal', 0.035), ('bigram', 0.035), ('disadvantage', 0.035), ('arxiv', 0.035), ('greatly', 0.034), ('entities', 0.034), ('accurate', 0.034), ('worked', 0.033), ('retained', 0.033), ('build', 0.032), ('aligned', 0.032), ('yarowsky', 0.032), ('named', 0.032), ('english', 0.032), ('label', 0.031), ('difficulties', 0.03), ('impossible', 0.03), ('easier', 0.029), ('unlabeled', 0.029), ('since', 0.028), ('cover', 0.027), ('multisource', 0.027), ('bakeoff', 0.027), ('imitate', 0.027), ('interlinear', 0.027), ('appeared', 0.027), ('xue', 0.027), ('improvements', 0.027), ('generality', 0.027), ('secondly', 0.027), ('making', 0.026), ('minor', 0.026), ('china', 0.026), ('limitation', 0.026), ('mixed', 0.026), ('tried', 0.025), ('built', 0.025), ('hu', 0.024), ('argmaxy', 0.024), ('badly', 0.024), ('field', 0.024), ('space', 0.024), ('giza', 0.023), ('improvement', 0.023)]
simIndex simValue paperId paperTitle
same-paper 1 1.0000002 97 acl-2013-Cross-lingual Projections between Languages from Different Families
Author: Mo Yu ; Tiejun Zhao ; Yalong Bai ; Hao Tian ; Dianhai Yu
Abstract: Cross-lingual projection methods can benefit from resource-rich languages to improve performances of NLP tasks in resources-scarce languages. However, these methods confronted the difficulty of syntactic differences between languages especially when the pair of languages varies greatly. To make the projection method well-generalize to diverse languages pairs, we enhance the projection method based on word alignments by introducing target-language word representations as features and proposing a novel noise removing method based on these word representations. Experiments showed that our methods improve the performances greatly on projections between English and Chinese.
2 0.49081704 70 acl-2013-Bilingually-Guided Monolingual Dependency Grammar Induction
Author: Kai Liu ; Yajuan Lu ; Wenbin Jiang ; Qun Liu
Abstract: This paper describes a novel strategy for automatic induction of a monolingual dependency grammar under the guidance of bilingually-projected dependency. By moderately leveraging the dependency information projected from the parsed counterpart language, and simultaneously mining the underlying syntactic structure of the language considered, it effectively integrates the advantages of bilingual projection and unsupervised induction, so as to induce a monolingual grammar much better than previous models only using bilingual projection or unsupervised induction. We induced dependency gram- mar for five different languages under the guidance of dependency information projected from the parsed English translation, experiments show that the bilinguallyguided method achieves a significant improvement of 28.5% over the unsupervised baseline and 3.0% over the best projection baseline on average.
3 0.41325825 136 acl-2013-Enhanced and Portable Dependency Projection Algorithms Using Interlinear Glossed Text
Author: Ryan Georgi ; Fei Xia ; William D. Lewis
Abstract: As most of the world’s languages are under-resourced, projection algorithms offer an enticing way to bootstrap the resources available for one resourcepoor language from a resource-rich language by means of parallel text and word alignment. These algorithms, however, make the strong assumption that the language pairs share common structures and that the parse trees will resemble one another. This assumption is useful but often leads to errors in projection. In this paper, we will address this weakness by using trees created from instances of Interlinear Glossed Text (IGT) to discover patterns of divergence between the lan- guages. We will show that this method improves the performance of projection algorithms significantly in some languages by accounting for divergence between languages using only the partial supervision of a few corrected trees.
4 0.32198387 98 acl-2013-Cross-lingual Transfer of Semantic Role Labeling Models
Author: Mikhail Kozhevnikov ; Ivan Titov
Abstract: Semantic Role Labeling (SRL) has become one of the standard tasks of natural language processing and proven useful as a source of information for a number of other applications. We address the problem of transferring an SRL model from one language to another using a shared feature representation. This approach is then evaluated on three language pairs, demonstrating competitive performance as compared to a state-of-the-art unsupervised SRL system and a cross-lingual annotation projection baseline. We also consider the contribution of different aspects of the feature representation to the performance of the model and discuss practical applicability of this method. 1 Background and Motivation Semantic role labeling has proven useful in many natural language processing tasks, such as question answering (Shen and Lapata, 2007; Kaisser and Webber, 2007), textual entailment (Sammons et al., 2009), machine translation (Wu and Fung, 2009; Liu and Gildea, 2010; Gao and Vogel, 2011) and dialogue systems (Basili et al., 2009; van der Plas et al., 2009). Multiple models have been designed to automatically predict semantic roles, and a considerable amount of data has been annotated to train these models, if only for a few more popular languages. As the annotation is costly, one would like to leverage existing resources to minimize the human effort required to construct a model for a new language. A number of approaches to the construction of semantic role labeling models for new languages have been proposed. On one end of the scale is unsupervised SRL, such as Grenager and Manning (2006), which requires some expert knowledge, but no labeled data. It clusters together arguments that should bear the same semantic role, but does not assign a particular role to each cluster. On the other end is annotating a new dataset from scratch. There are also intermediate options, which often make use of similarities between languages. This way, if an accurate model exists for one language, it should help simplify the construction of a model for another, related language. The approaches in this third group often use parallel data to bridge the gap between languages. Cross-lingual annotation projection systems (Pad o´ and Lapata, 2009), for example, propagate information directly via word alignment links. However, they are very sensitive to the quality of parallel data, as well as the accuracy of a sourcelanguage model on it. An alternative approach, known as cross-lingual model transfer, or cross-lingual model adaptation, consists of modifying a source-language model to make it directly applicable to a new language. This usually involves constructing a shared feature representation across the two languages. McDonald et al. (201 1) successfully apply this idea to the transfer of dependency parsers, using part-of- speech tags as the shared representation of words. A later extension of T ¨ackstr o¨m et al. (2012) enriches this representation with cross-lingual word clusters, considerably improving the performance. In the case of SRL, a shared representation that is purely syntactic is likely to be insufficient, since structures with different semantics may be realized by the same syntactic construct, for example “in August” vs “in Britain”. However with the help of recently introduced cross-lingual word represen1190 Proce dingsS o f ita h,e B 5u1lgsta Arinan,u Aaulg Musete 4ti-n9g 2 o0f1 t3h.e ? Ac s2s0o1ci3a Atiosnso fcoirat Cio nm foprut Caotimonpaulta Lti nognuails Lti cnsg,u piasgteics 1 90–120 , tations, such as the cross-lingual clustering mentioned above or cross-lingual distributed word representations of Klementiev et al. (2012), we may be able to transfer models of shallow semantics in a similar fashion. In this work we construct a shared feature representation for a pair of languages, employing crosslingual representations of syntactic and lexical information, train a semantic role labeling model on one language and apply it to the other one. This approach yields an SRL model for a new language at a very low cost, effectively requiring only a source language model and parallel data. We evaluate on five (directed) language pairs EN-ZH, ZH-EN, EN-CZ, CZ-EN and EN-FR, where EN, FR, CZ and ZH denote English, French, Czech and Chinese, respectively. The transferred model is compared against two baselines: an unsupervised SRL system and a model trained on the output of a cross-lingual annotation projection system. In the next section we will describe our setup, then in section 3 present the shared feature representation we use, discuss the evaluation data and other technical aspects in section 4, present the results and conclude with an overview of related work. – 2 Setup The purpose of the study is not to develop a yet another semantic role labeling system any existing SRL system can (after some modification) be used in this setup but to assess the practical applicability of cross-lingual model transfer to this – – problem, compare it against the alternatives and identify its strong/weak points depending on a particular setup. 2.1 Semantic Role Labeling Model We consider the dependency-based version of semantic role labeling as described in Haji cˇ et al. (2009) and transfer an SRL model from one language to another. We only consider verbal predicates and ignore the predicate disambiguation stage. We also assume that the predicate identification information is available in most languages it can be obtained using a relatively simple heuristic based on part-of-speech tags. The model performs argument identification and classification (Johansson and Nugues, 2008) separately in a pipeline first each candidate is classified as being or not being a head of an argument phrase with respect to the predicate in question and then each of the arguments is assigned a role from a given inventory. The model is factorized over arguments the decisions regarding the classification of different arguments are made in– – – dependently of each other. With respect to the use of syntactic annotation we consider two options: using an existing dependency parser for the target language and obtaining one by means of cross-lingual transfer (see section 4.2). Following McDonald et al. (201 1), we assume that a part-of-speech tagger is available for the target language. 2.2 SRL in the Low-resource Setting Several approaches have been proposed to obtain an SRL model for a new language with little or no manual annotation. Unsupervised SRL models (Lang and Lapata, 2010) cluster the arguments of predicates in a given corpus according to their semantic roles. The performance of such models can be impressive, especially for those languages where semantic roles correlate strongly with syntactic relation of the argument to its predicate. However, assigning meaningful role labels to the resulting clusters requires additional effort and the model’s parameters generally need some adjustment for every language. If the necessary resources are already available for a closely related language, they can be utilized to facilitate the construction of a model for the target language. This can be achieved either by means of cross-lingual annotation projection (Yarowsky et al., 2001) or by cross-lingual model transfer (Zeman and Resnik, 2008). This last approach is the one we are considering in this work, and the other two options are treated as baselines. The unsupervised model will be further referred to as UNSUP and the projection baseline as PROJ. 2.3 Evaluation Measures We use the F1 measure as a metric for the argument identification stage and accuracy as an aggregate measure of argument classification performance. When comparing to the unsupervised SRL system the clustering evaluation measures are used instead. These are purity and collocation 1191 N1Ximajx|Gj∩ Ci| CO =N1Xjmiax|Gj∩ Ci|, PU = where Ci is the set of arguments in the i-th induced cluster, Gj is the set of arguments in the jth gold cluster and N is the total number of arguments. We report the harmonic mean ofthe two (Lang and Lapata, 2011) and denote it F1c to avoid confusing it with the supervised metric. 3 Model Transfer The idea of this work is to abstract the model away from the particular source language and apply it to a new one. This setup requires that we use the same feature representation for both languages, for example part-of-speech tags and dependency relation labels should be from the same inventory. Some features are not applicable to certain lan- guages because the corresponding phenomena are absent in them. For example, consider a strongly inflected language and an analytic one. While the latter can usually convey the information encoded in the word form in the former one (number, gender, etc.), finding a shared feature representation for such information is non-trivial. In this study we will confine ourselves to those features that are applicable to all languages in question, namely: part-of-speech tags, syntactic dependency structures and representations of the word’s identity. 3.1 Lexical Information We train a model on one language and apply it to a different one. In order for this to work, the words of the two languages have to be mapped into a common feature space. It is also desirable that closely related words from both languages have similar representations in this space. Word mapping. The first option is simply to use the source language words as the shared representation. Here every source language word would have itself as its representation and every target word would map into a source word that corresponds to it. In other words, we supply the model with a gloss of the target sentence. The mapping (bilingual dictionary) we use is derived from a word-aligned parallel corpus, by identifying, for each word in the target language, the word in the source language it is most often aligned to. Cross-lingual clusters. There is no guarantee that each of the words in the evaluation data is present in our dictionary, nor that the corresponding source-language word is present in the training data, so the model would benefit from the ability to generalize over closely related words. This can, for example, be achieved by using cross-lingual word clusters induced in T ¨ackstr o¨m et al. (2012). We incorporate these clusters as features into our model. 3.2 Syntactic Information Part-of-speech Tags. We map part-of-speech tags into the universal tagset following Petrov et al. (2012). This may have a negative effect on the performance of a monolingual model, since most part-of-speech tagsets are more fine-grained than the universal POS tags considered here. For example Penn Treebank inventory contains 36 tags and the universal POS tagset only 12. Since the finergrained POS tags often reflect more languagespecific phenomena, however, they would only be useful for very closely related languages in the cross-lingual setting. The universal part-of-speech tags used in evaluation are derived from gold-standard annotation for all languages except French, where predicted ones had to be used instead. Dependency Structure. Another important aspect of syntactic information is the dependency structure. Most dependency relation inventories are language-specific, and finding a shared representation for them is a challenging problem. One could map dependency relations into a simplified form that would be shared between languages, as it is done for part-of-speech tags in Petrov et al. (2012). The extent to which this would be useful, however, depends on the similarity of syntactic-semantic in– terfaces of the languages in question. In this work we discard the dependency relation labels where the inventories do not match and only consider the unlabeled syntactic dependency graph. Some discrepancies, such as variations in attachment order, may be present even there, but this does not appear to be the case with the datasets we use for evaluation. If a target language is poor in resources, one can obtain a dependency parser for the target language by means of cross-lingual model transfer (Zeman and Resnik, 2008). We 1192 take this into account and evaluate both using the original dependency structures and the ones obtained by means of cross-lingual model transfer. 3.3 The Model The model we use is based on that of Bj ¨orkelund et al. (2009). It is comprised of a set of linear classifiers trained using Liblinear (Fan et al., 2008). The feature model was modified to accommodate the cross-lingual cluster features and the reranker component was not used. We do not model the interaction between different argument roles in the same predicate. While this has been found useful, in the cross-lingual setup one has to be careful with the assumptions made. For example, modeling the sequence of roles using a Markov chain (Thompson et al., 2003) may not work well in the present setting, especially between distant languages, as the order or arguments is not necessarily preserved. Most constraints that prove useful for SRL (Chang et al., 2007) also require customization when applied to a new language, and some rely on languagespecific resources, such as a valency lexicon. Taking into account the interaction between different arguments of a predicate is likely to improve the performance of the transferred model, but this is outside the scope of this work. 3.4 Feature Selection Compatibility of feature representations is necessary but not sufficient for successful model transfer. We have to make sure that the features we use are predictive of similar outcomes in the two languages as well. Depending on the pair of languages in question, different aspects of the feature representation will retain or lose their predictive power. We can be reasonably certain that the identity of an argument word is predictive of its semantic role in any language, but it might or might not be true of, for example, the word directly preceding the argument word. It is therefore important to pre- SCPDGylOespoSntreslTabunc1lra:obsFel-daitnguplrdoaeusntpagd-elronwfu-dcsopeyrnsd c.eylafguhtorsia mepgnrhs vent the model from capturing overly specific aspects of the source language, which we do by confining the model to first-order features. We also avoid feature selection, which, performed on the source language, is unlikely to help the model to better generalize to the target one. The experiments confirm that feature selection and the use of second-order features degrade the performance of the transferred model. 3.5 Feature Groups For each word, we use its part-of-speech tag, cross-lingual cluster id, word identity (glossed, when evaluating on the target language) and its dependency relation to its parent. Features associated with an argument word include the attributes of the predicate word, the argument word, its parent, siblings and children, and the words directly preceding and following it. Also included are the sequences of part-of-speech tags and dependency relations on the path between the predicate and the argument. Since we are also interested in the impact of different aspects of the feature representation, we divide the features into groups as summarized in table 1 and evaluate their respective contributions to the performance of the model. If a feature group is enabled the model has access to the corre– sponding source of information. For example, if only POS group is enabled, the model relies on the part-of-speech tags of the argument, the predicate and the words to the right and left of the argument word. If Synt is enabled too, it also uses the POS tags of the argument’s parent, children and siblings. Word order information constitutes an implicit group that is always available. It includes the Pos it ion feature, which indicates whether the argument is located to the left or to the right of the predicate, and allows the model to look up the attributes of the words directly preceding and following the argument word. The model we compare against the baselines uses all applicable feature groups (Deprel is only used in EN-CZ and CZ-EN experiments with original syntax). 4 Evaluation 4.1 Datasets and Preprocessing Evaluation of the cross-lingual model transfer requires a rather specific kind of dataset. Namely, the data in both languages has to be annotated 1193 with the same set of semantic roles following the same (or compatible) guidelines, which is seldom the case. We have identified three language pairs for which such resources are available: EnglishChinese, English-Czech and English-French. The evaluation datasets for English and Chinese are those from the CoNLL Shared Task 2009 (Haji ˇc et al., 2009) (henceforth CoNLL-ST). Their annotation in the CoNLL-ST is not identical, but the guidelines for “core” semantic roles are similar (Kingsbury et al., 2004), so we evaluate only on core roles here. The data for the second language pair is drawn from the Prague Czech-English Dependency Treebank 2.0 (Haji ˇc et al., 2012), which we converted to a format similar to that of CoNLL-ST1 . The original annotation uses the tectogrammatical representation (Haji ˇc, 2002) and an inventory of semantic roles (or functors), most of which are interpretable across various predicates. Also note that the syntactic anno- tation of English and Czech in PCEDT 2.0 is quite similar (to the extent permitted by the difference in the structure of the two languages) and we can use the dependency relations in our experiments. For English-French, the English CoNLL-ST dataset was used as a source and the model was evaluated on the manually annotated dataset from van der Plas et al. (201 1). The latter contains one thousand sentences from the French part ofthe Europarl (Koehn, 2005) corpus, annotated with semantic roles following an adapted version of PropBank (Palmer et al., 2005) guidelines. The authors perform annotation projection from English to French, using a joint model of syntax and semantics and employing heuristics for filtering. We use a model trained on the output of this projection system as one of the baselines. The evaluation dataset is relatively small in this case, so we perform the transfer only one-way, from English to French. The part-of-speech tags in all datasets were replaced with the universal POS tags of Petrov et al. (2012). For Czech, we have augmented the map- pings to account for the tags that were not present in the datasets from which the original mappings were derived. Namely, tag “t” is mapped to “VERB” and “Y” to “PRON”. We use parallel data to construct a bilingual dictionary used in word mapping, as well as in the projection baseline. For English-Czech – 1see http://www.ml4nlp.de/code-and-data/treex2conll and English-French, the data is drawn from Europarl (Koehn, 2005), for English-Chinese from MultiUN (Eisele and Chen, 2010). The word alignments were obtained using GIZA++ (Och and Ney, 2003) and the intersection heuristic. – 4.2 Syntactic Transfer In the low-resource setting, we cannot always rely on the availability of an accurate dependency parser for the target language. If one is not available, the natural solution would be to use crosslingual model transfer to obtain it. Unfortunately, the models presented in the previous work, such as Zeman and Resnik (2008), McDonald et al. (201 1) and T ¨ackstr o¨m et al. (2012), were not made available, so we reproduced the direct transfer algorithm of McDonald et al. (201 1), using Malt parser (Nivre, 2008) and the same set of features. We did not reimplement the projected transfer algorithm, however, and used the default training procedure instead of perceptron-based learning. The dependency structure thus obtained is, of course, only a rough approximation even a much more sophisticated algorithm may not perform well when transferring syntax between such languages as Czech and English, given the inherent difference in their structure. The scores are shown in table 2. We will henceforth refer to the syntactic annotations that were provided with the datasets as original, as opposed to the annotations obtained by means of syntactic transfer. – 4.3 Baselines Unsupervised Baseline: We are using a version of the unsupervised semantic role induction system of Titov and Klementiev (2012a) adapted to SetupUAS, % Table2:SyntaciE C ZcN HNt- rE ZaCFnN HZRsfer34 692567acuracy,unlabe dat- tachment score (percent). Note that in case of French we evaluate against the output of a supervised system, since manual annotation is not available for this dataset. This score does not reflect the true performance of syntactic transfer. 1194 the shared feature representation considered in order to make the scores comparable with those of the transfer model and, more importantly, to enable evaluation on transferred syntax. Note that the original system, tailored to a more expressive language-specific syntactic representation and equipped with heuristics to identify active/passive voice and other phenomena, achieves higher scores than those we report here. Projection Baseline: The projection baseline we use for English-Czech and English-Chinese is a straightforward one: we label the source side of a parallel corpus using the source-language model, then identify those verbs on the target side that are aligned to a predicate, mark them as predicates and propagate the argument roles in the same fashion. A model is then trained on the resulting training data and applied to the test set. For English-French we instead use the output of a fully featured projection model of van der Plas et al. (201 1), published in the CLASSiC project. 5 Results In order to ensure that the results are consistent, the test sets, except for the French one, were partitioned into five equal parts (of 5 to 10 thousand sentences each, depending on the dataset) and the evaluation performed separately on each one. All evaluation figures for English, Czech or Chinese below are the average values over the five subsets. In case of French, the evaluation dataset is too small to split it further, so instead we ran the evaluation five times on a randomly selected 80% sample of the evaluation data and averaged over those. In both cases the results are consistent over the subsets, the standard deviation does not exceed 0.5% for the transfer system and projection baseline and 1% for the unsupervised system. 5.1 Argument Identification We summarize the results in table 3. Argument identification is known to rely heavily on syntactic information, so it is unsurprising that it proves inaccurate when transferred syntax is used. Our simple projection baseline suffers from the same problem. Even with original syntactic information available, the performance of argument identification is moderate. Note that the model of (van der Plas et al., 2011), though relying on more expressive syntax, only outperforms the transferred system by 3% (F1) on this task. SetupSyntaxTRANSPROJ ZEC NH Z- EFCZNRHt r a n s 3462 1. 536 142 35. 4269 Table3EZ C:N H- CFEZANHZRrgumeon rt ig identf56 7ic13 a. t27903ion,21569t10ra. 3976nsferd model vs. projection baseline, F1. Most unsupervised SRL approaches assume that the argument identification is performed by some external means, for example heuristically (Lang and Lapata, 2011). Such heuristics or unsupervised approaches to argument identification (Abend et al., 2009) can also be used in the present setup. 5.2 Argument Classification In the following tables, TRANS column contains the results for the transferred system, UNSUP for the unsupervised baseline and PROJ for projection baseline. We highlight in bold the higher score where the difference exceeds twice the maximum of the standard deviation estimates of the two results. Table 4 presents the unsupervised evaluation results. Note that the unsupervised model performs as well as the transferred one or better where the – – SetupSyntaxTRANSUNSUP ZEC NH Z- EFCZNRHt r a n s 768 93648. 34627 6 5873. 1769 TableEZ C4NHZ:- FCEZANHZRrgumoe nr itg clasi78 fi94 3c. a25136tion,8 7 r9a4263n. 07 sferd model vs. unsupervised baseline in terms of the clustering metric F1c (see section 2.3). 1195 SetupSyntaxTRANSPROJ ZEC NH Z- EFCZNRHt r a n s 657 053. 1 36456419. 372 Table5EZ C:N H- CFEZANHZRrgumeon rt ig clasif657ic1936a. t170 ion,65 9t3804ra. 20847nsferd model vs. projection baseline, accuracy. original syntactic dependencies are available. In the more realistic scenario with transferred syn- tax, however, the transferred model proves more accurate. In table 5 we compare the transferred system with the projection baseline. It is easy to see that the scores vary strongly depending on the language pair, due to both the difference in the annotation scheme used and the degree of relatedness between the languages. The drop in performance when transferring the model to another language is large in every case, though, see table 6. SetupTargetSource Table6:MoCEZdHeNZ l- FECaZNRcH urac67 y53169o. 017nthes87 o25670u. r1245ceandtrge language using original syntax. The source language scores for English vary between language pairs because of the difference in syntactic annotation and role subset used. We also include the individual F1 scores for the top-10 most frequent labels for EN-CZ transfer with original syntax in table 7. The model provides meaningful predictions here, despite low overall accuracy. Most of the labels2 are self-explanatory: Patient (PAT), Actor (ACT), Time (TWHEN), Effect (EFF), Location (LOC), Manner (MANN), Addressee (ADDR), Extent (EXT). CPHR marks the 2http://ufal.mff.cuni.cz/∼toman/pcedt/en/functors.html LabelFreq.F1Re.Pr. recall and precision for the top-10 most frequent roles. nominal part of a complex predicate, as in “to have [a plan]CPHR”, and DIR3 indicates destination. 5.3 Additional Experiments We now evaluate the contribution of different aspects of the feature representation to the performance of the model. Table 8 contains the results for English-French. FeaturesOrigTrans ferent feature subsets, using original and transferred syntactic information. The fact that the model performs slightly better with transferred syntax may be explained by two factors. Firstly, as we already mentioned, the original syntactic annotation is also produced automatically. Secondly, in the model transfer setup it is more important how closely the syntacticsemantic interface on the target side resembles that on the source side than how well it matches the “true” structure of the target language, and in this respect a transferred dependency parser may have an advantage over one trained on target-language data. The high impact of the Glos s features here 1196 may be partly attributed to the fact that the mapping is derived from the same corpus as the evaluation data Europarl (Koehn, 2005) and partly by the similarity between English and French in terms of word order, usage of articles and prepositions. The moderate contribution of the crosslingual cluster features are likely due to the insufficient granularity of the clustering for this task. For more distant language pairs, the contributions of individual feature groups are less interpretable, so we only highlight a few observations. First of all, both EN-CZ and CZ-EN benefit noticeably from the use of the original syntactic annotation, including dependency relations, but not from the transferred syntax, most likely due to the low syntactic transfer performance. Both perform better when lexical information is available, although – – the improvement is not as significant as in the case of French only up to 5%. The situation with Chinese is somewhat complicated in that adding lexical information here fails to yield an improvement in terms of the metric considered. This is likely due to the fact that we consider only the core roles, which can usually be predicted with high accuracy based on syntactic information alone. – 6 Related Work Development of robust statistical models for core NLP tasks is a challenging problem, and adaptation of existing models to new languages presents a viable alternative to exhaustive annotation for each language. Although the models thus obtained are generally imperfect, they can be further refined for a particular language and domain using techniques such as active learning (Settles, 2010; Chen et al., 2011). Cross-lingual annotation projection (Yarowsky et al., 2001) approaches have been applied ex- tensively to a variety of tasks, including POS tagging (Xi and Hwa, 2005; Das and Petrov, 2011), morphology segmentation (Snyder and Barzilay, 2008), verb classification (Merlo et al., 2002), mention detection (Zitouni and Florian, 2008), LFG parsing (Wr o´blewska and Frank, 2009), information extraction (Kim et al., 2010), SRL (Pad o´ and Lapata, 2009; van der Plas et al., 2011; Annesi and Basili, 2010; Tonelli and Pianta, 2008), dependency parsing (Naseem et al., 2012; Ganchev et al., 2009; Smith and Eisner, 2009; Hwa et al., 2005) or temporal relation prediction (Spreyer and Frank, 2008). Interestingly, it has also been used to propagate morphosyntactic information between old and modern versions of the same language (Meyer, 2011). Cross-lingual model transfer methods (McDonald et al., 2011; Zeman and Resnik, 2008; Durrett et al., 2012; Søgaard, 2011; Lopez et al., 2008) have also been receiving much attention recently. The basic idea behind model transfer is similar to that of cross-lingual annotation projection, as we can see from the way parallel data is used in, for example, McDonald et al. (201 1). A crucial component of direct transfer approaches is the unified feature representation. There are at least two such representations of lexical information (Klementiev et al., 2012; T ¨ackstr o¨m et al., 2012), but both work on word level. This makes it hard to account for phenomena that are expressed differently in the languages considered, for example the syntactic function of a certain word may be indicated by a preposition, inflection or word order, depending on the language. Accurate representation of such information would require an extra level of abstraction (Haji ˇc, 2002). A side-effect ofusing adaptation methods is that we are forced to use the same annotation scheme for the task in question (SRL, in our case), which in turn simplifies the development of cross-lingual tools for downstream tasks. Such representations are also likely to be useful in machine translation. Unsupervised semantic role labeling methods (Lang and Lapata, 2010; Lang and Lapata, 2011; Titov and Klementiev, 2012a; Lorenzo and Cerisara, 2012) also constitute an alternative to cross-lingual model transfer. For an overview of of semi-supervised approaches we refer the reader to Titov and Klementiev (2012b). 7 Conclusion We have considered the cross-lingual model transfer approach as applied to the task of semantic role labeling and observed that for closely related languages it performs comparably to annotation projection approaches. It allows one to quickly construct an SRL model for a new language without manual annotation or language-specific heuristics, provided an accurate model is available for one of the related languages along with a certain amount of parallel data for the two languages. While an1197 notation projection approaches require sentenceand word-aligned parallel data and crucially depend on the accuracy of the syntactic parsing and SRL on the source side of the parallel corpus, cross-lingual model transfer can be performed using only a bilingual dictionary. Unsupervised SRL approaches have their advantages, in particular when no annotated data is available for any of the related languages and there is a syntactic parser available for the target one, but the annotation they produce is not always sufficient. In applications such as Information Retrieval it is preferable to have precise labels, rather than just clusters of arguments, for example. Also note that when applying cross-lingual model transfer in practice, one can improve upon the performance of the simplistic model we use for evaluation, for example by picking the features manually, taking into account the properties of the target language. Domain adaptation techniques can also be employed to adjust the model to the target language. Acknowledgments The authors would like to thank Alexandre Klementiev and Ryan McDonald for useful suggestions and T ¨ackstr o¨m et al. (2012) for sharing the cross-lingual word representations. This research is supported by the MMCI Cluster of Excellence. References Omri Abend, Roi Reichart, and Ari Rappoport. 2009. Unsupervised argument identification for semantic role labeling. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, ACL ’09, pages 28–36, Stroudsburg, PA, USA. Association for Computational Linguistics. Paolo Annesi and Roberto Basili. 2010. Cross-lingual alignment of FrameNet annotations through hidden Markov models. In Proceedings of the 11th international conference on Computational Linguistics and Intelligent Text Processing, CICLing’ 10, pages 12– 25, Berlin, Heidelberg. Springer-Verlag. Roberto Basili, Diego De Cao, Danilo Croce, Bonaventura Coppola, and Alessandro Moschitti. 2009. Cross-language frame semantics transfer in bilingual corpora. In Alexander F. Gelbukh, editor, Proceedings of the 10th International Conference on Computational Linguistics and Intelligent Text Pro- cessing, pages 332–345. Anders Bj ¨orkelund, Love Hafdell, and Pierre Nugues. 2009. Multilingual semantic role labeling. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL 2009): Shared Task, pages 43–48, Boulder, Colorado, June. Association for Computational Linguistics. Ming-Wei Chang, Lev Ratinov, and Dan Roth. 2007. Guiding semi-supervision with constraint-driven learning. In ACL. Chenhua Chen, Alexis Palmer, and Caroline Sporleder. 2011. Enhancing active learning for semantic role labeling via compressed dependency trees. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 183–191, Chiang Mai, Thailand, November. Asian Federation of Natural Language Processing. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. Proceedings of the Association for Computational Linguistics. Greg Durrett, Adam Pauls, and Dan Klein. 2012. Syntactic transfer using a bilingual lexicon. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1–1 1, Jeju Island, Korea, July. Association for Computational Linguistics. Andreas Eisele and Yu Chen. 2010. MultiUN: A multilingual corpus from United Nation documents. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA). Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, XiangRui Wang, and Chih-Jen Lin. 2008. LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research, 9: 1871–1874. Kuzman Ganchev, Jennifer Gillenwater, and Ben Taskar. 2009. Dependency grammar induction via bitext projection constraints. In Proceedings of the 47th Annual Meeting of the ACL, pages 369–377, Stroudsburg, PA, USA. Association for Computational Linguistics. Qin Gao and Stephan Vogel. 2011. Corpus expansion for statistical machine translation with semantic role label substitution rules. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 294–298, Portland, Oregon, USA. Trond Grenager and Christopher D. Manning. 2006. Unsupervised discovery of a statistical verb lexicon. In Proceedings of EMNLP. Jan Haji cˇ. 2002. Tectogrammatical representation: Towards a minimal transfer in machine translation. In Robert Frank, editor, Proceedings of the 6th International Workshop on Tree Adjoining Grammars 1198 and Related Frameworks (TAG+6), pages 216— 226, Venezia. Universita di Venezia. Jan Haji cˇ, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Ant o`nia Mart ı´, Llu ı´s M `arquez, Adam Meyers, Joakim Nivre, Sebastian Pad o´, Jan Sˇt eˇp a´nek, Pavel Stra nˇ a´k, Mihai Surdeanu, Nianwen Xue, and Yi Zhang. 2009. The CoNLL2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL 2009): Shared Task, pages 1–18, Boulder, Colorado. Jan Haji cˇ, Eva Haji cˇov a´, Jarmila Panevov a´, Petr Sgall, Ond ˇrej Bojar, Silvie Cinkov´ a, Eva Fuˇ c ´ıkov a´, Marie Mikulov a´, Petr Pajas, Jan Popelka, Ji ˇr´ ı Semeck´ y, Jana Sˇindlerov a´, Jan Sˇt eˇp a´nek, Josef Toman, Zde nˇka Ure sˇov a´, and Zden eˇk Zˇabokrtsk y´. 2012. Announcing Prague Czech-English dependency treebank 2.0. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Mehmet U gˇur Doˇ gan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey, May. European Language Resources Association (ELRA). Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel text. Natural Language Engineering, 11(3):3 11–325. Richard Johansson and Pierre Nugues. 2008. Dependency-based semantic role labeling of PropBank. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 69–78, Honolulu, Hawaii. Michael Kaisser and Bonnie Webber. 2007. Question answering based on semantic roles. In ACL Workshop on Deep Linguistic Processing. Seokhwan Kim, Minwoo Jeong, Jonghoon Lee, and Gary Geunbae Lee. 2010. A cross-lingual annotation projection approach for relation detection. In Proceedings of the 23rd International Conference on Computational Linguistics, COLING ’ 10, pages 564–571, Stroudsburg, PA, USA. Association for Computational Linguistics. Paul Kingsbury, Nianwen Xue, and Martha Palmer. 2004. Propbanking in parallel. In In Proceedings of the Workshop on the Amazing Utility of Parallel and Comparable Corpora, in conjunction with LREC’04. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of the International Conference on Computational Linguistics (COLING), Bombay, India. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Conference Proceedings: the tenth Machine Translation Summit, pages 79–86, Phuket, Thailand. AAMT. Joel Lang and Mirella Lapata. 2010. Unsupervised induction of semantic roles. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 939–947, Los Angeles, California, June. Association for Computational Linguistics. Joel Lang and Mirella Lapata. 2011. Unsupervised semantic role induction via split-merge clustering. In Proc. of Annual Meeting of the Association for Computational Linguistics (ACL). Ding Liu and Daniel Gildea. 2010. Semantic role features for machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), Beijing, China. Adam Lopez, Daniel Zeman, Michael Nossal, Philip Resnik, and Rebecca Hwa. 2008. Cross-language parser adaptation between related languages. In IJCNLP-08 Workshop on NLP for Less Privileged Languages, pages 35–42, Hyderabad, India, January. Alejandra Lorenzo and Christophe Cerisara. 2012. Unsupervised frame based semantic role induction: application to French and English. In Proceedings of the ACL 2012 Joint Workshop on Statistical Parsing and Semantic Processing of Morphologically Rich Languages, pages 30–35, Jeju, Republic of Korea, July. Association for Computational Linguistics. Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’ 11, pages 62–72, Stroudsburg, PA, USA. Association for Computational Linguistics. Paola Merlo, Suzanne Stevenson, Vivian Tsang, and Gianluca Allaria. 2002. A multi-lingual paradigm for automatic verb classification. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL’02), pages 207– 214, Philadelphia, PA. Roland Meyer. 2011. New wine in old wineskins?– Tagging old Russian via annotation projection from modern translations. Russian Linguistics, 35(2):267(15). Tahira Naseem, Regina Barzilay, and Amir Globerson. 2012. Selective sharing for multilingual dependency parsing. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, pages 629–637, Jeju Island, Korea, July. Association for Computational Linguistics. Joakim Nivre. 2008. Algorithms for deterministic incremental dependency parsing. Comput. Linguist., 34(4):513–553, December. 1199 Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1). Sebastian Pad o´ and Mirella Lapata. 2009. Crosslingual annotation projection for semantic roles. Journal of Artificial Intelligence Research, 36:307– 340. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31:71–105. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proceedings of LREC, May. Mark Sammons, Vinod Vydiswaran, Tim Vieira, Nikhil Johri, Ming wei Chang, Dan Goldwasser, Vivek Srikumar, Gourab Kundu, Yuancheng Tu, Kevin Small, Joshua Rule, Quang Do, and Dan Roth. 2009. Relation alignment for textual entailment recognition. In Text Analysis Conference (TAC). Burr Settles. 2010. Active learning literature survey. Computer Sciences Technical Report, 1648. Dan Shen and Mirella Lapata. 2007. Using semantic roles to improve question answering. In EMNLP. David A Smith and Jason Eisner. 2009. Parser adaptation and projection with quasi-synchronous grammar features. In Proceedings of the 2009 Confer- ence on Empirical Methods in Natural Language Processing, pages 822–831. Association for Computational Linguistics. Benjamin Snyder and Regina Barzilay. 2008. Crosslingual propagation for morphological analysis. In Proceedings of the 23rd national conference on Artificial intelligence. Anders Søgaard. 2011. Data point selection for crosslanguage adaptation of dependency parsers. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, volume 2 of HLT ’11, pages 682–686, Stroudsburg, PA, USA. Association for Computational Linguistics. Kathrin Spreyer and Anette Frank. 2008. Projectionbased acquisition of a temporal labeller. Proceedings of IJCNLP 2008. Oscar T¨ ackstr o¨m, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proc. of the Annual Meeting of the North American Association of Computational Linguistics (NAACL), pages 477– 487, Montr ´eal, Canada. Cynthia A. Thompson, Roger Levy, and Christopher D. Manning. 2003. A generative model for seman- tic role labeling. In Proceedings of the 14th European Conference on Machine Learning, ECML 2003, pages 397–408, Dubrovnik, Croatia. Ivan Titov and Alexandre Klementiev. 2012a. A Bayesian approach to unsupervised semantic role induction. In Proc. of European Chapter of the Association for Computational Linguistics (EACL). Ivan Titov and Alexandre Klementiev. 2012b. Semisupervised semantic role labeling: Approaching from an unsupervised perspective. In Proceedings of the International Conference on Computational Linguistics (COLING), Bombay, India, December. Sara Tonelli and Emanuele Pianta. 2008. Frame information transfer from English to Italian. In Proceedings of LREC 2008. Lonneke van der Plas, James Henderson, and Paola Merlo. 2009. Domain adaptation with artificial data for semantic parsing of speech. In Proc. 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 125–128, Boulder, Colorado. Lonneke van der Plas, Paola Merlo, and James Henderson. 2011. Scaling up automatic cross-lingual semantic role annotation. In Proceedings of the 49th Annual Meeting of the Association for Computa- tional Linguistics: Human Language Technologies, HLT ’ 11, pages 299–304, Stroudsburg, PA, USA. Association for Computational Linguistics. Alina Wr o´blewska and Anette Frank. 2009. Crosslingual projection of LFG F-structures: Building an F-structure bank for Polish. In Eighth International Workshop on Treebanks and Linguistic Theories, page 209. Dekai Wu and Pascale Fung. 2009. Can semantic role labeling improve SMT? In Proceedings of 13th Annual Conference of the European Association for Machine Translation (EAMT 2009), Barcelona. Chenhai Xi and Rebecca Hwa. 2005. A backoff model for bootstrapping resources for non-English languages. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 85 1–858, Stroudsburg, PA, USA. David Yarowsky, Grace Ngai, and Ricahrd Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of Human Language Technology Conference. Daniel Zeman and Philip Resnik. 2008. Crosslanguage parser adaptation between related lan- guages. In Proceedings of the IJCNLP-08 Workshop on NLP for Less Privileged Languages, pages 35– 42, Hyderabad, India, January. Asian Federation of Natural Language Processing. Imed Zitouni and Radu Florian. 2008. Mention detection crossing the language barrier. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1200
5 0.20178841 323 acl-2013-Simpler unsupervised POS tagging with bilingual projections
Author: Long Duong ; Paul Cook ; Steven Bird ; Pavel Pecina
Abstract: We present an unsupervised approach to part-of-speech tagging based on projections of tags in a word-aligned bilingual parallel corpus. In contrast to the existing state-of-the-art approach of Das and Petrov, we have developed a substantially simpler method by automatically identifying “good” training sentences from the parallel corpus and applying self-training. In experimental results on eight languages, our method achieves state-of-the-art results. 1 Unsupervised part-of-speech tagging Currently, part-of-speech (POS) taggers are available for many highly spoken and well-resourced languages such as English, French, German, Italian, and Arabic. For example, Petrov et al. (2012) build supervised POS taggers for 22 languages using the TNT tagger (Brants, 2000), with an average accuracy of 95.2%. However, many widelyspoken languages including Bengali, Javanese, and Lahnda have little data manually labelled for POS, limiting supervised approaches to POS tagging for these languages. However, with the growing quantity of text available online, and in particular, multilingual parallel texts from sources such as multilingual websites, government documents and large archives ofhuman translations ofbooks, news, and so forth, unannotated parallel data is becoming more widely available. This parallel data can be exploited to bridge languages, and in particular, transfer information from a highly-resourced language to a lesser-resourced language, to build unsupervised POS taggers. In this paper, we propose an unsupervised approach to POS tagging in a similar vein to the work of Das and Petrov (201 1). In this approach, — — pecina@ ufal .mff .cuni . c z a parallel corpus for a more-resourced language having a POS tagger, and a lesser-resourced language, is word-aligned. These alignments are exploited to infer an unsupervised tagger for the target language (i.e., a tagger not requiring manuallylabelled data in the target language). Our approach is substantially simpler than that of Das and Petrov, the current state-of-the art, yet performs comparably well. 2 Related work There is a wealth of prior research on building unsupervised POS taggers. Some approaches have exploited similarities between typologically similar languages (e.g., Czech and Russian, or Telugu and Kannada) to estimate the transition probabilities for an HMM tagger for one language based on a corpus for another language (e.g., Hana et al., 2004; Feldman et al., 2006; Reddy and Sharoff, 2011). Other approaches have simultaneously tagged two languages based on alignments in a parallel corpus (e.g., Snyder et al., 2008). A number of studies have used tag projection to copy tag information from a resource-rich to a resource-poor language, based on word alignments in a parallel corpus. After alignment, the resource-rich language is tagged, and tags are projected from the source language to the target language based on the alignment (e.g., Yarowsky and Ngai, 2001 ; Das and Petrov, 2011). Das and Petrov (201 1) achieved the current state-of-the-art for unsupervised tagging by exploiting high confidence alignments to copy tags from the source language to the target language. Graph-based label propagation was used to automatically produce more labelled training data. First, a graph was constructed in which each vertex corresponds to a unique trigram, and edge weights represent the syntactic similarity between vertices. Labels were then propagated by optimizing a convex function to favor the same tags for closely related nodes 634 Proce dingSsof oifa, th Beu 5l1gsarti Aan,An u aglu Mste 4e-ti9n2g 0 o1f3 t.he ?c A2s0s1o3ci Aatsiosonc fioartio Cno fmorpu Ctoamtiopnuatalt Lioin gauli Lsitnicgsu,i psatgices 634–639, ModelCoverageAccuracy Many-to-1 alignments88%68% 1-to-1 alignments 68% 78% 1-to-1 alignments: Top 60k sents 91% 80% Table 1: Token coverage and accuracy of manyto-one and 1-to-1 alignments, as well as the top 60k sentences based on alignment score for 1-to-1 alignments, using directly-projected labels only. while keeping a uniform tag distribution for unrelated nodes. A tag dictionary was then extracted from the automatically labelled data, and this was used to constrain a feature-based HMM tagger. The method we propose here is simpler to that of Das and Petrov in that it does not require convex optimization for label propagation or a feature based HMM, yet it achieves comparable results. 3 Tagset Our tagger exploits the idea ofprojecting tag information from a resource-rich to resource-poor language. To facilitate this mapping, we adopt Petrov et al.’s (2012) twelve universal tags: NOUN, VERB, ADJ, ADV, PRON (pronouns), DET (de- terminers and articles), ADP (prepositions and postpositions), NUM (numerals), CONJ (conjunctions), PRT (particles), “.” (punctuation), and X (all other categories, e.g., foreign words, abbreviations). These twelve basic tags are common across taggers for most languages. Adopting a universal tagset avoids the need to map between a variety of different, languagespecific tagsets. Furthermore, it makes it possible to apply unsupervised tagging methods to languages for which no tagset is available, such as Telugu and Vietnamese. 4 A Simpler Unsupervised POS Tagger Here we describe our proposed tagger. The key idea is to maximize the amount of information gleaned from the source language, while limiting the amount of noise. We describe the seed model and then explain how it is successively refined through self-training and revision. 4.1 Seed Model The first step is to construct a seed tagger from directly-projected labels. Given a parallel corpus for a source and target language, Algorithm 1provides a method for building an unsupervised tagger for the target language. In typical applications, the source language would be a better-resourced language having a tagger, while the target language would be lesser-resourced, lacking a tagger and large amounts of manually POS-labelled data. Algorithm 1 Build seed model Algorithm 1Build seed model 1:Tag source side. 2: Word align the corpus with Giza++ and remove the many-to-one mappings. 3: Project tags from source to target using the remaining 1-to-1 alignments. 4: Select the top n sentences based on sentence alignment score. 5: Estimate emission and transition probabilities. 6: Build seed tagger T. We eliminate many-to-one alignments (Step 2). Keeping these would give more POS-tagged tokens for the target side, but also introduce noise. For example, suppose English and French were the source and target language, respectively. In this case alignments such as English laws (NNS) to French les (DT) lois (NNS) would be expected (Yarowsky and Ngai, 2001). However, in Step 3, where tags are projected from the source to target language, this would incorrectly tag French les as NN. We build a French tagger based on English– French data from the Europarl Corpus (Koehn, 2005). We also compare the accuracy and coverage of the tags obtained through direct projection using the French Melt POS tagger (Denis and Sagot, 2009). Table 1confirms that the one-to-one alignments indeed give higher accuracy but lower coverage than the many-to-one alignments. At this stage of the model we hypothesize that highconfidence tags are important, and hence eliminate the many-to-one alignments. In Step 4, in an effort to again obtain higher quality target language tags from direct projection, we eliminate all but the top n sentences based on their alignment scores, as provided by the aligner via IBM model 3. We heuristically set this cutoff × to 60k to balance the accuracy and size of the seed model.1 Returning to our preliminary English– French experiments in Table 1, this process gives improvements in both accuracy and coverage.2 1We considered values in the range 60–90k, but this choice had little impact on the accuracy of the model. 2We also considered using all projected labels for the top 60k sentences, not just 1-to-1 alignments, but in preliminary experiments this did not perform as well, possibly due to the previously-observed problems with many-to-one alignments. 635 The number of parameters for the emission probability is |V | |T| where V is the vocabulary and aTb iilsi ttyh eis tag |s e×t. TTh| ew htrearnesi Vtio ins probability, on atnhed other hand, has only |T|3 parameters for the trigram hmaondde,l we use. TB|ecause of this difference in number of parameters, in step 5, we use different strategies to estimate the emission and transition probabilities. The emission probability is estimated from all 60k selected sentences. However, for the transition probability, which has less parameters, we again focus on “better” sentences, by estimating this probability from only those sen- tences that have (1) token coverage > 90% (based on direct projection of tags from the source language), and (2) length > 4 tokens. These criteria aim to identify longer, mostly-tagged sentences, which we hypothesize are particularly useful as training data. In the case of our preliminary English–French experiments, roughly 62% of the 60k selected sentences meet these criteria and are used to estimate the transition probability. For unaligned words, we simply assign a random POS and very low probability, which does not substantially affect transition probability estimates. In Step 6 we build a tagger by feeding the estimated emission and transition probabilities into the TNT tagger (Brants, 2000), an implementation of a trigram HMM tagger. 4.2 Self training and revision For self training and revision, we use the seed model, along with the large number of target language sentences available that have been partially tagged through direct projection, in order to build a more accurate tagger. Algorithm 2 describes this process of self training and revision, and assumes that the parallel source–target corpus has been word aligned, with many-to-one alignments removed, and that the sentences are sorted by alignment score. In contrast to Algorithm 1, all sentences are used, not just the 60k sentences with the highest alignment scores. We believe that sentence alignment score might correspond to difficulty to tag. By sorting the sentences by alignment score, sentences which are more difficult to tag are tagged using a more mature model. Following Algorithm 1, we divide sentences into blocks of 60k. In step 3 the tagged block is revised by comparing the tags from the tagger with those obtained through direct projection. Suppose source Algorithm 2 Self training and revision 1:Divide target language sentences into blocks of n sentences. 2: Tag the first block with the seed tagger. 3: Revise the tagged block. 4: Train a new tagger on the tagged block. 5: Add the previous tagger’s lexicon to the new tagger. 6: Use the new tagger to tag the next block. 7: Goto 3 and repeat until all blocks are tagged. language word wis is aligned with target language word wjt with probability p(wjt |wsi), Tis is the tag for wis using the tagger availa|bwle for the source language, and Tjt is the tag for wjt using the tagger learned for the > S, where S is a threshold which we heuristically set to 0.7, we replace Tjt by Tis. Self-training can suffer from over-fitting, in which errors in the original model are repeated and amplified in the new model (McClosky et al., 2006). To avoid this, we remove the tag of any token that the model is uncertain of, i.e., if p(wjt |wsi) < S and Tjt Tis then Tjt = Null. So, on th|ew target side, aligned words have a tag from direct projection or no tag, and unaligned words have a tag assigned by our model. Step 4 estimates the emission and transition target language. If p(wtj|wis) = probabilities as in Algorithm 1. In Step 5, emission probabilities for lexical items in the previous model, but missing from the current model, are added to the current model. Later models therefore take advantage of information from earlier models, and have wider coverage. 5 Experimental Results Using parallel data from Europarl (Koehn, 2005) we apply our method to build taggers for the same eight target languages as Das and Petrov (201 1) Danish, Dutch, German, Greek, Italian, Portuguese, Spanish and Swedish with English as the source language. Our training data (Europarl) is a subset of the training data of Das and Petrov (who also used the ODS United Nations dataset which we were unable to obtain). The evaluation metric and test data are the same as that used by Das and Petrov. Our results are comparable to theirs, although our system is penalized by having less training data. We tag the source language with the Stanford POS tagger (Toutanova et al., 2003). — — 636 DanishDutchGermanGreekItalianPortugueseSpanishSwedishAverage Seed model83.781.183.677.878.684.981.478.981.3 Self training + revision 85.6 84.0 85.4 80.4 81.4 86.3 83.3 81.0 83.4 Das and Petrov (2011) 83.2 79.5 82.8 82.5 86.8 87.9 84.2 80.5 83.4 Table 2: Token-level POS tagging accuracy for our seed model, self training and revision, and the method of Das and Petrov (201 1). The best results on each language, and on average, are shown in bold. 1 1 Iteration 2 2 3 1 1 2 2 3 Iteration Figure 1: Overall accuracy, accuracy on known tokens, accuracy on unknown tokens, and proportion of known tokens for Italian (left) and Dutch (right). Table 2 shows results for our seed model, self training and revision, and the results reported by Das and Petrov. Self training and revision improve the accuracy for every language over the seed model, and gives an average improvement of roughly two percentage points. The average accuracy of self training and revision is on par with that reported by Das and Petrov. On individual languages, self training and revision and the method of Das and Petrov are split each performs better on half of the cases. Interestingly, our method achieves higher accuracies on Germanic languages the family of our source language, English while Das and Petrov perform better on Romance languages. This might be because our model relies on alignments, which might be more accurate for more-related languages, whereas Das and Petrov additionally rely on label propagation. Compared to Das and Petrov, our model performs poorest on Italian, in terms of percentage point difference in accuracy. Figure 1 (left panel) shows accuracy, accuracy on known words, accuracy on unknown words, and proportion of known tokens for each iteration of our model for Italian; iteration 0 is the seed model, and iteration 3 1 is the final model. Our model performs poorly on unknown words as indicated by the low accuracy on unknown words, and high accuracy on known — — — words compared to the overall accuracy. The poor performance on unknown words is expected because we do not use any language-specific rules to handle this case. Moreover, on average for the final model, approximately 10% of the test data tokens are unknown. One way to improve the performance of our tagger might be to reduce the proportion of unknown words by using a larger training corpus, as Das and Petrov did. We examine the impact of self-training and revision over training iterations. We find that for all languages, accuracy rises quickly in the first 5–6 iterations, and then subsequently improves only slightly. We exemplify this in Figure 1 (right panel) for Dutch. (Findings are similar for other languages.) Although accuracy does not increase much in later iterations, they may still have some benefit as the vocabulary size continues to grow. 6 Conclusion We have proposed a method for unsupervised POS tagging that performs on par with the current state- of-the-art (Das and Petrov, 2011), but is substantially less-sophisticated (specifically not requiring convex optimization or a feature-based HMM). The complexity of our algorithm is O(nlogn) compared to O(n2) for that of Das and Petrov 637 (201 1) where n is the size of training data.3 We made our code are available for download.4 In future work we intend to consider using a larger training corpus to reduce the proportion of unknown tokens and improve accuracy. Given the improvements of our model over that of Das and Petrov on languages from the same family as our source language, and the observation of Snyder et al. (2008) that a better tagger can be learned from a more-closely related language, we also plan to consider strategies for selecting an appropriate source language for a given target language. Using our final model with unsupervised HMM methods might improve the final performance too, i.e. use our final model as the initial state for HMM, then experiment with differ- ent inference algorithms such as Expectation Maximization (EM), Variational Bayers (VB) or Gibbs sampling (GS).5 Gao and Johnson (2008) compare EM, VB and GS for unsupervised English POS tagging. In many cases, GS outperformed other methods, thus we would like to try GS first for our model. 7 Acknowledgements This work is funded by Erasmus Mundus European Masters Program in Language and Communication Technologies (EM-LCT) and by the Czech Science Foundation (grant no. P103/12/G084). We would like to thank Prokopis Prokopidis for providing us the Greek Treebank and Antonia Marti for the Spanish CoNLL 06 dataset. Finally, we thank Siva Reddy and Spandana Gella for many discussions and suggestions. References Thorsten Brants. 2000. TnT: A statistical part-ofspeech tagger. In Proceedings of the sixth conference on Applied natural language processing (ANLP ’00), pages 224–231 . Seattle, Washington, USA. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proceedings of 3We re-implemented label propagation from Das and Petrov (2011). It took over a day to complete this step on an eight core Intel Xeon 3.16GHz CPU with 32 Gb Ram, but only 15 minutes for our model. 4https://code.google.com/p/universal-tagger/ 5We in fact have tried EM, but it did not help. The overall performance dropped slightly. This might be because selftraining with revision already found the local maximal point. the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1 (ACL 2011), pages 600–609. Portland, Oregon, USA. Pascal Denis and Beno ıˆt Sagot. 2009. Coupling an annotated corpus and a morphosyntactic lexicon for state-of-the-art POS tagging with less human effort. In Proceedings of the 23rd PacificAsia Conference on Language, Information and Computation, pages 721–736. Hong Kong, China. Anna Feldman, Jirka Hana, and Chris Brew. 2006. A cross-language approach to rapid creation of new morpho-syntactically annotated resources. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’06), pages 549–554. Genoa, Italy. Jianfeng Gao and Mark Johnson. 2008. A comparison of bayesian estimators for unsupervised hidden markov model pos taggers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 344–352. Association for Computational Linguistics, Stroudsburg, PA, USA. Jiri Hana, Anna Feldman, and Chris Brew. 2004. A resource-light approach to Russian morphology: Tagging Russian using Czech resources. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP ’04), pages 222–229. Barcelona, Spain. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proceedings of the Tenth Machine Translation Summit (MT Summit X), pages 79–86. AAMT, Phuket, Thailand. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the main conference on Human Language Technology Conference ofthe North American Chapter of the Association of Computational Linguistics (HLT-NAACL ’06), pages 152–159. New York, USA. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 2089–2096. Istanbul, Turkey. Siva Reddy and Serge Sharoff. 2011. Cross language POS Taggers (and other tools) for Indian 638 languages: An experiment with Kannada using Telugu resources. In Proceedings of the IJCNLP 2011 workshop on Cross Lingual Information Access: Computational Linguistics and the Information Need of Multilingual Societies (CLIA 2011). Chiang Mai, Thailand. Benjamin Snyder, Tahira Naseem, Jacob Eisenstein, and Regina Barzilay. 2008. Unsupervised multilingual learning for POS tagging. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP ’08), pages 1041–1050. Honolulu, Hawaii. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Featurerich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Vol- ume 1 (NAACL ’03), pages 173–180. Edmonton, Canada. David Yarowsky and Grace Ngai. 2001 . Inducing multilingual POS taggers and NP bracketers via robust projection across aligned corpora. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies (NAACL ’01), pages 1–8. Pittsburgh, Pennsylvania, USA. 639
6 0.15810841 47 acl-2013-An Information Theoretic Approach to Bilingual Word Clustering
7 0.15418148 164 acl-2013-FudanNLP: A Toolkit for Chinese Natural Language Processing
8 0.15127175 345 acl-2013-The Haves and the Have-Nots: Leveraging Unlabelled Corpora for Sentiment Analysis
9 0.13410021 368 acl-2013-Universal Dependency Annotation for Multilingual Parsing
10 0.11728106 210 acl-2013-Joint Word Alignment and Bilingual Named Entity Recognition Using Dual Decomposition
11 0.11584473 80 acl-2013-Chinese Parsing Exploiting Characters
12 0.11420554 34 acl-2013-Accurate Word Segmentation using Transliteration and Language Model Projection
13 0.10868592 11 acl-2013-A Multi-Domain Translation Model Framework for Statistical Machine Translation
14 0.1080342 44 acl-2013-An Empirical Examination of Challenges in Chinese Parsing
15 0.10208998 369 acl-2013-Unsupervised Consonant-Vowel Prediction over Hundreds of Languages
16 0.10185826 193 acl-2013-Improving Chinese Word Segmentation on Micro-blog Using Rich Punctuations
17 0.09851636 128 acl-2013-Does Korean defeat phonotactic word segmentation?
18 0.094436504 134 acl-2013-Embedding Semantic Similarity in Tree Kernels for Domain Adaptation of Relation Extraction
19 0.092347093 256 acl-2013-Named Entity Recognition using Cross-lingual Resources: Arabic as an Example
20 0.092018507 29 acl-2013-A Visual Analytics System for Cluster Exploration
topicId topicWeight
[(0, 0.238), (1, -0.117), (2, -0.109), (3, 0.055), (4, 0.065), (5, -0.108), (6, -0.16), (7, 0.062), (8, 0.035), (9, -0.24), (10, 0.146), (11, -0.339), (12, 0.041), (13, 0.209), (14, -0.181), (15, 0.035), (16, 0.05), (17, -0.02), (18, 0.19), (19, -0.001), (20, 0.044), (21, 0.008), (22, 0.009), (23, -0.052), (24, -0.041), (25, -0.051), (26, 0.238), (27, 0.102), (28, -0.149), (29, -0.018), (30, 0.047), (31, -0.02), (32, 0.111), (33, -0.058), (34, -0.012), (35, -0.023), (36, 0.077), (37, 0.093), (38, -0.066), (39, 0.012), (40, -0.002), (41, 0.018), (42, 0.062), (43, 0.093), (44, -0.033), (45, 0.081), (46, -0.076), (47, -0.015), (48, -0.033), (49, -0.011)]
simIndex simValue paperId paperTitle
same-paper 1 0.95750642 97 acl-2013-Cross-lingual Projections between Languages from Different Families
Author: Mo Yu ; Tiejun Zhao ; Yalong Bai ; Hao Tian ; Dianhai Yu
Abstract: Cross-lingual projection methods can benefit from resource-rich languages to improve performances of NLP tasks in resources-scarce languages. However, these methods confronted the difficulty of syntactic differences between languages especially when the pair of languages varies greatly. To make the projection method well-generalize to diverse languages pairs, we enhance the projection method based on word alignments by introducing target-language word representations as features and proposing a novel noise removing method based on these word representations. Experiments showed that our methods improve the performances greatly on projections between English and Chinese.
2 0.8640393 70 acl-2013-Bilingually-Guided Monolingual Dependency Grammar Induction
Author: Kai Liu ; Yajuan Lu ; Wenbin Jiang ; Qun Liu
Abstract: This paper describes a novel strategy for automatic induction of a monolingual dependency grammar under the guidance of bilingually-projected dependency. By moderately leveraging the dependency information projected from the parsed counterpart language, and simultaneously mining the underlying syntactic structure of the language considered, it effectively integrates the advantages of bilingual projection and unsupervised induction, so as to induce a monolingual grammar much better than previous models only using bilingual projection or unsupervised induction. We induced dependency gram- mar for five different languages under the guidance of dependency information projected from the parsed English translation, experiments show that the bilinguallyguided method achieves a significant improvement of 28.5% over the unsupervised baseline and 3.0% over the best projection baseline on average.
3 0.84507406 136 acl-2013-Enhanced and Portable Dependency Projection Algorithms Using Interlinear Glossed Text
Author: Ryan Georgi ; Fei Xia ; William D. Lewis
Abstract: As most of the world’s languages are under-resourced, projection algorithms offer an enticing way to bootstrap the resources available for one resourcepoor language from a resource-rich language by means of parallel text and word alignment. These algorithms, however, make the strong assumption that the language pairs share common structures and that the parse trees will resemble one another. This assumption is useful but often leads to errors in projection. In this paper, we will address this weakness by using trees created from instances of Interlinear Glossed Text (IGT) to discover patterns of divergence between the lan- guages. We will show that this method improves the performance of projection algorithms significantly in some languages by accounting for divergence between languages using only the partial supervision of a few corrected trees.
4 0.70023936 98 acl-2013-Cross-lingual Transfer of Semantic Role Labeling Models
Author: Mikhail Kozhevnikov ; Ivan Titov
Abstract: Semantic Role Labeling (SRL) has become one of the standard tasks of natural language processing and proven useful as a source of information for a number of other applications. We address the problem of transferring an SRL model from one language to another using a shared feature representation. This approach is then evaluated on three language pairs, demonstrating competitive performance as compared to a state-of-the-art unsupervised SRL system and a cross-lingual annotation projection baseline. We also consider the contribution of different aspects of the feature representation to the performance of the model and discuss practical applicability of this method. 1 Background and Motivation Semantic role labeling has proven useful in many natural language processing tasks, such as question answering (Shen and Lapata, 2007; Kaisser and Webber, 2007), textual entailment (Sammons et al., 2009), machine translation (Wu and Fung, 2009; Liu and Gildea, 2010; Gao and Vogel, 2011) and dialogue systems (Basili et al., 2009; van der Plas et al., 2009). Multiple models have been designed to automatically predict semantic roles, and a considerable amount of data has been annotated to train these models, if only for a few more popular languages. As the annotation is costly, one would like to leverage existing resources to minimize the human effort required to construct a model for a new language. A number of approaches to the construction of semantic role labeling models for new languages have been proposed. On one end of the scale is unsupervised SRL, such as Grenager and Manning (2006), which requires some expert knowledge, but no labeled data. It clusters together arguments that should bear the same semantic role, but does not assign a particular role to each cluster. On the other end is annotating a new dataset from scratch. There are also intermediate options, which often make use of similarities between languages. This way, if an accurate model exists for one language, it should help simplify the construction of a model for another, related language. The approaches in this third group often use parallel data to bridge the gap between languages. Cross-lingual annotation projection systems (Pad o´ and Lapata, 2009), for example, propagate information directly via word alignment links. However, they are very sensitive to the quality of parallel data, as well as the accuracy of a sourcelanguage model on it. An alternative approach, known as cross-lingual model transfer, or cross-lingual model adaptation, consists of modifying a source-language model to make it directly applicable to a new language. This usually involves constructing a shared feature representation across the two languages. McDonald et al. (201 1) successfully apply this idea to the transfer of dependency parsers, using part-of- speech tags as the shared representation of words. A later extension of T ¨ackstr o¨m et al. (2012) enriches this representation with cross-lingual word clusters, considerably improving the performance. In the case of SRL, a shared representation that is purely syntactic is likely to be insufficient, since structures with different semantics may be realized by the same syntactic construct, for example “in August” vs “in Britain”. However with the help of recently introduced cross-lingual word represen1190 Proce dingsS o f ita h,e B 5u1lgsta Arinan,u Aaulg Musete 4ti-n9g 2 o0f1 t3h.e ? Ac s2s0o1ci3a Atiosnso fcoirat Cio nm foprut Caotimonpaulta Lti nognuails Lti cnsg,u piasgteics 1 90–120 , tations, such as the cross-lingual clustering mentioned above or cross-lingual distributed word representations of Klementiev et al. (2012), we may be able to transfer models of shallow semantics in a similar fashion. In this work we construct a shared feature representation for a pair of languages, employing crosslingual representations of syntactic and lexical information, train a semantic role labeling model on one language and apply it to the other one. This approach yields an SRL model for a new language at a very low cost, effectively requiring only a source language model and parallel data. We evaluate on five (directed) language pairs EN-ZH, ZH-EN, EN-CZ, CZ-EN and EN-FR, where EN, FR, CZ and ZH denote English, French, Czech and Chinese, respectively. The transferred model is compared against two baselines: an unsupervised SRL system and a model trained on the output of a cross-lingual annotation projection system. In the next section we will describe our setup, then in section 3 present the shared feature representation we use, discuss the evaluation data and other technical aspects in section 4, present the results and conclude with an overview of related work. – 2 Setup The purpose of the study is not to develop a yet another semantic role labeling system any existing SRL system can (after some modification) be used in this setup but to assess the practical applicability of cross-lingual model transfer to this – – problem, compare it against the alternatives and identify its strong/weak points depending on a particular setup. 2.1 Semantic Role Labeling Model We consider the dependency-based version of semantic role labeling as described in Haji cˇ et al. (2009) and transfer an SRL model from one language to another. We only consider verbal predicates and ignore the predicate disambiguation stage. We also assume that the predicate identification information is available in most languages it can be obtained using a relatively simple heuristic based on part-of-speech tags. The model performs argument identification and classification (Johansson and Nugues, 2008) separately in a pipeline first each candidate is classified as being or not being a head of an argument phrase with respect to the predicate in question and then each of the arguments is assigned a role from a given inventory. The model is factorized over arguments the decisions regarding the classification of different arguments are made in– – – dependently of each other. With respect to the use of syntactic annotation we consider two options: using an existing dependency parser for the target language and obtaining one by means of cross-lingual transfer (see section 4.2). Following McDonald et al. (201 1), we assume that a part-of-speech tagger is available for the target language. 2.2 SRL in the Low-resource Setting Several approaches have been proposed to obtain an SRL model for a new language with little or no manual annotation. Unsupervised SRL models (Lang and Lapata, 2010) cluster the arguments of predicates in a given corpus according to their semantic roles. The performance of such models can be impressive, especially for those languages where semantic roles correlate strongly with syntactic relation of the argument to its predicate. However, assigning meaningful role labels to the resulting clusters requires additional effort and the model’s parameters generally need some adjustment for every language. If the necessary resources are already available for a closely related language, they can be utilized to facilitate the construction of a model for the target language. This can be achieved either by means of cross-lingual annotation projection (Yarowsky et al., 2001) or by cross-lingual model transfer (Zeman and Resnik, 2008). This last approach is the one we are considering in this work, and the other two options are treated as baselines. The unsupervised model will be further referred to as UNSUP and the projection baseline as PROJ. 2.3 Evaluation Measures We use the F1 measure as a metric for the argument identification stage and accuracy as an aggregate measure of argument classification performance. When comparing to the unsupervised SRL system the clustering evaluation measures are used instead. These are purity and collocation 1191 N1Ximajx|Gj∩ Ci| CO =N1Xjmiax|Gj∩ Ci|, PU = where Ci is the set of arguments in the i-th induced cluster, Gj is the set of arguments in the jth gold cluster and N is the total number of arguments. We report the harmonic mean ofthe two (Lang and Lapata, 2011) and denote it F1c to avoid confusing it with the supervised metric. 3 Model Transfer The idea of this work is to abstract the model away from the particular source language and apply it to a new one. This setup requires that we use the same feature representation for both languages, for example part-of-speech tags and dependency relation labels should be from the same inventory. Some features are not applicable to certain lan- guages because the corresponding phenomena are absent in them. For example, consider a strongly inflected language and an analytic one. While the latter can usually convey the information encoded in the word form in the former one (number, gender, etc.), finding a shared feature representation for such information is non-trivial. In this study we will confine ourselves to those features that are applicable to all languages in question, namely: part-of-speech tags, syntactic dependency structures and representations of the word’s identity. 3.1 Lexical Information We train a model on one language and apply it to a different one. In order for this to work, the words of the two languages have to be mapped into a common feature space. It is also desirable that closely related words from both languages have similar representations in this space. Word mapping. The first option is simply to use the source language words as the shared representation. Here every source language word would have itself as its representation and every target word would map into a source word that corresponds to it. In other words, we supply the model with a gloss of the target sentence. The mapping (bilingual dictionary) we use is derived from a word-aligned parallel corpus, by identifying, for each word in the target language, the word in the source language it is most often aligned to. Cross-lingual clusters. There is no guarantee that each of the words in the evaluation data is present in our dictionary, nor that the corresponding source-language word is present in the training data, so the model would benefit from the ability to generalize over closely related words. This can, for example, be achieved by using cross-lingual word clusters induced in T ¨ackstr o¨m et al. (2012). We incorporate these clusters as features into our model. 3.2 Syntactic Information Part-of-speech Tags. We map part-of-speech tags into the universal tagset following Petrov et al. (2012). This may have a negative effect on the performance of a monolingual model, since most part-of-speech tagsets are more fine-grained than the universal POS tags considered here. For example Penn Treebank inventory contains 36 tags and the universal POS tagset only 12. Since the finergrained POS tags often reflect more languagespecific phenomena, however, they would only be useful for very closely related languages in the cross-lingual setting. The universal part-of-speech tags used in evaluation are derived from gold-standard annotation for all languages except French, where predicted ones had to be used instead. Dependency Structure. Another important aspect of syntactic information is the dependency structure. Most dependency relation inventories are language-specific, and finding a shared representation for them is a challenging problem. One could map dependency relations into a simplified form that would be shared between languages, as it is done for part-of-speech tags in Petrov et al. (2012). The extent to which this would be useful, however, depends on the similarity of syntactic-semantic in– terfaces of the languages in question. In this work we discard the dependency relation labels where the inventories do not match and only consider the unlabeled syntactic dependency graph. Some discrepancies, such as variations in attachment order, may be present even there, but this does not appear to be the case with the datasets we use for evaluation. If a target language is poor in resources, one can obtain a dependency parser for the target language by means of cross-lingual model transfer (Zeman and Resnik, 2008). We 1192 take this into account and evaluate both using the original dependency structures and the ones obtained by means of cross-lingual model transfer. 3.3 The Model The model we use is based on that of Bj ¨orkelund et al. (2009). It is comprised of a set of linear classifiers trained using Liblinear (Fan et al., 2008). The feature model was modified to accommodate the cross-lingual cluster features and the reranker component was not used. We do not model the interaction between different argument roles in the same predicate. While this has been found useful, in the cross-lingual setup one has to be careful with the assumptions made. For example, modeling the sequence of roles using a Markov chain (Thompson et al., 2003) may not work well in the present setting, especially between distant languages, as the order or arguments is not necessarily preserved. Most constraints that prove useful for SRL (Chang et al., 2007) also require customization when applied to a new language, and some rely on languagespecific resources, such as a valency lexicon. Taking into account the interaction between different arguments of a predicate is likely to improve the performance of the transferred model, but this is outside the scope of this work. 3.4 Feature Selection Compatibility of feature representations is necessary but not sufficient for successful model transfer. We have to make sure that the features we use are predictive of similar outcomes in the two languages as well. Depending on the pair of languages in question, different aspects of the feature representation will retain or lose their predictive power. We can be reasonably certain that the identity of an argument word is predictive of its semantic role in any language, but it might or might not be true of, for example, the word directly preceding the argument word. It is therefore important to pre- SCPDGylOespoSntreslTabunc1lra:obsFel-daitnguplrdoaeusntpagd-elronwfu-dcsopeyrnsd c.eylafguhtorsia mepgnrhs vent the model from capturing overly specific aspects of the source language, which we do by confining the model to first-order features. We also avoid feature selection, which, performed on the source language, is unlikely to help the model to better generalize to the target one. The experiments confirm that feature selection and the use of second-order features degrade the performance of the transferred model. 3.5 Feature Groups For each word, we use its part-of-speech tag, cross-lingual cluster id, word identity (glossed, when evaluating on the target language) and its dependency relation to its parent. Features associated with an argument word include the attributes of the predicate word, the argument word, its parent, siblings and children, and the words directly preceding and following it. Also included are the sequences of part-of-speech tags and dependency relations on the path between the predicate and the argument. Since we are also interested in the impact of different aspects of the feature representation, we divide the features into groups as summarized in table 1 and evaluate their respective contributions to the performance of the model. If a feature group is enabled the model has access to the corre– sponding source of information. For example, if only POS group is enabled, the model relies on the part-of-speech tags of the argument, the predicate and the words to the right and left of the argument word. If Synt is enabled too, it also uses the POS tags of the argument’s parent, children and siblings. Word order information constitutes an implicit group that is always available. It includes the Pos it ion feature, which indicates whether the argument is located to the left or to the right of the predicate, and allows the model to look up the attributes of the words directly preceding and following the argument word. The model we compare against the baselines uses all applicable feature groups (Deprel is only used in EN-CZ and CZ-EN experiments with original syntax). 4 Evaluation 4.1 Datasets and Preprocessing Evaluation of the cross-lingual model transfer requires a rather specific kind of dataset. Namely, the data in both languages has to be annotated 1193 with the same set of semantic roles following the same (or compatible) guidelines, which is seldom the case. We have identified three language pairs for which such resources are available: EnglishChinese, English-Czech and English-French. The evaluation datasets for English and Chinese are those from the CoNLL Shared Task 2009 (Haji ˇc et al., 2009) (henceforth CoNLL-ST). Their annotation in the CoNLL-ST is not identical, but the guidelines for “core” semantic roles are similar (Kingsbury et al., 2004), so we evaluate only on core roles here. The data for the second language pair is drawn from the Prague Czech-English Dependency Treebank 2.0 (Haji ˇc et al., 2012), which we converted to a format similar to that of CoNLL-ST1 . The original annotation uses the tectogrammatical representation (Haji ˇc, 2002) and an inventory of semantic roles (or functors), most of which are interpretable across various predicates. Also note that the syntactic anno- tation of English and Czech in PCEDT 2.0 is quite similar (to the extent permitted by the difference in the structure of the two languages) and we can use the dependency relations in our experiments. For English-French, the English CoNLL-ST dataset was used as a source and the model was evaluated on the manually annotated dataset from van der Plas et al. (201 1). The latter contains one thousand sentences from the French part ofthe Europarl (Koehn, 2005) corpus, annotated with semantic roles following an adapted version of PropBank (Palmer et al., 2005) guidelines. The authors perform annotation projection from English to French, using a joint model of syntax and semantics and employing heuristics for filtering. We use a model trained on the output of this projection system as one of the baselines. The evaluation dataset is relatively small in this case, so we perform the transfer only one-way, from English to French. The part-of-speech tags in all datasets were replaced with the universal POS tags of Petrov et al. (2012). For Czech, we have augmented the map- pings to account for the tags that were not present in the datasets from which the original mappings were derived. Namely, tag “t” is mapped to “VERB” and “Y” to “PRON”. We use parallel data to construct a bilingual dictionary used in word mapping, as well as in the projection baseline. For English-Czech – 1see http://www.ml4nlp.de/code-and-data/treex2conll and English-French, the data is drawn from Europarl (Koehn, 2005), for English-Chinese from MultiUN (Eisele and Chen, 2010). The word alignments were obtained using GIZA++ (Och and Ney, 2003) and the intersection heuristic. – 4.2 Syntactic Transfer In the low-resource setting, we cannot always rely on the availability of an accurate dependency parser for the target language. If one is not available, the natural solution would be to use crosslingual model transfer to obtain it. Unfortunately, the models presented in the previous work, such as Zeman and Resnik (2008), McDonald et al. (201 1) and T ¨ackstr o¨m et al. (2012), were not made available, so we reproduced the direct transfer algorithm of McDonald et al. (201 1), using Malt parser (Nivre, 2008) and the same set of features. We did not reimplement the projected transfer algorithm, however, and used the default training procedure instead of perceptron-based learning. The dependency structure thus obtained is, of course, only a rough approximation even a much more sophisticated algorithm may not perform well when transferring syntax between such languages as Czech and English, given the inherent difference in their structure. The scores are shown in table 2. We will henceforth refer to the syntactic annotations that were provided with the datasets as original, as opposed to the annotations obtained by means of syntactic transfer. – 4.3 Baselines Unsupervised Baseline: We are using a version of the unsupervised semantic role induction system of Titov and Klementiev (2012a) adapted to SetupUAS, % Table2:SyntaciE C ZcN HNt- rE ZaCFnN HZRsfer34 692567acuracy,unlabe dat- tachment score (percent). Note that in case of French we evaluate against the output of a supervised system, since manual annotation is not available for this dataset. This score does not reflect the true performance of syntactic transfer. 1194 the shared feature representation considered in order to make the scores comparable with those of the transfer model and, more importantly, to enable evaluation on transferred syntax. Note that the original system, tailored to a more expressive language-specific syntactic representation and equipped with heuristics to identify active/passive voice and other phenomena, achieves higher scores than those we report here. Projection Baseline: The projection baseline we use for English-Czech and English-Chinese is a straightforward one: we label the source side of a parallel corpus using the source-language model, then identify those verbs on the target side that are aligned to a predicate, mark them as predicates and propagate the argument roles in the same fashion. A model is then trained on the resulting training data and applied to the test set. For English-French we instead use the output of a fully featured projection model of van der Plas et al. (201 1), published in the CLASSiC project. 5 Results In order to ensure that the results are consistent, the test sets, except for the French one, were partitioned into five equal parts (of 5 to 10 thousand sentences each, depending on the dataset) and the evaluation performed separately on each one. All evaluation figures for English, Czech or Chinese below are the average values over the five subsets. In case of French, the evaluation dataset is too small to split it further, so instead we ran the evaluation five times on a randomly selected 80% sample of the evaluation data and averaged over those. In both cases the results are consistent over the subsets, the standard deviation does not exceed 0.5% for the transfer system and projection baseline and 1% for the unsupervised system. 5.1 Argument Identification We summarize the results in table 3. Argument identification is known to rely heavily on syntactic information, so it is unsurprising that it proves inaccurate when transferred syntax is used. Our simple projection baseline suffers from the same problem. Even with original syntactic information available, the performance of argument identification is moderate. Note that the model of (van der Plas et al., 2011), though relying on more expressive syntax, only outperforms the transferred system by 3% (F1) on this task. SetupSyntaxTRANSPROJ ZEC NH Z- EFCZNRHt r a n s 3462 1. 536 142 35. 4269 Table3EZ C:N H- CFEZANHZRrgumeon rt ig identf56 7ic13 a. t27903ion,21569t10ra. 3976nsferd model vs. projection baseline, F1. Most unsupervised SRL approaches assume that the argument identification is performed by some external means, for example heuristically (Lang and Lapata, 2011). Such heuristics or unsupervised approaches to argument identification (Abend et al., 2009) can also be used in the present setup. 5.2 Argument Classification In the following tables, TRANS column contains the results for the transferred system, UNSUP for the unsupervised baseline and PROJ for projection baseline. We highlight in bold the higher score where the difference exceeds twice the maximum of the standard deviation estimates of the two results. Table 4 presents the unsupervised evaluation results. Note that the unsupervised model performs as well as the transferred one or better where the – – SetupSyntaxTRANSUNSUP ZEC NH Z- EFCZNRHt r a n s 768 93648. 34627 6 5873. 1769 TableEZ C4NHZ:- FCEZANHZRrgumoe nr itg clasi78 fi94 3c. a25136tion,8 7 r9a4263n. 07 sferd model vs. unsupervised baseline in terms of the clustering metric F1c (see section 2.3). 1195 SetupSyntaxTRANSPROJ ZEC NH Z- EFCZNRHt r a n s 657 053. 1 36456419. 372 Table5EZ C:N H- CFEZANHZRrgumeon rt ig clasif657ic1936a. t170 ion,65 9t3804ra. 20847nsferd model vs. projection baseline, accuracy. original syntactic dependencies are available. In the more realistic scenario with transferred syn- tax, however, the transferred model proves more accurate. In table 5 we compare the transferred system with the projection baseline. It is easy to see that the scores vary strongly depending on the language pair, due to both the difference in the annotation scheme used and the degree of relatedness between the languages. The drop in performance when transferring the model to another language is large in every case, though, see table 6. SetupTargetSource Table6:MoCEZdHeNZ l- FECaZNRcH urac67 y53169o. 017nthes87 o25670u. r1245ceandtrge language using original syntax. The source language scores for English vary between language pairs because of the difference in syntactic annotation and role subset used. We also include the individual F1 scores for the top-10 most frequent labels for EN-CZ transfer with original syntax in table 7. The model provides meaningful predictions here, despite low overall accuracy. Most of the labels2 are self-explanatory: Patient (PAT), Actor (ACT), Time (TWHEN), Effect (EFF), Location (LOC), Manner (MANN), Addressee (ADDR), Extent (EXT). CPHR marks the 2http://ufal.mff.cuni.cz/∼toman/pcedt/en/functors.html LabelFreq.F1Re.Pr. recall and precision for the top-10 most frequent roles. nominal part of a complex predicate, as in “to have [a plan]CPHR”, and DIR3 indicates destination. 5.3 Additional Experiments We now evaluate the contribution of different aspects of the feature representation to the performance of the model. Table 8 contains the results for English-French. FeaturesOrigTrans ferent feature subsets, using original and transferred syntactic information. The fact that the model performs slightly better with transferred syntax may be explained by two factors. Firstly, as we already mentioned, the original syntactic annotation is also produced automatically. Secondly, in the model transfer setup it is more important how closely the syntacticsemantic interface on the target side resembles that on the source side than how well it matches the “true” structure of the target language, and in this respect a transferred dependency parser may have an advantage over one trained on target-language data. The high impact of the Glos s features here 1196 may be partly attributed to the fact that the mapping is derived from the same corpus as the evaluation data Europarl (Koehn, 2005) and partly by the similarity between English and French in terms of word order, usage of articles and prepositions. The moderate contribution of the crosslingual cluster features are likely due to the insufficient granularity of the clustering for this task. For more distant language pairs, the contributions of individual feature groups are less interpretable, so we only highlight a few observations. First of all, both EN-CZ and CZ-EN benefit noticeably from the use of the original syntactic annotation, including dependency relations, but not from the transferred syntax, most likely due to the low syntactic transfer performance. Both perform better when lexical information is available, although – – the improvement is not as significant as in the case of French only up to 5%. The situation with Chinese is somewhat complicated in that adding lexical information here fails to yield an improvement in terms of the metric considered. This is likely due to the fact that we consider only the core roles, which can usually be predicted with high accuracy based on syntactic information alone. – 6 Related Work Development of robust statistical models for core NLP tasks is a challenging problem, and adaptation of existing models to new languages presents a viable alternative to exhaustive annotation for each language. Although the models thus obtained are generally imperfect, they can be further refined for a particular language and domain using techniques such as active learning (Settles, 2010; Chen et al., 2011). Cross-lingual annotation projection (Yarowsky et al., 2001) approaches have been applied ex- tensively to a variety of tasks, including POS tagging (Xi and Hwa, 2005; Das and Petrov, 2011), morphology segmentation (Snyder and Barzilay, 2008), verb classification (Merlo et al., 2002), mention detection (Zitouni and Florian, 2008), LFG parsing (Wr o´blewska and Frank, 2009), information extraction (Kim et al., 2010), SRL (Pad o´ and Lapata, 2009; van der Plas et al., 2011; Annesi and Basili, 2010; Tonelli and Pianta, 2008), dependency parsing (Naseem et al., 2012; Ganchev et al., 2009; Smith and Eisner, 2009; Hwa et al., 2005) or temporal relation prediction (Spreyer and Frank, 2008). Interestingly, it has also been used to propagate morphosyntactic information between old and modern versions of the same language (Meyer, 2011). Cross-lingual model transfer methods (McDonald et al., 2011; Zeman and Resnik, 2008; Durrett et al., 2012; Søgaard, 2011; Lopez et al., 2008) have also been receiving much attention recently. The basic idea behind model transfer is similar to that of cross-lingual annotation projection, as we can see from the way parallel data is used in, for example, McDonald et al. (201 1). A crucial component of direct transfer approaches is the unified feature representation. There are at least two such representations of lexical information (Klementiev et al., 2012; T ¨ackstr o¨m et al., 2012), but both work on word level. This makes it hard to account for phenomena that are expressed differently in the languages considered, for example the syntactic function of a certain word may be indicated by a preposition, inflection or word order, depending on the language. Accurate representation of such information would require an extra level of abstraction (Haji ˇc, 2002). A side-effect ofusing adaptation methods is that we are forced to use the same annotation scheme for the task in question (SRL, in our case), which in turn simplifies the development of cross-lingual tools for downstream tasks. Such representations are also likely to be useful in machine translation. Unsupervised semantic role labeling methods (Lang and Lapata, 2010; Lang and Lapata, 2011; Titov and Klementiev, 2012a; Lorenzo and Cerisara, 2012) also constitute an alternative to cross-lingual model transfer. For an overview of of semi-supervised approaches we refer the reader to Titov and Klementiev (2012b). 7 Conclusion We have considered the cross-lingual model transfer approach as applied to the task of semantic role labeling and observed that for closely related languages it performs comparably to annotation projection approaches. It allows one to quickly construct an SRL model for a new language without manual annotation or language-specific heuristics, provided an accurate model is available for one of the related languages along with a certain amount of parallel data for the two languages. While an1197 notation projection approaches require sentenceand word-aligned parallel data and crucially depend on the accuracy of the syntactic parsing and SRL on the source side of the parallel corpus, cross-lingual model transfer can be performed using only a bilingual dictionary. Unsupervised SRL approaches have their advantages, in particular when no annotated data is available for any of the related languages and there is a syntactic parser available for the target one, but the annotation they produce is not always sufficient. In applications such as Information Retrieval it is preferable to have precise labels, rather than just clusters of arguments, for example. Also note that when applying cross-lingual model transfer in practice, one can improve upon the performance of the simplistic model we use for evaluation, for example by picking the features manually, taking into account the properties of the target language. Domain adaptation techniques can also be employed to adjust the model to the target language. Acknowledgments The authors would like to thank Alexandre Klementiev and Ryan McDonald for useful suggestions and T ¨ackstr o¨m et al. (2012) for sharing the cross-lingual word representations. This research is supported by the MMCI Cluster of Excellence. References Omri Abend, Roi Reichart, and Ari Rappoport. 2009. Unsupervised argument identification for semantic role labeling. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, ACL ’09, pages 28–36, Stroudsburg, PA, USA. Association for Computational Linguistics. Paolo Annesi and Roberto Basili. 2010. Cross-lingual alignment of FrameNet annotations through hidden Markov models. In Proceedings of the 11th international conference on Computational Linguistics and Intelligent Text Processing, CICLing’ 10, pages 12– 25, Berlin, Heidelberg. Springer-Verlag. Roberto Basili, Diego De Cao, Danilo Croce, Bonaventura Coppola, and Alessandro Moschitti. 2009. Cross-language frame semantics transfer in bilingual corpora. In Alexander F. Gelbukh, editor, Proceedings of the 10th International Conference on Computational Linguistics and Intelligent Text Pro- cessing, pages 332–345. Anders Bj ¨orkelund, Love Hafdell, and Pierre Nugues. 2009. Multilingual semantic role labeling. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL 2009): Shared Task, pages 43–48, Boulder, Colorado, June. Association for Computational Linguistics. Ming-Wei Chang, Lev Ratinov, and Dan Roth. 2007. Guiding semi-supervision with constraint-driven learning. In ACL. Chenhua Chen, Alexis Palmer, and Caroline Sporleder. 2011. Enhancing active learning for semantic role labeling via compressed dependency trees. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 183–191, Chiang Mai, Thailand, November. Asian Federation of Natural Language Processing. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. Proceedings of the Association for Computational Linguistics. Greg Durrett, Adam Pauls, and Dan Klein. 2012. Syntactic transfer using a bilingual lexicon. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1–1 1, Jeju Island, Korea, July. Association for Computational Linguistics. Andreas Eisele and Yu Chen. 2010. MultiUN: A multilingual corpus from United Nation documents. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA). Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, XiangRui Wang, and Chih-Jen Lin. 2008. LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research, 9: 1871–1874. Kuzman Ganchev, Jennifer Gillenwater, and Ben Taskar. 2009. Dependency grammar induction via bitext projection constraints. In Proceedings of the 47th Annual Meeting of the ACL, pages 369–377, Stroudsburg, PA, USA. Association for Computational Linguistics. Qin Gao and Stephan Vogel. 2011. Corpus expansion for statistical machine translation with semantic role label substitution rules. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 294–298, Portland, Oregon, USA. Trond Grenager and Christopher D. Manning. 2006. Unsupervised discovery of a statistical verb lexicon. In Proceedings of EMNLP. Jan Haji cˇ. 2002. Tectogrammatical representation: Towards a minimal transfer in machine translation. In Robert Frank, editor, Proceedings of the 6th International Workshop on Tree Adjoining Grammars 1198 and Related Frameworks (TAG+6), pages 216— 226, Venezia. Universita di Venezia. Jan Haji cˇ, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Ant o`nia Mart ı´, Llu ı´s M `arquez, Adam Meyers, Joakim Nivre, Sebastian Pad o´, Jan Sˇt eˇp a´nek, Pavel Stra nˇ a´k, Mihai Surdeanu, Nianwen Xue, and Yi Zhang. 2009. The CoNLL2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL 2009): Shared Task, pages 1–18, Boulder, Colorado. Jan Haji cˇ, Eva Haji cˇov a´, Jarmila Panevov a´, Petr Sgall, Ond ˇrej Bojar, Silvie Cinkov´ a, Eva Fuˇ c ´ıkov a´, Marie Mikulov a´, Petr Pajas, Jan Popelka, Ji ˇr´ ı Semeck´ y, Jana Sˇindlerov a´, Jan Sˇt eˇp a´nek, Josef Toman, Zde nˇka Ure sˇov a´, and Zden eˇk Zˇabokrtsk y´. 2012. Announcing Prague Czech-English dependency treebank 2.0. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Mehmet U gˇur Doˇ gan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey, May. European Language Resources Association (ELRA). Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel text. Natural Language Engineering, 11(3):3 11–325. Richard Johansson and Pierre Nugues. 2008. Dependency-based semantic role labeling of PropBank. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 69–78, Honolulu, Hawaii. Michael Kaisser and Bonnie Webber. 2007. Question answering based on semantic roles. In ACL Workshop on Deep Linguistic Processing. Seokhwan Kim, Minwoo Jeong, Jonghoon Lee, and Gary Geunbae Lee. 2010. A cross-lingual annotation projection approach for relation detection. In Proceedings of the 23rd International Conference on Computational Linguistics, COLING ’ 10, pages 564–571, Stroudsburg, PA, USA. Association for Computational Linguistics. Paul Kingsbury, Nianwen Xue, and Martha Palmer. 2004. Propbanking in parallel. In In Proceedings of the Workshop on the Amazing Utility of Parallel and Comparable Corpora, in conjunction with LREC’04. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of the International Conference on Computational Linguistics (COLING), Bombay, India. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Conference Proceedings: the tenth Machine Translation Summit, pages 79–86, Phuket, Thailand. AAMT. Joel Lang and Mirella Lapata. 2010. Unsupervised induction of semantic roles. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 939–947, Los Angeles, California, June. Association for Computational Linguistics. Joel Lang and Mirella Lapata. 2011. Unsupervised semantic role induction via split-merge clustering. In Proc. of Annual Meeting of the Association for Computational Linguistics (ACL). Ding Liu and Daniel Gildea. 2010. Semantic role features for machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), Beijing, China. Adam Lopez, Daniel Zeman, Michael Nossal, Philip Resnik, and Rebecca Hwa. 2008. Cross-language parser adaptation between related languages. In IJCNLP-08 Workshop on NLP for Less Privileged Languages, pages 35–42, Hyderabad, India, January. Alejandra Lorenzo and Christophe Cerisara. 2012. Unsupervised frame based semantic role induction: application to French and English. In Proceedings of the ACL 2012 Joint Workshop on Statistical Parsing and Semantic Processing of Morphologically Rich Languages, pages 30–35, Jeju, Republic of Korea, July. Association for Computational Linguistics. Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’ 11, pages 62–72, Stroudsburg, PA, USA. Association for Computational Linguistics. Paola Merlo, Suzanne Stevenson, Vivian Tsang, and Gianluca Allaria. 2002. A multi-lingual paradigm for automatic verb classification. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL’02), pages 207– 214, Philadelphia, PA. Roland Meyer. 2011. New wine in old wineskins?– Tagging old Russian via annotation projection from modern translations. Russian Linguistics, 35(2):267(15). Tahira Naseem, Regina Barzilay, and Amir Globerson. 2012. Selective sharing for multilingual dependency parsing. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, pages 629–637, Jeju Island, Korea, July. Association for Computational Linguistics. Joakim Nivre. 2008. Algorithms for deterministic incremental dependency parsing. Comput. Linguist., 34(4):513–553, December. 1199 Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1). Sebastian Pad o´ and Mirella Lapata. 2009. Crosslingual annotation projection for semantic roles. Journal of Artificial Intelligence Research, 36:307– 340. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31:71–105. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proceedings of LREC, May. Mark Sammons, Vinod Vydiswaran, Tim Vieira, Nikhil Johri, Ming wei Chang, Dan Goldwasser, Vivek Srikumar, Gourab Kundu, Yuancheng Tu, Kevin Small, Joshua Rule, Quang Do, and Dan Roth. 2009. Relation alignment for textual entailment recognition. In Text Analysis Conference (TAC). Burr Settles. 2010. Active learning literature survey. Computer Sciences Technical Report, 1648. Dan Shen and Mirella Lapata. 2007. Using semantic roles to improve question answering. In EMNLP. David A Smith and Jason Eisner. 2009. Parser adaptation and projection with quasi-synchronous grammar features. In Proceedings of the 2009 Confer- ence on Empirical Methods in Natural Language Processing, pages 822–831. Association for Computational Linguistics. Benjamin Snyder and Regina Barzilay. 2008. Crosslingual propagation for morphological analysis. In Proceedings of the 23rd national conference on Artificial intelligence. Anders Søgaard. 2011. Data point selection for crosslanguage adaptation of dependency parsers. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, volume 2 of HLT ’11, pages 682–686, Stroudsburg, PA, USA. Association for Computational Linguistics. Kathrin Spreyer and Anette Frank. 2008. Projectionbased acquisition of a temporal labeller. Proceedings of IJCNLP 2008. Oscar T¨ ackstr o¨m, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proc. of the Annual Meeting of the North American Association of Computational Linguistics (NAACL), pages 477– 487, Montr ´eal, Canada. Cynthia A. Thompson, Roger Levy, and Christopher D. Manning. 2003. A generative model for seman- tic role labeling. In Proceedings of the 14th European Conference on Machine Learning, ECML 2003, pages 397–408, Dubrovnik, Croatia. Ivan Titov and Alexandre Klementiev. 2012a. A Bayesian approach to unsupervised semantic role induction. In Proc. of European Chapter of the Association for Computational Linguistics (EACL). Ivan Titov and Alexandre Klementiev. 2012b. Semisupervised semantic role labeling: Approaching from an unsupervised perspective. In Proceedings of the International Conference on Computational Linguistics (COLING), Bombay, India, December. Sara Tonelli and Emanuele Pianta. 2008. Frame information transfer from English to Italian. In Proceedings of LREC 2008. Lonneke van der Plas, James Henderson, and Paola Merlo. 2009. Domain adaptation with artificial data for semantic parsing of speech. In Proc. 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 125–128, Boulder, Colorado. Lonneke van der Plas, Paola Merlo, and James Henderson. 2011. Scaling up automatic cross-lingual semantic role annotation. In Proceedings of the 49th Annual Meeting of the Association for Computa- tional Linguistics: Human Language Technologies, HLT ’ 11, pages 299–304, Stroudsburg, PA, USA. Association for Computational Linguistics. Alina Wr o´blewska and Anette Frank. 2009. Crosslingual projection of LFG F-structures: Building an F-structure bank for Polish. In Eighth International Workshop on Treebanks and Linguistic Theories, page 209. Dekai Wu and Pascale Fung. 2009. Can semantic role labeling improve SMT? In Proceedings of 13th Annual Conference of the European Association for Machine Translation (EAMT 2009), Barcelona. Chenhai Xi and Rebecca Hwa. 2005. A backoff model for bootstrapping resources for non-English languages. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 85 1–858, Stroudsburg, PA, USA. David Yarowsky, Grace Ngai, and Ricahrd Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of Human Language Technology Conference. Daniel Zeman and Philip Resnik. 2008. Crosslanguage parser adaptation between related lan- guages. In Proceedings of the IJCNLP-08 Workshop on NLP for Less Privileged Languages, pages 35– 42, Hyderabad, India, January. Asian Federation of Natural Language Processing. Imed Zitouni and Radu Florian. 2008. Mention detection crossing the language barrier. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1200
5 0.63507336 323 acl-2013-Simpler unsupervised POS tagging with bilingual projections
Author: Long Duong ; Paul Cook ; Steven Bird ; Pavel Pecina
Abstract: We present an unsupervised approach to part-of-speech tagging based on projections of tags in a word-aligned bilingual parallel corpus. In contrast to the existing state-of-the-art approach of Das and Petrov, we have developed a substantially simpler method by automatically identifying “good” training sentences from the parallel corpus and applying self-training. In experimental results on eight languages, our method achieves state-of-the-art results. 1 Unsupervised part-of-speech tagging Currently, part-of-speech (POS) taggers are available for many highly spoken and well-resourced languages such as English, French, German, Italian, and Arabic. For example, Petrov et al. (2012) build supervised POS taggers for 22 languages using the TNT tagger (Brants, 2000), with an average accuracy of 95.2%. However, many widelyspoken languages including Bengali, Javanese, and Lahnda have little data manually labelled for POS, limiting supervised approaches to POS tagging for these languages. However, with the growing quantity of text available online, and in particular, multilingual parallel texts from sources such as multilingual websites, government documents and large archives ofhuman translations ofbooks, news, and so forth, unannotated parallel data is becoming more widely available. This parallel data can be exploited to bridge languages, and in particular, transfer information from a highly-resourced language to a lesser-resourced language, to build unsupervised POS taggers. In this paper, we propose an unsupervised approach to POS tagging in a similar vein to the work of Das and Petrov (201 1). In this approach, — — pecina@ ufal .mff .cuni . c z a parallel corpus for a more-resourced language having a POS tagger, and a lesser-resourced language, is word-aligned. These alignments are exploited to infer an unsupervised tagger for the target language (i.e., a tagger not requiring manuallylabelled data in the target language). Our approach is substantially simpler than that of Das and Petrov, the current state-of-the art, yet performs comparably well. 2 Related work There is a wealth of prior research on building unsupervised POS taggers. Some approaches have exploited similarities between typologically similar languages (e.g., Czech and Russian, or Telugu and Kannada) to estimate the transition probabilities for an HMM tagger for one language based on a corpus for another language (e.g., Hana et al., 2004; Feldman et al., 2006; Reddy and Sharoff, 2011). Other approaches have simultaneously tagged two languages based on alignments in a parallel corpus (e.g., Snyder et al., 2008). A number of studies have used tag projection to copy tag information from a resource-rich to a resource-poor language, based on word alignments in a parallel corpus. After alignment, the resource-rich language is tagged, and tags are projected from the source language to the target language based on the alignment (e.g., Yarowsky and Ngai, 2001 ; Das and Petrov, 2011). Das and Petrov (201 1) achieved the current state-of-the-art for unsupervised tagging by exploiting high confidence alignments to copy tags from the source language to the target language. Graph-based label propagation was used to automatically produce more labelled training data. First, a graph was constructed in which each vertex corresponds to a unique trigram, and edge weights represent the syntactic similarity between vertices. Labels were then propagated by optimizing a convex function to favor the same tags for closely related nodes 634 Proce dingSsof oifa, th Beu 5l1gsarti Aan,An u aglu Mste 4e-ti9n2g 0 o1f3 t.he ?c A2s0s1o3ci Aatsiosonc fioartio Cno fmorpu Ctoamtiopnuatalt Lioin gauli Lsitnicgsu,i psatgices 634–639, ModelCoverageAccuracy Many-to-1 alignments88%68% 1-to-1 alignments 68% 78% 1-to-1 alignments: Top 60k sents 91% 80% Table 1: Token coverage and accuracy of manyto-one and 1-to-1 alignments, as well as the top 60k sentences based on alignment score for 1-to-1 alignments, using directly-projected labels only. while keeping a uniform tag distribution for unrelated nodes. A tag dictionary was then extracted from the automatically labelled data, and this was used to constrain a feature-based HMM tagger. The method we propose here is simpler to that of Das and Petrov in that it does not require convex optimization for label propagation or a feature based HMM, yet it achieves comparable results. 3 Tagset Our tagger exploits the idea ofprojecting tag information from a resource-rich to resource-poor language. To facilitate this mapping, we adopt Petrov et al.’s (2012) twelve universal tags: NOUN, VERB, ADJ, ADV, PRON (pronouns), DET (de- terminers and articles), ADP (prepositions and postpositions), NUM (numerals), CONJ (conjunctions), PRT (particles), “.” (punctuation), and X (all other categories, e.g., foreign words, abbreviations). These twelve basic tags are common across taggers for most languages. Adopting a universal tagset avoids the need to map between a variety of different, languagespecific tagsets. Furthermore, it makes it possible to apply unsupervised tagging methods to languages for which no tagset is available, such as Telugu and Vietnamese. 4 A Simpler Unsupervised POS Tagger Here we describe our proposed tagger. The key idea is to maximize the amount of information gleaned from the source language, while limiting the amount of noise. We describe the seed model and then explain how it is successively refined through self-training and revision. 4.1 Seed Model The first step is to construct a seed tagger from directly-projected labels. Given a parallel corpus for a source and target language, Algorithm 1provides a method for building an unsupervised tagger for the target language. In typical applications, the source language would be a better-resourced language having a tagger, while the target language would be lesser-resourced, lacking a tagger and large amounts of manually POS-labelled data. Algorithm 1 Build seed model Algorithm 1Build seed model 1:Tag source side. 2: Word align the corpus with Giza++ and remove the many-to-one mappings. 3: Project tags from source to target using the remaining 1-to-1 alignments. 4: Select the top n sentences based on sentence alignment score. 5: Estimate emission and transition probabilities. 6: Build seed tagger T. We eliminate many-to-one alignments (Step 2). Keeping these would give more POS-tagged tokens for the target side, but also introduce noise. For example, suppose English and French were the source and target language, respectively. In this case alignments such as English laws (NNS) to French les (DT) lois (NNS) would be expected (Yarowsky and Ngai, 2001). However, in Step 3, where tags are projected from the source to target language, this would incorrectly tag French les as NN. We build a French tagger based on English– French data from the Europarl Corpus (Koehn, 2005). We also compare the accuracy and coverage of the tags obtained through direct projection using the French Melt POS tagger (Denis and Sagot, 2009). Table 1confirms that the one-to-one alignments indeed give higher accuracy but lower coverage than the many-to-one alignments. At this stage of the model we hypothesize that highconfidence tags are important, and hence eliminate the many-to-one alignments. In Step 4, in an effort to again obtain higher quality target language tags from direct projection, we eliminate all but the top n sentences based on their alignment scores, as provided by the aligner via IBM model 3. We heuristically set this cutoff × to 60k to balance the accuracy and size of the seed model.1 Returning to our preliminary English– French experiments in Table 1, this process gives improvements in both accuracy and coverage.2 1We considered values in the range 60–90k, but this choice had little impact on the accuracy of the model. 2We also considered using all projected labels for the top 60k sentences, not just 1-to-1 alignments, but in preliminary experiments this did not perform as well, possibly due to the previously-observed problems with many-to-one alignments. 635 The number of parameters for the emission probability is |V | |T| where V is the vocabulary and aTb iilsi ttyh eis tag |s e×t. TTh| ew htrearnesi Vtio ins probability, on atnhed other hand, has only |T|3 parameters for the trigram hmaondde,l we use. TB|ecause of this difference in number of parameters, in step 5, we use different strategies to estimate the emission and transition probabilities. The emission probability is estimated from all 60k selected sentences. However, for the transition probability, which has less parameters, we again focus on “better” sentences, by estimating this probability from only those sen- tences that have (1) token coverage > 90% (based on direct projection of tags from the source language), and (2) length > 4 tokens. These criteria aim to identify longer, mostly-tagged sentences, which we hypothesize are particularly useful as training data. In the case of our preliminary English–French experiments, roughly 62% of the 60k selected sentences meet these criteria and are used to estimate the transition probability. For unaligned words, we simply assign a random POS and very low probability, which does not substantially affect transition probability estimates. In Step 6 we build a tagger by feeding the estimated emission and transition probabilities into the TNT tagger (Brants, 2000), an implementation of a trigram HMM tagger. 4.2 Self training and revision For self training and revision, we use the seed model, along with the large number of target language sentences available that have been partially tagged through direct projection, in order to build a more accurate tagger. Algorithm 2 describes this process of self training and revision, and assumes that the parallel source–target corpus has been word aligned, with many-to-one alignments removed, and that the sentences are sorted by alignment score. In contrast to Algorithm 1, all sentences are used, not just the 60k sentences with the highest alignment scores. We believe that sentence alignment score might correspond to difficulty to tag. By sorting the sentences by alignment score, sentences which are more difficult to tag are tagged using a more mature model. Following Algorithm 1, we divide sentences into blocks of 60k. In step 3 the tagged block is revised by comparing the tags from the tagger with those obtained through direct projection. Suppose source Algorithm 2 Self training and revision 1:Divide target language sentences into blocks of n sentences. 2: Tag the first block with the seed tagger. 3: Revise the tagged block. 4: Train a new tagger on the tagged block. 5: Add the previous tagger’s lexicon to the new tagger. 6: Use the new tagger to tag the next block. 7: Goto 3 and repeat until all blocks are tagged. language word wis is aligned with target language word wjt with probability p(wjt |wsi), Tis is the tag for wis using the tagger availa|bwle for the source language, and Tjt is the tag for wjt using the tagger learned for the > S, where S is a threshold which we heuristically set to 0.7, we replace Tjt by Tis. Self-training can suffer from over-fitting, in which errors in the original model are repeated and amplified in the new model (McClosky et al., 2006). To avoid this, we remove the tag of any token that the model is uncertain of, i.e., if p(wjt |wsi) < S and Tjt Tis then Tjt = Null. So, on th|ew target side, aligned words have a tag from direct projection or no tag, and unaligned words have a tag assigned by our model. Step 4 estimates the emission and transition target language. If p(wtj|wis) = probabilities as in Algorithm 1. In Step 5, emission probabilities for lexical items in the previous model, but missing from the current model, are added to the current model. Later models therefore take advantage of information from earlier models, and have wider coverage. 5 Experimental Results Using parallel data from Europarl (Koehn, 2005) we apply our method to build taggers for the same eight target languages as Das and Petrov (201 1) Danish, Dutch, German, Greek, Italian, Portuguese, Spanish and Swedish with English as the source language. Our training data (Europarl) is a subset of the training data of Das and Petrov (who also used the ODS United Nations dataset which we were unable to obtain). The evaluation metric and test data are the same as that used by Das and Petrov. Our results are comparable to theirs, although our system is penalized by having less training data. We tag the source language with the Stanford POS tagger (Toutanova et al., 2003). — — 636 DanishDutchGermanGreekItalianPortugueseSpanishSwedishAverage Seed model83.781.183.677.878.684.981.478.981.3 Self training + revision 85.6 84.0 85.4 80.4 81.4 86.3 83.3 81.0 83.4 Das and Petrov (2011) 83.2 79.5 82.8 82.5 86.8 87.9 84.2 80.5 83.4 Table 2: Token-level POS tagging accuracy for our seed model, self training and revision, and the method of Das and Petrov (201 1). The best results on each language, and on average, are shown in bold. 1 1 Iteration 2 2 3 1 1 2 2 3 Iteration Figure 1: Overall accuracy, accuracy on known tokens, accuracy on unknown tokens, and proportion of known tokens for Italian (left) and Dutch (right). Table 2 shows results for our seed model, self training and revision, and the results reported by Das and Petrov. Self training and revision improve the accuracy for every language over the seed model, and gives an average improvement of roughly two percentage points. The average accuracy of self training and revision is on par with that reported by Das and Petrov. On individual languages, self training and revision and the method of Das and Petrov are split each performs better on half of the cases. Interestingly, our method achieves higher accuracies on Germanic languages the family of our source language, English while Das and Petrov perform better on Romance languages. This might be because our model relies on alignments, which might be more accurate for more-related languages, whereas Das and Petrov additionally rely on label propagation. Compared to Das and Petrov, our model performs poorest on Italian, in terms of percentage point difference in accuracy. Figure 1 (left panel) shows accuracy, accuracy on known words, accuracy on unknown words, and proportion of known tokens for each iteration of our model for Italian; iteration 0 is the seed model, and iteration 3 1 is the final model. Our model performs poorly on unknown words as indicated by the low accuracy on unknown words, and high accuracy on known — — — words compared to the overall accuracy. The poor performance on unknown words is expected because we do not use any language-specific rules to handle this case. Moreover, on average for the final model, approximately 10% of the test data tokens are unknown. One way to improve the performance of our tagger might be to reduce the proportion of unknown words by using a larger training corpus, as Das and Petrov did. We examine the impact of self-training and revision over training iterations. We find that for all languages, accuracy rises quickly in the first 5–6 iterations, and then subsequently improves only slightly. We exemplify this in Figure 1 (right panel) for Dutch. (Findings are similar for other languages.) Although accuracy does not increase much in later iterations, they may still have some benefit as the vocabulary size continues to grow. 6 Conclusion We have proposed a method for unsupervised POS tagging that performs on par with the current state- of-the-art (Das and Petrov, 2011), but is substantially less-sophisticated (specifically not requiring convex optimization or a feature-based HMM). The complexity of our algorithm is O(nlogn) compared to O(n2) for that of Das and Petrov 637 (201 1) where n is the size of training data.3 We made our code are available for download.4 In future work we intend to consider using a larger training corpus to reduce the proportion of unknown tokens and improve accuracy. Given the improvements of our model over that of Das and Petrov on languages from the same family as our source language, and the observation of Snyder et al. (2008) that a better tagger can be learned from a more-closely related language, we also plan to consider strategies for selecting an appropriate source language for a given target language. Using our final model with unsupervised HMM methods might improve the final performance too, i.e. use our final model as the initial state for HMM, then experiment with differ- ent inference algorithms such as Expectation Maximization (EM), Variational Bayers (VB) or Gibbs sampling (GS).5 Gao and Johnson (2008) compare EM, VB and GS for unsupervised English POS tagging. In many cases, GS outperformed other methods, thus we would like to try GS first for our model. 7 Acknowledgements This work is funded by Erasmus Mundus European Masters Program in Language and Communication Technologies (EM-LCT) and by the Czech Science Foundation (grant no. P103/12/G084). We would like to thank Prokopis Prokopidis for providing us the Greek Treebank and Antonia Marti for the Spanish CoNLL 06 dataset. Finally, we thank Siva Reddy and Spandana Gella for many discussions and suggestions. References Thorsten Brants. 2000. TnT: A statistical part-ofspeech tagger. In Proceedings of the sixth conference on Applied natural language processing (ANLP ’00), pages 224–231 . Seattle, Washington, USA. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proceedings of 3We re-implemented label propagation from Das and Petrov (2011). It took over a day to complete this step on an eight core Intel Xeon 3.16GHz CPU with 32 Gb Ram, but only 15 minutes for our model. 4https://code.google.com/p/universal-tagger/ 5We in fact have tried EM, but it did not help. The overall performance dropped slightly. This might be because selftraining with revision already found the local maximal point. the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1 (ACL 2011), pages 600–609. Portland, Oregon, USA. Pascal Denis and Beno ıˆt Sagot. 2009. Coupling an annotated corpus and a morphosyntactic lexicon for state-of-the-art POS tagging with less human effort. In Proceedings of the 23rd PacificAsia Conference on Language, Information and Computation, pages 721–736. Hong Kong, China. Anna Feldman, Jirka Hana, and Chris Brew. 2006. A cross-language approach to rapid creation of new morpho-syntactically annotated resources. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’06), pages 549–554. Genoa, Italy. Jianfeng Gao and Mark Johnson. 2008. A comparison of bayesian estimators for unsupervised hidden markov model pos taggers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 344–352. Association for Computational Linguistics, Stroudsburg, PA, USA. Jiri Hana, Anna Feldman, and Chris Brew. 2004. A resource-light approach to Russian morphology: Tagging Russian using Czech resources. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP ’04), pages 222–229. Barcelona, Spain. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proceedings of the Tenth Machine Translation Summit (MT Summit X), pages 79–86. AAMT, Phuket, Thailand. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the main conference on Human Language Technology Conference ofthe North American Chapter of the Association of Computational Linguistics (HLT-NAACL ’06), pages 152–159. New York, USA. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 2089–2096. Istanbul, Turkey. Siva Reddy and Serge Sharoff. 2011. Cross language POS Taggers (and other tools) for Indian 638 languages: An experiment with Kannada using Telugu resources. In Proceedings of the IJCNLP 2011 workshop on Cross Lingual Information Access: Computational Linguistics and the Information Need of Multilingual Societies (CLIA 2011). Chiang Mai, Thailand. Benjamin Snyder, Tahira Naseem, Jacob Eisenstein, and Regina Barzilay. 2008. Unsupervised multilingual learning for POS tagging. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP ’08), pages 1041–1050. Honolulu, Hawaii. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Featurerich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Vol- ume 1 (NAACL ’03), pages 173–180. Edmonton, Canada. David Yarowsky and Grace Ngai. 2001 . Inducing multilingual POS taggers and NP bracketers via robust projection across aligned corpora. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies (NAACL ’01), pages 1–8. Pittsburgh, Pennsylvania, USA. 639
6 0.58769476 381 acl-2013-Variable Bit Quantisation for LSH
7 0.51510251 128 acl-2013-Does Korean defeat phonotactic word segmentation?
8 0.49041551 47 acl-2013-An Information Theoretic Approach to Bilingual Word Clustering
9 0.47879833 39 acl-2013-Addressing Ambiguity in Unsupervised Part-of-Speech Induction with Substitute Vectors
10 0.45828125 34 acl-2013-Accurate Word Segmentation using Transliteration and Language Model Projection
11 0.43678951 368 acl-2013-Universal Dependency Annotation for Multilingual Parsing
12 0.39584684 331 acl-2013-Stop-probability estimates computed on a large corpus improve Unsupervised Dependency Parsing
13 0.39561552 369 acl-2013-Unsupervised Consonant-Vowel Prediction over Hundreds of Languages
14 0.38911852 25 acl-2013-A Tightly-coupled Unsupervised Clustering and Bilingual Alignment Model for Transliteration
15 0.38442263 345 acl-2013-The Haves and the Have-Nots: Leveraging Unlabelled Corpora for Sentiment Analysis
16 0.37799853 276 acl-2013-Part-of-Speech Induction in Dependency Trees for Statistical Machine Translation
17 0.37299013 145 acl-2013-Exploiting Qualitative Information from Automatic Word Alignment for Cross-lingual NLP Tasks
18 0.35526779 295 acl-2013-Real-World Semi-Supervised Learning of POS-Taggers for Low-Resource Languages
19 0.35169968 356 acl-2013-Transfer Learning Based Cross-lingual Knowledge Extraction for Wikipedia
20 0.33828032 270 acl-2013-ParGramBank: The ParGram Parallel Treebank
topicId topicWeight
[(0, 0.056), (6, 0.055), (11, 0.062), (15, 0.016), (24, 0.035), (26, 0.084), (28, 0.011), (35, 0.065), (42, 0.083), (48, 0.053), (70, 0.043), (79, 0.018), (88, 0.127), (90, 0.03), (95, 0.151)]
simIndex simValue paperId paperTitle
1 0.96763206 345 acl-2013-The Haves and the Have-Nots: Leveraging Unlabelled Corpora for Sentiment Analysis
Author: Kashyap Popat ; Balamurali A.R ; Pushpak Bhattacharyya ; Gholamreza Haffari
Abstract: Expensive feature engineering based on WordNet senses has been shown to be useful for document level sentiment classification. A plausible reason for such a performance improvement is the reduction in data sparsity. However, such a reduction could be achieved with a lesser effort through the means of syntagma based word clustering. In this paper, the problem of data sparsity in sentiment analysis, both monolingual and cross-lingual, is addressed through the means of clustering. Experiments show that cluster based data sparsity reduction leads to performance better than sense based classification for sentiment analysis at document level. Similar idea is applied to Cross Lingual Sentiment Analysis (CLSA), and it is shown that reduction in data sparsity (after translation or bilingual-mapping) produces accuracy higher than Machine Translation based CLSA and sense based CLSA.
2 0.96589971 136 acl-2013-Enhanced and Portable Dependency Projection Algorithms Using Interlinear Glossed Text
Author: Ryan Georgi ; Fei Xia ; William D. Lewis
Abstract: As most of the world’s languages are under-resourced, projection algorithms offer an enticing way to bootstrap the resources available for one resourcepoor language from a resource-rich language by means of parallel text and word alignment. These algorithms, however, make the strong assumption that the language pairs share common structures and that the parse trees will resemble one another. This assumption is useful but often leads to errors in projection. In this paper, we will address this weakness by using trees created from instances of Interlinear Glossed Text (IGT) to discover patterns of divergence between the lan- guages. We will show that this method improves the performance of projection algorithms significantly in some languages by accounting for divergence between languages using only the partial supervision of a few corrected trees.
same-paper 3 0.94527209 97 acl-2013-Cross-lingual Projections between Languages from Different Families
Author: Mo Yu ; Tiejun Zhao ; Yalong Bai ; Hao Tian ; Dianhai Yu
Abstract: Cross-lingual projection methods can benefit from resource-rich languages to improve performances of NLP tasks in resources-scarce languages. However, these methods confronted the difficulty of syntactic differences between languages especially when the pair of languages varies greatly. To make the projection method well-generalize to diverse languages pairs, we enhance the projection method based on word alignments by introducing target-language word representations as features and proposing a novel noise removing method based on these word representations. Experiments showed that our methods improve the performances greatly on projections between English and Chinese.
4 0.93339825 111 acl-2013-Density Maximization in Context-Sense Metric Space for All-words WSD
Author: Koichi Tanigaki ; Mitsuteru Shiba ; Tatsuji Munaka ; Yoshinori Sagisaka
Abstract: This paper proposes a novel smoothing model with a combinatorial optimization scheme for all-words word sense disambiguation from untagged corpora. By generalizing discrete senses to a continuum, we introduce a smoothing in context-sense space to cope with data-sparsity resulting from a large variety of linguistic context and sense, as well as to exploit senseinterdependency among the words in the same text string. Through the smoothing, all the optimal senses are obtained at one time under maximum marginal likelihood criterion, by competitive probabilistic kernels made to reinforce one another among nearby words, and to suppress conflicting sense hypotheses within the same word. Experimental results confirmed the superiority of the proposed method over conventional ones by showing the better performances beyond most-frequent-sense baseline performance where none of SemEval2 unsupervised systems reached.
5 0.91886723 299 acl-2013-Reconstructing an Indo-European Family Tree from Non-native English Texts
Author: Ryo Nagata ; Edward Whittaker
Abstract: Mother tongue interference is the phenomenon where linguistic systems of a mother tongue are transferred to another language. Although there has been plenty of work on mother tongue interference, very little is known about how strongly it is transferred to another language and about what relation there is across mother tongues. To address these questions, this paper explores and visualizes mother tongue interference preserved in English texts written by Indo-European language speakers. This paper further explores linguistic features that explain why certain relations are preserved in English writing, and which contribute to related tasks such as native language identification.
6 0.91827464 41 acl-2013-Aggregated Word Pair Features for Implicit Discourse Relation Disambiguation
7 0.91618431 70 acl-2013-Bilingually-Guided Monolingual Dependency Grammar Induction
8 0.91491425 131 acl-2013-Dual Training and Dual Prediction for Polarity Classification
9 0.91157711 252 acl-2013-Multigraph Clustering for Unsupervised Coreference Resolution
10 0.90880644 292 acl-2013-Question Classification Transfer
11 0.90434432 196 acl-2013-Improving pairwise coreference models through feature space hierarchy learning
12 0.89967471 98 acl-2013-Cross-lingual Transfer of Semantic Role Labeling Models
13 0.89932746 327 acl-2013-Sorani Kurdish versus Kurmanji Kurdish: An Empirical Comparison
14 0.89707065 130 acl-2013-Domain-Specific Coreference Resolution with Lexicalized Features
15 0.89578438 211 acl-2013-LABR: A Large Scale Arabic Book Reviews Dataset
16 0.89469922 316 acl-2013-SenseSpotting: Never let your parallel data tie you to an old domain
17 0.89453197 9 acl-2013-A Lightweight and High Performance Monolingual Word Aligner
18 0.8942253 240 acl-2013-Microblogs as Parallel Corpora
19 0.89045954 264 acl-2013-Online Relative Margin Maximization for Statistical Machine Translation
20 0.88746834 267 acl-2013-PARMA: A Predicate Argument Aligner