acl acl2013 acl2013-251 knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Vladimir Eidelman ; Ke Wu ; Ferhan Ture ; Philip Resnik ; Jimmy Lin
Abstract: We present an open-source framework for large-scale online structured learning. Developed with the flexibility to handle cost-augmented inference problems such as statistical machine translation (SMT), our large-margin learner can be used with any decoder. Integration with MapReduce using Hadoop streaming allows efficient scaling with increasing size of training data. Although designed with a focus on SMT, the decoder-agnostic design of our learner allows easy future extension to other structured learning problems such as sequence labeling and parsing.
Reference: text
sentIndex sentText sentNum sentScore
1 of Linguistics 3 The iSchool Institute for Advanced Computer Studies University of Maryland {eide lman ,wuke , fture , re snik , j immyl in} @umd . [sent-4, score-0.132]
2 edu Abstract We present an open-source framework for large-scale online structured learning. [sent-5, score-0.287]
3 Developed with the flexibility to handle cost-augmented inference problems such as statistical machine translation (SMT), our large-margin learner can be used with any decoder. [sent-6, score-0.492]
4 Integration with MapReduce using Hadoop streaming allows efficient scaling with increasing size of training data. [sent-7, score-0.152]
5 Although designed with a focus on SMT, the decoder-agnostic design of our learner allows easy future extension to other structured learning problems such as sequence labeling and parsing. [sent-8, score-0.536]
6 1 Introduction Structured learning problems such as sequence labeling or parsing, where the output has a rich internal structure, commonly arise in NLP. [sent-9, score-0.189]
7 While batch learning algorithms adapted for structured learning such as CRFs (Lafferty et al. [sent-10, score-0.3]
8 , 2001) and structural SVMs (Joachims, 1998) have received much attention, online methods such as the structured perceptron (Collins, 2002) and a family of Passive-Aggressive algorithms (Crammer et al. [sent-11, score-0.287]
9 , 2006) have recently gained prominence across many tasks, including part-of-speech tagging (Shen, 2007), parsing (McDonald et al. [sent-12, score-0.043]
10 , 2005) and statistical machine translation (SMT) (Chiang, 2012), due to their ability to deal with large training sets and high-dimensional input representations. [sent-13, score-0.048]
11 Unlike batch learners, which must consider all examples when optimizing the objective, online learners operate in rounds, optimizing using one example or a handful of examples at a time. [sent-14, score-0.456]
12 This online nature offers several attractive properties, facilitating scaling to large training sets while remaining simple and offering fast convergence. [sent-15, score-0.349]
13 MIRA, the open source system1 described in this paper, implements an online largemargin structured learning algorithm based on MIRA (§2. [sent-17, score-0.353]
14 1), for cost-augmented online largeMscaIRleA training ,in f high-dimensional dfe oatnulrinee spaces. [sent-18, score-0.17]
15 Our contribution lies in providing the first published decoder-agnostic parallelization of MIRA with Hadoop for structured learning. [sent-19, score-0.235]
16 While the current demonstrated application focuses on large-scale discriminative training for machine translation, the learning algorithm is general with respect to the inference algorithm employed. [sent-20, score-0.08]
17 We are able to decouple our learner entirely from the MT decoder, allowing users to specify their own inference procedure through a simple text communication protocol (§2. [sent-21, score-0.554]
18 The lseimarnpeler only requires ka-tiboenst p output w (§i2th. [sent-23, score-0.064]
19 a tTuhree vectors, as well as the specification of a cost function. [sent-25, score-0.141]
20 Furthermore, our system can be extended to other structured learning problems with a min- imal amount of effort, simply by implementing a task-specific cost function and specifying an appropriate decoder. [sent-29, score-0.435]
21 Through Hadoop streaming, our system can take advantage of commodity clusters to handle large-scale training (§3), while also being capable loafr running itrna einnivnigro (n§m3)e,n wtsh ranging bfreoinmg a single machine to a PBS-managed batch cluster. [sent-30, score-0.178]
22 Experimental results (§4) show that it scales linearly and mmaenketasl fraesstu parameter tuning on large tuning sets for SMT practical. [sent-31, score-0.133]
23 1 Online Large-Margin Learning MIRA is a popular online large-margin structured learning method for NLP tasks (McDonald et al. [sent-33, score-0.287]
24 c A2s0s1o3ci Aatsiosonc fioartio Cno fmorpu Ctoamtiopnuatalt Lioin gauli Lsitnicgsu,i psatgices 19 –204, main intuition is that we want our model to enforce a margin between the correct and incorrect outputs of a sentence that agrees with our cost func- tion. [sent-39, score-0.285]
25 ∀y0 ∈ Y(xi), δfi(y0) ≥ ∆i(y0) ξi wt+1 = − where Y(xi) is the space of possible structured outputs we are able to produce from xi, and C is a regularization parameter that controls the size of the update. [sent-42, score-0.373]
26 With a passive-aggressive (PA) update, the ∀y0 constraint paabsosvive can g bree approximated by selecting the single most violated constraint, which maximizes y0 ← argmaxy∈Y(xi) w>f(xi, y) + ∆i(y). [sent-44, score-0.089]
27 This optimization problem is attractive because it reduces to a simple analytical solution, essentially performing a subgradient descent step with the step size adjusted based on each example: α ← min? [sent-45, score-0.238]
28 The cost can take any form as long as it decomposes across the local parts of the structure, just as the feature functions. [sent-52, score-0.144]
29 Cost-augmented Inference For most structured prediction problems in machine learning, yi ∈ Y(xi), that is, the model is able to produce, and∈ th Yu(sx score, the correct output structure, meaning = yi. [sent-54, score-0.408]
30 However, for certain NLP problems this may not be the case. [sent-55, score-0.075]
31 For instance in SMT, our model may not be able to produce or necessitates cost-augmented inference, where we select ← argmaxy∈Y(xi) w>f(xi, y) ∆i(y) y+ y+ − from the space gomf saxtructures our mod,ey)l can produce, to stand in for the correct output in optimization. [sent-56, score-0.266]
32 Our system was developed to handle both cases, with the decoder providing the k-best list to the learner, specifying whether to perform costaugmented selection. [sent-57, score-0.27]
33 Sparse Features While utilizing sparse features is a primary motivation for performing large-scale discriminative training, which features to use and how to learn their weights can have a large impact on the potential benefit. [sent-58, score-0.086]
34 To this end, we incorporate ‘1/‘2 regularization for joint feature selection in order to improve efficiency and counter overfitting effects (Simianer et al. [sent-59, score-0.105]
35 Furthermore, the PA update has a single learning rate η for all features, which specifies how much the feature weights can change at each update. [sent-61, score-0.146]
36 , language model) are observed far more frequently than sparse features (e. [sent-64, score-0.086]
37 , rule id), we may instead want to use a per-feature learning rate that allows larger steps for features that do not have much support. [sent-66, score-0.048]
38 Thus, we allow setting an adaptive per-feature learning rate (Green et al. [sent-67, score-0.048]
39 2 Learner/Decoder Communication Training requires communication between the decoder and the learner. [sent-72, score-0.219]
40 The decoder needs to receive weight updates and the input sentence from the learner; and the learner needs to receive k-best output with feature vectors from the decoder. [sent-73, score-0.712]
41 This is essentially all the required communication between the learner and the decoder. [sent-74, score-0.384]
42 Input sentence and weight updates Following common practice in machine translation, the learner encodes each input sentence as a singleline SGML entry named seg and sends it to the decoder. [sent-76, score-0.757]
43 In addition to the required sentence ID (useful in parallel processing), an optional delta field is used to encode the weight updates, as a sparse vector indexed by feature names. [sent-78, score-0.226]
44 First, for each name and upreach the correct reference translation, which prohibits our model from scoring it. [sent-79, score-0.05]
45 refer the reader date pair, a binary record consisting terminated of a null- string (name) and a double-precision floating point number in native byte order (update) is created. [sent-82, score-0.178]
wordName wordTfidf (topN-words)
[('seg', 0.33), ('xi', 0.287), ('learner', 0.233), ('kleine', 0.224), ('mira', 0.218), ('structured', 0.178), ('hadoop', 0.172), ('haus', 0.149), ('sgml', 0.149), ('smal', 0.149), ('decoder', 0.131), ('batch', 0.122), ('argmaxy', 0.115), ('smt', 0.114), ('crammer', 0.112), ('online', 0.109), ('id', 0.108), ('updates', 0.102), ('mapreduce', 0.101), ('cost', 0.099), ('update', 0.098), ('chiang', 0.096), ('communication', 0.088), ('streaming', 0.087), ('sparse', 0.086), ('specifying', 0.083), ('attractive', 0.082), ('inference', 0.08), ('problems', 0.075), ('learners', 0.071), ('receive', 0.071), ('byte', 0.066), ('ttuhree', 0.066), ('ost', 0.066), ('lman', 0.066), ('largemargin', 0.066), ('snik', 0.066), ('wt', 0.066), ('scaling', 0.065), ('output', 0.064), ('essentially', 0.063), ('fi', 0.061), ('dfe', 0.061), ('hou', 0.061), ('necessitates', 0.061), ('regularization', 0.059), ('hamming', 0.057), ('floating', 0.057), ('simianer', 0.057), ('rounds', 0.057), ('parallelization', 0.057), ('decouple', 0.057), ('delta', 0.057), ('optimizing', 0.057), ('handle', 0.056), ('umd', 0.055), ('protocol', 0.055), ('terminated', 0.055), ('sx', 0.052), ('sends', 0.052), ('labeling', 0.05), ('offering', 0.05), ('thi', 0.05), ('yy', 0.05), ('incurred', 0.05), ('ferhan', 0.05), ('duchi', 0.05), ('mcdonald', 0.05), ('correct', 0.05), ('produce', 0.05), ('translation', 0.048), ('rate', 0.048), ('agrees', 0.047), ('house', 0.047), ('analytical', 0.047), ('te', 0.047), ('tuning', 0.046), ('subgradient', 0.046), ('rre', 0.046), ('counter', 0.046), ('fc', 0.046), ('bleu', 0.045), ('outputs', 0.045), ('github', 0.045), ('violated', 0.045), ('decomposes', 0.045), ('incorrect', 0.044), ('constraint', 0.044), ('ix', 0.044), ('snover', 0.043), ('facilitating', 0.043), ('optional', 0.043), ('prominence', 0.043), ('xt', 0.042), ('specification', 0.042), ('able', 0.041), ('scales', 0.041), ('weight', 0.04), ('handful', 0.04), ('maryland', 0.04)]
simIndex simValue paperId paperTitle
same-paper 1 1.0 251 acl-2013-Mr. MIRA: Open-Source Large-Margin Structured Learning on MapReduce
Author: Vladimir Eidelman ; Ke Wu ; Ferhan Ture ; Philip Resnik ; Jimmy Lin
Abstract: We present an open-source framework for large-scale online structured learning. Developed with the flexibility to handle cost-augmented inference problems such as statistical machine translation (SMT), our large-margin learner can be used with any decoder. Integration with MapReduce using Hadoop streaming allows efficient scaling with increasing size of training data. Although designed with a focus on SMT, the decoder-agnostic design of our learner allows easy future extension to other structured learning problems such as sequence labeling and parsing.
2 0.29526308 264 acl-2013-Online Relative Margin Maximization for Statistical Machine Translation
Author: Vladimir Eidelman ; Yuval Marton ; Philip Resnik
Abstract: Recent advances in large-margin learning have shown that better generalization can be achieved by incorporating higher order information into the optimization, such as the spread of the data. However, these solutions are impractical in complex structured prediction problems such as statistical machine translation. We present an online gradient-based algorithm for relative margin maximization, which bounds the spread ofthe projected data while maximizing the margin. We evaluate our optimizer on Chinese-English and ArabicEnglish translation tasks, each with small and large feature sets, and show that our learner is able to achieve significant im- provements of 1.2-2 BLEU and 1.7-4.3 TER on average over state-of-the-art optimizers with the large feature set.
3 0.20453466 156 acl-2013-Fast and Adaptive Online Training of Feature-Rich Translation Models
Author: Spence Green ; Sida Wang ; Daniel Cer ; Christopher D. Manning
Abstract: We present a fast and scalable online method for tuning statistical machine translation models with large feature sets. The standard tuning algorithm—MERT—only scales to tens of features. Recent discriminative algorithms that accommodate sparse features have produced smaller than expected translation quality gains in large systems. Our method, which is based on stochastic gradient descent with an adaptive learning rate, scales to millions of features and tuning sets with tens of thousands of sentences, while still converging after only a few epochs. Large-scale experiments on Arabic-English and Chinese-English show that our method produces significant translation quality gains by exploiting sparse features. Equally important is our analysis, which suggests techniques for mitigating overfitting and domain mismatch, and applies to other recent discriminative methods for machine translation. 1
4 0.11935728 8 acl-2013-A Learner Corpus-based Approach to Verb Suggestion for ESL
Author: Yu Sawai ; Mamoru Komachi ; Yuji Matsumoto
Abstract: We propose a verb suggestion method which uses candidate sets and domain adaptation to incorporate error patterns produced by ESL learners. The candidate sets are constructed from a large scale learner corpus to cover various error patterns made by learners. Furthermore, the model is trained using both a native corpus and the learner corpus via a domain adaptation technique. Experiments on two learner corpora show that the candidate sets increase the coverage of error patterns and domain adaptation improves the performance for verb suggestion.
5 0.097605035 58 acl-2013-Automated Collocation Suggestion for Japanese Second Language Learners
Author: Lis Pereira ; Erlyn Manguilimotan ; Yuji Matsumoto
Abstract: This study addresses issues of Japanese language learning concerning word combinations (collocations). Japanese learners may be able to construct grammatically correct sentences, however, these may sound “unnatural”. In this work, we analyze correct word combinations using different collocation measures and word similarity methods. While other methods use well-formed text, our approach makes use of a large Japanese language learner corpus for generating collocation candidates, in order to build a system that is more sensitive to constructions that are difficult for learners. Our results show that we get better results compared to other methods that use only wellformed text. 1
6 0.089259051 221 acl-2013-Learning Non-linear Features for Machine Translation Using Gradient Boosting Machines
7 0.088235125 195 acl-2013-Improving machine translation by training against an automatic semantic frame based evaluation metric
8 0.084685378 173 acl-2013-Graph-based Semi-Supervised Model for Joint Chinese Word Segmentation and Part-of-Speech Tagging
9 0.08338964 361 acl-2013-Travatar: A Forest-to-String Machine Translation Engine based on Tree Transducers
10 0.083063722 334 acl-2013-Supervised Model Learning with Feature Grouping based on a Discrete Constraint
11 0.079206526 38 acl-2013-Additive Neural Networks for Statistical Machine Translation
12 0.077488296 328 acl-2013-Stacking for Statistical Machine Translation
13 0.076166756 19 acl-2013-A Shift-Reduce Parsing Algorithm for Phrase-based String-to-Dependency Translation
14 0.075796723 382 acl-2013-Variational Inference for Structured NLP Models
15 0.074517041 338 acl-2013-Task Alternation in Parallel Sentence Retrieval for Twitter Translation
16 0.074368283 181 acl-2013-Hierarchical Phrase Table Combination for Machine Translation
17 0.073170073 127 acl-2013-Docent: A Document-Level Decoder for Phrase-Based Statistical Machine Translation
18 0.072634421 164 acl-2013-FudanNLP: A Toolkit for Chinese Natural Language Processing
19 0.071715929 11 acl-2013-A Multi-Domain Translation Model Framework for Statistical Machine Translation
20 0.071560554 255 acl-2013-Name-aware Machine Translation
topicId topicWeight
[(0, 0.186), (1, -0.091), (2, 0.047), (3, 0.043), (4, -0.004), (5, 0.021), (6, 0.067), (7, -0.011), (8, -0.027), (9, 0.061), (10, -0.05), (11, 0.042), (12, -0.064), (13, -0.078), (14, 0.002), (15, 0.072), (16, -0.065), (17, 0.083), (18, 0.083), (19, -0.037), (20, 0.21), (21, 0.117), (22, -0.014), (23, 0.04), (24, 0.071), (25, 0.055), (26, 0.046), (27, 0.066), (28, -0.083), (29, -0.055), (30, 0.208), (31, 0.033), (32, -0.047), (33, -0.022), (34, 0.049), (35, 0.018), (36, -0.059), (37, 0.141), (38, 0.087), (39, -0.047), (40, 0.004), (41, -0.053), (42, 0.075), (43, -0.054), (44, -0.042), (45, 0.045), (46, 0.094), (47, -0.058), (48, 0.024), (49, 0.041)]
simIndex simValue paperId paperTitle
same-paper 1 0.95351708 251 acl-2013-Mr. MIRA: Open-Source Large-Margin Structured Learning on MapReduce
Author: Vladimir Eidelman ; Ke Wu ; Ferhan Ture ; Philip Resnik ; Jimmy Lin
Abstract: We present an open-source framework for large-scale online structured learning. Developed with the flexibility to handle cost-augmented inference problems such as statistical machine translation (SMT), our large-margin learner can be used with any decoder. Integration with MapReduce using Hadoop streaming allows efficient scaling with increasing size of training data. Although designed with a focus on SMT, the decoder-agnostic design of our learner allows easy future extension to other structured learning problems such as sequence labeling and parsing.
2 0.89992988 264 acl-2013-Online Relative Margin Maximization for Statistical Machine Translation
Author: Vladimir Eidelman ; Yuval Marton ; Philip Resnik
Abstract: Recent advances in large-margin learning have shown that better generalization can be achieved by incorporating higher order information into the optimization, such as the spread of the data. However, these solutions are impractical in complex structured prediction problems such as statistical machine translation. We present an online gradient-based algorithm for relative margin maximization, which bounds the spread ofthe projected data while maximizing the margin. We evaluate our optimizer on Chinese-English and ArabicEnglish translation tasks, each with small and large feature sets, and show that our learner is able to achieve significant im- provements of 1.2-2 BLEU and 1.7-4.3 TER on average over state-of-the-art optimizers with the large feature set.
3 0.82961106 156 acl-2013-Fast and Adaptive Online Training of Feature-Rich Translation Models
Author: Spence Green ; Sida Wang ; Daniel Cer ; Christopher D. Manning
Abstract: We present a fast and scalable online method for tuning statistical machine translation models with large feature sets. The standard tuning algorithm—MERT—only scales to tens of features. Recent discriminative algorithms that accommodate sparse features have produced smaller than expected translation quality gains in large systems. Our method, which is based on stochastic gradient descent with an adaptive learning rate, scales to millions of features and tuning sets with tens of thousands of sentences, while still converging after only a few epochs. Large-scale experiments on Arabic-English and Chinese-English show that our method produces significant translation quality gains by exploiting sparse features. Equally important is our analysis, which suggests techniques for mitigating overfitting and domain mismatch, and applies to other recent discriminative methods for machine translation. 1
4 0.74807686 24 acl-2013-A Tale about PRO and Monsters
Author: Preslav Nakov ; Francisco Guzman ; Stephan Vogel
Abstract: While experimenting with tuning on long sentences, we made an unexpected discovery: that PRO falls victim to monsters overly long negative examples with very low BLEU+1 scores, which are unsuitable for learning and can cause testing BLEU to drop by several points absolute. We propose several effective ways to address the problem, using length- and BLEU+1based cut-offs, outlier filters, stochastic sampling, and random acceptance. The best of these fixes not only slay and protect against monsters, but also yield higher stability for PRO as well as improved testtime BLEU scores. Thus, we recommend them to anybody using PRO, monsterbeliever or not. – 1 Once Upon a Time... For years, the standard way to do statistical machine translation parameter tuning has been to use minimum error-rate training, or MERT (Och, 2003). However, as researchers started using models with thousands of parameters, new scalable optimization algorithms such as MIRA (Watanabe et al., 2007; Chiang et al., 2008) and PRO (Hopkins and May, 2011) have emerged. As these algorithms are relatively new, they are still not quite well understood, and studying their properties is an active area of research. For example, Nakov et al. (2012) have pointed out that PRO tends to generate translations that are consistently shorter than desired. They have blamed this on inadequate smoothing in PRO’s optimization objective, namely sentencelevel BLEU+1, and they have addressed the problem using more sensible smoothing. We wondered whether the issue could be partially relieved simply by tuning on longer sentences, for which the effect of smoothing would naturally be smaller. To our surprise, tuning on the longer 50% of the tuning sentences had a disastrous effect on PRO, causing an absolute drop of three BLEU points on testing; at the same time, MERT and MIRA did not have such a problem. While investigating the reasons, we discovered hundreds of monsters creeping under PRO’s surface... Our tale continues as follows. We first explain what monsters are in Section 2, then we present a theory about how they can be slayed in Section 3, we put this theory to test in practice in Section 4, and we discuss some related efforts in Section 5. Finally, we present the moral of our tale, and we hint at some planned future battles in Section 6. 2 Monsters, Inc. PRO uses pairwise ranking optimization, where the learning task is to classify pairs of hypotheses into correctly or incorrectly ordered (Hopkins and May, 2011). It searches for a vector of weights w such that higher evaluation metric scores correspond to higher model scores and vice versa. More formally, PRO looks for weights w such that g(i, j) > g(i, j0) ⇔ hw (i, j) > hw (i, j0), where g is a local scoring fu hnction (typically, sentencelevel BLEU+1) and hw are the model scores for a given input sentence i and two candidate hypotheses j and j0 that were obtained using w. If g(i, j) > g(i, j0), we will refer to j and j0 as the positive and the negative example in the pair. Learning good parameter values requires negative examples that are comparable to the positive ones. Instead, tuning on long sentences quickly introduces monsters, i.e., corrupted negative examples that are unsuitable for learning: they are (i) much longer than the respective positive examples and the references, and (ii) have very low BLEU+1 scores compared to the positive examples and in absolute terms. The low BLEU+1 means that PRO effectively has to learn from positive examples only. 12 Proce dinSgosfi oa,f tB huel 5g1arsita, An Anu gauls Mt 4e-e9ti n2g01 o3f. th ?c e2 A0s1s3oc Aiastsio cnia fotiron C fo mrp Cuotmatpiounta tlio Lninaglu Li sntgicusi,s ptaicgses 12–17, Avg. Lengths Avg. BLEU+1 iter. pos neg ref. pos neg 1 45.2 44.6 46.5 52.5 37.6 2 3 4 5 ... 25 46.4 46.4 46.4 46.3 ... 47.9 70.5 261.0 250.0 248.0 ... 229.0 53.2 53.4 53.0 53.0 ... 52.5 52.8 52.4 52.0 52.1 ... 52.2 14.5 2.19 2.30 2.34 ... 2.81 Table 1: PRO iterations, tuning on long sentences. Table 1shows an optimization run of PRO when tuning on long sentences. We can see monsters after iterations in which positive examples are on average longer than negative ones (e.g., iter. 1). As a result, PRO learns to generate longer sentences, but it overshoots too much (iter. 2), which gives rise to monsters. Ideally, the learning algorithm should be able to recover from overshooting. However, once monsters are encountered, they quickly start dominating, with no chance for PRO to recover since it accumulates n-best lists, and thus also monsters, over iterations. As a result, PRO keeps jumping up and down and converges to random values, as Figure 1 shows. By default, PRO’s parameters are averaged over iterations, and thus the final result is quite mediocre, but selecting the highest tuning score does not solve the problem either: for example, on Figure 1, PRO never achieves a BLEU better than that for the default initialization parameters. iteration Figure 1: PRO tuning results on long sentences across iterations. The dark-gray line shows the tuning BLEU (left axis), the light-gray one is the hypothesis/reference length ratio (right axis). Figure 2 shows the translations after iterations 1, 3 and 4; the last two are monsters. The monster at iteration 3 is potentially useful, but that at iteration 4 is clearly unsuitable as a negative example. Optimizer Objective BLEU PROsent-BLEU+144.57 MERT corpus-BLEU 47.53 MIRA pseudo-doc-BLEU 47.80 PRO (6= objective)pseudo-doc-BLEU21.35 PMRIORA (6= =(6= o bojbejcetcivteiv)e) sent-BLEU+1 47.59 PMRIRO,A PC (6=-sm obojoectthiv,e g)roundfixed sent-BLEU+145.71 Table 2: PRO vs. MERT vs. MIRA. We also checked whether other popular optimizers yield very low BLEU scores at test time when tuned on long sentences. Lines 2-3 in Table 2 show that this is not the case for MERT and MIRA. Since they optimize objectives that are different from PRO’s,1 we further experimented with plugging MIRA’s objective into PRO and PRO’s objective into MIRA. The resulting MIRA scores were not much different from before, while PRO’s score dropped even further; we also found mon- sters. Next, we applied the length fix for PRO proposed in (Nakov et al., 2012); this helped a bit, but still left PRO two BLEU points behind and MIRA, and the monsters did not go away. We can conclude that the monster problem is PRO-specific, cannot be blamed on the objective function, and is different from the length bias. Note also that monsters are not specific to a dataset or language pair. We found them when tuning on the top-50% of WMT10 and testing on WMT1 1 for Spanish-English; this yielded a drop in BLEU from 29.63 (1M2/emEs/RprTes)la to 27.12 n(inPg/RtmOp.)1.1 MERT2 **REF** : but we have to close ranks with each other and realize that in unity there is strength while in division there is weakness . - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - **IT1** : but we are that we add our ranks to some of us and that we know that in the strength and weakness in **IT3** : , we are the but of the that that the , and , of ranks the the on the the our the our the some of we can include , and , of to the of we know the the our in of the of some people , force of the that that the in of the that that the the weakness Union the the , and **IT4** : namely Dr Heba Handossah and Dr Mona been pushed aside because a larger story EU Ambassador to Egypt Ian Burg highlighted 've dragged us backwards and dragged our speaking , never balme your defaulting a December 7th 1941 in Pearl Harbor ) we can include ranks will be joined by all 've dragged us backwards and dragged our $ 3 .8 billion in tourism income proceeds Chamber are divided among themselves : some ' ve dragged us backwards and dragged our were exaggerated . Al @-@ Hakim namely Dr Heba Handossah and Dr Mona December 7th 1941 in Pearl Harbor ) cases might be known to us December 7th 1941 in Pearl Harbor ) platform depends on combating all liberal policies Track and Field Federation shortened strength as well face several challenges , namely Dr Heba Handossah and Dr Mona platform depends on combating all liberal policies the report forecast that the weak structure Ftroai ngtkhsu et rahefef 2he : Ea t h xte ha,e motfo pstohlmeee r leafst eorfe wne c, et etr laonngs olfa t hei opnar a ofn sdo hhee oy fpwhaoitst hh a]r ee usisn i ostu tofra tnhes ilna tbiakoern s, haef ctoeokr it hee roant ainod nthse 1 t,h 3we aknonw d, 4. T@-@h eAl l tahes ft trwce o, tho ypotheses are monsters. 1See (Cherry and Foster, 2012) for details on objectives. 2Also, using PRO to initialize MERT, as implemented in Moses, yields 46.52 BLEU and monsters, but using MERT to initialize PRO yields 47.55 and no monsters. 13 3 Slaying Monsters: Theory Below we explain what monsters are and where they come from. Then, we propose various monster slaying techniques to be applied during PRO’s selection and acceptance steps. 3.1 What is PRO? PRO is a batch optimizer that iterates between (i) translation: using the current parameter values, generate k-best translations, and (ii) optimization: using the translations from all previous iterations, find new parameter values. The optimization step has four substeps: 1. Sampling: For each sentence, sample uniformly at random Γ = 5000 pairs from the set of all candidate translations for that sentence from all previous iterations. 2. Selection: From these sampled pairs, select those for which the absolute difference between their BLEU+1 scores is higher than α = 0.05 (note: this is 5 BLEU+1 points). 3. Acceptance: For each sentence, accept the Ξ = 50 selected pairs with the highest absolute difference in their BLEU+1 scores. 4. Learning: Assemble the accepted pairs for all sentences into a single set and use it to train a ranker to prefer the higher-scoring sentence in each pair. We believe that monsters are nurtured by PRO’s selection and acceptance policies. PRO’s selection step filters pairs involving hypotheses that differ by less than five BLEU+1 points, but it does not cut-off ones that differ too much based on BLEU+1 or length. PRO’s acceptance step selects Ξ = 50 pairs with the highest BLEU+1 differentials, which creates breeding ground for monsters since these pairs are very likely to include one monster and one good hypothesis. Below we discuss monster slaying geared towards the selection and acceptance steps of PRO. 3.2 Slaying at Selection In the selection step, PRO filters pairs for which the difference in BLEU+1 is less than five points, but it has no cut-off on the maximum BLEU+1 differentials nor cut-offs based on absolute length or difference in length. Here, we propose several selection filters, both deterministic and probabilistic. Cut-offs. A cut-off is a deterministic rule that filters out pairs that do not comply with some criteria. We experiment with a maximal cut-off on (a) the difference in BLEU+1 scores and (b) the difference in lengths. These are relative cut-offs because they refer to the pair, but absolute cut-offs that apply to each of the elements in the pair are also possible (not explored here). Cut-offs (a) and (b) slay monsters by not allowing the negative examples to get much worse in BLEU+1 or in length than the positive example in the pair. Filtering outliers. Outliers are rare or extreme observations in a sample. We assume normal distribution of the BLEU+1 scores (or of the lengths) of the translation hypotheses for the same source sentence, and we define as outliers hypotheses whose BLEU+1 (or length) is more than λ standard deviations away from the sample average. We apply the outlier filter to both the positive and the negative example in a pair, but it is more important for the latter. We experiment with values of λ like 2 and 3. This filtering slays monsters because they are likely outliers. However, it will not work if the population gets riddled with monsters, in which case they would become the norm. Stochastic sampling. Instead of filtering extreme examples, we can randomly sample pairs according to their probability of being typical. Let us assume that the values of the local scoring functions, i.e., the BLEU+1 scores, are distributed nor- mally: g(i, j) ∼ N(µ, σ2). Given a sample of hypothesis (tira,nj)sl ∼atio Nn(sµ {j} of the same source sentpeontchee i, we can ensstim {ja}te o σ empirically. Then, the difference ∆ = g(i, j) − g(i, j0) would be tdhisetr diibfufteerde normally w gi(thi, mean zero and variance 2σ2. Now, given a pair of examples, we can calculate their ∆, and we can choose to select the pair with some probability, according to N(0, 2σ2). 3.3 Slaying at Acceptance Another problem is caused by the acceptance mechanism of PRO: among all selected pairs, it accepts the top-Ξ with the highest BLEU+1 differentials. It is easy to see that these differentials are highest for nonmonster–monster pairs if such pairs exist. One way to avoid focusing primarily on such pairs is to accept a random set of pairs, among the ones that survived the selection step. One possible caveat is that we can lose some of the discriminative power of PRO by focusing on examples that are not different enough. Ξ 14 TESTING TUNING (run 1, it. 25, avg.) TEST(tune:full) PRO fix Avg. for 3 reruns BLEU StdDev Pos Lengths Neg Ref BLEU+1 Avg. for 3 reruns Pos Neg BLEU StdDev PRO (baseline)44.700.26647.9229.052.552.22.847.800.052 Max diff. cut-offBLEU+1 max=10†47.940.16547.949.649.449.439.947.770.035 BLEU+1 max=20 † 47.73 0.136 47.7 55.5 51.1 49.8 32.7 47.85 0.049 LEN max=5 † 48.09 0.021 46.8 47.0 47.9 52.9 37.8 47.73 0.051 LEN max=10 † 47.99 0.025 47.3 48.5 48.7 52.5 35.6 47.80 0.056 OutliersBLEU+1 λ=2.0†48.050.11946.847.247.752.239.547.470.090 BLEU+1 λ=3.0 LEN λ=2.0 LEN λ=3.0 47.12 46.68 47.02 1.348 2.005 0.727 47.6 49.3 48.2 168.0 82.7 163.0 53.0 53.1 51.4 51.7 52.3 51.4 3.9 5.3 4.2 47.53 47.49 47.65 0.038 0.085 0.096 Stoch. sampl.∆ BLEU+146.331.00046.8216.053.353.12.447.740.035 ∆ LEN 46.36 1.281 47.4 201.0 52.9 53.4 2.9 47.78 0.081 Table 3: Some fixes to PRO (select pairs with highest BLEU+1 differential, also require at least 5 BLEU+1 points difference). A dagger (†) indicates selection fixes that successfully get rid of monsters. 4 Attacking Monsters: Practice Below, we first present our general experimental setup. Then, we present the results for the various selection alternatives, both with the original acceptance strategy and with random acceptance. 4.1 Experimental Setup We used a phrase-based SMT model (Koehn et al., 2003) as implemented in the Moses toolkit (Koehn et al., 2007). We trained on all Arabic-English data for NIST 2012 except for UN, we tuned on (the longest-50% of) the MT06 sentences, and we tested on MT09. We used the MADA ATB segmentation for Arabic (Roth et al., 2008) and truecasing for English, phrases of maximal length 7, Kneser-Ney smoothing, and lexicalized reorder- ing (Koehn et al., 2005), and a 5-gram language model, trained on GigaWord v.5 using KenLM (Heafield, 2011). We dropped unknown words both at tuning and testing, and we used minimum Bayes risk decoding at testing (Kumar and Byrne, 2004). We evaluated the output with NIST’s scoring tool v.13a, cased. We used the Moses implementations of MERT, PRO and batch MIRA, with the –return-best-dev parameter for the latter. We ran these optimizers for up to 25 iterations and we used 1000-best lists. For stability (Foster and Kuhn, 2009), we performed three reruns of each experiment (tuning + evaluation), and we report averaged scores. 4.2 Selection Alternatives Table 3 presents the results for different selection alternatives. The first two columns show the testing results: average BLEU and standard deviation over three reruns. The following five columns show statistics about the last iteration (it. 25) of PRO’s tuning for the worst rerun: average lengths of the positive and the negative examples and average effective reference length, followed by average BLEU+1 scores for the positive and the negative examples in the pairs. The last two columns present the results when tuning on the full tuning set. These are included to verify the behavior of PRO in a nonmonster prone environment. We can see in Table 3 that all selection mechanisms considerably improve BLEU compared to the baseline PRO, by 2-3 BLEU points. However, not every selection alternative gets rid of monsters, which can be seen by the large lengths and low BLEU+1 for the negative examples (in bold). The max cut-offs for BLEU+1 and for lengths both slay the monsters, but the latter yields much lower standard deviation (thirteen times lower than for the baseline PRO!), thus considerably increasing PRO’s stability. On the full dataset, BLEU scores are about the same as for the original PRO (with small improvement for BLEU+1 max=20), but the standard deviations are slightly better. Rejecting outliers using BLEU+1 and λ = 3 is not strong enough to filter out monsters, but making this criterion more strict by setting λ = 2, yields competitive BLEU and kills the monsters. Rejecting outliers based on length does not work as effectively though. We can think of two possible reasons: (i) lengths are not normally distributed, they are more Poisson-like, and (ii) the acceptance criterion is based on the top-Ξ differentials based on BLEU+1, not based on length. On the full dataset, rejecting outliers, BLEU+1 and length, yields lower BLEU and less stability. 15 TESTING TUNING (run 1, it. 25, avg.) TEST(tune:full) Avg. for 3 reruns Lengths BLEU+1 Avg. for 3 reruns PRO fix BLEU StdDev Pos Neg Ref Pos Neg BLEU StdDev PRO (baseline)44.700.26647.9229.052.552.22.847.800.052 Rand. acceptPRO, rand††47.870.14747.748.548.7047.742.947.590.114 OutliersBLEU+1 λ=2.0, rand∗47.850.07848.248.448.947.543.647.620.091 BLEU+1 λ=3.0, rand 47.97 0.168 47.6 47.6 48.4 47.8 43.6 47.44 0.070 LEN λ=2.0, rand∗ 47.69 0.114 47.8 47.8 48.6 47.9 43.6 47.48 0.046 LEN λ=3.0, rand 47.89 0.235 47.8 48.0 48.7 47.7 43. 1 47.64 0.090 Stoch. sampl.∆ BLEU+1, rand∗47.990.08747.948.048.747.843.547.670.096 ∆ LEN, rand∗ 47.94 0.060 47.8 47.9 48.6 47.8 43.6 47.65 0.097 Table 4: More fixes to PRO (with random acceptance, no minimum BLEU+1). The (††) indicates that random acceptance kills monsters. The asterisk (∗) indicates improved stability over random acceptance. Reasons (i) and (ii) arguably also apply to stochastic sampling of differentials (for BLEU+1 or for length), which fails to kill the monsters, maybe because it gives them some probability of being selected by design. To alleviate this, we test the above settings with random acceptance. 4.3 Random Acceptance Table 4 shows the results for accepting training pairs for PRO uniformly at random. To eliminate possible biases, we also removed the min=0.05 BLEU+1 selection criterion. Surprisingly, this setup effectively eliminated the monster problem. Further coupling this with the distributional criteria can also yield increased stability, and even small further increase in test BLEU. For instance, rejecting BLEU outliers with λ = 2 yields comparable average test BLEU, but with only half the standard deviation. On the other hand, using the stochastic sampling of differentials based on either BLEU+1 or lengths improves the test BLEU score while increasing the stability across runs. The random acceptance has a caveat though: it generally decreases the discriminative power of PRO, yielding worse results when tuning on the full, nonmonster prone tuning dataset. Stochastic selection does help to alleviate this problem. Yet, the results are not as good as when using a max cut-off for the length. Therefore, we recommend using the latter as a default setting. 5 Related Work We are not aware of previous work that discusses the issue of monsters, but there has been work on a different, length problem with PRO (Nakov et al., 2012). We have seen that its solution, fix the smoothing in BLEU+1, did not work for us. The stability of MERT has been improved using regularization (Cer et al., 2008), random restarts (Moore and Quirk, 2008), multiple replications (Clark et al., 2011), and parameter aggregation (Cettolo et al., 2011). With the emergence of new optimization techniques, there have been studies that compare stability between MIRA–MERT (Chiang et al., 2008; Chiang et al., 2009; Cherry and Foster, 2012), PRO–MERT (Hopkins and May, 2011), MIRA– PRO–MERT (Cherry and Foster, 2012; Gimpel and Smith, 2012; Nakov et al., 2012). Pathological verbosity can be an issue when tuning MERT on recall-oriented metrics such as METEOR (Lavie and Denkowski, 2009; Denkowski and Lavie, 2011). Large variance between the results obtained with MIRA has also been reported (Simianer et al., 2012). However, none of this work has focused on monsters. 6 Tale’s Moral and Future Battles We have studied a problem with PRO, namely that it can fall victim to monsters, overly long negative examples with very low BLEU+1 scores, which are unsuitable for learning. We have proposed several effective ways to address this problem, based on length- and BLEU+1-based cut-offs, outlier filters and stochastic sampling. The best of these fixes have not only slayed the monsters, but have also brought much higher stability to PRO as well as improved test-time BLEU scores. These benefits are less visible on the full dataset, but we still recommend them to everybody who uses PRO as protection against monsters. Monsters are inherent in PRO; they just do not always take over. In future work, we plan a deeper look at the mechanism of monster creation in PRO and its possible connection to PRO’s length bias. 16 References Daniel Cer, Daniel Jurafsky, and Christopher Manning. 2008. Regularization and search for minimum error rate training. In Proc. of Workshop on Statistical Machine Translation, WMT ’08, pages 26–34. Mauro Cettolo, Nicola Bertoldi, and Marcello Federico. 2011. Methods for smoothing the optimizer instability in SMT. MT Summit XIII: the Machine Translation Summit, pages 32–39. Colin Cherry and George Foster. 2012. Batch tuning strategies for statistical machine translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT ’ 12, pages 427–436. David Chiang, Yuval Marton, and Philip Resnik. 2008. Online large-margin training of syntactic and structural translation features. In Proceedings ofthe Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 224–233. David Chiang, Kevin Knight, and Wei Wang. 2009. 11,001 new features for statistical machine transla- tion. In Proc. of the Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT ’09, pages 218–226. Jonathan Clark, Chris Dyer, Alon Lavie, and Noah Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proceedings of the Meeting of the Association for Computational Linguistics, ACL ’ 11, pages 176–181 . Michael Denkowski and Alon Lavie. 2011. Meteortuned phrase-based SMT: CMU French-English and Haitian-English systems for WMT 2011. Technical report, CMU-LTI-1 1-01 1, Language Technologies Institute, Carnegie Mellon University. George Foster and Roland Kuhn. 2009. Stabilizing minimum error rate training. In Proceedings of the Workshop on Statistical Machine Translation, StatMT ’09, pages 242–249. Kevin Gimpel and Noah Smith. 2012. Structured ramp loss minimization for machine translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT ’ 12, pages 221–231. Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Workshop on Statistical Machine Translation, WMT ’ 11, pages 187–197. Mark Hopkins and Jonathan May. 2011. Tuning as ranking. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, EMNLP ’ 11, pages 1352–1362. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, HLTNAACL ’03, pages 48–54. Philipp Koehn, Amittai Axelrod, Alexandra Birch Mayne, Chris Callison-Burch, Miles Osborne, and David Talbot. 2005. Edinburgh system description for the 2005 IWSLT speech translation evaluation. In Proceedings of the International Workshop on Spoken Language Translation, IWSLT ’05. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc. of the Meeting of the Association for Computational Linguistics, ACL ’07, pages 177–180. Shankar Kumar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, HLT-NAACL ’04, pages 169–176. Alon Lavie and Michael Denkowski. 2009. The METEOR metric for automatic evaluation of machine translation. Machine Translation, 23: 105–1 15. Robert Moore and Chris Quirk. 2008. Random restarts in minimum error rate training for statistical machine translation. In Proceedings of the International Conference on Computational Linguistics, COLING ’08, pages 585–592. Preslav Nakov, Francisco Guzm a´n, and Stephan Vogel. 2012. Optimizing for sentence-level BLEU+1 yields short translations. In Proceedings ofthe International Conference on Computational Linguistics, COLING ’ 12, pages 1979–1994. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the Meeting of the Association for Computational Linguistics, ACL ’03, pages 160–167. Ryan Roth, Owen Rambow, Nizar Habash, Mona Diab, and Cynthia Rudin. 2008. Arabic morphological tagging, diacritization, and lemmatization using lexeme models and feature ranking. In Proceedings of the Meeting of the Association for Computational Linguistics, ACL ’08, pages 117–120. Patrick Simianer, Stefan Riezler, and Chris Dyer. 2012. Joint feature selection in distributed stochastic learning for large-scale discriminative training in smt. In Proceedings of the Meeting of the Association for Computational Linguistics, ACL ’ 12, pages 11–21. Taro Watanabe, Jun Suzuki, Hajime Tsukada, and Hideki Isozaki. 2007. Online large-margin training for statistical machine translation. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL ’07, pages 764–773. 17
5 0.6807223 221 acl-2013-Learning Non-linear Features for Machine Translation Using Gradient Boosting Machines
Author: Kristina Toutanova ; Byung-Gyu Ahn
Abstract: In this paper we show how to automatically induce non-linear features for machine translation. The new features are selected to approximately maximize a BLEU-related objective and decompose on the level of local phrases, which guarantees that the asymptotic complexity of machine translation decoding does not increase. We achieve this by applying gradient boosting machines (Friedman, 2000) to learn new weak learners (features) in the form of regression trees, using a differentiable loss function related to BLEU. Our results indicate that small gains in perfor- mance can be achieved using this method but we do not see the dramatic gains observed using feature induction for other important machine learning tasks.
6 0.66048652 334 acl-2013-Supervised Model Learning with Feature Grouping based on a Discrete Constraint
7 0.59812331 328 acl-2013-Stacking for Statistical Machine Translation
8 0.59384668 277 acl-2013-Part-of-speech tagging with antagonistic adversaries
9 0.54217106 137 acl-2013-Enlisting the Ghost: Modeling Empty Categories for Machine Translation
10 0.51773387 195 acl-2013-Improving machine translation by training against an automatic semantic frame based evaluation metric
11 0.47144124 260 acl-2013-Nonconvex Global Optimization for Latent-Variable Models
12 0.46430448 127 acl-2013-Docent: A Document-Level Decoder for Phrase-Based Statistical Machine Translation
13 0.41596946 38 acl-2013-Additive Neural Networks for Statistical Machine Translation
14 0.41151977 382 acl-2013-Variational Inference for Structured NLP Models
15 0.40715963 236 acl-2013-Mapping Source to Target Strings without Alignment by Analogical Learning: A Case Study with Transliteration
16 0.40554047 8 acl-2013-A Learner Corpus-based Approach to Verb Suggestion for ESL
17 0.39154139 14 acl-2013-A Novel Classifier Based on Quantum Computation
18 0.38884372 110 acl-2013-Deepfix: Statistical Post-editing of Statistical Machine Translation Using Deep Syntactic Analysis
19 0.38714701 11 acl-2013-A Multi-Domain Translation Model Framework for Statistical Machine Translation
20 0.38655323 383 acl-2013-Vector Space Model for Adaptation in Statistical Machine Translation
topicId topicWeight
[(0, 0.075), (6, 0.067), (11, 0.097), (15, 0.02), (24, 0.032), (26, 0.042), (35, 0.057), (40, 0.017), (42, 0.088), (48, 0.066), (54, 0.185), (70, 0.034), (88, 0.025), (90, 0.029), (95, 0.102)]
simIndex simValue paperId paperTitle
1 0.87307352 69 acl-2013-Bilingual Lexical Cohesion Trigger Model for Document-Level Machine Translation
Author: Guosheng Ben ; Deyi Xiong ; Zhiyang Teng ; Yajuan Lu ; Qun Liu
Abstract: In this paper, we propose a bilingual lexical cohesion trigger model to capture lexical cohesion for document-level machine translation. We integrate the model into hierarchical phrase-based machine translation and achieve an absolute improvement of 0.85 BLEU points on average over the baseline on NIST Chinese-English test sets.
same-paper 2 0.85838521 251 acl-2013-Mr. MIRA: Open-Source Large-Margin Structured Learning on MapReduce
Author: Vladimir Eidelman ; Ke Wu ; Ferhan Ture ; Philip Resnik ; Jimmy Lin
Abstract: We present an open-source framework for large-scale online structured learning. Developed with the flexibility to handle cost-augmented inference problems such as statistical machine translation (SMT), our large-margin learner can be used with any decoder. Integration with MapReduce using Hadoop streaming allows efficient scaling with increasing size of training data. Although designed with a focus on SMT, the decoder-agnostic design of our learner allows easy future extension to other structured learning problems such as sequence labeling and parsing.
3 0.84768909 212 acl-2013-Language-Independent Discriminative Parsing of Temporal Expressions
Author: Gabor Angeli ; Jakob Uszkoreit
Abstract: Temporal resolution systems are traditionally tuned to a particular language, requiring significant human effort to translate them to new languages. We present a language independent semantic parser for learning the interpretation of temporal phrases given only a corpus of utterances and the times they reference. We make use of a latent parse that encodes a language-flexible representation of time, and extract rich features over both the parse and associated temporal semantics. The parameters of the model are learned using a weakly supervised bootstrapping approach, without the need for manually tuned parameters or any other language expertise. We achieve state-of-the-art accuracy on all languages in the TempEval2 temporal normalization task, reporting a 4% improvement in both English and Spanish accuracy, and to our knowledge the first results for four other languages.
4 0.73754936 358 acl-2013-Transition-based Dependency Parsing with Selectional Branching
Author: Jinho D. Choi ; Andrew McCallum
Abstract: We present a novel approach, called selectional branching, which uses confidence estimates to decide when to employ a beam, providing the accuracy of beam search at speeds close to a greedy transition-based dependency parsing approach. Selectional branching is guaranteed to perform a fewer number of transitions than beam search yet performs as accurately. We also present a new transition-based dependency parsing algorithm that gives a complexity of O(n) for projective parsing and an expected linear time speed for non-projective parsing. With the standard setup, our parser shows an unlabeled attachment score of 92.96% and a parsing speed of 9 milliseconds per sentence, which is faster and more accurate than the current state-of-the-art transitionbased parser that uses beam search.
5 0.73209423 156 acl-2013-Fast and Adaptive Online Training of Feature-Rich Translation Models
Author: Spence Green ; Sida Wang ; Daniel Cer ; Christopher D. Manning
Abstract: We present a fast and scalable online method for tuning statistical machine translation models with large feature sets. The standard tuning algorithm—MERT—only scales to tens of features. Recent discriminative algorithms that accommodate sparse features have produced smaller than expected translation quality gains in large systems. Our method, which is based on stochastic gradient descent with an adaptive learning rate, scales to millions of features and tuning sets with tens of thousands of sentences, while still converging after only a few epochs. Large-scale experiments on Arabic-English and Chinese-English show that our method produces significant translation quality gains by exploiting sparse features. Equally important is our analysis, which suggests techniques for mitigating overfitting and domain mismatch, and applies to other recent discriminative methods for machine translation. 1
6 0.73203504 264 acl-2013-Online Relative Margin Maximization for Statistical Machine Translation
7 0.73074162 83 acl-2013-Collective Annotation of Linguistic Resources: Basic Principles and a Formal Model
8 0.72832215 9 acl-2013-A Lightweight and High Performance Monolingual Word Aligner
9 0.72779417 18 acl-2013-A Sentence Compression Based Framework to Query-Focused Multi-Document Summarization
10 0.72769737 38 acl-2013-Additive Neural Networks for Statistical Machine Translation
11 0.72746813 333 acl-2013-Summarization Through Submodularity and Dispersion
12 0.7269581 68 acl-2013-Bilingual Data Cleaning for SMT using Graph-based Random Walk
13 0.72664475 70 acl-2013-Bilingually-Guided Monolingual Dependency Grammar Induction
14 0.72618514 132 acl-2013-Easy-First POS Tagging and Dependency Parsing with Beam Search
15 0.72575474 259 acl-2013-Non-Monotonic Sentence Alignment via Semisupervised Learning
16 0.72546631 164 acl-2013-FudanNLP: A Toolkit for Chinese Natural Language Processing
17 0.72535038 223 acl-2013-Learning a Phrase-based Translation Model from Monolingual Data with Application to Domain Adaptation
18 0.72485882 226 acl-2013-Learning to Prune: Context-Sensitive Pruning for Syntactic MT
19 0.72468758 343 acl-2013-The Effect of Higher-Order Dependency Features in Discriminative Phrase-Structure Parsing
20 0.72467959 123 acl-2013-Discriminative Learning with Natural Annotations: Word Segmentation as a Case Study