acl acl2012 acl2012-56 acl2012-56-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Geoffrey Zweig ; John C. Platt ; Christopher Meek ; Christopher J.C. Burges ; Ainur Yessenalina ; Qiang Liu
Abstract: This paper studies the problem of sentencelevel semantic coherence by answering SATstyle sentence completion questions. These questions test the ability of algorithms to distinguish sense from nonsense based on a variety of sentence-level phenomena. We tackle the problem with two approaches: methods that use local lexical information, such as the n-grams of a classical language model; and methods that evaluate global coherence, such as latent semantic analysis. We evaluate these methods on a suite of practice SAT questions, and on a recently released sentence completion task based on data taken from five Conan Doyle novels. We find that by fusing local and global information, we can exceed 50% on this task (chance baseline is 20%), and we suggest some avenues for further research.
J. Bellegarda. 2000. Exploiting latent semantic information in statistical language modeling. Proceedings of the IEEE, 88(8). Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2): 157 –166. Y. Benjamini and Y. Hochberg. 1995. Controlling the fase discovery rate: a practical and powerful approach to multiple testing. J. Royal Statistical Society B, 53(1):289–300. C. Burges, T. Shaked., E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. 2005. Learning to rank using gradient descent. In Proc. ICML, pages 89– 96. Eugene Charniak, Yasemin Altun, Rodrigo de Salvo Braz, Benjamin Garrett, Margaret Kosmala, Tomer Moscovich, Lixin Pang, Changhee Pyo, Ye Sun, Wei Wy, Zhongfa Yang, Shawn Zeller, and Lisa Zorn. 2000. Reading comprehension programs in a statistical-language-processing class. In Proceedings of the 2000 ANLP/NAACL Workshop on Reading comprehension tests as evaluation for computerbased language understanding sytems - Volume 6, ANLP/NAACL-ReadingComp ’00, pages 1–5. Association for Computational Linguistics. Stanley Chen and Joshua Goodman. 1999. An empirical study of smoothing techniques for language modeling. Computer Speech and Language, 13(4):359–393. S. Chen, L. Mangu, B. Ramabhadran, R. Sarikaya, and A. Sethy. 2009. Scaling shrinkage-based language models. In ASRU. S. Chen. 2009a. Performance prediction for exponential language models. In NAACL-HLT. S. Chen. 2009b. Shrinking exponential language models. In NAACL-HLT. P.R. Clarkson and R. Rosenfeld. 1997. Statistical language modeling using the CMU-Cambridge Toolkit. In Proceedings ESCA Eurospeech, http://www.speech.cs.cmu.edu/SLM/toolkit.html. N. Coccaro and D. Jurafsky. 1998. Towards better integration of semantic predictors in statistical language modeling. In Proceedings, ICSLP. Bollegala D., Matsuo Y., and Ishizuka M. 2009. Measuring the similarity between implicit semantic relations from the web. In World Wide Web Conference (WWW). S. Deerwester, S.T. Dumais, G.W. Furnas, T.K. Landauer, and R. Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(96). 609 T.G. Dietterich. 1998. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10: 1895–1923. T.G. Dietterich. 2000. Ensemble methods in machine learning. In International Workshop on Multiple Classifier Systems, pages 1–15. Springer-Verlag. Educational-Testing-Service. 2011. https://satonlinecourse.collegeboard.com/sr/digital assets/ assessment/pdf/0833a61 1-0a43-10c2-0148cc8c0087fb06-f.pdf. A. Emami, S. Chen, A. Ittycheriah, H. Soltau, and B. Zhao. 2010. Decoding with shrinkage-based language models. In Interspeech. Claudio Giuliano, Alfio Gliozzo, and Carlo Strapparava. 2007. Fbk-irst: Lexical substitution task exploiting domain and syntagmatic coherence. In Proceedings of the 4th International Workshop on Semantic Evaluations, SemEval ’07, pages 145–148, Stroudsburg, PA, USA. Association for Computational Linguistics. Samer Hassan, Andras Csomai, Carmen Banea, Ravi Sinha, and Rada Mihalcea. 2007. Unt: Subfinder: Combining knowledge sources for automatic lexical substitution. In Proceedings of the 4th International Workshop on Semantic Evaluations, SemEval ’07, pages 410–413, Stroudsburg, PA, USA. Association for Computational Linguistics. Lynette Hirschman, Mark Light, Eric Breck, and John D. Burger. 1999. Deep read: A reading comprehension system. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics. Thomas Landauer and Susan Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review, 104(2), pages 211– 240. Iddo Lev, Bill MacCartney, Christopher D. Manning, and Roger Levy. 2004. Solving logic puzzles: from robust processing to precise semantics. In Proceedings of the 2nd Workshop on Text Meaning and Interpretation, pages 9–16. Association for Computational Linguistics. Jarmasz M. and Szpakowicz S. 2003. Roget’s thesaurus and semantic similarity. In Recent Advances in Natural Language Processing (RANLP). Diana McCarthy and Roberto Navigli. 2007. Semeval2007 task 10: English lexical substitution task. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 48–53. Tomas Mikolov, Martin Karafiat, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of Interspeech 2010. Tomas Mikolov, Anoop Deoras, Stefan Kombrink, Lukas Burget, and Jan Cernocky. 2011a. Empirical evaluation and combination of advanced language modeling techniques. In Proceedings of Interspeech 2011. Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2011b. Extensions of recurrent neural network based language model. In Proceedings of ICASSP 2011. Saif Mohammed, Bonnie Dorr, and Graeme Hirst. 2008. Computing word pair antonymy. In Empirical Methods in Natural Language Processing (EMNLP). Saif M. Mohammed, Bonnie J. Dorr, Graeme Hirst, and Peter D. Turney. 2011. Measuring degrees of semantic opposition. Technical report, National Research Council Canada. Hwee Tou Ng, Leong Hwee Teo, and Jennifer Lai Pheng Kwan. 2000. A machine learning approach to answering questions for reading comprehension tests. In Proceedings of the 2000 Joint SIGDAT conference on Empirical methods in natural language processing and very large corpora: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics - Volume 13, EMNLP ’00, pages 124–132. J. Nocedal and S. Wright. 2006. Numerical Optimization. Springer-Verlag. Sebastian Pado and Mirella Lapata. 2007. Dependencybased construction of semantic space models. Computational Linguistics, 33 (2), pages 161–199. Princeton-Review. 2010. 11 Practice Tests for the SAT & PSAT, 2011 Edition. The Princeton Review. Ellen Riloff and Michael Thelen. 2000. A rule-based question answering system for reading comprehension tests. In Proceedings of the 2000 ANLP/NAACL Workshop on Reading comprehension tests as evaluationfor computer-based language understanding sytems - Volume 6, ANLP/NAACL-ReadingComp ’00, pages 13– 19. G. Salton, A. Wong, and C. S. Yang. 1975. A Vector Space Model for Automatic Indexing. Communications of the ACM, 18(1 1). Richard Socher, Cliff Chiung-Yu Lin, Andrew Y. Ng, and Christopher D. Manning. 2011. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 2011 International Conference on Machine Learning (ICML-2011). Ilya Sutskever, James Martens, and Geoffrey Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of the 2011 International Conference on Machine Learning (ICML-2011). E. Terra and C. Clarke. 2003. Frequency estimates for statistical word similarity measures. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). 610 Peter Turney and Michael Littman. 2005. Corpus-based learning of analogies and semantic relations. Machine Learning, 60 (1-3), pages 25 1–278. Peter D. Turney, Michael L. Littman, Jeffrey Bigham, and Victor Shnayder. 2003. Combining independent modules to solve multiple-choice synonym and analogy problems. In Recent Advances in Natural Language Processing (RANLP). Peter D. Turney. 2001. Mining the web for synonyms: PMI-IR versus LSA on TOEFL. In European Conference on Machine Learning (ECML). Peter Turney. 2008. A uniform approach to analogies, synonyms, antonyms, and associations. In International Conference on Computational Linguistics (COLING). T. Veale. 2004. Wordnet sits the sat: A knowledge-based approach to lexical analogy. In European Conference on Artificial Intelligence (ECAI). W. Wang, J. Auer, R. Parasuraman, I. Zubarev, D. Brandyberry, and M. P. Harper. 2000. A question answering system developed as a project in a natural language processing course. In Proceedings of the 2000 ANLP/NAACL Workshop on Reading comprehension tests as evaluation for computerbased language understanding sytems - Volume 6, ANLP/NAACL-ReadingComp ’00, pages 28–35. Deniz Yuret. 2007. Ku: word sense disambiguation by substitution. In Proceedings of the 4th International Workshop on Semantic Evaluations, SemEval ’07, pages 207–213, Stroudsburg, PA, USA. Association for Computational Linguistics. Geoffrey Zweig and Christopher J.C. Burges. 2011. The Microsoft Research sentence completion challenge. Technical Report MSR-TR-201 1-129, Microsoft.