acl acl2012 acl2012-194 acl2012-194-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Hiroshi Yamaguchi ; Kumiko Tanaka-Ishii
Abstract: The problem addressed in this paper is to segment a given multilingual document into segments for each language and then identify the language of each segment. The problem was motivated by an attempt to collect a large amount of linguistic data for non-major languages from the web. The problem is formulated in terms of obtaining the minimum description length of a text, and the proposed solution finds the segments and their languages through dynamic programming. Empirical results demonstrating the potential of this approach are presented for experiments using texts taken from the Universal Declaration of Human Rights and Wikipedia, covering more than 200 languages.
Beatrice Alex, Amit Dubey, and Frank Keller. 2007. Using foreign inclusion detection to improve parsing performance. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 151–160. Beatrice Alex. 2005. An unsupervised system for identifying english inclusions in german text. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, Student Research Workshop, pages 133–138. T.C. Bell, J.G. Cleary, and I. H. Witten. 1990. Text Compression. Prentice Hall. Dario Benedetto, Emanuele Caglioti, and Vittorio Loreto. 2002. Language trees and zipping. Physical Review Letters, 88(4). Rudi Cilibrasi and Paul Vit a´nyi. 2005. Clustering by compression. IEEE Transactions on Information Theory, 51(4): 1523–1545. John G. Cleary and Ian H. Witten. 1984. Data compression using adaptive coding and partial string matching. IEEE Transactions on Communications, 32:396–402. Martin Farach, Michiel Noordewier, Serap Savari, Larry Shepp, Abraham J. Wyner, and Jacob Ziv. 1994. On the entropy of dna: Algorithms and measurements based on memory and rapid convergence. In Proceedings of the Sixth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 48–57. Gregory Grefenstette. 1995. Comparing two language identification schemes. In Proceedings of 3rd International Conference on Statistical Analysis of Textual Data, pages 263–268. Marti A. Hearst. 1997. Texttiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics, 23(1):33–64. Patrick Juola. 1997. What can we do with small corpora? document categorization via cross-entropy. In Proceedings of an Interdisciplinary Workshop on Similarity and Categorization. Gen-itiro Kikui. 1996. Identifying the coding system and language of on-line documents on the internet. In Proceedings of 16th International Conference on Computational Linguistics, pages 652–657. Casanai Kruengkrai, Prapass Srichaivattana, Virach Sornlertlamvanich, and Hitoshi Isahara. 2005. Language identification based on string kernels. In Proceedings of the 5th International Symposium on Communications and Information Technologies, pages 926–929. Penelope Sibun and Jeffrey C. Reynar. 1996. Language identification: Examining the issues. In Proceedings of 5th Symposium on Document Analysis and Information Retrieval, pages 125–135. William J. Teahan and David J. Harper. 2001 . Using compression-based language models for text categorization. In Proceedings of the Workshop on Language Modeling and Information Retrieval, pages 83–88. William John Teahan. 2000. Text classification and segmentation using minimum cross-entropy. In RIAO, pages 943–961. Jacob Ziv and Abraham Lempel. 1977. A universal algorithm for sequential data compression. IEEE Transactions on Information Theory, 23(3):337–343. Appendix This Appendix lists all the languages contained in our data sets, as summarized in Table 1. For UDHR Latin Achinese, Achuar-Shiwiar, Adangme, Afrikaans, Aguaruna, Aja, Akuapem Akan, Akurio, Amahuaca, Amarakaeri, AmboPasco Quechua, Arabela, Arequipa-La Uni o´n Quechua, Arpitan, Asante Akan, Ash a´ninka, Ash e´ninka Pajonal, Asturian, Auvergnat Occitan, Ayacucho Quechua, Aymara, Baatonum, Balinese, Bambara, Baoul ´e, Basque, Bemba, Beti, Bikol, Bini, Bislama, Bokm a˚l Norwegian, Bora, Bosnian, Breton, Buginese, Cajamarca Quechua, Calder´ on Highland Quichua, CandoshiShapra, Caquinte, Cashibo-Cacataibo, Cashinahua, Catalan, Cebuano, Central Kanuri, Central Mazahua, Central Nahuatl, Chamorro, Chamula Tzotzil, Chayahuita, Chickasaw, Chiga, Chokwe, Chuanqiandian Cluster Miao, Chuukese, Corsican, Cusco Quechua, Czech, Dagbani, Danish, Dendi, Ditammari, Dutch, Eastern Maninkakan, Emiliano-Romagnolo, English, Esperanto, Estonian, Ewe, Falam Chin, Fanti, Faroese, Fijian, Filipino, Finnish, Fon, French, Friulian, Ga, Gagauz, Galician, Ganda, Garifuna, Gen, German, Gheg Albanian, Gonja, Guarani, G ¨uil´ a Zapotec, Haitian Creole, Haitian Creole (popular), Haka Chin, Hani, Hausa, Hawaiian, Hiligaynon, Huamal´ ıes-Dos de Mayo Hu´ anuco Quechua, Huautla Mazatec, Huaylas Ancash Quechua, Hungarian, Ibibio, Icelandic, Ido, Igbo, Iloko, Indonesian, Interlingua, Irish, Italian, Javanese, Jola-Fonyi, K’iche’, Kabiy `e, Kabuverdianu, Kalaallisut, Kaonde, Kaqchikel, Kasem, Kekch ´ı, Kimbundu, Kinyarwanda, Kituba, Konzo, Kpelle, Krio, Kurdish, Lamnso’, Languedocien Occitan, Latin, Latvian, Lingala, Lithuanian, Lozi, Luba-Lulua, Lunda, Luvale, Luxembourgish, Madurese, Makhuwa, Makonde, Malagasy, Maltese, Mam, Maori, Mapudungun, Margos-Yarowilca-Lauricocha Quechua, Marshallese, Mba, Mende, Metlat´ onoc Mixtec, Mezquital Otomi, Mi’kmaq, Miahuatl a´n Zapotec, Minangkabau, Mossi, Mozarabic, Murui Huitoto, M ´ıskito, Ndonga, Nigerian Pidgin, Nomatsiguenga, North Jun ı´n Quechua, Northeastern Dinka, Northern Conchucos Ancash Quechua, Northern Qiandong Miao, Northern Sami, Northern Kurdish, Nyamwezi, Nyanja, Nyemba, Nynorsk Norwegian, Nzima, Ojitl ´aan Chinantec, Oromo, Palauan, Pampanga, Papantla Totonac, Pedi, Picard, Pichis Ash e´ninka, Pijin, Pipil, Pohnpeian, Polish, laar, Purepecha, P a´ez, Quechua, Rarotongan, mansh, Romany, Rundi, Salinan, Samoan, Huastec, Sango, Sardinian, Scots, Scottish Portuguese, PuRomanian, RoSan Lu´ ıs Potos ´ı Gaelic, Serbian, 977 Serer, Seselwa Creole French, Sharanahua, Shipibo-Conibo, Shona, Slovak, Somali, Soninke, South Ndebele, Southern Dagaare, Southern Qiandong Miao, Southern Sotho, Spanish, Standard Malay, Sukuma, Sundanese, Susu, Swahili, Swati, Swedish, S ˜aotomense, Tahitian, Tedim Chin, Tetum, Tidikelt Tamazight, Timne, Tiv, Toba, Tojolabal, Tok Pisin, Tonga (Tonga Islands), Tonga (Zambia), Tsonga, Tswana, Turkish, Tzeltal, Umbundu, Upper Sorbian, Urarina, Uzbek, Veracruz Huastec, Vili, Vlax Romani, Walloon, Waray, Wayuu, Welsh, Western Frisian, Wolof, Xhosa, Yagua, Yanesha’, Yao, Yapese, Yoruba, Yucateco, Zhuang, Zulu Cyrillic Abkhazian, Belarusian, Bosnian, Bulgarian, Kazakh, Macedonian, Ossetian, Russian, Serbian, Tuvinian, Ukrainian, Yakut Arabic Standard Arabic Other Japanese, Korean, Mandarin Chinese, Modern Greek For Wiki Latin Afrikaans, Albanian, Aragonese, Aromanian, Arpitan, Asturian, Aymara, Azerbaijani, Bambara, Banyumasan, Basque, Bavarian, Bislama, Bosnian, Breton, Catal `a, Cebuano, Central Bikol, Chavacano, Cornish, Corsican, Crimean Tatar, Croatian, Czech, Danish, Dimli, Dutch, Dutch Low Saxon, EmilianoRomagnolo, English, Esperanto, Estonian, Ewe, Extremaduran, Faroese, Fiji Hindi, Finnish, French, Friulian, Galician, German, Gilaki, Gothic, Guarani, Hai//om, Haitian, Hakka Chinese, Hawaiian, Hungarian, Icelandic, Ido, Igbo, Iloko, Indonesian, Interlingua, Interlingue, Irish, Italian, Javanese, Kabyle, Kalaallisut, Kara-Kalpak, Kashmiri, Kashubian, Kongo, Korean, Kurdish, Ladino, Latin, Latvian, Ligurian, Limburgan, Lingala, Lithuanian, Lojban, Lombard, Low German, Lower Sorbian, Luxembourgish, Malagasy, Malay, Maltese, Manx, Maori, Mazanderani, Min Dong Chinese, Min Nan Chinese, Nahuatl, Narom, Navajo, Neapolitan, Northern Sami, Norwegian, Norwegian Nynorsk, Novial, Occitan, Old English, Pampanga, Pangasinan, Panjabi, Papiamento, Pennsylvania German, Piemontese, Pitcairn-Norfolk, Polish, Portuguese, Pushto, Quechua, Romanian, Romansh, Samoan, Samogitian Lithuanian, Sardinian, Saterfriesisch, Scots, Scottish Gaelic, SerboCroatian, Sicilian, Silesian, Slovak, Slovenian, Somali, Spanish, Sranan Tongo, Sundanese, Swahili, Swati, Swedish, Tagalog, Tahitian, Tarantino Sicilian, Tatar, Tetum, Tok Pisin, Tonga (Tonga Islands), Tosk Albanian, Tsonga, Tswana, Turkish, Turkmen, Uighur, Upper Sorbian, Uzbek, Venda, Venetian, Vietnamese, Vlaams, Vlax Romani, Volap ¨uk, V o˜ro, Walloon, Waray, Welsh, Western Frisian, Wolof, Yoruba, Zeeuws, Zulu Cyrillic Abkhazian, Bashkir, Belarusian, Bulgarian, Chuvash, Erzya, Kazakh, Kirghiz, Macedonian, Moksha, Moldovan, Mongolian, Old Belarusian, Ossetian, Russian, Serbian, Tajik, Udmurt, Ukrainian, Yakut Arabic Arabic, Egyptian Arabic, Gilaki, Mazanderani, Pushto, Uighur, Urdu Persian, Devanagari Bihari, Hindi, Marathi, Nepali, Newari, Sanskrit Other Amharic, Armenian, Assamese, Bengali, Bishnupriya, Burmese, Central Khmer, Chinese, Classical Chinese, Dhivehi, Gan Chinese, Georgian, Gothic, Gujarati, Hebrew, Japanese, Kannada, Lao, Malayalam, Modern Greek, Official Aramaic, Panjabi, Sinhala, Tamil, Telugu, Thai, Tibetan, Wu Chinese, Yiddish, Yue Chinese 978