emnlp emnlp2012 emnlp2012-60 emnlp2012-60-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Aciel Eshky ; Ben Allison ; Mark Steedman
Abstract: User simulation is frequently used to train statistical dialog managers for task-oriented domains. At present, goal-driven simulators (those that have a persistent notion of what they wish to achieve in the dialog) require some task-specific engineering, making them impossible to evaluate intrinsically. Instead, they have been evaluated extrinsically by means of the dialog managers they are intended to train, leading to circularity of argument. In this paper, we propose the first fully generative goal-driven simulator that is fully induced from data, without hand-crafting or goal annotation. Our goals are latent, and take the form of topics in a topic model, clustering together semantically equivalent and phonetically confusable strings, implicitly modelling synonymy and speech recognition noise. We evaluate on two standard dialog resources, the Communicator and Let’s Go datasets, and demonstrate that our model has substantially better fit to held out data than competing approaches. We also show that features derived from our model allow significantly greater improvement over a baseline at distinguishing real from randomly permuted dialogs.
Hua Ai and Diane J. Litman. 2008. Assessing dialog system user simulation evaluation measures using human judges. In Proceedings of ACL-08: HLT. Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag New York, Inc. Alan W. Black and Maxine Eskenazi. 2009. The Spoken Dialogue Challenge. In Proceedings of SIGDIAL 2009, SIGDIAL ’09. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993–1022, March. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27: 1– 27:27. Wieland Eckert, Esther Levin, and Roberto Pieraccini. 1997. User modeling for spoken dialogue system evaluation. In Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding. M. Ga˘ si´ c, F. Jurcicek, B. Thomson, K. Yu, and S. Young. 2011. On-line policy optimisation of spoken dia- logue via live interaction with human subjects. In Automatic Speech Recognition and Understanding, 2011 IEEE Workshop on, Hawaii, December. Kallirroi Georgila, James Henderson, and Oliver Lemon. 2005a. Learning user simulations for information state update dialogue systems. In Proceedings InterSpeech 2005. Kallirroi Georgila, Oliver Lemon, and James Henderson. 2005b. Automatic annotation of communicator dialogue data for learning dialogue strategies and user simulations. In Proceedings Ninth Workshop on the Semantics and Pragmatics of Dialogue. Kallirroi Georgila, James Henderson, and Oliver Lemon. 2006. User Simulation for Spoken Dialogue Systems: systems Learning and Evaluation. In Proceedings InterSpeech 2006. Thomas L. Griffiths and Mark Steyvers. 2004. Finding scientific topics. PNAS, 101(suppl. 1):5228–5235. SrinivasanJanarthanam and OliverLemon. 2009. Atwotier user simulation model for reinforcement learning of adaptive referring expression generation policies. In Proceedings of SIGDIAL 2009, SIGDIAL ’09, pages 120–123. Sangkeun Jung, Cheongjae Lee, Kyungduk Kim, Minwoo Jeong, and Gary Geunbae Lee. 2009. Datadriven user simulation for automated evaluation of spoken dialog systems. Computer Speech & Language, 23(4):479–509. Simon Keizer, Milica Gaˇ si´ c, Filip Jur cˇ´ ı cˇek, Fran ¸cois Mairesse, Blaise Thomson, Kai Yu, and Steve Young. 2010. Parameter estimation for agenda-based user simulation. In Proceedings of SIGDIAL 2010. Esther Levin and Roberto Pieraccini. 2000. A stochastic model of human-machine interaction for learning dialog strategies. In IEEE Transactions on Speech and Audio Processing. E. Levin, S. Narayanan, R. Pieraccini, K. Biatov, E. Bocchieri, G. Di Fabbrizio, W. Eckert, S. Lee, A. Pokrovsky, M. Rahim, P. Ruscitti, and M. Walker. 2000. The AT&T-DARPA; communicator mixedinitiative spoken dialog system. In In ICSLP. Radford Neal. 1991 . Bayesian Mixture Modeling by Monte Carlo Simulation. Technical report, University of Toronto. Olivier Pietquin. 2004. A Framework for Unsupervised Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence. Richard S. Sutton and Andrew G. Barto. 1998. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA. Jason D. Williams. 2008. Evaluating user simulations with the Cramer-von Mises divergence. Speech Communication, 50(10):829–846, October. Learning of Dialogue Strategies. Ph.D. thesis, Facult´ e Polytechnique de Mons, TCTS Lab (Belgique), apr. Jost Schatzmann, Kallirroi Georgila, and Steve Young. 2005. Quantitative evaluation of user simulation techniques for spoken dialogue systems. In Proceeings of 6th SIGDIAL Workshop. Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007a. Agenda-based user simulation for bootstrapping a POMDP dialogue system. In HLT-NAACL (Short Papers), NAACL-Short ’07. Jost Schatzmann, Blaise Thomson, and Steve Young. 2007b. Statistical User Simulation with a Hidden Agenda. In Proceedings 8th SIDdial Workshop on Discourse and Dialogue, September. Konrad Scheffler and Steve Young. 2002. Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning. In Proceedings of HLT 2002. Satinder P. Singh, Michael J. Kearns, Diane J. Litman, and Marilyn A. Walker. 2000. Empirical evaluation of a reinforcement learning spoken dialogue system. In 81