jmlr jmlr2006 jmlr2006-12 jmlr2006-12-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Hema Raghavan, Omid Madani, Rosie Jones
Abstract: We extend the traditional active learning framework to include feedback on features in addition to labeling instances, and we execute a careful study of the effects of feature selection and human feedback on features in the setting of text categorization. Our experiments on a variety of categorization tasks indicate that there is significant potential in improving classifier performance by feature re-weighting, beyond that achieved via membership queries alone (traditional active learning) if we have access to an oracle that can point to the important (most predictive) features. Our experiments on human subjects indicate that human feedback on feature relevance can identify a sufficient proportion of the most relevant features (over 50% in our experiments). We find that on average, labeling a feature takes much less time than labeling a document. We devise an algorithm that interleaves labeling features and documents which significantly accelerates standard active learning in our simulation experiments. Feature feedback can complement traditional active learning in applications such as news filtering, e-mail classification, and personalization, where the human teacher can have significant knowledge on the relevance of features. Keywords: active learning, feature selection, relevance feedback, term feedback, text classification
J. Allan. Topic detection and tracking. Kluwer Academic Publishers, 2002. 1683 R AGHAVAN , M ADANI AND J ONES D. Angluin. Computational learning theory: survey and selected bibliography. In Proceedings of the 24th Annual ACM Symposium on the Theory Computation, pages 351–369, 1992. P. Anick. Using terminological feedback for web search refinement: a log-based study. In Proceedings of SIGIR ’03: The 26th annual international ACM SIGIR conference on Research and development in information retrieval, pages 88–95, 2003. Y. Baram, R. El-Yaniv, and K. Luz. Online choice of active learning algorithms. In Proceedings of ICML 03: The 20th International Conference on Machine Learning, pages 19–26, 2003. E. B. Baum and K. Lang. Query learning can work poorly when human oracle is used. In International Joint Conference in Neural Netwroks, 1992. P. Beineke, T. Hastie, and S. Vaithyanathan. The sentimental factor: Improving review classification via human-provided information. In Proceedings of ACL 04: The 42nd Meeting of the Association for Computational Linguistics, Main Volume, pages 263–270, 2004. N. J. Belkin, C. Cool, D. Kelly, S. J. Lin, S. Y. Park, J. Perez-Carballo, and C. Sikora. Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval. Information Processing and Management, 37(3):403–434, 2001. J. Brank, M. Grobelnik, N. Milic-Frayling, and D. Mladenic. Feature selection using linear support vector machines. Technical report, Microsoft Research, 2002. C. C. Chang and C. J. Lin. Libsvm: a library for support vector machines. Available electronically at http://www.csie.ntu.edu.tw/cjlin/libsvm. J. Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:27–46, 1960. D. A. Cohn, L. Atlas, and R. E. Ladner. Improving generalization with active learning. Machine Learning, 15(2):201–221, 1994. W. B. Croft and R. Das. Experiments with query acquisition and use in document retrieval systems. In Proceedings of SIGIR ’90: The 13th annual international ACM SIGIR conference on Research and development in information retrieval, pages 349–368, 1990. G. DeJong and R. Mooney. Explanation-based generalization: an alternative view. Machine Learning, 1(2):145–176, 1986. F. Diaz and J. Allan. When less is more: Relevance feedback falls short and term expansion succeeds at HARD 2005. In Text REtrieval Conference (TREC 2005) Notebook. Dept. of Commerce, NIST, 2005. C. Domeniconi and D. Gunopulos. Incremental support vector machine construction. In Proceedings of ICDM 01:2001 IEEE International Conference on Data Mining, pages 589–592, 2001. S. Godbole, A. Harpale, S. Sarawagi, and S. Chakrabarti. Document classification through interactive supervision of document and term labels. In Proceedings of PKDD 04: The 8th European Conference on Principles and Practice of Knowledge Discovery in Databases, pages 185–196, 2004. 1684 ACTIVE L EARNING WITH F EEDBACK ON F EATURES AND I NSTANCES G. F. Hughes. On the mean accuracy of statistical pattern recognizers. IEEE Transactions on Information Theory, 14:55–63, 1968. T. Joachims. Text categorization with support vector machines: learning with many relevant features. In ECML 98: The 10th European Conference on Machine Learning, pages 137–142, 1998. T. Joachims. Transductive inference for text classification using support vector machines. In ICML ’99: Proceedings of the Sixteenth International Conference on Machine Learning, pages 200– 209, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1-55860-612-2. R. Jones. Learning to extract entities from labeled and unlabeled text. PhD thesis, Carnegie Mellon University, Pittsburgh, USA, 2005. G. R. Landis and G. G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33:159–174, 1977. K. Lang. Newsweeder: Learning to filter netnews. In Proceedings of ICML 95: The 12th International Conference on Machine Learning, pages 331–339, 1995. D. D. Lewis. Naive (Bayes) at forty: The independence assumption in information retrieval. In Proceedings of ECML 98: 10th European Conference on Machine Learning, pages 4–15, 1998. D. D. Lewis and J. Catlett. Heterogeneous uncertainty sampling for supervised learning. In Proceedings of ICML 94: The 11th International Conference on Machine Learning, pages 148–156, 1994. N. Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning, 2(4):285–318, 1988. D. J. Lizotte, O. Madani, and R. Greiner. Budgeted learning of naive-bayes classifiers. In Proceedings of UIA 03: The 19th Conference on Uncertainty in AI (UAI), 2003. A. K. McCallum. Bow: A toolkit for statistical language modeling, text retrieval, classification and clustering. Available electronically at http://www.cs.cmu.edu/∼mccallum/bow, 1996. T. Mitchell, R. Keller, and S. Kedar-Cabelli. Explanation-based generalization: A unifying view. Machine Learning, 1:47–80, 1986. M. J. Pazzani and D. Kibler. The role of prior knowledge in inductive learning. Machine Learning, 9 , 54-97., 9, 1992. M. Porter. An algorithm for suffix stripping. Automated Library and Information Systems, 14(3): 130–137, 1980. H. Raghavan, O. Madani, and R. Jones. Interactive feature selection. In Proceedings of IJCAI 05: The 19th International Joint Conference on Artificial Intelligence, pages 841–846, 2005. T. G. Rose, M. Stevenson, and M. Whitehead. The Reuters Corpus Vol. 1 - from yesterday’s news to tomorrow’s language resources. In Proceedings of International Conference on Language Resources and Evaluation, 2002. 1685 R AGHAVAN , M ADANI AND J ONES G. Salton. Automatic information organization and retrieval. McGraw Hill, 1968. R. Schapire, M. Rochery, M. Rahim, and N. Gupta. Incorporating prior knowledge into boosting. In Proceedings of ICML 02: The 19th International Conference on Machine Learning, 2002. G. Schohn and D. Cohn. Less is more: Active learning with support vector machines. In Proceedings of ICML 00: The 17th International Conference on Machine Learning, pages 839–846, 2000. F. Sebastiani. Machine learning in automated text categorization. ACM Computing Surveys, 34(1): 1–47, 2002. S. Tong and D. Koller. Support vector machine active learning with applications to text classification. Journal of Machine Learning Research, 2:45–66, 2002. ISSN 1533-7928. E. M. Voorhees and L. P. Buckland, editors. Text REtrieval Conference (TREC 2005) Notebook, 2005. Dept of Commerce, NIST. X. Wu and R. Srihari. Incorporating prior knowledge with weighted margin support vector machines. In Proceedings of KDD 04: Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 326–333, 2004. X. Zhu. Semi-supervised learning literature survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madison, 2005. 1686