iccv iccv2013 iccv2013-43 iccv2013-43-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Chengjiang Long, Gang Hua, Ashish Kapoor
Abstract: We present a noise resilient probabilistic model for active learning of a Gaussian process classifier from crowds, i.e., a set of noisy labelers. It explicitly models both the overall label noises and the expertise level of each individual labeler in two levels of flip models. Expectation propagation is adopted for efficient approximate Bayesian inference of our probabilistic model for classification, based on which, a generalized EM algorithm is derived to estimate both the global label noise and the expertise of each individual labeler. The probabilistic nature of our model immediately allows the adoption of the prediction entropy and estimated expertise for active selection of data sample to be labeled, and active selection of high quality labelers to label the data, respectively. We apply the proposed model for three visual recognition tasks, i.e, object category recognition, gender recognition, and multi-modal activity recognition, on three datasets with real crowd-sourced labels from Amazon Mechanical Turk. The experiments clearly demonstrated the efficacy of the proposed model.
[1] V. Ambati, S. Vogel, and J. Carbonell. Active learning and crowdsourcing for machine translation. In LREC, 2010. 1, 2
[2] R. U. Ashish Kapoor, Kristen Grauman and T. Darrell. Active learning with gaussian processes for object categorization. ICCV, 2007. 1, 6
[3] S. Chen, J. Zhang, G. Chen, and C. Zhang. What if the irresponsible teachers are dominating? a method of training on samples and
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21] clustering on teachers. In AAAI, 2010. 2 O. Dekel and O. Shamir. Good learners for evil teachers. In ICML, 2009. 2 O. Dekel and O. Shamir. Vox populi: Collecting high-quality labels from a crowd. In COLT, 2009. 2 J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248 –255, June 2009. 1, 4 P. Donmez, J. Carbonell, and J. Schneider. A probabilistic framework to learn from multiple annotators with time-varying accuracy. In SDM, 2010. 2 P. Donmez, J. G. Carbonell, and J. Schneider. Efficiently learning the accuracy of labeling sources for selective sampling. In SIGKDD, 2009. 2 S. Ebert, M. Fritz, and B. Schiele. Ralf: A reinforced active learning formulation for object class recognition. In CVPR, 2012. 1 M. Gibbs and D. Mackay. Variational gaussian process classifiers. T-NN, 2000. 2 A. Kapoor, G. Hua, A. Akbarzadeh, and S. Baker. Which faces to tag: Adding prior constraints into active learning. ICCV, 2009. 1 H.-C. Kim and Z. Ghahramani. Outlier robust gaussian process classification. In SSPR/SPR, pages 896–905, 2008. 2, 3, 6 A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. 1 Y. Lin, F. Lv, S. Zhu, M. Yang, T. Cour, K. Yu, L. Cao, and T. Huang. Large-scale image classification: Fast feature extraction and svm training. In CVPR, 2011. 1, 4 C. Loy, T. Hospedales, T. Xiang, and S. Gong. Stream-based joint exploration-exploitation active learning. In CVPR, 2012. 1 T. Minka. A family of algorithms for approximate Bayesian inference. Ph.d. thesis, MIT, 2001. 1, 2, 3 R. M. Neal. Monte carlo implementation of gaussian process models for bayesian regression and classification. Technical Report CRGTR972, University of Toronto, 1997. 2 M. Opper and O. Winther. Gaussian processes for classification: Mean-field algorithms. NC, 2000. 2 V. C. Raykar and S. Yu. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. JMLR, 2012. 2 V. C. Raykar, S. Yu, L. H. Zhao, A. Jerebko, C. Florin, G. H. Valadez, L. Bogoni, and L. Moy. Supervised learning from multiple experts: whom to trust when everyone lies a bit. In ICML, 2009. 2 J. Sanchez and F. Perronnin. High-dimensional signature compres-
[22]
[23]
[24]
[25]
[26]
[27]
[28]
[29]
[30] sion for large-scale image classification. In CVPR, 2011. 1 E. H. Spriggs, F. D. L. Torre, and M. Hebert. Temporal segmentation and activity classification from first-person sensing. In CVPRW, 2009. 5 S. Vijayanarasimhan and K. Grauman. Large-scale live active learning: Training object detectors with crawled data and crowds. In CVPR, 2011. 1, 2 P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In NIPS, 2010. 2 P. Welinder and P. Perona. Online crowdsourcing: Rating annotators and obtaining cost-effective labels. In CVPRW, 2010. 2 C. Williams and D. Barber. Bayesian classification with gaussian processes. T-PAMI, 20(12): 1342 –1351, dec 1998. 2, 3 Y. Yan, R. Rosales, G. Fung, and J. Dy. Active learning from multiple knowledge sources. In AISTATS, 2012. 2, 6 Y. Yan, R. Rosales, G. Fung, and J. G. Dy. Active learning from crowds. ICML, pages 1161–1 168, 2011. 2, 6 L. Zhao, G. Sukthankar, and R. Sukthankar. Incremental relabeling for active learning with noisy crowdsourced annotations. In SocialCom, 2011. 2, 5 C. Zhu, R. H. Byrd, P. Lu, and J. Nocedal. Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization. ACM Trans. Math. Softw., 1997. 4 333000000777