nips nips2012 nips2012-225 nips2012-225-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Binbin Lin, Sen Yang, Chiyuan Zhang, Jieping Ye, Xiaofei He
Abstract: Multi-task learning (MTL) aims to improve generalization performance by learning multiple related tasks simultaneously and identifying the shared information among tasks. Most of existing MTL methods focus on learning linear models under the supervised setting. We propose a novel semi-supervised and nonlinear approach for MTL using vector fields. A vector field is a smooth mapping from the manifold to the tangent spaces which can be viewed as a directional derivative of functions on the manifold. We argue that vector fields provide a natural way to exploit the geometric structure of data as well as the shared differential structure of tasks, both of which are crucial for semi-supervised multi-task learning. In this paper, we develop multi-task vector field learning (MTVFL) which learns the predictor functions and the vector fields simultaneously. MTVFL has the following key properties. (1) The vector fields MTVFL learns are close to the gradient fields of the predictor functions. (2) Within each task, the vector field is required to be as parallel as possible which is expected to span a low dimensional subspace. (3) The vector fields from all tasks share a low dimensional subspace. We formalize our idea in a regularization framework and also provide a convex relaxation method to solve the original non-convex problem. The experimental results on synthetic and real data demonstrate the effectiveness of our proposed approach. 1
[1] A. Agarwal, H. D. III, and S. Gerber. Learning multiple tasks using manifold regularization. In Advances in Neural Information Processing Systems 23, pages 46–54. 2010.
[2] R. K. Ando and T. Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817–1853, 2005.
[3] A. Argyriou, C. A. Micchelli, M. Pontil, and Y. Ying. A spectral regularization framework for multi-task structure learning. In Advances in Neural Information Processing Systems 20, pages 25–32. 2008.
[4] B. Bakker and T. Heskes. Task clustering and gating for bayesian multitask learning. Journal of Machine Learning Research, 4:83–99, 2003.
[5] M. Belkin, P. Niyogi, and V. Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 7:2399–2434, December 2006.
[6] S. Ben-David, J. Gehrke, and R. Schuller. A theoretical framework for learning from a pool of disparate data sources. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 443–449, 2002.
[7] S. Ben-David and R. Schuller. Exploiting task relatedness for mulitple task learning. In Conference on Learning Theory, pages 567–580, 2003.
[8] S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
[9] A. Carlson, J. Betteridge, R. C. Wang, E. R. Hruschka, Jr., and T. M. Mitchell. Coupled semisupervised learning for information extraction. In Proceedings of the third ACM international conference on Web search and data mining, pages 101–110, 2010.
[10] O. Chapelle, B. Sch¨ lkopf, and A. Zien, editors. Semi-Supervised Learning. MIT Press, 2006. o
[11] A. Defant and K. Floret. Tensor Norms and Operator Ideals. North-Holland Mathematics Studies, North-Holland, Amsterdam, 1993.
[12] T. Evgeniou, C. A. Micchelli, and M. Pontil. Learning multiple tasks with kernel methods. Journal of Machine Learning Research, 6:615–637, 2005.
[13] G. H. Golub and C. F. V. Loan. Matrix computations. Johns Hopkins University Press, 3rd edition, 1996.
[14] L. Jacob, F. Bach, and J.-P. Vert. Clustered multi-task learning: A convex formulation. In Advances in Neural Information Processing Systems 21, pages 745–752. 2009.
[15] J. Lafferty and L. Wasserman. Statistical analysis of semi-supervised regression. In Advances in Neural Information Processing Systems 20, pages 801–808, 2007.
[16] J. M. Lee. Introduction to Smooth Manifolds. Springer Verlag, New York, 2nd edition, 2003.
[17] B. Lin, C. Zhang, and X. He. Semi-supervised regression via parallel field regularization. In Advances in Neural Information Processing Systems 24, pages 433–441. 2011.
[18] Q. Liu, X. Liao, and L. Carin. Semi-supervised multitask learning. In Advances in Neural Information Processing Systems 20, pages 937–944. 2008.
[19] F. Wang, X. Wang, and T. Li. Semi-supervised multi-task learning with task regularizations. In Proceedings of the 2009 Ninth IEEE International Conference on Data Mining, pages 562– 568. IEEE Computer Society, 2009.
[20] Y. Xue, X. Liao, L. Carin, and B. Krishnapuram. Multi-task learning for classification with dirichlet process priors. Journal of Machine Learning Research, 8:35–63, 2007. 9