jmlr jmlr2012 jmlr2012-96 jmlr2012-96-reference knowledge-graph by maker-knowledge-mining

96 jmlr-2012-Refinement of Operator-valued Reproducing Kernels


Source: pdf

Author: Haizhang Zhang, Yuesheng Xu, Qinghui Zhang

Abstract: This paper studies the construction of a refinement kernel for a given operator-valued reproducing kernel such that the vector-valued reproducing kernel Hilbert space of the refinement kernel contains that of the given kernel as a subspace. The study is motivated from the need of updating the current operator-valued reproducing kernel in multi-task learning when underfitting or overfitting occurs. Numerical simulations confirm that the established refinement kernel method is able to meet this need. Various characterizations are provided based on feature maps and vector-valued integral representations of operator-valued reproducing kernels. Concrete examples of refining translation invariant and finite Hilbert-Schmidt operator-valued reproducing kernels are provided. Other examples include refinement of Hessian of scalar-valued translation-invariant kernels and transformation kernels. Existence and properties of operator-valued reproducing kernels preserved during the refinement process are also investigated. Keywords: vector-valued reproducing kernel Hilbert spaces, operator-valued reproducing kernels, refinement, embedding, translation invariant kernels, Hessian of Gaussian kernels, Hilbert-Schmidt kernels, numerical experiments


reference text

N. Aronszajn. Theory of reproducing kernels. Trans. Amer. Math. Soc., 68:337–404, 1950. S. K. Berberian. Notes on Spectral Theory. Van Nostrand, New York, 1966. M. S. Birman and M. Z. Solomjak. Spectral Theory of Self-Adjoint Operators in Hilbert Space. D. Reidel Publishing Company, Dordrecht, Holland, 1987. S. Bochner. Lectures on Fourier Integrals with an Author’s Supplement on Monotonic Functions, Stieltjes Integrals, and Harmonic Analysis. Annals of Mathematics Studies 42, Princeton University Press, New Jersey, 1959. J. Burbea and P. Masani. Banach and Hilbert Spaces of Vector-valued Functions. Pitman Research Notes in Mathematics 90, Boston, MA, 1984. A. Caponnetto, C. A. Micchelli, M. Pontil and Y. Ying. Universal multi-task kernels. Journal of Machine Learning Research, 9:1615–1646, 2008. C. Carmeli, E. De Vito and A. Toigo. Vector valued reproducing kernel Hilbert spaces of integrable functions and Mercer theorem. Anal. Appl., 4:377–408, 2006. C. Carmeli, E. De Vito, A. Toigo and V. Umanita. Vector valued reproducing kernel Hilbert spaces and universality. Anal. Appl., 8:19–61, 2010. J. B. Conway. A Course in Functional Analysis. 2nd Edition, Springer-Verlag, New York, 1990. F. Cucker and S. Smale. On the mathematical foundations of learning. Bull. Amer. Math. Soc., 39:1– 49, 2002. F. Cucker and D. X. Zhou. Learning Theory: An Approximation Theory Viewpoint. Cambridge University Press, Cambridge, 2007. I. Daubechies. Ten Lectures on Wavelets. CBMS-NSF Regional Conference Series in Applied Mathematics 61, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 1992. J. Diestel and J.J. Uhl, Jr. Vector Measures. American Mathematical Society, Providence, 1977. T. Evgeniou, C. A. Micchelli and M. Pontil. Learning multiple tasks with kernel methods. Journal of Machine Learning Research, 6:615–637, 2005. T. Evgeniou, M. Pontil and T. Poggio. Regularization networks and support vector machines. Adv. Comput. Math., 13:1–50, 2000. 135 Z HANG , X U AND Z HANG P. A. Fillmore. Notes on Operator Theory. Van Nostrand Company, New York, 1970. R. A. Horn and C. B. Johnson. Topics in Matrix Analysis. Cambridge University Press, Cambridge, 1991. S. Lowitzsh. Approximation and interpolation employing divergence-free radial basis functions with applications. Ph.D. Thesis, Texas A&M; University, College Station, Texas, 2003. S. Mallat. Multiresolution approximations and wavelet orthonormal bases of L2 (R). Trans. Amer. Math. Soc., 315:69–87, 1989. C. A. Micchelli and M. Pontil. On learning vector-valued functions. Neural Comput., 17:177–204, 2005. C. A. Micchelli, Y. Xu and H. Zhang. Universal kernels. Journal of Machine Learning Research, 7:2651–2667, 2006. S. Mukherjee and Q. Wu. Estimation of gradients and coordinate covariation in classification. Journal of Machine Learning Research, 7:2481-2514, 2006. S. Mukherjee and D. X. Zhou. Learning coordinate covariances via gradients. Journal of Machine Learning Research, 7:519–549, 2006. G. B. Pedrick. Theory of reproducing kernels for Hilbert spaces of vector valued functions. Technical Report 19, University of Kansas, 1957. W. Rudin. Real and Complex Analysis. 3rd Edition, McGraw-Hill, New York, 1987. B. Sch¨ lkopf and A. J. Smola. Learning with Kernels: Support Vector Machines, Regularization, o Optimization, and Beyond. MIT Press, Cambridge, Mass, 2002. J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge, 2004. V. N. Vapnik. Statistical Learning Theory. Wiley, New York, 1998. H. Wendland. Divergence-free kernel methods for approximating the Stokes problem. SIAM J. Numer. Anal., 47:3158–3179, 2009. Y. Xu and H. Zhang. Refinable kernels. Journal of Machine Learning Research, 8:2083–2120, 2007. Y. Xu and H. Zhang. Refinement of reproducing kernels. Journal of Machine Learning Research, 10:107–140, 2009. Y. Ying and C. Campbell. Learning coordinate gradients with multi-task kernels. In COLT, 2008. H. Zhang, Y. Xu, and J. Zhang. Reproducing kernel Banach spaces for machine learning. Journal of Machine Learning Research, 10:2741–2775, 2009. H. Zhang and L. Zhao. On the inclusion relation of reproducing kernel Hilbert spaces. Anal. Appl., accetped subject to minor revision, arXiv:1106.4075, 2011. 136