nips nips2012 nips2012-32 nips2012-32-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Christoph Sawade, Niels Landwehr, Tobias Scheffer
Abstract: We address the problem of comparing the risks of two given predictive models—for instance, a baseline model and a challenger—as confidently as possible on a fixed labeling budget. This problem occurs whenever models cannot be compared on held-out training data, possibly because the training data are unavailable or do not reflect the desired test distribution. In this case, new test instances have to be drawn and labeled at a cost. We devise an active comparison method that selects instances according to an instrumental sampling distribution. We derive the sampling distribution that maximizes the power of a statistical test applied to the observed empirical risks, and thereby minimizes the likelihood of choosing the inferior model. Empirically, we investigate model selection problems on several classification and regression tasks and study the accuracy of the resulting p-values. 1
[1] M. Balcan, A. Beygelzimer, and J. Langford. Agnostic active learning. In Proceedings of the 23rd International Conference on Machine Learning, 2006.
[2] A. Beygelzimer, S. Dasgupta, and J. Langford. Importance weighted active learning. In Proceedings of the 26th International Conference on Machine Learning, 2009.
[3] J. Geweke. Bayesian inference in econometric models using monte carlo integration. Econometrica, 57(6):1317–1339, 1989.
[4] J. S. Liu. Monte Carlo Strategies in Scientific Computing. Springer, 2001.
[5] O. Madani, D. J. Lizotte, and R. Greiner. Active model selection. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, 2004.
[6] O. Maron and A. W. Moore. Hoeffding races: Accelerating model selection search for classification and function approximation. In Proceedings of the 6th Annual Conference on Neural Information Processing Systems, 1993.
[7] Carl Edward Rasmussen and Christopher Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
[8] C. Sawade, N. Landwehr, S. Bickel, and T. Scheffer. Active risk estimation. In Proceedings of the 27th International Conference on Machine Learning, 2010.
[9] C. Sawade, N. Landwehr, and T. Scheffer. Active estimation of f-measures. In Proceedings of the 23rd Annual Conference on Neural Information Processing Systems, 2010.
[10] T. Scheffer and S. Wrobel. Finding the most interesting patterns in a database quickly by using sequential sampling. Journal of Machine Learning Research, 3:833–862, 2003.
[11] D. Sheskin. Handbook of Parametric and Nonparametric Statistical Procedures. Chapman & Hall, 2004.
[12] L. Wasserman. All of Statistics: a Concise Course in Statistical Inference. Springer, 2004. 9