nips nips2008 nips2008-159 nips2008-159-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Patrice Bertail, Stéphan J. Clémençcon, Nicolas Vayatis
Abstract: This paper is devoted to thoroughly investigating how to bootstrap the ROC curve, a widely used visual tool for evaluating the accuracy of test/scoring statistics in the bipartite setup. The issue of confidence bands for the ROC curve is considered and a resampling procedure based on a smooth version of the empirical distribution called the ”smoothed bootstrap” is introduced. Theoretical arguments and simulation results are presented to show that the ”smoothed bootstrap” is preferable to a ”naive” bootstrap in order to construct accurate confidence bands. 1
[1] P. Bertail, S. Cl´ mencon, and N. Vayatis. On constructing accurate confidence bands for ROC curves e ¸ through smooth resampling, http://hal.archives-ouvertes.fr/hal-00335232/fr/. Technical report, 2008.
[2] S. Cl´ mencon, G. Lugosi, and N. Vayatis. Ranking and scoring using empirical risk minimization. Proe ¸ ceedings of COLT 2005, Eds P. Auer and R. Meir, LNAI 3559, Springer, 2005.
[3] S. Cl´ mencon and N. Vayatis. Ranking the best instances. Journal of Machine Learning Research, e ¸ 5:197–227, 2007.
[4] W. Cohen, R. Schapire, and Y. Singer. Learning to order things. Journal of Artificial Intelligence Research, 10:243–270, 1999.
[5] M. Csorgo and P. Revesz. Strong approximations in probability and statistics. Academic Press, 1981.
[6] B. Efron. Bootstrap methods: another look at the jacknife. Annals of Statistics, 7:1–26, 1979.
[7] M. Falk and R. Reiss. Weak convergence of smoothed and nonsmoothed bootstrap quantile estimates. Annals of Probability, 17:362–371, 1989.
[8] T. Fawcett. ROC graphs: Notes and practical considerations for data mining researchers. Technical Report HPL 2003-4), 5:197–227, 2003.
[9] P. Flach. The geometry of roc space: understanding machine learning metrics through roc isometrics. In T. Fawcett and N. Mishra, editors, Proc. 20th International Conference on Machine Learning (ICML’03), AAAI Press, 86:194–201, 2003.
[10] Y. Freund, R. Iyer, R. Schapire, and Y. Singer. An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4:933–969, 2003.
[11] P. Ghosal and J. Gu. Bayesian ROC curve estimation under binormality using a partial likelihood based on ranks. Submitted for publication, 2007.
[12] P. Ghosal and J. Gu. Strong approximations for resample quantile process and application to ROC methodology. Submitted for publication, 2007.
[13] A. Girling. ROC confidence bands: An empirical evaluation. Journal of the Royal Statistical Society, Series B, 62:367–382, 2000.
[14] D. Green and J. Swets. Signal detection theory and psychophysics. Wiley, NY, 1966.
[15] P. Hall. On the number of bootstrap simulations required to construct a confidence interval. Annals of Statistics, 14:1453–1462, 1986.
[16] P. Hall and R. Hyndman. Improved methods for bandwidth selection when estimating ROC curves. Statistics and Probability Letters, 64:181–189, 2003.
[17] P. Hall, R. Hyndman, and Y. Fan. Nonparametric confidence intervals for receiver operating characteristic curves. Biometrika, 91:743–750, 2004.
[18] F. Hsieh and B. Turnbull. Nonparametric and semi-parametric statistical estimation of the ROC curve. The Annals of Statistics, 24:25–40, 1996.
[19] S. Macskassy and F. Provost. Confidence bands for ROC curves: methods and an empirical study. In Proceedings of the first Workshop on ROC Analysis in AI (ROCAI-2004) at ECAI-2004, 2004.
[20] S. Macskassy, F. Provost, and S. Rosset. Bootstrapping the ROC curve: an empirical evaluation. In Proceedings of ICML-2005 Workshop on ROC Analysis in Machine Learning (ROCML-2005), 2005.
[21] S. Macskassy, F. Provost, and S. Rosset. ROC confidence bands: An empirical evaluation. In Proceedings of the 22nd International Conference on Machine Learning (ICML-2005), 2005.
[22] B. Silverman and G. Young. The bootstrap: to smooth or not to smooth? Biometrika, 74:469–479, 1987. 8