cvpr cvpr2013 cvpr2013-72 cvpr2013-72-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Xiaodi Hou, Alan Yuille, Christof Koch
Abstract: For an ill-posed problem like boundary detection, human labeled datasets play a critical role. Compared with the active research on finding a better boundary detector to refresh the performance record, there is surprisingly little discussion on the boundary detection benchmark itself. The goal of this paper is to identify the potential pitfalls of today’s most popular boundary benchmark, BSDS 300. In the paper, we first introduce a psychophysical experiment to show that many of the “weak” boundary labels are unreliable and may contaminate the benchmark. Then we analyze the computation of f-measure and point out that the current benchmarking protocol encourages an algorithm to bias towards those problematic “weak” boundary labels. With this evidence, we focus on a new problem of detecting strong boundaries as one alternative. Finally, we assess the performances of 9 major algorithms on different ways of utilizing the dataset, suggesting new directions for improvements.
[1] S. Alpert, M. Galun, R. Basri, and A. Brandt. Image segmentation by probabilistic bottom-up aggregation and cue integration. In CVPR 2007. IEEE Conference on, pages 1–8. IEEE, 2007. 1
[2] P. Arbelaez. Boundary extraction in natural images using ultrametric contour maps. In CVPR Workshop, 2006. IEEE Conference on, pages 182–182. IEEE, 2006. 3
[3] P. Arbel ´aez, B. Hariharan, C. Gu, S. Gupta, L. Bourdev, and J. Malik. Semantic segmentation using regions and parts. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 3378–3385. IEEE, 2012. 6
[4] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour detection and hierarchical image segmentation. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 33(5):898–916, 2011. 3, 7
[5] E. Borenstein and S. Ullman. Class-specific, top-down segmentation. Computer Vision–ECCV 2002, pages 639–641, 2002. 1
[6] T. Cour, F. Benezit, and J. Shi. Spectral segmentation with multiscale graph decomposition. In CVPR, 2005. IEEE Conference on, volume 2, pages 1124–1 13 1. IEEE, 2005. 6
[7] P. Dollar, Z. Tu, and S. Belongie. Supervised learning of
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16] edges and object boundaries. In CVPR, 2006 IEEE Conference on, volume 2, pages 1964–1971. IEEE, 2006. 3 M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html. 1 P. Felzenszwalb and D. McAllester. A min-cover approach for finding salient curves. In CVPR Workshop, 2006. IEEE Conference on, pages 185–185. IEEE, 2006. 3 V. Ferrari, T. Tuytelaars, and L. Van Gool. Object detection by contour segment networks. Computer Vision–ECCV 2006, pages 14–28, 2006. 6 A. Goldberg and R. Kennedy. An efficient cost scaling algorithm for the assignment problem. Mathematical Programming, 71(2): 153–177, 1995. 2 R. Kennedy, J. Gallier, and J. Shi. Contour cut: identifying salient contours in images by solving a hermitian eigenvalue problem. In CVPR, 2011. IEEE Conference on, pages 2065– 2072. IEEE, 2011. 3 I. Kokkinos. Boundary detection using f-measure-, filterand feature-(f 3) boost. Computer Vision–ECCV 2010, pages 650–663, 2010. 3 T. Leung and J. Malik. Contour continuity in region based image segmentation. Computer Vision–ECCV 1998, pages 544–559, 1998. 6 D. Martin, C. Fowlkes, and J. Malik. Learning to detect natural image boundaries using local brightness, color, and texture cues. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 26(5):530–549, 2004. 2, 3 D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25] human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Computer Vision–ICCV 2001. IEEE International Conference on, volume 2, pages 416–423. IEEE, 2001 . 1, 2, 3 D. Martin, J. Malik, and D. Patterson. An Empirical Approach to Grouping and Segmentaqtion. Computer Science Division, University of California, 2003. 2 V. Raykar, S. Yu, L. Zhao, A. Jerebko, C. Florin, G. Valadez, L. Bogoni, and L. Moy. Supervised learning from multiple experts: Whom to trust when everyone lies a bit. In ICML, 2009. ACM Conference on, pages 889–896. ACM, 2009. 3 X. Ren. Multi-scale improves boundary detection in natural images. Computer Vision–ECCV 2008, pages 533–545, 2008. 3 X. Ren and L. Bo. Discriminatively trained sparse code gradients for contour detection. Advances in Neural Information Processing Systems, 25, 2012. 2, 3 X. Ren, C. Fowlkes, and J. Malik. Figure/ground assignment in natural images. Computer Vision–ECCV2006, pages 614– 627, 2006. 6 A. Sorokin and D. Forsyth. Utility data annotation with amazon mechanical turk. In CVPR Workshops, 2008. IEEE Conference on, pages 1–8. IEEE, 2008. 3 A. Torralba and A. Efros. Unbiased look at dataset bias. In CVPR, 2011. IEEE Conference on, pages 1521–1528. IEEE, 2011. 3 S. Vittayakorn and J. Hays. Quality assessment for crowdsourced object annotations. In Proceedings of the British machine vision conference, pages 109–1, 2011. 3 S. Wang, T. Kubota, and J. Siskind. Salient boundary de- tection using ratio contour. Advances in Neural Information Processing Systems, 16, 2003. 3
[26] P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In In Proc. of NIPS, pages 2424–2432, 2010. 3
[27] Q. Zhu, G. Song, and J. Shi. Untangling cycles for contour grouping. In Computer Vision–ICCV2007. IEEE Conference on, pages 1–8. IEEE, 2007. 3 222111223088