nips nips2012 nips2012-40 knowledge-graph by maker-knowledge-mining

40 nips-2012-Analyzing 3D Objects in Cluttered Images

Source: pdf

Author: Mohsen Hejrati, Deva Ramanan

Abstract: We present an approach to detecting and analyzing the 3D conﬁguration of objects in real-world images with heavy occlusion and clutter. We focus on the application of ﬁnding and analyzing cars. We do so with a two-stage model; the ﬁrst stage reasons about 2D shape and appearance variation due to within-class variation (station wagons look different than sedans) and changes in viewpoint. Rather than using a view-based model, we describe a compositional representation that models a large number of effective views and shapes using a small number of local view-based templates. We use this model to propose candidate detections and 2D estimates of shape. These estimates are then reﬁned by our second stage, using an explicit 3D model of shape and viewpoint. We use a morphable model to capture 3D within-class variation, and use a weak-perspective camera model to capture viewpoint. We learn all model parameters from 2D annotations. We demonstrate state-of-the-art accuracy for detection, viewpoint estimation, and 3D shape reconstruction on challenging images from the PASCAL VOC 2011 dataset. 1

Reference: text

Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 edu Abstract We present an approach to detecting and analyzing the 3D conﬁguration of objects in real-world images with heavy occlusion and clutter. [sent-5, score-0.331]

2 We do so with a two-stage model; the ﬁrst stage reasons about 2D shape and appearance variation due to within-class variation (station wagons look different than sedans) and changes in viewpoint. [sent-7, score-0.562]

3 We use a morphable model to capture 3D within-class variation, and use a weak-perspective camera model to capture viewpoint. [sent-11, score-0.381]

4 We demonstrate state-of-the-art accuracy for detection, viewpoint estimation, and 3D shape reconstruction on challenging images from the PASCAL VOC 2011 dataset. [sent-13, score-0.507]

5 1 Introduction Figure 1: We describe two-stage models for detecting and analyzing the 3D shape of objects in unconstrained images. [sent-14, score-0.35]

6 In the ﬁrst stage, our models reason about 2D appearance and shape using variants of deformable part models (DPMs). [sent-15, score-0.515]

7 We use global mixtures of trees with local mixtures of gradient-based part templates (top-left). [sent-16, score-0.836]

8 Global mixtures capture constraints on visibility and shape (headlights are only visible in certain views at certain locations), while local mixtures capture constraints on appearance (headlights look different in different views). [sent-17, score-1.449]

9 We feed this output to our second stage, which directly reasons about 3D shape and camera viewpoint. [sent-19, score-0.391]

10 We develop a model that detects cars, estimates camera viewpoint, and recovers 3D landmarks conﬁgurations and their visibility with state-of-the-art accuracy. [sent-31, score-0.751]

11 It does so by reasoning about appearance, 3D shape, and camera viewpoint through the use of 2D structured, relational classiﬁers and 3D geometric subspace models. [sent-32, score-0.475]

12 While deformable models and pictorial structures [10, 31, 11] are known to successfully model articulation, 3D viewpoint is still not well understood. [sent-33, score-0.315]

13 We introduce a two-stage approach that ﬁrst reasons about 2D shape and appearance variation, and then reasons explicitly about 3D shape and viewpoint given 2D correspondences from the ﬁrst stage. [sent-41, score-0.806]

14 2D shape and appearance: Our ﬁrst stage models 2D shape and appearance using a variant of deformable part models (DPMs) designed to produce reliable 2D landmark correspondences. [sent-43, score-1.195]

15 We use global mixtures of trees with local mixtures of “part” or landmark templates. [sent-45, score-1.147]

16 Global mixtures capture constraints on visibility and shape (headlights are only visible in certain views at certain locations), while local mixtures capture constraints on appearance (headlights look different in different views). [sent-46, score-1.449]

17 One salient aspect of our 2D model is that it reports 2D locations of all landmarks including occluded ones, each augmented with a visibility ﬂag. [sent-48, score-0.868]

18 3D shape and viewpoint: Our second layer processes the 2D output of our ﬁrst stage, incorporating global shape constraints arising from 3D shape variation and viewpoint. [sent-49, score-0.939]

19 To capture viewpoint constraints, we model landmarks as weak-perspective projections of a 3D object. [sent-50, score-0.599]

20 To capture withinclass variation, we model the 3D shape of any object instance as a linear combination of 3D basis shapes. [sent-51, score-0.411]

21 Crucially, we make use of occlusion reports generated by our local view-based templates to estimate morphable 3D shape and camera viewpoint. [sent-53, score-0.792]

22 Voting-based methods: One approach to detection and viewpoint classiﬁcation is based on bottomup geometric voting, using a Hough transform or geometric hashing. [sent-55, score-0.411]

23 Our approach differs in that we require no initial feature detection stage, and instead we reason about all possible geometric conﬁgurations and occlusion states. [sent-59, score-0.323]

24 One classic representation for encoding such visibility constraints is an aspect graph [5]. [sent-65, score-0.317]

25 [33] model such topological constraints with global mixtures with varying tree structures. [sent-66, score-0.475]

26 Our model is similar to such approaches, except that we use a decomposable notion of aspect; we simultaneously reason about global and semi-local changes in visibility using local part mixtures with global co-occurrence constraints. [sent-67, score-0.93]

27 Our approach is closely related to the recent work of [22], which also uses a deformable part model (DPM) to capture viewpoint variation in cars. [sent-71, score-0.452]

28 Our model differs in that we directly reason about the location of fully-occluded landmarks, we model an exponential number of viewpoints by using a compositional representation, and we produce continuous 3D shapes and camera viewpoints associated with each detection using only 2D training data. [sent-73, score-0.6]

29 To estimate such models from 2D data, we adapt methods designed for tracking morphable shapes to 3D object category recognition [29, 28]. [sent-75, score-0.311]

30 Our model differs its encoding of occlusion states using local mixtures, as well as the introduction of global mixtures that enforce occlusions and spatial geometry consistent with changes in 3D viewpoint. [sent-79, score-0.719]

31 A notable aspect of our model is that we estimate landmark locations for all parts in all views, even when they are fully occluded. [sent-85, score-0.516]

32 We consider different relational graphs Gm = (V, Em ) where Em connects pairs of landmarks constrained to have consistent locations and local mixtures in global mixture m. [sent-88, score-1.074]

33 Note that we deﬁne a template even for mixtures ti corresponding to fully-occluded states. [sent-97, score-0.361]

34 Unlike the remaining terms in our scoring function, the local appearance model is not dependent on the global mixture/viewpoint. [sent-100, score-0.373]

35 We show that this independence allows our model to compose together different local mixtures to model a single global viewpoint. [sent-101, score-0.463]

36 Notably, this spring depends on part i and j, the local mixture components of part i and j, and the global mixture m. [sent-105, score-0.605]

37 The last term γijmj deﬁnes a co-occurrence score associated with instancing local mixture ti and tj , and global mixture m. [sent-108, score-0.632]

38 This encodes the 3 constraint that, if the left front headlight is occluded due to self occlusion, the left front wheel is also likely occluded. [sent-109, score-0.317]

39 Intuitively, different relational structures may help because occluded landmarks tend to be localized with less reliability. [sent-113, score-0.574]

40 Even for a ﬁxed global mixture m, our model can generate an exponentially-large set of appearances |V |T , where T is the number of local mixture types. [sent-115, score-0.481]

41 To see that the inner maximization can be optimized by DP, let us deﬁne zi = (pi , ti ) to denote both the discrete pixel position and discrete mixture type of part i. [sent-119, score-0.316]

42 2 Learning We assume we are given training data consisting of image-landmark triplet {In , pin , oin }, where landmarks are augmented with an additional discrete visibility ﬂag oin . [sent-123, score-0.771]

43 We use oin ∈ {0, 1, 2} to denote visible, self-occlusion, and other-occlusion respectively, where other occlusion corresponds to a landmark that is occluded by another object (or the image border). [sent-125, score-0.893]

44 We now show how to augment this training set with local mixtures labels tin , global mixtures labels mn , and global edge structures Em . [sent-126, score-1.094]

45 Essentially, we infer such mixture labels using probabilistic algorithms for generating local/global clusters of 2D landmark conﬁgurations. [sent-127, score-0.572]

46 We construct a “local-geometric-context” vector for each part, and obtain landmark mixture labels by grouping landmark instances with similar local geometry. [sent-130, score-1.052]

47 Speciﬁcally, for each landmark i and image n, we construct a K-element vector gin that deﬁnes the 2D relative location of a landmark with respect to the other K landmarks in instance n, normalized for the size of that training instance. [sent-131, score-1.25]

48 This means that, for landmark i, a third of its T local mixtures will model visible instances in the training set, a third will model self-occlusions, and a third will capture other-occlusions. [sent-136, score-0.833]

49 Learning relational structure: Given local mixture labels tin , we simultaneously learn global mixtures mn and edge structure Em with a probabilistic model of zin = (pin , tin ). [sent-137, score-0.933]

50 We ﬁnd the global mixtures and edge structure that maximizes the probability of the observed {zin } labels. [sent-138, score-0.416]

51 Learning structure and global mixtures: To simultaneously learn global mixture labels mn and edge structures associated with each mixture Em , we use an EM algorithm for learning mixtures of trees [20, 15]. [sent-144, score-0.909]

52 We demonstrate that our latently-estimated global mixtures are crucial for high-performance in 3D reasoning. [sent-149, score-0.386]

53 To do so, let us write the landmark position labels pn , local mixtures labels tn , and global mixture label mn collectively as yn . [sent-152, score-1.156]

54 We use it to propose detections with associated landmarks positions p∗ . [sent-164, score-0.459]

55 In this section, we describe a 3D shape and viewpoint model for reﬁning p∗ . [sent-165, score-0.448]

56 Consider 2D views of a single rigid object; 2D landmark positions must obey epipolar geometry constraints. [sent-166, score-0.503]

57 (2) We assume the 3D landmarks of all object instances can be written as linear combinations of a few basis shapes. [sent-171, score-0.49]

58 Let us write the set of detected landmark positions as p∗ as a 2 × K matrix where K = |V |. [sent-172, score-0.447]

59 Inference: Given 2D landmark locations p∗ and a known set of 3D basis shapes B i , inference corresponds to minimizing (6). [sent-175, score-0.594]

60 This means that we can associate each detection with shape basis coefﬁcients αi (which allows us to reconstruct the 3D shape) and camera viewpoint R. [sent-179, score-0.753]

61 One can estimate such a basis given training data with labeled 2D landmark positions by casting this as nonrigid structure from motion (SFM) problem. [sent-184, score-0.536]

62 Stack all 2D landmarks from N training images into a 2N × K matrix. [sent-185, score-0.412]

63 Notably, our 2D appearance model provides location estimates for occluded landmarks. [sent-191, score-0.322]

64 Recall that our learning formulation requires all landmarks (including occluded ones) to be labeled in training data. [sent-194, score-0.51]

65 Manually labeling the positions of occluded landmarks can be ambiguous. [sent-195, score-0.554]

66 Instead, we use the estimated shape basis and camera viewpoints to infer/correct the locations of occluded landmarks. [sent-196, score-0.746]

67 5 Experiments Datasets: To evaluate our model, we focus on car detection and 3D landmark estimation in cluttered, real-world datasets with severe occlusions. [sent-197, score-0.607]

68 We labeled a subset of 500 images from the PASCAL VOC 2011 dataset [9] with locations and visibility states of 20 car landmarks. [sent-198, score-0.482]

69 36% of landmarks are not visible due to self-occlusion, while 21% of landmarks are not visible due to occlusion by another object (or truncation due to the image border). [sent-200, score-1.108]

70 Hence over half our landmarks are occluded, making our dataset considerably more difﬁcult than those typically used for landmark localization or 3D viewpoint estimation. [sent-201, score-1.027]

71 We also compare results on a more standard viewpoint dataset from [1], which consists of 200 relatively “clean” cars from the PASCAL VOC 2007 dataset, marked with 40 discrete viewpoint class labels. [sent-203, score-0.479]

72 Implementation: We modify the publically-available code of [31] and [28] to learn our models, setting the number of local mixtures T = 9, the number of global mixtures M = 50, and the number of basis shapes nB = 5. [sent-204, score-0.823]

73 Learning our 2D deformable model takes roughly 4 hours, while learning our 3D shape model takes less than a minute. [sent-206, score-0.343]

74 Evaluation: Given an image, our algorithm produces multiple detections, each with 3D landmark locations, visibility ﬂags, and camera viewpoints. [sent-209, score-0.801]

75 To evaluate our output, we assume test images are marked with ground-truth cars, each annotated with ground-truth 2D landmarks and visibility ﬂags. [sent-212, score-0.657]

76 We evaluate 2D landmark localization (LP) by counting the fraction of predicted 6 80 Our model 60 Arie−Nachimson and Basri 40 Glasner et al. [sent-215, score-0.464]

77 We also ﬁnd that modeling the effects of shape due to global changes in 3D viewpoint is crucial for both detection and landmark localization. [sent-275, score-1.126]

78 We evaluate landmark visibility prediction (VP) by counting the number of landmarks whose predicted visibility state matches the ground-truth, where landmarks may be “visible”, “self-occluded”, or “other-occluded”. [sent-278, score-1.599]

79 Finally to evaluate viewpoint classiﬁcation (VC), we compare predicted camera viewpoints with ground-truth viewpoints on the standard benchmark of [1]. [sent-281, score-0.507]

80 Given a test instance, we run our detector, estimate the camera rotation R, and report the reconstructed 2D landmarks generated using the estimated R. [sent-284, score-0.506]

81 Then we produce a quantized viewpoint label by matching the reconstructions to landmark locations for a reference image (provided in the dataset). [sent-285, score-0.779]

82 This latter model is equivalent in structure to a state-of-the-art model for car detection and viewpoint estimation [22], which trains a DPM using supervision provided by a 3D CAD model. [sent-294, score-0.414]

83 5%), but does noticeably better than both view-based models for landmark prediction. [sent-298, score-0.403]

84 We produce landmark visibility (VP) estimates from our multiview baselines by predicting a ﬁxed set of visibility labels conditioned on the view-based mixture. [sent-303, score-0.991]

85 We should note that accurate landmark localization is crucial for estimating the 3D shape of the detected instance. [sent-304, score-0.702]

86 The top image (in the pair) shows the output of our tree model, and the bottom shows our 3D shape reconstruction, following the notational conventions of Fig. [sent-308, score-0.335]

87 Our morphable 3D model adapts to the shape of the car, producing different reconstructions for SUVs and sedans (row 2, columns 2-3). [sent-311, score-0.487]

88 Recall that our tree model explicitly reasons about changes in visibility due to self-occlusions versus occlusions from other objects, manifested as local mixture templates. [sent-312, score-0.574]

89 In some cases, the estimated 3D shape is misaligned due to extreme shape variation of the car instance (e. [sent-316, score-0.624]

90 “Local” refers to a single tree model with local mixtures only, while “Global” refers to our global mixtures of trees. [sent-321, score-0.762]

91 However, in terms of landmark prediction, “Global” strongly outperforms “Local”, 69. [sent-324, score-0.403]

92 We use these predicted landmarks to estimate 3D shape below. [sent-327, score-0.591]

93 3D Shape: Our 3D shape model reports back a z depth value for each landmark (x, y) position. [sent-328, score-0.641]

94 We further analyze this by looking at the improvement in localization accuracy of ground-truth landmarks that are visible (73. [sent-333, score-0.483]

95 Local templates corresponding to occluded mixtures will be less accurate, and so will beneﬁt more from a 3D shape model. [sent-341, score-0.683]

96 Conclusion: We have described a geometric model for detecting and estimating the 3D shape of objects in heavily cluttered, occluded, real-world images. [sent-342, score-0.365]

97 Our model differs from typical multiview approaches by reasoning about local changes in landmark appearance and global changes in visibility and shape, through the aid of a morphable 3D model. [sent-343, score-1.289]

98 While our model is similar to prior work in terms of detection performance, it produces signiﬁcantly better estimates of 2D/3D landmarks and camera positions, and quantiﬁably improves localization of occluded landmarks. [sent-344, score-0.829]

99 Face detection, pose estimation, and landmark localization in the wild. [sent-551, score-0.507]

100 Revisiting 3d geometric models for accurate object shape and pose. [sent-558, score-0.376]

similar papers computed by tfidf model

tfidf for this paper:

wordName wordTfidf (topN-words)

[('landmark', 0.403), ('landmarks', 0.353), ('mixtures', 0.248), ('visibility', 0.245), ('shape', 0.238), ('viewpoint', 0.21), ('occluded', 0.157), ('morphable', 0.156), ('camera', 0.153), ('global', 0.138), ('occlusion', 0.128), ('appearance', 0.12), ('mixture', 0.115), ('ti', 0.113), ('deformable', 0.105), ('detection', 0.105), ('car', 0.099), ('pi', 0.098), ('vp', 0.093), ('object', 0.09), ('cluttered', 0.087), ('ijmj', 0.087), ('locations', 0.079), ('local', 0.077), ('sfm', 0.076), ('tj', 0.074), ('viewpoints', 0.072), ('dpm', 0.07), ('headlights', 0.069), ('oin', 0.069), ('tin', 0.069), ('visible', 0.069), ('ap', 0.067), ('shapes', 0.065), ('cvpr', 0.065), ('relational', 0.064), ('detections', 0.062), ('localization', 0.061), ('cars', 0.059), ('images', 0.059), ('spring', 0.056), ('views', 0.056), ('labels', 0.054), ('occlusions', 0.054), ('lp', 0.054), ('pj', 0.054), ('part', 0.052), ('sedans', 0.052), ('tree', 0.051), ('variation', 0.049), ('nb', 0.048), ('geometric', 0.048), ('objects', 0.047), ('basis', 0.047), ('compositional', 0.046), ('em', 0.046), ('glasner', 0.046), ('headlight', 0.046), ('wheel', 0.046), ('image', 0.046), ('location', 0.045), ('mv', 0.045), ('positions', 0.044), ('multiview', 0.044), ('pose', 0.043), ('differs', 0.042), ('basri', 0.042), ('nonrigid', 0.042), ('workshops', 0.042), ('reconstructions', 0.041), ('templates', 0.04), ('stage', 0.039), ('constraints', 0.038), ('scoring', 0.038), ('pascal', 0.038), ('mn', 0.038), ('zi', 0.036), ('appearances', 0.036), ('capture', 0.036), ('arie', 0.035), ('nachimson', 0.035), ('pin', 0.035), ('wagons', 0.035), ('aspect', 0.034), ('front', 0.034), ('detectors', 0.034), ('trees', 0.033), ('zj', 0.033), ('analyzing', 0.033), ('gm', 0.033), ('detecting', 0.032), ('heavy', 0.032), ('voc', 0.032), ('changes', 0.032), ('iccv', 0.031), ('dpms', 0.031), ('station', 0.031), ('zin', 0.031), ('edge', 0.03), ('yn', 0.029)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.99999952 40 nips-2012-Analyzing 3D Objects in Cluttered Images

Author: Mohsen Hejrati, Deva Ramanan

2 0.22562881 1 nips-2012-3D Object Detection and Viewpoint Estimation with a Deformable 3D Cuboid Model

Author: Sanja Fidler, Sven Dickinson, Raquel Urtasun

Abstract: This paper addresses the problem of category-level 3D object detection. Given a monocular image, our aim is to localize the objects in 3D by enclosing them with tight oriented 3D bounding boxes. We propose a novel approach that extends the well-acclaimed deformable part-based model [1] to reason in 3D. Our model represents an object class as a deformable 3D cuboid composed of faces and parts, which are both allowed to deform with respect to their anchors on the 3D box. We model the appearance of each face in fronto-parallel coordinates, thus effectively factoring out the appearance variation induced by viewpoint. Our model reasons about face visibility patters called aspects. We train the cuboid model jointly and discriminatively and share weights across all aspects to attain efﬁciency. Inference then entails sliding and rotating the box in 3D and scoring object hypotheses. While for inference we discretize the search space, the variables are continuous in our model. We demonstrate the effectiveness of our approach in indoor and outdoor scenarios, and show that our approach signiﬁcantly outperforms the stateof-the-art in both 2D [1] and 3D object detection [2]. 1

3 0.18723412 330 nips-2012-Supervised Learning with Similarity Functions

Author: Purushottam Kar, Prateek Jain

Abstract: We address the problem of general supervised learning when data can only be accessed through an (indeﬁnite) similarity function between data points. Existing work on learning with indeﬁnite kernels has concentrated solely on binary/multiclass classiﬁcation problems. We propose a model that is generic enough to handle any supervised learning task and also subsumes the model previously proposed for classiﬁcation. We give a “goodness” criterion for similarity functions w.r.t. a given supervised learning task and then adapt a well-known landmarking technique to provide efﬁcient algorithms for supervised learning using “good” similarity functions. We demonstrate the effectiveness of our model on three important supervised learning problems: a) real-valued regression, b) ordinal regression and c) ranking where we show that our method guarantees bounded generalization error. Furthermore, for the case of real-valued regression, we give a natural goodness deﬁnition that, when used in conjunction with a recent result in sparse vector recovery, guarantees a sparse predictor with bounded generalization error. Finally, we report results of our learning algorithms on regression and ordinal regression tasks using non-PSD similarity functions and demonstrate the effectiveness of our algorithms, especially that of the sparse landmark selection algorithm that achieves signiﬁcantly higher accuracies than the baseline methods while offering reduced computational costs. 1

4 0.18149997 201 nips-2012-Localizing 3D cuboids in single-view images

Author: Jianxiong Xiao, Bryan Russell, Antonio Torralba

Abstract: In this paper we seek to detect rectangular cuboids and localize their corners in uncalibrated single-view images depicting everyday scenes. In contrast to recent approaches that rely on detecting vanishing points of the scene and grouping line segments to form cuboids, we build a discriminative parts-based detector that models the appearance of the cuboid corners and internal edges while enforcing consistency to a 3D cuboid model. Our model copes with different 3D viewpoints and aspect ratios and is able to detect cuboids across many different object categories. We introduce a database of images with cuboid annotations that spans a variety of indoor and outdoor scenes and show qualitative and quantitative results on our collected database. Our model out-performs baseline detectors that use 2D constraints alone on the task of localizing cuboid corners. 1

5 0.17047296 8 nips-2012-A Generative Model for Parts-based Object Segmentation

Author: S. Eslami, Christopher Williams

Abstract: The Shape Boltzmann Machine (SBM) [1] has recently been introduced as a stateof-the-art model of foreground/background object shape. We extend the SBM to account for the foreground object’s parts. Our new model, the Multinomial SBM (MSBM), can capture both local and global statistics of part shapes accurately. We combine the MSBM with an appearance model to form a fully generative model of images of objects. Parts-based object segmentations are obtained simply by performing probabilistic inference in the model. We apply the model to two challenging datasets which exhibit signiﬁcant shape and appearance variability, and ﬁnd that it obtains results that are comparable to the state-of-the-art. There has been signiﬁcant focus in computer vision on object recognition and detection e.g. [2], but a strong desire remains to obtain richer descriptions of objects than just their bounding boxes. One such description is a parts-based object segmentation, in which an image is partitioned into multiple sets of pixels, each belonging to either a part of the object of interest, or its background. The signiﬁcance of parts in computer vision has been recognized since the earliest days of the ﬁeld (e.g. [3, 4, 5]), and there exists a rich history of work on probabilistic models for parts-based segmentation e.g. [6, 7]. Many such models only consider local neighborhood statistics, however several models have recently been proposed that aim to increase the accuracy of segmentations by also incorporating prior knowledge about the foreground object’s shape [8, 9, 10, 11]. In such cases, probabilistic techniques often mainly differ in how accurately they represent and learn about the variability exhibited by the shapes of the object’s parts. Accurate models of the shapes and appearances of parts can be necessary to perform inference in datasets that exhibit large amounts of variability. In general, the stronger the models of these two components, the more performance is improved. A generative model has the added beneﬁt of being able to generate samples, which allows us to visually inspect the quality of its understanding of the data and the problem. Recently, a generative probabilistic model known as the Shape Boltzmann Machine (SBM) has been used to model binary object shapes [1]. The SBM has been shown to constitute the state-of-the-art and it possesses several highly desirable characteristics: samples from the model look realistic, and it generalizes to generate samples that differ from the limited number of examples it is trained on. The main contributions of this paper are as follows: 1) In order to account for object parts we extend the SBM to use multinomial visible units instead of binary ones, resulting in the Multinomial Shape Boltzmann Machine (MSBM), and we demonstrate that the MSBM constitutes a strong model of parts-based object shape. 2) We combine the MSBM with an appearance model to form a fully generative model of images of objects (see Fig. 1). We show how parts-based object segmentations can be obtained simply by performing probabilistic inference in the model. We apply our model to two challenging datasets and ﬁnd that in addition to being principled and fully generative, the model’s performance is comparable to the state-of-the-art. 1 Train labels Train images Test image Appearance model Joint Model Shape model Parsing Figure 1: Overview. Using annotated images separate models of shape and appearance are trained. Given an unseen test image, its parsing is obtained via inference in the proposed joint model. In Secs. 1 and 2 we present the model and propose efﬁcient inference and learning schemes. In Sec. 3 we compare and contrast the resulting joint model with existing work in the literature. We describe our experimental results in Sec. 4 and conclude with a discussion in Sec. 5. 1 Model We consider datasets of cropped images of an object class. We assume that the images are constructed through some combination of a ﬁxed number of parts. Given a dataset D = {Xd }, d = 1...n of such images X, each consisting of P pixels {xi }, i = 1...P , we wish to infer a segmentation S for the image. S consists of a labeling si for every pixel, where si is a 1-of-(L+1) encoded variable, and L is the ﬁxed number of parts that combine to generate the foreground. In other words, si = (sli ), P l = 0...L, sli 2 {0, 1} and l sli = 1. Note that the background is also treated as a ‘part’ (l = 0). Accurate inference of S is driven by models for 1) part shapes and 2) part appearances. Part shapes: Several types of models can be used to deﬁne probabilistic distributions over segmentations S. The simplest approach is to model each pixel si independently with categorical variables whose parameters are speciﬁed by the object’s mean shape (Fig. 2(a)). Markov Random Fields (MRFs, Fig. 2(b)) additionally model interactions between nearby pixels using pairwise potential functions that efﬁciently capture local properties of images like smoothness and continuity. Restricted Boltzmann Machines (RBMs) and their multi-layered counterparts Deep Boltzmann Machines (DBMs, Fig. 2(c)) make heavy use of hidden variables to efﬁciently deﬁne higher-order potentials that take into account the conﬁguration of larger groups of image pixels. The introduction of such hidden variables provides a way to efﬁciently capture complex, global properties of image pixels. RBMs and DBMs are powerful generative models, but they also have many parameters. Segmented images, however, are expensive to obtain and datasets are typically small (hundreds of examples). In order to learn a model that accurately captures the properties of part shapes we use DBMs but also impose carefully chosen connectivity and capacity constraints, following the structure of the Shape Boltzmann Machine (SBM) [1]. We further extend the model to account for multi-part shapes to obtain the Multinomial Shape Boltzmann Machine (MSBM). The MSBM has two layers of latent variables: h1 and h2 (collectively H = {h1 , h2 }), and deﬁnes a P Boltzmann distribution over segmentations p(S) = h1 ,h2 exp{ E(S, h1 , h2 |✓s )}/Z(✓s ) where X X X X X 1 2 E(S, h1 , h2 |✓s ) = bli sli + wlij sli h1 + c 1 h1 + wjk h1 h2 + c2 h2 , (1) j j j j k k k i,l j i,j,l j,k k where j and k range over the ﬁrst and second layer hidden variables, and ✓s = {W 1 , W 2 , b, c1 , c2 } are the shape model parameters. In the ﬁrst layer, local receptive ﬁelds are enforced by connecting each hidden unit in h1 only to a subset of the visible units, corresponding to one of four patches, as shown in Fig. 2(d,e). Each patch overlaps its neighbor by b pixels, which allows boundary continuity to be learned at the lowest layer. We share weights between the four sets of ﬁrst-layer hidden units and patches, and purposely restrict the number of units in h2 . These modiﬁcations signiﬁcantly reduce the number of parameters whilst taking into account an important property of shapes, namely that the strongest dependencies between pixels are typically local. 2 h2 1 1 h S S (a) Mean h S (b) MRF h2 h2 h1 S S (c) DBM b (d) SBM (e) 2D SBM Figure 2: Models of shape. Object shape is modeled with undirected graphical models. (a) 1D slice of a mean model. (b) Markov Random Field in 1D. (c) Deep Boltzmann Machine in 1D. (d) 1D slice of a Shape Boltzmann Machine. (e) Shape Boltzmann Machine in 2D. In all models latent units h are binary and visible units S are multinomial random variables. Based on Fig. 2 of [1]. k=1 k=2 k=3 k=1 k=2 k=3 k=1 k=2 k=3 ⇡ l=0 l=1 l=2 Figure 3: A model of appearances. Left: An exemplar dataset. Here we assume one background (l = 0) and two foreground (l = 1, non-body; l = 2, body) parts. Right: The corresponding appearance model. In this example, L = 2, K = 3 and W = 6. Best viewed in color. Part appearances: Pixels in a given image are assumed to have been generated by W ﬁxed Gaussians in RGB space. During pre-training, the means {µw } and covariances {⌃w } of these Gaussians are extracted by training a mixture model with W components on every pixel in the dataset, ignoring image and part structure. It is also assumed that each of the L parts can have different appearances in different images, and that these appearances can be clustered into K classes. The classes differ in how likely they are to use each of the W components when ‘coloring in’ the part. The generative process is as follows. For part l in an image, one of the K classes is chosen (represented by a 1-of-K indicator variable al ). Given al , the probability distribution deﬁned on pixels associated with part l is given by a Gaussian mixture model with means {µw } and covariances {⌃w } and mixing proportions { lkw }. The prior on A = {al } speciﬁes the probability ⇡lk of appearance class k being chosen for part l. Therefore appearance parameters ✓a = {⇡lk , lkw } (see Fig. 3) and: a p(xi |A, si , ✓ ) = p(A|✓a ) = Y l Y l a sli p(xi |al , ✓ ) p(al |✓a ) = = Y Y X YY l l k w lkw N (xi |µw , ⌃w ) !alk !sli (⇡lk )alk . , (2) (3) k Combining shapes and appearances: To summarize, the latent variables for X are A, S, H, and the model’s active parameters ✓ include shape parameters ✓s and appearance parameters ✓a , so that p(X, A, S, H|✓) = Y 1 p(A|✓a )p(S, H|✓s ) p(xi |A, si , ✓a ) , Z( ) i (4) where the parameter adjusts the relative contributions of the shape and appearance components. See Fig. 4 for an illustration of the complete graphical model. During learning, we ﬁnd the values of ✓ that maximize the likelihood of the training data D, and segmentation is performed on a previously-unseen image by querying the marginal distribution p(S|Xtest , ✓). Note that Z( ) is constant throughout the execution of the algorithms. We set via trial and error in our experiments. 3 n H ✓a si al H xi L+1 ✓s S X A P Figure 4: A model of shape and appearance. Left: The joint model. Pixels xi are modeled via appearance variables al . The model’s belief about each layer’s shape is captured by shape variables H. Segmentation variables si assign each pixel to a layer. Right: Schematic for an image X. 2 Inference and learning Inference: We approximate p(A, S, H|X, ✓) by drawing samples of A, S and H using block-Gibbs Markov Chain Monte Carlo (MCMC). The desired distribution p(S|X, ✓) can then be obtained by considering only the samples for S (see Algorithm 1). In order to sample p(A|S, H, X, ✓) we consider the conditional distribution of appearance class k being chosen for part l which is given by: Q P ·s ⇡lk i ( w lkw N (xi |µw , ⌃w )) li h Q P i. p(alk = 1|S, X, ✓) = P (5) K ·sli r=1 ⇡lr i( w lrw N (xi |µw , ⌃w )) Since the MSBM only has edges between each pair of adjacent layers, all hidden units within a layer are conditionally independent given the units in the other two layers. This property can be exploited to make inference in the shape model exact and efﬁcient. The conditional probabilities are: X X 1 2 p(h1 = 1|s, h2 , ✓) = ( wlij sli + wjk h2 + c1 ), (6) j k j i,l p(h2 k 1 = 1|h , ✓) = ( X k 2 wjk h1 j + c2 ), j (7) j where (y) = 1/(1 + exp( y)) is the sigmoid function. To sample from p(H|S, X, ✓) we iterate between Eqns. 6 and 7 multiple times and keep only the ﬁnal values of h1 and h2 . Finally, we draw samples for the pixels in p(S|A, H, X, ✓) independently: P 1 exp( j wlij h1 + bli ) p(xi |A, sli = 1, ✓) j p(sli = 1|A, H, X, ✓) = PL . (8) P 1 1 m=1 exp( j wmij hj + bmi ) p(xi |A, smi = 1, ✓) Seeding: Since the latent-space is extremely high-dimensional, in practice we ﬁnd it helpful to run several inference chains, each initializing S(1) to a different value. The ‘best’ inference is retained and the others are discarded. The computation of the likelihood p(X|✓) of image X is intractable, so we approximate the quality of each inference using a scoring function: 1X Score(X|✓) = p(X, A(t) , S(t) , H(t) |✓), (9) T t where {A(t) , S(t) , H(t) }, t = 1...T are the samples obtained from the posterior p(A, S, H|X, ✓). If the samples were drawn from the prior p(A, S, H|✓) the scoring function would be an unbiased estimator of p(X|✓), but would be wildly inaccurate due to the high probability of missing the important regions of latent space (see e.g. [12, p. 107-109] for further discussion of this issue). Learning: Learning of the model involves maximizing the log likelihood log p(D|✓a , ✓s ) of the training dataset D with respect to the model parameters ✓a and ✓s . Since training is partially supervised, in that for each image X its corresponding segmentation S is also given, we can learn the parameters of the shape and appearance components separately. For appearances, the learning of the mixing coefﬁcients and the histogram parameters decomposes into standard mixture updates independently for each part. For shapes, we follow the standard deep 4 Algorithm 1 MCMC inference algorithm. 1: procedure I NFER(X, ✓) 2: Initialize S(1) , H(1) 3: for t 2 : chain length do 4: A(t) ⇠ p(A|S(t 1) , H(t 1) , X, ✓) 5: S(t) ⇠ p(S|A(t) , H(t 1) , X, ✓) 6: H(t) ⇠ p(H|S(t) , ✓) 7: return {S(t) }t=burnin:chain length learning literature closely [13, 1]. In the pre-training phase we greedily train the model bottom up, one layer at a time. We begin by training an RBM on the observed data using stochastic maximum likelihood learning (SML; also referred to as ‘persistent CD’; [14, 13]). Once this RBM is trained, we infer the conditional mean of the hidden units for each training image. The resulting vectors then serve as the training data for a second RBM which is again trained using SML. We use the parameters of these two RBMs to initialize the parameters of the full MSBM model. In the second phase we perform approximate stochastic gradient ascent in the likelihood of the full model to ﬁnetune the parameters in an EM-like scheme as described in [13]. 3 Related work Existing probabilistic models of images can be categorized by the amount of variability they expect to encounter in the data and by how they model this variability. A signiﬁcant portion of the literature models images using only two parts: a foreground object and its background e.g. [15, 16, 17, 18, 19]. Models that account for the parts within the foreground object mainly differ in how accurately they learn about and represent the variability of the shapes of the object’s parts. In Probabilistic Index Maps (PIMs) [8] a mean partitioning is learned, and the deformable PIM [9] additionally allows for local deformations of this mean partitioning. Stel Component Analysis [10] accounts for larger amounts of shape variability by learning a number of different template means for the object that are blended together on a pixel-by-pixel basis. Factored Shapes and Appearances [11] models global properties of shape using a factor analysis-like model, and ‘masked’ RBMs have been used to model more local properties of shape [20]. However, none of these models constitute a strong model of shape in terms of realism of samples and generalization capabilities [1]. We demonstrate in Sec. 4 that, like the SBM, the MSBM does in fact possess these properties. The closest works to ours in terms of ability to deal with datasets that exhibit signiﬁcant variability in both shape and appearance are the works of Bo and Fowlkes [21] and Thomas et al. [22]. Bo and Fowlkes [21] present an algorithm for pedestrian segmentation that models the shapes of the parts using several template means. The different parts are composed using hand coded geometric constraints, which means that the model cannot be automatically extended to other application domains. The Implicit Shape Model (ISM) used in [22] is reliant on interest point detectors and deﬁnes distributions over segmentations only in the posterior, and therefore is not fully generative. The model presented here is entirely learned from data and fully generative, therefore it can be applied to new datasets and diagnosed with relative ease. Due to its modular structure, we also expect it to rapidly absorb future developments in shape and appearance models. 4 Experiments Penn-Fudan pedestrians: The ﬁrst dataset that we considered is Penn-Fudan pedestrians [23], consisting of 169 images of pedestrians (Fig. 6(a)). The images are annotated with ground-truth segmentations for L = 7 different parts (hair, face, upper and lower clothes, shoes, legs, arms; Fig. 6(d)). We compare the performance of the model with the algorithm of Bo and Fowlkes [21]. For the shape component, we trained an MSBM on the 684 images of a labeled version of the HumanEva dataset [24] (at 48 ⇥ 24 pixels; also ﬂipped horizontally) with overlap b = 4, and 400 and 50 hidden units in the ﬁrst and second layers respectively. Each layer was pre-trained for 3000 epochs (iterations). After pre-training, joint training was performed for 1000 epochs. 5 (c) Completion (a) Sampling (b) Diffs ! ! ! Figure 5: Learned shape model. (a) A chain of samples (1000 samples between frames). The apparent ‘blurriness’ of samples is not due to averaging or resizing. We display the probability of each pixel belonging to different parts. If, for example, there is a 50-50 chance that a pixel belongs to the red or blue parts, we display that pixel in purple. (b) Differences between the samples and their most similar counterparts in the training dataset. (c) Completion of occlusions (pink). To assess the realism and generalization characteristics of the learned MSBM we sample from it. In Fig. 5(a) we show a chain of unconstrained samples from an MSBM generated via block-Gibbs MCMC (1000 samples between frames). The model captures highly non-linear correlations in the data whilst preserving the object’s details (e.g. face and arms). To demonstrate that the model has not simply memorized the training data, in Fig. 5(b) we show the difference between the sampled shapes in Fig. 5(a) and their closest images in the training set (based on per-pixel label agreement). We see that the model generalizes in non-trivial ways to generate realistic shapes that it had not encountered during training. In Fig. 5(c) we show how the MSBM completes rectangular occlusions. The samples highlight the variability in possible completions captured by the model. Note how, e.g. the length of the person’s trousers on one leg affects the model’s predictions for the other, demonstrating the model’s knowledge about long-range dependencies. An interactive M ATLAB GUI for sampling from this MSBM has been included in the supplementary material. The Penn-Fudan dataset (at 200 ⇥ 100 pixels) was then split into 10 train/test cross-validation splits without replacement. We used the training images in each split to train the appearance component with a vocabulary of size W = 50 and K = 100 mixture components1 . We additionally constrained the model by sharing the appearance models for the arms and legs with that of the face. We assess the quality of the appearance model by performing the following experiment: for each test image, we used the scoring function described in Eq. 9 to evaluate a number of different proposal segmentations for that image. We considered 10 randomly chosen segmentations from the training dataset as well as the ground-truth segmentation for the test image, and found that the appearance model correctly assigns the highest score to the ground-truth 95% of the time. During inference, the shape and appearance models (which are deﬁned on images of different sizes), were combined at 200 ⇥ 100 pixels via M ATLAB’s imresize function, and we set = 0.8 (Eq. 8) via trial and error. Inference chains were seeded at 100 exemplar segmentations from the HumanEva dataset (obtained using the K-medoids algorithm with K = 100), and were run for 20 Gibbs iterations each (with 5 iterations of Eqs. 6 and 7 per Gibbs iteration). Our unoptimized M ATLAB implementation completed inference for each chain in around 7 seconds. We compute the conditional probability of each pixel belonging to different parts given the last set of samples obtained from the highest scoring chain, assign each pixel independently to the most likely part at that pixel, and report the percentage of correctly labeled pixels (see Table 1). We ﬁnd that accuracy can be improved using superpixels (SP) computed on X (pixels within a superpixel are all assigned the most common label within it; as with [21] we use gPb-OWT-UCM [25]). We also report the accuracy obtained, had the top scoring seed segmentation been returned as the ﬁnal segmentation for each image. Here the quality of the seed is determined solely by the appearance model. We observe that the model has comparable performance to the state-of-the-art but pedestrianspeciﬁc algorithm of [21], and that inference in the model signiﬁcantly improves the accuracy of the segmentations over the baseline (top seed+SP). Qualitative results can be seen in Fig. 6(c). 1 We obtained the best quantitative results with these settings. The appearances exhibited by the parts in the dataset are highly varied, and the complexity of the appearance model reﬂects this fact. 6 Table 1: Penn-Fudan pedestrians. We report the percentage of correctly labeled pixels. The ﬁnal column is an average of the background, upper and lower body scores (as reported in [21]). FG BG Upper Body Lower Body Head Average Bo and Fowlkes [21] 73.3% 81.1% 73.6% 71.6% 51.8% 69.5% MSBM MSBM + SP 70.7% 71.6% 72.8% 73.8% 68.6% 69.9% 66.7% 68.5% 53.0% 54.1% 65.3% 66.6% Top seed Top seed + SP 59.0% 61.6% 61.8% 67.3% 56.8% 60.8% 49.8% 54.1% 45.5% 43.5% 53.5% 56.4% Table 2: ETHZ cars. We report the percentage of pixels belonging to each part that are labeled correctly. The ﬁnal column is an average weighted by the frequency of occurrence of each label. BG Body Wheel Window Bumper License Light Average ISM [22] 93.2% 72.2% 63.6% 80.5% 73.8% 56.2% 34.8% 86.8% MSBM 94.6% 72.7% 36.8% 74.4% 64.9% 17.9% 19.9% 86.0% Top seed 92.2% 68.4% 28.3% 63.8% 45.4% 11.2% 15.1% 81.8% ETHZ cars: The second dataset that we considered is the ETHZ labeled cars dataset [22], which itself is a subset of the LabelMe dataset [23], consisting of 139 images of cars, all in the same semiproﬁle view (Fig. 7(a)). The images are annotated with ground-truth segmentations for L = 6 parts (body, wheel, window, bumper, license plate, headlight; Fig. 7(d)). We compare the performance of the model with the ISM of Thomas et al. [22], who also report their results on this dataset. The dataset was split into 10 train/test cross-validation splits without replacement. We used the training images in each split to train both the shape and appearance components. For the shape component, we trained an MSBM at 50 ⇥ 50 pixels with overlap b = 4, and 2000 and 100 hidden units in the ﬁrst and second layers respectively. Each layer was pre-trained for 3000 epochs and joint training was performed for 1000 epochs. The appearance model was trained with a vocabulary of size W = 50 and K = 100 mixture components and we set = 0.7. Inference chains were seeded at 50 exemplar segmentations (obtained using K-medoids). We ﬁnd that the use of superpixels does not help with this dataset (due to the poor quality of superpixels obtained for these images). Qualitative and quantitative results that show the performance of model to be comparable to the state-of-the-art ISM can be seen in Fig. 7(c) and Table 2. We believe the discrepancy in accuracy between the MSBM and ISM on the ‘license’ and ‘light’ labels to mainly be due to ISM’s use of interest-points, as they are able to locate such ﬁne structures accurately. By incorporating better models of part appearance into the generative model, we expect to see this discrepancy decrease. 5 Conclusions and future work In this paper we have shown how the SBM can be extended to obtain the MSBM, and presented a principled probabilistic model of images of objects that exploits the MSBM as its model for part shapes. We demonstrated how object segmentations can be obtained simply by performing MCMC inference in the model. The model can also be treated as a probabilistic evaluator of segmentations: given a proposal segmentation it can be used to estimate its likelihood. This leads us to believe that the combination of a generative model such as ours, with a discriminative, bottom-up segmentation algorithm could be highly effective. We are currently investigating how textured appearance models, which take into account the spatial structure of pixels, affect the learning and inference algorithms and the performance of the model. Acknowledgments Thanks to Charless Fowlkes and Vittorio Ferrari for access to datasets, and to Pushmeet Kohli and John Winn for valuable discussions. AE has received funding from the Carnegie Trust, the SORSAS scheme, and the IST Programme under the PASCAL2 Network of Excellence (IST-2007-216886). 7 (a) Test (c) MSBM (b) Bo and Fowlkes (d) Ground truth Background Hair Face Upper Shoes Legs Lower Arms (d) Ground truth (c) MSBM (b) Thomas et al. (a) Test Figure 6: Penn-Fudan pedestrians. (a) Test images. (b) Results reported by Bo and Fowlkes [21]. (c) Output of the joint model. (d) Ground-truth images. Images shown are those selected by [21]. Background Body Wheel Window Bumper License Headlight Figure 7: ETHZ cars. (a) Test images. (b) Results reported by Thomas et al. [22]. (c) Output of the joint model. (d) Ground-truth images. Images shown are those selected by [22]. 8 References [1] S. M. Ali Eslami, Nicolas Heess, and John Winn. The Shape Boltzmann Machine: a Strong Model of Object Shape. In IEEE CVPR, 2012. [2] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The PASCAL Visual Object Classes (VOC) Challenge. International Journal of Computer Vision, 88:303–338, 2010. [3] Martin Fischler and Robert Elschlager. The Representation and Matching of Pictorial Structures. IEEE Transactions on Computers, 22(1):67–92, 1973. [4] David Marr. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. Freeman, 1982. [5] Irving Biederman. Recognition-by-components: A theory of human image understanding. Psychological Review, 94:115–147, 1987. [6] Ashish Kapoor and John Winn. Located Hidden Random Fields: Learning Discriminative Parts for Object Detection. In ECCV, pages 302–315, 2006. [7] John Winn and Jamie Shotton. The Layout Consistent Random Field for Recognizing and Segmenting Partially Occluded Objects. In IEEE CVPR, pages 37–44, 2006. [8] Nebojsa Jojic and Yaron Caspi. Capturing Image Structure with Probabilistic Index Maps. In IEEE CVPR, pages 212–219, 2004. [9] John Winn and Nebojsa Jojic. LOCUS: Learning object classes with unsupervised segmentation. In ICCV, pages 756–763, 2005. [10] Nebojsa Jojic, Alessandro Perina, Marco Cristani, Vittorio Murino, and Brendan Frey. Stel component analysis. In IEEE CVPR, pages 2044–2051, 2009. [11] S. M. Ali Eslami and Christopher K. I. Williams. Factored Shapes and Appearances for Partsbased Object Understanding. In BMVC, pages 18.1–18.12, 2011. [12] Nicolas Heess. Learning generative models of mid-level structure in natural images. PhD thesis, University of Edinburgh, 2011. [13] Ruslan Salakhutdinov and Geoffrey Hinton. Deep Boltzmann Machines. In AISTATS, volume 5, pages 448–455, 2009. [14] Tijmen Tieleman. Training restricted Boltzmann machines using approximations to the likelihood gradient. In ICML, pages 1064–1071, 2008. [15] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. “GrabCut”: interactive foreground extraction using iterated graph cuts. ACM SIGGRAPH, 23:309–314, 2004. [16] Eran Borenstein, Eitan Sharon, and Shimon Ullman. Combining Top-Down and Bottom-Up Segmentation. In CVPR Workshop on Perceptual Organization in Computer Vision, 2004. [17] Himanshu Arora, Nicolas Loeff, David Forsyth, and Narendra Ahuja. Unsupervised Segmentation of Objects using Efﬁcient Learning. IEEE CVPR, pages 1–7, 2007. [18] Bogdan Alexe, Thomas Deselaers, and Vittorio Ferrari. ClassCut for unsupervised class segmentation. In ECCV, pages 380–393, 2010. [19] Nicolas Heess, Nicolas Le Roux, and John Winn. Weakly Supervised Learning of ForegroundBackground Segmentation using Masked RBMs. In ICANN, 2011. [20] Nicolas Le Roux, Nicolas Heess, Jamie Shotton, and John Winn. Learning a Generative Model of Images by Factoring Appearance and Shape. Neural Computation, 23(3):593–650, 2011. [21] Yihang Bo and Charless Fowlkes. Shape-based Pedestrian Parsing. In IEEE CVPR, 2011. [22] Alexander Thomas, Vittorio Ferrari, Bastian Leibe, Tinne Tuytelaars, and Luc Van Gool. Using Recognition and Annotation to Guide a Robot’s Attention. IJRR, 28(8):976–998, 2009. [23] Bryan Russell, Antonio Torralba, Kevin Murphy, and William Freeman. LabelMe: A Database and Tool for Image Annotation. International Journal of Computer Vision, 77:157–173, 2008. [24] Leonid Sigal, Alexandru Balan, and Michael Black. HumanEva. International Journal of Computer Vision, 87(1-2):4–27, 2010. [25] Pablo Arbelaez, Michael Maire, Charless C. Fowlkes, and Jitendra Malik. From Contours to Regions: An Empirical Evaluation. In IEEE CVPR, 2009. 9

6 0.15929006 344 nips-2012-Timely Object Recognition

7 0.15553878 180 nips-2012-Learning Mixtures of Tree Graphical Models

8 0.1499335 42 nips-2012-Angular Quantization-based Binary Codes for Fast Similarity Search

9 0.14533134 106 nips-2012-Dynamical And-Or Graph Learning for Object Shape Modeling and Detection

10 0.14321464 357 nips-2012-Unsupervised Template Learning for Fine-Grained Object Recognition

11 0.13832577 103 nips-2012-Distributed Probabilistic Learning for Camera Networks with Missing Data

12 0.11254552 209 nips-2012-Max-Margin Structured Output Regression for Spatio-Temporal Action Localization

13 0.10791832 303 nips-2012-Searching for objects driven by context

14 0.10314551 185 nips-2012-Learning about Canonical Views from Internet Image Collections

15 0.098223425 360 nips-2012-Visual Recognition using Embedded Feature Selection for Curvature Self-Similarity

16 0.089935236 81 nips-2012-Context-Sensitive Decision Forests for Object Detection

17 0.08453095 83 nips-2012-Controlled Recognition Bounds for Visual Learning and Exploration

18 0.084438384 311 nips-2012-Shifting Weights: Adapting Object Detectors from Image to Video

19 0.082542129 168 nips-2012-Kernel Latent SVM for Visual Recognition

20 0.078504667 339 nips-2012-The Time-Marginalized Coalescent Prior for Hierarchical Clustering

similar papers computed by lsi model

lsi for this paper:

topicId topicWeight

[(0, 0.199), (1, 0.037), (2, -0.181), (3, -0.044), (4, 0.068), (5, -0.138), (6, 0.003), (7, -0.073), (8, -0.018), (9, -0.007), (10, 0.003), (11, 0.027), (12, 0.137), (13, -0.202), (14, 0.062), (15, 0.228), (16, 0.033), (17, -0.111), (18, -0.073), (19, -0.026), (20, 0.041), (21, 0.005), (22, 0.004), (23, 0.073), (24, -0.071), (25, 0.094), (26, 0.063), (27, -0.021), (28, -0.043), (29, -0.016), (30, -0.004), (31, -0.118), (32, -0.015), (33, 0.005), (34, 0.007), (35, -0.016), (36, -0.047), (37, -0.033), (38, 0.007), (39, 0.015), (40, -0.113), (41, 0.065), (42, -0.049), (43, -0.092), (44, -0.053), (45, 0.048), (46, 0.016), (47, 0.038), (48, 0.066), (49, -0.019)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.94877148 40 nips-2012-Analyzing 3D Objects in Cluttered Images

Author: Mohsen Hejrati, Deva Ramanan

2 0.89652741 201 nips-2012-Localizing 3D cuboids in single-view images

Author: Jianxiong Xiao, Bryan Russell, Antonio Torralba

3 0.86754006 1 nips-2012-3D Object Detection and Viewpoint Estimation with a Deformable 3D Cuboid Model

Author: Sanja Fidler, Sven Dickinson, Raquel Urtasun

4 0.72466427 357 nips-2012-Unsupervised Template Learning for Fine-Grained Object Recognition

Author: Shulin Yang, Liefeng Bo, Jue Wang, Linda G. Shapiro

Abstract: Fine-grained recognition refers to a subordinate level of recognition, such as recognizing different species of animals and plants. It differs from recognition of basic categories, such as humans, tables, and computers, in that there are global similarities in shape and structure shared cross different categories, and the differences are in the details of object parts. We suggest that the key to identifying the ﬁne-grained differences lies in ﬁnding the right alignment of image regions that contain the same object parts. We propose a template model for the purpose, which captures common shape patterns of object parts, as well as the cooccurrence relation of the shape patterns. Once the image regions are aligned, extracted features are used for classiﬁcation. Learning of the template model is efﬁcient, and the recognition results we achieve signiﬁcantly outperform the stateof-the-art algorithms. 1

5 0.71608466 8 nips-2012-A Generative Model for Parts-based Object Segmentation

Author: S. Eslami, Christopher Williams

6 0.60355788 106 nips-2012-Dynamical And-Or Graph Learning for Object Shape Modeling and Detection

7 0.585109 137 nips-2012-From Deformations to Parts: Motion-based Segmentation of 3D Objects

8 0.57186544 185 nips-2012-Learning about Canonical Views from Internet Image Collections

9 0.56059271 344 nips-2012-Timely Object Recognition

10 0.55874145 103 nips-2012-Distributed Probabilistic Learning for Camera Networks with Missing Data

11 0.54724491 209 nips-2012-Max-Margin Structured Output Regression for Spatio-Temporal Action Localization

12 0.53196913 303 nips-2012-Searching for objects driven by context

13 0.52327043 223 nips-2012-Multi-criteria Anomaly Detection using Pareto Depth Analysis

14 0.51332843 311 nips-2012-Shifting Weights: Adapting Object Detectors from Image to Video

15 0.50371188 81 nips-2012-Context-Sensitive Decision Forests for Object Detection

16 0.49581805 2 nips-2012-3D Social Saliency from Head-mounted Cameras

17 0.49348524 360 nips-2012-Visual Recognition using Embedded Feature Selection for Curvature Self-Similarity

18 0.48712403 210 nips-2012-Memorability of Image Regions

19 0.46208817 235 nips-2012-Natural Images, Gaussian Mixtures and Dead Leaves

20 0.46041566 168 nips-2012-Kernel Latent SVM for Visual Recognition

similar papers computed by lda model

lda for this paper:

topicId topicWeight

[(0, 0.033), (21, 0.048), (38, 0.083), (39, 0.013), (42, 0.016), (54, 0.015), (74, 0.512), (76, 0.098), (80, 0.055), (92, 0.026)]

similar papers list:

simIndex simValue paperId paperTitle

1 0.91321081 202 nips-2012-Locally Uniform Comparison Image Descriptor

Author: Andrew Ziegler, Eric Christiansen, David Kriegman, Serge J. Belongie

Abstract: Keypoint matching between pairs of images using popular descriptors like SIFT or a faster variant called SURF is at the heart of many computer vision algorithms including recognition, mosaicing, and structure from motion. However, SIFT and SURF do not perform well for real-time or mobile applications. As an alternative very fast binary descriptors like BRIEF and related methods use pairwise comparisons of pixel intensities in an image patch. We present an analysis of BRIEF and related approaches revealing that they are hashing schemes on the ordinal correlation metric Kendall’s tau. Here, we introduce Locally Uniform Comparison Image Descriptor (LUCID), a simple description method based on linear time permutation distances between the ordering of RGB values of two image patches. LUCID is computable in linear time with respect to the number of pixels and does not require ﬂoating point computation. 1

same-paper 2 0.90897202 40 nips-2012-Analyzing 3D Objects in Cluttered Images

Author: Mohsen Hejrati, Deva Ramanan

3 0.90634561 360 nips-2012-Visual Recognition using Embedded Feature Selection for Curvature Self-Similarity

Author: Angela Eigenstetter, Bjorn Ommer

Abstract: Category-level object detection has a crucial need for informative object representations. This demand has led to feature descriptors of ever increasing dimensionality like co-occurrence statistics and self-similarity. In this paper we propose a new object representation based on curvature self-similarity that goes beyond the currently popular approximation of objects using straight lines. However, like all descriptors using second order statistics, ours also exhibits a high dimensionality. Although improving discriminability, the high dimensionality becomes a critical issue due to lack of generalization ability and curse of dimensionality. Given only a limited amount of training data, even sophisticated learning algorithms such as the popular kernel methods are not able to suppress noisy or superﬂuous dimensions of such high-dimensional data. Consequently, there is a natural need for feature selection when using present-day informative features and, particularly, curvature self-similarity. We therefore suggest an embedded feature selection method for SVMs that reduces complexity and improves generalization capability of object models. By successfully integrating the proposed curvature self-similarity representation together with the embedded feature selection in a widely used state-of-the-art object detection framework we show the general pertinence of the approach. 1

4 0.84678096 337 nips-2012-The Lovász ϑ function, SVMs and finding large dense subgraphs

Author: Vinay Jethava, Anders Martinsson, Chiranjib Bhattacharyya, Devdatt Dubhashi

Abstract: The Lov´ sz ϑ function of a graph, a fundamental tool in combinatorial optimizaa tion and approximation algorithms, is computed by solving a SDP. In this paper we establish that the Lov´ sz ϑ function is equivalent to a kernel learning problem a related to one class SVM. This interesting connection opens up many opportunities bridging graph theoretic algorithms and machine learning. We show that there exist graphs, which we call SVM − ϑ graphs, on which the Lov´ sz ϑ function a can be approximated well by a one-class SVM. This leads to novel use of SVM techniques for solving algorithmic problems in large graphs e.g. identifying a √ 1 planted clique of size Θ( n) in a random graph G(n, 2 ). A classic approach for this problem involves computing the ϑ function, however it is not scalable due to SDP computation. We show that the random graph with a planted clique is an example of SVM − ϑ graph. As a consequence a SVM based approach easily identiﬁes the clique in large graphs and is competitive with the state-of-the-art. We introduce the notion of common orthogonal labelling and show that it can be computed by solving a Multiple Kernel learning problem. It is further shown that such a labelling is extremely useful in identifying a large common dense subgraph in multiple graphs, which is known to be a computationally difﬁcult problem. The proposed algorithm achieves an order of magnitude scalability compared to state of the art methods. 1

5 0.81794447 339 nips-2012-The Time-Marginalized Coalescent Prior for Hierarchical Clustering

Author: Levi Boyles, Max Welling

Abstract: We introduce a new prior for use in Nonparametric Bayesian Hierarchical Clustering. The prior is constructed by marginalizing out the time information of Kingman’s coalescent, providing a prior over tree structures which we call the Time-Marginalized Coalescent (TMC). This allows for models which factorize the tree structure and times, providing two beneﬁts: more ﬂexible priors may be constructed and more efﬁcient Gibbs type inference can be used. We demonstrate this on an example model for density estimation and show the TMC achieves competitive experimental results. 1

6 0.76753438 3 nips-2012-A Bayesian Approach for Policy Learning from Trajectory Preference Queries

7 0.70054513 357 nips-2012-Unsupervised Template Learning for Fine-Grained Object Recognition

8 0.69680923 185 nips-2012-Learning about Canonical Views from Internet Image Collections

9 0.69437432 274 nips-2012-Priors for Diversity in Generative Latent Variable Models

10 0.68532878 201 nips-2012-Localizing 3D cuboids in single-view images

11 0.65984929 176 nips-2012-Learning Image Descriptors with the Boosting-Trick

12 0.64841419 8 nips-2012-A Generative Model for Parts-based Object Segmentation

13 0.6358785 210 nips-2012-Memorability of Image Regions

14 0.6217913 101 nips-2012-Discriminatively Trained Sparse Code Gradients for Contour Detection

15 0.61034626 235 nips-2012-Natural Images, Gaussian Mixtures and Dead Leaves

16 0.60950732 106 nips-2012-Dynamical And-Or Graph Learning for Object Shape Modeling and Detection

17 0.60567117 303 nips-2012-Searching for objects driven by context

18 0.58651716 1 nips-2012-3D Object Detection and Viewpoint Estimation with a Deformable 3D Cuboid Model

19 0.58464068 146 nips-2012-Graphical Gaussian Vector for Image Categorization

20 0.58020663 168 nips-2012-Kernel Latent SVM for Visual Recognition