nips nips2003 nips2003-156 knowledge-graph by maker-knowledge-mining

156 nips-2003-Phonetic Speaker Recognition with Support Vector Machines

Source: pdf

Author: William M. Campbell, Joseph P. Campbell, Douglas A. Reynolds, Douglas A. Jones, Timothy R. Leek

Abstract: A recent area of signiﬁcant progress in speaker recognition is the use of high level features—idiolect, phonetic relations, prosody, discourse structure, etc. A speaker not only has a distinctive acoustic sound but uses language in a characteristic manner. Large corpora of speech data available in recent years allow experimentation with long term statistics of phone patterns, word patterns, etc. of an individual. We propose the use of support vector machines and term frequency analysis of phone sequences to model a given speaker. To this end, we explore techniques for text categorization applied to the problem. We derive a new kernel based upon a linearization of likelihood ratio scoring. We introduce a new phone-based SVM speaker recognition approach that halves the error rate of conventional phone-based approaches.

Reference: text

Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 edu Abstract A recent area of signiﬁcant progress in speaker recognition is the use of high level features—idiolect, phonetic relations, prosody, discourse structure, etc. [sent-13, score-0.652]

2 A speaker not only has a distinctive acoustic sound but uses language in a characteristic manner. [sent-14, score-0.569]

3 Large corpora of speech data available in recent years allow experimentation with long term statistics of phone patterns, word patterns, etc. [sent-15, score-0.645]

4 We propose the use of support vector machines and term frequency analysis of phone sequences to model a given speaker. [sent-17, score-0.517]

5 We derive a new kernel based upon a linearization of likelihood ratio scoring. [sent-19, score-0.208]

6 We introduce a new phone-based SVM speaker recognition approach that halves the error rate of conventional phone-based approaches. [sent-20, score-0.543]

7 1 Introduction We consider the problem of text-independent speaker veriﬁcation. [sent-21, score-0.437]

8 Traditional speaker recognition systems use features based upon the spectral content (e. [sent-23, score-0.598]

9 For instance, a system relying only on acoustics might have difﬁculty conﬁrming that an individual speaking on a land-line telephone is the the same as an individual speaking on a cell phone [3]. [sent-30, score-0.527]

10 Second, traditional systems also rely upon seemingly different methods than human listeners [4]. [sent-31, score-0.143]

11 Human listeners are aware of prosody, word choice, pronunciation, accent, and other speech habits (laughs, etc. [sent-32, score-0.158]

12 An exciting area of recent development pioneered by Doddington [5] is the use of “high level” features for speaker recognition. [sent-35, score-0.463]

13 In Doddington’s idiolect work, word N -grams from conversations were used to characterize a particular speaker. [sent-36, score-0.118]

14 More recent systems have used a variety of approaches involving phone sequences [6], pronunciation modeling [7], and prosody [8]. [sent-37, score-0.565]

15 For this paper, we concentrate on the use of phone sequences [6]. [sent-38, score-0.467]

16 The processing for this type of system uses acoustic information to obtain sequences of phones for a given conversation and then discards the acoustic waveform. [sent-39, score-0.711]

17 Thus, processing is done at the level of terms (symbols) consisting of, for example, phones or phone N -grams. [sent-40, score-0.575]

18 In Section 2, we discuss the NIST extended data speaker recognition task. [sent-42, score-0.569]

19 1, we present a method for obtaining a phone stream. [sent-44, score-0.433]

20 2 shows the structure of the SVM phonetic speaker recognition system. [sent-46, score-0.652]

21 Section 4 discusses how we construct a kernel for speaker recognition using term weighting techniques for document classiﬁcation. [sent-47, score-0.712]

22 We derive a new kernel based upon a linearization of a likelihood ratio. [sent-48, score-0.159]

23 Finally, Section 5 shows the applications of our methods and illustrates the dramatic improvement in performance possible over standard phone-based N -gram speaker recognition methods. [sent-49, score-0.543]

24 2 The NIST extended data task Experiments for the phone-based speaker recognition experiments were performed based upon the NIST 2003 extended data task [9]. [sent-50, score-0.685]

25 Each potential training utterance in the NIST extended data task consisted of a conversation side that was nominally of length 5 minutes recorded over a land-line telephone. [sent-52, score-0.546]

26 Each conversation side consisted of a speaker having a conversation on a topic selected by an automatic operator; conversations were typically between unfamiliar individuals. [sent-53, score-1.44]

27 For training, a given split contains speakers to be recognized (target speakers) and impostor speakers; the remaining splits could be used to construct models describing the statistics of the general population—a “background” model. [sent-56, score-0.128]

28 Training a speaker model was performed by using statistics from 1, 2, 4, 8, or 16 conversation sides. [sent-58, score-0.915]

29 This simulated a situation where the system could use longer term statistics and become “familiar” with the individual; this longer term training allows one to explore techniques which might mimic more what human listeners perceive about an individual’s speech habits. [sent-59, score-0.358]

30 A large number of speakers and tests were available; for instance, for 8 conversation training, 739 distinct target speakers were used and 11, 171 true trials and 17, 736 false trials were performed. [sent-60, score-0.68]

31 1 Phone Sequence Extraction Phone sequence extraction for the speaker recognition process is performed using the phone recognition system (PPRLM) designed by Zissman [11] for language identiﬁcation. [sent-63, score-1.277]

32 Each phone is modeled in a gender-dependent context-independent (monophone) manner using a three-state hidden Markov model (HMM). [sent-65, score-0.433]

33 Phone recognition is performed with a Viterbi search using a fully connected null-grammar network of monophones; note that no explicit language model is used in the decoding process. [sent-66, score-0.243]

34 The phone recognition system encompassed multiple languages—English (EG), German (GE), Japanese (JA), Mandarin (MA), and Spanish (SP). [sent-67, score-0.571]

35 In earlier phonetic speaker recog- nition work [6], it was found that these multiple streams were useful for improving accuracy. [sent-68, score-0.583]

36 The phone recognizers were trained using the OGI multilanguage corpus which had been hand labeled by native speakers. [sent-69, score-0.528]

37 After a “raw” phone stream was obtained from PPRLM, additional processing was performed to increase robustness. [sent-70, score-0.499]

38 First, speech activity detection marks were used to eliminate phone segments where no speech was present. [sent-71, score-0.619]

39 The idea in this case is to capture some of the ways in which a speaker interacts with others—does the speaker pause frequently, etc. [sent-74, score-0.874]

40 Finally, all phones with short duration were removed (less than 3 frames). [sent-76, score-0.142]

41 2 Phonetic SVM System Our system for speaker recognition using phone sequences is shown in Figure 1. [sent-78, score-1.042]

42 A phone sequence is derived using each of the language phone recognizers and then post-processing is performed on the sequence as discussed in Section 3. [sent-83, score-1.104]

43 After this step, the phone sequence is vectorized by computing frequencies of N -grams—this process will be discussed in Section 4. [sent-85, score-0.487]

44 This vector is then introduced into a SVM using the speaker’s model in the appropriate language and a score per language is produced. [sent-88, score-0.246]

45 These scores are then fused using a linear weighting to produce a ﬁnal score for the test utterance. [sent-90, score-0.21]

46 The ﬁnal score is compared to a threshold and a reject or accept decision is made based upon whether the score was below or above the threshold, respectively. [sent-91, score-0.139]

47 An interesting aspect of the system in Figure 1 is that it uses multiple streams of phones in different languages. [sent-92, score-0.211]

48 First, the system can be used without modiﬁcation for speakers in multiple languages. [sent-94, score-0.135]

49 Second, although not obvious, from experimentation we show that phone streams different from the language being spoken provide complimentary information for speaker recognition. [sent-95, score-1.032]

50 A third point is that the system may also work in other languages not represented in the phone streams. [sent-97, score-0.53]

51 It is known that in the case of language identiﬁcation, language characterization can be performed even if a phone recognizer is not available in that particular language [11]. [sent-98, score-0.844]

52 We treat each conversation side in the corpus as a “document. [sent-100, score-0.521]

53 ” From each of these conversation sides we derive a single (sparse) vector of weighted probabilities. [sent-101, score-0.517]

54 The conversations sides not in the current split (see Section 2) are used as a background. [sent-104, score-0.184]

55 That is, all conversation sides not in the current split are used as the class for SVM target value of −1. [sent-105, score-0.542]

56 Note that this strategy ensures that speakers that are used as impostors are “unseen” in the training data. [sent-106, score-0.148]

57 4 Kernel Construction Possibly the most important aspect of the process of phonetic speaker recognition is the selection of the kernel for the SVM. [sent-107, score-0.685]

58 Of particular interest is a kernel which will preserve the identity cues a particular individual might present in their phone sequence. [sent-108, score-0.466]

59 Our ﬁrst step of kernel construction is the selection of probabilities to describe the phone stream. [sent-110, score-0.504]

60 For a phone sequence, we produce N -grams by the standard transformation of the stream; e. [sent-113, score-0.433]

61 , for bigrams (2-grams) the sequence of phones, t1 , t2 , . [sent-115, score-0.157]

62 , tn , is transformed to the sequence of bigrams of phones t1 t2 , t2 t3 , . [sent-118, score-0.361]

63 That is, suppose we are considering unigrams and bigrams of phones, and the unique unigrams and bigrams are designated d1 , . [sent-123, score-0.394]

64 These probabilities then become entries in a vector v describing the conversation side v = [p(d1 ) . [sent-130, score-0.513]

65 (2) In general, the vector v will be sparse since the conversation side will not contain all potential unigrams, bigrams, etc. [sent-137, score-0.475]

66 A second step of kernel construction is the selection of the “document component” of term weighting for the entries of the vector v in (2) and the normalization of the resulting vector. [sent-138, score-0.169]

67 By term weighting we mean that for each entry, vi , of the vector v, we multiply by a “collection” (or background) component, wi , for that entry. [sent-139, score-0.136]

68 From the background section of the corpus we compute the frequency of a particular N -gram using conversation sides as the item analogous to a document. [sent-143, score-0.667]

69 , if we let DF (t i ) be the number of conversation sides where a particular N -gram, ti , is observed, then our resulting term-weighted vector has entries vi log # of conversation sides in background DF (ti ) . [sent-146, score-1.229]

70 An alternate method of term weighting may be derived using the following strategy. [sent-149, score-0.136]

71 Suppose that we have two conversation sides from speakers, spk1 and spk2 . [sent-150, score-0.517]

72 Further suppose that the sequence of N -grams (for ﬁxed N ) in each conversation side is t1 ,t2 , . [sent-151, score-0.501]

73 We can build a “model” based upon the conversation sides for each speaker consisting of the probability of N -grams, p(di |spkj ). [sent-162, score-1.009]

74 We then compute the likelihood ratio of the ﬁrst conversation side as is standard in veriﬁcation problems [1]; a linearization of the likelihood ratio computation will serve as the kernel. [sent-163, score-0.676]

76 Note that the strategy used for constructing a kernel is part of a general process of ﬁnding kernels based upon training on one instance and testing upon another instance [2]. [sent-173, score-0.188]

77 Five language phone recognizers were used—English (EG), German (GE), Japanese (JA), Mandarin (MA), and Spanish (SP). [sent-177, score-0.584]

78 The resulting phone sequences were vectorized as unigram and bigram probabilities (2). [sent-178, score-0.625]

79 Both the standard TFIDF term weighting (3) and the log-likelihood ratio (TFLLR) term weighting (6) methods were used. [sent-179, score-0.321]

80 Comparisons of performance for different strategies were typically done with 8 conversation training and English phone streams since these were representative of overall performance. [sent-183, score-0.958]

81 Table 1: Comparison of different term weighting strategies, English only scores, 8 conversation training Term Weighting Method EER TFIDF 7. [sent-184, score-0.624]

82 Table 1 shows the results for two different weightings, TFIDF (3) and TFLLR (6), using English phones only and 8 training conversations. [sent-187, score-0.187]

83 We next considered the effect on performance of the language of the phone stream for the 8 conversation training case. [sent-193, score-1.054]

84 Figure 2 shows a DET plot (a ROC plot with a special scale [15]) with results corresponding to the 5 language phone streams. [sent-194, score-0.535]

85 The best performing system in the ﬁgure is an equal fusion of all scores from the SVM outputs for each language and has an EER of 3. [sent-195, score-0.193]

86 1 that other languages do provide signiﬁcant speaker recognition information. [sent-199, score-0.608]

87 5 1 2 5 10 20 False Alarm probability (in %) 40 Figure 2: DET plot for the 8 conversation training case with varying languages and TFLLR weighting. [sent-206, score-0.553]

88 We also see that even for 1 training conversation, the SVM system provides signiﬁcant speaker characterization ability. [sent-225, score-0.536]

89 Figure 3 shows DET plots comparing the performance of the standard log likelihood ratio method [6] to our new SVM method using the TFLLR weighting. [sent-226, score-0.113]

90 We show log likelihood results based on both bigrams and trigrams; in addition, a slightly more complex model involving discounting of probabilities is used. [sent-227, score-0.233]

91 6 Conclusions and future work An exciting new application of SVMs to speaker recognition was shown. [sent-233, score-0.569]

92 By computing frequencies of phones in conversations, speaker characterization was performed. [sent-234, score-0.601]

93 A new kernel was introduced based on the standard method of log likelihood ratio scoring. [sent-235, score-0.146]

94 Campbell, “Generalized linear discriminant sequence kernels for speaker recognition,” in Proceedings of the International Conference on Acoustics Speech and Signal Processing, 2002, pp. [sent-249, score-0.463]

95 O’Leary, “Estimation of handset nonlinearity with application to speaker recognition,” IEEE Trans. [sent-257, score-0.437]

96 Crystal, “Speaker veriﬁcation by human listeners: Experiments comparing human and machine performance using the NIST 1998 speaker evaluation data,” Digital Signal Processing, vol. [sent-263, score-0.483]

97 Godfrey, and Jaime Hern´ ndez-Cordero, “Gender-dependent phonetic refraction for speaker recognition,” in Proa ceedings of the International Conference on Acoustics Speech and Signal Processing, 2002, pp. [sent-273, score-0.546]

98 Campbell, “Conditional pronunciation aˇ i a modeling in speaker detection,” in Proceedings of the International Conference on Acoustics Speech and Signal Processing, 2003, pp. [sent-279, score-0.486]

99 Godfrey, “Modeling prosodic dynamics for speaker recognition,” in Proceedings of the International Conference on Acoustics Speech and Signal Processing, 2003, pp. [sent-283, score-0.437]

100 Martin, “The NIST year 2003 speaker recognition evaluation plan,” http://www. [sent-287, score-0.543]

similar papers computed by tfidf model

tfidf for this paper:

wordName wordTfidf (topN-words)

[('conversation', 0.443), ('speaker', 0.437), ('phone', 0.433), ('dj', 0.228), ('phones', 0.142), ('bigrams', 0.131), ('tfllr', 0.115), ('phonetic', 0.109), ('nist', 0.109), ('svm', 0.109), ('recognition', 0.106), ('background', 0.104), ('speakers', 0.103), ('language', 0.102), ('eer', 0.1), ('speech', 0.093), ('dm', 0.091), ('weighting', 0.086), ('conversations', 0.085), ('ge', 0.085), ('sides', 0.074), ('english', 0.073), ('tfidf', 0.073), ('doddington', 0.066), ('ja', 0.066), ('unigrams', 0.066), ('eg', 0.065), ('listeners', 0.065), ('reynolds', 0.065), ('languages', 0.065), ('tn', 0.062), ('acoustics', 0.062), ('ti', 0.059), ('sp', 0.057), ('upon', 0.055), ('campbell', 0.054), ('term', 0.05), ('eers', 0.049), ('fused', 0.049), ('pprlm', 0.049), ('pronunciation', 0.049), ('prosody', 0.049), ('recognizers', 0.049), ('trigrams', 0.049), ('unigram', 0.049), ('ratio', 0.049), ('recognizer', 0.048), ('corpus', 0.046), ('corpora', 0.046), ('iv', 0.046), ('training', 0.045), ('alarm', 0.043), ('postprocessing', 0.043), ('bigram', 0.043), ('weightings', 0.043), ('score', 0.042), ('tk', 0.041), ('det', 0.041), ('linearization', 0.039), ('probabilities', 0.038), ('streams', 0.037), ('miss', 0.036), ('performed', 0.035), ('sequences', 0.034), ('kernel', 0.033), ('scores', 0.033), ('godfrey', 0.033), ('idiolect', 0.033), ('llr', 0.033), ('mandarin', 0.033), ('przybocki', 0.033), ('quatieri', 0.033), ('svmtorch', 0.033), ('zissman', 0.033), ('side', 0.032), ('likelihood', 0.032), ('system', 0.032), ('log', 0.032), ('false', 0.031), ('stream', 0.031), ('di', 0.031), ('acoustic', 0.03), ('idf', 0.028), ('vectorized', 0.028), ('spanish', 0.028), ('veri', 0.028), ('eurospeech', 0.026), ('cepstral', 0.026), ('exciting', 0.026), ('united', 0.026), ('fusion', 0.026), ('silence', 0.026), ('sequence', 0.026), ('extended', 0.026), ('split', 0.025), ('human', 0.023), ('japanese', 0.023), ('douglas', 0.023), ('experimentation', 0.023), ('characterization', 0.022)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 1.0000005 156 nips-2003-Phonetic Speaker Recognition with Support Vector Machines

Author: William M. Campbell, Joseph P. Campbell, Douglas A. Reynolds, Douglas A. Jones, Timothy R. Leek

2 0.24051064 9 nips-2003-A Kullback-Leibler Divergence Based Kernel for SVM Classification in Multimedia Applications

Author: Pedro J. Moreno, Purdy P. Ho, Nuno Vasconcelos

Abstract: Over the last years signiﬁcant efforts have been made to develop kernels that can be applied to sequence data such as DNA, text, speech, video and images. The Fisher Kernel and similar variants have been suggested as good ways to combine an underlying generative model in the feature space and discriminant classiﬁers such as SVM’s. In this paper we suggest an alternative procedure to the Fisher kernel for systematically ﬁnding kernel functions that naturally handle variable length sequence data in multimedia domains. In particular for domains such as speech and images we explore the use of kernel functions that take full advantage of well known probabilistic models such as Gaussian Mixtures and single full covariance Gaussian models. We derive a kernel distance based on the Kullback-Leibler (KL) divergence between generative models. In effect our approach combines the best of both generative and discriminative methods and replaces the standard SVM kernels. We perform experiments on speaker identiﬁcation/veriﬁcation and image classiﬁcation tasks and show that these new kernels have the best performance in speaker veriﬁcation and mostly outperform the Fisher kernel based SVM’s and the generative classiﬁers in speaker identiﬁcation and image classiﬁcation. 1

3 0.16919874 60 nips-2003-Eigenvoice Speaker Adaptation via Composite Kernel Principal Component Analysis

Author: James T. Kwok, Brian Mak, Simon Ho

Abstract: Eigenvoice speaker adaptation has been shown to be effective when only a small amount of adaptation data is available. At the heart of the method is principal component analysis (PCA) employed to ﬁnd the most important eigenvoices. In this paper, we postulate that nonlinear PCA, in particular kernel PCA, may be even more effective. One major challenge is to map the feature-space eigenvoices back to the observation space so that the state observation likelihoods can be computed during the estimation of eigenvoice weights and subsequent decoding. Our solution is to compute kernel PCA using composite kernels, and we will call our new method kernel eigenvoice speaker adaptation. On the TIDIGITS corpus, we found that compared with a speaker-independent model, our kernel eigenvoice adaptation method can reduce the word error rate by 28–33% while the standard eigenvoice approach can only match the performance of the speaker-independent model. 1

4 0.094671048 162 nips-2003-Probabilistic Inference of Speech Signals from Phaseless Spectrograms

Author: Kannan Achan, Sam T. Roweis, Brendan J. Frey

Abstract: Many techniques for complex speech processing such as denoising and deconvolution, time/frequency warping, multiple speaker separation, and multiple microphone analysis operate on sequences of short-time power spectra (spectrograms), a representation which is often well-suited to these tasks. However, a signiﬁcant problem with algorithms that manipulate spectrograms is that the output spectrogram does not include a phase component, which is needed to create a time-domain signal that has good perceptual quality. Here we describe a generative model of time-domain speech signals and their spectrograms, and show how an efﬁcient optimizer can be used to ﬁnd the maximum a posteriori speech signal, given the spectrogram. In contrast to techniques that alternate between estimating the phase and a spectrally-consistent signal, our technique directly infers the speech signal, thus jointly optimizing the phase and a spectrally-consistent signal. We compare our technique with a standard method using signal-to-noise ratios, but we also provide audio ﬁles on the web for the purpose of demonstrating the improvement in perceptual quality that our technique offers. 1

5 0.091085881 5 nips-2003-A Classification-based Cocktail-party Processor

Author: Nicoleta Roman, Deliang Wang, Guy J. Brown

Abstract: At a cocktail party, a listener can selectively attend to a single voice and filter out other acoustical interferences. How to simulate this perceptual ability remains a great challenge. This paper describes a novel supervised learning approach to speech segregation, in which a target speech signal is separated from interfering sounds using spatial location cues: interaural time differences (ITD) and interaural intensity differences (IID). Motivated by the auditory masking effect, we employ the notion of an ideal time-frequency binary mask, which selects the target if it is stronger than the interference in a local time-frequency unit. Within a narrow frequency band, modifications to the relative strength of the target source with respect to the interference trigger systematic changes for estimated ITD and IID. For a given spatial configuration, this interaction produces characteristic clustering in the binaural feature space. Consequently, we perform pattern classification in order to estimate ideal binary masks. A systematic evaluation in terms of signal-to-noise ratio as well as automatic speech recognition performance shows that the resulting system produces masks very close to ideal binary ones. A quantitative comparison shows that our model yields significant improvement in performance over an existing approach. Furthermore, under certain conditions the model produces large speech intelligibility improvements with normal listeners. 1 In t ro d u c t i o n The perceptual ability to detect, discriminate and recognize one utterance in a background of acoustic interference has been studied extensively under both monaural and binaural conditions [1, 2, 3]. The human auditory system is able to segregate a speech signal from an acoustic mixture using various cues, including fundamental frequency (F0), onset time and location, in a process that is known as auditory scene analysis (ASA) [1]. F0 is widely used in computational ASA systems that operate upon monaural input – however, systems that employ only this cue are limited to voiced speech [4, 5, 6]. Increased speech intelligibility in binaural listening compared to the monaural case has prompted research in designing cocktail-party processors based on spatial cues [7, 8, 9]. Such a system can be applied to, among other things, enhancing speech recognition in noisy environments and improving binaural hearing aid design. In this study, we propose a sound segregation model using binaural cues extracted from the responses of a KEMAR dummy head that realistically simulates the filtering process of the head, torso and external ear. A typical approach for signal reconstruction uses a time-frequency (T-F) mask: T-F units are weighted selectively in order to enhance the target signal. Here, we employ an ideal binary mask [6], which selects the T-F units where the signal energy is greater than the noise energy. The ideal mask notion is motivated by the human auditory masking phenomenon, in which a stronger signal masks a weaker one in the same critical band. In addition, from a theoretical ASA perspective, an ideal binary mask gives a performance ceiling for all binary masks. Moreover, such masks have been recently shown to provide a highly effective front-end for robust speech recognition [10]. We show for mixtures of multiple sound sources that there exists a strong correlation between the relative strength of target and interference and estimated ITD/IID, resulting in a characteristic clustering across frequency bands. Consequently, we employ a nonparametric classification method to determine decision regions in the joint ITDIID feature space that correspond to an optimal estimate for an ideal mask. Related models for estimating target masks through clustering have been proposed previously [11, 12]. Notably, the experimental results by Jourjine et al. [12] suggest that speech signals in a multiple-speaker condition obey to a large extent disjoint orthogonality in time and frequency. That is, at most one source has a nonzero energy at a specific time and frequency. Such models, however, assume input directly from microphone recordings and head-related filtering is not considered. Simulation of human binaural hearing introduces different constraints as well as clues to the problem. First, both ITD and IID should be utilized since IID is more reliable at higher frequencies than ITD. Second, frequency-dependent combinations of ITD and IID arise naturally for a fixed spatial configuration. Consequently, channel-dependent training should be performed for each frequency band. The rest of the paper is organized as follows. The next section contains the architecture of the model and describes our method for azimuth localization. Section 3 is devoted to ideal binary mask estimation, which constitutes the core of the model. Section 4 presents the performance of the system and a quantitative comparison with the Bodden [7] model. Section 5 concludes our paper. 2 M od el a rch i t ect u re a n d a zi mu t h locali zat i o n Our model consists of the following stages: 1) a model of the auditory periphery; 2) frequency-dependent ITD/IID extraction and azimuth localization; 3) estimation of an ideal binary mask. The input to our model is a mixture of two or more signals presented at different, but fixed, locations. Signals are sampled at 44.1 kHz. We follow a standard procedure for simulating free-field acoustic signals from monaural signals (no reverberations are modeled). Binaural signals are obtained by filtering the monaural signals with measured head-related transfer functions (HRTF) from a KEMAR dummy head [13]. HRTFs introduce a natural combination of ITD and IID into the signals that is extracted in the subsequent stages of the model. To simulate the auditory periphery we use a bank of 128 gammatone filters in the range of 80 Hz to 5 kHz as described in [4]. In addition, the gains of the gammatone filters are adjusted in order to simulate the middle ear transfer function. In the final step of the peripheral model, the output of each gammatone filter is half-wave rectified in order to simulate firing rates of the auditory nerve. Saturation effects are modeled by taking the square root of the signal. Current models of azimuth localization almost invariably start with Jeffress’s crosscorrelation mechanism. For all frequency channels, we use the normalized crosscorrelation computed at lags equally distributed in the plausible range from –1 ms to 1 ms using an integration window of 20 ms. Frequency-dependent nonlinear transformations are used to map the time-delay axis onto the azimuth axis resulting in a cross-correlogram structure. In addition, a ‘skeleton’ cross-correlogram is formed by replacing the peaks in the cross-correlogram with Gaussians of narrower widths that are inversely proportional to the channel center frequency. This results in a sharpening effect, similar in principle to lateral inhibition. Assuming fixed sources, multiple locations are determined as peaks after summating the skeleton cross-correlogram across frequency and time. The number of sources and their locations computed here, as well as the target source location, feed to the next stage. 3 B i n a ry ma s k est i mat i on The objective of this stage of the model is to develop an efficient mechanism for estimating an ideal binary mask based on observed patterns of extracted ITD and IID features. Our theoretical analysis for two-source interactions in the case of pure tones shows relatively smooth changes for ITD and IID with the relative strength R between the two sources in narrow frequency bands [14]. More specifically, when the frequencies vary uniformly in a narrow band the derived mean values of ITD/IID estimates vary monotonically with respect to R. To capture this relationship in the context of real signals, statistics are collected for individual spatial configurations during training. We employ a training corpus consisting of 10 speech utterances from the TIMIT database (see [14] for details). In the two-source case, we divide the corpus in two equal sets: target and interference. In the three-source case, we select 4 signals for the target set and 2 interfering sets of 3 signals each. For all frequency channels, local estimates of ITD, IID and R are based on 20-ms time frames with 10 ms overlap between consecutive time frames. In order to eliminate the multi-peak ambiguity in the cross-correlation function for mid- and high-frequency channels, we use the following strategy. We compute ITDi as the peak location of the cross-correlation in the range 2π / ω i centered at the target ITD, where ω i indicates the center frequency of the ith channel. On the other hand, IID and R are computed as follows: ∑ t s i2 (t )     Ri = ∑ ∑ t li2 (t ) , t s i2 (t ) + ∑ ∑ t ri2 (t ) t ni2 (t )     IIDi = 20 log10 where l i and ri refer to the left and right peripheral output of the ith channel, respectively, s i refers to the output for the target signal, and ni that for the acoustic interference. In computing IIDi , we use 20 instead of 10 in order to compensate for the square root operation in the peripheral model. Fig. 1 shows empirical results obtained for a two-source configuration on the training corpus. The data exhibits a systematic shift for both ITD and IID with respect to the relative strength R. Moreover, the theoretical mean values obtained in the case of pure tones [14] match the empirical ones very well. This observation extends to multiple-source scenarios. As an example, Fig. 2 displays histograms that show the relationship between R and both ITD (Fig. 2A) and IID (Fig. 2B) for a three-source situation. Note that the interfering sources introduce systematic deviations for the binaural cues. Consider a worst case: the target is silent and two interferences have equal energy in a given T-F unit. This results in binaural cues indicating an auditory event at half of the distance between the two interference locations; for Fig. 2, it is 0° - the target location. However, the data in Fig. 2 has a low probability for this case and shows instead a clustering phenomenon, suggesting that in most cases only one source dominates a T-F unit. B 1 1 R R A theoretical empirical 0 -1 theoretical empirical 0 -15 1 ITD (ms) 15 IID (dB) Figure 1. Relationship between ITD/IID and relative strength R for a two-source configuration: target in the median plane and interference on the right side at 30°. The solid curve shows the theoretical mean and the dash curve shows the data mean. A: The scatter plot of ITD and R estimates for a filter channel with center frequency 500 Hz. B: Results for IID for a filter channel with center frequency 2.5 kHz. A B 1 C 10 1 IID s) 0.5 0 -10 IID (d B) 10 ) (dB R R 0 -0.5 m ITD ( -10 -0.5 m ITD ( s) 0.5 Figure 2. Relationship between ITD/IID and relative strength R for a three-source configuration: target in the median plane and interference at -30° and 30°. Statistics are obtained for a channel with center frequency 1.5 kHz. A: Histogram of ITD and R samples. B: Histogram of IID and R samples. C: Clustering in the ITD-IID space. By displaying the information in the joint ITD-IID space (Fig. 2C), we observe location-based clustering of the binaural cues, which is clearly marked by strong peaks that correspond to distinct active sources. There exists a tradeoff between ITD and IID across frequencies, where ITD is most salient at low frequencies and IID at high frequencies [2]. But a fixed cutoff frequency that separates the effective use of ITD and IID does not exist for different spatial configurations. This motivates our choice of a joint ITD-IID feature space that optimizes the system performance across different configurations. Differential training seems necessary for different channels given that there exist variations of ITD and, especially, IID values for different center frequencies. Since the goal is to estimate an ideal binary mask, we focus on detecting decision regions in the 2-dimensional ITD-IID space for individual frequency channels. Consequently, supervised learning techniques can be applied. For the ith channel, we test the following two hypotheses. The first one is H 1 : target is dominant or Ri > 0.5 , and the second one is H 2 : interference is dominant or Ri < 0.5 . Based on the estimates of the bivariate densities p( x | H 1 ) and p( x | H 2 ) the classification is done by the maximum a posteriori decision rule: p( H 1 ) p( x | H 1 ) > p( H 2 ) p( x | H 2 ) . There exist a plethora of techniques for probability density estimation ranging from parametric techniques (e.g. mixture of Gaussians) to nonparametric ones (e.g. kernel density estimators). In order to completely characterize the distribution of the data we use the kernel density estimation method independently for each frequency channel. One approach for finding smoothing parameters is the least-squares crossvalidation method, which is utilized in our estimation. One cue not employed in our model is the interaural time difference between signal envelopes (IED). Auditory models generally employ IED in the high-frequency range where the auditory system becomes gradually insensitive to ITD. We have compared the performance of the three binaural cues: ITD, IID and IED and have found no benefit for using IED in our system after incorporating ITD and IID [14]. 4 Pe rfo rmanc e an d c omp arison The performance of a segregation system can be assessed in different ways, depending on intended applications. To extensively evaluate our model, we use the following three criteria: 1) a signal-to-noise (SNR) measure using the original target as signal; 2) ASR rates using our model as a front-end; and 3) human speech intelligibility tests. To conduct the SNR evaluation a segregated signal is reconstructed from a binary mask using a resynthesis method described in [5]. To quantitatively assess system performance, we measure the SNR using the original target speech as signal: ∑ t 2 s o (t ) ∑ SNR = 10 log 10 (s o (t ) − s e (t ))2 t where s o (t ) represents the resynthesized original speech and s e (t ) the reconstructed speech from an estimated mask. One can measure the initial SNR by replacing the denominator with s N (t ) , the resynthesized original interference. Fig. 3 shows the systematic results for two-source scenarios using the Cooke corpus [4], which is commonly used in sound separation studies. The corpus has 100 mixtures obtained from 10 speech utterances mixed with 10 types of intrusion. We compare the SNR gain obtained by our model against that obtained using the ideal binary mask across different noise types. Excellent results are obtained when the target is close to the median plane for an azimuth separation as small as 5°. Performance degrades when the target source is moved to the side of the head, from an average gain of 13.7 dB for the target in the median plane (Fig. 3A) to 1.7 dB when target is at 80° (Fig. 3B). When spatial separation increases the performance improves even for side targets, to an average gain of 14.5 dB in Fig. 3C. This performance profile is in qualitative agreement with experimental data [2]. Fig. 4 illustrates the performance in a three-source scenario with target in the median plane and two interfering sources at –30° and 30°. Here 5 speech signals from the Cooke corpus form the target set and the other 5 form one interference set. The second interference set contains the 10 intrusions. The performance degrades compared to the two-source situation, from an average SNR of about 12 dB to 4.1 dB. However, the average SNR gain obtained is approximately 11.3 dB. This ability of our model to segregate mixtures of more than two sources differs from blind source separation with independent component analysis. In order to draw a quantitative comparison, we have implemented Bodden’s cocktail-party processor using the same 128-channel gammatone filterbank [7]. The localization stage of this model uses an extended cross-correlation mechanism based on contralateral inhibition and it adapts to HRTFs. The separation stage of the model is based on estimation of the weights for a Wiener filter as the ratio between a desired excitation and an actual one. Although the Bodden model is more flexible by incorporating aspects of the precedence effect into the localization stage, the estimation of Wiener filter weights is less robust than our binary estimation of ideal masks. Shown in Fig. 5, our model shows a considerable improvement over the Bodden system, producing a 3.5 dB average improvement. A B C 20 20 10 10 10 0 0 0 -10 SNR (dB) 20 -10 -10 N0 N1 N2 N3 N4 N5 N6 N7 N8 N9 N0 N1 N2 N3 N4 N5 N6 N7 N8 N9 N0 N1 N2 N3 N4 N5 N6 N7 N8 N9 Figure 3. Systematic results for two-source configuration. Black bars correspond to the SNR of the initial mixture, white bars indicate the SNR obtained using ideal binary mask, and gray bars show the SNR from our model. Results are obtained for speech mixed with ten intrusion types (N0: pure tone; N1: white noise; N2: noise burst; N3: ‘cocktail party’; N4: rock music; N5: siren; N6: trill telephone; N7: female speech; N8: male speech; N9: female speech). A: Target at 0°, interference at 5°. B: Target at 80°, interference at 85°. C: Target at 60°, interference at 90°. 20 0 SNR (dB) SNR (dB) 5 -5 -10 -15 -20 10 0 -10 N0 N1 N2 N3 N4 N5 N6 N7 N8 N9 Figure 4. Evaluation for a three-source configuration: target at 0° and two interfering sources at –30° and 30°. Black bars correspond to the SNR of the initial mixture, white bars to the SNR obtained using the ideal binary mask, and gray bars to the SNR from our model. N0 N1 N2 N3 N4 N5 N6 N7 N8 N9 Figure 5. SNR comparison between the Bodden model (white bars) and our model (gray bars) for a two-source configuration: target at 0° and interference at 30°. Black bars correspond to the SNR of the initial mixture. For the ASR evaluation, we use the missing-data technique as described in [10]. In this approach, a continuous density hidden Markov model recognizer is modified such that only acoustic features indicated as reliable in a binary mask are used during decoding. Hence, it works seamlessly with the output from our speech segregation system. We have implemented the missing data algorithm with the same 128-channel gammatone filterbank. Feature vectors are obtained using the Hilbert envelope at the output of the gammatone filter. More specifically, each feature vector is extracted by smoothing the envelope using an 8-ms first-order filter, sampling at a frame-rate of 10 ms and finally log-compressing. We use the bounded marginalization method for classification [10]. The task domain is recognition of connected digits, and both training and testing are performed on acoustic features from the left ear signal using the male speaker dataset in the TIDigits database. A 100 B 100 Correctness (%) Correctness (%) Fig. 6A shows the correctness scores for a two-source condition, where the male target speaker is located at 0° and the interference is another male speaker at 30°. The performance of our model is systematically compared against the ideal masks for four SNR levels: 5 dB, 0 dB, -5 dB and –10 dB. Similarly, Fig. 6B shows the results for the three-source case with an added female speaker at -30°. The ideal mask exhibits only slight and gradual degradation in recognition performance with decreasing SNR and increasing number of sources. Observe that large improvements over baseline performance are obtained across all conditions. This shows the strong potential of applying our model to robust speech recognition. 80 60 40 20 5 dB Baseline Ideal Mask Estimated Mask 0 dB −5 dB 80 60 40 20 5 dB −10 dB Baseline Ideal Mask Estimated Mask 0 dB −5 dB −10 dB Figure 6. Recognition performance at different SNR values for original mixture (dotted line), ideal binary mask (dashed line) and estimated mask (solid line). A. Correctness score for a two-source case. B. Correctness score for a three-source case. Finally we evaluate our model on speech intelligibility with listeners with normal hearing. We use the Bamford-Kowal-Bench sentence database that contains short semantically predictable sentences [15]. The score is evaluated as the percentage of keywords correctly identified, ignoring minor errors such as tense and plurality. To eliminate potential location-based priming effects we randomly swap the locations for target and interference for different trials. In the unprocessed condition, binaural signals are produced by convolving original signals with the corresponding HRTFs and the signals are presented to a listener dichotically. In the processed condition, our algorithm is used to reconstruct the target signal at the better ear and results are presented diotically. 80 80 Keyword score (%) B100 Keyword score (%) A 100 60 40 20 0 0 dB −5 dB −10 dB 60 40 20 0 Figure 7. Keyword intelligibility score for twelve native English speakers (median values and interquartile ranges) before (white bars) and after processing (black bars). A. Two-source condition (0° and 5°). B. Three-source condition (0°, 30° and -30°). Fig. 7A gives the keyword intelligibility score for a two-source configuration. Three SNR levels are tested: 0 dB, -5 dB and –10 dB, where the SNR is computed at the better ear. Here the target is a male speaker and the interference is babble noise. Our algorithm improves the intelligibility score for the tested conditions and the improvement becomes larger as the SNR decreases (61% at –10 dB). Our informal observations suggest, as expected, that the intelligibility score improves for unprocessed mixtures when two sources are more widely separated than 5°. Fig. 7B shows the results for a three-source configuration, where our model yields a 40% improvement. Here the interfering sources are one female speaker and another male speaker, resulting in an initial SNR of –10 dB at the better ear. 5 C onclu si on We have observed systematic deviations of the ITD and IID cues with respect to the relative strength between target and acoustic interference, and configuration-specific clustering in the joint ITD-IID feature space. Consequently, supervised learning of binaural patterns is employed for individual frequency channels and different spatial configurations to estimate an ideal binary mask that cancels acoustic energy in T-F units where interference is stronger. Evaluation using both SNR and ASR measures shows that the system estimates ideal binary masks very well. A comparison shows a significant improvement in performance over the Bodden model. Moreover, our model produces substantial speech intelligibility improvements for two and three source conditions. A c k n ow l e d g me n t s This research was supported in part by an NSF grant (IIS-0081058) and an AFOSR grant (F49620-01-1-0027). A preliminary version of this work was presented in 2002 ICASSP. References [1] A. S. Bregman, Auditory Scene Analysis, Cambridge, MA: MIT press, 1990. [2] J. Blauert, Spatial Hearing - The Psychophysics of Human Sound Localization, Cambridge, MA: MIT press, 1997. [3] A. Bronkhorst, “The cocktail party phenomenon: a review of research on speech intelligibility in multiple-talker conditions,” Acustica, vol. 86, pp. 117-128, 2000. [4] M. P. Cooke, Modeling Auditory Processing and Organization, Cambridge, U.K.: Cambridge University Press, 1993. [5] G. J. Brown and M. P. Cooke, “Computational auditory scene analysis,” Computer Speech and Language, vol. 8, pp. 297-336, 1994. [6] G. Hu and D. L. Wang, “Monaural speech separation,” Proc. NIPS, 2002. [7] M. Bodden, “Modeling human sound-source localization and the cocktail-party-effect,” Acta Acoustica, vol. 1, pp. 43-55, 1993. [8] C. Liu et al., “A two-microphone dual delay-line approach for extraction of a speech sound in the presence of multiple interferers,” J. Acoust. Soc. Am., vol. 110, pp. 32183230, 2001. [9] T. Whittkop and V. Hohmann, “Strategy-selective noise reduction for binaural digital hearing aids,” Speech Comm., vol. 39, pp. 111-138, 2003. [10] M. P. Cooke, P. Green, L. Josifovski and A. Vizinho, “Robust automatic speech recognition with missing and unreliable acoustic data,” Speech Comm., vol. 34, pp. 267285, 2001. [11] H. Glotin, F. Berthommier and E. Tessier, “A CASA-labelling model using the localisation cue for robust cocktail-party speech recognition,” Proc. EUROSPEECH, pp. 2351-2354, 1999. [12] A. Jourjine, S. Rickard and O. Yilmaz, “Blind separation of disjoint orthogonal signals: demixing N sources from 2 mixtures,” Proc. ICASSP, 2000. [13] W. G. Gardner and K. D. Martin, “HRTF measurements of a KEMAR dummy-head microphone,” MIT Media Lab Technical Report #280, 1994. [14] N. Roman, D. L. Wang and G. J. Brown, “Speech segregation based on sound localization,” J. Acoust. Soc. Am., vol. 114, pp. 2236-2252, 2003. [15] J. Bench and J. Bamford, Speech Hearing Tests and the Spoken Language of HearingImpaired Children, London: Academic press, 1979.

6 0.063587591 191 nips-2003-Unsupervised Context Sensitive Language Acquisition from a Large Corpus

7 0.054163273 144 nips-2003-One Microphone Blind Dereverberation Based on Quasi-periodicity of Speech Signals

8 0.053884964 1 nips-2003-1-norm Support Vector Machines

9 0.049319949 100 nips-2003-Laplace Propagation

10 0.047464635 159 nips-2003-Predicting Speech Intelligibility from a Population of Neurons

11 0.045191105 186 nips-2003-Towards Social Robots: Automatic Evaluation of Human-Robot Interaction by Facial Expression Classification

12 0.04309532 113 nips-2003-Learning with Local and Global Consistency

13 0.04288014 160 nips-2003-Prediction on Spike Data Using Kernel Algorithms

14 0.041437641 70 nips-2003-Fast Algorithms for Large-State-Space HMMs with Applications to Web Usage Analysis

15 0.038624987 135 nips-2003-Necessary Intransitive Likelihood-Ratio Classifiers

16 0.037009001 134 nips-2003-Near-Minimax Optimal Classification with Dyadic Classification Trees

17 0.036561295 177 nips-2003-Simplicial Mixtures of Markov Chains: Distributed Modelling of Dynamic User Profiles

18 0.035372719 111 nips-2003-Learning the k in k-means

19 0.03511776 125 nips-2003-Maximum Likelihood Estimation of a Stochastic Integrate-and-Fire Neural Model

20 0.034446977 15 nips-2003-A Probabilistic Model of Auditory Space Representation in the Barn Owl

similar papers computed by lsi model

lsi for this paper:

topicId topicWeight

[(0, -0.127), (1, -0.041), (2, 0.038), (3, -0.065), (4, -0.04), (5, 0.031), (6, 0.082), (7, -0.11), (8, -0.018), (9, 0.176), (10, 0.184), (11, -0.104), (12, -0.193), (13, -0.125), (14, 0.153), (15, -0.14), (16, 0.167), (17, -0.037), (18, 0.003), (19, -0.053), (20, -0.056), (21, -0.041), (22, -0.035), (23, 0.043), (24, -0.014), (25, -0.05), (26, 0.066), (27, -0.031), (28, -0.139), (29, 0.071), (30, 0.023), (31, 0.108), (32, -0.026), (33, 0.026), (34, 0.006), (35, 0.201), (36, -0.122), (37, 0.012), (38, 0.162), (39, 0.048), (40, 0.113), (41, 0.066), (42, 0.103), (43, -0.069), (44, 0.0), (45, -0.174), (46, 0.071), (47, -0.096), (48, 0.036), (49, -0.071)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.96568745 156 nips-2003-Phonetic Speaker Recognition with Support Vector Machines

Author: William M. Campbell, Joseph P. Campbell, Douglas A. Reynolds, Douglas A. Jones, Timothy R. Leek

2 0.75310546 60 nips-2003-Eigenvoice Speaker Adaptation via Composite Kernel Principal Component Analysis

Author: James T. Kwok, Brian Mak, Simon Ho

3 0.71793491 9 nips-2003-A Kullback-Leibler Divergence Based Kernel for SVM Classification in Multimedia Applications

Author: Pedro J. Moreno, Purdy P. Ho, Nuno Vasconcelos

4 0.46421781 162 nips-2003-Probabilistic Inference of Speech Signals from Phaseless Spectrograms

Author: Kannan Achan, Sam T. Roweis, Brendan J. Frey

5 0.31741393 5 nips-2003-A Classification-based Cocktail-party Processor

Author: Nicoleta Roman, Deliang Wang, Guy J. Brown

6 0.31244013 144 nips-2003-One Microphone Blind Dereverberation Based on Quasi-periodicity of Speech Signals

7 0.28455797 28 nips-2003-Application of SVMs for Colour Classification and Collision Detection with AIBO Robots

8 0.26966763 147 nips-2003-Online Learning via Global Feedback for Phrase Recognition

9 0.26845652 1 nips-2003-1-norm Support Vector Machines

10 0.24780484 191 nips-2003-Unsupervised Context Sensitive Language Acquisition from a Large Corpus

11 0.23761514 178 nips-2003-Sparse Greedy Minimax Probability Machine Classification

12 0.22822386 159 nips-2003-Predicting Speech Intelligibility from a Population of Neurons

13 0.22016835 70 nips-2003-Fast Algorithms for Large-State-Space HMMs with Applications to Web Usage Analysis

14 0.21992308 160 nips-2003-Prediction on Spike Data Using Kernel Algorithms

15 0.21757065 123 nips-2003-Markov Models for Automated ECG Interval Analysis

16 0.20657215 135 nips-2003-Necessary Intransitive Likelihood-Ratio Classifiers

17 0.19059096 181 nips-2003-Statistical Debugging of Sampled Programs

18 0.1846389 100 nips-2003-Laplace Propagation

19 0.18310146 173 nips-2003-Semi-supervised Protein Classification Using Cluster Kernels

20 0.17652366 113 nips-2003-Learning with Local and Global Consistency

similar papers computed by lda model

lda for this paper:

topicId topicWeight

[(0, 0.016), (11, 0.024), (30, 0.022), (35, 0.044), (53, 0.074), (71, 0.586), (76, 0.026), (85, 0.036), (91, 0.065), (99, 0.013)]

similar papers list:

simIndex simValue paperId paperTitle

1 0.97079289 133 nips-2003-Mutual Boosting for Contextual Inference

Author: Michael Fink, Pietro Perona

Abstract: Mutual Boosting is a method aimed at incorporating contextual information to augment object detection. When multiple detectors of objects and parts are trained in parallel using AdaBoost [1], object detectors might use the remaining intermediate detectors to enrich the weak learner set. This method generalizes the efficient features suggested by Viola and Jones [2] thus enabling information inference between parts and objects in a compositional hierarchy. In our experiments eye-, nose-, mouth- and face detectors are trained using the Mutual Boosting framework. Results show that the method outperforms applications overlooking contextual information. We suggest that achieving contextual integration is a step toward human-like detection capabilities. 1 In trod u ction Classification of multiple objects in complex scenes is one of the next challenges facing the machine learning and computer vision communities. Although, real-time detection of single object classes has been recently demonstrated [2], naïve duplication of these detectors to the multiclass case would be unfeasible. Our goal is to propose an efficient method for detection of multiple objects in natural scenes. Hand-in-hand with the challenges entailing multiclass detection, some distinct advantages emerge as well. Knowledge on position of several objects might shed light on the entire scene (Figure 1). Detection systems that do not exploit the information provided by objects on the neighboring scene will be suboptimal. A B Figure 1: Contextual spatial relationships assist detection A. in absence of facial components (whitened blocking box) faces can be detected by context (alignment of neighboring faces). B. keyboards can be detected when they appear under monitors. Many human and computer vision models postulate explicitly or implicitly that vision follows a compositional hierarchy. Grounded features (that are innate/hardwired and are available prior to learning) are used to detect salient parts, these parts in turn enable detection of complex objects [3, 4], and finally objects are used to recognize the semantics of the entire scene. Yet, a more accurate assessment of human performance reveals that the visual system often violates this strictly hierarchical structure in two ways. First, part and whole detection are often evidently interacting [5, 6]. Second, several layers of the hierarchy are occasionally bypassed to enable swift direct detection. This phenomenon is demonstrated by gist recognition experiments where the semantic classification of an entire scene is performed using only minimal low level feature information [7]. The insights emerging from observing human perception were adopted by the object detection community. Many object detection algorithms bypass stages of a strict compositional hierarchy. The Viola & Jones (VJ) detector [2] is able to perform robust online face detection by directly agglomerating very low-level features (rectangle contrasts), without explicitly referring to facial parts. Gist detection from low-level spatial frequencies was demonstrated by Oliva and Torralba [8]. Recurrent optimization of parts and object constellation is also common in modern detection schemes [9]. Although Latent Semantic Analysis (making use of object cooccurrence information) has been adapted to images [10], the existing state of object detection methods is still far from unifying all the sources of visual contextual information integrated by the human perceptual system. Tackling the context integration problem and achieving robust multiclass object detection is a vital step for applications like image-content database indexing and autonomous robot navigation. We will propose a method termed Mutual Boosting to incorporate contextual information for object detection. Section 2 will start by posing the multiclass detection problem from labeled images. In Section 3 we characterize the feature sets implemented by Mutual Boosting and define an object's contextual neighborhood. Section 4 presents the Mutual Boosting framework aimed at integrating contextual information and inspired by the recurrent inferences dominating the human perceptual system. An application of the Mutual Boosting framework to facial component detection is presented in Section 5. We conclude with a discussion on the scope and limitations of the proposed framework. 2 Problem setting and basic notation Suppose we wish to detect multiple objects in natural scenes, and that these scenes are characterized by certain mutual positions between the composing objects. Could we make use of these objects' contextual relations to improve detection? Perceptual context might include multiple sources of information: information originating from the presence of existing parts, information derived from other objects in the perceptual vicinity and finally general visual knowledge on the scene. In order to incorporate these various sources of visual contextual information Mutual Boosting will treat parts, objects and scenes identically. We will therefore use the term object as a general term while referring to any entity in the compositional hierarchy. Let M denote the cardinality of the object set we wish to detect in natural scenes. Our goal is to optimize detection by exploiting contextual information while maintaining detection time comparable to M individual detectors trained without such information. We define the goal of the multiclass detection algorithm as generating M intensity maps Hm=1,..,M indicating the likelihood of object m appearing at different positions in a target image. We will use the following notation (Figure 2): • H0+/H0-: raw image input with/without the trained objects (A1 & A2) • Cm[i]: labeled position of instance i of object m in image H0+ • Hm: intensity map output indicating the likelihood of object m appearing in different positions in the image H0 (B) B. Hm A2. H0- A1. H0+ Cm[1] Cm[2] Cm[1] Cm[2] Figure 2: A1 & A2. Input: position of positive and negative examples of eyes in natural images. B. Output: Eye intensity (eyeness) detection map of image H0+ 3 F e a t u r e se t a n d c o n t e x t u a l wi n d o w g e n e r a l i za t i o n s The VJ method for real-time object-detection included three basic innovations. First, they presented the rectangle contrast-features, features that are evaluated efficiently, using an integral-image. Second, VJ introduced AdaBoost [1] to object detection using rectangle features as weak learners. Finally a cascade method was developed to chain a sequence of increasingly complex AdaBoost learners to enable rapid filtering of non-relevant sections in the target image. The resulting cascade of AdaBoost face detectors achieves a 15 frame per second detection speed, with 90% detection rate and 2x10-6 false alarms. This detection speed is currently unmatched. In order to maintain efficient detection and in order to benchmark the performance of Mutual Boosting we will adopt the rectangle contrast feature framework suggested by VJ. It should be noted that the grayscale rectangle features could be naturally extended to any image channel that preserves the semantics of summation. A diversified feature set (including color features, texture features, etc.) might saturate later than a homogeneous channel feature set. By making use of features that capture the object regularities well, one can improve performance or reduce detection time. VJ extract training windows that capture the exact area of the training faces. We term this the local window approach. A second approach, in line with our attempt to incorporate information from neighboring parts or objects, would be to make use of training windows that capture wide regions around the object (Figure 3)1. A B Figure 3: A local window (VJ) and a contextual window that captures relative position information from objects or parts around and within the detected object. 1 Contextual neighborhoods emerge by downscaling larger regions in the original image to a PxP resolution window. The contextual neighborhood approach contributes to detection when the applied channels require a wide contextual range as will be demonstrated in the Mutual Boosting scheme presented in the following section2. 4 Mutual Boosting The AdaBoost algorithm maintains a clear distinction between the boosting level and the weak-learner training level. The basic insight guiding the Mutual Boosting method reexamines this distinction, stipulating that when multiple objects and parts are trained simultaneously using AdaBoost; any object detector might combine the previously evolving intermediate detectors to generate new weak learners. In order to elaborate this insight it should first be noted that while training a strong learner using 100 iterations of AdaBoost (abbreviated AB100) one could calculate an intermediate strong learner at each step on the way (AB2 - AB99). To apply this observation for our multiclass detection problem we simultaneously train M object detectors. At each boosting iteration t the M detectors (ABmt-1) emerging at the previous stage t-1, are used to filter positive and negative3 training images, thus producing intermediate m-detection maps Hmt-1 (likelihood of object m in the images4). Next, the Mutual Boosting stage takes place and all the existing Hmt-1 maps are used as additional channels out of which new contrast features are selected. This process gradually enriches the initial grounded features with composite contextual features. The composite features are searched on a PxP wide contextual neighborhood region rather than the PxP local window (Figure 3). Following a dynamic programming approach in training and detection, Hm=1,..,M detection maps are constantly maintained and updated so that the recalculation of Hmt only requires the last chosen weak learner WLmn*t to be evaluated on channel Hn*t-1 of the training image (Figure 4). This evaluation produces a binary detection layer that will be weighted by the AdaBoost weak-learner weighting scheme and added to the previous stage map5. Although Mutual Boosting examines a larger feature set during training, an iteration of Mutual Boosting detection of M objects is as time-consuming as performing an AdaBoost detection iteration for M individual objects. The advantage of Mutual Boosting emerges from introducing highly informative feature sets that can enhance detection or require fewer boosting iterations. While most object detection applications extract a local window containing the object information and discard the remaining image (including the object positional information). Mutual Boosting processes the entire image during training and detection and makes constant use of the information characterizing objects’ relative-position in the training images. As we have previously stated, the detected objects might be in various levels of a compositional hierarchy (e.g. complex objects or parts of other objects). Nevertheless, Mutual Boosting provides a similar treatment to objects, parts and scenes enabling any compositional structure of the data to naturally emerge. We will term any contextual reference that is not directly grounded to the basic features, as a cross referencing of objects6. 2 The most efficient size of the contextual neighborhoods might vary, from the immediate to the entire image, and therefore should be empirically learned. 3 Images without target objects (see experimental section below) 4 Unlike the weak learners, the intermediate strong learners do not apply a threshold 5 In order to optimize the number of detection map integral image recalculations these maps might be updated every k (e.g. 50) iterations rather than at each iteration. 6 Scenes can be crossed referenced as well if scene labels are available (office/lab etc.). H0+/0- positive / negative raw images Cm[i] position of instance i of object m=1,..,M in image H0+ initialize boosting-weights of instances i of object m to 1 initialize detection maps Hm+0/Hm-0 to 0 Input Initialization For t=1,…,T For m=1,..,M and n=0,..,M (A) cutout & downscale local (n=0) or contextual (n>0) windows (WINm) of instances i of object m (at Cm[i]), from all existing images Hnt-1 For m=1,..,M normalize boosting-weights of object m instances [1] (B1&2) select map Hn*t-1 and weak learner WLmn* that minimize error on WINm decrease boosting-weights of instances that WLmn* labeled correctly [1] (C) DetectionLayermn* ← WLmn*(Hn*t-1) calculate α mt the weak learner contribution factor from the empirical error [1] (D) update m-detection map Hmt ← Hmt-1 + αmt DetectionLayermn * Return strong learner ABmT including WLmn*1,..,T and αm1,..,T (m=1,..,M) H0± raw image H1± . . . Hn*± (A) WIN m0 WL m0 (B1) . . . Hm± (A) WIN m1 (B2) WL m1 (B1) (B2) m detection map (A) WIN mn* WL (B1) (D) (C) Detection Layer mn* mn* Figure 4: Mutual Boosting Diagram & Pseudo code. Each raw image H0 is analyzed by M object detection maps Hm=1,.,M, updated by iterating through four steps: (A) cutout & downscale from existing maps H n=0,..,M t-1 a local (n=0) or contextual (n>0) PxP window containing a neighborhood of object m (B1&2) select best performing map Hn* and weak learner WLmn* that optimize object m detection (C) run WLmn* on Hn* map to generate a new binary m-detection layer (D) add m-detection layer to existing detection map Hm. [1] Standard AdaBoost stages are not elaborated To maintain local and global natural scene statistics, negative training examples are generated by pairing each image with an image of equal size that does not contain the target objects and by centering the local and contextual windows of the positive and negative examples on the object positions in the positive images (see Figure 2). By using parallel boosting and efficient rectangle contrast features, Mutual Boosting is capable of incorporating many information inferences (references in Figure 5): • Features could be used to directly detect parts and objects (A & B) • Objects could be used to detect other (or identical) objects in the image (C) • Parts could be used to detect other (or identical) nearby parts (D & E) • Parts could be used to detect objects (F) • Objects could be used to detect parts A. eye feature from raw image B. face feature from raw image C. face E. mouth feature from eye feature from face detection image detection image F. face feature from mouth D. eye feature from eye detection image detection image Figure 5: A-E. Emerging features of eyes, mouths and faces (presented on windows of raw images for legibility). The windows’ scale is defined by the detected object size and by the map mode (local or contextual). C. faces are detected using face detection maps HFace, exploiting the fact that faces tend to be horizontally aligned. 5 Experiments A. Pd In order to test the contribution of the Mutual Boosting process we focused on detection of objects in what we term a face-scene (right eye, left eye, nose, mouth and face). We chose to perform contextual detection in the face-scene for two main reasons. First as detailed in Figure 5, face scenes demonstrate a range of potential part and object cross references. Second, faces have been the focus of object detection research for many years, thus enabling a systematic result comparison. Experiment 1 was aimed at comparing the performance of Mutual Boosting to that of naïve independently trained object detectors using local windows. Pfa Figure 6: A. Two examples of the CMU/MIT face database. B. Mutual Boosting and AdaBoost ROCs on the CMU/MIT face database. Face-scene images were downloaded from the web and manually labeled7. Training relied on 450 positive and negative examples (~4% of the images used by VJ). 400 iterations of local window AdaBoost and contextual window Mutual Boosting were performed on the same image set. Contextual windows encompassed a region five times larger in width and height than the local windows8 (see Figure 3). 7 By following CMU database conventions (R-eye, L-eye, Nose & Mouth positions) we derive both the local window position and the relative position of objects in the image 8 Local windows were created by downscaling objects to 25x25 grids Test image detection maps emerge from iteratively summing T m-detection layers (Mutual Boosting stages C&D;). ROC performance on the CMU/MIT face database (see sample images in Figure 6A) was assessed using a threshold on position Cm[i] that best discriminated the final positive and negative detection maps Hm+/-T. Figure 6B demonstrates the superiority of Mutual Boosting to grounded feature AdaBoost. A. COV 0.25 COV 1.00 COV 4.00 Equal error performance Our second experiment was aimed at assessing the performance of Mutual Boosting as we change the detected configurations’ variance. Assuming normal distribution of face configurations we estimated (from our existing labeled set) the spatial covariance between four facial components (noses, mouths and both eyes). We then modified the covariance matrix, multiplying it by 0.25, 1 or 4 and generated 100 artificial configurations by positioning four contrasting rectangles in the estimated position of facial components. Although both Mutual Boosting and AdaBoost performance degraded as the configuration variance increased, the advantage of Mutual Boosting persists both in rigid and in varying configurations9 (Figure 7). MB sigma=0.25 MB sigma=1.00 MB sigma=4.00 AB sigma=0.25 AB sigma=1.00 AB sigma=4.00 Boosting iteration Figure 7: A. Artificial face configurations with increasing covariance B. MB and AB Equal error rate performance on configurations with varying covariance as a function of boosting iterations. 6 D i s c u s s i on While evaluating the performance of Mutual Boosting it should be emphasized that we did not implement the VJ cascade approach; therefore we only attempt to demonstrate that the power of a single AdaBoost learner could be augmented by Mutual Boosting. The VJ detector is rescaled in order to perform efficient detection of objects in multiple scales. For simplicity, scale of neighboring objects and parts was assumed to be fixed so that a similar detector-rescaling approach could be followed. This assumption holds well for face-scenes, but if neighboring objects may vary in scale a single m-detection map will not suffice. However, by transforming each m-detection image to an m-detection cube, (having scale as the third dimension) multi-scale context detection could be achieved10. The dynamic programming characteristic of Mutual Boosting (simply reusing the multiple position and scale detections already performed by VJ) will ensure that the running time of varying scale context will only be doubled. It should be noted that the facescene is highly structured and therefore it is a good candidate for demonstrating 9 In this experiment the resolution of the MB windows (and the number of training features) was decreased so that information derived from the higher resolution of the parts would be ruled out as an explaining factor for the Mutual Boosting advantage. This procedure explains the superior AdaBoost performance in the first boosting iteration. 10 By using an integral cube, calculating the sum of a cube feature (of any size) requires 8 access operations (only double than the 4 operations required in the integral image case). Mutual Boosting; however as suggested by Figure 7B Mutual Boosting can handle highly varying configurations and the proposed method needs no modification when applied to other scenes, like the office scene in Figure 111. Notice that Mutual Boosting does not require a-priori knowledge of the compositional structure but rather permits structure to naturally emerge in the cross referencing pattern (see examples in Figure 5). Mutual Boosting could be enhanced by unifying the selection of weak-learners rather than selecting an individual weak learner for each object detector. Unified selection is aimed at choosing weak learners that maximize the entire object set detection rate, thus maximizing feature reuse [11]. This approach is optimal when many objects with common characteristics are trained. Is Mutual Boosting specific for image object detection? Indeed it requires labeled input of multiple objects in a scene supplying a local description of the objects as well as information on their contextual mutual positioning. But these criterions are shared by other complex

2 0.96516103 108 nips-2003-Learning a Distance Metric from Relative Comparisons

Author: Matthew Schultz, Thorsten Joachims

Abstract: This paper presents a method for learning a distance metric from relative comparison such as “A is closer to B than A is to C”. Taking a Support Vector Machine (SVM) approach, we develop an algorithm that provides a ﬂexible way of describing qualitative training data as a set of constraints. We show that such constraints lead to a convex quadratic programming problem that can be solved by adapting standard methods for SVM training. We empirically evaluate the performance and the modelling ﬂexibility of the algorithm on a collection of text documents. 1

same-paper 3 0.96209234 156 nips-2003-Phonetic Speaker Recognition with Support Vector Machines

Author: William M. Campbell, Joseph P. Campbell, Douglas A. Reynolds, Douglas A. Jones, Timothy R. Leek

4 0.96163708 142 nips-2003-On the Concentration of Expectation and Approximate Inference in Layered Networks

Author: Xuanlong Nguyen, Michael I. Jordan

Abstract: We present an analysis of concentration-of-expectation phenomena in layered Bayesian networks that use generalized linear models as the local conditional probabilities. This framework encompasses a wide variety of probability distributions, including both discrete and continuous random variables. We utilize ideas from large deviation analysis and the delta method to devise and evaluate a class of approximate inference algorithms for layered Bayesian networks that have superior asymptotic error bounds and very fast computation time. 1

5 0.94705236 96 nips-2003-Invariant Pattern Recognition by Semi-Definite Programming Machines

Author: Thore Graepel, Ralf Herbrich

Abstract: Knowledge about local invariances with respect to given pattern transformations can greatly improve the accuracy of classiﬁcation. Previous approaches are either based on regularisation or on the generation of virtual (transformed) examples. We develop a new framework for learning linear classiﬁers under known transformations based on semideﬁnite programming. We present a new learning algorithm— the Semideﬁnite Programming Machine (SDPM)—which is able to ﬁnd a maximum margin hyperplane when the training examples are polynomial trajectories instead of single points. The solution is found to be sparse in dual variables and allows to identify those points on the trajectory with minimal real-valued output as virtual support vectors. Extensions to segments of trajectories, to more than one transformation parameter, and to learning with kernels are discussed. In experiments we use a Taylor expansion to locally approximate rotational invariance in pixel images from USPS and ﬁnd improvements over known methods. 1

6 0.76122773 117 nips-2003-Linear Response for Approximate Inference

7 0.65365791 132 nips-2003-Multiple Instance Learning via Disjunctive Programming Boosting

8 0.6153214 163 nips-2003-Probability Estimates for Multi-Class Classification by Pairwise Coupling

9 0.61414158 41 nips-2003-Boosting versus Covering

10 0.61146259 145 nips-2003-Online Classification on a Budget

11 0.61046141 9 nips-2003-A Kullback-Leibler Divergence Based Kernel for SVM Classification in Multimedia Applications

12 0.60768718 121 nips-2003-Log-Linear Models for Label Ranking

13 0.60091531 100 nips-2003-Laplace Propagation

14 0.60028982 47 nips-2003-Computing Gaussian Mixture Models with EM Using Equivalence Constraints

15 0.60019702 122 nips-2003-Margin Maximizing Loss Functions

16 0.59344929 23 nips-2003-An Infinity-sample Theory for Multi-category Large Margin Classification

17 0.57873368 187 nips-2003-Training a Quantum Neural Network

18 0.57427961 85 nips-2003-Human and Ideal Observers for Detecting Image Curves

19 0.57424253 103 nips-2003-Learning Bounds for a Generalized Family of Bayesian Posterior Distributions

20 0.57011497 192 nips-2003-Using the Forest to See the Trees: A Graphical Model Relating Features, Objects, and Scenes