acl acl2011 acl2011-95 knowledge-graph by maker-knowledge-mining

95 acl-2011-Detection of Agreement and Disagreement in Broadcast Conversations

Source: pdf

Author: Wen Wang ; Sibel Yaman ; Kristin Precoda ; Colleen Richey ; Geoffrey Raymond

Abstract: We present Conditional Random Fields based approaches for detecting agreement/disagreement between speakers in English broadcast conversation shows. We develop annotation approaches for a variety of linguistic phenomena. Various lexical, structural, durational, and prosodic features are explored. We compare the performance when using features extracted from automatically generated annotations against that when using human annotations. We investigate the efficacy of adding prosodic features on top of lexical, structural, and durational features. Since the training data is highly imbalanced, we explore two sampling approaches, random downsampling and ensemble downsampling. Overall, our approach achieves 79.2% (precision), 50.5% (recall), 61.7% (F1) for agreement detection and 69.2% (precision), 46.9% (recall), and 55.9% (F1) for disagreement detection, on the English broadcast conversation data. 1 ?yIntroduction In ?ythis work, we present models for detecting agre?yement/disagreement (denoted (dis)agreement) betwy?een speakers in English broadcast conversation show?ys. The Broadcast Conversation (BC) genre differs from the Broadcast News (BN) genre in that it is?y more interactive and spontaneous, referring to freey? speech in news-style TV and radio programs and consisting of talk shows, interviews, call-in prog?yrams, live reports, and round-tables. Previous y? y?This work was performed while the author was at ICSI. syaman@us . ibm .com, graymond@ s oc .uc sb . edu work on detecting (dis)agreements has been focused on meeting data. (Hillard et al., 2003), (Galley et al., 2004), (Hahn et al., 2006) used spurt-level agreement annotations from the ICSI meeting corpus (Janin et al., 2003). (Hillard et al., 2003) explored unsupervised machine learning approaches and on manual transcripts, they achieved an overall 3-way agreement/disagreement classification ac- curacy as 82% with keyword features. (Galley et al., 2004) explored Bayesian Networks for the detection of (dis)agreements. They used adjacency pair information to determine the structure of their conditional Markov model and outperformed the results of (Hillard et al., 2003) by improving the 3way classification accuracy into 86.9%. (Hahn et al., 2006) explored semi-supervised learning algorithms and reached a competitive performance of 86.7% 3-way classification accuracy on manual transcriptions with only lexical features. (Germesin and Wilson, 2009) investigated supervised machine learning techniques and yields competitive results on the annotated data from the AMI meeting corpus (McCowan et al., 2005). Our work differs from these previous studies in two major categories. One is that a different definition of (dis)agreement was used. In the current work, a (dis)agreement occurs when a responding speaker agrees with, accepts, or disagrees with or rejects, a statement or proposition by a first speaker. Second, we explored (dis)agreement detection in broadcast conversation. Due to the difference in publicity and intimacy/collegiality between speakers in broadcast conversations vs. meet- ings, (dis)agreement may have different character374 Proceedings ofP thoer t4l9atnhd A, Onrnuegaoln M,e Jeuntineg 19 o-f2 t4h,e 2 A0s1s1o.c?i ac t2io0n11 fo Ar Cssoocmiaptuiotanti foonra Clo Lminpguutiast i ocns:aslh Loirntpgaupisetrics , pages 374–378, istics. Different from the unsupervised approaches in (Hillard et al., 2003) and semi-supervised approaches in (Hahn et al., 2006), we conducted supervised training. Also, different from (Hillard et al., 2003) and (Galley et al., 2004), our classification was carried out on the utterance level, instead of on the spurt-level. Galley et al. extended Hillard et al.’s work by adding features from previous spurts and features from the general dialog context to infer the class of the current spurt, on top of features from the current spurt (local features) used by Hillard et al. Galley et al. used adjacency pairs to describe the interaction between speakers and the relations between consecutive spurts. In this preliminary study on broadcast conversation, we directly modeled (dis)agreement detection without using adjacency pairs. Still, within the conditional random fields (CRF) framework, we explored features from preceding and following utterances to consider context in the discourse structure. We explored a wide variety of features, including lexical, structural, du- rational, and prosodic features. To our knowledge, this is the first work to systematically investigate detection of agreement/disagreement for broadcast conversation data. The remainder of the paper is organized as follows. Section 2 presents our data and automatic annotation modules. Section 3 describes various features and the CRF model we explored. Experimental results and discussion appear in Section 4, as well as conclusions and future directions. 2 Data and Automatic Annotation In this work, we selected English broadcast conversation data from the DARPA GALE program collected data (GALE Phase 1 Release 4, LDC2006E91; GALE Phase 4 Release 2, LDC2009E15). Human transcriptions and manual speaker turn labels are used in this study. Also, since the (dis)agreement detection output will be used to analyze social roles and relations of an interacting group, we first manually marked soundbites and then excluded soundbites during annotation and modeling. We recruited annotators to provide manual annotations of speaker roles and (dis)agreement to use for the supervised training of models. We de- fined a set of speaker roles as follows. Host/chair is a person associated with running the discussions 375 or calling the meeting. Reporting participant is a person reporting from the field, from a subcommittee, etc. Commentator participant/Topic participant is a person providing commentary on some subject, or person who is the subject of the conversation and plays a role, e.g., as a newsmaker. Audience participant is an ordinary person who may call in, ask questions at a microphone at e.g. a large presentation, or be interviewed because of their presence at a news event. Other is any speaker who does not fit in one of the above categories, such as a voice talent, an announcer doing show openings or commercial breaks, or a translator. Agreements and disagreements are composed of different combinations of initiating utterances and responses. We reformulated the (dis)agreement detection task as the sequence tagging of 11 (dis)agreement-related labels for identifying whether a given utterance is initiating a (dis)agreement opportunity, is a (dis)agreement response to such an opportunity, or is neither of these, in the show. For example, a Negative tag question followed by a negation response forms an agreement, that is, A: [Negative tag] This is not black and white, is it? B: [Agreeing Response] No, it isn’t. The data sparsity problem is serious. Among all 27,071 utterances, only 2,589 utterances are involved in (dis)agreement as initiating or response utterances, about 10% only among all data, while 24,482 utterances are not involved. These annotators also labeled shows with a variety of linguistic phenomena (denoted language use constituents, LUC), including discourse markers, disfluencies, person addresses and person mentions, prefaces, extreme case formulations, and dialog act tags (DAT). We categorized dialog acts into statement, question, backchannel, and incomplete. We classified disfluencies (DF) into filled pauses (e.g., uh, um), repetitions, corrections, and false starts. Person address (PA) terms are terms that a speaker uses to address another person. Person mentions (PM) are references to non-participants in the conversation. Discourse markers (DM) are words or phrases that are related to the structure of the discourse and express a relation between two utter- ances, for example, I mean, you know. Prefaces (PR) are sentence-initial lexical tokens serving functions close to discourse markers (e.g., Well, I think that...). Extreme case formulations (ECF) are lexical patterns emphasizing extremeness (e.g., This is the best book I have ever read). In the end, we manually annotated 49 English shows. We preprocessed English manual transcripts by removing transcriber annotation markers and noise, removing punctuation and case information, and conducting text normalization. We also built automatic rule-based and statistical annotation tools for these LUCs. 3 Features and Model We explored lexical, structural, durational, and prosodic features for (dis)agreement detection. We included a set of “lexical” features, including ngrams extracted from all of that speaker’s utterances, denoted ngram features. Other lexical features include the presence of negation and acquiescence, yes/no equivalents, positive and negative tag questions, and other features distinguishing different types of initiating utterances and responses. We also included various lexical features extracted from LUC annotations, denoted LUC features. These additional features include features related to the presence of prefaces, the counts of types and tokens of discourse markers, extreme case formulations, disfluencies, person addressing events, and person mentions, and the normalized values of these counts by sentence length. We also include a set of features related to the DAT of the current utterance and preceding and following utterances. We developed a set of “structural” and “durational” features, inspired by conversation analysis, to quantitatively represent the different participation and interaction patterns of speakers in a show. We extracted features related to pausing and overlaps between consecutive turns, the absolute and relative duration of consecutive turns, and so on. We used a set of prosodic features including pause, duration, and the speech rate of a speaker. We also used pitch and energy of the voice. Prosodic features were computed on words and phonetic alignment of manual transcripts. Features are computed for the beginning and ending words of an utterance. For the duration features, we used the average and maximum vowel duration from forced align- ment, both unnormalized and normalized for vowel identity and phone context. For pitch and energy, we 376 calculated the minimum, maximum,E range, mean, standard deviation, skewnesSs and kurEtosis values. A decision tree model was useSd to comEpute posteriors fFrom prosodic features and Swe used cuEmulative binnFing of posteriors as final feSatures , simEilar to (Liu et aFl., 2006). As ilPlu?stErajtSed?F i?n SectionS 2, we refEormulated the F(dis)agrePe?mEEejnSt? Fdet?ection taSsk as a seqEuence tagging FproblemP. EWEejS u?sFe?d the MalSlet packagEe (McCallum, 2F002) toP i?mEEpjSle?mFe?nt the linSear chain CERF model for FsequencPe ?tEEagjSgi?nFg.? A CRFS is an undEirected graphiFcal modPe?lEE EthjSa?t Fde?fines a glSobal log-lEinear distributFion of Pthe?EE sjtaSt?eF (o?r label) Ssequence E conditioned oFn an oPbs?EeErvjaSt?ioFn? sequencSe, in our case including Fthe sequPe?nEcEej So?fF Fse?ntences S and the corresponding sFequencPe ?oEEf jfSea?Ftur?es for this sequence of sentences F. TheP ?mEEodjSe?l Fis? optimized globally over the entire seqPue?nEEcejS. TFh?e CRF model is trained to maximize theP c?oEEnjdSit?iFon?al log-likelihood of a given training set P?EEjS? F?. During testing, the most likely sequence E is found using the Viterbi algorithm. One of the motivations of choosing conditional random fields was to avoid the label-bias problem found in hidden Markov models. Compared to Maximum Entropy modeling, the CRF model is optimized globally over the entire sequence, whereas the ME model makes a decision at each point individually without considering the context event information. 4 Experiments All (dis)agreement detection results are based on nfold cross-validation. In this procedure, we held out one show as the test set, randomly held out another show as the dev set, trained models on the rest of the data, and tested the model on the heldout show. We iterated through all shows and computed the overall accuracy. Table 1 shows the results of (dis)agreement detection using all features except prosodic features. We compared two conditions: (1) features extracted completely from the automatic LUC annotations and automatically detected speaker roles, and (2) features from manual speaker role labels and manual LUC annotations when man- ual annotations are available. Table 1 showed that running a fully automatic system to generate automatic annotations and automatic speaker roles produced comparable performance to the system using features from manual annotations whenever available. Table 1: Precision (%), recall (%), and F1 (%) of (dis)agreement detection using features extracted from manual speaker role labels and manual LUC annotations when available, denoted Manual Annotation, and automatic LUC annotations and automatically detected speaker roles, denoted Automatic Annotation. AMuatnoumaltAicn Aontaoitantio78P91.5Agr4eR3em.26en5tF671.5 AMuatnoumal tAicn Aontaoitanio76P04D.13isag3rR86e.56emn4F96t.176 We then focused on the condition of using features from manual annotations when available and added prosodic features as described in Section 3. The results are shown in Table 2. Adding prosodic features produced a 0.7% absolute gain on F1 on agreement detection, and 1.5% absolute gain on F1 on disagreement detection. Table 2: Precision (%), recall (%), and F1 (%) of (dis)agreement detection using manual annotations without and with prosodic features. w /itohp ro s o d ic 8 P1 .58Agr4 e34Re.m02en5t F76.125 w i/tohp ro s o d ic 7 0 PD.81isag43r0R8e.15eme5n4F19t.172 Note that only about 10% utterances among all data are involved in (dis)agreement. This indicates a highly imbalanced data set as one class is more heavily represented than the other/others. We suspected that this high imbalance has played a major role in the high precision and low recall results we obtained so far. Various approaches have been studied to handle imbalanced data for classifications, 377 trying to balaNnce the class distribution in the training set by eithNer oversaNmpling the minority class or downsamplinNg the maNjority class. In this preliminary study of NNsamplingN Napproaches for handling imbalanced dataN NNfor CRF Ntraining, we investigated two apprNoaches, rNNandom dNownsampling and ensemble dowNnsamplinNgN. RandoNm downsampling randomly dowNnsamples NNthe majorNity class to equate the number Nof minoritNNy and maNjority class samples. Ensemble NdownsampNNling is a N refinement of random downsamNpling whiNNch doesn’Nt discard any majority class samNples. InstNNead, we pNartitioned the majority class samNples into NN subspaNces with each subspace containiNng the samNe numbNer of samples as the minority clasNs. Then wNe train N CRF models, each based on thNe minoritNy class samples and one disjoint partitionN Nfrom the N subspaces. During testing, the posterioNr probability for one utterance is averaged over the N CRF models. The results from these two sampling approaches as well as the baseline are shown in Table 3. Both sampling approaches achieved significant improvement over the baseline, i.e., train- ing on the original data set, and ensemble downsampling produced better performance than downsampling. We noticed that both sampling approaches degraded slightly in precision but improved significantly in recall, resulting in 4.5% absolute gain on F1 for agreement detection and 4.7% absolute gain on F1 for disagreement detection. Table 3: Precision (%), recall (%), and F1 (%) of (dis)agreement detection without sampling, with random downsampling and ensemble downsampling. Manual annotations and prosodic features are used. BERansedlmoinbedwonsampling78P19D.825Aisagr4e8R0.m7e5n6 tF701. 2 EBRa ns ne dlmoinmbel dodwowns asmamp lin gn 67 09. 8324 046. 8915 351. 892 In conclusion, this paper presents our work on detection of agreements and disagreements in English broadcast conversation data. We explored a variety of features, including lexical, structural, durational, and prosodic features. We experimented these features using a linear-chain conditional random fields model and conducted supervised training. We observed significant improvement from adding prosodic features and employing two sampling approaches, random downsampling and ensemble downsampling. Overall, we achieved 79.2% (precision), 50.5% (recall), 61.7% (F1) for agreement detection and 69.2% (precision), 46.9% (recall), and 55.9% (F1) for disagreement detection, on English broadcast conversation data. In future work, we plan to continue adding and refining features, explore dependencies between features and contextual cues with respect to agreements and disagreements, and investigate the efficacy of other machine learning approaches such as Bayesian networks and Support Vector Machines. Acknowledgments The authors thank Gokhan Tur and Dilek HakkaniT u¨r for valuable insights and suggestions. This work has been supported by the Intelligence Advanced Research Projects Activity (IARPA) via Army Research Laboratory (ARL) contract number W91 1NF-09-C-0089. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, ARL, or the U.S. Government. References M. Galley, K. McKeown, J. Hirschberg, and E. Shriberg. 2004. Identifying agreement and disagreement in conversational speech: Use ofbayesian networks to model pragmatic dependencies. In Proceedings of ACL. S. Germesin and T. Wilson. 2009. Agreement detection in multiparty conversation. In Proceedings of International Conference on Multimodal Interfaces. S. Hahn, R. Ladner, and M. Ostendorf. 2006. Agreement/disagreement classification: Exploiting unlabeled data using constraint classifiers. In Proceedings of HLT/NAACL. 378 D. Hillard, M. Ostendorf, and E. Shriberg. 2003. Detection of agreement vs. disagreement in meetings: Training with unlabeled data. In Proceedings of HLT/NAACL. A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, and C. Wooters. 2003. The ICSI Meeting Corpus. In Proc. ICASSP, Hong Kong, April. Yang Liu, Elizabeth Shriberg, Andreas Stolcke, Dustin Hillard, Mari Ostendorf, and Mary Harper. 2006. Enriching speech recognition with automatic detection of sentence boundaries and disfluencies. IEEE Transactions on Audio, Speech, and Language Processing, 14(5): 1526–1540, September. Special Issue on Progress in Rich Transcription. Andrew McCallum. 2002. Mallet: A machine learning for language toolkit. http://mallet.cs.umass.edu. I. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, W. Post, D. Reidsma, and P. Wellner. 2005. The AMI meeting corpus. In Proceedings of Measuring Behavior 2005, the 5th International Conference on Methods and Techniques in Behavioral Research.

Reference: text

Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 Box 218, Yorktown Heights, NY 10598, USA Ungiversity of California, Santa Barbara, CA, USA , fwwang precoda , col leeng @ speech . [sent-23, score-0.057]

2 com, Abstract We present Conditional Random Fields based approaches for detecting agreement/disagreement between speakers in English broadcast conversation shows. [sent-25, score-0.511]

3 We develop annotation approaches for a variety of linguistic phenomena. [sent-26, score-0.075]

4 Various lexical, structural, durational, and prosodic features are explored. [sent-27, score-0.337]

5 We compare the performance when using features extracted from automatically generated annotations against that when using human annotations. [sent-28, score-0.189]

6 We investigate the efficacy of adding prosodic features on top of lexical, structural, and durational features. [sent-29, score-0.552]

7 Since the training data is highly imbalanced, we explore two sampling approaches, random downsampling and ensemble downsampling. [sent-30, score-0.365]

8 9% (F1) for disagreement detection, on the English broadcast conversation data. [sent-37, score-0.498]

9 edu work on detecting (dis)agreements has been focused on meeting data. [sent-55, score-0.029]

10 , 2006) used spurt-level agreement annotations from the ICSI meeting corpus (Janin et al. [sent-59, score-0.31]

11 , 2003) explored unsupervised machine learning approaches and on manual transcripts, they achieved an overall 3-way agreement/disagreement classification ac- curacy as 82% with keyword features. [sent-62, score-0.218]

12 , 2004) explored Bayesian Networks for the detection of (dis)agreements. [sent-64, score-0.231]

13 They used adjacency pair information to determine the structure of their conditional Markov model and outperformed the results of (Hillard et al. [sent-65, score-0.094]

14 , 2006) explored semi-supervised learning algorithms and reached a competitive performance of 86. [sent-69, score-0.076]

15 7% 3-way classification accuracy on manual transcriptions with only lexical features. [sent-70, score-0.186]

16 In the current work, a (dis)agreement occurs when a responding speaker agrees with, accepts, or disagrees with or rejects, a statement or proposition by a first speaker. [sent-75, score-0.214]

17 Due to the difference in publicity and intimacy/collegiality between speakers in broadcast conversations vs. [sent-77, score-0.302]

18 Different from the unsupervised approaches in (Hillard et al. [sent-81, score-0.032]

19 , 2004), our classification was carried out on the utterance level, instead of on the spurt-level. [sent-86, score-0.063]

20 ’s work by adding features from previous spurts and features from the general dialog context to infer the class of the current spurt, on top of features from the current spurt (local features) used by Hillard et al. [sent-89, score-0.422]

21 used adjacency pairs to describe the interaction between speakers and the relations between consecutive spurts. [sent-91, score-0.153]

22 In this preliminary study on broadcast conversation, we directly modeled (dis)agreement detection without using adjacency pairs. [sent-92, score-0.423]

23 Still, within the conditional random fields (CRF) framework, we explored features from preceding and following utterances to consider context in the discourse structure. [sent-93, score-0.482]

24 We explored a wide variety of features, including lexical, structural, du- rational, and prosodic features. [sent-94, score-0.337]

25 To our knowledge, this is the first work to systematically investigate detection of agreement/disagreement for broadcast conversation data. [sent-95, score-0.548]

26 Section 2 presents our data and automatic annotation modules. [sent-97, score-0.043]

27 Section 3 describes various features and the CRF model we explored. [sent-98, score-0.076]

28 2 Data and Automatic Annotation In this work, we selected English broadcast conversation data from the DARPA GALE program collected data (GALE Phase 1 Release 4, LDC2006E91; GALE Phase 4 Release 2, LDC2009E15). [sent-100, score-0.393]

29 Human transcriptions and manual speaker turn labels are used in this study. [sent-101, score-0.307]

30 Also, since the (dis)agreement detection output will be used to analyze social roles and relations of an interacting group, we first manually marked soundbites and then excluded soundbites during annotation and modeling. [sent-102, score-0.404]

31 We recruited annotators to provide manual annotations of speaker roles and (dis)agreement to use for the supervised training of models. [sent-103, score-0.451]

32 Host/chair is a person associated with running the discussions 375 or calling the meeting. [sent-105, score-0.105]

33 Reporting participant is a person reporting from the field, from a subcommittee, etc. [sent-106, score-0.161]

34 Commentator participant/Topic participant is a person providing commentary on some subject, or person who is the subject of the conversation and plays a role, e. [sent-107, score-0.45]

35 Audience participant is an ordinary person who may call in, ask questions at a microphone at e. [sent-110, score-0.19]

36 a large presentation, or be interviewed because of their presence at a news event. [sent-112, score-0.031]

37 Other is any speaker who does not fit in one of the above categories, such as a voice talent, an announcer doing show openings or commercial breaks, or a translator. [sent-113, score-0.152]

38 Agreements and disagreements are composed of different combinations of initiating utterances and responses. [sent-114, score-0.345]

39 We reformulated the (dis)agreement detection task as the sequence tagging of 11 (dis)agreement-related labels for identifying whether a given utterance is initiating a (dis)agreement opportunity, is a (dis)agreement response to such an opportunity, or is neither of these, in the show. [sent-115, score-0.398]

40 For example, a Negative tag question followed by a negation response forms an agreement, that is, A: [Negative tag] This is not black and white, is it? [sent-116, score-0.081]

41 Among all 27,071 utterances, only 2,589 utterances are involved in (dis)agreement as initiating or response utterances, about 10% only among all data, while 24,482 utterances are not involved. [sent-119, score-0.48]

42 These annotators also labeled shows with a variety of linguistic phenomena (denoted language use constituents, LUC), including discourse markers, disfluencies, person addresses and person mentions, prefaces, extreme case formulations, and dialog act tags (DAT). [sent-120, score-0.37]

43 We categorized dialog acts into statement, question, backchannel, and incomplete. [sent-121, score-0.05]

44 Person address (PA) terms are terms that a speaker uses to address another person. [sent-125, score-0.152]

45 Person mentions (PM) are references to non-participants in the conversation. [sent-126, score-0.039]

46 Discourse markers (DM) are words or phrases that are related to the structure of the discourse and express a relation between two utter- ances, for example, I mean, you know. [sent-127, score-0.149]

47 Prefaces (PR) are sentence-initial lexical tokens serving functions close to discourse markers (e. [sent-128, score-0.18]

48 Extreme case formulations (ECF) are lexical patterns emphasizing extremeness (e. [sent-134, score-0.087]

49 We preprocessed English manual transcripts by removing transcriber annotation markers and noise, removing punctuation and case information, and conducting text normalization. [sent-138, score-0.27]

50 We also built automatic rule-based and statistical annotation tools for these LUCs. [sent-139, score-0.043]

51 3 Features and Model We explored lexical, structural, durational, and prosodic features for (dis)agreement detection. [sent-140, score-0.413]

52 We included a set of “lexical” features, including ngrams extracted from all of that speaker’s utterances, denoted ngram features. [sent-141, score-0.06]

53 Other lexical features include the presence of negation and acquiescence, yes/no equivalents, positive and negative tag questions, and other features distinguishing different types of initiating utterances and responses. [sent-142, score-0.525]

54 We also included various lexical features extracted from LUC annotations, denoted LUC features. [sent-143, score-0.167]

55 These additional features include features related to the presence of prefaces, the counts of types and tokens of discourse markers, extreme case formulations, disfluencies, person addressing events, and person mentions, and the normalized values of these counts by sentence length. [sent-144, score-0.503]

56 We also include a set of features related to the DAT of the current utterance and preceding and following utterances. [sent-145, score-0.139]

57 We developed a set of “structural” and “durational” features, inspired by conversation analysis, to quantitatively represent the different participation and interaction patterns of speakers in a show. [sent-146, score-0.241]

58 We extracted features related to pausing and overlaps between consecutive turns, the absolute and relative duration of consecutive turns, and so on. [sent-147, score-0.297]

59 We used a set of prosodic features including pause, duration, and the speech rate of a speaker. [sent-148, score-0.337]

60 Prosodic features were computed on words and phonetic alignment of manual transcripts. [sent-150, score-0.186]

61 For the duration features, we used the average and maximum vowel duration from forced align- ment, both unnormalized and normalized for vowel identity and phone context. [sent-152, score-0.236]

62 For pitch and energy, we 376 calculated the minimum, maximum,E range, mean, standard deviation, skewnesSs and kurEtosis values. [sent-153, score-0.042]

63 A decision tree model was useSd to comEpute posteriors fFrom prosodic features and Swe used cuEmulative binnFing of posteriors as final feSatures , simEilar to (Liu et aFl. [sent-154, score-0.411]

64 One of the motivations of choosing conditional random fields was to avoid the label-bias problem found in hidden Markov models. [sent-202, score-0.116]

65 Compared to Maximum Entropy modeling, the CRF model is optimized globally over the entire sequence, whereas the ME model makes a decision at each point individually without considering the context event information. [sent-203, score-0.036]

66 4 Experiments All (dis)agreement detection results are based on nfold cross-validation. [sent-204, score-0.155]

67 In this procedure, we held out one show as the test set, randomly held out another show as the dev set, trained models on the rest of the data, and tested the model on the heldout show. [sent-205, score-0.068]

68 Table 1 shows the results of (dis)agreement detection using all features except prosodic features. [sent-207, score-0.492]

69 We compared two conditions: (1) features extracted completely from the automatic LUC annotations and automatically detected speaker roles, and (2) features from manual speaker role labels and manual LUC annotations when man- ual annotations are available. [sent-208, score-1.015]

70 Table 1 showed that running a fully automatic system to generate automatic annotations and automatic speaker roles produced comparable performance to the system using features from manual annotations whenever available. [sent-209, score-0.64]

71 176 We then focused on the condition of using features from manual annotations when available and added prosodic features as described in Section 3. [sent-217, score-0.636]

72 7% absolute gain on F1 on agreement detection, and 1. [sent-220, score-0.276]

73 Table 2: Precision (%), recall (%), and F1 (%) of (dis)agreement detection using manual annotations without and with prosodic features. [sent-222, score-0.684]

74 172 Note that only about 10% utterances among all data are involved in (dis)agreement. [sent-229, score-0.15]

75 This indicates a highly imbalanced data set as one class is more heavily represented than the other/others. [sent-230, score-0.14]

76 We suspected that this high imbalance has played a major role in the high precision and low recall results we obtained so far. [sent-231, score-0.092]

77 Various approaches have been studied to handle imbalanced data for classifications, 377 trying to balaNnce the class distribution in the training set by eithNer oversaNmpling the minority class or downsamplinNg the maNjority class. [sent-232, score-0.261]

78 In this preliminary study of NNsamplingN Napproaches for handling imbalanced dataN NNfor CRF Ntraining, we investigated two apprNoaches, rNNandom dNownsampling and ensemble dowNnsamplinNgN. [sent-233, score-0.192]

79 RandoNm downsampling randomly dowNnsamples NNthe majorNity class to equate the number Nof minoritNNy and maNjority class samples. [sent-234, score-0.263]

80 Ensemble NdownsampNNling is a N refinement of random downsamNpling whiNNch doesn’Nt discard any majority class samNples. [sent-235, score-0.086]

81 InstNNead, we pNartitioned the majority class samNples into NN subspaNces with each subspace containiNng the samNe numbNer of samples as the minority clasNs. [sent-236, score-0.089]

82 Then wNe train N CRF models, each based on thNe minoritNy class samples and one disjoint partitionN Nfrom the N subspaces. [sent-237, score-0.05]

83 During testing, the posterioNr probability for one utterance is averaged over the N CRF models. [sent-238, score-0.063]

84 The results from these two sampling approaches as well as the baseline are shown in Table 3. [sent-239, score-0.096]

85 Both sampling approaches achieved significant improvement over the baseline, i. [sent-240, score-0.096]

86 , train- ing on the original data set, and ensemble downsampling produced better performance than downsampling. [sent-242, score-0.265]

87 We noticed that both sampling approaches degraded slightly in precision but improved significantly in recall, resulting in 4. [sent-243, score-0.143]

88 5% absolute gain on F1 for agreement detection and 4. [sent-244, score-0.431]

89 Table 3: Precision (%), recall (%), and F1 (%) of (dis)agreement detection without sampling, with random downsampling and ensemble downsampling. [sent-246, score-0.501]

90 892 In conclusion, this paper presents our work on detection of agreements and disagreements in English broadcast conversation data. [sent-254, score-0.697]

91 We explored a variety of features, including lexical, structural, durational, and prosodic features. [sent-255, score-0.337]

92 We experimented these features using a linear-chain conditional random fields model and conducted supervised training. [sent-256, score-0.192]

93 We observed significant improvement from adding prosodic features and employing two sampling approaches, random downsampling and ensemble downsampling. [sent-257, score-0.731]

94 9% (F1) for disagreement detection, on English broadcast conversation data. [sent-264, score-0.498]

95 In future work, we plan to continue adding and refining features, explore dependencies between features and contextual cues with respect to agreements and disagreements, and investigate the efficacy of other machine learning approaches such as Bayesian networks and Support Vector Machines. [sent-265, score-0.304]

96 Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. [sent-270, score-0.043]

97 Identifying agreement and disagreement in conversational speech: Use ofbayesian networks to model pragmatic dependencies. [sent-280, score-0.342]

98 Enriching speech recognition with automatic detection of sentence boundaries and disfluencies. [sent-321, score-0.155]

similar papers computed by tfidf model

tfidf for this paper:

wordName wordTfidf (topN-words)

[('dis', 0.429), ('prosodic', 0.261), ('hillard', 0.223), ('broadcast', 0.209), ('luc', 0.201), ('agreement', 0.197), ('conversation', 0.184), ('downsampling', 0.163), ('detection', 0.155), ('speaker', 0.152), ('utterances', 0.15), ('durational', 0.143), ('initiating', 0.13), ('annotations', 0.113), ('manual', 0.11), ('person', 0.105), ('disagreement', 0.105), ('crf', 0.104), ('ensemble', 0.102), ('hahn', 0.099), ('prefaces', 0.098), ('imbalanced', 0.09), ('markers', 0.085), ('agreements', 0.084), ('galley', 0.081), ('roles', 0.076), ('features', 0.076), ('explored', 0.076), ('duration', 0.075), ('germesin', 0.065), ('janin', 0.065), ('manjority', 0.065), ('mccowan', 0.065), ('soundbites', 0.065), ('spurt', 0.065), ('disagreements', 0.065), ('discourse', 0.064), ('sampling', 0.064), ('utterance', 0.063), ('disfluencies', 0.063), ('denoted', 0.06), ('adjacency', 0.059), ('precoda', 0.057), ('arl', 0.057), ('speakers', 0.057), ('participant', 0.056), ('formulations', 0.056), ('structural', 0.055), ('icsi', 0.053), ('response', 0.05), ('dialog', 0.05), ('class', 0.05), ('iarpa', 0.05), ('thep', 0.05), ('precision', 0.047), ('extreme', 0.046), ('fields', 0.045), ('ami', 0.045), ('transcriptions', 0.045), ('ostendorf', 0.045), ('recall', 0.045), ('efficacy', 0.043), ('vowel', 0.043), ('annotation', 0.043), ('gale', 0.043), ('absolute', 0.043), ('shriberg', 0.042), ('pitch', 0.042), ('nt', 0.041), ('networks', 0.04), ('minority', 0.039), ('dat', 0.039), ('mentions', 0.039), ('energy', 0.038), ('opportunity', 0.038), ('posteriors', 0.037), ('consecutive', 0.037), ('conversations', 0.036), ('gain', 0.036), ('random', 0.036), ('globally', 0.036), ('conditional', 0.035), ('held', 0.034), ('ic', 0.033), ('statement', 0.033), ('transcripts', 0.032), ('approaches', 0.032), ('presence', 0.031), ('negation', 0.031), ('lexical', 0.031), ('adding', 0.029), ('ro', 0.029), ('detecting', 0.029), ('release', 0.029), ('thne', 0.029), ('hain', 0.029), ('lincoln', 0.029), ('microphone', 0.029), ('disagrees', 0.029), ('pausing', 0.029)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 1.0000004 95 acl-2011-Detection of Agreement and Disagreement in Broadcast Conversations

Author: Wen Wang ; Sibel Yaman ; Kristin Precoda ; Colleen Richey ; Geoffrey Raymond

2 0.23491667 257 acl-2011-Question Detection in Spoken Conversations Using Textual Conversations

Author: Anna Margolis ; Mari Ostendorf

Abstract: We investigate the use of textual Internet conversations for detecting questions in spoken conversations. We compare the text-trained model with models trained on manuallylabeled, domain-matched spoken utterances with and without prosodic features. Overall, the text-trained model achieves over 90% of the performance (measured in Area Under the Curve) of the domain-matched model including prosodic features, but does especially poorly on declarative questions. We describe efforts to utilize unlabeled spoken utterances and prosodic features via domain adaptation.

3 0.22259627 228 acl-2011-N-Best Rescoring Based on Pitch-accent Patterns

Author: Je Hun Jeon ; Wen Wang ; Yang Liu

Abstract: In this paper, we adopt an n-best rescoring scheme using pitch-accent patterns to improve automatic speech recognition (ASR) performance. The pitch-accent model is decoupled from the main ASR system, thus allowing us to develop it independently. N-best hypotheses from recognizers are rescored by additional scores that measure the correlation of the pitch-accent patterns between the acoustic signal and lexical cues. To test the robustness of our algorithm, we use two different data sets and recognition setups: the first one is English radio news data that has pitch accent labels, but the recognizer is trained from a small amount ofdata and has high error rate; the second one is English broadcast news data using a state-of-the-art SRI recognizer. Our experimental results demonstrate that our approach is able to reduce word error rate relatively by about 3%. This gain is consistent across the two different tests, showing promising future directions of incorporating prosodic information to improve speech recognition.

4 0.14955951 21 acl-2011-A Pilot Study of Opinion Summarization in Conversations

Author: Dong Wang ; Yang Liu

Abstract: This paper presents a pilot study of opinion summarization on conversations. We create a corpus containing extractive and abstractive summaries of speaker’s opinion towards a given topic using 88 telephone conversations. We adopt two methods to perform extractive summarization. The first one is a sentence-ranking method that linearly combines scores measured from different aspects including topic relevance, subjectivity, and sentence importance. The second one is a graph-based method, which incorporates topic and sentiment information, as well as additional information about sentence-to-sentence relations extracted based on dialogue structure. Our evaluation results show that both methods significantly outperform the baseline approach that extracts the longest utterances. In particular, we find that incorporating dialogue structure in the graph-based method contributes to the improved system performance.

5 0.12757356 83 acl-2011-Contrasting Multi-Lingual Prosodic Cues to Predict Verbal Feedback for Rapport

Author: Siwei Wang ; Gina-Anne Levow

Abstract: Verbal feedback is an important information source in establishing interactional rapport. However, predicting verbal feedback across languages is challenging due to languagespecific differences, inter-speaker variation, and the relative sparseness and optionality of verbal feedback. In this paper, we employ an approach combining classifier weighting and SMOTE algorithm oversampling to improve verbal feedback prediction in Arabic, English, and Spanish dyadic conversations. This approach improves the prediction of verbal feedback, up to 6-fold, while maintaining a high overall accuracy. Analyzing highly weighted features highlights widespread use of pitch, with more varied use of intensity and duration.

6 0.10132807 260 acl-2011-Recognizing Authority in Dialogue with an Integer Linear Programming Constrained Model

7 0.09596923 312 acl-2011-Turn-Taking Cues in a Human Tutoring Corpus

8 0.086761974 194 acl-2011-Language Use: What can it tell us?

9 0.083830737 118 acl-2011-Entrainment in Speech Preceding Backchannels.

10 0.083428815 185 acl-2011-Joint Identification and Segmentation of Domain-Specific Dialogue Acts for Conversational Dialogue Systems

11 0.074445367 223 acl-2011-Modeling Wisdom of Crowds Using Latent Mixture of Discriminative Experts

12 0.071971551 101 acl-2011-Disentangling Chat with Local Coherence Models

13 0.070336707 119 acl-2011-Evaluating the Impact of Coder Errors on Active Learning

14 0.070284218 77 acl-2011-Computing and Evaluating Syntactic Complexity Features for Automated Scoring of Spontaneous Non-Native Speech

15 0.067208588 157 acl-2011-I Thou Thee, Thou Traitor: Predicting Formal vs. Informal Address in English Literature

16 0.066962503 203 acl-2011-Learning Sub-Word Units for Open Vocabulary Speech Recognition

17 0.066655338 39 acl-2011-An Ensemble Model that Combines Syntactic and Semantic Clustering for Discriminative Dependency Parsing

18 0.066389941 53 acl-2011-Automatically Evaluating Text Coherence Using Discourse Relations

19 0.062278006 269 acl-2011-Scaling up Automatic Cross-Lingual Semantic Role Annotation

20 0.060727894 7 acl-2011-A Corpus for Modeling Morpho-Syntactic Agreement in Arabic: Gender, Number and Rationality

similar papers computed by lsi model

lsi for this paper:

topicId topicWeight

[(0, 0.159), (1, 0.051), (2, -0.048), (3, -0.001), (4, -0.144), (5, 0.16), (6, 0.033), (7, -0.031), (8, -0.007), (9, 0.051), (10, 0.001), (11, -0.012), (12, -0.053), (13, 0.031), (14, 0.007), (15, 0.011), (16, -0.076), (17, -0.05), (18, 0.026), (19, -0.104), (20, 0.023), (21, -0.034), (22, -0.137), (23, 0.225), (24, 0.05), (25, 0.109), (26, 0.185), (27, -0.088), (28, -0.092), (29, 0.022), (30, -0.074), (31, -0.033), (32, 0.114), (33, -0.062), (34, -0.011), (35, 0.009), (36, 0.096), (37, -0.006), (38, -0.041), (39, -0.027), (40, -0.035), (41, 0.026), (42, -0.029), (43, -0.009), (44, -0.066), (45, -0.021), (46, -0.002), (47, -0.076), (48, -0.004), (49, -0.036)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.93768847 95 acl-2011-Detection of Agreement and Disagreement in Broadcast Conversations

Author: Wen Wang ; Sibel Yaman ; Kristin Precoda ; Colleen Richey ; Geoffrey Raymond

2 0.87191492 228 acl-2011-N-Best Rescoring Based on Pitch-accent Patterns

Author: Je Hun Jeon ; Wen Wang ; Yang Liu

3 0.82910854 257 acl-2011-Question Detection in Spoken Conversations Using Textual Conversations

Author: Anna Margolis ; Mari Ostendorf

4 0.81123066 83 acl-2011-Contrasting Multi-Lingual Prosodic Cues to Predict Verbal Feedback for Rapport

Author: Siwei Wang ; Gina-Anne Levow

5 0.74564099 223 acl-2011-Modeling Wisdom of Crowds Using Latent Mixture of Discriminative Experts

Author: Derya Ozkan ; Louis-Philippe Morency

Abstract: In many computational linguistic scenarios, training labels are subjectives making it necessary to acquire the opinions of multiple annotators/experts, which is referred to as ”wisdom of crowds”. In this paper, we propose a new approach for modeling wisdom of crowds based on the Latent Mixture of Discriminative Experts (LMDE) model that can automatically learn the prototypical patterns and hidden dynamic among different experts. Experiments show improvement over state-of-the-art approaches on the task of listener backchannel prediction in dyadic conversations.

6 0.70705432 118 acl-2011-Entrainment in Speech Preceding Backchannels.

7 0.69424844 312 acl-2011-Turn-Taking Cues in a Human Tutoring Corpus

8 0.54855263 77 acl-2011-Computing and Evaluating Syntactic Complexity Features for Automated Scoring of Spontaneous Non-Native Speech

9 0.49301234 301 acl-2011-The impact of language models and loss functions on repair disfluency detection

10 0.47972339 260 acl-2011-Recognizing Authority in Dialogue with an Integer Linear Programming Constrained Model

11 0.47584495 249 acl-2011-Predicting Relative Prominence in Noun-Noun Compounds

12 0.42453712 306 acl-2011-Towards Style Transformation from Written-Style to Audio-Style

13 0.39139917 194 acl-2011-Language Use: What can it tell us?

14 0.38544855 21 acl-2011-A Pilot Study of Opinion Summarization in Conversations

15 0.37218103 165 acl-2011-Improving Classification of Medical Assertions in Clinical Notes

16 0.37081116 157 acl-2011-I Thou Thee, Thou Traitor: Predicting Formal vs. Informal Address in English Literature

17 0.35862374 215 acl-2011-MACAON An NLP Tool Suite for Processing Word Lattices

18 0.35057676 320 acl-2011-Unsupervised Discovery of Domain-Specific Knowledge from Text

19 0.34583238 74 acl-2011-Combining Indicators of Allophony

20 0.34546834 119 acl-2011-Evaluating the Impact of Coder Errors on Active Learning

similar papers computed by lda model

lda for this paper:

topicId topicWeight

[(5, 0.046), (17, 0.058), (26, 0.016), (31, 0.012), (37, 0.079), (39, 0.044), (41, 0.09), (44, 0.259), (55, 0.031), (59, 0.042), (72, 0.06), (88, 0.015), (91, 0.046), (96, 0.125)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.79336953 95 acl-2011-Detection of Agreement and Disagreement in Broadcast Conversations

Author: Wen Wang ; Sibel Yaman ; Kristin Precoda ; Colleen Richey ; Geoffrey Raymond

2 0.77322519 135 acl-2011-Faster and Smaller N-Gram Language Models

Author: Adam Pauls ; Dan Klein

Abstract: N-gram language models are a major resource bottleneck in machine translation. In this paper, we present several language model implementations that are both highly compact and fast to query. Our fastest implementation is as fast as the widely used SRILM while requiring only 25% of the storage. Our most compact representation can store all 4 billion n-grams and associated counts for the Google n-gram corpus in 23 bits per n-gram, the most compact lossless representation to date, and even more compact than recent lossy compression techniques. We also discuss techniques for improving query speed during decoding, including a simple but novel language model caching technique that improves the query speed of our language models (and SRILM) by up to 300%.

3 0.75243455 150 acl-2011-Hierarchical Text Classification with Latent Concepts

Author: Xipeng Qiu ; Xuanjing Huang ; Zhao Liu ; Jinlong Zhou

Abstract: Recently, hierarchical text classification has become an active research topic. The essential idea is that the descendant classes can share the information of the ancestor classes in a predefined taxonomy. In this paper, we claim that each class has several latent concepts and its subclasses share information with these different concepts respectively. Then, we propose a variant Passive-Aggressive (PA) algorithm for hierarchical text classification with latent concepts. Experimental results show that the performance of our algorithm is competitive with the recently proposed hierarchical classification algorithms.

4 0.73789489 295 acl-2011-Temporal Restricted Boltzmann Machines for Dependency Parsing

Author: Nikhil Garg ; James Henderson

Abstract: We propose a generative model based on Temporal Restricted Boltzmann Machines for transition based dependency parsing. The parse tree is built incrementally using a shiftreduce parse and an RBM is used to model each decision step. The RBM at the current time step induces latent features with the help of temporal connections to the relevant previous steps which provide context information. Our parser achieves labeled and unlabeled attachment scores of 88.72% and 91.65% respectively, which compare well with similar previous models and the state-of-the-art.

5 0.71787059 12 acl-2011-A Generative Entity-Mention Model for Linking Entities with Knowledge Base

Author: Xianpei Han ; Le Sun

Abstract: Linking entities with knowledge base (entity linking) is a key issue in bridging the textual data with the structural knowledge base. Due to the name variation problem and the name ambiguity problem, the entity linking decisions are critically depending on the heterogenous knowledge of entities. In this paper, we propose a generative probabilistic model, called entitymention model, which can leverage heterogenous entity knowledge (including popularity knowledge, name knowledge and context knowledge) for the entity linking task. In our model, each name mention to be linked is modeled as a sample generated through a three-step generative story, and the entity knowledge is encoded in the distribution of entities in document P(e), the distribution of possible names of a specific entity P(s|e), and the distribution of possible contexts of a specific entity P(c|e). To find the referent entity of a name mention, our method combines the evidences from all the three distributions P(e), P(s|e) and P(c|e). Experimental results show that our method can significantly outperform the traditional methods. 1

6 0.67483699 265 acl-2011-Reordering Modeling using Weighted Alignment Matrices

7 0.6069212 65 acl-2011-Can Document Selection Help Semi-supervised Learning? A Case Study On Event Extraction

8 0.60236543 126 acl-2011-Exploiting Syntactico-Semantic Structures for Relation Extraction

9 0.60000044 196 acl-2011-Large-Scale Cross-Document Coreference Using Distributed Inference and Hierarchical Models

10 0.59985697 119 acl-2011-Evaluating the Impact of Coder Errors on Active Learning

11 0.59830749 324 acl-2011-Unsupervised Semantic Role Induction via Split-Merge Clustering

12 0.59739369 58 acl-2011-Beam-Width Prediction for Efficient Context-Free Parsing

13 0.59729147 32 acl-2011-Algorithm Selection and Model Adaptation for ESL Correction Tasks

14 0.59723139 40 acl-2011-An Error Analysis of Relation Extraction in Social Media Documents

15 0.5971781 209 acl-2011-Lexically-Triggered Hidden Markov Models for Clinical Document Coding

16 0.59710246 108 acl-2011-EdIt: A Broad-Coverage Grammar Checker Using Pattern Grammar

17 0.59564704 190 acl-2011-Knowledge-Based Weak Supervision for Information Extraction of Overlapping Relations

18 0.59334671 145 acl-2011-Good Seed Makes a Good Crop: Accelerating Active Learning Using Language Modeling

19 0.59141254 86 acl-2011-Coreference for Learning to Extract Relations: Yes Virginia, Coreference Matters

20 0.59120739 246 acl-2011-Piggyback: Using Search Engines for Robust Cross-Domain Named Entity Recognition