Andrew B. Goldberg et. al, NAACL 2009 Reading Group: 26 August, 2015 Presenter: Sapna Negi, UNLP, INSIGHT NUIG

Transcription

1 May All Your Wishes Come True: A Study of Wishes and How to Recognize Them Andrew B. Goldberg et. al, NAACL 2009 Reading Group: 26 August, 2015 Presenter: Sapna Negi, UNLP, INSIGHT NUIG

2 Outline: Part 1 Statistics of what people wish for new year - Wish corpus: New year (2008) wishes Manual Labels: topic distribution of wishes - Automatic topic based clustering of text from the wishes

3 Outline: Part 2 New task: Automatic detection of wish expressions in texts - Data: product reviews, political discussions - Methodology: Supervised Classification - Evaluation

4 Part 1: Statistics of the Wish Corpus

5 Wish Corpus : Statistics In December 2007, web users sent in their wishes for the new year Virtual Wishing Wall (website). Were dropped during Times Square new year ball drop. Over 100,000 wishes collected Optional state/country location entered by the wisher Average length : 8 tokens Used: 89,574 wishes written in English

6 Wish Distributions: Topics Manually annotated random subsample of 5,000 wishes Individual requests: i wish for a new puppy solicitations: call me , visit website.com sinister: to take over the world

7 Wish Distributions: Scope Manually annotated random subsample of 5,000 wishes

8 Wish Distributions: Geography Manually annotated random sample of 5,000 wishes - Valid location information: Covers all US states and other Continents, excluding Antarctica - Two categories: US(3600) and Non- US(400)

9 Wish Distributions: Geography and Topics Do the topics/scopes differ by geography? - 2 Pearson Chi square tests - Each for topic and scope - Statistically significant differences between topics/scope for US and Non -US

10 Wish Distributions: Geography Manually annotated random sample of 5,000 wishes - Valid location information: Two categories: US(3600) and Non- US(400) My opinion: Stratified random sampling, rather than random sampling. 1. Divide the full corpus into US and non-us 2. Equal no. of wishes chosen randomly from each

11 Wish Corpus: Latent Topic Modeling Same topics appear in the full corpus? So far manual labeling of 5000 wishes Automatic identification of topics in ~90000 : Verify the manual labels Each wish is a short document 12 clusters/topics

12 Wish Corpus: Latent Topic Modeling

13 Wish Corpus: Latent Topic Modeling My Opinion: Automatic analysis of wish corpus Verify/Evaluate the analysis with manual labels - The statistics with manual label were on a sample (not representative) - No evaluation of topic modeling based on manual labels - No statistics of the full corpus after topic modeling

14 Part 2 Automatic Detection of Wishes

15 Wish Detection: Motivation Why detect wishes? Wishes add a novel dimension to sentiment analysis, opinion mining Knowing what people explicitly want, not just what they like or dislike Great camera. Indoor shots with a flash are not quite as good as 35mm. I wish the camera had a higher optical zoom so that I could take even better wildlife photos. Automatic wish detector can provide political value & business intelligence

16 Wish Detection: Task definition Novel NLP task: Given sentence S, classify S as wish or non-wish Target Domains: Product reviews, political discussions Approach: - Label training data in target domains - Supervised learning, SVM classifier - Use the wish templates extracted from Wish corpus, as features

17 Wish Detection: Datasets 2 small corpora Manually labeled sentences as wish or non-wish: - Electronic product reviews: sentences, 12% wishes - Political discussion board: 6379 sentences. 34% wishes Download from

18 Wish Detection: Baselines 1. Manual: - Rule based classifier - 13 templates identified by native English speakers. Eg. I wish_, I hope_, if only_ 2. Words as features: - SVM light - Use words as features

19 Wish Detection: Learning wish templates Approach: Automatically extract wish templates from new year wishes. Use wish templates as features. Key idea: Exploit redundancy in the content of wishes - Short wish formats -> only content : world peace, health and happiness - Longer wish formats -> content + other words : I wish for world peace, I wish for health and happiness Extracted template: I wish for _

20 Wish Detection: Learning wish templates Formally, build a bipartite graph: Two kind of nodes: Content nodes c C on left, Template nodes t T on right Two kind of edges: c t (weighted by # times content appears in the template) t c (weighted by # times template matches a content node) Content Template

21 Wish Detection: Ranking wish templates Useful templates match many complete wishes but few content-only wishes Rank all template nodes t by score(t) = in(t) - out(t) Subtracting the out-degree eliminates bad templates that contain specific topical content (e.g., and happiness )

22 Wish Detection: Wish Templates as features Some of the templates

23 Wish Detection: Results 10-fold cross validation, linear classifier (SVM light using default parameters)

24 Wish Detection: Results Dataset Manual Words Templates Words + Templates Politics Products Metrics used: Area Under curve (AUC)

25 Conclusion Positives: - Introduced wish corpus - Proposed a new task - Wish tagged dataset made available - Wish templates, which are transferable to other domains - Improved results over the baseline

26 Conclusion Negatives: - Part 1 do not say much, except that wish corpus could be used for template extraction - The wish corpus statistics are questionable - Details of data annotation are not provided, annotation of wishes can be quite subjective.

27 Conclusion Relevance to my work - Suggestion Mining (I wish there was a kettle in the room, they should provide electric kettle in the room) - Suggestions might be in the form of wishes. - New results for wish detection (Negi and Buitelaar, 2015)

28 Conclusion Relevance to my work: - Suggestion Mining (I wish there was a kettle in the room, they should provide electric kettle in the room) - Suggestions might be in the form of wishes. - New results for wish detection (Negi and Buitelaar, 2015) Goldberg et. al. Negi and Buitelaar Politics % Products % Improvement