Can Deep Learning solve the Sentiment Analysis Problem? Mark Cieliebak Zurich University of Applied Sciences

Transcription

1 Can Deep Learning solve the Sentiment Analysis Problem? Mark Cieliebak Zurich University of Applied Sciences Annual Meeting of SGAICO Swiss Group for Artificial Intelligence and Cognitive Science

2 Outline 1. What is sentiment analysis? 2. How good are "classical" approaches? 3. Does deep learning solve the problem? Mark Cieliebak 2

3 About Me Mark Cieliebak Institute of Applied Information Technology (InIT) ZHAW, Winterthur Website: Software Engineering Open Data Research Interests Automated Test Generation Text Analytics Mark Cieliebak 3

4 What is Sentiment Analysis " WiFi Analytics is a free Android app that I find very handy when it comes to troubleshooting and monitoring a home network. "[1] Mark Cieliebak 4

5 Sample Application: Social Media Monitoring [7] Text Analytics Components: Find relevant documents Hot topic Analysis Sentiment analysis Mark Cieliebak 5

6 Flavours of Sentiment Analysis Document Based Sentence Based Target-Specific Rating Prediction Mark Cieliebak 6

7 Classic Approaches to Sentiment Analysis Rule-Based Corpus-Based Predicted Label [3] [4] Mark Cieliebak 7

8 Simple Sentiment Analysis Idea: Count number of positive and negative words "This camera is great[+1]." +1 (pos) "I find it beautiful[+1] and good[+1]." +2 (pos) "It looks terrible[-1]." -1 (neg) "This car has a blue color." 0 (neu) Use Sentiment-Dictionary: POSITIVE: NEUTRAL: NEGATIVE: great hello bad love see hate nice I ugly Mark Cieliebak 8

9 Sample Rules Detect Booster Words: "The car is really very expensive[ ]." New Category "Mixed": "This car has an appealing[+1] design and comfortable[+1] seats, but it is expensive[-1]. " Negation: Invert only score of words occuring after the negation: "The car is appealing[+3] and I do not[*-1] find it expensive[-2]" I do not find the car expensive and it is appealing. Need to understand the sentence Mark Cieliebak 9

10 Linguistic Analysis Sentence Sentence Conj. Sentence Noun Phrase Verb Phrase Verb Adverb Verb Noun Phrase Adj. Noun Phrase Verb Phrase Det. Det Noun Det. Verb Participle I do not find the car expensive and it is appealing -> RULE: Invert scores of words being in the same phrases as negation. I do not find the car expensive[+2] and it is appealing[+3]. +5 (pos) Mark Cieliebak 10

11 Rule-Based Sentiment Analysis Most Important Issues: - Requires good hand-crafted rules - Hard to transfer to new tasks or languages [5] - Does not work well for texts with bad grammer (Twitter) Mark Cieliebak 11

12 Classic Approaches to Sentiment Analysis Rule-Based Corpus-Based Predicted Label [3] [4] Mark Cieliebak 12

13 Corpus-Based Sentiment Analysis Predicted Label [4] Mark Cieliebak 13

14 Corpus-Based Sentiment Analysis Annotated Corpus Sentence This analysis is good. It looks awful. This car has a blue color. This car has an appealing design, comfortable seats, but it is expensive. This car has a very appealing design, comfortable seats, but it is really expensive. This analysis is not good. This car has an appealing design, comfortable seats and it is not expensive. This movie was like a horror event. This car is appealing and is not expensive. Polarity Pos Neg Neu Mix Mix Neg Mix Neg Mix Mark Cieliebak 14

15 Sample Features for Tweets Word ngrams: presence or absence of contiguous sequences of 1, 2, 3, and 4 tokens; noncontiguous ngrams POS: the number of occurrences of each part-of-speech tag Sentiment Lexica: each word annotated with tonality score ( ) Negation: the number of negated contexts Punctuation: the number of contiguous sequences of exclamation marks, question marks, and both exclamation and question marks Emoticons: presence or absence, last token is a positive or negative emoticon; Hashtags: the number of hashtags; Elongated words: the number of words with one character repeated (e.g. soooo ) from: Mohammad et al., SemEval Mark Cieliebak 15

16 Corpus-Based Sentiment Analysis Most Important Issues: - Requires large annotated corpora - Depends on good features [6] Mark Cieliebak 16

17 How good are Sentiment Analysis Tools? Mark Cieliebak 17

18 Quick Poll "How good are state-of-the-art sentiment analysis tools?" Short texts: 1-2 sentences from Twitter, news, reviews etc. Three-class classification: positive, negative, other Accuracy = # correct docs # docs Accuracy <50% 50-60% 60-70% 70-80% 80-90% >90% Votes Mark Cieliebak 21

19 Accuracy Tool Accuracy 0,8 0,7 0,6 Best Tool per Corpus Worst Tool per Corpus Avg. 61% 40% 0,5 0,4 0,3 0,2 [14] Mark Cieliebak 22

20 Accuracy Tool Accuracy 0,8 0,7 0,6 Best Tool per Corpus Worst Tool per Corpus Overall Best Tool Avg. 61% 40% 59% 0,5 0,4 0,3 0, Mark Cieliebak 23

21 Take-Home Lesson Accuracy of best commercial tool on arbitrary short texts is 59% Mark Cieliebak 24

22 Approaches to Sentiment Analysis Rule-Based Corpus-Based Pr edi cte d La bel Deep Learning [9] [8] Mark Cieliebak 25

23 Deep Learning on Text It's all about Word Vectors! Mark Cieliebak 26

24 Word2Vec [9] Huge set of text samples (billions of words) Extract dictionary Word-Matrix: k-dimensional vector for each word (k typically ) Word vector initialized randomly Train word vectors to predict next words, given a sequence of words from sample text Major contributions by Bengio et al. 2003, Collobert&Weston 2008, Socher et al. 2011, Mikolov et al Mark Cieliebak 27

25 The Magic of Word Vectors [10] King - Man + Woman Queen Live Demo on 100b words from Google News dataset: Mark Cieliebak 28

26 Relations Learned by Word2Vec [11] Mark Cieliebak 29

27 Using Word Vectors in NLP Collobert et al., 2011: SENNA: Generic NLP System based on word vectors No manual feature engineering Solves many NLP-Tasks as good as benchmark systems [12] Mark Cieliebak 30

28 Deep Learning and Sentiment Maas et al., 2011 Enrich word vectors with sentiment context Capture semantic of words (unsupervised) and sentiment (supervised) in parallel, using multiple learning tasks wonderful amazing terrible awful Mark Cieliebak 31

29 Deep Learning and Sentiment Socher et al. 2013: Word Vectors do not help for Sentiment Analysis Recursive Neural Tensor Networks Representing sentence structures as trees while adding sentiment annotations at same time Restricted to single, well-structured sentences [13] Mark Cieliebak 32

30 Deep Learning and Sentiment Quoc and Mikolov, 2014: "Paragraph Vectors" Add context (sentence, paragraph, document) to word vectors during training Improves many existing approaches [9] Mark Cieliebak 33

31 Does Deep Learning solve the Sentiment Analysis Problem? Mark Cieliebak 34

32 Conclusion: Deep Learning for Sentiment Small improvements, not revolution Very recent research, not yet "end of the story" SemEval 2015 will be benchmark Mark Cieliebak 35

33 Talk in Short! 1. Classic approaches are rule-based or corpus-based 2. State-of-the-art tools classify 4 out of 10 docs wrong 3. Deep Learning does not need hand-crafted features 4. Deep Learning improves existing benchmarks Mark Cieliebak 36

34 Thank You! Mark Cieliebak Zurich University of Applied Sciences (ZHAW) Winterthur, Switzerland Website: [15] Mark Cieliebak 37