Sentiment Classification. in a Nutshell. Cem Akkaya, Xiaonan Zhang

Sentiment Classification in a Nutshell Cem Akkaya, Xiaonan Zhang

Outline Problem Definition Level of Classification Evaluation Mainstream Method Conclusion

Problem Definition Sentiment is the overall emotion, judgment or opinion towards the subject matter expressed by the author Sentiment Classification is the task of determining the overall sentiment properties of a text (e.g. document, sentence) binary ternary scale based

Applications Review Classification Well as usual Keanu Reeves is nothing special, but surprisingly, the very talented + Laurence Fishbourne is not good either. I hope they do not shoot a sequel The actors are first grade + and it has a really well thought out story line. I've seen it 10 times and I'll watch it a few more. Enjoy!

Applications Review Classification Well as usual Keanu Reeves is nothing special, but surprisingly, the very talented + Laurence Fishbourne is not good either. I hope they do not shoot a sequel The actors are first grade + and it has a really well thought out story line. I've seen it 10 times and I'll watch it a few more. Enjoy!

Applications Review Classification Well as usual Keanu Reeves is nothing special, but surprisingly, the very talented + Laurence Fishbourne is not good either, I hope they do not shoot a sequel The actors are first grade + and it has a really well thought out story line. I've seen it 10 times and I'll watch it a few more. Enjoy!

Applications Product review mining Opinion Question Answering Trends & Buzz Analysis Document Summarization

Classification Level - Document level Assumption: one-sided opinion throughout the whole document review classification buzz analysis Similar to topic based document classification Pang et. al 2002 (83%) < (above 90%) sentiment of a text piece depends on context, world knowledge and even the narrative style

Classification Level - Sentence level Assumption: one-sided opinion throughout the whole sentence More fine-grained more elaborate applications opinion question answering document summarization

Classification Level - Feature level Concerned with sentiments expressed towards certain aspects of a topic 3-step approach identification of targets identification of the opinions about the target determination of the polarity of the identified opinions Product review mining topic product aspect features Although the picture quality is very good, I am dissatisfied with the battery life

Evaluation Manually created Gold standard datasets Annotator aggreement Kappa statistics Evaluation metric : accuracy Evaluated on different annotator sets separately intersection of different annotator sets

Mainstream Methods Aggregating opinion words Good story and excellent performances!

Mainstream Methods Aggregating opinion words Good + story and excellent + performances! One global classifier Good story and excellent performances! f1, f2, fn Classifier

Mainstream Methods Aggregating opinion words One global classifier

Aggregating opinion words Questions How to pick representative words? What is the semantic orientation (SO) of a word? How to aggregate the SO scores?

How to pick representative words? Opinion words Words: Adjectives [1], Verbs, Adverbs, Nouns and/or their combinations [2] Phrases containing adjectives and adverbs [3] Lexico-syntactic patterns: way with <np>, expense of <np> [4] [1]. Hatzivassiloglou & McKeown 1997, Kamps & Marx 2002, Andreevskaia & Bergler 2006 [2]. Esuli & Sebastiani 2006,Yu and Hatzivassiloglou 2003 [3]. Turney 2002, Takamura, Inui & Okumura 2007 [4]. Riloff & Wiebe 2003

What is the SO of a word? Manually compiled lexicon Corpus based training Constraints-based Co-occurrence based Dictionary-based extraction

Manually Compiled Lexicon Expert-identified words + semantic orientation positive + negative + strength (Tong 2001) positive + negative + neutral (Das and Chen 2001) Pros Able to incorporate expert domain knowledge Easy to utilize the sentiment strength information Cons Domain specific: inflexible, unadaptable Human intuition may not be correct or comprehensive (Pang et al. 2002)

Corpus Mining: Constraints Based (Hatzivassiloglou and McKeown 1997) Utilize linguistic constraints Parallel structure Nice and comfortable, Nice and stupid* and /or / either or / neither nor, but Morphological relationship: thoughtful vs. thoughtless Best accuracy of 92.37%

Corpus Mining: Constraints Based (Hatzivassiloglou and McKeown 1997) Supervised link prediction Unsupervised clustering scenic nice handsome painful terrible fun expensive comfortable Wiebe et al. EUROLAN SUMMER SCHOOL 2007

Corpus Mining: Co-occurrence Based (Turney 2002, Yu and Hatzivassiloglou 2003) Assumption: words with similar SO tend to co-occur Basic Procedure Select Seed Set: S+ and S- Compute SO score of other words based on cooccurrence statistics unsupervised/semi-supervised learning

Turney 2002 {excellent, poor} Altavista Seeds: S+, S- Separate Corpus co-occurrence statistics Pointwise Mutual Information (PMI) P( w1 NEAR w2) PMI ( w1, w2) = ; p( w1) p( w2) SO( w) = PMI ( w, excellent) - PMI ( w, poor) Opinion Words = Seeds + Similar words

Yu and Hatzivassiloglou 2003 Subsets of 1,336 adjectives from (Hatzivassiloglou and McKeown 1997) Seeds: S+, S- Separate Corpus WSJ co-occurrence statistics Modified Log-Likelihood Ratio Freq( Wi, POS j, S + ) + ε Freq( Wall, POS j, S + ) L( Wi, POS j ) = log Freq( Wi, POS j, S ) + ε Freq( Wall, POS j, S ) Opinion Words = Seeds + Similar words

Dictionary-based approach Utilize on-line dictionaries: esp. WordNet Synonym gloss Similar word Morphologically related word antonym

Use WordNet Relations A semantic network Similar to corpus-based methods! An initial seed set Expand via relations (bootstrapping) Synonym/hyponym: same SO Antonym: opposite SO Caveat: SO conflicts (e.g. cheap vs. expensive, cheap and cheesy) Heuristics Combined evidence from multiple occurrences In-context disambiguation

Hu & Liu 2004 marvelous Seeds 30 common adjectives: {great, fantastic, nice, cool..} {bad, dull } defective terrific S+ S- boring neat outstanding inferior spoiled synonym Conflict resolution: use the first found orientation antonym

Kim & Hovy 2004 Similar seed set expansion Seeds: 23+ /21- verbs, 15+/19- adjectives 2 iterations of bootstraping: 2840+ /3239- verbs, 5880+/ 6233- adjectives More informed conflict resolution Soft classification with combined evidence Using idea of document classification: Naïve Bayes arg max P( c w) = arg max P( c) P( w c) = arg max = arg max P( c) P( syn P( c) 1 P( f syn k 2 c).. syn count n c) ( fk, synset ( w))

Using WordNet Gloss (Esuli and Sebastiani 2005, Andreevskaia and Bergler 2006) Gloss: a collection of semantically related words (a special case of co-occurrence) Pros (compared to using relations) more words in gloss than in synsets applicable to any word with a gloss in WordNet Cons May contain irrelevant words

Esuli & Sebastiani 2005 Synonym, antonym, hyponym Gloss (vs. synonyms) as classifier input Naive Bayesian, SVM, PrTFIDF

Esuli & Sebastiani 2005 HM Test Set 657+/ 679- adjectives, from Hatzivassiloglou & McKeown, 1997 Method Hatzivassiloglou and McKeown, 1997 [Turney and Littman, 2003] AV-NEAR [Turney and Littman, 2003] 7M-NEAR [Esuli and Sebastiani, 2005] Accuracy% 78.08 87.13 80.31 87.38 TL 1,614 +/1,982- terms from the General Inquirer lexicon. [Turney and Littman, 2003] AV-NEAR [Turney and Littman, 2003] 7M-NEAR [Esuli and Sebastiani, 2005] 82.84 76.06 83.09

Popescu & Etzioni 2005 Combining Corpus and Dictionary Utilize semantic relationship between words (constraints) Conjunction and disjunction Morphological relationship Manually-designed syntactic dependency rules WordNet relations (synonym, antonym, IS-A) Conflict resolution In context disambiguation Find the SO of word w (positive, negative, or neutral) in the context of associated product feature f and sentences s

Popescu & Etzioni 2005 Unsupervised learning Relaxation labeling in Computer Vision Iteratively search for a global label assignment that maximally satisfy local constraint Initialization: PMI score as in (Turney, 2002) Evaluation Compared with PMI++, Hu++ Increased precision (78% vs. 72% for PMI++ and 75% for Hu++) Gain at context-sensitive words

3 Ways to Label SO of a Word Method Semantic Lexicon Advantage Usu. one time effort; incorporate expert knowledge Disadvantage Inflexible; limited coverage Corpus Based Reliable and adaptable (with large and appropriate corpus) Require a large amount of data; require seed set Dictionary No (manually labeled) data is needed Require seed set; inapplicable to words outside the dictionary

How to aggregate? Simple and widely used: take sum/average (Turney 2002) (Yu and Hatzivassiloglou 2003) (Hu and Liu 2004) Tried but with limited success: take harmonic mean and geometric mean (Kim and Hovy 2004) More complex: feed SO of words to a document-level classifier (Das and Chen 2001) Naive Classifier (take average) Vector Distance Classifier Discriminant-Based Classifier Adjective-Adverb Phrase Classifier Bayesian Classifier Combined Classifier (k-voting)

Mainstream Methods Aggregating opinion words One global classifier

One Global Classifier Look at the input document as a whole (do not care about SO of words) Extract features to represent the document Classify the document based on the features Supervised / Semi-supervised

Pang et al. 2002 Sentiment classification of movie reviews Simply plain text-classification Use Naïve Bayes, Maximum Entropy, SVM Features: unigram, bigram, POS, text position SVM + Unigram presence works best (82.9%) Worse result than that of topic categorization thwarted expectations : I think the material the screen writers had to work with should have resulted in a more engaging story. Not every sentence is about the movie Need some understanding of context/discourse structure

Progress along the Line Pang et al. 2002 Whitelaw et al., 2005: Add Appraisal Groups Information; Attitude & Orientation appraisal features + unigram: 90.2% accuracy Time Pang & Lee 2004: Classification based only on the most subjective sentences. 86.4% accuracy with 60% words Pang & Lee 2005: Extend to numerical rating. First run a standard n-ary classifier, then alter the outputs to assign similar labels to similar reviews (with metric labeling)

And many others (Dave et al. 2003): n-grams and many variations on n- grams (stemming, statistical substitution, proper name substitution, dependency parsing, negating modification, substrings ) (Goldberg and Zhu 2006): Rating inference using a semi-supervised graph-based approach, encoding the difference among review labels as the distance between them in the graph (Cui et al. 2006): compare 3 classifiers (Winnow, a generative n-gram language model, a discriminative classifier by (Shalev-Shwartz et al. 2004)) on product review classification using very large corpus ( ~100K)

Method Comparison Method Advantage Disadvantage Aggregate Word SO Less training time; flexible combination; apply to various levels of classification Require prior knowledge (e.g. seed set); lower accuracy (so far reported) A Global Classifier prior-knowledge free; better accuracy (so far reported) Usu. require a reasonablesized labeled data; take long time to train; applicable mostly at sentence level or higher

Conclusion Complex task Emerging discipline Promising industrial applications

Resources Polarity Lexicons HMADJ (Hatzivassiloglou and McKeown 1997) MAINICHI (Takamura et al. 2006) OPINION LEXICON (Wilson et al. 2005) GENERAL INQUIRER (Stone et al. 1966) SENTIWORDNET (Esuli and Sebastiani 2006).

Resources Datasets MOVIEREVIEWSET (Pang and Lee 2004) MPQACORPUS (Wiebe et al. 2005) PRODUCTREVIEWSET (Yi et al. 2003) BOOKREVIEWSET (Aue and Gamon, 2005) SENTENCESET (Kim and Hovy 2004)

Thanks! xiaonanz@cs.cmu.edu cem@cs.pitt.edu

References Cui, H., V. Mittal, et al. (2006). Comparative Experiments on Sentiment Classification for Online Product Reviews. Proceedings of AAAI-06, the 21st National Conference on Artificial Intelligence, Boston, US, AAAI Press. Das, S. R. and M. Y. Chen (2001). Yahoo! for Amazon: Sentiment Parsing from Small Talk on the Web. Proceedings of EFA 2001, European Finance Association Annual Conference, Barcelona, ES. Dave, K., S. Lawrence, et al. (2003). Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. Proceedings of WWW-03, 12th International Conference on the {World Wide Web}, Budapest, HU, ACM Press. Esuli, A. and F. Sebastiani (2005). Determining the semantic orientation of terms through gloss classification. Proceedings of CIKM-05, the ACM SIGIR Conference on Information and Knowledge Management, Bremen,DE, ACM Press. Esuli, A. and F. Sebastiani (2006). SentiWordNet: A Publicly Available Lexical Resource for Opinion Mining. In Proceedings of LREC-06, 5th Conference on Language Resources and Evaluation, Genova, IT, pages 417-422 Gamon, M. (2004). Sentiment classification on customer feedback data: noisy data, large feature vectors, and the role of linguistic analysis. Proceeding of COLING-04, the 20th International Conference on Computational Linguistics, Geneva, CH. Goldberg, A. B. and X. Zhu (2006). Seeing stars when there aren't many stars: Graph-based semi-supervised learning for sentiment categorization. HLT-NAACL 2006 Workshop on Textgraphs: Graph-based Algorithms for Natural Language Processing, New York, NY.

References Hu, M. and B. Liu (2004). Mining and summarizing customer reviews. Proceedings of KDD '04, the ACM SIGKDD international conference on Knowledge discovery and data mining, Seattle, US, ACM Press. Hurst, Hatzivassiloglou, V. and K. R. McKeown (1997). Predicting the semantic orientation of adjectives. Proceedings of ACL-97, 35th Annual Meeting of the Association for Computational Linguistics, Madrid, ES, Association for Computational Linguistics. Kamps, J., M. Marx, et al. (2004). Using WordNet to measure semantic orientation of adjectives. Proceedings of LREC-04, 4th International Conference on Language Resources and Evaluation, Lisbon, PT. Kim, S.-M. and E. Hovy (2004). Determining the Sentiment of Opinions. Proceedings COLING-04, the Conference on Computational Linguistics, Geneva, CH. Lehrer, A. (1974). Semantic Fields and Lexical Structure. Amsterdam, North Holland. Pang, B., Lee, L., and Vaithyanathan, S. (2002). Thumbs up? sentiment classification using machine learning techniques. In Proceedings of EMNLP-02, the Conference on Empirical Methods in Natural Language Processing, pages 79 86, Philadelphia, US. Association for Computational Linguistics. Pang, B. and L. Lee (2004). A Sentimental Education: Sentiment Analysis using Subjectivity Summarization based on Minimum Cuts. Proceedings of ACL-04, 42nd Meeting of the Association for Computational Linguistics, Barcelona, ES, Association for Computational Linguistics.

References Pang, B. and L. Lee (2005). Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. Proceedings of ACL-05, 43nd Meeting of the Association for Computational Linguistics, Ann Arbor, US, Association for Computational Linguistics. Popescu, A.-M. and O. Etzioni (2005). Extracting Product Features and Opinions from Reviews. Proceedings of HLT-EMNLP-05, the Human Language Technology Conference/Conference on Empirical Methods in Natural Language Processing, Vancouver, CA. Tong, R. M. (2001). An operational system for detecting and tracking opinions in online discussions. Working Notes of the ACM SIGIR 2001 Workshop on Operational Text Classification, New York, NY, ACM. Turney, P. (2002). Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews. Proceedings of ACL-02, 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, US, Association for Computational Linguistics. Turney, P. D. and M. L. Littman (2003). Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems 21(4): 315--346. Wiebe, J., T. Wilson, et al. (2005). Annotating Expressions of Opinions and Emotions in Language. Language Resources and Evaluation 1(2): 0-0. Whitelaw, C., Garg, N., and Argamon, S. (2005). Using appraisal taxonomies for sentiment analysis. In Proceedings of MCLC-05, the 2nd Midwest Computational Linguistic Colloquium, Columbus, US.

References Yi, J., T. Nasukawa, et al. (2003). Sentiment Analyzer: Extracting Sentiments about a Given Topic using Natural Language Processing Techniques. Proceeding of ICDM- 03, the 3ird IEEE International Conference on Data Mining, Melbourne, US, IEEE Computer Society. Mullen, T. and N. Collier (2004). Sentiment analysis using support vector machines with diverse information sources. Proceedings of EMNLP-04, 9th Conference on Empirical Methods in Natural Language Processing, Barcelon, ES. Yu, H. and V. Hatzivassiloglou (2003). Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences. Proceedings of EMNLP-03, 8th Conference on Empirical Methods in Natural Language Processing, Sapporo, JP. Zhu, X., Z. Ghahramani, et al. (2003). Semi-supervised learning using Gaussian fields and harmonic functions. ICML-03, 20th International Conference on Machine Learning. Nasukawa T. and J. Yi. (2003). Sentiment analysis: Capturing favorability using natural language processing. In K-CAP2003. Wilson, T., J. Wiebe, et al. (2005). Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. HLT-EMNLP-2005 Stone, P., D. Dunphy, et al. (1966). The General Inquirer: A Computer Approach to Content Analysis. MIT Press, Cambridge.