Big Data and Opinion Mining: Challenges and Opportunities

Transcription

1 Big Data and Opinion Mining: Challenges and Opportunities Dr. Nikolaos Korfiatis Director Frankfurt Big Data Lab JW Goethe University Frankfurt, Germany /~nkorf

2 Agenda Opinion Mining and Sentiment Analysis What is it? How do we do it? Why is important? Big Data and Opinion Mining Why opinion mining is a problem of big data Applications: Opinion Fraud and Schilling

3 Opinions and Markets: A little bit of history The infamous information market historical case of Nathan Rothchild

4 A more recent example Any clue as to what happened here?

5 Ups?

6 Opinion Mining and Sentiment Analysis The computational study of opinions, sentiments and emotions expressed in textual information. It encompasses a set of data mining tools and workflows aimed on identifying an expressed opinion in a piece of text It traces its origins on NLP, Data Mining and Information Retrieval Two approaches: polarity (simpler) and sentiment(harder)

7 Some Applications Product review mining: What features of the iphone do customers like and which do they dislike? Review classification (polarity): Is a review positive or negative towards a product or service? Tracking sentiment toward topics over time: What is the general belief about the economy? (eg. IFO index) Prediction (election outcomes, market trends): Who is going to win the elections? Is the market going up or down?

8 Predicting the election outcome Politwi (

9 Twitter predicts the Gallup Poll of consumer confidence O'Connor, Balasubramanyan,Routledge, and Smith(2010) From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series

10 But not only the Gallup Poll Bollen,Mao and Zeng.(2011) Twitter mood predicts the stock market,journal of Computational Science Issue 2, Vol 1.,

11 Types of Opinion Mining Opinion Extraction: Extracting opinions from a text which has a source Opinion Classification(through subjectivity analysis): Identify if the text contains a positive, negative or neutral opinion Sentiment Assessment: Identify affection or affective states (Happiness, anger, fear etc.)

12 Sentiment analysis is the detection of attitudes (Dan Jurafsky, Stanford Univ.) 1. Holder (source) of attitude 2. Target (aspect) of attitude 3. Type of attitude From a set of types Like, love, hate, value, desire, etc. Or (more commonly) simple weighted polarity: positive, negative, neutral, together with strength 4. Text containing the attitude Sentence or entire document

13 An example (Movie Reviews) This film should be brilliant. It sounds like a great plot, the actors are first grade, and the supporting cast is good as well, and Stallone is attempting to deliver a good performance. However, it can t hold up. Well as usual Keanu Reeves is nothing special, but surprisingly, the very talented Laurence Fishbourne is not so good either, I was surprised.

14 Where Big Data comes to place Regenerating an opinion list Opinion mining is a computationally hard case Every piece of opinionated text has to be accessed against an opinion lexicon The lexicon itself has to be adjusted based on the domain Example Workflow (Rill et al., 2012): Some example workflow

15 Sentiment analysis and affection Scherer s (1984) typology of affective states (expression of an attitude) Emotion: brief organically synchronized evaluation of a major event angry, sad, joyful, fearful, ashamed, proud, elated Mood: diffuse non-caused low-intensity long-duration change in subjective feeling cheerful, gloomy, irritable, listless, depressed, buoyant Interpersonal stances: affective stance toward another person in a specific interaction friendly, flirtatious, distant, cold, warm, supportive, contemptuous Attitudes: enduring, affectively colored beliefs, dispositions towards objects or persons liking, loving, hating, valuing, desiring Personality traits: stable personality dispositions and typical behavior tendencies nervous, anxious, reckless, morose, hostile, jealous

16 Big Data Workflow Challenges to address Loading data from heterogeneus sources (Flume) Store them in a format independent filesystem (HDFS) Access the opinion lexicon (HCATALOG) Manage and scale the resources needed (Ambari) Example Architecture Hortonworks data platform white paper

17 Applications: Opinion Fraud Fake opinions planted by shills in order to direct consumers to a specific product Amazon Reviews, Roten Tomatoes, ebay reviews

18 Not only individuals do it

19 Shills on online consumer reviews Amazon reviews Are there any fake reviews on a product review page? Hu et al Fraud detection in online consumer reviews Fake Reviews are planted in the beginning

20 Other strategies e.g. ebay Dellarocas, C. (2000). Immunizing online reputation reporting systems against unfair ratings and discriminatory behavior. Unfair behavior (from buyers): Shills do not only affect the decision to purchase but also to not purchase (bad mouthing) Discriminatory behavior (from sellers): Good serviceto few that will increase the rating (e.g. in a hotel only to the customers that book through tripadvisor/booking.com)

21 The helpfulness of Big Data Opinion mining is of particular interest to cases where consumer insights are needed Cost of running such an infrastructure is over the expected benefit to SME s and small retailers Cloud platforms make the opinion mining task relatively inexpensive and easy to scale IAAS business models gain momentum (e.g. Amazon EC2, Google Compute Engine)

22 Open Challenges Domain independence Opinion lexicons are semantically attached to the corpus that is used for their generation Minimum sampling size of input for an efficient sentiment detection procedure More connectors needed (e.g. extensions to Apache Flume)

23 THANK YOU!