Why Semantic Analysis is Better than Sentiment Analysis. A White Paper by T.R. Fitz-Gibbon, Chief Scientist, Networked Insights

Transcription

1 Why Semantic Analysis is Better than Sentiment Analysis A White Paper by T.R. Fitz-Gibbon, Chief Scientist, Networked Insights

2 Why semantic analysis is better than sentiment analysis I like it, I don t like it or I have no opinion sentiment is widely used to measure how customers view a company s products and services. After all, who doesn t want to be liked? But does sentiment tell you what you really need to know? Sometimes it does, for example, when you want to understand what people are saying that could affect your brand image. Or you may be interested in how your product fares in a straight-up comparison with a competitor s. Other times, though, sentiment may not provide the insights you re after. This can be especially true when you re trying to wade through the huge numbers of mentions and comments appearing in the social media world. A promising alternative to sentiment analysis is semantic analysis. Don t be turned off by the name. Simply put, semantic analysis is a way to distill and create structure around mountains of unstructured data blog posts, social network chatter, tweets and more without preconceived ideas of whether or how they are related. Networked Insights new Topic Discovery Engine (TDE) is a semantic analysis system finely tuned to discover topics in social media posts. Semantic analysis refers to a group of methods that allow machines to discover the fundamental patterns of words or phrases that act as building blocks in a large set of text. Topics, themes, sentiment and similar elements of meaning appear as intricate weavings of those fundamental patterns. In fact, a valuable type of semantic analysis is topic discovery: the summarization of large amounts of text by automatically discovering the topics and themes within. Networked Insights new Topic Discovery Engine (TDE) is a semantic analysis system finely tuned to discover topics in social media posts. networkedinsights.com info@networkedinsights.com Networked Insights, Inc. 2011, all rights reserved 2

3 By grouping social media posts based on semantic similarity, rather than preset sentiment categories such as positive, negative and neutral, TDE can help you uncover important information for example, what exactly people are saying about your product or service; where and how they use it; the features they use most; and the enhancements or new offerings they re interested in. All of this information can ultimately drive product development, new revenue streams, and strategies for marketing, advertising and media planning. Why sentiment falls short One problem with sentiment analysis is what it cannot tell you because it only considers a small amount of the available data. Our experience shows that, on average, only about 10 percent of posts actually contain sentiment, either positive or negative and that s a generous estimate (Figure 1). This means nine out of 10 posts are neutral, revealing no sentiment, and are effectively being ignored by the analysis. Thus, with sentiment analysis you re making decisions based on what only 10 percent of the posts are saying. The 90 percent of posts that do not reveal sentiment are not all irrelevant; they just don t fall cleanly into the restrictive positive-negative view of semantics or meaning that sentiment analysis adheres to. For example, many posts about a particular smartphone may come from dedicated, loyal fans who simply have questions about using the device. These are potentially valuable posts as they indicate what users want from the device, problems they may be having with its and features that could be improved. However, customer questions such as these are rarely classified as positive or negative, so they would be missed by sentiment analysis. A second problem with sentiment analysis deals with statistical confidence in data. All methods of sentiment analysis rely on example data to design, test or validate the analysis. The accuracy and value of sentiment analysis is directly dependent on the quality or confidence of the example data Percentage of posts that contain sentiment Positive Negative None Unknown Data is based on a 500-post sentiment study we conducted. The posts were classified by 20 people each. Posts were assigned to a sentiment category based on a majority vote. Only about 10% of posts were found to contain sentiment. Figure 1 networkedinsights.com info@networkedinsights.com Networked Insights, Inc. 2011, all rights reserved 3

4 Because sentiment is subjective, this example data is based on majority opinion rather than truth. For practical reasons, we cannot determine the majority opinion of all readers for each post. Instead, the example data is obtained from a small sample of human readers labeling posts with the type of sentiment they contain (for example: positive, negative or neutral). Many companies report that, on average, approximately 65 to 75 percent of readers agree on the sentiment of a post. Assuming one of these companies asks four people about the sentiment of each post, which is very likely, statistics tells us that the company is no more than 35 percent confident it actually has a positive post when its readers identify one. The graph at the right demonstrates this fact. Data with such low confidence is a poor foundation for sentiment analysis and largely leaves it up to chance ask a different set of four readers or use a different set of posts, and results could be drastically different. Sentiment analysis is not inherently bad; for particular types of questions, it may be the right tool. But if you use it, make sure the data underlying the analysis is sound and valuable data is not being ignored. Semantic analysis gives you much more If you really want to discover and understand the conversations around your company, products, services and brand, you need to be open to what all of the data tells you. Semantic analysis is a better way to do that than sentiment analysis for several reasons. In contrast to sentiment analysis, semantic analysis can take every post from a data set into account and can even identify clear trends within groups of posts. Confidence intervals for a sample size of four readers Percent agreement % 35% Sentiment of a post When three out of four readers agree on the sentiment of a post, 35% is the highest confidence interval that ensures a majority of readers would considered a post positive. Normally, statistical significance at the 95% level is desired (for research and opinion polls). Most sentiment data only achieves statistical significance at the 35% level. Thus, most sentiment data is not statistically significant (at the 95% level). networkedinsights.com info@networkedinsights.com Networked Insights, Inc. 2011, all rights reserved 4

5 It s not limited to a positive-negative framework and doesn t exclude neutral posts, unlike sentiment analysis in the smartphone example previously discussed. In this way, semantic analysis gives you clear insights into what s happening in the aggregate across a large number of posts without your having to read all of them, an inefficient or impossible task. In short, semantic analysis can find any trend in the data as long as it exists in significant enough numbers. Another important advantage of semantic analysis is that it isn t restricted by a narrow view of meaning or semantics. Sentiment, after all, is semantics: What is the author trying to communicate in this post? But people rarely post to a social network with the intent of simply expressing that they either like or dislike a product, company or idea; most forms of meaning are more complex and varied. Semantic analysis reveals the meaning or topics that sentiment analysis ignores. A final advantage of semantic analysis is unique to Networked Insights. Our TDE uses an advanced form of semantic analysis to produce topic trees it organizes the topics it discovers into a tree-like structure, allowing you to drill into a topic to see the subtopics within it. A tree structure is highly effective for organizing large amounts of data. It makes the process of finding valuable insights, quite literally, exponentially faster than having to search a flat set of topics. In the end, it s about you and what you re looking for Ultimately, you are the best judge of information about your company. You understand your domain best, which topics are important and which are not. At the same time, it s important to inject subjectivity into the process as late as possible to avoid biasing the analytic results. Android Tablet Networked Insights topic tree using semantic analysis Android guess ipad 2 specs Android, Google HTC Flyer, tablet Android Honeycomb ipad 2 Motorola Xoom PlayBook, RIM price drop Verizon leak next gen ipads Our TDE uses an advanced form of semantic analysis to produce topic trees it organizes the topics it discovers into a tree-like structure, allowing you to drill into a topic to see the subtopics within it. The size of the node represents volume of conversation. buy an ipad retina display dual core iphone 4 networkedinsights.com info@networkedinsights.com Networked Insights, Inc. 2011, all rights reserved 5

6 Semantic analysis with TDE considers these factors. Rather than having a machine or human readers judge the subjective sentiment of every post and then aggregate some output, TDE groups similar posts and summarizes the topics. Then, at the last stage, you or another qualified professional can examine the output and decide which topics are relevant, which are not and what they mean in the given context. A tool for these times Social media information is expanding at a challenging pace, and valuable nuggets can come from the most unexpected places. Semantic analysis with TDE can help you harness and make sense of it all. Most exciting, automatic topic discovery with TDE gives you tremendous latitude around how you approach the analysis. You don t have to be certain about what you re looking for. Instead, it s a journey to discovery, not a set path that may lead to inadequate insights or misleading conclusions. With TDE s semantic analysis, you can cost-effectively learn volumes about how your company and your products and services are being judged in the marketplace so much that you ll have little time to be sentimental. Social media information is expanding at a challenging pace, and valuable nuggets can come from the most unexpected places. Semantic analysis with TDE can help you harness and make sense of it all. We love the challenge of finding insights in all this data our challenge is your success! Networked Insights was founded in 2006 by industry leaders and seasoned entrepreneurs in the fields of social media and customer intelligence. Headquarters are in Madison, WI, with offices in New York and Chicago. T.R. Fitz-Gibbon is the chief scientist at Networked Insights. His team designs the Natural Language Processing and Artificial Intelligence algorithms that power the company s software. His background is in electrical engineering, computer engineering, and computer science with a focus on machine learning. T.R. s passion lies in using machine learning and big-data techniques to find great solutions to problems that are too large and complex to have perfect solutions. networkedinsights.com info@networkedinsights.com Networked Insights, Inc. 2011, all rights reserved 6