Automated Text Analytics. Testing Manual Processing against Automated Listening

Transcription

1 Automated Text Analytics Testing Manual Processing against Automated Listening

2 Contents Executive Summary... 3 Why Is Manual Analysis Inaccurate and Automated Text Analysis on Target?... 3 Text Analytics Provides Consistency and Precision... 5 Molasses vs. Lighting... 5 Automated Text Analytics Enables More Input for Greater Accuracy... 6 Real- World Examples of Automated Customer Listening and Text Analytics... 6 Why is this important?... 8 About KANA Software, Inc KANA Software, Inc. 840 W California Ave, Suite 100, Sunnyvale CA [email protected] PAGE 2

3 Executive Summary Although automated customer listening programs using text analytics are increasingly popular, marketing-, customer- experience, and social- media professionals often hold the misconception that manual analysis of customer conversations is an adequate and accurate way of analyzing the voice of their customers. However, a closer look at the process of manually monitoring customer comments reveals increased error rates, high costs, and delayed time to insight especially in comparison with automated text analytics. This paper explores the shortcomings of manual conversation analysis, and demonstrates the value of automation technologies like KANA Experience Analytics a customer listening and text analytics system to increase analytical accuracy and time- to- insight across customer listening initiatives. Why Is Manual Analysis Inaccurate and Automated Text Analysis on Target? A person can generally classify a single user- generated comment more accurately than any automated text analytics system. But for classifying hundreds of messages, automated text analytics systems do just as well as humans. For classifying thousands, or hundreds of thousands of customer verbatim comments, the performance of automated text analysis is always better than manual. In addition to improved accuracy, automated classification can perform the task in a fraction of the time and cost as human efforts. This difference in using an automated text analytics system is reflected in the high cost inherent in market- research oriented services that analyze feedback, social media conversations, and other voice- of- the- customer data. In comparison with automated listening and text analytics, the factors which render manual categorization ineffective are a combination of subjectivity and fatigue. When a person analyzes and classifies text documents, the first one encountered gets lots of attention and careful thought. The second gets not quite so much attention, the third less still... and by the time a person gets to the 100th document he or she has begun to suffer from wavering attention and fatigue. By the time a person gets to the 500th, the accuracy with which they perform text analysis and assign categories has seriously deteriorated. Beyond a few thousand documents, the main result of manual categorization isn t better information it s eye strain. Moreover, every individual will categorize records according to their own interpretation and experience, resulting in often- inconsistent results (e.g., see real- world examples below). Even a single individual s judgment will drift over time, and it is not uncommon to apply different weightings of topic and sentiment to the same 2012 KANA Software, Inc. 840 W California Ave, Suite 100, Sunnyvale CA [email protected] PAGE 3

4 messages. Automated listening and text analytics categorize input with greater accuracy and consistency. KANA has conducted exhaustive experiments to test the effectiveness of manual classification, under conditions designed to be as favorable to manual analysis as possible. In one widely documented experiment, a subject was required to classify 100 documents into 5 categories (G. Foster, 2006). Even for this easy task, human performance when it comes to manual text analysis is far from perfect. Overall accuracy analyzing and classifying text records for the human subjects of the experiment was in other words, humans got 87.9% of the classifications correct. We also tested an automated text analytics system. To stack the deck in favor of the humans, we used only 100 comments to train the system far fewer than we would tolerate in real world circumstances then tested it on exactly the same textual data submitted to the humans for analysis. Automated text analysis scored an accuracy of 84.4%. Although manual coding scored better, the difference was quite small, far less than the difference between human performance and perfection. This testing of automated vs. manual text analysis was designed to be as flattering to manual analysis as possible a very small sample, with far too little training data for a machine learning system to optimize analytical accuracy. But even with these advantages, the superiority of manual coding was minimal. In our experiment, we heavily favored manual coding above automated text analysis by making the task easy: only 100 documents, only 5 categories, so there were only 500 document- category assignments to determine. In the real world, it is not practical to perform textual analysis with fewer than 500 documents, or with fewer than 20 categories, making for 10,000 document- category assignments. And that s a minimal case if fatigue and repetitiveness are important factors with only 500 text documents assigned for analysis, imagine how important they are with 10,000. As examined above, imagine how a person would perform analyzing and classifying 5,000 text documents across 100 categories. In standard existing client deployments, it is common for the KANA solution to classify over 500,000 documents across 200+ categories KANA Software, Inc. 840 W California Ave, Suite 100, Sunnyvale CA [email protected] PAGE 4

5 Text Analytics Provides Consistency and Precision As discussed, automated text analytics systems are proven to be more accurate than manual analysis for large data sets. In order for a business to have confidence in their analytics, accuracy and consistency are a must. However, as important as accuracy, is the need to categorize records consistently in the same category. This is important for businesses that require data trend analysis. Without precision, the data wavers over time inconsistently, thereby throwing off the most fundamental of analytical inquiries: How is my data trending? Automated listening solutions that use text analytics can accurately provide much deeper analytical categorization and support consistent trending over time. Molasses vs. Lighting Another big disadvantage of manual analysis over automated text analytics is that it s achingly slow, especially if you want to do a good job. A human has to read the message, then make a decision to include or exclude it for every category (messages can fit more than one category). If we want to do a careful job of analyzing and categorizing text records, we can expect to expend at least 5 seconds for each document- category assignment. Coding 5,000 documents against 100 categories will then take 2,500,000 seconds, or 28.9 days. That s full 24- hour days; in terms of a typical 8- hour work day, that s 86.8 work days nearly 3 man- months of labor required to do the job. Automated text analytics systems can do the entire task in a matter of seconds. Manual text analysis is inherently slow. This relegates any manual solution to a backward looking what happened historically position vs. the ability of an automated text analytics system to interpret the data in real- time. Having real- time insight into posted categories allows automated text analytics solutions to trigger any number of dynamic events that are context sensitive: specific , site redirect, case placed in call center system, etc. Automated listening solutions can support real- time analysis and automated reaction to various categories being posted KANA Software, Inc. 840 W California Ave, Suite 100, Sunnyvale CA [email protected] PAGE 5

6 Automated Text Analytics Enables More Input for Greater Accuracy Unlike automated listening and text analysis, manual reading imposes severe constraints on how large a sample can be included because of the time required to do the work. The usual sample size for manually- coded studies is 500 documents; that s close to the minimum required for meaningful statistics, and close to the maximum practical time commitment. With automated text analytics systems, it is not necessary to reduce comments to a random sample of 500. Instead, automated text analytics systems can evaluate every document of interest. This means that the statistics generated from the study will be vastly more precise. It also means that very rare events can be detected and studied events that are so rare that a sample of 500 may not include any of them and can be monitored accurately. And some of those rare events can be critically important to your business. Of course automated text analysis is not perfect, but there is no perfect system, and for moderate to large samples, automated text analytics systems are the quickest and most accurate option. Sampling removes the intricacies of the complete data landscape. This relegates any manual solution to an overall, general perspective of the data vs. an automated text analytics solution s ability to interpret emerging topics hidden deep in the detailed data. Having insight into statistical changes in the use of various words allows automated solutions to trigger an alert for further analysis. Manual review of a 1% sample won t likely catch that, for example, the unclassified topic battery failure has increased from 10:100,000 posts to 20:100,000 posts. In this way, automated listening systems that use text analytics support sifting through massive amounts of data to catch statistically significant emerging topics and alert business users. Real- World Examples of Automated Customer Listening and Text Analytics We have had the opportunity to compare the performance of manual vs. automated text analysis, when the manual effort was completed by research professionals. A potential client was considering using KANA Experience Analytics, and also considering using the services of a company which takes a random sample of customer comments and classifies them manually. They decided to undertake a head- to- head comparison. The task was to identify which customer comments, in a random sample of 500, complained that the product was too expensive KANA Software, Inc. 840 W California Ave, Suite 100, Sunnyvale CA [email protected] PAGE 6

7 The professional sampling company made a strong case, pointing out several examples for which manual text analysis was correct while automated text analysis was wrong, because of the inability of automated systems to handle subtleties of language. However, Experience Analytics performance showed even more impressive results, pointing out numerous examples for which automated coding was correct while manual analysis was wrong but not because of subtleties of the language. Automated text analytics systems make complex mistakes, but humans tend to make simple mistakes. The humans missed most of the messages that obviously fit the category, and often coded identical messages differently. In fact when overall performance was evaluated, manual analysis was dramatically inferior to automated categorization. Measured by the correlation coefficient a better measure of accuracy manual analysis scored while Experience Analytics scored In more concrete terms, humans missed 72% of the messages which fit the category (e.g., out of 47 messages complaining that the product was too expensive, they only identified 13. The automated text analytics system found 45). When the number of verbatim records for textual analysis grows large, manual performance decreases rapidly. It s also well to note that automated text analytics systems exhibit perfect consistency in a large number of cases, humans classified identical messages differently, while an automated system is guaranteed to classify identical messages identically, in every case. The cost benefits of automation similarly outweighed the expense and time delays involved in manual text analysis and categorization. A KANA customer the world s largest software company - was receiving over 500,000 direct feedback messages per month prior to implementing Experience Analytics. With an average cost of $1 per record manually processed, the company could not afford to manually analyze and report on all customer comments and was instead sampling only 5% of total inbound message volumes. Experience Analytics was able to reduce this cost of processing to pennies and enabled the inclusion of every single message in its analysis and generate accurate and highly thorough insight reports and alerts in real- time. Another KANA client receiving 15,000 messages per month in their online community was passing these messages on to their call center for manual categorization and analysis. Each message was reviewed by a service representative who tagged it to a selected topic area the resulting summaries were often inconsistent and subjective (e.g., it is difficult for a human to consider anything more than 3-4 classification codes at a time, whereas machine processing can handle thousands). The cost of this manual effort was estimated at $4.00 per record processed prior to automating text analytics and listening with Experience Analytics, and the resulting insights were of questionable quality and published only when the backlog of messages could be read/addressed (oftentimes too late to be of use for marketing departments) KANA Software, Inc. 840 W California Ave, Suite 100, Sunnyvale CA [email protected] PAGE 7

8 Why is this important? The growing importance of customer listening and social media monitoring initiatives requires systems that produce accurate insights on a timely basis. In today s competitive environment, it is no longer acceptable to wait for research teams to complete their analyses in order to act. Furthermore, manual categorization efforts have been proven to be less reliable than Experience Analytics automated listening, text analysis and categorization capabilities. Nimble companies interested in delivering highly relevant products and customer experiences now have the ability to listen continuously and generate highly accurate insights in real- time that account for every single customer voice of interest. About KANA Software, Inc. KANA makes every customer experience a good experience. A global leader in customer service solutions delivered on- premise or in the cloud, KANA lets organizations take complete control over customer service interactions, so they can take care of customers, while managing costs and reinforcing brand. By unifying and maintaining context for customer journeys across agent, web, social and mobile experiences, KANA solutions have reduced handling time, increased resolution rates and improved net promoter score (NPS) at more than 900 enterprises, including half of the Global 100 and more than 250 government agencies. KANA is based in Silicon Valley, California and has offices worldwide. Contact us at [email protected] and visit us at KANA Software, Inc. 840 W California Ave, Suite 100, Sunnyvale CA [email protected] PAGE 8