CAPTURING THE VALUE OF UNSTRUCTURED DATA: INTRODUCTION TO TEXT MINING

Transcription

1 CAPTURING THE VALUE OF UNSTRUCTURED DATA: INTRODUCTION TO TEXT MINING Mary-Elizabeth ( M-E ) Eddlestone Principal Systems Engineer, Analytics SAS Customer Loyalty, SAS Institute, Inc.

2 Is there valuable information locked away in your unstructured data? 2

3 CURRENT SITUATION: COMMON QUESTIONS ABOUT TEXTUAL DATA SOURCES Are there hidden insights within text data sources that can help my organization? Such as call center notes, s, news, government filings, social media How can I leverage on our textual data sources? What value can it bring? Can I also use text data to analyze and predict the future? To reduce fraud, reduce churn, improve sales, reduce costs How can I leverage on both unstructured and structured data sources? Customer data + Customer feedback? Need to leverage the most from text data!

4 WHAT IF YOU COULD. Extract key information from text data? e.g. people, places, companies See how things are related to each other? Across a large number of documents and messages? Discover main ideas/ topics across all documents and messages Find patterns across non/text data, that can predict the future

5 WHAT IF YOU COULD Discover new insights from large text data sources Extract key patterns from text data to predict the future Customers Discover current topics about your products from customer opinions Find patterns within customer feedback, that predicts good interest in upsell opportunities Fraud Public Opinion Detect anomalies from usual topics described in text reports, text applications or feedback Understand previously unknown issues/ concerns, from citizen discussions on twitter/ forums Find patterns in reports that may seem to predict/ relate to suspicious behavior Extract key opinions from citizen feedback to forecast citizen sentiments in the near future

6 WHERE IS TEXT MINING USED? Text Mining has numerous applications in any industry Government Detect fraudulent activity. Spot emerging trends and public concerns. Finance Retention of current customer base using call center transcriptions or transcribed audio. Identification of potentially fraudulent activities. Insurance Identify fraudulent claims. Track competitive intelligence. Brand management Retail Manufacturing Telecommunications Life Sciences Identify the most profitable customers and the underlying reasons for their loyalty. Brand management Reduce time to detect root cause of product issues. Identify trends in market segments. Help prevent churn and suggest up-sell/cross-sell opportunities for individual customers. Identify adverse events. Recommend appropriate research materials.

7 TEXT MINING

8 SAS Text Analytics Domain-Driven Information Organization and Access Analysis-Driven Predictive Modeling, Discover Trends and Patterns SAS Enterprise Content Categorization SAS Ontology Management SAS Text Miner SAS Sentiment Analysis

9 SAS TEXT MINER Is a complete solution, to discover insights or predict behaviour and outcomes by leveraging on data mining capabilities of SAS Enterprise Miner and SAS natural language processing (NLP)/ advanced linguistic technologies. What is Concept Extraction? To automatically locate and extract the key information from documents based on the rules & advanced linguistic logic What is Concept Linking? To look within a large corpus of text documents to discover how concepts/ key information are associated/ linked with each other. What is Topic Discovery? To analyse a large corpus of text documents to discover topics by grouping messages that has very similar content.

10 HOW DOES TEXT MINING WORK? EXPLORING & DISCOVERING INSIGHTS 1. Input text messages e.g. twitter data, reports, , news, forum messages 2. Parse & explore Text Data break down text and explore relationships of key concepts such as persons, places, organizations 3. Discover Topics cluster documents of similar content and describe them with important key words

11 HOW DOES TEXT MINING WORK? DISCOVER PATTERNS FOR PREDICTIVE MODELING 1. Input text messages with relevant structured data e.g. , call center notes, applications 2. Parse Text Data and Discover Topics Break down text into structured data, group messages of similar content 3. Predictive Modeling with text data text data input into models may provide reliable info to predict outcome & behavior Customer data Predict activity that is likely fraudulent

12 WHAT CAN WE DISCOVER? Discover relationships between concepts described in large corpus of text data how are persons, places, organizations related? Discover topics mentioned in text data what are main topics mentioned? What are the rare topics? Discover patterns related to structured data e.g. how is feedback related to customer purchase behavior?

13 EXAMPLE DISCOVERING INSIGHTS FROM CUSTOMER COMPLAINT DATA From customer complaints to engineer logs to legal documents, it is a considerable challenge to draw insights from large amounts of information, and usually unfeasible via manual means. This is even more difficult when we wish to detect concepts and patterns within the documents, in order to find trends and detect high risk events How can we analyse millions of documents quickly and identify key patterns and cases of high risk? (e.g. risk of fraudulent activity) THE DRIVER SIDE SEAT BELT SOMETIMES FAILS TO RETRACT. WHEN I PULLED THE BELT OUT, IT STAYED OUT AND WOULD NOT RETRACT. I INSPECTED THE AREA AND FOUND NO INTERFERENCE. THIS HAPPENED ON A SAT. I DROVE THE VEHICLE SAT. AND SUN WITH A FAULTY BELT. I CALLED THE DEALERS SERVICE DEPT. TOLD THEM THE PROBLEM BUT COULDN'T GET IN FOR A WEEK.

14 EXAMPLE DISCOVERING INSIGHTS FROM CUSTOMER COMPLAINT DATA SAS Text Miner automates manual comprehension of text documents, uncovering relationships and trends of concepts mentioned across documents, allowing drill down analysis and integrated with predictive modeling within SAS Enterprise Miner. In this example, we look at a large database of car faults Here, SAS Text Miner runs a Text Parsing processing on thousands of reports of car faults Recognizing and extracting entities and parts of speech Supporting a wide range of languages Into a detailed term/ document matrix Allowing us deeper analysis/ visualization of insights Car Fault Records THE DRIVER SIDE SEAT BELT SOMETIMES FAILS TO RETRACT. WHEN I PULLED THE BELT OUT, IT STAYED OUT AND WOULD NOT RETRACT. I INSPECTED THE AREA AND FOUND NO INTERFERENCE

15 EXAMPLE DISCOVERING INSIGHTS FROM CUSTOMER COMPLAINT DATA This allows us to discover relationships between concepts across all messages e.g. what is usually mentioned with issues such as brake problems? Discover topics mentioned in text data e.g. Understand the main topics: dealerships Uncover the emerging topics: Battery issues Discover patterns related to structured data e.g. Complaints on engine trouble have a higher chance of car accidents

16 EXAMPLE DISCOVERING INSIGHTS FROM CUSTOMER COMPLAINT DATA How does this help? Discovery of new insights/ topics: Text data forum messages, s, logs, records typically contain rich, yet sparse/ uncommon insights. Text mining allows you to: Parse and extract information from text data Reliably filter and retain important information Automatically group documents into similar topics, allowing discovery of important/ large topics or rare/ small topics Text mining input in Predictive modeling: Documents and records often contain important facts that can reliably predict outcomes for e.g. any mention of bad maintenance habits will likely result in earlier car failure Empowered by SAS Natural Language Processing and wide multi language support, Text mining discovers key trends within large amounts of text, to be used as clean, reliable input in data mining analysis.

17 BENEFITS SAS Text Miner helps your organization to: Uncover previously undetected associations and relationships Get a complete view data, and drill down to specific documents for more insight Automate time-consuming tasks of reading and understanding text. Analyse both text and non-text data produce predictive models that spot more opportunities and recognize trends more accurately Discover hidden patterns from text data for insights and predictive modeling!

18 SAS TEXT MINER

19 SAS TEXT MINER ANALYTICAL WORKFLOW Text Mining Raw Data Model with Structured and Unstructured Data

20 EXAMPLE TEXT MINING PROCESS FLOWS

21 EXAMPLE TEXT MINING PROCESS FLOWS Start with a table that contains either: - Documents saved as a variable (column) - A column that points to physical text files

22 EXAMPLE INPUT DATA VARIABLE CONTAINS FULL TEXT

23 EXAMPLE INPUT DATA VARIABLE CONTAINS POINTER TO TEXT FILE

24 EXAMPLE TEXT MINING PROCESS FLOWS Apply natural language processing algorithms to parse the documents and quantify information about the terms in the corpus.

25 TEXT PARSING NODE Tokenization - break sentences or documents into terms Stemming - identify the root form of a word (run, runs, running, ran, etc.) Synonyms Remove low-information words such as a, an, and the (stop list) Part of speech identification (noun, verb, etc.) Identify Standard and Custom Entities (names, places, etc.) Multiword terms or phrases ( blue screen of death ) Import custom entities, facts, and events as defined in SAS Enterprise Content Categorization (ECC) Include negation entities from SAS ECC for Sentiment Analysis

26 SUPPORTED LANGUAGES Arabic, Chinese, Dutch, English, French, German, Italian, Japanese, Korean, Polish, Portuguese, Spanish, and Swedish, Czech, Danish, Finnish, Greek, Hebrew, Hungarian, Indonesian, Norwegian, Romanian, Russian, Slovak, Thai, Turkish, Vietnamese, Russian, Greek, Vietnamese, Turkish, Czech, Indonesian, Thai, Danish, Norwegian, Slovak, Finnish, Romanian, Hebrew, Hungarian, Korean New in SAS 9.3

27 EXAMPLE TEXT MINING PROCESS FLOWS Perform spell-checking and refine synonym lists. Discover related concepts using Concept Linking. Perform full text search. Subset documents and/or terms for further analysis.

28 TEXT FILTER NODE Spell checking Concept Linking Full text search Define additional synonyms Sub-setting management of terms and documents that are passed to subsequent nodes

29 FILTER VIEWER

30 SAS Text Mining

31 CONCEPT LINKING

32 EXAMPLE TEXT MINING PROCESS FLOWS Analyze the documents to create topics and assign each document to one or more topics. In addition to derived topics, users can add their own topic definitions.

33 TEXT TOPIC NODE Multiple topics per document Soft clustering using rotated SVD (PROC SVD followed by PROC FACTOR) Allows automatic creation of single and multi-word topics User defined topics and editing of automatic topics

34 INTERACTIVE TOPIC VIEWER

35 EXAMPLE TEXT MINING PROCESS FLOWS Analyze the documents to create clusters and assign each document to a single cluster.

36 CLUSTER VIEWER

37 CLUSTER VIEWER

38 EXAMPLE TEXT MINING PROCESS FLOWS Clusters can be further explored using the Segment Profile node to identify factors that differentiate data segments from the population.

39 SEGMENT PROFILE The Segment Profile node is available on the Assess tab of Enterprise Miner. It allows the examination of segmented or clustered data to identify factors that differentiate data segments from the population.

40 SEGMENT PROFILE

41 EXAMPLE TEXT MINING PROCESS FLOWS: PREDICTION Several methods are available to use the unstructured data to create predictions.

42 WHERE IS TEXT MINING USED? Text Mining has numerous applications in any industry Government Detect fraudulent activity. Spot emerging trends and public concerns. Finance Retention of current customer base using call center transcriptions or transcribed audio. Identification of potentially fraudulent activities. Insurance Identify fraudulent claims. Track competitive intelligence. Brand management Retail Manufacturing Telecommunications Life Sciences Identify the most profitable customers and the underlying reasons for their loyalty. Brand management Reduce time to detect root cause of product issues. Identify trends in market segments. Help prevent churn and suggest up-sell/cross-sell opportunities for individual customers. Identify adverse events. Recommend appropriate research materials.

43 BENEFITS SAS Text Miner helps your organization to: Uncover previously undetected associations and relationships Get a complete view data, and drill down to specific documents for more insight Automate time-consuming tasks of reading and understanding text. Analyse both text and non-text data produce predictive models that spot more opportunities and recognize trends more accurately Discover hidden patterns from text data for insights and predictive modeling!

44 LEARNING MORE

45 SAS TEXT MINER RESOURCES SAS Text Miner Product Web Site SAS Text Miner Technical Support Web Site SAS Text Miner Technical Forum (Join Today!) SAS Training Data Miner Training Path: Courses for SAS Text Miner:

46 Step-bystep how-to guide

47 Data for the step-bystep how-to guide

48 DISCUSSION FORUMS

49 DISCUSSION FORUMS

50 COMPLIMENTARY ON-DEMAND WORKSHOPS

51 THANK YOU FOR USING SAS!