Enriching Customer Data With New Customer Insights Using Big Data And Analytics

Transcription

1 Enriching Customer Data With New Customer Insights Using Big Data And Analytics Mike Ferguson Managing Director Intelligent Business Strategies Swiss BI Day Geneva, October 2015 About Mike Ferguson Mike Ferguson is Managing Director of Intelligent Business Strategies Limited. As an analyst and consultant he specialises in business intelligence, data management and enterprise business integration. With over 34 years of IT experience, Mike has consulted for dozens of companies, spoken at events all over the world and written numerous articles. Formerly he was a principal and co-founder of Codd and Date Europe Limited the inventors of the Relational Model, a Chief Architect at Teradata on the Teradata DBMS and European Managing Director of DataBase Associates. mferguson@intelligentbusiness.biz Tel/Fax (+44)

2 Topics The increasing power of the customer New Requirement - The Customer Intelligent Omni-Channel Front Office Why is new data needed for business survival? Using Big Data and analytics to enrich customer insight for competitive advantage Assessing new data sources to determine business value Text analytics and social media data Graph analytics Clickstream analytics Re-analysing enriched customer data for data-driven growth Integrating enriched customer data into the omni-channnel front office 3 Business Survival: Today Customers Are Increasingly Well Informed BEFORE They Buy On the web the customer is king New competitors More choice Primarily B2C but B2B is increasingly following the same route Easy to find Easy to compare Easy to find ratings On the move Voice of the customer Important new data sources for analysis Search data Clickstream data from web logs (including tracker data) Sentiment and social network influencer data 4 2

3 Customer Power Comparison Web Sites Are Having A Major Influence On B2C And Some B2B Buying Behaviour MySupermarket (retailer prices) Google flights (flight prices) Pricegrabber (CPG prices) USwitch (broadband prices) GoCompare (car insurance) CompareTheMarket (energy prices) 5 With Customers So Well Informed, Quality Of Product And Service Plus Smooth Operations Become Very Important Customer sentiment will be the judge Eliminate process errors 6 3

4 Requirement Is Consistent Customer Treatment Across All Channels The Smart Omni-Channel Front-Office Customers Improve customer engagement Front-Office Operations E-commerce application Customer service apps Sales Force automation apps Marketing applications Customer facing outlet applications OMNI-CHANNEL Predictive analytics Common transaction services Personalised customer Insight Personalised customer recommendations Prescriptive analytics Improve marketing, sales and service via on-demand access to insights and recommendation services relevant to each customer 7 For Most Organisations A Customer Master Data Management System Holds What Details About Customers Customer master data Transaction data MDM sales R ops distribution finance OLTP systems C customer U D DW & marts EDW mart Customer and other insights 8 4

5 Key Questions? Is traditional customer identity data in your master management system enough? Do you have all the attributes in your Customer MDM system that could be of value to your business? Do you know about all the relationships that your customers have in your Customer MDM system that could be of value to your business? Do you have all the insights about your customers either in your DW or your MDM system that could be of value to your business? 9 Why New Data? Huge Demand To Enrich Customer Master Data Source: IBM Redbook - Information Governance Principles and Practices for a Big Data Landscape 10 5

6 Improving Customer Experience Via Time Series Analysis Of All Customer Interactions OMNI channel analysis analyse all customer interactions across all channels Customer DNA identity behavioural data data social data 11 New Data - Do You Collect Data From All Inbound And Outbound Customer Interaction Points? Direct mail In-store POS Kiosks Websites Search Online advertising sites Mobile devices SMS/ MMS (inbound and outbound) Social Media Systems of Engagement Do you know how your customers interact? Customer service Call centres Client centres Events web events and physical events 12 6

7 Social Networks Are Getting Significant Business Interest Primarily From Marketing Profiles (e.g. LinkedIn) Social Graph Image source: Ratings / Likes / Dislikes Comments (e.g. Twitter) 13 The Business Value Of Social Networks And Social Network Analytics What are the dominant relationships? Who are the influencers? Sentiment? Circle of trusted friends? Influencers? How valuable? How valuable is the network? 14 7

8 Popular Big Web Data Analytic Applications That Can Help Enrich Master Data Clickstream analytics Site navigation behaviour (session) analysis Paths to buy, paths to abandonment, what else they looked at Improve customer experience and conversion Associate clicks with customers & prospects Social network influencer analysis Graph analytics for influencer behavioural impact analysis Target the influencer marketing campaign effectiveness 15 Today Both Structured And Multi-Structured Data Are Needed For Deeper Insight Often un-modelled and may not be well understood Often a schema is defined and data is well understood Multistructured data Structured data Click stream web log data Customer interaction data Social interaction data Sensor data Rich media data (video, audio) External content Documents Internal web content Seismic data (oil & gas) OLTP system data Data warehouse data Personal data stores, e.g. Excel, Access 16 8

9 Using Big Data And Hadoop To Enrich Customer Knowledge In MDM And DW Systems MDM DW & marts R contains clean, high value data C Prod Cust U Asset EDW D mart New high value Insights (pub/sub) other data sandbox Data Refinery sandbox sandbox Transform & Cleanse Data in Hadoop (Spark or MapReduce) Parse & Prepare Data in Hadoop (Spark or MapReduce) Discover data in Hadoop ELT work -flow Data Reservoir (raw data) Load data into Hadoop 17 The Problem With Enriching Customer Master Data There could be potentially hundreds of possible data sources to choose from Which ones would add the highest value? How do you assess the candidate data sources? Who decides which ones to choose? How long does it currently take you to currently get business agreement on adding new attributes to a customer MDM system? How long does the data from a candidate data source retain its business value? Can you use big data technology to help you decide which data sources are important? Yes!! Load into HDFS and run search over it to quickly explore it Ingest into HBase and run queries to quickly see what s of value 18 9

10 Data Deluge - Search Offers A Way To Quickly Explore Multi-Structured Data Sources To Assess Their Value Search and BI Lucene search engine technology is part of the Hadoop software stack Free-form ad hoc analysis of multistructured data Sales by Region search indexes content BI Systems DWs & Data Marts ECMS WWW 19 Search On Hadoop - Data Scientists Can Quickly Explore Newly Loaded Multi-Structured Data BI Tools, Applications, Mashups index index Index partition e.g. Social Data Platforms HDFS files 20 10

11 Big Data Analysis - Exploratory Analysis of Multi-Structured Data In Hadoop Via Search, e.g. Lucene Or IBM BigIndex File servers Use massively parallel Map Reduce to build a partitioned search index Web sites BI Tools, Applications, Mashups CMS LOAD Image server index index Index partition index partitions Collab tools Web feeds Useful for analysing un-modelled semi-structured content that is not well understood 21 Hadoop Search Based Analytics - Product Example Splunk Hunk (Splunk on Hadoop) 22 11

12 Assessing New Sources To Enrich Customer Data Is A Collaborative Process You Need Business In The Loop We need all relevant people to help determine high value data sources Data Scientist Data Steward Business data expert IT Data Architect Goal: Enrich CUSTOMER data for better marketing Business analyst sandbox sandbox Business data expert IT Data Architect Business data expert IT Developer We need to capture discussions, share exploratory results, rate data, prioritise projects 23 Once You Have Assessed The Value You Can Start Data Science Project(s) To Acquire New Data For example additional data about customers could come from: Social media data Professional life Lifestyle Relationships Likes/dislikes Sentiment - positive or negative opinion Intent - wants to buy, travel, etc. Ownership - products owned (could be from competitors) Interests - Could be short-lived In-bound customer Sentiment Call centre notes 24 12

13 Going Beyond Basic Identity Master Data E.g. Extending / Enriching Customer MDM Customer interaction data Customer attitude data Chat / transcripts Call centre notes Click stream Person-to-person dialogue Opinions Preferences Needs and desires R C enriched customer customer U D Customer behaviour data Orders Payments Transaction history Usage history Customer descriptive data MDM system Attributes with master data Characteristics services Relationships Demographics The objective is to create the best Customer dimension possible using additional internal and external data sources 25 Source: MDM Enriching Customer Data Which Data Sources Potentially Require Big Data Analytics To Derive Insight? Customer interaction data Customer attitude data Chat / transcripts Call centre notes Click stream Person-to-person dialogue Opinions Preferences Needs and desires CRM, web logs CRM, social media data, review web sites R C enriched customer U Potential big data sources D Customer behaviour data Orders Payments Transaction history sensor data, Usage history web logs Customer descriptive data MDM system Attributes with master data Characteristics services social media Relationships data, SEC Demographics filings The objective is to create the best Customer dimension possible using additional internal and external data sources Source: MDM 26 13

14 Enriching Customer Data Need To Consider Volume, Variety And Velocity Of Valuable New Data Sources Customer interaction data unstructured Chat / transcripts Call centre notes Click stream Person-to-person dialogue CRM, web logs Customer attitude data data unstructured data CRM, social media data, review web sites Opinions Preferences Needs and desires semistructured data R High volume undiscovered structured data C enriched customer Potential big data sources U D Customer behaviour data High velocity, high volume semistructured data Orders Payments Transaction history sensor data, Usage history web logs Customer descriptive data MDM system semiattributes with master data structured data Characteristics services Relationships social media Demographics data The objective is to create the best Customer dimension possible using additional internal and external data sources 27 Source: MDM Enriching Customer Data Different Platforms Optimised For Different Analytical Workloads Are Needed Big Data workloads result in multiple platforms now being needed for analytical processing Real-time stream processing & decision m gmt Streaming analytics Graph analysis Investigative analysis, Data refinery Advanced Analytic (multi-structured data) NoSQL DB NoSQL DBMS DW & marts EDW e.g. graph DB Streaming data Data mining, model development Hadoop data store mart Data Warehouse RDBMS Advanced Analytics (structured data) DW Appliance Analytical RDBMS Traditional query, reporting & analysis 28 14

15 Key Point! Several Different Types Of Big Data Analytic Workloads Can Be Used To Enrich Customer Data Text analytics to get new structured data attributes from millions of documents e.g. SEC filings, tweets, reviews Sentiment analytics for customer opinion Graph analytics for discovery of new customer relationships Clickstream analytics for customer interaction behaviour You can also combine these to find new data E.g. Text analytics to extract new data feeding graph analytics to find relationships in extracted data 29 New Data Sources - What Are We looking To Extract From Social Media Data Sources? Additional Organisation data Additional Person data e.g. hobbies, Interests, desires Social Data Platforms Professional data e.g. employers Product ownership data Intent Sentiment Unknown Relationships Requires several techniques: 1. JSON schema extraction 2. Text analytics for entity extraction 3. Clickstream analysis 4. Graph analytics for relationship discovery analysis R enrich C customer U D MDM System DW & marts HDFS files EDW mart 30 15

16 Social Media Data Challenges A Person Could Have Multiple Social Personas 31 Enriching Customer MDM - Extracting LinkedIn Social Profile Data Via Their REST API Most social media sites have APIs to access informaton Additional Person data e.g. education, interests Professional data e.g. employers, skills LinkedIn returns data in JSON or XML formats Source: LinkedIn 32 16

17 Enriching Customer MDM - Text Analysis Can Help Extract Structure From Unstructured Data Case management Fault management and field service optimisation Voice of the customer Sentiment analytics Competitor analysis Media coverage analysis How much is TEXT worth to your business? Improve pharma drug trials Unstructured content is hard to analyse 33 Using Text Analytics To Extract Additional Data From Unstructured Content Requirement is automatic recognition of people, organisations, addresses This can be a computationally intensive process involving complex character-level operations such as pattern matching On large volumes, scalability matters 34 17

18 The Text Analytics Process Key Tasks Extract raw text (html, pdf, ps, gif) Tag parts of speech nouns & verbs Tokenize Detect term boundaries Tag named entities Detect sentence boundaries Parse Person, place, organization, gene, chemical Determine co-reference Extract knowledge 35 Text Analytics Applications - What Is Sentiment Analysis? Definition The process of determining a sentiment score from text Additional Person data e.g. hobbies, Interests, desires Why do it? Responding to negative sentiment quickly is important to improving Intent Sentiment customer satisfaction and loyalty and protecting brand Data sources Contact centre customer interactions, e.g. , SMS.. Twitter, Facebook, review web sites Basic sentiment analysis Classifies the polarity of a document, sentence or other text Positive, Negative, Neutral Advanced sentiment analysis Beyond polarity" sentiment classification looks, at emotional states such as "angry, "sad, "happy Source: Mining Text to Pinpoint Customer Reactions to Products M Turner 36 18

19 Sentiment Analysis Text Analytics Entity Extraction Is Needed To Derive Structure From Unstructured Content Note: Not everyone is on Twitter!! Some people have > 1 Twitter account Challenges Emoticons ( :-) :-< :0) ) Twitter hashtags #bigdata Yoda speak Slang / vernacular / abbreviations Sarcasm Ambiguity Spam Multiple languages 37 (source: Crunchbase) Sentiment Analytics The Process Of Associating Terms With Sentiment Ratings Source: Mining Text to Pinpoint Customer Reactions to Products M Turner Drill down For customers who rated product x low, how many of them mentioned smell 38 19

20 Sentiment Analysis Visualisation Example Sentiment Histograms Source: Pardee Center Research Report: Connecting the Dots: Information Visualization and Text Analysis of the Searchlight Project Newsletter, Feb The Social Profile And Sentiment Analytics Can Be Matched To Master Data In Hadoop Using Fuzzy Matching Social Data Platforms Text Analysis Customer Engagement Management Social Media Aggregators Hive tables MapReduce or Spark sentiment scoring application Scored sentiment and Social profile data HDFS files Twitter Firehose MySpace Klout Amazon Facebook reddit Flickr Youtube bit.ly CRM applications Probabilistic ( fuzzy ) matching critical fields enrich Analyse / Index / Deliver R C enriched U customer D R MDM System Ccustomer U D MDM System 40 20

21 Sentiment Analysis Could Be Done On The Cloud While Matching Could Be Done In-House Social Data Platforms Text Analysis Customer Engagement Management Social Media Aggregators On-the-cloud Hive tables MapReduce or Spark sentiment scoring application Scored sentiment and Social profile data HDFS files Probabilistic ( fuzzy ) matching enrich critical fields R C enriched U customer D R MDM System On-premises Twitter Firehose MySpace Klout Amazon Facebook reddit Flickr Youtube bit.ly CRM applications Analyse / Index / Deliver Ccustomer U D MDM System 41 Running A Master Data Matching Engine On Hadoop As A MapReduce Job Matching People With Social Interactions Product Example: IBM InfoSphere MDM BigMatch PME Source: IBM 42 21

22 Where Are We? - Enriching Customer Master Data With New Relationships Using Graph Analysis Customer interaction data Customer attitude data Chat / transcripts Call centre notes Clickstream Person-to-person dialogue Opinions Preferences Needs and desires R C Enriched U customer customer D Customer behaviour data Customer descriptive data MDM system Attributes with master data Characteristics services Relationships Demographics Orders Payments Transaction history Usage history Click stream navigation The objective is to create the best Customer dimension possible using additional internal and external data sources Source: MDM 43 Graph Analytics Use Cases Financial crimes Anti-money laundering, fraud Government benefits fraud Insurance fraud Crime prevention and counter terrorism Social network influencer analysis Route optimisation Airlines, supply/distribution chain, logistics Life sciences (Bioinformatics) Medical research, Disease pathologies MDM - Identify new relationships 44 22

23 Graph Analytics Example - Social Network Relationships Analysis As graphs get more complex you don t know the relationships and the less likely you would be in successfully partitioning the data Image source: Mashable.com 45 Graph Analysis Verticies And Edges - What Can Be Vertices? Edge (can have direction) Vertex (can have properties) Source: Teradata Vertex 46 23

24 Graph Analysis Edges Are Often More Valuable Than Vertices Source: Teradata 47 There Are A Range Of Graph Analytics Algorithms - E.g. Teradata Aster Prepackaged Graph Algorithms Source: Teradata 48 24

25 Graph Analysis Algorithm Example - Eigen Centrality Could Highlight Important Influencers 49 Exploratory Graph Analysis - Social Graph Analysis And Visualisation 50 25

26 Using Text And Graph Analytics To Enrich Customer Data - Entity Flow From SEC Filings Millions of documents Filing timeline 2005 Event subsidiaries, insider, 5%, 10% owner, banking subsidiaries 2013 employment, director, officer insider, 5% owner, 10% owner SEC/FDIC Filings of Financial Companies Company Extract Integrate Person borrower, lender Loan Security Entity-centric view Source: IBM 51 Using Text AND Graph Big Data Analytics To Enrich Customer Data - Detailed Entity Flow Overview Product Example: IBM Big Match and BigInsights Nutch segments U.S. Crawl S.E.C Securities and Exchange Commission Nutch crawl for SEC. Source: IBM Entity Integration Per document, incremental Parse and Extract using AQL Load Over all documents (non-incremental) Part 1 Part 2 RESTful API Manual download for FDIC filings. Text Analytics PostCrawl JSON (Nested Entities) JSON JSON Query Layer Single machine Hadoop Graph Store 52 26

27 Information Extracted From SEC filings The information from the following SEC documents can be extracted and consolidated into entities No extractor run. Convert from XML to JSON. We get people and companies from here and the transactions between them. Forms 3/4/5 XML to Json Forms SC Extract 5% or more Beneficial Ownership reports Forms 13F Extract Institutional Investment Manager Reports. Holdings. Forms 8 / 10 / DEF Extract Core Financial Information: Biographies, Loan Agreements, Merger & Acquisitions, Appointments & Resignations, Committees, Board Positions, etc. Source: IBM 53 Information Extracted From SEC Filings Current Events merger and acquisition bankruptcy change of officers and directors material definitive agreements Subsidiaries list subsidiaries of a company Forms 8-K Forms 3/4/5, SC 13D, SC 13G, 10-K, FDIC Call Report subsidiaries, insider, 5%, 10% owner, banking subsidiaries Event Forms 3/4/5, SC 13D, SC 13G employment, director, officer Shareholders related institutional managers Holdings in different securities Forms 10-K, DEF 14A, 8-K, 3/4/5, 13F, SC 13D, SC 13G, FDIC Call Report Officers & Directors mention bio range, age, current position, past position signed by committee membership insider, 5% owner, 10% owner Company Person borrower, lender Forms 10-K, 10-Q, 8-K Forms 10-K, DEF 14A, 8-K, 3/4/5 Loan Security Reference SEC table Loan Agreements loan summary details counterparties (borrower, lender, other agents) commitments Source: IBM Insider filings transactions holdings Insider relationship Forms 13F, Forms 3/4/5 5% beneficial ownership owner issuer % owned date 54 27

28 Enriching Customer Master Data Do You Attach Insights To The Master Data Entity, Relationships Or Both enrich enrich new relationship Image Source: by Andy Hogg 55 Where Are We? - Enriching Customer Master Data With Clickstream Interaction Behaviour Insight Customer interaction data Chat / transcripts Call centre notes Clickstream Person-to-person dialogue Customer attitude data What do logged in Opinions customers Preferences do and look Needs and desires at on-line? R C Enriched U customer customer D Customer behaviour data Customer descriptive data MDM system Attributes with master data Characteristics services Relationships Demographics Orders Payments Transaction history Usage history Click stream navigation The objective is to create the best Customer dimension possible using additional internal and external data sources Source: MDM 56 28

29 A Common Way To Capture Weblog Data To Bring Into Hadoop HDFS Is Using Apache Flume Flume Master, is a separate service with knowledge of all the physical and logical nodes in a Flume installation Flume Sinks include HDFS sink - supports writing Avro files with arbitrary schemas Solr sink with ETL capabilities. HBase Source: Cloudera 57 Exploratory Analysis Of Clickstream Data In Hadoop E.g. Weblog Data In Hortonworks 58 29

30 Putting Structure On Clickstream Data - Creating A Hive View Over The Weblog Data 59 ClickStream Data With A Hive Schema Allows The Data To Be Queried & Joined With Other Data, e.g. CRM And Product Data 60 30

31 Teradata Aster Discovery Portfolio Clickstream Visualisation Examples Source: Teradata 61 Analysing Enriched Customer Data Can Improve Accuracy Of Next Best Action To Be Taken Option 1 Life events Additional Organisation data Additional Person data e.g. hobbies, Interests, desires Professional data e.g. employers Behaviour Product ownership data Intent score Sentiment score Unknown Relationships enrich R C Enriched U customer analyse D Next best action Enriched MDM System Option 2 Life events Additional Organisation data Additional Person data e.g. hobbies, Interests, desires Professional data e.g. employers Behaviour Product ownership data Intent score Sentiment score Unknown Relationships DW & marts enrich analyse EDW mart Next best action 62 31

32 Distributed Execution Of Analytics In A Data Refinery Process E.g. RapidMiner 63 Use Analytics On Enriched Master Data To Top Up Todays Calls Into Salesforce.com, e.g. RapidMiner 64 32

33 Achieving Consistent Customer Treatment Via On-Demand Access To Common Smart Analytical Master Data Services Customers Improve customer engagement Front-Office Operations E-commerce application Customer service app Sales Force automation app Customer facing outlet applications Marketing application OMNI-CHANNEL Enterprise Service Bus Analytical services R C Enriched customer U D Smart master data & master data services Improve marketing, sales and service via ondemand access to smart master data with insight on each and every each customer available through all channels 65 Conclusions B2C and B2B customers are becoming very powerful because they are getting informed before they buy This means loyalty is becoming cheap and so organisations have to try harder to keep customers To understand customers better, companies need more data Big data analytics can be very effective in providing new insights to enrich customer data in DW and MDM systems Integrating analytical and decision services that analyse enriched customer data into OLTP applications can deliver significant competitive advantage 66 33

34 Thank You! Tel/Fax (+44)