UDA EXPERIENCES. Cesar Rojas Director of Product Marketing Data Science & Hadoop

Transcription

1 UDA EXPERIENCES Cesar Rojas Director of Product Marketing Data Science & Hadoop Spring Teradata User Group Meetings: Los Angeles

2 Did you know? LA = "El Pueblo de Nuestra Señora la Reina de los Ángeles de Porciúncula" 2 Spring 2014 Teradata User Groups Talk

3 Agenda Section 1 What is changing? > New engineering and business possibilities Section 2 Teradata s Point of View and Big Data Strategy > Why UDA? Section 3 Who s Adopted? Why? > Case Studies Section 4 Getting Started > Tips and Typical Obstacles Section 5 More Info / Resources > Q&A 3 Spring 2014 Teradata User Groups Talk

4 SECTION 1: WHAT IS CHANGING? 6 KEY ASPECTS OF BIG DATA

5 Yet Another Slide with the V s (YASV) The V s and Info Density 1 5 Spring 2014 Teradata User Groups Talk

6 Typical Big Data Use Case Experiences New data types and very fast new analytics techniques 2 Retail Omni channel consumer analysis Time to Insight (post data load & prep*) Hours Days 2 or 4 Weeks Behavioral based affinity analysis Multi-touch marketing Attribution Event-based customer attrition Omni channel path to purchase Personalized recommendation Enhanced behavioral segmentation Healthcare, Insurance Multi-Event path to surgery Reduction in claim review (Text analytics) Predictive Medicare complaint reduction Path to policy purchase Driving behavior telematics analysis Finance. Communications Cross channel fraud analysis Client behavioral call center analysis Multi event-based churn prediction Broker surveillance analysis Manufacturing, CPG, Media Aircraft multi event sensor path analysis Set-top box viewer behavioral analysis Optimized Movie recommendations Multi-channel customer SAT index Golden path web registration analysis * Directional estimates based on past proof-of-values 6 Spring 2014 Teradata User Groups Talk

7 New Content and Data Structures 3 Relational - Columnar Relational Row Stores And Hybrid Semi- Structured Unstructured Pre-Defined Early Binding Undefined Late Binding When do you figure out what the structure of the data is? 7 Spring 2014 Teradata User Groups Talk

8 New Approaches: Early and Late Binding All Data has structure, and analytics requires teasing out the structure. The key decision is when to best apply the structure based on intended use of the data 4 Early Binding : Schema Needed on Writing Into the Database Source System RDBMS Effort: Data Structuring Business Users Late Binding : Schema not needed until Reading Source System File Systems Effort: Data Structuring Developers and Data Scientists 8 Spring 2014 Teradata User Groups Talk

9 Faster: Visualization of Big Data Discover Insights More Quickly 5 Chord Flow Hierarchy Affinity and Graphs 9 Spring 2014 Teradata User Groups Talk

10 You Now Have Choices Pick 6 IDW Methodology when Big Data Methodology when there are increases in number of: decreases in Analyses, Integrated Data Sources & Reuse of Data, plus increases in: Analyses (Concurrency, Throughput, SLAs, Ease of Use) Integrated Data Sources (Access Complexity and High IO) Reuse of Data (schema-on-write, business rule changes, governance) Data Variety (no schema, sparse data) High Intensity, Batch Computation (High CPU) Logic Complexity (Procedural Language Processing) And with needs for.. Fine Grain Security Data Quality and Integrity High Availability Fast Response Times, especially for Operational Intelligence (Active) Integrated Data Warehousing is cost advantaged when the above is true Development Costs Maintenance Costs Usage Costs And with needs for.. Extreme Data Ingest Rates Fast exploration of new data types like graphs, social networks Custom Development Big Data Methodologies are cost advantaged when the above is true Acquisition Costs Development Costs Usage Costs NOT either-or: pick the right tools for the job 10 Spring 2014 Teradata User Groups Talk

11 Putting It All Together: Explore Big Data Separate Discovery from Production PRODUCTION SYSTEMS Dashboards Contact Center FACTS IN AN INTEGRATED DATA WAREHOUSE Promote New Insights EXPLORATION DISCOVERY SYSTEMS Web Store Access Known Facts BI Analysts Analysis Of known facts Analysis Of unknown or discovered facts 11 Spring 2014 Teradata User Groups Talk

12 SECTION 2: TERADATA S POINT OF VIEW AND 5-STEP STRATEGY

13 Realize Big Data is an Evolution not a Revolution Flow DATA -> INSIGHTS -> ACTIONS Data s data. I don t like the term Big Data because it means nothing. Predictions Events Patterns Hypothesis Testing Strategic Actions Operational Actions Flow BIG DATA -> INSIGHTS -> ACTIONS Is the Ultimate USE of Big Data Different? No. 13 Spring 2014 Teradata User Groups Talk

14 Step 1: Extend Teradata With New Data Types Teradata Data Warehouse XML <xml /> XML :25: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; enus) AppleWebKit/ (KHTML, like Gecko) Version/5.0.3 Safari/ " elcome/" " client=psyab&hl=en&source =hp&q=oklahoma+state&pb x=1&oq weblogs JSON 14 Spring 2014 Teradata User Groups Talk

15 Step 2: Buy Aster, a SQL M/R Discovery Tool Extensive Software Library, Connectors, and Appliance Platform Teradata Aster Discovery Platform Innovations Teradata acquires Aster Aster Big Analytics Appliance Aster Lens H H H Aster 5, SQL-H Aster 5.10, Aster Lens 15 Spring 2014 Teradata User Groups Talk

16 Step 3: Visualization, Web Checkout Sankey Analysis Good for analyzing where people drop out on the web Customers who have reached the checkout process follow an ideal path. deliveryslots > deliveryinformation > coupons > substitutions > paymentinfo > orderconfirmation Determine how and when (and ultimately, why) customers deviate from this path. Discover obstacles preventing purchase and optimize visitor flow through the web site.. 16 Spring 2014 Teradata User Groups Talk

17 Use Visualization To See Insights Faster 17 Spring 2014 Teradata User Groups Talk

18 Visualization: License Plate Recognition Open-source software grabbed, used by Aster COE in 1 day! Applications in government and fleet management SELECT * FROM RecognizeNumberPlate( ON anpr.vehiclelogs imagecol('recognizedobject') ); Each country has a different number plate format, configured in a syntax file. We have it working for UK, Germany 18 Spring 2014 Teradata User Groups Talk

19 Capture, Store, Refine Step 4: Provide a Dual Platform Hadoop Strategy (Hadoop or Teradata 1700) Hadoop Data Platform: Raw data capture History or long term storage Low cost archival Transformations DATA PLATFORM Data Mining Structured, semi-structured Sessionize, remove XML tags, extract key words Machine Learning Programmer Simple math at scale Data Scientists Batch processing Languages 1700 Data Platform: All of the above PLUS database capabilities Price comparable to Hadoop ANALYTIC TOOLS USERS 19 Spring 2014 Teradata User Groups Talk

20 Step 5: Build a Unified Data Architecture Hybrid approach Endorsed by Gartner Group Logical Data Warehouse Extensible > Multiple engines > Multiple connectors Optimizable over time > Data placement > Computation placement Entering 2014, the hype around replacing the data warehouse gives way to the more sensible strategy of augmenting it. The influence of the logical data warehouse has created a situation in which multiple repository strategies are now expected, even from a single vendor. -Gartner Data Warehouse MQ (3/14) 20 Spring 2014 Teradata User Groups Talk

21 TERADATA UNIFIED DATA ARCHITECTURE System Conceptual View ERP MOVE MANAGE ACCESS Marketing Marketing Executives SCM CRM INTEGRATED DATA WAREHOUSE Applications Operational Systems Images DATA PLATFORM Business Intelligence Customers Partners Audio and Video TERADATA DATABASE Data Mining Frontline Workers Machine Logs Text TERADATA DATABASE HORTONWORKS HADOOP INTEGRATED DISCOVERY PLATFORM Math and Stats Business Analysts Data Scientists Languages Web and Social Engineers TERADATA ASTER DATABASE SOURCES ANALYTIC TOOLS & APPS USERS

22 SECTION 3: WHO HAS ADOPTED UDA? WHY? WHAT ARE THEY DOING? WHAT VALUE ARE THEY SEEING?

23 A Sampler of Customers 24 Spring 2014 Teradata User Groups Talk

24 Big Data Best of Partners Dallas 2013 Big Data Types... Tweets during Hurricanes Doctor Notes about Patients Remote Control Clicks on the TV Web Pathways (like DNA) Web Clicks, then Call Center Interactive Voice Response (IVR) Button pushes Voice Recordings of Customers Calling Customer Service Customer Care Agent Notes/Text Smart Utility Meters Market Baskets of Professionals Doing Home Projects 25 Spring 2014 Teradata User Groups Talk

25 FAREASTONE (TAIWAN)

26 UDA at FarEasTone (FET) Big Data Journey Impact Situation Deeper customer insights from social, better customer models Better operations models, better security models Faster analytical processes Excellent experience gained in understanding architecture choices 27 Spring 2014 Teradata User Groups Talk Taiwan telco, 7M subscribers, $3B UDS revenues. In 60 seconds, collect a huge amount of data, various types Problem How to integrate and monetize the data. How to provide value-add for the LOBs. Explore right architecture to handle the massive volumes of data and new requirements. Solution Used Hadoop, Aster, and Teradata with UDA to capture, analyze, integrate, and take action on social data, network data, security breach data. Learned where to place data and computation optimally.

27 Data What FET Collects in 60 sec 28 Spring 2014 Teradata User Groups Talk

28 29 Spring 2014 Teradata User Groups Talk

29 More Data Sources Consumers 30 Spring 2014 Teradata User Groups Talk

30 More Data Sources Mobile Network 31 Spring 2014 Teradata User Groups Talk

31 Business Drivers for Big Data 32 Spring 2014 Teradata User Groups Talk

32 New Architecture, Processing Roles 33 Spring 2014 Teradata User Groups Talk

33 34 Spring 2014 Teradata User Groups Talk

34 Old Vs. New Applications 35 Spring 2014 Teradata User Groups Talk

35 36 Spring 2014 Teradata User Groups Talk

36 We are in LA What about media? 37 Spring 2014 Teradata User Groups Talk

37 COMCAST

38 UDA at Comcast Increase TV Viewer Insight Situation Need to understand customer behavior across channels, get a 360 degree view, glean insights to better set ad rates Problem Silos of information, no ability to collect and analyze the 3.5 TB / day of clickstream data on viewer habits. Solution Used Hadoop and Aster to capture/analyze the data. Use visualizations to more deeply understand customer patterns. Impact Deeper customer insights Better predictive models Faster analytical processes Better insights about viewers will lead to better ratesetting for ads 39 Spring 2014 Teradata User Groups Talk

39 The Problem Art project from school? OR Goal is to tease apart to see groups of customers with similar viewing patterns 40 Spring 2014 Teradata User Groups Talk

40 Integrate the Data in Teradata Aster 41 Spring 2014 Teradata User Groups Talk

41 Insights: Clusters of People Watching Channels Aggregate Customer Affinity to Channels 42 Spring 2014 Teradata User Groups Talk

42 Insights: Drill Down Differences between HD and Standard Channels? 43 Spring 2014 Teradata User Groups Talk

43 Insight Drilldown: Differences between Segments and Channels? 44 Spring 2014 Teradata User Groups Talk

44 Popular Show Drilldowns How do people get to Game of Thrones for example? And where do they go after? 45 Spring 2014 Teradata User Groups Talk

45 46 Spring 2014 Teradata User Groups Talk

46 47 Spring 2014 Teradata User Groups Talk

47 FOUR TYPICAL STEPS THAT ALL BIG DATA CASES HAVE IN COMMON

48 Step 1: Acquire Multi-Channel View : Data All types of interaction data from all channels Start with 2-3 channels and grow over time DATA PLATFORM DISCOVERY PLATFORM Discovery Platform Online,Social Store Call center ATM Branch Survey Text Category (Intent) Page Type, Actions, Tweets Branch ID, Mgr ID, Event Event ATM ID, Event, $$ Teller ID, Event Teller ID, Question ID, Response ID 49 Spring 2014 Teradata User Groups Talk

49 Step 2: Recreate Journeys Recreate the Customer Behavior Through Identify the target customer or groups of customers Identify the sessions in time Stitch together sessions to recreate cross-channel journey 07:05:32 09:20:23 09:25:32 11:05:48 1:05:06 1:35:12 1:42:58 1:45:14 3:05:58 4:15:22 50 Spring 2014 Teradata User Groups Talk

50 Step 3: Find Patterns, Events : Insights Deliver valuable insight to lines of business resulting from deep analysis of all of your data, all of the time Highly visual, exploratory Customer Paths To Attrition Golden Path to Application Submit Fraud Paths 51 Spring 2014 Teradata User Groups Talk

51 Step 4: Operationalize Insights: Action! Attrition Propensity Model Exploit Active Data Warehousing HADOOP OR TERADATA CAPTURE STORE REFINE Teradata Aster Discovery Platform Attrition Path Flag Attrition Score Path Code # Days to Cancel Sentiment Index Teradata Data Warehouse Campaign Management ( Teradata Applications) Real Time Interaction Manager Customer Path Identified Attrition Risk Scored Retention Campaign for At Risk High Value Customers 52 Spring 2014 Teradata User Groups Talk

52 Step 5: Use Big Data for Personalization Unified Data Architecture for Digital Marketing Data Customer Interaction Manager Channels Product / SKU Inventory Teradata Customer Data Warehouse Segmenting Campaign Automation Leads Digital Marketing Center In- Store Call Center Consumer Group Sales / POS Teradata Aster Reporting Trending Rules , SMS Responses Real-Time Interaction Manager Mobile Celebrus Web Data Hadoop Content & Offers Rules Real-Time Decisioning Web Responses Feedback 53 Spring 2014 Teradata User Groups Talk

53 SECTION 4 TIPS DOING BIG DATA PROJECTS HOW TO GET STARTED

54 How To Get Started Your Teradata Data Warehouse = the foundation > You are much of the way there if you already have Teradata Good News! Start small & simple pick a pilot Big Data (UDA) project > Clearly define business value > Map out the business enhancement, process, & users > Find an executive sponsor > Do a POC try out Aster and Hadoop > Communicate your success 55 New Hire Feb Teradata Confidential

55 Top 5 Problems To Avoid Why Can t Companies Seize the Data-Driven Opportunities? 1. No Visionary / Leader 2. Simple Inertia, Legacy, No Rock the Boat unwilling to explore new technologies 3. No Strong Business Alignment IT groups off doing something in the basement without a business justification 4. No Culture of managing by the numbers, accountability, ROI proof points, communicating the value 5. Lack of Talent Data Scientists 56 New Hire Feb Teradata Confidential

56 Do More With Your Data Leverage your Teradata Investment Strategic Intelligence Better, Faster Decisions? Widespread Intelligence CxO Sure! Operations Front lines EDW Touch points ATM 57 New Hire Feb Teradata Confidential

57 Be Creative What Can YOUR COMPANY Do? Understand how customers consume data services- what apps do they download, how & where they use them? Know where your customers are, how/why they are spending, and how you can serve them better Generate new revenue sensors and The internet of Things Know which processes in your company and with your partner eco-system are working or not, and fix them Understand which of your customers are unhappy with your service and who are they publically sharing their opinions with Have clarity into which channels are responsible for attribution and optimize marketing spend correlation 58 New Hire Feb Teradata Confidential

58 WANT MORE INFO? HOW TO FIND MORE STORIES, WHITE PAPERS, WEBINARS

59 Teradata.com Content Unified Data Architecture (also web pages for Aster and Hadoop) 60 New Hire Feb Teradata Confidential

60 More Info UDA in Action White Paper 16 page report Aimed at business people, project managers, technical people Provides comprehensive overview of the Teradata point of view on Big Data Key themes: > evolution not revolution, incremental architecture investments, > survey data on projects, > details on the technical components, > two customer examples 61 New Hire Feb Teradata Confidential

61 More Info Available NOW: CITO Research Report 13 page report at business people, project managers, people considering big data projects Strategic thinking piece Co-authored by Scott Gnau 62 New Hire Feb Teradata Confidential

62 More on Aster? More on Hadoop? 63 New Hire Feb Teradata Confidential

63 Tech info: UDA and Hadoop Content E-Book on UDA > View Where Hadoop Fits in Your Data Warehouse Architecture > View Busting Common Hadoop Myths > View TCO of Big Data Richard Winter 64 New Hire Feb Teradata Confidential

64 QUESTIONS?

65 BACKUP SLIDES

66 DISCOVER FINANCIAL

67 UDA at Discover Financial Services Big Data Proof of Concept for Customer Care Situation Explore: can Big Data improve customer care? Problem Need to see paths of how customers interact with Discover s web site and customer care system to understand where improvements can be made. Solution Used Aster Discovery Platform to find new areas for improvement, drive re-imagining of apps Impact Developed predictive model for call reasons Data and discovery processes now support big data Can see breakage between web and contact center channels Proactive/personalized customer calls 68 New Hire Feb Teradata Confidential

68 69 New Hire Feb Teradata Confidential

69 Call Center Interactive Voice Response Decision Tree Analysis on Button Push Sequences 70 New Hire Feb Teradata Confidential

70 Merging and Matching on Aster Step 1: Calls + Memos Step 2: Web + Calls/Memos 71 New Hire Feb Teradata Confidential

71 Find 1 or more Pages followed by a Call Solution Design: Time-series sequence of page-hits followed by call events are then matched using Aster s npath SQL- MR function. SELECT * FROM npath( ON page_call rec_typt = PAGE AS p, rec_type = CALL AS c Pattern( p+.c )); Page-Hit Call / Memo Page-Hit Page-Hit Page-Hit Call / Memo npath: Page+ Call On-Line Page Sequences Leading To a Phone Call 72 New Hire Feb Teradata Confidential

72 Sessionize Across Web and Care Center 73 New Hire Feb Teradata Confidential

73 RMS (Recognition Mining Synthesis) - Predictive Model 74 New Hire Feb Teradata Confidential