Welcome. Host: Eric Kavanagh. The Briefing Room. Twitter Tag: #briefr

Transcription

1 The Briefing Room

2 Welcome Host: Eric Kavanagh Twitter Tag: #briefr The Briefing Room

3 Mission! Reveal the essential characteristics of enterprise software, good and bad! Provide a forum for detailed analysis of today s innovative technologies! Give vendors a chance to explain their product to savvy analysts! Allow audience members to pose serious questions... and get answers! Twitter Tag: #briefr The Briefing Room

4 JANUARY: Big Data February: Analytics March: Open Source April: Intelligence Twitter Tag: #briefr The Briefing Room

5 Big Data Copyrighted property. May not be copied or downloaded without permission from 123RF Limited. NEW SOURCES New Insights NEW Challenges Twitter Tag: #briefr The Briefing Room

6 Analyst: Robin Bloor Robin Bloor is Chief Analyst at The Bloor Group Twitter Tag: #briefr The Briefing Room

7 Teradata Aster! Teradata is known for its data analytics solutions with a focus on integrated data warehousing, big data analytics and business applications! It offers a broad suite of technology platforms and solutions; data management applications; and data mining capabilities! Teradata Aster is its MapReduce platform to handle big data analytics on multi-structured data Twitter Tag: #briefr The Briefing Room

8 Steve Wooledge Steve is Senior Director of Product Marketing for Teradata Aster and has 10 years of industry experience. Twitter Tag: #briefr The Briefing Room

9 Bringing Big Data into the Light: Teradata Big Analytics Appliance Steve Wooledge Sr. Director, Product Marketing, Teradata Aster January 2013

10 TOPICS WHAT IS DIFFERENT ABOUT BIG DATA ANALYTICS? MAKING BIG ANALYTICS & DISCOVERY FAST AND EASY TERADATA ASTER BIG ANALYTICS APPLIANCE Confidential and proprietary. Copyright 2012 Teradata Corporation. Copyright 2012 Teradata Corporation. 10

11 Copyright 2012 Teradata Corporation. What is Different about Big Analytics and Discovery?

12 The Lytro and Big Data 12

13 Interactive, Living Pictures 13

14 See Your Business in High-Definition Big Analytics & Discovery Unlocks Hidden Value Classic BI Structured & Repeatable Analysis Business determines what questions to ask IT structures the data to answer those questions Capture only what s needed 14

15 See Your Business in High-Definition Big Analytics & Discovery Unlocks Hidden Value Classic BI Structured & Repeatable Analysis Business determines what questions to ask IT structures the data to answer those questions Capture only what s needed IT delivers a platform for storing, refining, and analyzing all data sources Capture in case it s needed Big Data Analytics Multi-structured & Iterative Analysis Business explores data for questions worth answering 15

16 Iterative Analytics Accelerates Discovery Analytical Idea Operational DB or EDW Operationalize or Move On Zero-ETL Data Load/Integration Evaluate Results SQL and non-sql Analysis 16

17 Iterative Analytics Accelerates Discovery Analytical Idea Operational DB or EDW Operationalize or Move On 5x Faster Discovery Process with Aster - Hours vs. Days Evaluate Results Zero-ETL Data Load/Integration SQL and non-sql Analysis 17

18 Need for a Unified Data Architecture for New Insights Enabling Any User for Any Data Type from Data Capture to Analysis Java, C/C++, Python, R, SAS, SQL, Excel, BI, Visualization Discover and Explore Reporting and Execution in the Enterprise Capture, Store and Refine Audio/ Video Images Docs Text Web & Social Machine Logs CRM SCM ERP 18

19 Big Data Comes with BIG HEADACHES Even free software like Hadoop is causing companies to spend more money Many CIOs believe data is inexpensive because storage has become inexpensive. But data is inherently messy it can be wrong, it can be duplicative, and it can be irrelevant which means it requires handling, which is where the real expenses come in. Through 2015, 85% of Fortune 500 organizations will be unable to exploit big data for competitive advantage. Source: The Wall Street Journal. CIOs Big Problem with Big Data. Aug 2012 Source: Gartner. Information Innovation: Innovation Key Initiative Overview. April

20 UNIFIED DATA ARCHITECTURE Data Scientists Quants Customers / Partners Front-Line Workers Engineers Business Analysts Executives Operational Systems LANGUAGES MATH & STATS DATA MINING BUSINESS INTELLIGENCE APPLICATIONS DISCOVERY PLATFORM INTEGRATED DATA WAREHOUSE CAPTURE STORE REFINE AUDIO & VIDEO IMAGES TEXT WEB & SOCIAL MACHINE LOGS CRM SCM ERP 20

21 UNIFIED DATA ARCHITECTURE Data Scientists Quants Customers / Partners Front-Line Workers Engineers Business Analysts Executives Operational Systems LANGUAGES MATH & STATS DATA MINING BUSINESS INTELLIGENCE APPLICATIONS Big Data Analytics DISCOVERY PLATFORM INTEGRATED DATA WAREHOUSE CAPTURE STORE REFINE Big Data Management AUDIO & VIDEO IMAGES TEXT WEB & SOCIAL MACHINE LOGS CRM SCM ERP 21

22 TERADATA UNIFIED DATA ARCHITECTURE Data Scientists Quants Customers / Partners Front-Line Workers Engineers Business Analysts Executives Operational Systems LANGUAGES MATH & STATS DATA MINING BUSINESS INTELLIGENCE APPLICATIONS DISCOVERY PLATFORM INTEGRATED DATA WAREHOUSE CAPTURE STORE REFINE AUDIO & VIDEO IMAGES TEXT WEB & SOCIAL MACHINE LOGS CRM SCM ERP 22

23 TERADATA UNIFIED DATA ARCHITECTURE Data Scientists Quants Customers / Partners Front-Line Workers Engineers Business Analysts Executives Operational Systems VIEWPOINT LANGUAGES MATH & STATS DATA MINING BUSINESS INTELLIGENCE APPLICATIONS SUPPORT DISCOVERY PLATFORM Aster Teradata Connector INTEGRATED DATA WAREHOUSE Aster Connector for Hadoop SQL-H SQL-H Teradata Connector for Hadoop Aster Loader CAPTURE STORE REFINE Teradata Loader 23 Copyright AUDIO & VIDEO 2013 Teradata IMAGES Corporation. TEXT WEB & SOCIAL MACHINE LOGS CRM SCM ERP

24 Shift from a Single Platform to an Ecosystem Big Data requirements are solved by a range of platforms including analytical databases, discovery platforms and NoSQL solutions beyond Hadoop. Source: Big Data Comes of Age. EMA and 9sight Consulting. Nov

25 How Does Big Analytics and Discovery Add Business Value? Copyright 2012 Teradata Corporation.

26 Customers Getting Deeper Insights 26

27 Customer Behavior Analysis BI Tools Database Tools Monitoring Tools CORRESPOND- ENCE DATA BRANCH TELLER DATA ONLINE BANKING DATA STORE VISION PLATFORM DATA CALL CENTER DATA CUSTOMER PROFILE DATA CUSTOMER SURVEY DATA 27

28 Events Preceding Account Closure 28

29 Events Preceding Account Closure 29

30 Events Preceding Account Closure SELECT * FROM npath ( ON ( SELECT WHERE u.event_description IN ( SELECT aper.event FROM attrition_paths_event_rank aper ORDER BY aper.count DESC LIMIT 10) ) PATTERN ('(OTHER EVENT){1,20}$') SYMBOLS ( ) RESULT ( ) ) ) n; Interactive Analytics Reducing the Noise to find the Signal 30

31 Events Preceding Account Closure 31

32 Events Preceding Account Closure SELECT * FROM npath ( ON ( ) PARTITION BY sba_id ORDER BY datestamp MODE (NONOVERLAPPING) PATTERN ('(OTHER_EVENT FEE_EVENT)+') SYMBOLS ( event LIKE '%REVERSE FEE%' AS FEE_EVENT, event NOT LIKE '%REVERSE FEE%' AS OTHER_EVENT) RESULT ( ) ) n; Closed Accounts Fee Reversal Seems to Be a Signal 32

33 Example: Online Checkout Flow Analysis (6 Stages) Customers who have reached the checkout process follow an ideal path. deliveryslots > deliveryinformation > coupons > substitutions > paymentinfo > orderconfirmation Determine how and when (and ultimately, why) customers deviate from this path. Discover obstacles preventing purchase and optimize visitor flow through the web site. The Aster SQL-MapReduce Framework enables a variety of different path visualizations. 33

34 Example: Online Checkout Flow Analysis (mid-stage) Customers who have reached the checkout process follow an ideal path. deliveryslots > deliveryinformation > coupons > substitutions > paymentinfo > orderconfirmation Determine how and when (and ultimately, why) customers deviate from this path. Discover obstacles preventing purchase and optimize visitor flow through the web site. The Aster SQL-MapReduce Framework enables a variety of different path visualizations. 34

35 How Do We Make Big Analytics & Discovery Possible? Copyright 2012 Teradata Corporation.

36 Key Requirements of a Discovery Platform Democratize Big Data & Maximize Enterprise Adoption 36

37 Key Requirements of a Discovery Platform 1 Highly Efficient & Performant Big Data Platform That Allows Quick Iterations Democratize Big Data & Maximize Enterprise Adoption 37

38 Key Requirements of a Discovery Platform 1 Highly Efficient & Performant Big Data Platform That Allows Quick Iterations 2 Hybrid Capabilities that Provide both Legacy (SQL, BI) and New (MapReduce) Interfaces Democratize Big Data & Maximize Enterprise Adoption 38

39 Key Requirements of a Discovery Platform 1 Highly Efficient & Performant Big Data Platform That Allows Quick Iterations 2 Hybrid Capabilities that Provide both Legacy (SQL, BI) and New (MapReduce) Interfaces 3 Significant Out-of-the-Box Analytical Apps that Minimize Development Democratize Big Data & Maximize Enterprise Adoption 39

40 Teradata Aster Big Analytics Appliance First Deeply Integrated SQL, MapReduce and Hadoop Appliance UNIQUE FEATURES 1. Integrated, modular Aster Database and 100% Open-Source Hortonworks HDP 2. First and only ANSI SQL & HCatalog integration via SQL-H 3. Industry s only ANSI-standard SQL & MapReduce integration via SQL-MapReduce 4. Industry s most manageable & supportable Apache Hadoop appliance via Teradata Viewpoint & TVI 5. Most complete MapReduce App Portfolio with 60+ pre-built MapReduce functions 6. Fully engineered and supported by Teradata, with Level-4 support by Hortonworks world-class Hadoop team Benefits Leverage existing investments in standard BI, ETL tools & people with SQL skills Industry s highest performance platform for Big Analytics Lowest TCO (technology + people), highest ROI, and fastest time to value 40

41 Unified Big Data Analytics Architecture Integrated Analytics and Navigation BI Tools, SQL, ETL BIG ANALYTICS APPLIANCE TERADATA IDW Unified Big Analytics Architecture Behavior Sentiments Multi- Structured Data Discovery Platform Unstructured Data Facebook Twitter Pinterest Revenue Social Media Iterative Information Discovery Operationalized Analytics Best Decision Possible 41

42 Teradata Aster Big Analytics Appliance Solution Value Add NEW SQL BI Tools Analytic SQL Apps Hadoop Tools Single vendor for lowest TCO Common system management tools Common Management, Troubleshooting, and Support NEW Aster MapReduce Portfolio of Functions SQL SQL- MapReduce SQL-H Aster Database Hive, Pig, Supports standard BI and ETL tools Use Hadoop tools like Hive and Pig Analytics Library w/ 50+ functions SQL interface to MapReduce and Hadoop Pre-tuned HDFS and MapReduce parameters for Big Data workloads Store and manage data in Apache Hadoop or Aster Database NEW InfiniBand (40 GB/s) Interconnect Fabric NEW Big Analytics Appliance Hardware Processing, storage, and networking designed for Big Data workloads 40 GB/s InfiniBand network 42

43 Aster MapReduce Analytics Portfolio The App Store of Big Data PATH ANALYSIS Discover Patterns in Rows of Sequential Data TEXT ANALYSIS Derive Patterns and Extract Features in Textual Data STATISTICAL ANALYSIS High-Performance Processing of Common Statistical Calculations SEGMENTATION Discover Natural Groupings of Data Points MARKETING ANALYTICS Analyze Customer Interactions to Optimize Marketing Decisions DATA TRANSFORMATION Transform Data for More Advanced Analysis 43

44 ESG Benchmark Report Summary Third Party Validation of Aster and Hadoop Fit Scope Identical hardware for Aster and Hadoop Clickstream, sentiment, and traditional retail data Compare time to insight and time to develop RESULTS Discovery Process: Aster 5x Faster Analytics: Aster 35x Faster (range: 4 416x) Development: Aster 3x Faster Loading: Hadoop 1.8x Faster Transforms: Hadoop 1.3x Faster Hadoop MapReduce Aster SQL-MapReduce 6 Hours 32 Hours Aster 5x Faster Discovery Cycle-Time (Development + Execution Time) FULL REPORT AVAILABLE AT 44

45 Comparing Advanced Analytic Development and Execution Example: Determine Spikes In Hourly Pageviews Apache Hadoop Execute 5 Write Java MR job to group records by pagename and find all pages <100 pageviews/hr Sort by the yy/mm/dd and hour fields Java reduce phase to place all same-keyed records into temporary arrays Compute counts for low/high/low hourly page views Create custom partitioner Create custom grouping comparator Create custom key comparator Execute each Mapper and Reducer Multiple passes of data Save output to flat files making it unstructured, No relational semantics and preventing use of DB interfaces (e.g. ODBC/JDBC) Retrieve results with other tools (e.g., SSH/FTP) Source: Enterprise Strategy Group, Lab Validation Report, September

46 Comparing Advanced Analytic Development and Execution Example: Determine Spikes In Hourly Pageviews Apache Hadoop Teradata Aster 1 2 Write Java MR job to group records by pagename and find all pages <100 pageviews/hr Sort by the yy/mm/dd and hour fields Java reduce phase to place all same-keyed records into temporary arrays Compute counts for low/high/low hourly page views 1 Execute Use Aster npath Input parameters in SQL as regular expressions Single Pass of the data SQL handles group-by, counts, sorts MapReduce perform regular pattern matching over a sequence of rows 3+ Create custom partitioner Create custom grouping comparator Create custom key comparator 3 Outputs written to relational table Use SQL or BI tools to visualize results Execute Execute each Mapper and Reducer Multiple passes of data 5 Save output to flat files making it unstructured, No relational semantics and preventing use of DB interfaces (e.g. ODBC/JDBC) Retrieve results with other tools (e.g., SSH/FTP) Source: Enterprise Strategy Group, Lab Validation Report, September

47 Comparing Advanced Analytic Development and Execution Example: Determine Spikes In Hourly Pageviews Apache Hadoop Teradata Aster 1 2 Write Java MR job to group records by pagename and find all pages <100 pageviews/hr Sort by the yy/mm/dd and hour fields Java reduce phase to place all same-keyed records into temporary arrays Compute counts for low/high/low hourly page views 1 Execute Use Aster npath Input parameters in SQL as regular expressions Single Pass of the data SQL handles group-by, counts, sorts MapReduce perform regular pattern matching over a sequence of rows 3+ Create custom partitioner Create custom grouping comparator Create custom key comparator 3 Outputs written to relational table Use SQL or BI tools to visualize results Execute Execute each Mapper and Reducer Multiple passes of data 5 Save output to flat files making it unstructured, No relational semantics and preventing use of DB interfaces (e.g. ODBC/JDBC) Retrieve results with other tools (e.g., SSH/FTP) Development Time: 4 hours Execution Time: 149 seconds Development Time: 1 hour (4x faster) Execution Time: 3 seconds (50x faster) 47

48 Comparing Advanced Analytic Development and Execution Example: Determine Spikes In Hourly Pageviews Apache Hadoop By using SQL-MapReduce, 1 Aster and takes find all pages fewer <100 pageviews/hr steps to Sort by the yy/mm/dd and hour fields develop analytics 2 Execute Write Java MR job to group records by pagename Java reduce phase to place all same-keyed records into temporary arrays Compute counts for low/high/low hourly page views Rather Create than custom using partitioner MapReduce 3+ Create custom grouping comparator processing Create custom for key each comparator step in the analysis, SQL is used in place Execute each Mapper and Reducer of a Map Multiple (or passes Reduce) of data phase and MapReduce is used only in Save output to flat files making it unstructured, steps No that relational cannot semantics and be preventing use of 5 DB interfaces (e.g. ODBC/JDBC) expressed Retrieve results in with SQL. other tools (e.g., SSH/FTP) Teradata Aster This is also why the execution Use npath time in Aster is much faster. 1 Execute Input parameters in SQL as regular expressions Single Pass of the data SQL handles group-by, counts, sorts MapReduce perform regular pattern matching over a sequence of rows Map or Outputs Reduce written to relational requires table data 3 Use SQL or BI tools to visualize results shuffling and produces higher latency than SQL Development Time: 4 hours Execution Time: 149 seconds Source: Enterprise Strategy Group, Lab Validation Report, September 2012 Development Time: 1 hour (4x faster) Execution Time: 3 seconds (50x faster) 48

49 Teradata Aster Big Analytics Appliance Key Innovations Copyright 2012 Teradata Corporation.

50 Aster SQL-H A Business User s Bridge to Analyze Hadoop Data Aster SQL-H Gives Analysts and Data Scientists a Better Way to Analyze Data Stored in Hadoop Allow standard ANSI SQL access to Hadoop data Leverage existing BI tool and enable self service Enable 50+ prebuilt SQL- MapReduce Apps and IDE Data Data Filtering Aster: SQL-H Hadoop MR Hive Pig NEW HCatalog Hadoop Layer: HDFS 50 Copyright 2012 Teradata Corporation.

51 Aster SQL-H Analytic Query Against Hadoop Data Iterative Data Discovery and Exploration SQL-H Makes Data Accessible for Multi-Channel Analysis in Aster Discovery Platform SQL-H ORIGINAL CALL CENTER DATA STORED IN HADOOP IVR Log Files with Associated Metadata RECORDS TRANSFORMED IN HADOOP Converted to HCatalog Table and Metadata 51 Copyright 2012 Teradata Corporation.

52 Hortonworks Data Platform Enterprise-Ready Hadoop The ONLY 100% open source data platform for Hadoop Tightly aligned with core Apache code lines All code committed back to open source Engineered integration with Teradata Viewpoint and Ambari HCatalog - centralized metadata services for easy data sharing Dependable full stack high availability Capacity scheduler for better multi-tenancy Intuitive graphical data integration tools 52

53 Teradata Viewpoint Integration Easier, Faster, and Better System Management Common Management Console for Aster, Teradata and Apache Hadoop Aster-Specific Portlets Aster Node Monitoring Aster Completed Processes Trend/ Visualization Portlets Capacity Heat Map Metrics Graph Metrics Analysis Query Portlets Query Monitor Admin Portlets Teradata System Roles Manager Other Portlets System Health Canary queries Aster Alerting 53

54 Teradata Vital Infrastructure (TVI) Integrated hardware & software solution for systems management PROACTIVE RELIABILITY, AVAILABILITY, AND MANAGEABILITY 1U server virtualizes system and cabinet management software Server Management VMS Cabinet Management Interface Controller (CMIC) Service Work Station (SWS) Automatically installed on base/first cabinet VMS allows full rack solutions without additional cabinet for traditional SWS Eliminates need for expansion racks, reducing customers floor space & energy costs Supports Teradata hardware and Aster/Hadoop software TVI Support for Aster and Hadoop 62 70% of Incidents Discovered through TVI 54

55 Copyright 2012 Teradata Corporation. How Can You Get Started? Aster Express

56 Making it easy to try Aster Big Analytics Solutions Aster Express, Aster Live, Aster Big Analytics Appliance Aster Express Aster Live Aster Big Analytics Appliance 56

57 Aster Express Tutorials Make it Easy to Start 57

58 Teradata Aster Big Analytics Appliance Summary Bring Big Data to Life with Big Analytics & Discovery INDUSTRY S FIRST UNIFIED BIG ANALYTICS APPLIANCE UNIFIED INTERFACES FOR ITERATIVE SQL AND MAPREDUCE ANALYTICS TERADATA-TRUSTED RELIABILITY, AVAILABILITY & MANAGEABILITY EASY TO DEPLOY, MANAGE & USE Get Started Now! asterdata.com/asterexpress 58

59

60 When to Use Which? The best approach by workload and data type Processing as a Function of Schema Requirements and Stage of Data Pipeline Low Cost Storage and Fast Loading Data Pre- Processing, Refining, Cleansing Simple math at scale (Score, filter, sort, avg., count...) Joins, Unions, Aggregates Analytics (Iterative and data mining) Reporting Stable Schema Teradata/ Hadoop Financial Analysis, Ad-Hoc/OLAP Enterprise-Wide BI and Reporting Spatial/Temporal Active Execution Teradata Teradata Teradata Teradata Teradata Evolving Schema Hadoop Interactive Data Discovery Aster / Aster / Web Hadoop Clickstream, Hadoop Set-Top Box Analysis CDRs, Sensor Logs, JSON Aster Aster Aster (SQL + Aster MapReduce Analytics) Format, No Schema Social Feeds, Text, Image Processing Aster Hadoop Audio/Video Hadoop Hadoop Storage Aster and Refining Aster (MapReduce Aster Analytics) Storage and Batch Transformations 60

61 When to Use Which? The best approach by workload and data type Processing as a Function of Schema Requirements and Stage of Data Pipeline Low Cost Storage and Fast Loading Data Pre- Processing, Refining, Cleansing Simple math at scale (Score, filter, sort, avg., count...) Joins, Unions, Aggregates Analytics (Iterative and data mining) Reporting Stable Schema Teradata/ Hadoop Teradata Teradata Teradata Teradata Teradata Evolving Schema Hadoop Aster / Hadoop Aster / Hadoop Aster Aster Aster (SQL + Aster MapReduce Analytics) Format, No Schema Aster Hadoop Hadoop Hadoop Aster Aster (MapReduce Aster Analytics) 61

62 Perceptions & Questions Analyst: Robin Bloor Twitter Tag: #briefr The Briefing Room

63 The Bloor Group

64 The Bloor Group Big Data Is About Analytics DATA AIN T WHAT IT USED TO BE Machine generated data (logs) Web data Social media data Public data services Supply chain data Real-time data flows THE ANALOGY OF STRIP-MINING IS RELEVANT BECAUSE THE SCALE OF DATA ANALYTICS HAS EXPANDED DRAMATICALLY

65 The Data Analytics Issue The Bloor Group

66 What Hadoop Is NOT A MULTIUSER HIGHLY TUNED ENGINE AN ANALYTICS PLATFORM A SOLUTION But it IS: A USEFUL, FLEXIBLE AND VERY ECONOMIC DATA STORE WITH PLUG-INS The Bloor Group

67 The Bloor Group About Data Analytics It is all about TIME TO INSIGHT as long as that is followed by action Fast time to insight requires FLEXIBLE management of high performance data flows - for the benefit of the data analyst The data analyst needs to be able to MARSHAL the data Then maybe, just maybe, he will deserve the title of DATA SCIENTIST

68 The Bloor Group Clearly the Teradata Aster Big Analytics Appliance is a powerful data flow engine, so:! How does Aster Data achieve its performance lift with MapReduce?! How is it most usually deployed?! Can it do data cleansing in flight?! Can it perform analytic tasks?

69 The Bloor Group! Why an appliance? What is gained and what is sacrificed?! Which sectors/businesses do you expect to be able to make best use of this technology?! Which companies/products do you regard as competitors (either direct or near)?! Which companies/products do you partner with?! How does the appliance fit in the cloud?

70 Twitter Tag: #briefr The Briefing Room

71 Upcoming Topics This month: Big Data February: Analytics March: Open Source April: Intelligence Twitter Tag: #briefr The Briefing Room

72 Thank You for Your Attention Twitter Tag: #briefr The Briefing Room