Transcription

1 Welcome to Today s Web Seminar! March 15, :00PM ET Sponsored by: Hosted by:

2 Eric Kavanagh is the host of DM Radio and Information Management's Webcasts. He is a veteran journalist and consultant t with 20 years of experience in print, broadcast and online media. For the past 10 years, he has focused on enterprise technology and information management. Recently, he helped spur adoption of the Federal Funding Accountability and Transparency Act, which led to the deployment of USASpending.gov, a Web-based portal that provides detailed financial information about all federal contracts and grants. Prior to joining the magazine, Eric was web editor for TDWI. He also founded and runs Mobius Media, a strategic communications firm based in New Orleans. You can him at dmradio@sourcemedia.comcom

3 Mike Ferguson is Managing Director of Intelligent Business Strategies Limited. As an analyst and consultant he specializes in business intelligence, data management and enterprise business integration. With over 29 years of IT experience, Mike has consulted for dozens of companies, spoken at events all over the world and written numerous articles. He is an expert on the B-EYE-Network in UK, Germany, Netherlands, Spain, India, Scandinavia and China. Formerly he was a principal and co-founder of Codd and Date Europe Limited the inventors of the Relational Model, a Chief Architect at Teradata on the Teradata DBMS and European Managing Director of DataBase Associates. mferguson@intelligentbusiness.biz Tel/Fax (+44)

4 Francois Ajenstat Software Ajenstat is Director, Product Management at Tableau As Director of Product Management, Francois Ajenstat is responsible for product strategy and product planning combining customer and partner feedback to set the strategic t vision i and leading the execution of new features and products. Prior to joining Tableau, Francois worked at Microsoft for 10 years in a number of different groups including SQL Server, Office and Trustworthy Computing. Prior to joining Microsoft, Francois worked for Cognos Corporation (acquired by IBM) leading strategic alliances with key industry partners such as IBM, HP, and Microsoft..

5 To Ask a Question: 1. Click Q&A pane 2. Type question in the Type a question for the presenter field 3. Click the Ask button Use the scroll button to scroll to your answer if necessary

6 Analysing and Visualizing Big Data What s Possible? Mike Ferguson Managing Director Intelligent Business Strategies March 2011

7 Topics Types of big data Popular big data analytic applications Challenges when managing and analysing big data Scalability options for data integration in big data environments Data storage options for big data analytics Analytical database Vs NoSQL Hadoop Map/Reduce Big data analytics query performance enablers Using data visualisation and in-memory data in a big data environment 7

8 Types of BIG Data Web data Clickstream data E-commerce logs Online gaming Online advertising Social networks Vertical industries i transaction ti data Telecommunications Mobile phone and fixed line call data records (CDRs) Geo-location data Retail transaction data Pharmaceutical Drug Test data 8

9 Web Data Example: Social Networks The World Is Obsessed With Facebook 500m 500 active ti monthly thl users Source: Over 500 BILLION page views per month 25 billion new pieces of data per month Petabytes of data (>10TB per day) 9

10 Types of BIG Data Sensors Enable An Internet Of Things To Generate Huge Volumes of Data Sensor data Temperature Light Vibration Location (e.g. inside smart phones) Airflow Liquid flow Pressure Chemical Picture Source: The Economist, 4th Nov 2010 A Special Report on Smart Systems E.g. RFID tag 10

11 Types of Big Data Unstructured Content How much is TEXT worth to your business? 11

12 Popular BIG Data Analytic Applications Web Data Clickstream analytics Site navigation behaviour (session analysis) On-line advertising click through analysis On-line advertising success V failure rates Combine with customer and/or login data to associate clickstream with specific people E-commerce purchase analytics Online shopping cart basket analysis Recommendation effectiveness analysis Combine with customer data to associate on-line customer segmentation with buying behaviour etc. Social network influencer analysis Graph analytics for Influencer behavioural impact analysis Target the influencer marketing campaign effectiveness 12

13 Popular BIG Data Analytic Applications Sensor Data Supply/distribution chain optimisation Sustainability analytics e.g. energy optimisation Asset management Location based advertising (mobile phones) Grid health monitoring Electricity, water, oil pipelines, mobile phone cell network Smart metering Fraud Traffic optimisation People flow optimisation e.g. Heathrow T5 Smart Buildings Complex Event Processing (CEP) AND traditional analysis of sensor data are increasingly common 13

14 Popular BIG Data Analytic Applications Unstructured Data Case management Fault management and field service optimisation Voice of the customer Sentiment analytics Competitor analysis Media coverage analysis 14

15 Challenges With BIG Data Some analytic apps require low latency data capture Online advertising Fraud detection Location specific advertising (mobile devices) May be a problem if it also has to be integrated with other data BIG data capture Requires data integration tool connectivity to web logs, clickstream, mobile data, online gaming, web analytics.. Data transformation and integration with other data sources Data loading CDC is key, micro-batch loading, parallel load, pre-calculated data Data storage where do you put it? Maintaining historical data Analysing large data volumes e.g. online Vs. batch analysis 15

16 Scaling Data Integration for BIG Data - Options 1. Data integration software that uses ELT processing Data integration software leverages the power of a MPP DBMS to deal with very large data volumes 2. Massively parallel data integration software Specially developed data integration software that executes every function in parallel e.g. Pervasive DataRush 3. Data integration accelerators Integrate with data integration software products to boost transformation performance e.g. Syncsort DMExpress Exploitation of Hadoop Map/Reduce from within the Data Integration software Both accelerator inside Hadoop and Hadoop used in DI 4. Specialist data integration software E.g. Facebook Scribe log aggregation software (open source) 16

17 Leveraging Hadoop For Cloud Based Data Integration On Massive Volumes Of Data To Bring Insight Into A DW Hundreds d of Cloud Data e.g. Deriving insight from huge terabytes up volumes of social web content on to petabytes sites like twitter, facebook. Digg, myspace, tripadvisor, Linkedin.for sentiment analytics Extract Operational systems Cloud Data Transform D I DW HDFS Map/ Reduce apps e.g. eg PIG, IBM JAQL relevant insight 17

18 Data Storage Options for BIG Data Analytics Classic RDBMS E.g. Analytic DBMS E.g. NoSQL option E.g. Hybrid - Analytic DBMS with Hadoop Map/Reduce integration E.g. 18

19 Data Storage Options for BIG Data Analytics - Why Are People Selecting Analytical DBMSs? Copyright TechTarget, BIResearch, IT Market Strategy, 2010 Source: Analytic Platforms: Beyond the Traditional Data Warehouse BeyeNetwork Custom Research Report, White & Adrian,

20 Data Storage Options for BIG Data Analytics - Hadoop HDFS Partitions Data Over Thousands of Servers Source: Cloudera 20

21 What Is Hadoop MapReduce? A restricted programming model (NOT a programming language) Designed to interpret data at time of processing Handles big data - bulk data processing, scalable storage across machine clusters (think read-only file system) Shared nothing architecture t Hadoop M/R is NOT a good match where: Low latency is critical for accessing data Processing a small subset of the data within a large data set Real-time processing of data that must be immediately processed Hadoop is not normally a RDMS competitor It expands the opportunity to work with a broader range of content Source:IBM 21

22 The NoSQL BIG Data Warehouse Option - Hive Data Warehouse On A Hadoop Cluster Source: Facebook's Petabyte Scale Data Warehouse using Hive and Hadoop Map/Reduce jobs invoked by HiveQL to analyse the data with M/R results written back into Hive tables 22

23 Hybrid RDBMS/NoSQL Analytics and Map/Reduce - Aster Data Aster Data offers a suite of map/reduce analytic functions to accelerate development of rich applications Analytic Solution Analytic Application Business Logic Aster Data Analytic Foundation Source: AsterData SQL and SQL-MR Data Suite of pre-built functions for common operations - Path Pattern analysis Sessionization - Statistical Approximate percentiles and counts Correlation Regression Averages - Relational l Basket analysis Graph analysis Tokenization 23

24 Upstream Hadoop Feeding Into Traditional Warehousing Is A Popular Emerging Pattern Source: IBM 24

25 BIG Data Analytics Query Performance Enablers Speed up I/O Processing with Solid State Disks (SSDs) Minimising I/O is a key enabler Filter data close to the disk e.g. FPGA processors Columnar database query processing Data compression Scan sharing Query 1 data Query 2 reuse Query 3 Query 4 In-database analytics and pushdown optimisation i In-memory data Columnar in-memory storage, compression for most popular data 25

26 Columnar Vs. Row based Data Stores Columnar Databases Can Significantly Reduce I/O E.g. SELECT avg(price) FROM tickstore WHERE symbol = GM and date = 1/17/2007 Column Store Reads 3 columns GM GM GM AAPL ,000 10,000 12,500 9,000 NYSE 1/17/2007 NYSE 1/17/2007 NYSE 1/17/2007 NQDS 1/17/2007 Row Store Reads All 5 columns Column compression GM ,000 NYSE 1/17/2007 / GM ,000 NYSE 1/17/2007 GM ,500 NYSE 1/17/2007 AAPL ,000 NQDS 1/17/2007 RLE GM GM, GM GM AAPL ,000 10, ,500 9,000 RLE 1/17/2007, 1/17/2007 1/17/2007 1/17/2007 GM,4 1/17/2007,5 AAPL,1 26

27 BIG Data Query Performance - Pushdown Optimisation and In-Database Analytics General trend to running analytical function much closer to the data to improve performance Often referred to as in-database analytics Objective is to achieve pushdown optimisation to exploit powerful analytics in the DBMS rather than take big data out and do this outside the DBMS Examples of in-database analytics include Predictive and statistical models running inside the DBMS MapReduce external table functions running in the DBMS server Analytical functions written in other languages and running in the DBMS In-Database Analytics Data Prep Data Mining Predictive Analytics Spatial R Analytics Scientific Analytics 27

28 Data Visualisation and In-Memory Data In a BIG Data Environment It is not always practical to bring all data into memory in BIG data environments BI servers and data visualisation tools need to exploit pushdown optimisation and din-database analytics to leverage the power of parallel computing Columnar in-memory data stores and data compression allow a lot of data to be brought into memory for visualisation and detailed analysis Shared access to popular in-memory data will boost performance for multiple concurrent users Browser or desktop In-memory data Data Prep Data Mining Data visualisation e.g. Dashboards In-Database Analytics Spatial R Analytics BI In-memory server(s) pushdown optimization Predictive Analytics Scientific Analytics 28

29 Thank You! 29

30 Additional Resources Web Seminar Resources + For a copy of the presentation workbook and to hear the web seminar on-demand go to Q & A + If you have a question, please type it in the panel for an immediate reply or contact us via or phone. Michael Ferguson + Intelligent Business Strategies Francois Ajenstat + Tableau Software +@ajenstat + fajenstat@tableausoftware.com + (206) x5483 All rights reserved Tableau Software Inc.