EMC CUSTOMER UPDATE. 31 mei 2011 Fort Voordorp. Bart Sjerps. Greenplum Data Warehouse. Copyright 2011 EMC Corporation. All rights reserved.

Transcription

1 EMC CUSTOMER UPDATE 31 mei 2011 Fort Voordorp Bart Sjerps Greenplum Data Warehouse 1

2 Introduction & Agenda What is Data warehousing? And what s Business Intelligence? Evolution in the Data Warehouse Business purpose Classic DWH architecture Present and future challenges EMC Solution Greenplum 2

3 What is a Data Warehouse? A Datawarehouse is not Vendor and consultant proclamations aside, a data warehouse is not: A project With a specific end date A product you buy from a vendor Like an ODS (such as SCT s) A canned warehouse supplied by istrategy Cognos ReportNet A database schema or instance Like Oracle Or SQL Server A cut-down version of your live transactional database According to Ralph Kimball and Joe Caserta, a data warehouse is: A system that extracts, cleans, conforms, and delivers source data into a dimensional data store and then supports and implements querying and analysis for the purpose of decision making. Another def.: The union of all the enterprise s data marts Aside: The Kimball model is not without some critics: E.g., Bill Inmon 3

4 History: From reports to advanced analytics Early days: run a simple report against the OLTP Database Run heavy batch reports against OLTP Database Dayly, weekly, monthly, year-end, adhoc Run custom queries against OLTP Database (using standard reporting tools) First use of what later became Business Intelligence, getting (market) knowledge from large amounts of information Offload databases for reporting and querying only Implemented as 1:1 copies, or custom designed databases (the first pure Data Warehouses) Need for Extract, Transform, Load tools (ETL) Evolved into OLAP (Online Analytical Processing); specialized methods for running Analytics This required special reporting tools as well Note: Running Batch and reporting on OLTP kills OLTP response time and performance 4

5 Classic vs. Next-gen business intelligence Old-style Datawarehousing: New style Datawarehousing: Frequently run reports/batch Built by programmers, optimized for performance and minimizing resource usage, requires huge developer and DBA efforts This is achieved by classic tuning such as using table indexes, partitioning, SQL optimization Very efficient but only for predictable queries Ad-hoc queries against OLTP data Can kill OLTP service levels, therefore this is often offloaded against prod database copy Optimizing using tricks such as materialized views Classic tuning fails (because it s unpredictable) DWH misused for pieces of business process Now mission-critical! Consider HA / DR / Compliancy Does not replace classic DWH! Get as much data from as many sources as possible Web, data feeds, legacy systems, smart electronics, etc etc Clean it up and modify it for analytics using ETL tools This is very resource intensive and typically requires long processing times Loading in the DWH can be problematic Classic DB systems again use workarounds for speeding it up Data needs to be as up-to-date as possible (less than 24 hours old) Build multi-dimensional databases That can have holes with missing data Build specialized data-marts Optimized by purpose Contains sub-set of all data 5

6 DW/BI purpose Risk Management Credit risk, Operational risk, Market risk Financial Analysis Customer credit, Cash inflow, Key financial ratios/performance Fraud Detection Internal fraud, External fraud Compliance Data integration, Reporting, Audit Customer Intelligence Behavior analysis, Spend & Value analysis, Portfolio analysis, Scorecard and Rating applications Performance Management Analysis of business methodologies, metrics, processes and systems 6

7 Classic Data Warehouse Architecture Data Sources CRM ERP Supply Chain Web Financial Systems Call Center Extract, Transform, Load (ETL) Fast Loading Staging Staging DWH Elastic Scale Data Mart Data Mart Data Mart Data Mart Reports ETL Data Warehouse Data Analysis 7

8 Datamart Sprawl EDW ~10 % of data Data is everywhere and growing 44X data growth by s of data marts Shadow databases Critical business insight is outside EDW Centralized legacy systems are expensive Data marts and personal databases ~90% of data System expansion is slow and process heavy Proprietary HW systems lag behind open systems innovation Traditional solutions cannot scale to meet the DW/BI challenges 8

9 Business Intelligence Challenges (1) Related to Infrastructure Higher service levels DWH not allowed to be down for a few days Need for backup/recovery/dr No SPOF, high-availability architecture Don t forget security, auditing, compliancy, data leakage prevention, customer privacy considerations (think Facebook and Google) Massive growth According to research firms, unstructured data will be biggest growth factor for companies Business Intelligence is #2 Soon we will see datawarehouses 100 s of Terabytes in size (And the first Petabyte customers) Business people want to store more and more in the DWH 9

10 Business Intelligence Challenges (2) Related to Infrastructure Loading time DWH needs to have up-to-date info Load times of multiple days is simply no longer acceptable 24H is max (for the whole process, not just loading) Long term, drive to real-time (ouch!) Scan time (how long does it take to run a query) More data More impatient end users More ad-hoc queries Cannot optimize this anymore with classic SQL tuning and database tricks & magic 10

11 Business Intelligence Challenges (3) Related to Infrastructure And finally New paradigms Multi-dimensional OLAP databases In-memory statistical calculations Needs to load a data subset in memory real quick Web users accessing BI data Of course, through web applications Massive scale-up in # of parallel transactions 11

12 What Takes Up the Most Time? You may be surprised to learn what DW step takes the most time Try guessing which: Hardware 30 Physical database setup Database design ETL OLAP setup st Qtr 2nd Qtr 3rd Qtr 4th Qtr Hardware East Database ETL West Schemas North OLAP tools Acc. to Kimball & Caserta, ETL will eat up 70% of the time. Other analysts give estimates ranging from 50% to 80%. The most often underestimated part of the warehouse project! 12

13 Data In. Decisions Out. 13

14 Database sizes over the years: -1996: 11 TB, in a Teradata Database -1999: 130 TB, in a Teradata Database - May 2008: 2 PB, Yahoo 14

15 It s Time for a Change... Yesterday s Data Warehouse and Analytic Infrastructure Proprietary Expensive Centralized, Monolithic Process-Heavy Batch Summarized Slow The Greenplum Future Commodity Cost-Effective Distributed Self-Service Real-Time Deep Agile

16 Greenplum True market disruption 20 Terabytes 20 kw, 8 Racks $20M 70 Terabytes 20 kw, 6 Racks $7M 100 Terabytes 12 kw, 2 Racks $1.8M 16

17 Industry Recognition: 2009 Gartner Magic Quadrant Gartner: Strengths Scale Mixed workloads Cloud ready Self service Low cost Concerns Company size Fixed by EMC R&D budget Fixed by EMC 17 *2007 was our first year on the MQ Source: Gartner (January 2010)

18 Greenplum Database Data in. Decisions out. Fastest Data Loading Advanced analytics Data in Scatter/Gather Streaming for the world s fastest data loading Eliminate data load bottlenecks Clean and integrate new data Several loading options ranging from bulk load updates to microbatching for near real-time processing In Database Analytics Optimized for fast query execution and linear scalability Move processing closer to data Shared nothing MPP scale-out architecture Computing is automatically optimized and distributed across resources Provides the best concurrent multi-workload performance Decisions out Unified data access for greater insight and value from data Enable parallel analysis across the enterprise Open platform with broad language support Certified enterprise connectivity and integration with most BI, ETL and management products 18

19 Greenplum Database Architecture Overview 19 19

20 Deployment models Greenplum Community Edition Free downloadable Limited to 2 segment servers All software is enabled Greenplum Software Only I.e. run on Vsphere / Vblock Or on standard (Intel) servers Greenplum DCD Appliance Pre-configured, tested, supported, plug & play Huge bandwidth DCD Appliance hybrid DAS / SAN 20

21 Architecture of Greenplum DCA Flexible framework for processing large datasets Process large datasets with support for both SQL and MapReduce etc SQL MapReduce UDF s: R,Java,C,Python,Perl ODBC, JDBC, OLEDB BI/ETL Tools Master servers optimize queries for the most efficient query execution Master Master Interconnect for continuous pipelining of data processing Segment servers process queries close to the data in parallel Segment Segment Segment Segment Segment MPP Scatter/Gather streaming for fast loading of data 21

22 Architecture Based on PostgreSQL (open source) database 15+ years of development Feature-rich, mission critical-ready Greenplum adds features on top of PostgreSQL Very low development cost (compared to traditional RDBMS vendors) Linear Scale-out Parallel loading Not depending on classic (OLTP) RDBMS tricks Special indexes, materialized views, 22

23 Greenplum vs OLTP DB architecture Greenplum Shared nothing master aggregates data Scale out only limited by master servers Optimized for Analytical processing As few indexes as possible (you don t know what you need anyway) Divide & Conquer Limited locking mechanisms -> no OLTP 23 OLTP Shared everything Cache fusion Scale out limited to ~ 4 nodes (if lucky) Optimized for transaction processing (OLTP) Highly optimized indexes - Extensively tuned for known queries Distribute workload (if possible) Full locking -> OLTP optimized

24 Greenplum DCA Instant Price/Performance Leadership EMC Greenplum DCA (1 rack) Oracle Exadata X2-8 (Full Rack) Netezza TwinFin 12 (Full Rack) Teradata 2580 (Full Rack) Architecture MPP Shared-Nothing MPP Shared-Disk MPP Shared-Nothing MPP Shared-Nothing Servers 16 Cores Scan (GB/s) w/o compression 192 Intel E5670 (2.93 GHz) 2 DB 14 Storage 128 DB 112 Storage Intel X Intel E5460 (3.16GHz) + 96 FPGA (2.26 GHz) + 96 FPGA 32 Intel Nehalem (2.66 GHz) 24 GB/Sec 25 GB/Sec 10 GB/Sec 10 GB/Sec Load (TB/Hr) >10TB/Hr 5TB/Hr 2 TB/Hr TBD Capacity (TB) Usable w/o Compression Capacity (TB) Usable w/ Compression 36 TB (600GB) 28 TB (600 GB) 32 TB 15 TB (450 GB) 144 TB 112 TB 128 TB 20 TB Largest Multi-Rack Configuration Max DB Cores Multi-rack Configuration 24 racks 8 racks 10 racks 10 racks

25 Greenplum 4.0: Self-Healing Fault-Tolerance Master Servers Network interconnect Segment Servers Master Servers 1. Segment server fails 2. Mirror segments take over, with no loss of service 3. Segment server is restored or replaced 4. Mirror segments restore primary via differential recovery (while online) Greenplum Database 4.0 enhances fault tolerance using a self-healing physical block replication architecture In Sync Key benefits of this architecture are: Resync ing Fast differential recovery and catch-up (always read-write) Change Tracking Improved write performance and reduced network load 25

26 Greenplum Data Computing Appliance Performance, Scalability, Reliability and Reduced TCO for DW/BI Environments Extreme Performance Optimized for fast query execution and unmatched data loading Rapidly Deployable Purpose-build data warehousing appliance Highly Available Self healing and fully redundant Elastic Scalability Expand capacity and performance online 26 Reduced TCO Consolidate Data Marts for lower costs Private Cloud Ready Data and computing are automatically optimized and distributed Advanced Backup and DR Leverage industry-leading Data Domain backup and recovery

27 Greenplum DCA Available Configurations Half rack (GP100) and Full rack (GP1000) GP100 GP1000 Expansion Bus 8 X Segment Servers Interconnect Bus 2X Master Servers 8 X Segment Servers 27

28 Greenplum DCA Specifications Half rack (GP100) and Full rack (GP1000) GP100 - Half Rack GP Full Rack Master Servers 2 Master Servers 2 Master Servers Segment Servers 8 Segment Servers 16 Segment Servers Memory per Server 48 GB 48 GB Total Memory 384 GB 768 GB Segment HDD s (SAS) Useable Capacity (uncompressed) Useable Capacity (compressed) 18 TB 36 TB 72 TB 144 TB Scan Rate 12 GB/Sec 24 GB/Sec Data Load Rate 5TB/Hour 10TB/Hour 28

29 Customers by Industry Financial Services Telco Media & Internet Retail Gov t & Health/Ins. 29

30 Overall DCD Partner Network and growing Hardware Vendors BI / ETL Tools Solutions & OEM Consultants And Resellers 30

31 Enabling Business Intelligence Through Virtual Enterprise Data Warehousing 31