IBM Netezza Taking advantage of the new wealth of information to make more intelligent decisions at the time of impact

Transcription

1 IBM Netezza Taking advantage of the new wealth of information to make more intelligent decisions at the time of impact Carlo Marchesi TechSales / System Engineer IBM Netezza 1

2 Würden Sie Google nutzen Wenn es Sie 3 Tage und 7 Personen kostet, um ein Suchergebnis zu bekommen? 2

3 Tage für eine einzige Abfrage andauerndes Tuning Nahezu 70% aller Data Warehouse Anwendungen leiden unter Leistungseinschränkungen der unterschiedlichsten Art. - Gartner 2010 Magic Quadrant Nur für Experten durchschaubar Monate bis zur ersten Nutzung 3

4 Traditionelle Data Warehouse Anwendungen sind einfach zu komplex Sie basieren auf Datenbanken die für die Transaktionsverarbeitung optimiert wurden NICHT um die Anforderungen von fortschrittlichen Analysen auf großen Datenbeständen abzubilden Zu komplexe Infrastruktur Zu komplizierter Einsatz Zu viel Tuning notwendig Zu ineffiziente Analysen Zu viel Personal für die Wartung Zu kostspielig im Betrieb Zu zeitraubend für schnelle Antworten 4

5 Agenda What is Netezza Customer Cases Architecture Hardware Scalability High Availability Integration ETL BI Analytics Backup & Restore Migration IBM Warehousing Offerings 5

6 IBM Netezza WHAT IS NETEZZA 6

7 Netezza s Market-Leading Evolution World s First Analytic Data Warehouse Appliance TwinFin with i-class Advanced Analytics (300X Performance) World s First Petabyte Data Warehouse Appliance TwinFin (150X Performance) World s First 100 TB Data Warehouse Appliance Impact NPS Series (50X Performance) World s First Data Warehouse Appliance NPS 8000 Series Source: If applicable, describe source origin 7

8 Appliance simplicity Integrated database, server & storage x faster than traditional systems Peta-scale data capacity Advanced analytics Standard interfaces Trust with compliance & auditing 8

9 Eine echte Appliance ermöglicht - Speed 15,000 users running 800,000+ queries per day (50X faster than before) when something took 24 hours I could only do so much with it, but when something takes 10 seconds, I may be able to completely rethink the business process - SVP Application Development, Nielsen Source: 9 9

10 Eine echte Appliance ermöglicht - Simplicity Up an running 6 months before having any training MONTHS Allowing the business users access to the Netezza box was what sold it. Steve Taff, Executive Dir. of IT Services DAYS WEEKS 10

11 Eine echte Appliance ermöglicht - Scalability 1 PB on Netezza (including DR) 7 years of historical data % annual data growth NYSE has replaced an Oracle IO relational database with a data warehousing appliance from Netezza, allowing it to conduct rapid searches of 650 terabytes of data. ComputerWeekly.com Source:

12 Eine echte Appliance ermöglicht - Smarts 30% redemption increase Using results derived from Netezza system, Catalina Marketing is able to give shoppers (more relevant) coupons at point of sale for items they would like want to buy in future visits. Editorial Director, DM Review 12

13 IBM Netezza ARCHITECTURE 13

14 The Netezza TwinFin Appliance Disk Enclosures Slice of User Data Swap and Mirror partitions High speed data streaming SMP Hosts S-Blades (with FPGA-based Database Accelerator) SQL Compiler Query Plan Optimize Admin Processor & streaming DB logic High-performance database engine streaming joins, aggregations, sorts, etc. 14

15 S-Blade Components SAS Expander Module DRAM SAS Expander Module Dual-Core FPGA Intel Quad-Core IBM BladeCenter Server Netezza DB Accelerator

16 The Netezza AMPP Architecture FPGA Memory CPU Advanced Analytics FPGA Memory CPU Host Hosts ETL BI FPGA Memory CPU Loader Disk Enclosures S-Blades Network Fabric Applications Netezza Appliance

17 Asymmetric Massively Parallel Processing SOLARIS AIX Netezza TwinFin Appliance Client TRU64 HP-UX WINDOWS LINUX ODBC 3.X JDBC Type 4 OLE-DB SQL/92 SQL Compiler 1 2 S-Blade Processor & streaming DB logic S-Blade Processor & streaming DB logic Query Plan Execution Engine 3 S-Blade Processor & streaming DB logic Quell- Systeme ETL Server DBA CLI 3rd Party Apps High-Speed Loader/Unloader Optimize Admin Front End SMP Host Netzwerk 920 High-Performance Database Engine Streaming joins, aggregations, sorts S-Blade Processor & streaming DB logic Massively Parallel Intelligent Storage High Performance Loader

18 Asymmetric Massively Parallel Processing SOLARIS AIX Netezza TwinFin Appliance Client TRU64 HP-UX WINDOWS LINUX SQL SQL Compiler Snippets S-Blade Processor & streaming DB logic S-Blade Processor & streaming DB logic Query Plan Execution Engine 3 S-Blade 1 Processor 2 & 3 streaming DB logic Quell- Systeme ETL Server DBA CLI 3rd Party Apps High-Speed Loader/Unloader Optimize Admin SQL Front End SMP Host Netzwerk 920 High-Performance Database Engine Streaming joins, aggregations, sorts S-Blade Processor & streaming DB logic Massively Parallel Intelligent Storage High Performance Loader

19 Our Secret Sauce select DISTRICT, PRODUCTGRP, sum(nrx) from MTHLY_RX_TERR_DATA where MONTH = ' ' and MARKET = and SPECIALTY = 'GASTRO' FPGA Core CPU Core Uncompress Project Restrict, Complex VisibilityJoins, Aggs, etc. Slice of table MTHLY_RX_TERR_DATA (compressed) select DISTRICT, PRODUCTGRP, sum(nrx) where MONTH = ' ' and MARKET = and SPECIALTY = 'GASTRO' sum(nrx)

20 Asymmetric Massively Parallel Processing SOLARIS AIX Netezza TwinFin Appliance Client TRU64 HP-UX WINDOWS LINUX ODBC 3.X JDBC Type 4 OLE-DB SQL/92 SQL Compiler Consolidate 1 2 S-Blade S-Blade Processor & streaming DB logic Processor & streaming DB logic Query Plan Execution Engine 3 S-Blade 1 Processor 2 & 3 streaming DB logic Quell- Systeme ETL Server DBA CLI 3rd Party Apps High-Speed Loader/Unloader Optimize Admin Front End SMP Host Netzwerk 920 High-Performance Database Engine Streaming joins, aggregations, sorts, etc. S-Blade Processor & streaming DB logic Massively Parallel Intelligent Storage High Performance Loader

21 TwinFin 24 Specification 16 (8*2) Disk Enclosures 192 (96*2) 1TB SAS Drives (8 hot spares) RAID 1 Mirroring 2 Hosts (Active-Passive): 24 Cores (Quad-Core Intel 2.6 GHz) 96 GB Memory 4x146 GB SAS Drives Red Hat Linux 5 64-bit 10G Internal Network 24 Netezza S-Blades: 192 Core s ( Intel Quad-Core 2.5 GHz) 192 FPGA s ( 125 MHz ) 384 GB DDR2 RAM (1+TB compressed) Linux 64-bit Kernel User Data Capacity: Data Scan Speed: Load Speed (per system): 250 TB 290 TB/hr 2.0 TB/hr Power/Rack: 7,400 Watts Cooling/Rack: 25,500 BTU/Hour

22 HIGH AVAILABILITY 22

23 Disk Mirroring and Failover Primary Mirror Temp All user data and temp space mirrored Disk failures transparent to queries and transactions Failed drives automatically regenerated Bad sectors automatically rewritten or relocated

24 S-Blade Failover S-Blades Drives automatically reassigned to active S-Blades within a chassis Read-only queries that have not returned data yet automatically restarted. Transactions and loads interrupted

25 INTEGRATION 25

26 Loading and Quering Netezza Datenintegration Reporting & Analyse Ab Initio Business Objects/SAP Composite Software Expressor Software Oracle GoldenGate Informatica IBM Information Server (Datastage) Oracle Data Integrator (Sunopsis) WisdomForce Talend Open Studio laden SQL ODBC JDBC OLE-DB SQL ODBC JDBC OLE-DB lesen Actuate Business Objects/SAP Cognos (IBM) Information Builders Kalido KXEN MicroStrategy Oracle OBIEE QlikTech Quest Software SAS SPSS (IBM) Unica (IBM) 26

27 IBM Netezza Analytics for Client Business Manager BI & Visualization IBM Cognos, SAS, Microstrategy, Business Objects, ESRI, MS Excel Analyst Model Building & Scoring IBM SPSS, Revolution Analytics, Fuzzy Logix, SAS, R Developer Custom Analytics R, Hadoop, Java, C, C++, Python, Fortran IBM Netezza appliance Data Preparation Geospatial Analytics Data Mining Advanced Statistics Predictive Analytics

28 Advanced Analytics the Traditional Way SAS Data Warehouse Analytics Grid Data SQL ETL Demand Forecasting ETL R, S+ ETL SQL C/C++, Java, Python, Fortran, Fraud Detection SQL 28

29 Advanced Analytics with IBM Netezza Analytic Tools Data Warehouse Analytics Grid Data SQL ETL Demand Forecasting ETL ETL SQL C/C++, Java, Python, Fortran, Fraud Detection SQL 29

30 Advanced Analytics with IBM Netezza Analytic Tools Demand Forecasting Fraud Detection 30

31 IBM Netezza Analytics Technical Overview Client Revolution Analytics Analytic Packages IBM SPSS Modeler SAS Software Development Kit User-Defined Eclipse Client Extensions Plug-in Adaptors for R, Hadoop Java, C, C++, Python, Fortran Client Connectors / Partner Engines (e.g., Fuzzy Logix) IBM Netezza Analytics IBM Netezza data warehouse appliance Hadoop R Spatial Matrix In-Database Analytics Data Data Prep Mining Predictive Analytics Language Support & Adaptors IBM Netezza AMPP Platform 31

32 IBM In-Database Analytics Analytic Task Data Exploration and Discovery Data Transformation Model Building Model Diagnostics Model Scoring Function Types Data Profiling Descriptive Statistics Principal Components Analysis Bayesian Network Standardization Normalization Binning Missing Data Treatment Classification: Decision Trees, Naïve Bayes Estimation: Linear Regression, Regression Trees Clustering: k-means, Hierarchical Association: a Priori, FP-Growth Mean/Relative Error Statistics Hypothesis Testing Classification/Misclassification Statistics Linked to Model Building functions 32

33 MONITORING 33

34 nzportal: Monitoring and Capacity Planning Visual representation of Query History database Search for and compare runtimes of identical queries

35 BACKUP & RESTORE 35

36 Backup and Restore Features Enterprise-class integration and certification with 3rd-party BAR tools Simplify deployment with leading backup and restore tools X/Open Backup Services API (XBSA) support Certification with Veritas NetBackup from Symantec Certification with IBM Tivoli Storage Manager Incremental backup and restore Significantly shorten backup times compared to full backups Available in NZBACKUP utility Full or partial restore options Sun Mon Tue Wed Thu Fri Sat Full Diff Diff Cum Diff Diff Diff

37 MIGRATION 37

38 Oracle to Netezza Conversion Tool The mission Tool to automatically convert Oracle SQL (and PL/SQL) to Netezza SQL (and NZPLSQL) To minimize manual effort during code conversion from Oracle to Netezza. Assessment Estimation Conversion Testing Size and complexity of Data Assessment data fed into an Estimation tool Tool to minimize manual conversion of Oracle Code Correctness of Syntax / Semantics Complexity of Database Objects / Data types Complexity of PL/SQL codebase (if any) Development of a tool like MEET in plan The following Oracle entities can be converted to Netezza equivalents. Data type compatibility Are Netezza best practices followed? Oracle Data Types and Formats Data Definition statements Data Manipulation statements Oracle PL/SQL & Stored Procedures

39 Conversion Capability - Today Oracle Datatypes Character Datatypes (CHAR,VARCHAR2,LONG,NVARCHAR2,NCHAR, RAW,LONG RAW) Numeric Datatypes (NUMBER,NUMERIC,FLOAT,DEC,INT,SMALLINT,REAL,DOUBLE,) Temporal Datatypes (DATE,TIMESTAMP, TIMESTAMP with timezones, INTERVAL DAY/YEAR) Large Objects (BFILE,BLOB,CLOB,NCLOB) Basic Oracle DDLs Create Statements (TABLES,VIEWS, SEQUENCES,MATERIALIZED VIEWS,USERS,GROUPS) Alter Statements Rename Statements Truncate Basic Oracle DMLs INSERTs UPDATEs DELETEs SELECTs (Including Oracle s OUTER JOIN (+) syntax) Correlated Subqueries in UPDATE statements PL/SQL & Stored Procedures Stored Proc declarations with and without parameters Nested Stored Procedures (Will be Unnested for Netezza) PL Constructs (loops) Oracle Exceptions Output statements (DBMS_OUTPUT) Anything ANSI does not require conversion!

40 What cannot be converted (non-exhaustive) Oracle Datatypes RowID Datatypes (ROWID,UROWID) X Oracle DDLs INDEX related statements X TABLESPACE related statements X All other Oracle db objects unsupported in Netezza X Basic Oracle DMLs CURSOR Operations ( Oracle + format) X PL/SQL & Stored Procedures RECORD Variable declarations and operations X Oracle EXCEPTIONS X Output statements (DBMS_OUTPUT) X Key: X - Not Required by Netezza X Manual conversion will be advised

41 National Stock Exchange : Netezza PoC In Progress The PoC is underway with the following Generic success criteria Task Current Time NSE s Expectation Results on Netezza Data Load Rate 40 GB per Hour 400 GB per Hour (10x) 1.1 TB per Hour Data Compression 65% Compression > 65% > 75% ETL processing 70 mins 7 mins (10x) 2m30Secs BO queries 70 mins 3.5 mins (20x) 1m30Secs Intraday Processing 6Hrs 30Mins 39 mins (10x) In Progress EOD Processing 6Hrs 30Mins 39 mins (10x) In Progress NSE s current environment is Oracle on a 16 Core HP server Netezza configuration being used for the PoC TwinFin12 Competitiors Exadata, Greenplum

42 IBM DATA WAREHOUSE OFFERINGS 42

43 Appliance family for data life-cycle management Netezza-100 Smart Analytics System 5710 Netezza-1000 Dev & Test System Complete BI Solution Data Warehouse High Performance Analytics Netezza High Capacity Queryable Archiving Back-up / DR 1 TB to 10 TB 1.5 TB to 13 TB 1 TB to 1.5 PB 100 TB to 10 PB 43

44 IBM DB2 Analytics Accelerator zenterprise IBM DB2 Analytics Acelerator Admin Client Data Studio Foundation DB2 Analytics Accelerator Admin Plug-in OSA-Express3 10 GbE Primary Private Service Network 10Gb Backup Online Transactions BladeCenter Analytic Queries DB2 for z/os (Data Warehouse) enabled for IBM DB2 Analytics Accelerator z/os: Recognized leader in Security, Availability, & Recoverability for OLTP Netezza: Recognized leader in cost-effective high speed deep analytics

45 Netezza Differentiators & Business Impact Price/Performance Leader Data mart to enterprise warehouses Low TCO, fast time to value Deliver analytics to the masses Speed Scalability Simple Smart Hardwarebased data streaming True MPP offers enterprise scale-out Black-box appliance with no tuning or storage administration Built-in advanced analytics pushed deep into database 45

46 USE CASES 46

47 Netezza Deal Scenarios 1 2 EDW Replacement Install Next-gen EDW 3 High Value, Focused Data Marts 4 5 Retail Analytics Out of the Box SAP BW Co-exist / Replacement

48 Netezza Deal Scenarios Solution Target Accounts Target Audience Value Prop/ Differentiated Positioning 1. Teradata, EDW Replacement Netezza Twin Fin Tired of paying TD/ORCL maintenance and cost for expansion. Not using a lot of TD/ORCL proprietary applications on the platform. Trying to change business (add new analytic capabilities). Sponsor exists in account who will be a change agent. Typically, CIO will sponsor this or SVP of BI / DW at a larger account. Establish EDW that s faster, simpler, cheaper (esp TCO) than Teradata, Oracle, et al. Unique architectural approach (esp DB DW) enables CIO to achieve operating leverage over life of technology. Better serve needs of business immediately and over long run architectural features enable biz growth and change cheaply, easily, quickly (vs. ORCL, etc.).

49 Yum! Brands - Teradata EDW Replacement A $11B global quick serve leader with 37,000 restaurants in 110 countries. Brands include KFC, Pizza Hut, Taco Bell, Long John Silver s, and A&W. Challenge Solution Result Growing Business w/ expanding demand for BI and analytics BI user satisfaction was declining Cost/Performance Appliances were maturing Good Time for a Change Goal: Better, Faster, Cheaper BI Lower TCO on legacy playform, by replacing Teradata Separate platforms for high performance analytics and BI from operational reporting. 3x POC w/ Teradata, DataAllegro and Netezza Netezza was 2-4x faster than Teradata Netezza deployed in 3 mo for forst brand, <1 yr for all brands BI Platform deployed to 1,400 users Queries run 2-11x faster than previously with Teradata 60% reduction in TCO over Teradata Shift in focus from admin task to BI and analytics 49

50 Details: IBM Netezza is better value than Oracle Exadata Oracle Exadata Results In IBM Netezza Client Advantage Costs Acquisition cost can exceed $7M per rack Hardware $1M Software is more than $6M! High maintenance and software subscription Continuing high admin costs High cost of ownership Low, transparent initial cost Simple install requires no additional professional services Standard maintenance includes hardware /software support & software upgrades Netezza Migrator for easy migration of Oracle applications Fast deployment & time-tovalue Easily understood, predictable costs Minimal extra services so easier to budget for Netezza Smart Very limited push-down analytics RAC bottleneck for analytic performance Poor analytic performance Push down of many diverse analytics (SAS, R, Gnu, etc.) with Netezza i-class Fast time to insight Advanced analytics on big data easily accessible Simplicity Complexity of Oracle Real Application Clusters (RAC) Constant tuning for performance Complex maintenance and patch process Complex administration True Appliance with HW/SW created to provide high performance for data warehousing No tuning More time spent delivering business value rather than tuning for acceptable performance Scalability No proof points of scaling Backup/restore bottlenecks Parallelism for joins limited to RAC tier Scalability Bottlenecks Proven scalability at peta-scale No scalability bottlenecks No restrictions on business and data growth Speed Tuned for OLTP (e.g. FlashCache) RAC unfit for data warehouse workloads Poor data warehouse performance Appliance tuned for data warehousing and advanced analytics Highest data warehouse performance Operational Simplicity Architecture Two layer: Clustered SMP DB Layer (RAC) Shared disk MPP Storage Layer Compromised performance True MPP with FPGA acceleration of processing in each MPP node Best architecture for data warehouse and advanced analytics due to minimization of contention/bottlenecks

51 Netezza Deal Scenarios Solution Target Accounts Target Audience Value Prop/ Differentiated Positioning 2. Install Next-gen EDW Netezza Twin Fin Less mature retailer on outdated and/or disorganized technologies. Looking to rationalize data platforms and achieve better performance as efficiently as possible. Sponsor exists in account who will be a change agent. Typically, CIO will sponsor this. Establish EDW that s faster, simpler, cheaper (esp TCO) than alternative vendors Teradata, Oracle, et al. Unique architectural approach (esp DB DW) enables CIO to achieve operating leverage over life of technology. Better serve needs of business immediately and over long run architectural features enable biz growth and change cheaply, easily, quickly (vs. ORCL, etc.).

52 Carter s Next-gen EDW Leading children s apparel retailer, with 6 brands, more than 375 Carter s and OshKosh retail stores and distribution through national department stores, chain and specialty stores, and discount retailers. In 2009 Carter s revenue was $1.68B Challenge Solution Result No single view of the biz 2005 acquired OshKosh Multiple brands, product hierarchies, distribution channels, planning and allocation systems Could not support detailed analysis: SKU Store Trasaction Missing SLAs Poor Performance: 12 hrs for an analytic report Oracle system admin was absorbing too many resources 52 Netezza: Most straightforward, minimal administration, lowest TCO Exadata: Essentially a bigger, faster version of existing platform; same administration requirements and Oracle limitations; platform partner concerns HP, SUN? Teradata: In transition phase from big DW player to appliance vendor Consistently meeting SLAs Faster, better analytics: fresh sales trends data, market basket analysis (NEW), imminent stock-out (NEW) 1 platform, 1 data model, 1 product hierarchy for all brands and channels Reduced admin. reports cut from 120 to hour analytical data reformat reduced to 90 minutes on Netezza

53 Netezza Deal Scenarios 3. High Value, Focused Data Marts Solution Target Accounts Target Audience Value Prop/ Differentiate d Positioning Netezza Twin Fin Has new strategic initiative involving big data, big analytics (a la customer hub, marketing analytics). Struggling to make merchandising data warehouse functional (e.g., Oracle Retail RDW port). Typically larger, Tier 1 retailers (e.g., Best Buy, Target, Tesco). Project can be implemented alongside existing EDW to enhance/ extend. Looking for speed-to-value for business with minimal disruption to existing IT foundation. Position with Chief Marketing Officer, Chief Merchant, and/or CIO. Establish high value data hub for business in weeks. Enable apps and/or analysis quickly to support key business initiatives (e.g., multi-channel customer experience). Realize value of investment in merchandising OLTP (Oracle Retail RDW). Can easily, cheaply evolve to play bigger role in EDW if desired.

54 Netezza Deal Scenarios Solution Target Accounts Target Audience Value Prop/ Differentiated Positioning 4. Retail Analytics out of the box Netezza Retail Analytic Appliance Under invested in technology and looking to scale/ evolve technology to support maturing business with minimal risk and business disruption. Has no DW/BI or is not happy with existing functionality. Haven t capitalized on BI investment (MS, Cognos) and/or data in merchandising OLTP apps. Has too many reports, and/or has widespread use of Excel offline. Avoid established Teradata accounts, where unlikely to replace just for this solution. Position with Chief Merchant / GMM, VP of Planning, Head of Stores, CFO and/or CIO. Only complete, proven solution HW, BI & content (data model, ETL, metrics, reports, dashboards, playbooks) out-of-the-box deployed at leading retailers. Fast, Efficient, Low Risk - Live in 4-8 weeks with less risk and cost than build or alternative offerings. Page 54

55 Summary Key Messages Financial Netezza is far less expensive than comparable competitive systems Netezza is the ONLY technology that provides operating leverage for IT / CFO. Technical Netezza is designed as a data warehouse appliance. Our performance is driven by an elegant design, not by brute force. Designed for simplicity drives speed to value for application development. Business Simplicity means that you can better service the business by taking off the guard rails. Let the business analyze at the speed of thought in the context of a business process. Business people don t do well when they are constrained by IT. They are creative by nature. Competitive Everybody talks about having an appliance, but the only thing they have in common with Netezza is that they are in a single cabinet. Customers buy because of price / performance, while they stick with us because of simplicity. 55

56 Netezza und SAP Einfache Integration von SAP und Nicht-SAP Daten Schnittstellen zur bestehenden BI Lösung Cognos Ergänzung oder kompletter Ersatz von SAP BW Mit SAP BW: Basierend auf SAP OpenHub Interface Ohne SAP BW: Integration basierend auf SAP ECC Einfache Skalierbarkeit ohne zusätzliche Komplexität

57 Netezza und SAP mit NewFrontiers NewFrontiers liefert vordefiniert: ETL-Prozess Datenmodell InfoProbes (Reports) Rasche und einfache Integration POC in 10 Tagen Referenzen: Heineken Unilever Generali Roche ING... InfoProbes reporting, analysis alerts, dashboards analytics DataMart(s) ODS Staging ETL/ELT process Non SAP

58 Test Drive TwinFin Your Data. Your Site. Our Appliance. 58