IBM Informix Warehouse Accelerator (IWA)

Transcription

1 Fred Ho Informix Development Sept 4, 2013 IBM Informix Warehouse Accelerator (IWA) 1

2 Agenda Data Warehouse Trends IWA Technology Overview IWA Customers and Partners IWA Reference Architecture and Competition IWA Roadmap and Features

3 Database and Data Warehousing Industry TRENDS 3

4 Data Warehousing Workload & Optimizations Data Warehousing/OLAP workload are inherently more complex than OLTP transactions and reasons are well-documented. Ways to overcome that include: Building Indexes Partitioning of data Building cubes, MOLAP / ROLAP / HOLAP Query tuning Appliances that add a new layer of hardware to perform I/O for DBMS Mixed-Workload always a challenge DBMS needs to be built to handle such a workload 4

5 Third Generation of Database Technology According to IDC s Article (Carl Olofson) Feb st Generation: Vendor proprietary databases of IMS, IDMS, Datacom 2nd Generation: RDBMS for Open Systems Dependent on disk layout, limitations in scalability and disk I/O 3rd Generation: IDC Predicts that within 5 years: Most data warehouses will be stored in a columnar fashion Most OLTP database will either be augmented by an in-memory database (IMDB) or reside entirely in memory Most large-scale database servers will achieve horizontal scalability through clustering 5

6 Data Warehouse Trends for the CIO, Data Warehouse Appliances: DW appliances are not a new concept. Most vendors have developed an appliance offering or promote certified configurations. Main reason for consideration is simplicity. The Resurgence of Data Marts: Data marts can be used to optimize DW by offloading part of the workload, returning greater performance to the warehousing environment. Column-Store DBMSs: CIOs should be aware that their current DBMS vendor may offer a column-store solution. Don t just buy a column-store-only DBMS because a column store was recommended by your team. In-Memory DBMSs: IMDBMS technology also introduces a higher probability that analytics and transactional systems can share the same database. Source:TheState of DataWarehousing in 2011; 1/31/2011; Gartner MarkBeyer,Roxane Edjlali,Donald Feinberg(IDNumber: G ) 6

7 In-Memory DBMSs (IMDBMS) Have Plenty of Promises Promise that they will transform the way data is provisioned and consumed The extreme performance enable true real-time decision making on any shape and volume of data Powers unique business innovation and competitive differentiation in today s high-speed business environment Gartner estimates that IMDBMS will replace 25% of traditional data warehouse and OLTP systems by 2016 Examples are: Real-time pricing for airlines or banks evaluating assets for rebalancing of portfolio Lower TCO by reducing the need for separate OLTP and data warehouse copies of data and by eliminating need for cubes, aggregates and indexes. Gartner ID Number: G , Publication Date: 8 Sept 2011, Authors: Roxane Edjlali, Donald Feinberg 7

8 IDC WW RDBMS Forecast As a result of the near-ubiquitous adoption of 64-bit processors, precipitous declines in the price of main memory, and the need for greater transaction throughput, database technology is migrating from a diskbased paradigm to a memory-based one. Data is increasingly stored in memory, protected by redundant replication and asynchronous logging (for recovery), and organized for highly efficient retrieval and update. 8

9 Does the 21st-Century "Big Data" Warehouse Mean the End of the Enterprise Data Warehouse? Key Findings: Organizations that deploy an EDW almost all create second and third data warehouses or marts to support additional user needs (judging from up to 90% of the data warehouse inquiries received from Gartner clients), despite strict instructions to use the EDW. The architectural style of a data warehouse is usually determined by the available skills and tools, and secondarily by time-to-delivery. Source: Gartner; Mark Beyer, Donald Feinberg (ID Number G

10 Market Research on IWA: What do the Analysts Say? White Paper by Bloor Research: IBM Informix in Hybrid Workload Environments, August 2012 for hybrid environments, Informix has a number of unique capabilities that cannot be matched by either conventional data warehouse vendors or traditional data warehouses White Paper by Ovum Research: Informix Accelerates Analytic Integration into OLTP, July 2012 Supporting both operational and analytic workloads with the same system is a relatively unique idea that is also being pursued by rivals such as Oracle. Ovum believes that IBM now has a strong story to tell with IWA. Magic Quadrant for Data Warehouse Database Management Systems by Gartner, Jan 2013 IBM has a vast number of products that use in-memory computing, including soliddb, but its only in-memory solution for the data warehousing and analytical market is Informix Warehouse Edition. To download these and other papers, go to: 10

11 IBM Informix Warehouse Accelerator (IWA) IWA TECHNOLOGY OVERVIEW 11

12 Informix Warehouse Accelerator (IWA): Overview and Seamlessly Integration with Informix/IDS SQL Queries (from apps) Informix Warehouse Accelerator Informix Query Router 64-bit IDS Database SQL Results (via DRDA) TCP/IP Linux x86_64 Query Processor Compressed DB partition Bulk Loader Informix: Routes SQL queries to the Accelerator User need not change SQL or applications Can always run query in Informix if not accelerated Informix Warehouse Accelerator: Connects to Informix via TCP/IP Analizes, compresses and loads In-Memory a copy of (portion of) Informix warehouse Proceses routed SQL queries with extraorinary speed Returns results/answer back to Informix/IDS Informix Warehouse Accelerator (IWA) transparently accelerates Informix warehouse/analytic queries up to 100 times or more!

13 IWA Technology Innovations provide: Extreme unparallel analytics speed for fast business decisions 64 bit Intel/AMD Prcessors Compresion TB of RAM Memoria No Need for Aggregate Tables Row and Column Store Number of occurrences Common Values Rare Values Intelligent Frequence Paritioning SIMD Predicates Evaluation on Compressed Data

14 IWA: Breakthrough Technologies for Extreme Performance Extreme Compression Required because RAM is the limiting factor. Row & Columnar Database Row format within IDS for transactional workloads and columnar data access via accelerator for OLAP queries. Multi-core and Vector Optimized Algorithms Avoiding locking or synchronization In Memory Database 3 rd generation database technology avoids I/O. Compression allows huge databases to be completely memory resident Predicate evaluation on compressed data Often scans w/o decompression during evaluation Frequency Partitioning Enabler for the effective parallel access of the compressed data for scanning. Horizontal and Vertical Partition Elimination. Massive Parallelism All cores are used within used for queries 14

15 How Fast is IWA? Columnar In-Memory Analytics with Unprecedented Performance 15

16 IBM Informix editions: New Value-Added Software Bundles 16

17 Informix Warehouse Accelerator (IWA) CUSTOMERS AND PARTNERS 17

18 Some IWA Customers by Sector: Retail, Government, Transportation, E&U 18

19 Some IWA Customers by Sector: Telecommunications, Insurance, Financial, IT Services 19

20 Real-Life Productive IWA in Government Agency in LATAM 20

21 IWA Architecture and Use (same Government Agency in LATAM) Sources/OLTP IDS > FCx AIX on pseries BI Tools ETL: Consolidate and aggregate data in IDS to later source and build MSAS cubes ETL IWA FCx Linux x86_64 IWA Real-time Analytics Cubes built in Microsoft (MSAS) 37x to 456x faster!! 21

22 Europe s Largest Power company tackles the Smart Meter Big Data challenge with Informix TimeSeries + In-Memory Accelerator (IWA) E.ON Metering (EMTG) is the centre of excellence for the development and commercialization of smart energy solutions and technologies and part of Europe s largest Power and Gas company E.ON EMTG operates a sophisticated Smart Meter data infrastructure based on IBM Informix TimeSeries technology in combination with Informix In-Memory Warehouse Accelerator IBM Information Management products currently used: Informix Ultimate Warehouse Edition Cognos Business Intelligence 10 22

23 Some of our IWA Business Partners 23

24 Informix Warehouse Accelerator (IWA) REFERENCE ARCHITECTURE & COMPETITION 24

25 IBM Informix Presentation 12.1 Template Full Version Current Customer BI Architecture with Informix Prod A Prod B Prod C E T L Informix DW E T L Data Mart 1 Data Mart 2 Data Mart 3 BI Prod D Prod E Sun T3 Solaris IUE v11.50 Intel servers MS Windows MS SQLserver Cubes IUE : Informix Ultimate Edition Source If Applicable 25

26 IBM Informix Presentation 12.1 Template Full Version Target Referenced Architecture with IWA Prod A Warehouse Accelerator Prod B Prod C E T L Informix DW BI Prod D Prod E Sun T3 Solaris IUWE v11.70 IUWE : Informix Ultimate Warehouse Edition Source If Applicable 26

27 IWA s Industry Positioning and Competitors DW Appliance DataAllegro (Microsoft) Dataupia Greenplum (EMC) Kognito Netezza (IBM) Columnar Database Calpont Exasol Infobright ParAccel Sand Technology Vertica (HP) Sybase IQ (SAP) In-Memory OLAP Tools QlikTech/QlikView Applix TM-1 (IBM-Cognos) Exalytics (Oracle) PALO In-Memory Data Warehouse HANA (SAP) IWA (IBM) 27

28 Informix Warehouse Accelerator (IWA) ROADMAP & NEW FEATURES 28

29 IBM Informix Warehouse/Analytics Present Roadmap Informix xC2 IWA 1st Release On SMP Informix xC3 Workload Analysis Tool More Locales Data Currency Informix xC4 IGWE IWA on Blade Server Informix xC5 Partition Refresh Load from Secondary Solaris on Intel Informix xC6 Partition Refresh Load from Secondary Solaris on Intel Informix xC7 Partition Refresh Load from Secondary node in Cluster Solaris on Intel Informix xC1 Bundled Cognos BI & SPSS Automatic incremental refresh Trickle feed (continuous refresh) Accelerate new SQL & OLAP queries Admin IWA using OAT & built-in functions Right/Real-Time In-Memory Analytics Big Data on Sensor data (TimeSeries+IWA) Informix xC2 Coming soon

30 Summary: Key Value Propositions for the Informix Market State-of-the-art query accelerator for current OLTP customers Ideal for an embedded database in a single machine environment Leverage partner-based solutions & sales Fully compatible with existing Informix architecture Deploy in HA configurations Integration with OAT Ideal for SMB Commodity based hardware, configurable memory size, no expensive interconnect required Low cost entry, scaling via cluster Appliance not always a fit Seamless integration with TimeSeries for Big Machine data

31 Informix12.10:SimplyPowerful 28

32 (CRANFORD,SC) SAPs SAPs 7.0 UNICODE SAP ECC 6.0 Basic 3850 (Paxville, DC) 3850 (TULSA DC) 3850 (TIGERTON QC) 3850 DUNNINGTON 3850 x5 (NEHALEM- (HC) EX, 8C) Informix 12.1 IWA 12.10: Highlights of New Features and Benefits IWA HW & SW Breakthrough Technology Innovations Multi-Core Parallelism & Intel 64 SIMD tech. Massive Parallel Scaling for Loads & Queries In-Memory storage & query Fast Storage Backup for Recoverability Row and Column storage Compression & query processing on compressed data Intelligent Partitioning No Aggregate Tables, No Indexes Insert Only on Delta SAPs Extreme performance (10x up to 200x faster complex queries) using Low cost commodity HW, transparent integration with Informix ORDBMS In 12.10, we made IWA even more accessible by providing: Automatic incremental (partition-level) refresh and trickle feed (continuous refresh) Support for smart sensors/meters data (Time Series) Big Data (on sensors data) Additional SQL capabilities for common OLAP queries Integrated administration via Open Admin Tool (OAT) and SQL API functions

33 IBM Informix 12.10: Performance for Real-Time Operational Analytics and Big Data on Sensor Data NEW! IWA Support for UNION queries and additional SQL support NEW! Much faster operational analytics and enhanced OLAP capabilities Enhanced integration with Cognos for much faster Cognos BI IDS and IWA support for OLAP functions and windowed aggregates 33

34 IBM Informix 12.10: Performance for Real-Time Operational Analytics and Big Data on Sensor Data NEW! Trickle Feed (Continuous Refresh) Automated, continuous updates/refresh of changed data from Informix into IWA for speed-of-thought analysis of real-time data ifx_setuptricklefeed Tracks changes in Dimensions Tracks inserts in Fact tables Automated updates in IWA datamart 34

35 IBM Informix 12.10: Performance for Real-Time Operational Analytics and Big Data on Sensor Data NEW! Automatic Partition-Level Refresh NEW! IWA administration through OpenAdmin Tool (OAT) and SQL API functions 35

36 IBM Informix 12.10: Performance for Real-Time Operational Analytics and Big Data on Sensor Data NEW! Informix TimeSeries + Informix Warehouse Accelerator Integration Provides real-time analytics of stored sensor data (Big Data for sensors/meters) 36

37

38